Libraries.io February Progress Update

Continuing from last month with the regular progress updates on Libraries.io since securing a grant from both the Sloan Foundation and , here’s what we got up to in February 2017.

BitBucket and GitLab support

The biggest feature to ship was support for finding and parsing dependency information from source code repositories hosted on GitLab and Bitbucket. Previously Libraries.io could only pull in information about GitHub repositories and it’s taken a few months to untangle the hard coded GitHub logic throughout the application. We’re slowly spidering both services and so far have around around 300,000 repositories on GitLab and Bitbucket, this helps give a much better picture of the whole open source ecosystem rather than just the one on GitHub. The large refactoring to get that shipped also means in the future we can add support for other hosts like , and self-hosted GitLab repositories much more easily.

Other Notable Features

During the great repository refactoring, we also made switched the approach to parsing dependency information we were taking. Previously we had a separate node.js web service call , which the background workers interacted with over http, but now we’ve switched to using the ruby library for parsing dependencies, which runs directly in the background workers. With this we reduced the amount of different web services we needed to run, simplified a whole lot of code and everything is in one language, this also means that Libraries.io is now sharing it’s parser with , so an improvement to the library benefits both services. We’ve started to codify the infrastructure and server configuration that Libraries.io with Ansible, it’s all being done in the open on the new infrastructure repository: I also rolled out a new small service to replace our usage of the npm rss feed, which was pretty terrible, instead directly follows the changes feed and produces a list of recently updated project names for to consume every 60 seconds.

Ben has been working to improve Libraries.io’s documentation throughout February. He re-wrote much of the old, developer-centred documentation to make Libraries.io a more welcoming project for designers, copywriters, and users alike. You can check it out at A big part of this has involved re-writing our and the templates we use for and s, which we’ve rolled out to all the repositories in the Libraries.io organisation. In line with our improvements to documentation we’ve also improved our ticketing. Inquisitive, would-be contributors can now find issues to pick up by size, skill or type and can easily find issue to help others with. We even have a label for first-time contributors, explanations of each label can be found on the . And finally, in return for financial support we made a number of commitments to the Alfred P. Sloan Foundation and, as of January, the Ford Foundation. You can now see those applications and our commitments to these institutions in our new . A full list of all changes right across the Libraries.io org on GitHub is available in , generated by:

Unseen Open Source Infrastructure

At the start of February I wrote a post about , along with an showing the most depended upon repositories with very few stars. The blog post went down well and has helped kicked off a number of conversations around how we can use Libraries.io data to pinpoint important projects that might need more support.

Statistics

42,594 new libraries and 265,766 new versions found, bringing the total to 2,154,734 libraries indexed. 701 commits, 33 pull requests and 220 issues opened across all Libraries.io repositories on GitHub: Libraries.io has also discovered over 20 million open source repositories from GitHub, GitLab and Bitbucket.

Contributors

This month we had a quite a few new contributors whom we would like to thank:

Plans for March

The first week of March we’ll be continuing our documentation push, getting involved in , Ben has written more about that . After that I’ll be focusing on beefing up the postgres and elasticsearch infrastructure and refactoring the organisation/user mapping to be less GitHub specific and bring in support for GitLab and Bitbucket Users and Teams. As always, follow us on Twitter at , and for more updates.