5 Metrics Engineering Managers Can Extract from Pull Requests

Many Engineering Managers promote Pull Requests as part of the development workflow. It’s a consolidated practice that brings . It consists of comparing the changes of a branch with the repository’s base branch (conventionally called master).Pull Requests provide useful and actionable metrics. However, following the wrong metrics cause distortions and bring more disadvantages than benefits. In this blog post, I list some metrics I consider useful.

Measuring process, not individuals

Before getting into the metrics, I want to make a disclaimer: don’t use these numbers to compare individuals.Sometimes a hard-to-find bug requires a single line of code to be fixed, and that single line took a week of work. I’ve witnessed it many times in my career.I’ve also witnessed Engineering Managers encouraging developers to open pull requests with too many changes that it was unpractical to review. They usually reinforce that by telling everyone these developers are productive, that they are doing the hard job when others are taking the easiest ones.Measuring individuals through pull requests may even be unfair. A developer dedicated to maintaining a legacy codebase tends to be slower than another one, which works on a greenfield project.That’s why measuring Pull Requests is tricky. Engineering Managers can’t use pull request data to asses individuals. If you do pull requests, then you want your team to collaborate. In this practice, collaboration is the core value. Developers’ effort can’t be measured just by how many pull requests are open or merged. Even worst, effort does not represent its size.However, managers can use pull requests metrics to understand the team dynamics and act appropriately to correct behaviors before things get out of track. Here are 5 suggestions of what to measure:

- Time to Merge
- Pull Request Lead Time
- Pull Request Size
- Pull Request Flow Ratio
- Pull Requests Discussions

Pull Request Lead Time

The lead-time metric gives you an idea of how many times (usually in days) pull requests take to be merged or closed. You can calculate lead time by repository or considering all repositories of the organization. The latest option gives you a clearer idea of your team dynamics.To find this number, you need to track every pull request. Save the date and time for each pull request when opened, and then, when it’s merged, save it too. The formula is easy. Just perform a simple average for the difference of dates.This metric is especially useful for raising questions and start investigations before it’s too late. A good practice is to measure this number over time so that you can spot trends and behaviors more pragmatically.

Time to Merge

In general, pull requests are open with some work in progress, which means that measuring pull request lead time doesn’t tell you the whole story.

Time to Merge is how much time it takes for the first commit of a branch to reach master. In practice, the math is simple. It’s the timestamp of the oldest commit of a branch minus the timestamp of the merge commit.

Comparing these two metrics (pull requests lead time and time to merge) can be surprising. Let’s see an example.In the above scenario, a pull request is taking an average time of three days to be merged. It is pretty good. But the Time to Merge is 15 days. It means that developers work an average of 12 days (15 – 3) before opening a pull request. It means a developer opens a pull request once in two weeks. I definitely would investigate why.Let’s see another example:In this case, developers open one pull request by week (13 – 8 = 5 days), much better than the previous example. However, pull requests are taking eight days to be merged or closed. There are tons of reasons why the lead time is that high. The number is not telling you the reason, but it indicates where the problem may be.

Pull Request Size

Time to Merge can vary, and the causes are many. The average pull request size is one of them. Depending on how many lines of code changes, it requires more or less effort to review.Ironically, developers tend to merge long pull requests faster. People get kind of lazy to perform thorough reviews when there are too many things going on. So, they immediately approve changes.There are two problems here: your time to merge goes up, and your quality goes down.Short pull requests are usually particular. They add a single feature, fix a single bug, or improve a little part of the codebase. They are very straightforward. It makes developers to review with attention and guided by a goal, lessening pull request lead time, and time to merge. It also mitigates the risks of bugs or security breaches.

The Pull Request Size is the average of total lines of code added plus the total lines of code removed. The higher the number, the bigger the pull request is.

Pull Request Flow Ratio

The Pull Request Flow Ratio is the sum of the opened pull request in a day divided by the sum of closed pull requests in that same day. This metric shows whether your team works in a healthy proportion.

Merging pull requests and deploying to production is a good thing. It adds value to the final user. However, when your team closes more pull requests than opens, soon pull request queue starves. It means there may be a hiatus in the delivery.On the other hand, the more pull requests are open, the higher the probability of increasing its lead time. Creating a task force to review and merge them in batches is the usual practice I have seen in the market. Sooner or later, a hiatus in delivery emerges.You need to make sure your team merges pull requests in a ratio as close as they open, the closer to 1:1, the better. That’s why to keep an eye on the ratio is essential. A healthy pace makes your flow to be smoother and more predictable. The challenge here is to deliver value to the final user consistently.

Pull Requests Discussions

When I say discussion, I mean the number of comments and reactions per pull request. On the contrary to social media posts, too much engagement in pull requests leads to inefficiency, and I show you why.

Measuring the number of comments and reactions for each pull request gives you an idea of how your team collaborates. Collaboration is great, and as leaders, we want to endorse it. However, after a certain level, discussions slow down development.Some pull requests require more attention than others, and the team must contribute to the threads. But if it happens too often, there may be something wrong. Maybe your team is not aligned, or maybe the software requirements are not precise enough. Misalignment discussions are not collaboration. It is time waste.In the opposite scenario, having almost zero engagement means code review is not part of the team’s habits. It’s your responsibility to foster this culture.In summary, you need to find an ideal number based on your team size and distribution. It can’t be too much, and can’t be too little. More importantly, you need to act on pull requests with useless discussions to keep your team productive.

Conclusion

Engineering Managers can extract lots of useful metrics from pull requests. In this blog post, I shared five metrics extracted from Pull Requests and how they help engineering managers to understand how their team works. Although they are pretty simple, they give you visibility of your team’s dynamics. That’s crucial for acting before problems arise.