Ask HN: How do you measure eng team productivity?
17 comments
I have a better idea, measure how good you are at turnover. How negative is it when someone leaves? How easy is it for someone to join and be productive?
Well, specifically, you can measure onboarding, which is a proxy:
How quickly can someone get started on a feature and ship it to production? How obvious is it for them to follow your teams procedures, guardrails, tests, review processes, etc and go?
Maximizing this means maximizing the available colleagues that can work on your codebase and sets up a positive culture of enablement. It means maintaining your code is not an inscrutable mess.
Well, specifically, you can measure onboarding, which is a proxy:
How quickly can someone get started on a feature and ship it to production? How obvious is it for them to follow your teams procedures, guardrails, tests, review processes, etc and go?
Maximizing this means maximizing the available colleagues that can work on your codebase and sets up a positive culture of enablement. It means maintaining your code is not an inscrutable mess.
The organisation is like a complex organism, and engineering is merely piece of the whole. Each piece contributes to and depends on each other (roughly) to make a business function. Therefore, the word "engineering productivity" by necessity should be put under the category of "intangibles" (in most cases).
Still, we can consider some interesting thought experiments. Say, you start with a purely technical team; there are no marketers, no HR, no managers, etc, and the team gets a product going in the market. In such a limited case, one can characterise productivity with "revenue per engineer" (essentially - total revenue / number of engineers).
However, in reality, if the marketers, the HR, and various managers, leaders and salesman don't do their jobs, despite having a great product, revenue will approximate towards zero overtime. In reverse, if the marketers merely do their job perfectly, but there are no engineers delivering the product, again there won't be any profits.
Despite all these caveats, let me share why I think "revenue per employee" seems like the most useful metric here. Consider the "average revenue per employee" for these two companies:
ADP: $284,453
Apple: $2,560,571
Google: $2,020,329
Apple/Google are 10x ADP. I'd argue a good part of this 10x difference we see in the output primarily due to the differences in engineering sophistication. The engineers in these more profitable orgs are doing qualitatively different "types" of engineering (hardware? prototyping? deep research? higher standards?). Increasing engineering prowess, trying more sophisticated projects seems like the biggest boost to the bottom line.
Still, we can consider some interesting thought experiments. Say, you start with a purely technical team; there are no marketers, no HR, no managers, etc, and the team gets a product going in the market. In such a limited case, one can characterise productivity with "revenue per engineer" (essentially - total revenue / number of engineers).
However, in reality, if the marketers, the HR, and various managers, leaders and salesman don't do their jobs, despite having a great product, revenue will approximate towards zero overtime. In reverse, if the marketers merely do their job perfectly, but there are no engineers delivering the product, again there won't be any profits.
Despite all these caveats, let me share why I think "revenue per employee" seems like the most useful metric here. Consider the "average revenue per employee" for these two companies:
ADP: $284,453
Apple: $2,560,571
Google: $2,020,329
Apple/Google are 10x ADP. I'd argue a good part of this 10x difference we see in the output primarily due to the differences in engineering sophistication. The engineers in these more profitable orgs are doing qualitatively different "types" of engineering (hardware? prototyping? deep research? higher standards?). Increasing engineering prowess, trying more sophisticated projects seems like the biggest boost to the bottom line.
I worked in a company who measured productivity by attributing a score based on the Fibonacci scale. But they were doing this when the feature was done.
By doing that, it was fairly easy to give a relevant score to each feature.
Engineering team was doing this to measure solely engineering productivity, but product team could do this as well to measure added value to the company.
By tracking the start/end date as well for each feature, you can then measure productivity at any time scale you want.
By doing that, it was fairly easy to give a relevant score to each feature.
Engineering team was doing this to measure solely engineering productivity, but product team could do this as well to measure added value to the company.
By tracking the start/end date as well for each feature, you can then measure productivity at any time scale you want.
That’s very interesting, what is the score based on?
It was supposed to represent "how big" the feature was.
As it's subjective and vague, the score was suggested by the engineer in charge of the feature and then (rapidly) approved by the team during daily meetings.
But I guess this principle could be applied with a score representing whatever you want to track, and however you want to track it.
As it's subjective and vague, the score was suggested by the engineer in charge of the feature and then (rapidly) approved by the team during daily meetings.
But I guess this principle could be applied with a score representing whatever you want to track, and however you want to track it.
Why do you want to measure productivity?
You already know where you are. A measurement doesn’t help your team ship better or faster.
If you’re shipping product improvements at a regular cadence, and the customers are happy, then there’s no problem with productivity.
If you’re shipping poor quality, identify why and improve that.
If you’re shipping so slow that it’s negatively impacting customers, identify why and improve that.
Shoot me an email, I’d be happy to offer more perspective or advice.
You already know where you are. A measurement doesn’t help your team ship better or faster.
If you’re shipping product improvements at a regular cadence, and the customers are happy, then there’s no problem with productivity.
If you’re shipping poor quality, identify why and improve that.
If you’re shipping so slow that it’s negatively impacting customers, identify why and improve that.
Shoot me an email, I’d be happy to offer more perspective or advice.
Interesting to note that two of your decision points are on "shipping product improvements at a regular cadence" or "shipping so slow that it's negatively impacting customers". Both relate to a productivity discussion, ie "how to define what it means to 'ship' as an engineering team". In that sense, we're aligned on why it's important to measure productivity.
It would however be a mistake to try to optimize for maximum productivity in a vacuum, but note that this is not what this thread is about. This thread is about the unit of measurement that people use in their teams. In that sense it's purely a research / curiosity question
The material output of programmers is code. In that sense, programmer productivity is (|additions| + |deletions|) / time. Economically, it’s (|additions| + |deletions|) / wages paid.
This is how I would quantitatively measure the productivity of members on the teams I work with. If you look at this measure across your team, I think you’ll find it roughly matches your intuitive notion of who the most productive programmers are. I also think you’ll find story size positively correlated with the magnitude of code change or time.
The thing is, measuring how fast programmers output software isn’t all that useful.
This is how I would quantitatively measure the productivity of members on the teams I work with. If you look at this measure across your team, I think you’ll find it roughly matches your intuitive notion of who the most productive programmers are. I also think you’ll find story size positively correlated with the magnitude of code change or time.
The thing is, measuring how fast programmers output software isn’t all that useful.
This feels like you're measuring lines of code as productivity, but most experienced engineers would disagree that lines of code is a good metric. Am I misunderstanding your assertion?
Productivity is the ratio of output to input.
Code is the output of a programmer. Time (or wages) is the primary economic input to software development.
Productivity is not a measure of whether something is worth doing. It says nothing about whether the code is valuable or worth writing. A factory has increased productivity if it has produced more widgets with the same inputs, even if nobody wants those widgets or those widgets pollute the environment.
Similarly, programmers do not choose what software to write. They are employed to write software someone else decides is valuable to the business. Their productivity is independent of whether the software has business value.
Consider a one line program. A programmer that produces that program in 5 minutes is more productive than one who produces it in 1 hour.
Stories with sizes are proxies for magnitude of code change or time. A larger project is larger because it takes more time or more code change, or both. Story size/points is an attempt to estimate this. So it is more accurate to measure the actual code change and time taken.
It’s true that the same program can be written with different amounts of code. But this is largely irrelevant at the organization level, where programmers are not writing multiple versions of same program in different languages and toolsets. A team is typically writing software within the context of a particular toolchain and runtime environment.
Code is the output of a programmer. Time (or wages) is the primary economic input to software development.
Productivity is not a measure of whether something is worth doing. It says nothing about whether the code is valuable or worth writing. A factory has increased productivity if it has produced more widgets with the same inputs, even if nobody wants those widgets or those widgets pollute the environment.
Similarly, programmers do not choose what software to write. They are employed to write software someone else decides is valuable to the business. Their productivity is independent of whether the software has business value.
Consider a one line program. A programmer that produces that program in 5 minutes is more productive than one who produces it in 1 hour.
Stories with sizes are proxies for magnitude of code change or time. A larger project is larger because it takes more time or more code change, or both. Story size/points is an attempt to estimate this. So it is more accurate to measure the actual code change and time taken.
It’s true that the same program can be written with different amounts of code. But this is largely irrelevant at the organization level, where programmers are not writing multiple versions of same program in different languages and toolsets. A team is typically writing software within the context of a particular toolchain and runtime environment.
I think, first principle is to build something that people want.
If you build this way, I feel more comfortable than building something fast, that at the end has close to zero usage.
2 - 7 eng dev teams is a small one. It means, just put people on top of things and things should develop faster than in a an average team in any corporation.
This approach/mindset might not work in corporations etc. since they want to have "reports"
If you build this way, I feel more comfortable than building something fast, that at the end has close to zero usage.
2 - 7 eng dev teams is a small one. It means, just put people on top of things and things should develop faster than in a an average team in any corporation.
This approach/mindset might not work in corporations etc. since they want to have "reports"
I agree 100%, you need to have a strong product foundation to turn good output into good business outcome. A productive team that delivers features no one wants is pointless. Both go hand in hand, but as far as engineering management goes, it is still a challenge to find a good productivity metric. What’s used in your organization @przeor and what do you think of it?
> However, anything that does not bring obvious value (bug fixing, refactoring) should be counted out of output in terms of productivity
Not counting refactoring as output makes sense, because it's more of repositioning for future output, and/or recreation.
But if fixing bugs doesn't bring value, why do you do it?
Not counting refactoring as output makes sense, because it's more of repositioning for future output, and/or recreation.
But if fixing bugs doesn't bring value, why do you do it?
Well, productivity measures how fast you deliver new value. Fixing a bug is about repairing value that you thought you had delivered but actually did not. Counting bugs as "value" would be double counting.
I understand your point, but by doing this you will probably end up with nobody in the team caring about fixing bugs.
That's a problem because sometimes fixing bugs brings more value than adding a new feature or optimizing something.
That's a problem because sometimes fixing bugs brings more value than adding a new feature or optimizing something.
> Fixing a bug is about repairing value that you thought you had delivered but actually did not.
In that case it's important to classify issues correctly. Email sending worked yesterday but not today because Office365 changed some SMTP requirements (say ciphers)?
Customer will call it a bug, but according to your definition it should be classified as a feature.
In that case it's important to classify issues correctly. Email sending worked yesterday but not today because Office365 changed some SMTP requirements (say ciphers)?
Customer will call it a bug, but according to your definition it should be classified as a feature.
Look at the SPACE framework, its meant as a super() to DORA that is focused on developer productivity vs. systems performance: https://queue.acm.org/detail.cfm?id=3454124
I’m looking for testimonies and knowledge: - how is productivity measured in your team if you’re an engineer? In different teams if you’re at a management level? Conversely, is it not measured and do you know why? - how is productivity measured in companies you know, either through firsthand experience or well-documented sources? (Company articles, books) - are you aware of research in this area?
On a last side note, I would count research displayed in Accelerate out of the productivity question. The DORA metrics don’t delve into the complex details of “what is value for my team” and focus on the release level. The books and research are great, but arguably not about engineering productivity.