Ask HN: How do you assess developer productivity?
32 comments
It’s the ultimate mistake people keep making about programmers’ work.
That it’s linear or at least can be approximated with some linear formula, even if multidimensional.
The question itself implies that there’s some linear “productivity” scale where you can put all your developers and see who’s more and who’s less productive.
Because of this fallacy still dominating management minds - we developers can for example hardly justify 2x, 3x salary raise, even though we realize sometimes that our impact and value is as much higher relative to some peers.
The brutal truth is that the contribution of a developer to any complex and big enough project is rarely proportional to any measurable metrics. And barely ever can be reliably connected to such a metrics.
developer A can just sit around few days and invent simple yet scalable decision with just 1 PR, while developer B would be writing thousands of lines of a good looking code.
But this great looking and even well documented code might kill the whole product at some point in near future, even because of some new requirements that nobody could expect (except that developer A, who wasnt promoted for bad “productivity” and quit).
Add to this the fact that often you can’t really know who was good and who was the bad developer - at the moment both had good arguments.
So, my point is that big software products are essentially non linear, dynamic and poorly predictable beasts.
I don’t have ready answers on “how to decide who’s gonna be promoted”.
But i feel the farther you go from trying to “engineer” and “deeplearn” this issue and closer to trusting intuitions of experienced dev leaders, to treating developers as people not machines, considering each project as a unique thing, with unique approaches — the closer you to the truth and to real improvements.
That it’s linear or at least can be approximated with some linear formula, even if multidimensional.
The question itself implies that there’s some linear “productivity” scale where you can put all your developers and see who’s more and who’s less productive.
Because of this fallacy still dominating management minds - we developers can for example hardly justify 2x, 3x salary raise, even though we realize sometimes that our impact and value is as much higher relative to some peers.
The brutal truth is that the contribution of a developer to any complex and big enough project is rarely proportional to any measurable metrics. And barely ever can be reliably connected to such a metrics.
developer A can just sit around few days and invent simple yet scalable decision with just 1 PR, while developer B would be writing thousands of lines of a good looking code.
But this great looking and even well documented code might kill the whole product at some point in near future, even because of some new requirements that nobody could expect (except that developer A, who wasnt promoted for bad “productivity” and quit).
Add to this the fact that often you can’t really know who was good and who was the bad developer - at the moment both had good arguments.
So, my point is that big software products are essentially non linear, dynamic and poorly predictable beasts.
I don’t have ready answers on “how to decide who’s gonna be promoted”.
But i feel the farther you go from trying to “engineer” and “deeplearn” this issue and closer to trusting intuitions of experienced dev leaders, to treating developers as people not machines, considering each project as a unique thing, with unique approaches — the closer you to the truth and to real improvements.
It's best to give up the idea of measuring software productivity entirely. Almost no one would seriously wonder how to measure artistic productivity and software engineering is a lot more like art than most people realize because it is fundamentally a creative activity.
Software is also too abstract to be measured the same way assembly line productivity is measured so there is no point in trying to impose a metric based productivity framework on top of software engineering. It's unlikely any such effort will be successful or will lead to any positive outcomes.
Software is also too abstract to be measured the same way assembly line productivity is measured so there is no point in trying to impose a metric based productivity framework on top of software engineering. It's unlikely any such effort will be successful or will lead to any positive outcomes.
It's best to give up? Almost no one would seriously wonder?Software is also too abstract?
Developer productivity questions will always occur in the minds of the people signing the front of the paycheck.
Developer productivity questions will always occur in the minds of the people signing the front of the paycheck.
Then it's best to disabuse them of such notions. If they want a consistent product that is produced by the operation of an assembly line then they should invest in factories instead of software and software engineers.
Any large tech company has teams dedicated to improving developer productivity. So how do you assess the quality of those teams' work, or that they're worth the investment, or help them prioritize projects? Just don't bother?
Do you have an example of something these teams have developed that has actually made software engineers more productive?
No - I've worked with several teams with "productivity" right in the name, and to be honest, I'm usually skeptical when these teams claim that their work will "improve productivity" and am interested in how to evaluate their claims. If we just give up completely because the space is hard to measure, there's no check on these teams that keeps them actually working on important things, vs. making changes that they subjectively prefer.
Software engineering is a form of process automation which by its nature is a form of productivity improvement. So if some teams are not streamlining and consistently improving business processes then asking another team to help them do so with more software is kinda nonsensical.
For a while I worked on build tools and CI pipelines and I can honestly say that I never improved anyone's productivity. Most of my work was figuring out how to remove performance bottlenecks that unwitting software engineers would invariably add to the build and test process because a manager was breathing down their neck about delivering some feature yesterday. If business processes are dysfunctional then that's not a software productivity problem and can not be fixed by adding more software.
For a while I worked on build tools and CI pipelines and I can honestly say that I never improved anyone's productivity. Most of my work was figuring out how to remove performance bottlenecks that unwitting software engineers would invariably add to the build and test process because a manager was breathing down their neck about delivering some feature yesterday. If business processes are dysfunctional then that's not a software productivity problem and can not be fixed by adding more software.
Can you define developer productivity? Before you can measure something you need to have a definition. Measuring PRs and commits is like measuring a writer’s productivity by how many sheets of paper they use.
A useful metric might be derived from business value added vs. cost of programming effort, but then how well can the organization measure business value? Or the cost of programmer effort?
A useful metric might be derived from business value added vs. cost of programming effort, but then how well can the organization measure business value? Or the cost of programmer effort?
Writers do measure productivity by pages written… sure, they may not write a best seller, but there’s a date they start writing and a date the book is published and the book has a word count.
I agree though that lines of code or number of pages doesn’t have much of a relationship to revenue.
I agree though that lines of code or number of pages doesn’t have much of a relationship to revenue.
Pages and word count are after-the-fact measures and don’t have a relationship to either perceived value or productivity. Nor does the time it takes to write a book. Schedule and length may be imposed by an editor or publisher, just like programmers get from managers.
Code that adds business value (or reduces business costs), delivered within schedule and budget, is what management looks for. But is that What we mean when we programmers talk about productivity? I don’t think so.
Code that adds business value (or reduces business costs), delivered within schedule and budget, is what management looks for. But is that What we mean when we programmers talk about productivity? I don’t think so.
Writing is a bad analogy for programming.
In some respects, yes. But both are creative and technical processes that show huge variance in quality and value across writers and programmers. No analogy is perfect but if we intend to measure programmer productivity we can start by looking at how that’s measured in similar activities.
I only glance at quantitative things like once a year come review time. Talking with people is a much better way to assess their performance. Usually people are hardest on themselves. Everyone is different, has different life circumstances and performs differently.
In general I think the premise of the question is wrong. It's taking a factory building widgets approach to a creative process. You'd have to have standardized measurable inputs like product requirements that can be worked by anyone on the team. Everyone on the team has to have the same level of experience with the technology used so any learning curve is the same for everyone. Proper standardized estimation needs to be done on every piece of work and then measure the time used to do the work. This is like CMM Level 5 type of utter nonsense, in my opinion.
Also, its more often the case that the inputs (requirements) shift and change as they get worked on which will not be reflected in any quantitative way.
In general I think the premise of the question is wrong. It's taking a factory building widgets approach to a creative process. You'd have to have standardized measurable inputs like product requirements that can be worked by anyone on the team. Everyone on the team has to have the same level of experience with the technology used so any learning curve is the same for everyone. Proper standardized estimation needs to be done on every piece of work and then measure the time used to do the work. This is like CMM Level 5 type of utter nonsense, in my opinion.
Also, its more often the case that the inputs (requirements) shift and change as they get worked on which will not be reflected in any quantitative way.
There is no quantitative way to do it as any measure will be gamed or will not be representative of all the dimensions of what makes a successful project.
High performing teams that I've been on rely on developers to meet their own given estimates and effectively communicate. It is pretty obvious when someone is repeatedly dodging as they will be unable to convincingly explain their status to their own team members.
Basically, each team member holds each other accountable. This only works if they have strong, shared incentives.
High performing teams that I've been on rely on developers to meet their own given estimates and effectively communicate. It is pretty obvious when someone is repeatedly dodging as they will be unable to convincingly explain their status to their own team members.
Basically, each team member holds each other accountable. This only works if they have strong, shared incentives.
Lines of code. Yes, it's easy to game. Don't use it for bonuses.
Yes, it varies by person and how they close their brackets, how they write comments, whether they use functional, or how they break up formulas. Compare a person to themselves.
Coding is an iterative process. Write stuff. Stick placeholders everywhere. Remove stuff and replace it with better stuff. LOC measures the stuff.
Does it measure architecture? The biggest argument for architecture is that it slows things down at first, but speeds it later. Great, measure LOC for both stages.
If you see people writing 1000 LOC/day early then slowing to 50, that's probably underengineering or overengineering. It should be more or less steady, slowing a bit when they get to the edges of the familiar code and need more R&D, then speeding up again.
Your devs spent some time documenting and refactoring? How's LOC rate after that? You added 3 newbies to the team? How does the LOC change for the seniors training them? Do the meetings discussing code increase or decrease LOC? How much does No Meeting Day or remote work change productivity? How effective are 4 day weeks or days when everyone goes home at 5?
It's not perfect, but it works for many of the things you'd like to measure, as long as you manage the downsides. If you're trying to reward productivity, it throws it all off.
Yes, it varies by person and how they close their brackets, how they write comments, whether they use functional, or how they break up formulas. Compare a person to themselves.
Coding is an iterative process. Write stuff. Stick placeholders everywhere. Remove stuff and replace it with better stuff. LOC measures the stuff.
Does it measure architecture? The biggest argument for architecture is that it slows things down at first, but speeds it later. Great, measure LOC for both stages.
If you see people writing 1000 LOC/day early then slowing to 50, that's probably underengineering or overengineering. It should be more or less steady, slowing a bit when they get to the edges of the familiar code and need more R&D, then speeding up again.
Your devs spent some time documenting and refactoring? How's LOC rate after that? You added 3 newbies to the team? How does the LOC change for the seniors training them? Do the meetings discussing code increase or decrease LOC? How much does No Meeting Day or remote work change productivity? How effective are 4 day weeks or days when everyone goes home at 5?
It's not perfect, but it works for many of the things you'd like to measure, as long as you manage the downsides. If you're trying to reward productivity, it throws it all off.
I'm putting feelers out for a tool I'm building in the same vein: https://bldbtr.tech/.
What makes engineering metrics hard is that every team is different. We can talk about cycle time, deploy frequency, bug counts, but not every team can pull the same levers and those levers don't always yield the same results. So the teams and the manager need to know how to create metrics that matter and metrics that the team has the ability to affect.
You also need to take into account what is organizationally important. I've been in startups that valued deployments more than quality and enterprises in compliance heavy industries that valued quality over deployments. If your metrics don't align with your culture, it never feels right.
Then the biggest thing is to not compare teams. This is a game of "better than yesterday", not a competition. As a team, you want to find what equilibrium is and use your retro to find ways to iterate. As you have more discussions around performance, you'll find more things to measure, you just need to decide if it's worth it.
What makes engineering metrics hard is that every team is different. We can talk about cycle time, deploy frequency, bug counts, but not every team can pull the same levers and those levers don't always yield the same results. So the teams and the manager need to know how to create metrics that matter and metrics that the team has the ability to affect.
You also need to take into account what is organizationally important. I've been in startups that valued deployments more than quality and enterprises in compliance heavy industries that valued quality over deployments. If your metrics don't align with your culture, it never feels right.
Then the biggest thing is to not compare teams. This is a game of "better than yesterday", not a competition. As a team, you want to find what equilibrium is and use your retro to find ways to iterate. As you have more discussions around performance, you'll find more things to measure, you just need to decide if it's worth it.
Either you end up counting things that are poor indicators of performance (points, lines of code, commits, etc), or it's completely subjective.
My company basically uses the subjective, using the opinions of 3-4 people. Although some break policy and do things like point counting anyways.
My company basically uses the subjective, using the opinions of 3-4 people. Although some break policy and do things like point counting anyways.
Doesn't seem to me that you can really measure productivity at all. How do you compare : (a) dev that writes tons of code and makes things over complex and not maintainable (b) dev that writes hardly any code, procrastinates, but their code is v high quality, simple, has no problems (c) dev that cranks out a lot of code but is horrible to work with and very good teammates quit (d) dev whose code output isn't high but that's cos they do tons of useful mentoring leading to other people's output being good
Most methods are poor. PRs and commits can easily be gamed and generally the best developers are the ones with succinct small changes that deliver high value.
Two ways I have found to measure a developer's productivity are...
1. Based on the feedback of their peers. It helps to know the team members and the team's dynamics though.
2. Based on the $$$ the team bring in. Depending on how the team operate this could be far from their control though, a lot of decisions about the product will be out of the developer's hands.
Two ways I have found to measure a developer's productivity are...
1. Based on the feedback of their peers. It helps to know the team members and the team's dynamics though.
2. Based on the $$$ the team bring in. Depending on how the team operate this could be far from their control though, a lot of decisions about the product will be out of the developer's hands.
Look for track record of deliverables and adjust for human energy. If you see a good track record of things getting done, and then see a drop off, adjust for human energy. Humans can’t be at 100% all the time.
If you measure clinically, then you disrespect what a human being is. You can’t say ‘well you never missed a deadline and now you are, so you must suck or have given up’, this is a really bad thing to do. Empathy is key here.
In other words, you cannot put a number on this, I’m sorry. The eye test is crucial here versus the data.
If you measure clinically, then you disrespect what a human being is. You can’t say ‘well you never missed a deadline and now you are, so you must suck or have given up’, this is a really bad thing to do. Empathy is key here.
In other words, you cannot put a number on this, I’m sorry. The eye test is crucial here versus the data.
If you're more interested in team performance metrics, read Accelerate. The primary metrics from Accelerate that I use are Deployment Frequency and Lead Time.
> As an experiment we’ve starred building an app that measures things like PRs and commits for each developer.
Ever heard of perverse incentives? This repeats errors made in the early 80s.
Ever heard of perverse incentives? This repeats errors made in the early 80s.
This 100%. Also, to the OP, what goal do you want the engineers to achieve? More lines of code? That means more bugs and sometimes it means a worse implementation. More PRs? Well that’s just dumb.
You want engineers to solve business problems. Sometimes that involves code, sometimes it doesn’t. Can you quantify and qualify “solutions to business problems”? Do you want solutions that bring in thousands of dollars or millions? Over what time? Requiring how many resources to achieve? Etc etc.
I hope I don’t sound hostile, it’s just that, when done poorly, the kinds of solutions you mentioned can literally bankrupt a business. It also shows a gross misunderstanding of how incentives work, and what you’re wanting to achieve.
You want engineers to solve business problems. Sometimes that involves code, sometimes it doesn’t. Can you quantify and qualify “solutions to business problems”? Do you want solutions that bring in thousands of dollars or millions? Over what time? Requiring how many resources to achieve? Etc etc.
I hope I don’t sound hostile, it’s just that, when done poorly, the kinds of solutions you mentioned can literally bankrupt a business. It also shows a gross misunderstanding of how incentives work, and what you’re wanting to achieve.
Goodhart's law - https://en.wikipedia.org/wiki/Goodhart%27s_law
> When a measure becomes a target, it ceases to be a good measure.
> When a measure becomes a target, it ceases to be a good measure.
I'm gonna live by that from now on.
Then make an OKR that you’ll say it to yourself everyday for 91 days.
How often do goals get met, on time? If you can rely on estimates, and get good results, you've got a productive developer. The one time I managed someone as a programmer, I really, really lucked out, and he was great at it. We would discuss something that had to be done on Tuesday, I might get a question or two, otherwise nothing until he showed it to me done on Friday, and it was always up to snuff.
my general metric is, per task:
( for complexity_points i use https://ciandt.com/us/en-us/complexitypoints )
( complexity_points - bugs_reported ) / total_time
( for complexity_points i use https://ciandt.com/us/en-us/complexitypoints )
As an experiment we’ve starred building an app that measures things like PRs and commits for each developer. I’m not sure this is the best way to solve this problem. I’m curious how engineering managers philosophically think about measuring the performance of their team and if engineering managers are using any quantitative measures how do they do it? How do high performing software teams measure performance?