How we use tennis rankings for fixing bugs(refactoring.substack.com)
refactoring.substack.com
How we use tennis rankings for fixing bugs
https://refactoring.substack.com/p/how-we-use-tennis-rankings-for-fixing
44 comments
Doesn't this optimize for those that work endless hours?
Hi, author here! I think it's a risk you have if your company culture encourages working long hours. We never did that tbh and people know we don't have that expectation.
We also didn't couple strong incentives (e.g. money) with this process, we always kept it at a "fun" level. It's been more than enough to keep people engaged
We also didn't couple strong incentives (e.g. money) with this process, we always kept it at a "fun" level. It's been more than enough to keep people engaged
You say that now but what about Mary dev who consistently pulls high effort tasks and does not finish them in the right timeframe because she goes home on time? When someone consistently shows up in the bottom of the chart because they work regular hours instead of overtime, it will be demotivating, cause your employees to resent those at the bottom for not working hard enough, and will creep into performance reviews.
I'm in the same boat as other posters, I am a hard no on this idea and would never subject my employees to it. When you build employee-facing metrics like this, you end up judging your employees by those metrics. Build smarter metrics.
ETA: "your company culture" is dictated by things like this. So the incentive it creates in your culture is to work longer hours so you can get to the top. This is literally building company culture, and trying to say you can avoid the negative incentive by having a better culture is missing the point.
I'm in the same boat as other posters, I am a hard no on this idea and would never subject my employees to it. When you build employee-facing metrics like this, you end up judging your employees by those metrics. Build smarter metrics.
ETA: "your company culture" is dictated by things like this. So the incentive it creates in your culture is to work longer hours so you can get to the top. This is literally building company culture, and trying to say you can avoid the negative incentive by having a better culture is missing the point.
I understand what you say — we never judged anyone on such metrics, but I agree I didn't do a great job at elaborating on that. Like you said, the metric is very simple, it's just an indication that is useful to understand bottlenecks, and to have some fun in a tight-knit startup team (that doesn't work long hours) where people don't like that much fixing bugs :)
Like others have said, I think the line between this being healthy and useful, and this becoming a nightmare is fine, and it gets probably harder the bigger the company is.
Thank you for your feedback anyway!
Like others have said, I think the line between this being healthy and useful, and this becoming a nightmare is fine, and it gets probably harder the bigger the company is.
Thank you for your feedback anyway!
> we never judged anyone on such metrics
I think you need to be aware and selective in who "we" is here. Do you mean management-"we", peer-"we", personal-"I"? The post you're replying to did a great job of calling out that all of these (and more) could have very different viewpoints on this.
I think you need to be aware and selective in who "we" is here. Do you mean management-"we", peer-"we", personal-"I"? The post you're replying to did a great job of calling out that all of these (and more) could have very different viewpoints on this.
Do the developers think it's fun? Or do they know the boss thinks it's fun?
tl;dr: You have effectively instituted a grading rubric for your entire engineering team, the way a manager grades their worst engineers. As a result, your entire team is now motivated to act in accordance with the way bad engineers typically perform.
---
I completely agree with the other responders.
At the BARE minimum, you could have changed this "leaderboard" to display only the top three, and coded the app in a way as to hide the remaining developers, as a way of celebrating the "best" developers this week, instead of ranking everyone on the team (and thereby punishing those on the bottom).
Your dev team is not a sales team.
If its your management team deciding task effort level, then you have compounding problems because they do not understand engineering impact. If it is your engineering team choosing engineering effort for each task, you still have created a perverse incentive.
If I implemented this sort of ranking on my team, BY FAR my best developers would consistently be on the bottom of the list. And this list, would in turn, motivate them to stop working on the hard ambiguous problems I need them to solve, and motivate them to steal as many items as possible that my junior engineers could otherwise have accomplished.
Even if my best engineers stuck to high-effort tasks (which they now are motivated not to do), not all high effort tasks are the same difficulty, and they now have a very strong incentive to only work on the easiest tasks in each difficulty level.
If I was on your development team, the introduction of this leadership board would be a Huge, Neon sign to me that code quality and engineering decision making no longer matters, all that matters for job security is writing as much as possible, as quickly as possible.
What's that? My poorly thought out code will result in numerous more bugs than if I had actually put thought into it to begin with? Well guess what? I'm the developer who best knows how to fix those bugs, so the more bugs I create now, the more I'll be able to fix next sprint too! Then I'll still be on top of the leaderboard!
Sure, you might have some internal processes to prevent this, but those are now on a timeline to deprecation as this leadership board comes to define your team culture. And even if your engineering team's internal culture is strong enough to overcome this blight, the game will simply become either how to best ignore this leadership board without upsetting management, or alternatively how to most act as described in the previous paragraph without getting caught.
This reeks of project management applied to engineering, without understanding the nuances of engineering.
---
I completely agree with the other responders.
At the BARE minimum, you could have changed this "leaderboard" to display only the top three, and coded the app in a way as to hide the remaining developers, as a way of celebrating the "best" developers this week, instead of ranking everyone on the team (and thereby punishing those on the bottom).
Your dev team is not a sales team.
If its your management team deciding task effort level, then you have compounding problems because they do not understand engineering impact. If it is your engineering team choosing engineering effort for each task, you still have created a perverse incentive.
If I implemented this sort of ranking on my team, BY FAR my best developers would consistently be on the bottom of the list. And this list, would in turn, motivate them to stop working on the hard ambiguous problems I need them to solve, and motivate them to steal as many items as possible that my junior engineers could otherwise have accomplished.
Even if my best engineers stuck to high-effort tasks (which they now are motivated not to do), not all high effort tasks are the same difficulty, and they now have a very strong incentive to only work on the easiest tasks in each difficulty level.
If I was on your development team, the introduction of this leadership board would be a Huge, Neon sign to me that code quality and engineering decision making no longer matters, all that matters for job security is writing as much as possible, as quickly as possible.
What's that? My poorly thought out code will result in numerous more bugs than if I had actually put thought into it to begin with? Well guess what? I'm the developer who best knows how to fix those bugs, so the more bugs I create now, the more I'll be able to fix next sprint too! Then I'll still be on top of the leaderboard!
Sure, you might have some internal processes to prevent this, but those are now on a timeline to deprecation as this leadership board comes to define your team culture. And even if your engineering team's internal culture is strong enough to overcome this blight, the game will simply become either how to best ignore this leadership board without upsetting management, or alternatively how to most act as described in the previous paragraph without getting caught.
This reeks of project management applied to engineering, without understanding the nuances of engineering.
I don't think so. I read the full blog series, and I am interested in some of the ideas.
It's just for "maintenance" tasks which are limited to 8-12 hours per week. Developers give a task (in a weekly meeting) an effort of low (2hrs), medium (4), or high (8) and choose which tasks they want to get done in the week. If a developer is knocking out 3+ high effort tasks in a week there is 1) a problem with estimating effort and/or 2) a problem with the developer spending way too much time on maintenance tasks. Either one can be addressed.
I am not sure if I would try these ideas out for myself, but I think most of the criticisms here on HN are addressed in the full blog series.
It's just for "maintenance" tasks which are limited to 8-12 hours per week. Developers give a task (in a weekly meeting) an effort of low (2hrs), medium (4), or high (8) and choose which tasks they want to get done in the week. If a developer is knocking out 3+ high effort tasks in a week there is 1) a problem with estimating effort and/or 2) a problem with the developer spending way too much time on maintenance tasks. Either one can be addressed.
I am not sure if I would try these ideas out for myself, but I think most of the criticisms here on HN are addressed in the full blog series.
I'd be curious to see the long term (1-2 year) consequences of this. Whether it leads to burnout/attrition or greater employee satisfaction.
I think it all depends on how you approach it at a management level.
It might become a nightmare, with people burning out over this, or it might play out as a healthy way of tracking metrics on people's work to find ways of improving over time.
People should feel safe about the fact that they are not personally judged over this (or any metric about their work), that we are in this together and we use "points" to understand issues and how to improve together
It might become a nightmare, with people burning out over this, or it might play out as a healthy way of tracking metrics on people's work to find ways of improving over time.
People should feel safe about the fact that they are not personally judged over this (or any metric about their work), that we are in this together and we use "points" to understand issues and how to improve together
You call the people with the highest scores top performers. You're going to ignore that in performance reviews?
These "tennis rankings" sound awfully similar to an idea often used in kindergarten classrooms of awarding the children with gold stars or stickers for completing tasks.
http://changingminds.org/disciplines/teaching/classroom_mana...
I'm sure it can be helpful to have some kind of feedback for productivity and output but it could also be seen as a bit patronizing and or paternalistic if you're not careful.
I'm sure it can be helpful to have some kind of feedback for productivity and output but it could also be seen as a bit patronizing and or paternalistic if you're not careful.
I agree about the risk. I think this is true for each system you put in place to give feedback on productivity — the difference between "healthy" and "patronizing" lies in good management
I find treating developers like children and making them compete with each other for points very insulting.
The implication is that developers are incapable of working on what they are supposed to do by default and introducing a "game" like this is what motivates them to do better.
Do any other job families have to go through similar "break down" and detailed probing of their deliverables?
The implication is that developers are incapable of working on what they are supposed to do by default and introducing a "game" like this is what motivates them to do better.
Do any other job families have to go through similar "break down" and detailed probing of their deliverables?
A lot of job families have very measurable performance metics. From marketing counting leads to sales counting dollars and delivery people counting packages delivered.
Software development on average has a lot more complaining about it. I think that's partly because a typical good developer highly values autonomy, and partly because solid software metics are so hard to define.
Software development on average has a lot more complaining about it. I think that's partly because a typical good developer highly values autonomy, and partly because solid software metics are so hard to define.
But those metrics matter so they aren’t really comparable. These bug/feature points are completely made up.
Nurses have to take care of n number of patients they're assigned or their shift is considered a failure. This is often at odds with reality, where patients may be neglected so that every patient can be met within the total time given.
It's not really 1:1 but it's similarly myopic from management.
It's not really 1:1 but it's similarly myopic from management.
Many jobs have very precise performance metrics — just think of sales.
I believe metrics are not good or bad per se, it just depends how you use them.
Of course you may use numbers to treat developers like children. But you may use them also to understand problems and find ways to improve, individually or all together as a team.
I don't think the solution is not to measure things at all :)
I believe metrics are not good or bad per se, it just depends how you use them.
Of course you may use numbers to treat developers like children. But you may use them also to understand problems and find ways to improve, individually or all together as a team.
I don't think the solution is not to measure things at all :)
Who said not to measure things at all?
I think the issue is not that someone's output is being measured it's that the chosen metric is not matching the devs own conception of the value that they bring to the table. For example management might start tracking LoC per Dev and this will lead to discontentment for all the correct reasons.
The halving of points every week something doesn't get done sounds like a great way to motivate engineers to finish tasks on time, but in reality most of these tasks aren't slipping because people aren't motivated - it's because they're de-prioritized by whoever's in charge of product decisions, or because the scope is bigger than originally estimated. If a task slips by a week, now it's worth half points, and quarter points the week after that etc. The motivation to "get it over the finish line" is also going to slump exponentially for anyone optimizing for rewards points, isn't that what we're trying to avoid in the first place?
Halving the points every week something doesn't get done is exactly what you'd want if your goal is to encourage developers to either fix things quickly or don't bother at all, and abandon anything that turns out to be bigger than expected.
It's very strange that the author is claiming that it'll encourage getting things done.
It's very strange that the author is claiming that it'll encourage getting things done.
Great, so you “invented” automated stack ranking based on some bogus metric?
If someone at my company tried to pull shit like this and they weren’t shut down immediately, then I would promptly start looking for my next job.
If someone at my company tried to pull shit like this and they weren’t shut down immediately, then I would promptly start looking for my next job.
His full blog series on this explains it better. To my understanding, this “ranking” is only for maintenance tasks. He also sets “maintenance” at 8-12 hours maximum per week. So it’s just an interesting way of making a game out of doing potentially not-fun maintenance tasks.
He also explains that the developers pick their own maintenance tasks for the week, so it’s an additional way to make people accountable for having decent estimates.
When I look at it through that lens I don’t find it nearly as off putting.
He also explains that the developers pick their own maintenance tasks for the week, so it’s an additional way to make people accountable for having decent estimates.
When I look at it through that lens I don’t find it nearly as off putting.
This brings up a different but related question that I've been trying to find an algorithm answer to:
If people are willing to give only pairwise comparisons of options ("I'd rather do project B instead of A, and C instead of D"), is there a handy algorithm that produces from this a ranked list? You encounter this a lot where people don't know enough to give a prioritization of a full set, but know their little corner of preferences.
If people are willing to give only pairwise comparisons of options ("I'd rather do project B instead of A, and C instead of D"), is there a handy algorithm that produces from this a ranked list? You encounter this a lot where people don't know enough to give a prioritization of a full set, but know their little corner of preferences.
Who is the Goat?
> developers started feeling this part of the work wasn't very important, and gradually moved to spending more time on delivering regular features from the Sprint, neglecting the bugs.
Do your developers maybe have too much autonomy, or lack an appreciation for the way bugs impact users/customers?
I've never worked somewhere that we didn't prioritise the backlog, with input from the Product team. You didn't just get to skip fixing bugs because you didn't think it was fun or interesting. Fixing bugs is part of professional engineering. And, lucky us, they're not actually always boring.
Perhaps I missed something from the rest of the blog post, but it seems like a lot of this system could have been unnecessary if the backlog was properly prioritised and/or management made sure developers were aware that bugs have to be fixed. If you have to assign bug tickets to people because they're not getting voluntarily chosen, then you do that.
Edit: I agree with the point of the section that I took my quote from, which is that you need to have retrospectives/team meetings, work on bottlenecks and processes, etc... But none of those are an excuse for not fixing bugs, they're just hindrances.
Do your developers maybe have too much autonomy, or lack an appreciation for the way bugs impact users/customers?
I've never worked somewhere that we didn't prioritise the backlog, with input from the Product team. You didn't just get to skip fixing bugs because you didn't think it was fun or interesting. Fixing bugs is part of professional engineering. And, lucky us, they're not actually always boring.
Perhaps I missed something from the rest of the blog post, but it seems like a lot of this system could have been unnecessary if the backlog was properly prioritised and/or management made sure developers were aware that bugs have to be fixed. If you have to assign bug tickets to people because they're not getting voluntarily chosen, then you do that.
Edit: I agree with the point of the section that I took my quote from, which is that you need to have retrospectives/team meetings, work on bottlenecks and processes, etc... But none of those are an excuse for not fixing bugs, they're just hindrances.
That's good feedback thank you!
We tried to prioritize the backlog of small tasks against the rest of regular Sprint tasks, but we couldn't make it work very well (tried to write it in the post)
It's not that people skipped bugs entirely, but we found more success by setting up a fixed time every week
At the end the whole process is exactly what you said: a system to make sure the backlog is properly prioritized and managed, with developers assigning bugs to themselves and telling the rest of the company when they will address them
We tried to prioritize the backlog of small tasks against the rest of regular Sprint tasks, but we couldn't make it work very well (tried to write it in the post)
It's not that people skipped bugs entirely, but we found more success by setting up a fixed time every week
At the end the whole process is exactly what you said: a system to make sure the backlog is properly prioritized and managed, with developers assigning bugs to themselves and telling the rest of the company when they will address them
This is dumb. Bugs have costs. Costs to exist and costs to fix.
Eg: my software has a bug, with a workaround that only affects 20% of my clients, once a month. This bug costs $20 each occurrence. So call this a $400/mo bug.
But we've got other bugs are $4000/mo bugs. Or $6k OTC bugs.
And we know what our dev time costs.
So, stop gaming devs against a broken guessing game.
Use real numbers, if you care about your code. This bug costs $400/mo to exist - spend $4000 of your dev time to fix it.
It's not: can you fix in time-guess for points - that depreciate over time (reducing motivation)
It should be: managers will leave you alone for 16h so you can work this one problem to death. Devs can then finish their tasks and gain confidence and pride.
Gaming devs against each other is not a good long term strategy. Measure the priority and allocate resources from that.
Eg: my software has a bug, with a workaround that only affects 20% of my clients, once a month. This bug costs $20 each occurrence. So call this a $400/mo bug.
But we've got other bugs are $4000/mo bugs. Or $6k OTC bugs.
And we know what our dev time costs.
So, stop gaming devs against a broken guessing game.
Use real numbers, if you care about your code. This bug costs $400/mo to exist - spend $4000 of your dev time to fix it.
It's not: can you fix in time-guess for points - that depreciate over time (reducing motivation)
It should be: managers will leave you alone for 16h so you can work this one problem to death. Devs can then finish their tasks and gain confidence and pride.
Gaming devs against each other is not a good long term strategy. Measure the priority and allocate resources from that.
> This bug costs $20 each occurrence.
Knowing how much a bug costs is not always trivial.
Knowing how much a bug costs is not always trivial.
I would argue that figuring out the real cost of a bug is not trivial most of the time.
Well, you know how much time costs right? And you know how much time each client waste when the bug occurs? If you don't then fix that. These are values that can be measured from real world, empirical.
Guessing how long to fix, or how many "points" a bug will take is complete BS next to real-world values.
Why make devs guess at the future when you could have business manages measure the present?
Guessing how long to fix, or how many "points" a bug will take is complete BS next to real-world values.
Why make devs guess at the future when you could have business manages measure the present?
I'm not sure client's time wasted is the right way to measure. For example, I would prioritize a high-risk security related bug (regardless of its effect on my systems performance) over a memory leak.
Right! And my point is that it's easier to measure present cost of $THING. And to compare to other so-measured things than it is to guess at time to fix. You can assign some cost to that security thing can't you? You can assign a risk and then assign cost to that risk.
I never stated that client time was the thing to measure. I'm saying (now repeating) that PRESENT COST is a better basis than FUTURE TIME GUESS.
The whole problem is management wants to maximize dev-time so they try to cram bugs to make your time full - based on a guess. It's ass backwards.
Measure the problem then budget resources to fix.
I never stated that client time was the thing to measure. I'm saying (now repeating) that PRESENT COST is a better basis than FUTURE TIME GUESS.
The whole problem is management wants to maximize dev-time so they try to cram bugs to make your time full - based on a guess. It's ass backwards.
Measure the problem then budget resources to fix.
> Guessing how long to fix, or how many "points" a bug will take is complete BS next to real-world values.
The difficulty is that "how long to fix" literally has to be a guess. Even with up front investment to investigate the nature of the bug and how to go about fixing it, the end result is still a guess, just a more educated guess.
Unless you're exceptionally good at predicting the future. We should have a sidebar chat if this applies to you.
The difficulty is that "how long to fix" literally has to be a guess. Even with up front investment to investigate the nature of the bug and how to go about fixing it, the end result is still a guess, just a more educated guess.
Unless you're exceptionally good at predicting the future. We should have a sidebar chat if this applies to you.
I've played this game before, but it ends up being really difficult to estimate the cost of a bug. E.g. what's the cost of not upgrading <libraryX> from v1.2.0 to v1.5.0? Or a theoretical security bug that you think is going to be quite hard to hit, but would be ruinous if it was exploited? Is that bug 0.001% or 0.01% likely to be triggered?
In practice it's impossible to get accurate ROI estimates for large chunks of the work you do. So being honest about that and using an approximation often doesn't cost you any appreciable accuracy -- and you will definitely spend less time trying to figure out how to precisely cost your bugs.
Don't get me wrong, I'm in favour of using ROI as a lens for prioritizing bugs, I've just never seen anyone make a credible estimate that's more accurate than the nearest order-of-magnitude.
I'm sure this is much easier the bigger you get; Google can estimate how many clicks are lost due to a bug, and clicks are of known value, so you can make a case for this bug being worth $Xm. But for startups I think this is unfeasible because you're working with such small sample sizes.
In practice it's impossible to get accurate ROI estimates for large chunks of the work you do. So being honest about that and using an approximation often doesn't cost you any appreciable accuracy -- and you will definitely spend less time trying to figure out how to precisely cost your bugs.
Don't get me wrong, I'm in favour of using ROI as a lens for prioritizing bugs, I've just never seen anyone make a credible estimate that's more accurate than the nearest order-of-magnitude.
I'm sure this is much easier the bigger you get; Google can estimate how many clicks are lost due to a bug, and clicks are of known value, so you can make a case for this bug being worth $Xm. But for startups I think this is unfeasible because you're working with such small sample sizes.
> We decided the process had two major goals: 1. Complete as much work as possible within the allocated time. 2. Be reliable and complete things respecting our own deadlines.
The choice of 1. as not just a goal, but the number one goal, is a big red flag for me. "As much work as possible" means everyone is incentivized to pick the lowest hanging fruit and simply bang out as many of them as possible. This means any difficult issues simply get postponed as long as they possibly can be, and when someone does pick them up, they won't be happy because their "ranking" will likely suffer. Difficult problems can often be the ones you should address first instead, since they can have larger consequences on the rest of your codebase.
The choice of 1. as not just a goal, but the number one goal, is a big red flag for me. "As much work as possible" means everyone is incentivized to pick the lowest hanging fruit and simply bang out as many of them as possible. This means any difficult issues simply get postponed as long as they possibly can be, and when someone does pick them up, they won't be happy because their "ranking" will likely suffer. Difficult problems can often be the ones you should address first instead, since they can have larger consequences on the rest of your codebase.
This is so backwards.
Points should never be reported or gamed between teams or within them. The second that happens all you get is inflation and that helps nobody other than people who think "chart up and to the right == OK". Points are planning tools. That's it. No company metrics, board level metrics, bonuses, performance evaluations, etc should be measured on team or individual point velocity ever. Period.
You DO need to measure the teams output and the way to do that is to measure two things at once. You need to measure the deployment frequency AND the quality of the application at the same time. Report your deployment frequency and uptime together in one page if you have to. Evaluate teams based on team output. Evaluate individuals based on known criteria - the least of which will be coding.
You DO need to measure the teams output and the way to do that is to measure two things at once. You need to measure the deployment frequency AND the quality of the application at the same time. Report your deployment frequency and uptime together in one page if you have to. Evaluate teams based on team output. Evaluate individuals based on known criteria - the least of which will be coding.
I think it's a mistake to track the metrics at a individual level, and an even bigger one to do it publicly. Your team owns your product, not your individual devs. So by measuring each team member individually, you end up mis-aligning incentives.
Shameless plug: I'm also interested in assigning cost and prioritization to things when it's difficult to do so mathematically or trivially. I'm building a system to do so using Relative comparison. It's in the VERY early stages but: https://www.makepriorities.com
This is actually a pretty good article, in that it exposes the process well and documents it with measurable data (and proof of why it works)
I’m not sure I understand the point. This is to somehow make developers want to fix bugs, rather than build features, right? It seems like an odd way to go about it, but kudos to them if they’re all enjoying it.
The problem will come when some bright spark decides this should be linked to compensation, and then the metric will be gamed to death and become a rope to hang people with
The problem will come when some bright spark decides this should be linked to compensation, and then the metric will be gamed to death and become a rope to hang people with