> One of the main challenges when dealing with technical debt has been the lack of a way to measure it. To help overcome that problem, CISQ/OMG led the development of an Automated Technical Debt (ATD) measurement standard, which is currently being updated with a new version expected in 2023.
I'm highly skeptical about the tech debt measurement algorithm this article purports to be developing.
Google researchers recently published a paper on their attempts to measure technical debt.
They tested 117 metrics that were proposed as potential indicators.
Regressions were used to test each metric to see whether it could predict an engineer’s perceptions of technical debt.
No single metric or combination of metrics were found to be valid indicators.
"I'm a bit pessimistic that much of this research is being driven by large orgs in collaboration with researchers who aren't sufficiently independent."
I'm one of the co-authors of the study. I think your sentiment is valid but what you describe is true for most fields of research: conflicts of interest can be a problem.
I can attest to the fact that the researchers behind this study have extensive backgrounds in academic research and hold themselves to high standards. If nothing else, not doing so risks putting individual reputations on the line.
> Developer experience encompasses how developers feel about, think about, and value their work.9 In prior research, we identified more than 25 sociotechnical factors that affect DevEx. For example, interruptions, unrealistic deadlines, and friction in development tools negatively affect DevEx, while having clear tasks, well-organized code, and pain-free releases improve it.
"Are there any plans to figure out objective ways to measure productivity"
You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies.
"You could look at literally any objective measure to proxy actual productivity and be better off than this"
It's fairly well-established in research (and in practice) that there is no objective measure of developer productivity. Metrics like lines of code, number of pull requests, velocity points are incredibly poor proxies.
If you read the introduction of the paper, you'll see that the aim of this paper is to give managers and developers concrete data to use to help get buy-in on investing in developer experience from business leaders.
I'm one of the co-authors of the study. Your critique is valid though by research standards, for this type of study, our sample is sufficient. We are planning to replicate this study on a larger scale in the future, though!
Accelerate - this book has become an excuse for managers to spend outrageous money implementing metrics like lead time and deployment frequency to measure teams, whereas the book actually advises something very different.
I'm a developer and am right there with you. But if you're a decades-old corporation with 10,000 engineers, you need some set of signals to help guide improvements to tools and processes, right? This benefits developers, and there should be a set of signals to enable this.
"What should we measure to improve developer productivity?" is a decades-old problem for leaders with no clear solution.
There finally seems to be some level of consensus that output metrics like lines of code, # of PRs, and commits, are an ineffective approach.
Lean metrics like cycle time and lead time can be a helpful high-level diagnostic, but they're far from an indicator of effectiveness or productivity.
A new approach being adopted by many organizations is to focus on the actual experiences of developers... the things that slow them down or frustrate them... and turn these into measurements that guide improvement. I'm the founder of getdx.com where we're publishing research on this: http://paper.getdx.com
Right now we're focused on interviewing people in internal-facing roles, but we're exploring doing a series that explores interesting vendors and solutions.
We interview developers and leaders who work on "platform" teams (e.g. devex, devprod, infrastructure) that focus on making developers happier and more productive. If you're interested in this kind of work, or the subject of developer productivity, I think you'd enjoy checking it out.
A couple recent episodes include an interview with a former Dropbox engineer about how they approached measuring productivity, and a conversation with a couple of engineers from LinkedIn who've built a system for gathering real-time feedback about their internal developer tools.
The hard truth is that using velocity to measure performance will guarantee that developers bias their estimates to ensure they make themselves look good, thereby making your "estimates" useless.
Anyone who's built software knows that estimates are guesses and are bound to change once the work actually begins (not to mention if scope changes midway through).
Additionally, velocity does not "count" work happening outside of tickets, e.g., helping with a customer support issue, assisting another developer with legacy code, reviewing other people's work, spending time on feature planning.
I'm highly skeptical about the tech debt measurement algorithm this article purports to be developing.
Google researchers recently published a paper on their attempts to measure technical debt.
They tested 117 metrics that were proposed as potential indicators.
Regressions were used to test each metric to see whether it could predict an engineer’s perceptions of technical debt.
No single metric or combination of metrics were found to be valid indicators.