Good DevEx increases productivity. Here is the data(github.blog)
github.blog
Good DevEx increases productivity. Here is the data
https://github.blog/2024-01-23-good-devex-increases-productivity/
130 comments
I'm one of the co-authors of the study. Your critique is valid though by research standards, for this type of study, our sample is sufficient. We are planning to replicate this study on a larger scale in the future, though!
Are there any plans to figure out objective ways to measure productivity and what distinguishes “good devex” from “bad devex”?
I’ve worked at a lot of big tech companies that do surveys about internal tooling and every year it’s rated as a weak spot, across years and companies this seemed like a consistent trend.
And yet everyone had teams dedicated to improving various aspects of devex so it’s unclear if these teams are just improving the wrong things or if productivity really is improving and it’s something else (eg the amount of code debt grows faster than devex improvements or people are asked to go faster than the devex improvements can keep up or the devex is being improved but the size of the survey means not enough people feel it because you optimize smaller subsets of engineering orgs).
That’s another thing to be mindful about large scale and small scale surveys - the latter might be sampling specific teams adopting the tool whereas the former might find there’s no way to make everyone happy and it all turns into a wash.
I’ve worked at a lot of big tech companies that do surveys about internal tooling and every year it’s rated as a weak spot, across years and companies this seemed like a consistent trend.
And yet everyone had teams dedicated to improving various aspects of devex so it’s unclear if these teams are just improving the wrong things or if productivity really is improving and it’s something else (eg the amount of code debt grows faster than devex improvements or people are asked to go faster than the devex improvements can keep up or the devex is being improved but the size of the survey means not enough people feel it because you optimize smaller subsets of engineering orgs).
That’s another thing to be mindful about large scale and small scale surveys - the latter might be sampling specific teams adopting the tool whereas the former might find there’s no way to make everyone happy and it all turns into a wash.
"Are there any plans to figure out objective ways to measure productivity"
You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies.
Here are some examples: https://newsletter.pragmaticengineer.com/p/measuring-develop...
"... and what distinguishes 'good devex' from 'bad devex'"
Yes to this. This is an ongoing effort - we have two previous journal papers that touch on this which may be of interest to you:
- https://getdx.com/research/devex-what-actually-drives-produc...
- https://getdx.com/research/conceptual-framework-for-develope...
You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies.
Here are some examples: https://newsletter.pragmaticengineer.com/p/measuring-develop...
"... and what distinguishes 'good devex' from 'bad devex'"
Yes to this. This is an ongoing effort - we have two previous journal papers that touch on this which may be of interest to you:
- https://getdx.com/research/devex-what-actually-drives-produc...
- https://getdx.com/research/conceptual-framework-for-develope...
> You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies.
No those are metrics you’re suggesting. There are better ones as someone else mentioned (time to get code from PR to production, some way of measuring the quality of work getting to production, etc). Yes, the obvious metrics are poor and better metrics are difficult to measure and quantify. And obviously no single metric is going to capture something as multidimensional as code development.
Also, the link you reference doesn’t support your argument.
> Across the board, all companies shared that they use both qualitative and quantitative measures
Throughout it discusses that people do use quantitative metrics to help guide their analysis and none of them try to do the obviously naive ones as mentioned.
This isn’t intended as a critique, but as an engineering profession anything that isn’t quantifiable means it’s open to interpretation and argument weakening forward progress to be restricted to what becomes adopted as industry standard which is in many ways more of a popularity contest of fads rather than concrete technical improvement.
No those are metrics you’re suggesting. There are better ones as someone else mentioned (time to get code from PR to production, some way of measuring the quality of work getting to production, etc). Yes, the obvious metrics are poor and better metrics are difficult to measure and quantify. And obviously no single metric is going to capture something as multidimensional as code development.
Also, the link you reference doesn’t support your argument.
> Across the board, all companies shared that they use both qualitative and quantitative measures
Throughout it discusses that people do use quantitative metrics to help guide their analysis and none of them try to do the obviously naive ones as mentioned.
This isn’t intended as a critique, but as an engineering profession anything that isn’t quantifiable means it’s open to interpretation and argument weakening forward progress to be restricted to what becomes adopted as industry standard which is in many ways more of a popularity contest of fads rather than concrete technical improvement.
Makes sense - I agree with you. My mistake on assuming you were referring to metrics like the ones I listed.
Asking people whether the developer experience is good or bad is not going to be the most efficient of approaches: It's ultimately asking for a mood. When teams are asked what they are spending a lot of time on than they wish they shouldn't, you can at least see what are the heavier pain points. It doesn't help if your developer experience budget is zero, but it can at least organize the useful alternatives.
In most places I've worked at, the a survey asking for specific pain points gets great results, because the worst time sinks stick out like a sore thumb, especially if you have workers that have worked in high quality organizations.
In most places I've worked at, the a survey asking for specific pain points gets great results, because the worst time sinks stick out like a sore thumb, especially if you have workers that have worked in high quality organizations.
Those places did detailed surveys but the results were uninteresting; it’s always something abstract like long compile times, long CI times, etc. then the next year the company presents all the concrete ways they made things faster and the same themes repeat.
There are three problems. The first is that people who can accurately point out the problem are outweighed by a bunch of people who are just unhappy and generate a response for the sake of participation / being prompted. This means that you can address the pain point you think you’ve identified only for sentiment to remain unchanged so you try to tackle the next point and the cycle repeats.
The second is that things like “slow compile times” may have hundreds of different reasons so if you improve compile times in aggregate by 20% you’ve not solved anyone’s specific day to day compile time pain point where they normally spend many 10s compiling which sees a reduction to 8s but the expensive compile which runs infrequently (both because less needed or because it’s so slow) takes 1-4 hours. It’ll maybe see a reduction into the 48min-3.2 hour range which is substantial but not enough to be felt or it may be unaffected because the improvements aren’t measuring that slow build and it’s not a focus (eg maybe it has a bad dependency chain that pulls in way too much). The causes of why that’s slow can be hard to tease out correctly and engineers are incentivized to make the “biggest bang for the buck” changes that sound impressive and quantifiable (20% reduction in compile times across the company vs I made this team happier and it’ll maybe show up in an end of year survey if I’m lucky)
The third is that the rate at which certain kinds of things get worse (long CI and compile times) keeps up or usually outperforms the pace at which things get better (eg 1000 developers adding to compile and test times cannot be beat by a team of 10 engineers spending their time on speeding things up).
There are three problems. The first is that people who can accurately point out the problem are outweighed by a bunch of people who are just unhappy and generate a response for the sake of participation / being prompted. This means that you can address the pain point you think you’ve identified only for sentiment to remain unchanged so you try to tackle the next point and the cycle repeats.
The second is that things like “slow compile times” may have hundreds of different reasons so if you improve compile times in aggregate by 20% you’ve not solved anyone’s specific day to day compile time pain point where they normally spend many 10s compiling which sees a reduction to 8s but the expensive compile which runs infrequently (both because less needed or because it’s so slow) takes 1-4 hours. It’ll maybe see a reduction into the 48min-3.2 hour range which is substantial but not enough to be felt or it may be unaffected because the improvements aren’t measuring that slow build and it’s not a focus (eg maybe it has a bad dependency chain that pulls in way too much). The causes of why that’s slow can be hard to tease out correctly and engineers are incentivized to make the “biggest bang for the buck” changes that sound impressive and quantifiable (20% reduction in compile times across the company vs I made this team happier and it’ll maybe show up in an end of year survey if I’m lucky)
The third is that the rate at which certain kinds of things get worse (long CI and compile times) keeps up or usually outperforms the pace at which things get better (eg 1000 developers adding to compile and test times cannot be beat by a team of 10 engineers spending their time on speeding things up).
Having been on the teams that improve developer experience, the problem is that one of my hands gives while the other takes away. I can address every complaint a developer has about the company platform, but at the same time requirements change. As a company grows they start caring more about security and firewalling between different data and services which makes developing harder and more annoying.
For your first question about good devex, there are definitely some objective ways to measure it.
* The time it takes for completed code to be deployed to production * count of manual interventions it takes to get the code deployed * count of other people that have to get involved for a given deploy * how long it takes a new employee to set up their dev environment, count of manual steps involved
Stuff like that
* The time it takes for completed code to be deployed to production * count of manual interventions it takes to get the code deployed * count of other people that have to get involved for a given deploy * how long it takes a new employee to set up their dev environment, count of manual steps involved
Stuff like that
I do love to see more research here, thank you!
Having done internal developer & analyst tooling work (and used DX), this type of survey is great for internal prioritization when you have dedicated capacity for improvements.
I'd be curious to see more about organizational outcomes, as this is piece of DevOps/DevEx data that I feel is weakest and requires the most faith. DORA did some research here, but it's still not always enough to convince leadership.
Having done internal developer & analyst tooling work (and used DX), this type of survey is great for internal prioritization when you have dedicated capacity for improvements.
I'd be curious to see more about organizational outcomes, as this is piece of DevOps/DevEx data that I feel is weakest and requires the most faith. DORA did some research here, but it's still not always enough to convince leadership.
Agreed with this. We're working on it!
For the uninitiated among us - can you share more context on the research standards and the reasoning behind it? I'm interested and would like this to influence some decisions I have but would like to understand the confidence here :).
I agree. I think this is just one of those things that we know is true but can't easily be proven via the scientific method.
I expect some HN readers will balk at the idea that we can know things but can't prove them scientifically but only because they haven't really thought about it. E.g. does giving variables sensible names reduce bug counts? Obviously. But it's very difficult to prove even something as obvious as that.
I expect some HN readers will balk at the idea that we can know things but can't prove them scientifically but only because they haven't really thought about it. E.g. does giving variables sensible names reduce bug counts? Obviously. But it's very difficult to prove even something as obvious as that.
It's peculiar that "developer productivity" is even a term. It doesn't have a generally-agreed-upon definition. I don't think a definition is even possible.
So, we keep trying to come up with objective measures of a term that is impossible to objectively measure. Isn't that weird?
Why not "developer happiness", or, if that's too hippy-dippy for the suits, "developer satisfaction"? Surely we can reason that, all other things being equal, happier developers are more productive.
Then, surveys like this one make perfect sense. "Hey, how satisfied are you with the tools / processes / whathaveyou at your employer?" That's a valid question, and collecting a pile of answers does start to yield some useful insights into what works and what doesn't.
But no, we're trying to use "developer productivity" instead, as though we're all simple machines that provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee.
So, we keep trying to come up with objective measures of a term that is impossible to objectively measure. Isn't that weird?
Why not "developer happiness", or, if that's too hippy-dippy for the suits, "developer satisfaction"? Surely we can reason that, all other things being equal, happier developers are more productive.
Then, surveys like this one make perfect sense. "Hey, how satisfied are you with the tools / processes / whathaveyou at your employer?" That's a valid question, and collecting a pile of answers does start to yield some useful insights into what works and what doesn't.
But no, we're trying to use "developer productivity" instead, as though we're all simple machines that provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee.
I think in general, that's the industrial-age worldview of the "suits", where a business is a money-making machine, with worker productivity measured as an objective quantity, a mechanical force that converts to profit. It's a kind of scientism, a worship of numbers and charts, like economics.
> It doesn't have a generally-agreed-upon definition.
It just means how productive developers are. How much they get done. I don't see what's peculiar about that.
Is it peculiar that "productivity" is a term?
> provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee
That is pretty much the case, though you don't need to word it so awfully.
We provide value by spending our time to producing useful software. The winning business model (all other factors being equal; which they never are) is the one that is able to help developers waste less time when producing said software.
See now it doesn't sound evil.
It just means how productive developers are. How much they get done. I don't see what's peculiar about that.
Is it peculiar that "productivity" is a term?
> provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee
That is pretty much the case, though you don't need to word it so awfully.
We provide value by spending our time to producing useful software. The winning business model (all other factors being equal; which they never are) is the one that is able to help developers waste less time when producing said software.
See now it doesn't sound evil.
Okay, let's walk through this together.
Explain to me what makes a developer "productive". What, specifically, are you referring to when you define it as, "How much they get done"?
Explain to me what makes a developer "productive". What, specifically, are you referring to when you define it as, "How much they get done"?
I am not sure what's hard to understand. Developers have tasks to do right? They are generally employed to do specific things.
Productivity is how quickly they do those things.
If I hire two developers to make me a website, and they both make websites of similar quality but one gets it done in half the time then he is twice as productive. It's not complicated.
Measuring productivity is of course extremely difficult, hence my original comment about the difficulty of proving productivity improvements. But defining it is not especially difficult.
Productivity is how quickly they do those things.
If I hire two developers to make me a website, and they both make websites of similar quality but one gets it done in half the time then he is twice as productive. It's not complicated.
Measuring productivity is of course extremely difficult, hence my original comment about the difficulty of proving productivity improvements. But defining it is not especially difficult.
You're being vague -- whether intentionally or unintentionally, I'm not sure -- and beginning to contradict yourself.
How is it that productivity as a concept is "not complicated" but also extremely difficult to measure? If productivity has a simple definition, then it should be simple to measure, right? Likewise, doesn't the difficulty in measuring it suggest that maybe it's equally difficult to define?
I am a team lead for an organization; we have 100 tickets in our issue tracker. I have three developers. In the same one-week period, Developer Alfred closed a single issue that had been open for over a year and involved a number of core systems that were interacting in a subtle way and causing intermittent outages; Developer Beth closed 5 issues that were all feature requests from paying clients; and Developer Charlie closed 50 issues that were all small bug fixes.
Which developer was the most productive?
I'm not asking for a measure here. No absolute value or unit of measure required. This is purely comparative. If "productivity" has any meaning at all -- and especially if it's "not complicated" -- then this should have a straightforward answer. How should I, as the team lead, evaluate the relative productivity of my three developers?
How is it that productivity as a concept is "not complicated" but also extremely difficult to measure? If productivity has a simple definition, then it should be simple to measure, right? Likewise, doesn't the difficulty in measuring it suggest that maybe it's equally difficult to define?
I am a team lead for an organization; we have 100 tickets in our issue tracker. I have three developers. In the same one-week period, Developer Alfred closed a single issue that had been open for over a year and involved a number of core systems that were interacting in a subtle way and causing intermittent outages; Developer Beth closed 5 issues that were all feature requests from paying clients; and Developer Charlie closed 50 issues that were all small bug fixes.
Which developer was the most productive?
I'm not asking for a measure here. No absolute value or unit of measure required. This is purely comparative. If "productivity" has any meaning at all -- and especially if it's "not complicated" -- then this should have a straightforward answer. How should I, as the team lead, evaluate the relative productivity of my three developers?
> How is it that productivity as a concept is "not complicated" but also extremely difficult to measure?
I should have been more specific really: it's difficult to measure for (most) programming. Productivity is work/time, and time is obviously easy to measure, but for programming the amount of work involved in a task is really really difficult to know because tasks vary so much.
If you're just building widgets in a factory and they're all exactly the same then productivity is trivial to measure because the work to produce each thing is the same.
But that's not the only reason it's near impossible to set up productivity experiments. Ideally you want two identical people to do the exact same task with only one variable differing (e.g. do they have an IDE or Notepad). But you can't do that. People vary hugely in skill, and you can't have the same person do the same task twice in a row because they'll obviously be quicker the second time.
> Which developer was the most productive?
Very difficult to know because - as I already said - measuring productivity is extremely difficult. Does that mean that productivity is somehow vague or difficult to define? No, absolutely not.
Also... I said it's extremely difficult. That doesn't mean you can't do it at all. Surely you have worked with people who are bad at their job and don't get anything done? Or maybe you're lucky enough to have worked with one or two 10x developers? (It's not a myth; you just might not have met any.)
I should have been more specific really: it's difficult to measure for (most) programming. Productivity is work/time, and time is obviously easy to measure, but for programming the amount of work involved in a task is really really difficult to know because tasks vary so much.
If you're just building widgets in a factory and they're all exactly the same then productivity is trivial to measure because the work to produce each thing is the same.
But that's not the only reason it's near impossible to set up productivity experiments. Ideally you want two identical people to do the exact same task with only one variable differing (e.g. do they have an IDE or Notepad). But you can't do that. People vary hugely in skill, and you can't have the same person do the same task twice in a row because they'll obviously be quicker the second time.
> Which developer was the most productive?
Very difficult to know because - as I already said - measuring productivity is extremely difficult. Does that mean that productivity is somehow vague or difficult to define? No, absolutely not.
Also... I said it's extremely difficult. That doesn't mean you can't do it at all. Surely you have worked with people who are bad at their job and don't get anything done? Or maybe you're lucky enough to have worked with one or two 10x developers? (It's not a myth; you just might not have met any.)
This seems like it's getting a bit repetitive at this point. I argue that a concept that is difficult to measure necessarily has a vague, useless, or exceedingly complex definition. I'm not seeing a good argument to the contrary.
Yes, I get that the idea of programmer productivity seems simple. It's just how much "stuff" they do! But, once you start trying to work out the details of what "stuff" is and how to evaluate programmer productivity, it all suddenly turns into a big hairy pear-shaped mess. To me, this makes the concept of programmer productivity pretty darn useless.
And yes, I've worked with or observed a pretty wide variety of developers. As is the case with any other industry, there are some geniuses and other people that are just fantastic -- but also, they tend to be people who have found the right role for their personality and interests. I've also seen a lot of fair-to-middling people who had unsung talents but were stuck keeping the lights on somewhere. I don't think any of this is helpful for understanding the meaning of productivity.
Managers have been trying to figure out how to evaluate developer productivity for at least 25 years -- that would've been about the time I read my first trade rag. All they've managed to do is generate a lot of human misery. You'd think 25 years would be long enough for people to come around to the idea that maybe this developer productivity thing they're chasing is an illusion. But ... naaah.
Yes, I get that the idea of programmer productivity seems simple. It's just how much "stuff" they do! But, once you start trying to work out the details of what "stuff" is and how to evaluate programmer productivity, it all suddenly turns into a big hairy pear-shaped mess. To me, this makes the concept of programmer productivity pretty darn useless.
And yes, I've worked with or observed a pretty wide variety of developers. As is the case with any other industry, there are some geniuses and other people that are just fantastic -- but also, they tend to be people who have found the right role for their personality and interests. I've also seen a lot of fair-to-middling people who had unsung talents but were stuck keeping the lights on somewhere. I don't think any of this is helpful for understanding the meaning of productivity.
Managers have been trying to figure out how to evaluate developer productivity for at least 25 years -- that would've been about the time I read my first trade rag. All they've managed to do is generate a lot of human misery. You'd think 25 years would be long enough for people to come around to the idea that maybe this developer productivity thing they're chasing is an illusion. But ... naaah.
What does similar quality mean, it they're different in multiple axes? Maybe one is easier to use but the other is faster, or one is better tested but the other is written in a more modular way that won't require big refactors to make changes?
This is the crux of the problem, I think, programmer output can be analyzed in so many different axes that have different weights in different industries that measuring productivity objectively is incredibly complex.
This is the crux of the problem, I think, programmer output can be analyzed in so many different axes that have different weights in different industries that measuring productivity objectively is incredibly complex.
> Good developer experience increases self-reported productivity.
So do ineffective DX, most times. There's "proper" way to assess productivity gains that don't rely on subjective self-reporting (eg the DORA metrics). The "improving feedback loops" item on this report makes sense on that light - the other two are questionable as a direct proxy
So do ineffective DX, most times. There's "proper" way to assess productivity gains that don't rely on subjective self-reporting (eg the DORA metrics). The "improving feedback loops" item on this report makes sense on that light - the other two are questionable as a direct proxy
> DORA metrics
I generally agree with your point, with the caveat that metrics like DORA are the first to be gamed when applied to an organization.
It's the usual cycle of applying the metrics first in an informal way, then using it for team/process evaluations as it seems to be somewhat reliable, then teams start to reajust how they make the tickets and the deploy cycle in ways that dramatically increase the metrics while reality hasn't changed much.
Analyzing productivity with completely objective metrics will I think always bear these flaws (I'd assume even measuring things at the highest level as "wages vs revenue" would be gamed on how wages and revenue are calculated ?)
I generally agree with your point, with the caveat that metrics like DORA are the first to be gamed when applied to an organization.
It's the usual cycle of applying the metrics first in an informal way, then using it for team/process evaluations as it seems to be somewhat reliable, then teams start to reajust how they make the tickets and the deploy cycle in ways that dramatically increase the metrics while reality hasn't changed much.
Analyzing productivity with completely objective metrics will I think always bear these flaws (I'd assume even measuring things at the highest level as "wages vs revenue" would be gamed on how wages and revenue are calculated ?)
when applied as a metric, yes - so is any metric that becomes a goal (goodhart's law?). To _assess_ an organization, they're pretty solid
> So do ineffective DX, most times.
I don't find this hard to believe, but do you know of any research showing this?
I don't find this hard to believe, but do you know of any research showing this?
I haven't looked for research tbh, just personal experience. It's kinda hard to prove a negative tbh (how do you organize a research that can measure if something subjectively "bad" made people subjectively less effective?).
I've been to multiple organizations that invested heavily in DX, and as an external observer you could see they became actually slower (eg one went from test-in-prod and deploy-to-prod to a convoluted CI pipeline) - it made engineers feel like they were doing a lot less work (you just push code and forget), but in reality, an entire department was hired to deal with issues in prod, and the teams didn't really deliver anything faster at all. But 100% of the devs swore it was better/faster/more efficient.
On another, the added free time converted directly into more complexity, so tons of overengineered microservices (sometimes to the tune of 2-3x the number of individuals on a team) ensued - it got so easy to do X, everyone overdid it.
I've been to multiple organizations that invested heavily in DX, and as an external observer you could see they became actually slower (eg one went from test-in-prod and deploy-to-prod to a convoluted CI pipeline) - it made engineers feel like they were doing a lot less work (you just push code and forget), but in reality, an entire department was hired to deal with issues in prod, and the teams didn't really deliver anything faster at all. But 100% of the devs swore it was better/faster/more efficient.
On another, the added free time converted directly into more complexity, so tons of overengineered microservices (sometimes to the tune of 2-3x the number of individuals on a team) ensued - it got so easy to do X, everyone overdid it.
My role is DevOps but I find myself gravitating towards improving the developer experience, which I think is a worthy subcategory in the DevOps practice.
The team I'm on previously didn't have any automated way to set up developer workstations, now we've got an Ansible script that does most of everything they need in less than an hour. (Our VPN is slow as cold molasses) Previously, setup was ad-hoc, leading to troubleshooting questions like "did you follow instructions in this forwarded email, this one section from a wiki page, and a copy/paste comment in chat?". Now it's "did you run the script?".
Similarly, the concept of the "Internal Developer Platform" might be interesting to some: https://internaldeveloperplatform.org/core-components/
It's kind of amazing to me that there are whole teams out there that struggle with awful environments that actively put up barriers to efficient coding.
The team I'm on previously didn't have any automated way to set up developer workstations, now we've got an Ansible script that does most of everything they need in less than an hour. (Our VPN is slow as cold molasses) Previously, setup was ad-hoc, leading to troubleshooting questions like "did you follow instructions in this forwarded email, this one section from a wiki page, and a copy/paste comment in chat?". Now it's "did you run the script?".
Similarly, the concept of the "Internal Developer Platform" might be interesting to some: https://internaldeveloperplatform.org/core-components/
It's kind of amazing to me that there are whole teams out there that struggle with awful environments that actively put up barriers to efficient coding.
I suppose I could be called DevOps, and I've always seen that as part of the job - making the infrastructure as easy for the devs to work with as possible. They've got their own specialities to worry about; infrastructure shouldn't be one of them. I've heard the role called somewhat derogatorily "helpdesk for devs", but that is part of the job, and I embrace it. I also embrace "yaml wrangler" - it's not my favorite thing to do, but it does sometimes need wrangling. I suppose I've always seen the role as a generalist one who has some understanding of the overall system, and how to glue (or duct tape) it all together.
I've been called a Systems Wrangler before. It was pointed out to me that I love to wrestle with a new unfamiliar system, "taming" it to work with our stuff.
I think I prefer that to some other choice descriptors that have been applied to me! ;) "Git Whisperer" is another one that is amusing to me, considering the difference in meaning between "Wrangler" and "Whisperer". Does that mean Git requires a subtle approach?
I think I prefer that to some other choice descriptors that have been applied to me! ;) "Git Whisperer" is another one that is amusing to me, considering the difference in meaning between "Wrangler" and "Whisperer". Does that mean Git requires a subtle approach?
Part of this work used to be called toolsmith, but it doesn't quite cover the job description.
devops and dx share the same spirit of reducing friction
at my job i can't stop leaning toward them because I can count in blood the amount of energy, time and money wasted on the thousands papercuts stacked over time
one needs to find a balance of fluidifying the daily logistics involved, it amps up joy and reduce stress.. leaves more time for problem solving and creativity.
sadly most devs / managers couldn't care less about that
at my job i can't stop leaning toward them because I can count in blood the amount of energy, time and money wasted on the thousands papercuts stacked over time
one needs to find a balance of fluidifying the daily logistics involved, it amps up joy and reduce stress.. leaves more time for problem solving and creativity.
sadly most devs / managers couldn't care less about that
I've a similar background, and have worked recently w/ a handful of teams with atrocious, self-imposed developer experience. I think that applying concepts like SLOs and error budgets to relevant developer-centric activities can really help focus the team and other stakeholders well before the experience passes some threshold of terrible.
This is practically a tenet for me now:
If you tolerate painful problems you are capable of fixing, you're a masochist. If you expect others to tolerate them, you're a sadist.
I often work with people who are more than happy to have someone fix problems. But I also work with people who don't just denigrate such work, but are obstructionist. You're expending energy to stop other people from making the system better. There's something deeply wrong with such people.
If you tolerate painful problems you are capable of fixing, you're a masochist. If you expect others to tolerate them, you're a sadist.
I often work with people who are more than happy to have someone fix problems. But I also work with people who don't just denigrate such work, but are obstructionist. You're expending energy to stop other people from making the system better. There's something deeply wrong with such people.
> You're expending energy to stop other people from making the system better. There's something deeply wrong with such people.
One explanation is emotional attachment to a system/technology/solution/process they helped put in place. If someone wants to improve it by heavily modifying or God forbid getting rid of it, they will fight back because they feel personally attacked for it. It is wrong, but it happens a lot.
One explanation is emotional attachment to a system/technology/solution/process they helped put in place. If someone wants to improve it by heavily modifying or God forbid getting rid of it, they will fight back because they feel personally attacked for it. It is wrong, but it happens a lot.
I don't know if it would help or not to mention that the existing system helped to get the company to where it is today, when delivering the "now is the time to get rid of this" pitch.
Acknowledging how long the existing system has been live would hopefully be a reminder that you're not trying to erase their achievement.
Acknowledging how long the existing system has been live would hopefully be a reminder that you're not trying to erase their achievement.
It will help, but the receiving party has to be reasonable enough (or "emotionally mature" enough) to fully grasp it. It's usually not the case with people that go nuclear on these topics.
I haven't quite had 'nuclear' but I know what you two are talking about. "But we would have gone out of business 3 years ago if that weren't written!"
Yes. Three years ago. And it's not three years ago.
Yes. Three years ago. And it's not three years ago.
"What brought you here won't bring you there". As cheap psychology as it might sound, it's actually true most of the times.
> You're expending energy to stop other people from making the system better. There's something deeply wrong with such people.
Lurking somewhere inside everyone is a sublimated death wish.
Lurking somewhere inside everyone is a sublimated death wish.
In my experience workstation setup scripts can be a waste of time to implement and maintain in certain contexts; like everything it depends.
I completely agree. In my experience what, what works is:
1) Development containers. They're not for every situation, but if they suit your situation (prebuilt base images, private repo, long build times that can be split out into common layers, probably remote dev with a good IDE that natively supports them, i.e. a Jetbrains IDE), they're amazing. They completely remove the need for "works on my machine" as you deploy a multi stage built app from the same base container image.
2) Single "5 step" setup page. The setup guide should be: Install git/p4, install toolchain with versions, log in, clone, `make build`/`cmake --build` / `npm run dev`. Depending on what you work on, "run dependencies in a container" (database, grafana/prometheus, log agent) might be there too. Everyone who is onboarded follows these steps, and their first two tasks, in your teams issue tracker are "fix anything that didn't work in setup guide", and "make a PR", both of which should happen on day 1/2.
1) Development containers. They're not for every situation, but if they suit your situation (prebuilt base images, private repo, long build times that can be split out into common layers, probably remote dev with a good IDE that natively supports them, i.e. a Jetbrains IDE), they're amazing. They completely remove the need for "works on my machine" as you deploy a multi stage built app from the same base container image.
2) Single "5 step" setup page. The setup guide should be: Install git/p4, install toolchain with versions, log in, clone, `make build`/`cmake --build` / `npm run dev`. Depending on what you work on, "run dependencies in a container" (database, grafana/prometheus, log agent) might be there too. Everyone who is onboarded follows these steps, and their first two tasks, in your teams issue tracker are "fix anything that didn't work in setup guide", and "make a PR", both of which should happen on day 1/2.
> developers who carve out significant time for deep work enjoy a 50% productivity boost.
Makes sense from personal experience. At my company we have 1 no-meeting day a week (Thursday). Especially since I've gotten into management I've found Thursday to be my favorite day and I probably get ~70% of the actual code work I do in a given week done on that day. If you don't have something like this, I really recommend it!
Makes sense from personal experience. At my company we have 1 no-meeting day a week (Thursday). Especially since I've gotten into management I've found Thursday to be my favorite day and I probably get ~70% of the actual code work I do in a given week done on that day. If you don't have something like this, I really recommend it!
I often think the once-a-week-no-meeting-days should be reversed to once-a-week-meeting-days. Put all status, retrospective, planning and other recurring meetings into this one day so people can actually work the rest of the week.
During Covid my company had a 32 hour week for a year and we lost zero productivity because the first thing that was eliminated were the meetings that just fill time but achieve nothing.
During Covid my company had a 32 hour week for a year and we lost zero productivity because the first thing that was eliminated were the meetings that just fill time but achieve nothing.
Regular meetings and calls can't go over twice a week period. For me, it's one of the biggest reasons for unproductivity. I've worked on a lot of projects and the amount of meetings was directly correlated to how much everything was delayed.
In all cases, the people who pushed the most for meetings were the ones who contributed the least, and in fact had a negative impact. Nowadays it's a nonnegotiable rule from the start in our agency when taking on new clients.
In all cases, the people who pushed the most for meetings were the ones who contributed the least, and in fact had a negative impact. Nowadays it's a nonnegotiable rule from the start in our agency when taking on new clients.
Matches my experience, too.
Unfortunately, some companies approach culture and DevEx as an additive system where they keep piling more meetings, more engagement, more activities, and more process on top of everyone’s responsibilities.
When a company approaches DevEx and culture by subtracting things that aren’t working it’s like a breath of fresh air.
Unfortunately, some companies approach culture and DevEx as an additive system where they keep piling more meetings, more engagement, more activities, and more process on top of everyone’s responsibilities.
When a company approaches DevEx and culture by subtracting things that aren’t working it’s like a breath of fresh air.
I recommend scaling that to no meetings on most days.
One of the things to be aware of is that two meeting free days won't double that output. You still do a lot of valuable "stuff" in that meeting time. But, a reliable period of uninterrupted focus time is rocket fuel for an IC. I get mornings for free (I live in the UK and work mostly with the US), and it's insane how efficient I am for it.
> We finally have data to back up the benefits of developer experience
Any manager who was waiting on the "data" (read: sponsored study) that you need to invest in proper tooling for your devs isn't going to do anything significantly different because of it.
Any manager who was waiting on the "data" (read: sponsored study) that you need to invest in proper tooling for your devs isn't going to do anything significantly different because of it.
You need to think like a desperate manager. Yes, I know, I know.
The job of a manager is to make good choices and decisions. In the absence of an obvious best choice, they have work backward from both success and failure scenarios, including the political and social impacts.
In the absence of external studies and data, if they make an investment in this area and it turns out not to help, they will be left telling their boss things like "everyone says this is right, not sure why my project didn't work."
But a flimsy fig leaf that comes from outside and smells objective is a perfect CYA tool that allows the manager to take a risk while anticipating the need to manage and contain failure.
Does that describe a healthy dynamic and a smart organization? No, but it's reality at many places.
The job of a manager is to make good choices and decisions. In the absence of an obvious best choice, they have work backward from both success and failure scenarios, including the political and social impacts.
In the absence of external studies and data, if they make an investment in this area and it turns out not to help, they will be left telling their boss things like "everyone says this is right, not sure why my project didn't work."
But a flimsy fig leaf that comes from outside and smells objective is a perfect CYA tool that allows the manager to take a risk while anticipating the need to manage and contain failure.
Does that describe a healthy dynamic and a smart organization? No, but it's reality at many places.
Sometimes the manager knows it, but the manager have managers who need to have some external justification to cover their assess. It is basically the same principle that drives hiring consultants to say exactly the same thing the team have been saying for years.
This feels like an uncreative statement though, as there are many different ways to use this data.
For example, what if an engineering manager needs to convince finance for headcount to hire a devex employee?
For example, what if an engineering manager needs to convince finance for headcount to hire a devex employee?
+1
If you read the introduction of the paper, you'll see that the aim of this paper is to give managers and developers concrete data to use to help get buy-in on investing in developer experience from business leaders.
https://queue.acm.org/detail.cfm?id=3639443
If you read the introduction of the paper, you'll see that the aim of this paper is to give managers and developers concrete data to use to help get buy-in on investing in developer experience from business leaders.
https://queue.acm.org/detail.cfm?id=3639443
This is just the whitepaper to cover the manager's manager ass.
Having worked as a software developer in several different areas of enterprise IT, the vast majority of the work is about "Improving {process} to increase {team}'s productivity." Businesses already know, or at least hope, that "Good {worker}Ex increases productivity" and push endlessly towards that objective. It's quite possibly the only reason non-software companies hire software developers.
It's just that IT departments routinely fail to be included in that. So the IT workers get burnt out working on projects they don't understand to make someone else's job easier, while their job gets continually worse because everything is an IT problem now. (And in many cases, nothing actually improves for the business either, because the project only considered the "happy path" so they are still spending most their time dealing with the special cases, but now have to go through IT who are too busy doing the same thing for everyone else.)
It's just that IT departments routinely fail to be included in that. So the IT workers get burnt out working on projects they don't understand to make someone else's job easier, while their job gets continually worse because everything is an IT problem now. (And in many cases, nothing actually improves for the business either, because the project only considered the "happy path" so they are still spending most their time dealing with the special cases, but now have to go through IT who are too busy doing the same thing for everyone else.)
'Member when DevOps was about improving DX? I member.
Since every organization lost sight of the fact that DevOps should be serving the Developers as a customer - DevEx was a low priority and DevOps became about infrastructure purism.
Since every organization lost sight of the fact that DevOps should be serving the Developers as a customer - DevEx was a low priority and DevOps became about infrastructure purism.
IMHO: DevOps was about a simple thing.
"One throat to choke." If there is a problem, you have operations, and development mixed together to produce one nexus of responsibility.
The moment someone says "I can't.... because I'm not a..." in those situation you've lost the point.
The whole point is... It IS your problem as a team. FIX IT. You have everything. Make it happen.
Now a days... the thread has been totally lost. I've gone back to development. I prefer tossing the dead cats over the fence over receiving them.
Either way... the poor cat is dead... we didn't work together to save it :(
"One throat to choke." If there is a problem, you have operations, and development mixed together to produce one nexus of responsibility.
The moment someone says "I can't.... because I'm not a..." in those situation you've lost the point.
The whole point is... It IS your problem as a team. FIX IT. You have everything. Make it happen.
Now a days... the thread has been totally lost. I've gone back to development. I prefer tossing the dead cats over the fence over receiving them.
Either way... the poor cat is dead... we didn't work together to save it :(
That's a shame, because Github actions is IME the worst productivity killer since you can't test locally- which makes iterating painfully long.
That has nothing to do with GitHub Actions. That’s just poor setup
GitHub Actions is predicated on push n' pray SDLC. Actions is also showing some of the warts that exist from being forked from the old Azure DevOps TFS pipelines.
I haven't used GitLab CI runner too often, but the carrot of being able to use a runner on my local machine is extremely appealing.
I haven't used GitLab CI runner too often, but the carrot of being able to use a runner on my local machine is extremely appealing.
It's really amazing. Github Actions has been an absolute curse on my team, it's bundled with other Github and Azure charges so finance convinced us to abandon CircleCI and I have nothing but regrets.
Now imagine having to work with AWS CodePipeline+CodeBuild (because it's always one more service with AWS). The worst CI tool I've ever worked with, but it goes into the AWS bill and no one ever questions that so that's what we're using.
That's most automation and you almost need an obsessive personality for it.
Lots of room for innovation for sure. But if you think actions is bad try iterating on Windows slip streaming.
Lots of room for innovation for sure. But if you think actions is bad try iterating on Windows slip streaming.
What's wrong with local runners like nektos/act?
I haven't used personally but curious if it has some shortcomings they don't mention.
I haven't used personally but curious if it has some shortcomings they don't mention.
They are not 1:1 parity with the hosted version, they are hard to run, and they are slow. When I tried GA I often had to just test on the hosted service, and debugging on the hosted service is so slow and painful. I honestly am very surprised that GA is so popular, it's probably the worst CI I've used and I thought I was missing something.
That's surprising.
GA workflows are basically yaml lists of docker container names to run. I thought it would've been trivial to emulate. At least much easier than any other system.
GA workflows are basically yaml lists of docker container names to run. I thought it would've been trivial to emulate. At least much easier than any other system.
It's not hard to put all the meat in containers run via a Makefile.
Then to test locally you just do "make check".
The problem comes if you start using pre-made actions instead of simple commands.
Then to test locally you just do "make check".
The problem comes if you start using pre-made actions instead of simple commands.
What’s the crowd favorite self-hosted alternative these days?
The other day I was looking up Unix find command on Wikipedia (https://en.wikipedia.org/wiki/Find_(Unix)). It lead me down a rabbit hole to a project called Programmer's Workbench (https://en.wikipedia.org/wiki/PWB/UNIX).
Notable firsts in PWB include:
The Source Code Control System, the first UNIX revision control system, written by Marc J. Rochkind
The remote job entry batch-submission system
The PWB shell, written by John R. Mashey, which preceded Steve Bourne's Bourne shell
The restricted shell (rsh), an option of the PWB shell, used to create widely-available logins for status-checking, trouble-reporting, but made safe by restricting commands
The troff -mm (memorandum) macro package, written by John R. Mashey and Dale W. Smith
Utilities like find, cpio, expr, all three written by Dick Haight, xargs, egrep and fgrep
yacc and lex, which, though not written specifically for PWB, were available outside of Bell Labs for the first time in the PWB distribution
Amazing.
Notable firsts in PWB include:
The Source Code Control System, the first UNIX revision control system, written by Marc J. Rochkind
The remote job entry batch-submission system
The PWB shell, written by John R. Mashey, which preceded Steve Bourne's Bourne shell
The restricted shell (rsh), an option of the PWB shell, used to create widely-available logins for status-checking, trouble-reporting, but made safe by restricting commands
The troff -mm (memorandum) macro package, written by John R. Mashey and Dale W. Smith
Utilities like find, cpio, expr, all three written by Dick Haight, xargs, egrep and fgrep
yacc and lex, which, though not written specifically for PWB, were available outside of Bell Labs for the first time in the PWB distribution
Amazing.
The PWB might well be called a "human-end" computer; like "front-end" and "back-end" computers, it improves productivity by efficient specialization.
- An Introduction to the Programmer's Workbench, T. A. Dolotta and J. R. Mashey
- An Introduction to the Programmer's Workbench, T. A. Dolotta and J. R. Mashey
Shocking discovery, instead of fighting the tools we can do the actual stuff.
edit: to be a bit positive, at least this is neatly presentable to managers who might want to axe DX improvement projects.
edit: to be a bit positive, at least this is neatly presentable to managers who might want to axe DX improvement projects.
I worked for a large corporation. The corporation had a program to hire coop students, both from high school (limited to 4 hrs a day for 5 months), and from university (different universities had different programs, some had 4 month engagements, others than 16 month long ones).
This had positive effect of forcing our internal documentation for developers to be top notch, so that students / new hires were up to speed as fast as possible.
If your corporation has resources to hire co-op students, use this as opportunity to look into what DevEx pieces are missing, what you can improve. This will improve your teams docs/devex and will help your team better interface with other teams within the corporation.
This had positive effect of forcing our internal documentation for developers to be top notch, so that students / new hires were up to speed as fast as possible.
If your corporation has resources to hire co-op students, use this as opportunity to look into what DevEx pieces are missing, what you can improve. This will improve your teams docs/devex and will help your team better interface with other teams within the corporation.
A few years ago I worked for a startup that was recently acquired. We were absolutely minting money, but all the devs got these little Dell ultrabooks with dual-core 15W i7 CPUs. It took around 20 minutes to build the frontend of our web app. People generally tolerated it, but I rocked the boat and rallied coworkers until we all got 6-core 45W i7 laptops. I was lobbying for my sanity, but obviously reducing build times by 10x adds up really quickly and the “managing out” from higher ups that I received subsequently was kind of insane considering the man-hour savings.
Do we have a precise definition for what "DevEx" means?
Talking about productivity by itself is already pretty hard, trowing "DevEx" into the mix seems to me like the perfect recipe to sell snake oil.
Talking about productivity by itself is already pretty hard, trowing "DevEx" into the mix seems to me like the perfect recipe to sell snake oil.
See this previous paper: https://queue.acm.org/detail.cfm?id=3595878
The authors tacked on a mini-survey to DX's regular survey offering, limiting participants to people who work at orgs willing to pay for DX's survey offering. It was not extraordinarily expensive, but the fact that they as recently as a month ago had public pricing and no longer do suggests it's probably more than it was at something like $15 PUPM.
I'm a bit pessimistic that much of this research is being driven by large orgs in collaboration with researchers who aren't sufficiently independent. That's not to say that this sort of work isn't valuable, or that the research is fraudulent, but many people who work on software don't work for these sorts of orgs, and it's not clear to me how those people will be able to participate in research going forward.
Much of this work also feels like it's becoming the industry equivalent of something like the Social Determinants of Health[1]. People with greater job satisfaction and more uninterrupted time are more productive? Is that the best we can do?
[1] https://health.gov/healthypeople/priority-areas/social-deter...
I'm a bit pessimistic that much of this research is being driven by large orgs in collaboration with researchers who aren't sufficiently independent. That's not to say that this sort of work isn't valuable, or that the research is fraudulent, but many people who work on software don't work for these sorts of orgs, and it's not clear to me how those people will be able to participate in research going forward.
Much of this work also feels like it's becoming the industry equivalent of something like the Social Determinants of Health[1]. People with greater job satisfaction and more uninterrupted time are more productive? Is that the best we can do?
[1] https://health.gov/healthypeople/priority-areas/social-deter...
"I'm a bit pessimistic that much of this research is being driven by large orgs in collaboration with researchers who aren't sufficiently independent."
I'm one of the co-authors of the study. I think your sentiment is valid but what you describe is true for most fields of research: conflicts of interest can be a problem.
I can attest to the fact that the researchers behind this study have extensive backgrounds in academic research and hold themselves to high standards. If nothing else, not doing so risks putting individual reputations on the line.
I'm one of the co-authors of the study. I think your sentiment is valid but what you describe is true for most fields of research: conflicts of interest can be a problem.
I can attest to the fact that the researchers behind this study have extensive backgrounds in academic research and hold themselves to high standards. If nothing else, not doing so risks putting individual reputations on the line.
Thank you for the response! Though I think in taking the rest of the context of my criticism away you've avoided the main point of it, which isn't conflict of interest, but whom your research serves to benefit. For example, it would be a shame if cancer research were carried out only on rich or educated participants. That doesn't mean that you shouldn't work with the population you have access to, but perhaps you should give some thought to how you can broaden participation.
Developer experience is hard to quantify, and is not a simple as correlating with subjective input of "how devs feel." SPACE and DORA are attempts to get objective about engineering productivity. Some teams prefer to use a more subjective approach here as well. There aren't any great answers for equivalents in developer experience, yet.
There are some obvious "tooling" considerations that impact developer experience. For the most part, these are related to how much devs can "self-service" the various parts of the SDLC, and how good the user experience of that service flow is. For teams that want to get a great experience without reinventing the wheel, internal developer platforms like what we're building at Coherence (withcoherence.com) are a great answer.
We've also got a free 1-minute PR velocity scorecard that you can try: https://www.withcoherence.com/post/measure-your-pr-efficienc...
There are some obvious "tooling" considerations that impact developer experience. For the most part, these are related to how much devs can "self-service" the various parts of the SDLC, and how good the user experience of that service flow is. For teams that want to get a great experience without reinventing the wheel, internal developer platforms like what we're building at Coherence (withcoherence.com) are a great answer.
We've also got a free 1-minute PR velocity scorecard that you can try: https://www.withcoherence.com/post/measure-your-pr-efficienc...
We definitely need daily meetings to discuss this and find collaborative solutions :-)
But seriously, is this a surprise to anybody?
But seriously, is this a surprise to anybody?
Of course good DevEx increases productivity. Also see Joel on Software:
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
Good DevEx is incredibly important.
I'm not so sure it should be obsessed on more than the Customer and User experience, though.
If DevEx doesn't create Devs who understand users better and help put them in the users shoes, I would think there is room for Good DevEx to improve.
I'm not so sure it should be obsessed on more than the Customer and User experience, though.
If DevEx doesn't create Devs who understand users better and help put them in the users shoes, I would think there is room for Good DevEx to improve.
Getting feedback rapidly is part of good DX, so IMO getting feedback from customers should be accounted for in the cycle. They help each other.
I agree, but many developers are promised to not have to deal with customers.
Then again, I lean towards the idea that separating product management and product ownership results in a disconnected product from users.
Then again, I lean towards the idea that separating product management and product ownership results in a disconnected product from users.
I wonder how much having an office where you can close the door would improve DevEx
Sometimes just taking a break, sitting and contemplating life over a shared meal is best.
But sometimes bringing forward data like this is the right choice. It's what we need as we look to the future.
Very cool.
But sometimes bringing forward data like this is the right choice. It's what we need as we look to the future.
Very cool.
I work in DevEx...it's pretty clear the market does not value DevEx as an investment area right now for companies of all kinds. There are higher priorities, but I'm eager to see areas where the counter-factual might exist.
Instead of using metrics like "productivity" I would like to see a more detailed analysis describing ways in which DevEx oriented productivity contributes to impact/outcomes on an idiosyncratic basis and how it might generalize.
Instead of using metrics like "productivity" I would like to see a more detailed analysis describing ways in which DevEx oriented productivity contributes to impact/outcomes on an idiosyncratic basis and how it might generalize.
One big issue in developer productivity that seems less overlooked now is testing and test turnover time. I guess you could argue this falls under the responsiveness banner in the article, but that doesn't really discuss the extensiveness of testing and focuses mostly on things like review feedback.
Testing in general is great because as your corpus gets ever larger you can be more confident that just running the testsuite is sufficient to catch regressions. That’s why large projects these days often require every behavioral change or bug fix to have new tests to cover the intended behavior and additional negative tests as well (eg don’t just test the new behaviour/fix works, you have to add tests for neighboring behavior as well). For many simple fixes the time to make tests is actually greater than the time to make the fix itself because you reduce the overall development time for the entire project in doing so.
Over time that makes new development substantially easier and less worry inducing, but if you’re constantly adding new tests you get a new problem: eventually running all the tests just takes an absurd amount of time, even spread over all the cores (and depending on what your tests do it can make your computer unusable at the same time).
I recall at some point the webkit bugzilla added bots so you could queue automatic test and commit flags, so you could locally test just the subset of tests you expected to be impacted and then rely on the automated infrastructure doing the full tests prior to merging which helped there.
I vastly prefer that to "merge and rely on CI bots catching regressions" because it trivially leads to you having local checkouts that have failures and then you have to go back and forth verifying the local failures are existing in the tree, etc. In my experience that model seems much more prone to having regularly broken bots, and that breakage leads to normalization of broken bots and general distrust of local failures ("my change couldn't have caused this breakage, so it must already be broken in the tree and CI hasn't caught up")
You get similar issues with build times and there isn’t really a solution for that in the C/C++ model of compilation - if the wrong header is touched you end up in a miserable near world build misery pit, and it doesn’t seem like there’s any real solution for that yet. This feeds into the last point as well: if you get local failures but you don't have confidence that the build isn't already broken there's much less incentive to rebuild the entire tree without your changes to confirm the "impossible by your change" regression is not in fact caused by your changes.
Testing in general is great because as your corpus gets ever larger you can be more confident that just running the testsuite is sufficient to catch regressions. That’s why large projects these days often require every behavioral change or bug fix to have new tests to cover the intended behavior and additional negative tests as well (eg don’t just test the new behaviour/fix works, you have to add tests for neighboring behavior as well). For many simple fixes the time to make tests is actually greater than the time to make the fix itself because you reduce the overall development time for the entire project in doing so.
Over time that makes new development substantially easier and less worry inducing, but if you’re constantly adding new tests you get a new problem: eventually running all the tests just takes an absurd amount of time, even spread over all the cores (and depending on what your tests do it can make your computer unusable at the same time).
I recall at some point the webkit bugzilla added bots so you could queue automatic test and commit flags, so you could locally test just the subset of tests you expected to be impacted and then rely on the automated infrastructure doing the full tests prior to merging which helped there.
I vastly prefer that to "merge and rely on CI bots catching regressions" because it trivially leads to you having local checkouts that have failures and then you have to go back and forth verifying the local failures are existing in the tree, etc. In my experience that model seems much more prone to having regularly broken bots, and that breakage leads to normalization of broken bots and general distrust of local failures ("my change couldn't have caused this breakage, so it must already be broken in the tree and CI hasn't caught up")
You get similar issues with build times and there isn’t really a solution for that in the C/C++ model of compilation - if the wrong header is touched you end up in a miserable near world build misery pit, and it doesn’t seem like there’s any real solution for that yet. This feeds into the last point as well: if you get local failures but you don't have confidence that the build isn't already broken there's much less incentive to rebuild the entire tree without your changes to confirm the "impossible by your change" regression is not in fact caused by your changes.
Even if good devex didn't increase productivity, it increases developer happiness and as a developer, I will try to tithe my budgeted time to improve it and hide it among ambiguous tasks. When I feel I can't, I sense I am at a toxic workplace.
And while I've never left a job primarily due to developer experience alone, it's always the second reason in mind which makes me want to never go back.
And while I've never left a job primarily due to developer experience alone, it's always the second reason in mind which makes me want to never go back.
I tend to spend as much -or more- time on the backend of my apps, as I do, on the frontend (frequently more time).
Not only is it important to make developer experience smoother, it is also important to make it safer. When a dev makes a mistake, it can cause big problems, so it's important to make their tools easy to use and understand.
Not only is it important to make developer experience smoother, it is also important to make it safer. When a dev makes a mistake, it can cause big problems, so it's important to make their tools easy to use and understand.
Then why do you guys completely ignore https://github.com/orgs/community/discussions/10369 ?
I feel like the people in that issue are doing something wrong. I've never unintentionally started a review instead of submitting a comment. I feel the buttons are pretty clear about which does what.
As someone bitten in the ass by this, it's a UX issue. If it were just me, fine - I'm an idiot. But it's enough people that come looking for this exact same problem.
They are doing something wrong, and having enough people doing the same thing wrong is exactly why the design of a product might need to be re-considered.
This happens to me semi-regularly. There's a mode difference where if you're reviewing, the comments don't get sent, when normally those comments are sent immediately.
It's rare that I complete reviews in a single session, so I often don't realize I'm in the middle of a review when I go back to the PR.
It's rare that I complete reviews in a single session, so I often don't realize I'm in the middle of a review when I go back to the PR.
> I feel like the people in that issue are doing something wrong.
And there are UX choices that could result in fewer people doing something wrong.
And there are UX choices that could result in fewer people doing something wrong.
[deleted]
This seems like a tautology because what we think of as a good “experience” is something that lets us get our work done quickly. No one ever said a tool had great devex but was hard to get stuff done with.
This required an entire commissioned research study and published article...?
You can think of it as an Ad for GitHub Copilot
Yes.
> “Certain technologies like GitHub Copilot can help developers better understand their code and future-proof their productivity.”
I would have never thought that. /s
> “Certain technologies like GitHub Copilot can help developers better understand their code and future-proof their productivity.”
I would have never thought that. /s
Ever seen developers that are frustrated by technical debt and whose business leaders don't seem to care? This study is aimed to help with that.
>Adobe is an example of a company that recognizes the value of providing an effective working environment for developers. CJ Dotson, senior PM of developer productivity at Adobe, notes that “when technology is what you sell, investments in DevEx are not optional. Adobe’s investment in DevEx leads to higher developer satisfaction and better business outcomes.”
Ok, a bit of a fluff, but sure, it's just a first paragraph of a chapter, surely they will expand on it in the next one and actually give some examples. After all, they wouldn't make this a one-of statement just to mention Adibe by name and get that sponsorship money, right? Right?!
Ok, a bit of a fluff, but sure, it's just a first paragraph of a chapter, surely they will expand on it in the next one and actually give some examples. After all, they wouldn't make this a one-of statement just to mention Adibe by name and get that sponsorship money, right? Right?!
Disclaimer: didn't read the paper yet. But the assertions are "Feel X% more Y", which is hardly hard data. Seems like the methodology was just interviews?
Good DevEx is incredibly important. Nice to see actual data supporting what is already known anecdotally.
> Using work design theory, we created a set of research hypotheses, developed questions, and surveyed more than 20 industry-diverse companies.
So it's a bunch of surveys, not experimental data; AFAICT nothing here distinguishes ex. "devs who understand their code are productive" from "devs who are productive develop an understanding of the code". But forget correlation/causation, because it turns out their claims - even in the linked "research" is about how people feel, not their actual effectiveness - choice quotes are
> developers who report a high degree of understanding of their code feel 42% more productive
or
> Developers who have a significant amount of time carved out for deep work feel 50 percent more productive compared with those lacking in dedicated time
Notice that there's no attempt to objectively see if those feelings are true! SLoC might be poor, but you could look at literally any objective measure to proxy actual productivity and be better off than this!
So for all that I'd like to believe them, and all that the claims could actually be true, this is just a bunch of meaningless nonsense that barely qualifies as "data" and certainly shouldn't be taken seriously as "research".
So it's a bunch of surveys, not experimental data; AFAICT nothing here distinguishes ex. "devs who understand their code are productive" from "devs who are productive develop an understanding of the code". But forget correlation/causation, because it turns out their claims - even in the linked "research" is about how people feel, not their actual effectiveness - choice quotes are
> developers who report a high degree of understanding of their code feel 42% more productive
or
> Developers who have a significant amount of time carved out for deep work feel 50 percent more productive compared with those lacking in dedicated time
Notice that there's no attempt to objectively see if those feelings are true! SLoC might be poor, but you could look at literally any objective measure to proxy actual productivity and be better off than this!
So for all that I'd like to believe them, and all that the claims could actually be true, this is just a bunch of meaningless nonsense that barely qualifies as "data" and certainly shouldn't be taken seriously as "research".
"You could look at literally any objective measure to proxy actual productivity and be better off than this"
It's fairly well-established in research (and in practice) that there is no objective measure of developer productivity. Metrics like lines of code, number of pull requests, velocity points are incredibly poor proxies.
It's fairly well-established in research (and in practice) that there is no objective measure of developer productivity. Metrics like lines of code, number of pull requests, velocity points are incredibly poor proxies.
Lines of code is a much better proxy than reported productivity.
It stops being a good proxy if you use it to reward or punish developers, compare different languages, or different types of software. But if you don't do those, it's quite good.
Velocity points is worse, because the current culture implies it will be used to reward or punish developers. But it's probably still better than reported productivity.
In fact, reported productivity is probably one of the worst metrics around. People can't even keep track of how long they spend programing, much less of how much they accomplish.
It stops being a good proxy if you use it to reward or punish developers, compare different languages, or different types of software. But if you don't do those, it's quite good.
Velocity points is worse, because the current culture implies it will be used to reward or punish developers. But it's probably still better than reported productivity.
In fact, reported productivity is probably one of the worst metrics around. People can't even keep track of how long they spend programing, much less of how much they accomplish.
"Lines of code is a much better proxy than reported productivity."
Researchers who specialize in this field largely disagree with this. Here's a round-up: https://getdx.com/blog/measuring-developer-activity/
Researchers who specialize in this field largely disagree with this. Here's a round-up: https://getdx.com/blog/measuring-developer-activity/
You noticed that those quotes are almost all about metrics for rewarding or punishing developers, right?
The point of collecting data is to act on it.
But in this case the individual devs aren't rewarded/punished
Alright, I'm willing to consider that their claims are not merely unsupported but impossible to support, but that's not a better look exactly.
>>>> It's fairly well-established in research (and in practice) that there is no objective measure of developer productivity.
Oh, there is.
Money.
How much do you spend on eng to earn a buck (eng value, infrastructure). Impact of your next project on the bottom line. How much can you save by making a change.
So many bullshit features from engineers, and product people that have FUCK ALL impact on the bottom line. But hey they made you happy, or looked good on your resume to get you the next job...
Oh, there is.
Money.
How much do you spend on eng to earn a buck (eng value, infrastructure). Impact of your next project on the bottom line. How much can you save by making a change.
So many bullshit features from engineers, and product people that have FUCK ALL impact on the bottom line. But hey they made you happy, or looked good on your resume to get you the next job...
Private offices for focus without distractions will become a thing again.
Compare salary vs office rent
Compare salary vs office rent
What's DevEx?
> Developer experience encompasses how developers feel about, think about, and value their work.9 In prior research, we identified more than 25 sociotechnical factors that affect DevEx. For example, interruptions, unrealistic deadlines, and friction in development tools negatively affect DevEx, while having clear tasks, well-organized code, and pain-free releases improve it.
https://queue.acm.org/detail.cfm?id=3595878
https://queue.acm.org/detail.cfm?id=3595878
Developer Experience, i.e. the developer's "quality of life" while developing. Things like good tooling, good documentation, and lack of distractions contribute to DevEx.
A shipping company, like VPS or DHT.
Or, "Developer Experience", if you prefer.
Or, "Developer Experience", if you prefer.
"Keep your feedback loops short" is the main one :)
Incoming semi-related rant.
This doesn't apply to LLMs writing code, and it is quite an aside, but I think we have destroyed a lot of ways the web works by chasing a "good" dev experience.
It is "nicer" to write JSX and Typescript in a Next.js project than it was to write jQuery in inline script tags on WordPress. No doubt. But I never had issues maintaining, scaling, updating, or improving those old sites. And I definitely wasn't nickel and dimed for every feature/service I needed to add.
I didn't have to wait for builds, or compiling, or worry about the deployment Node.js being a different version than the local Node.js. Or "open source" framework features being locked to certain hosts (looking at you RSC and Next.13) and their platforms. And the worst thing, coming back to a project after a couple years and it being impossible to get working because all the tooling has changed and/or been abandoned.
I recently had a client from 8 years ago want to move their hosting. They had been sitting on a PHP/Laravel-based CMS, hosted on DigitalOcean, paying $10 a month for ~7 years.
I haven't deployed any websites in the last 3 years that I think could even survive that long. Let alone even run locally after that time. The site could be running fine, but the tooling to work on it, or the platforms supporting it, will break. But my dev experience writing them has been, Chefs kiss!
React will be updated, and perfectly workable things will be deprecated, the tools to build/bundle code will be deprecated or abandoned for ones with a "better experience", and the hosting providers will drop support for older Node.js versions.
Maybe this speaks more to the disposable nature of the web these days, but I feel like websites specifically started getting more brittle and complicated when we started chasing dev experience over user experience.
This doesn't apply to LLMs writing code, and it is quite an aside, but I think we have destroyed a lot of ways the web works by chasing a "good" dev experience.
It is "nicer" to write JSX and Typescript in a Next.js project than it was to write jQuery in inline script tags on WordPress. No doubt. But I never had issues maintaining, scaling, updating, or improving those old sites. And I definitely wasn't nickel and dimed for every feature/service I needed to add.
I didn't have to wait for builds, or compiling, or worry about the deployment Node.js being a different version than the local Node.js. Or "open source" framework features being locked to certain hosts (looking at you RSC and Next.13) and their platforms. And the worst thing, coming back to a project after a couple years and it being impossible to get working because all the tooling has changed and/or been abandoned.
I recently had a client from 8 years ago want to move their hosting. They had been sitting on a PHP/Laravel-based CMS, hosted on DigitalOcean, paying $10 a month for ~7 years.
I haven't deployed any websites in the last 3 years that I think could even survive that long. Let alone even run locally after that time. The site could be running fine, but the tooling to work on it, or the platforms supporting it, will break. But my dev experience writing them has been, Chefs kiss!
React will be updated, and perfectly workable things will be deprecated, the tools to build/bundle code will be deprecated or abandoned for ones with a "better experience", and the hosting providers will drop support for older Node.js versions.
Maybe this speaks more to the disposable nature of the web these days, but I feel like websites specifically started getting more brittle and complicated when we started chasing dev experience over user experience.
I think you're getting some causality twisted - websites started getting more brittle (if they did?) when they started getting more complex, and they started getting more complex precisely because better dev tooling enabled it.
Oh true. That is a good point. I didn't think of it that way
I think this is mostly your own personal experience and likely a bit rose tinted.
It is perfectly possible to deploy a webapp in React that will last for 10 years more or less how a website from 10 years ago could still function today. Also jQuery's very existence was due to how much of a disaster writing webapps was in the past with the host of compatibility issues between not only browsers but just within versions of the same browser. Also there's no issue scaling WordPress? WordPress is notoriously slow and hard to scale. It's a great option for getting a basic website up and running quickly but by God is it a slow and bloated mess and while you can performance tune it, it's not at all a simple or straight forward process and it certainly wasn't easier 10 years ago.
I've been a web developer since 1998 and it's definitely much easier to deploy, maintain and scale websites today than it ever was not to mention that websites today play a much larger role with the rise of mobile computing than they did back then.
To the extent that it was easier to do anything in the past, it was because the concerns of the past were much smaller than they are today. Security was a much lesser concern 10-15 years ago than now. Writing webapps that work across multiple devices also was a much lower priority. Compatibility was a complete mess, many sites were developed to just work on a specific version of Internet Explorer. The idea that 10 years ago there were minimal problems deploying websites or maintaining them is likely you remembering the good parts and forgetting the bad or that your websites didn't need much scale or didn't address many of the modern concerns that exist today. Things were pretty bad back then to be frank and it's much easier today than it ever was.
It is perfectly possible to deploy a webapp in React that will last for 10 years more or less how a website from 10 years ago could still function today. Also jQuery's very existence was due to how much of a disaster writing webapps was in the past with the host of compatibility issues between not only browsers but just within versions of the same browser. Also there's no issue scaling WordPress? WordPress is notoriously slow and hard to scale. It's a great option for getting a basic website up and running quickly but by God is it a slow and bloated mess and while you can performance tune it, it's not at all a simple or straight forward process and it certainly wasn't easier 10 years ago.
I've been a web developer since 1998 and it's definitely much easier to deploy, maintain and scale websites today than it ever was not to mention that websites today play a much larger role with the rise of mobile computing than they did back then.
To the extent that it was easier to do anything in the past, it was because the concerns of the past were much smaller than they are today. Security was a much lesser concern 10-15 years ago than now. Writing webapps that work across multiple devices also was a much lower priority. Compatibility was a complete mess, many sites were developed to just work on a specific version of Internet Explorer. The idea that 10 years ago there were minimal problems deploying websites or maintaining them is likely you remembering the good parts and forgetting the bad or that your websites didn't need much scale or didn't address many of the modern concerns that exist today. Things were pretty bad back then to be frank and it's much easier today than it ever was.
Don't get me wrong, I agree with you that things are easier for developers.
I followed a lot of the (let’s face it) hype and marketing around going headless and using hosted services. And the experience is pretty great for a developer.
But I've had clients who aren't happy with waiting for builds, paying per seat for a CMS, the turnaround of bug fixes and features, having to buy new services that used to be included, and also keep on top of the 100s of dependencies or changes among those services. All while being constantly charged for these "pleasures".
I followed a lot of the (let’s face it) hype and marketing around going headless and using hosted services. And the experience is pretty great for a developer.
But I've had clients who aren't happy with waiting for builds, paying per seat for a CMS, the turnaround of bug fixes and features, having to buy new services that used to be included, and also keep on top of the 100s of dependencies or changes among those services. All while being constantly charged for these "pleasures".
surely they upgraded their OS, and the packages to patch security vulnerabilities, right?
Long uptimes aren't always great, when it comes at the expense of security.
Long uptimes aren't always great, when it comes at the expense of security.
[deleted]
Great data.
Now please fix github actions so it isn't such a PITA to develop workflows in. (I'd start with deprecating yaml actions, personally. Step 2 would be a local dev runner.)
Now please fix github actions so it isn't such a PITA to develop workflows in. (I'd start with deprecating yaml actions, personally. Step 2 would be a local dev runner.)
Agree, there is an open source runner called act, but when I tried it (briefly) it seemed to give different results from actual actions which is unfortunate
An official local runner would be fantastic
An official local runner would be fantastic
Good developer experience increases self-reported productivity. I do suspect that it correlates with actual productivity and business outcomes, especially around reducing the productivity loss that comes from error remediation.