Somewhere along the way we forgot about software craftsmanship(pcloadletter.dev)
pcloadletter.dev
Somewhere along the way we forgot about software craftsmanship
https://www.pcloadletter.dev/blog/craftsmanship/
65 comments
As with everything else, it's all about incentives. If well-crafted software made more money, then all we'd have would be craftsmanship. But it doesn't.
There is plenty of software which has to be reliable and without surprises, and that is where you will find craftsmanship. Except that doesn't usually make as much money as "big tech", so it won't pay as much and that is the problem. People want to have their cake and eat it too - spend countless hours on problems that tickle their intellectual curiosity and sense of achievement, while also being paid handsomely.
There is plenty of software which has to be reliable and without surprises, and that is where you will find craftsmanship. Except that doesn't usually make as much money as "big tech", so it won't pay as much and that is the problem. People want to have their cake and eat it too - spend countless hours on problems that tickle their intellectual curiosity and sense of achievement, while also being paid handsomely.
> If well-crafted software made more money, then all we'd have would be craftsmanship. But it doesn't.
The problem is that people fall for shiny but low-quality things that break or are hacked into after being used for a week. And then they can't get their money back or cancel their subscription, or they are simply too lazy for it.
We're basically asked to produce a quick dopamine hit, instead of rock-solid long-term solutions.
The problem is that people fall for shiny but low-quality things that break or are hacked into after being used for a week. And then they can't get their money back or cancel their subscription, or they are simply too lazy for it.
We're basically asked to produce a quick dopamine hit, instead of rock-solid long-term solutions.
There's a market for disposable software as much as there is for permanent software.
Disposable software is more like the fast food industry. You're looking to minimize cost. Craftsmenship is not ideal in that situation. I find a lot of B2C software fits into this category. If it's trying to get a monthly subscription, it's probably disposable software. You'll want something that works well to steal them from the competition, but probably will only capture people's attention for a few years before some new competition arrives and steals your customers.
Unless it's reducing costs by automating some business function (like a bank's UI). Then it's important that it just works and is stable.
On the other hand, a lot of B2B software needs to work for a long time without breaking, so it's easier to justify paying for something stable and reliable. You don't want your public API to be unreliable or your clients will leave.
I'd agree that the highest paid engineers are most likely to be building B2C at the big SV companies, but I've made a pretty good living working mostly in the B2B space. I tend to enjoy it more because the business wants reliability. Usually when I'm arriving the project is in major disrepair so they need someone to clean and fix it all up, and they are motivated to pay for it.
Disposable software is more like the fast food industry. You're looking to minimize cost. Craftsmenship is not ideal in that situation. I find a lot of B2C software fits into this category. If it's trying to get a monthly subscription, it's probably disposable software. You'll want something that works well to steal them from the competition, but probably will only capture people's attention for a few years before some new competition arrives and steals your customers.
Unless it's reducing costs by automating some business function (like a bank's UI). Then it's important that it just works and is stable.
On the other hand, a lot of B2B software needs to work for a long time without breaking, so it's easier to justify paying for something stable and reliable. You don't want your public API to be unreliable or your clients will leave.
I'd agree that the highest paid engineers are most likely to be building B2C at the big SV companies, but I've made a pretty good living working mostly in the B2B space. I tend to enjoy it more because the business wants reliability. Usually when I'm arriving the project is in major disrepair so they need someone to clean and fix it all up, and they are motivated to pay for it.
I think quality does make more money, but there is no direct relationship. It is much easier to see the very direct relationship between quality and the expense, so that’s where the focus will be.
This is my take as well. Quality is important, but at least in big tech, it's not what's going to get the quick win on the stock market. But when your product is falling apart within a few years, management won't have the faintest idea why. Must be bad engineers.
If what you were saying was accurate and taken at face value, then we would see companies and products simply crumbling every few years, for seemingly unknown reasons.
There is definitely some amount of truth to this but, by definition, it cannot be as bad as you portrey it.
There is definitely some amount of truth to this but, by definition, it cannot be as bad as you portrey it.
Quality takes time. So the trade-off is well-crafted software vs. more software, but of lower quality. If "more" makes more money than "better, but less", then you can't be surprised that management chooses "more".
I mean, given a choice between two boxes, one of which said "twice as many features", and the other said "crashes half as often", I suspect the majority of users would choose the "twice as many features" box.
I mean, given a choice between two boxes, one of which said "twice as many features", and the other said "crashes half as often", I suspect the majority of users would choose the "twice as many features" box.
On a wider scale, this is Scott Alexander's Moloch -- people who ship an Electron app now would outcompete people who would spend and 3x more on writing native clients for each platform in 3 different languages, and 3x as long to polish the codebase.
Exactly.
Electron is just evolution at work. The OS vendors had decades to fix their system UI frameworks and make it easier to write cross-platform UI applications.
But they didn't because vendor lock-in was always more important to them, and that's how you get Electron. You reap what you sow ;)
Electron has gone a bit overboard with the 'everything but the kitchen sink' philosophy, but the basic idea of a platform-independent application development framework is sound.
Electron is just evolution at work. The OS vendors had decades to fix their system UI frameworks and make it easier to write cross-platform UI applications.
But they didn't because vendor lock-in was always more important to them, and that's how you get Electron. You reap what you sow ;)
Electron has gone a bit overboard with the 'everything but the kitchen sink' philosophy, but the basic idea of a platform-independent application development framework is sound.
I'm investigating free pascal and Lazarus to see if viable multiplatform UI framework. Certainly much smaller binaries
0. We need this done yesterday.
1. Moving targets. As the code becomes usable it becomes clear what is really needed doing and how it should be done. And it's certainly not what was defined at the start of the project.
2. So the prototype/PoC works, thank you! - "job done" - more than good enough. No time to go refactor for the many little "todos" in the code that say " this code needs to be revisited because ..." as the next project "needed yesterday" is already here...
Craftmanship is for academics and people that don't work in the real world because it needs time, lots of it.
1. Moving targets. As the code becomes usable it becomes clear what is really needed doing and how it should be done. And it's certainly not what was defined at the start of the project.
2. So the prototype/PoC works, thank you! - "job done" - more than good enough. No time to go refactor for the many little "todos" in the code that say " this code needs to be revisited because ..." as the next project "needed yesterday" is already here...
Craftmanship is for academics and people that don't work in the real world because it needs time, lots of it.
> Craftmanship is for academics and people that don't work in the real world because it needs time, lots of it.
I think the biggest benefit of craftsmanship is actually flexibility. Well-crafted code has a property of malleability without becoming a tangled mess. Its parts tend to be more orthogonal and less coupled.
I don't think that it's a binary choice of craftsmanship or shipping code; this may be true if you're a younger engineer. Mid- and senior- engineers can build well-crafted systems with less of a tradeoff for speed afforded by having more experience building and knowing where to add that malleability and where it would be wasted.
I think the biggest benefit of craftsmanship is actually flexibility. Well-crafted code has a property of malleability without becoming a tangled mess. Its parts tend to be more orthogonal and less coupled.
I don't think that it's a binary choice of craftsmanship or shipping code; this may be true if you're a younger engineer. Mid- and senior- engineers can build well-crafted systems with less of a tradeoff for speed afforded by having more experience building and knowing where to add that malleability and where it would be wasted.
I've dedicated entire years over my multi-decade career prepping software for changes that never happen. Not once has a big change been correctly prepped for ahead of time.
So my teams don't waste time on it any more.
So my teams don't waste time on it any more.
In my experience the best way to prepare for changes is to write dumb code.
Dumb code, easily revertible has been, by far, the most effective way to prepare for changes in my experience.
Anything trying to be too smart, too generalised, too decoupled with plugable parts and it has a huge potential to become an entangled mess. While dumb code can be read and reasoned about quickly, if a change is easily revertible you can plug back the dumb code and try again without losing momentum.
Anything trying to be too smart, too generalised, too decoupled with plugable parts and it has a huge potential to become an entangled mess. While dumb code can be read and reasoned about quickly, if a change is easily revertible you can plug back the dumb code and try again without losing momentum.
> too decoupled with plugable parts
Decoupling is good.
But if you build a plugin system now you have many more problems.
Decoupling is good.
But if you build a plugin system now you have many more problems.
Decoupling and composition are absolutely powerful tools to make good code but unfortunately I've seen both being used to create ad hoc plugin systems that were simply absurd to deal with in microservices.
"I've invested in individual stocks and never made double significant returns. Therefore I no longer invest in individual stocks."
Totally fair. It is a skill in and of itself.
Totally fair. It is a skill in and of itself.
> I think the biggest benefit of craftsmanship is actually flexibility.
Exactly. Take the craftsmanship of floating things on water. Yes you will be able to assemble a raft and you might far with it, but you will have a better/safer ride if you have a craftsman who knows what's needed and may invent something useful for this journey.
Exactly. Take the craftsmanship of floating things on water. Yes you will be able to assemble a raft and you might far with it, but you will have a better/safer ride if you have a craftsman who knows what's needed and may invent something useful for this journey.
Well... academic code is even worse as it has no incentives other than those (user bases are ~1-2 people and the paper is the product).
As long as it worked briefly to produce the paper results it's "gtg".
This is soooo true; "Tests?"
Yep so write it right the first time alright. Tonight! This is the rite!
This is why you put delay loops or similar into poc, and demo any u proposalsi as a sketch so there is no misunderstanding it's not production ready.
It is also true that a clunky and inelegant piece of code that does its job is infinitely better than a pristinely written code that solves a slightly wrong problem.
So iterating quickly and being agile is bad for the code quality, but it is probably good for finding exactly what is the right problem to solve.
So iterating quickly and being agile is bad for the code quality, but it is probably good for finding exactly what is the right problem to solve.
> Unfortunately, "impact" is almost always measured by what features you shipped rather than considering the impact your code had on the long-term maintainability of the codebase.
I've been trying to communicate this with absolutely no luck at all. Just coding is not considered "impact", meanwhile there are a ton of bugs and features take ridiculously long to complete. I believe this is major misunderstanding or lack of knowledge as to how software should work, thinking it's analogous to building hardware where you need more and more people to scale productivity.
I've been trying to communicate this with absolutely no luck at all. Just coding is not considered "impact", meanwhile there are a ton of bugs and features take ridiculously long to complete. I believe this is major misunderstanding or lack of knowledge as to how software should work, thinking it's analogous to building hardware where you need more and more people to scale productivity.
Yeah, this 100%. Which is why I suspect a 'winning' strategy for a career in software engineering is 'get involved in big new projects early on, take care of the low hanging fruit without much care for code quality or maintainability, and saunter off into the sunset before you have to deal with the consequences of your actions'.
Meanwhile the folks stuck behind end up with the blame for these people's mistakes and the problems caused by their work, and get treated like poor performers because they can't just rush out features at the same lightning pace.
Meanwhile the folks stuck behind end up with the blame for these people's mistakes and the problems caused by their work, and get treated like poor performers because they can't just rush out features at the same lightning pace.
It drives me nuts. People cramming “features” into the app with reckless disregard. Moving through the codebase becomes akin to walking through quicksand.
Have you tried showing results of code qualify scanning tooling such as sonarqube or sonarcloud? This should show a downward pressure on quality.
Impact should be measured as "features and their (monetary) value", not only "feature". Btw, I don't buy "long-term maintainability of the codebase". Ok, _I_ buy it, sometimes; my son doesn't.
This above may be sad regarding the art side of things, but, well, it is what it is.
This above may be sad regarding the art side of things, but, well, it is what it is.
[deleted]
The absence or presence of craftsmanship is primarily a differentiator in which sandboxes you get to play.
You simply cannot build e.g. a viable dbms without at least a modicum of software craftsmanship. If you attempt such a thing with the shovelware mindset of most user-facing applications, it will simply never come together.
You simply cannot build e.g. a viable dbms without at least a modicum of software craftsmanship. If you attempt such a thing with the shovelware mindset of most user-facing applications, it will simply never come together.
Oh it'll come together, and quite quickly too! Moments before eating all your data :P
I serve as tech lead on a team of five people - each with wildly varying levels of experience. Those five people are spread out across two entirely separate projects. Tell me - how does one foster craftsmanship in that environment?
Demonstrate it yourself in practice?
I worked in such an environment and that’s how it’s done. Demonstrate it to your team, set expectations, have conversations about what matters. Encourage improved processes.
I worked in such an environment and that’s how it’s done. Demonstrate it to your team, set expectations, have conversations about what matters. Encourage improved processes.
It's extremely difficult to change a pre-existing culture.
As the people turn over naturally, make extra effort to hire people who want to work that way. However, expect that they will clash with the existing team so try to bring them into the "craftsmenship conspiracy". Two good hires might change a team of five, and three definitely will. If your average engineer stays two years, that'll mean you're replacing 2.5 of them every year, so you could aim for that by 2025.
Also, maybe start vocalizing and demonstrating what it is you want to see. Start a reading group around craftsmenship. Find ways to inspire them about how great it will be when you've got very few production defects or whatever it is that makes that team demoralized. As they say, "If you want to build a ship, don’t drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea."
The other thing is reward it. Not monetarily, but just reward effort that is improving the system. They will need time for these activities. When I manage my teams that develop long lived software I build in at least 33-50% of all time programming for technical debt repayment. There's no doubt in their minds that quality is extremely important. I'll also be extremely aggressive about abolishing deadlines. Unless there's a legitimate emergency, I'll assume all features are done when they are done. Also expect that they will find things to improve you never thought of or think is silly. Generally it's better to good people loose on technical debt than it is to be a dictator.
With the right people, this is the fertilizer that lets them produce extremely high quality work at a good consistent pace, for decades.
As the people turn over naturally, make extra effort to hire people who want to work that way. However, expect that they will clash with the existing team so try to bring them into the "craftsmenship conspiracy". Two good hires might change a team of five, and three definitely will. If your average engineer stays two years, that'll mean you're replacing 2.5 of them every year, so you could aim for that by 2025.
Also, maybe start vocalizing and demonstrating what it is you want to see. Start a reading group around craftsmenship. Find ways to inspire them about how great it will be when you've got very few production defects or whatever it is that makes that team demoralized. As they say, "If you want to build a ship, don’t drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea."
The other thing is reward it. Not monetarily, but just reward effort that is improving the system. They will need time for these activities. When I manage my teams that develop long lived software I build in at least 33-50% of all time programming for technical debt repayment. There's no doubt in their minds that quality is extremely important. I'll also be extremely aggressive about abolishing deadlines. Unless there's a legitimate emergency, I'll assume all features are done when they are done. Also expect that they will find things to improve you never thought of or think is silly. Generally it's better to good people loose on technical debt than it is to be a dictator.
With the right people, this is the fertilizer that lets them produce extremely high quality work at a good consistent pace, for decades.
Actual crafts, as in hand crafts are either wildly expensive or imported from the 3rd world. You can make them yourself, but then it's probably expensive not just in equipment but in time.
With software craftsmanship, I think there are the same options.
I don't think anyone forgot, so much as craftsmanship of all kinds got more expensive if you want it done here.
And the money seems to matter to a lot of people more than ideal quality does. Far fewer people buy a (reasonably-priced if expensive) hand-made table from Amish craftsmen than buy (overpriced but cheap) Ikea garbage.
With software craftsmanship, I think there are the same options.
I don't think anyone forgot, so much as craftsmanship of all kinds got more expensive if you want it done here.
And the money seems to matter to a lot of people more than ideal quality does. Far fewer people buy a (reasonably-priced if expensive) hand-made table from Amish craftsmen than buy (overpriced but cheap) Ikea garbage.
> "impact" is almost always measured by what features you shipped rather than considering the impact your code had on the long-term maintainability of the codebase
A new feature = an increment in major version of your product, for example v1.0 -> v2.0
Bug fixing and other maintenance work = an increment in minor version, for example v1.0 - v1.1
No matter how many bug fixes or perf improvement you did, v1.999999 is always less than v2.0. It's a pretty sad fact.
A new feature = an increment in major version of your product, for example v1.0 -> v2.0
Bug fixing and other maintenance work = an increment in minor version, for example v1.0 - v1.1
No matter how many bug fixes or perf improvement you did, v1.999999 is always less than v2.0. It's a pretty sad fact.
It’s hard to fathom how much information technology is still in its infancy. Probably, one day, people will simply pay for software, and in that universe craftsmanship matters. Until then, the current conditions are largely collateral damage of adtech and SaSS.
Still, it’ll pass. Software is just hard, and business is even harder. Eventually people who are good at both will win, but it takes time to sort itself out.
Still, it’ll pass. Software is just hard, and business is even harder. Eventually people who are good at both will win, but it takes time to sort itself out.
Every project you goto, its allready a catastrophe.
No tests, patchy expectations, strange visionary gurus with even stranger requirements, which of course change all the time and make no sense.
No money for a rewrite, eternally add oriels to it.
Frameworks, that are either old - but reliable, or new and ever changing.
Wish, you could join companies and refuse to participate in old projects.
Is this not simply because software is no longer art, but commerce?
If you're making something for someone else, your soul isn't in it. So who cares?
If you're making something for someone else, your soul isn't in it. So who cares?
It was commerce in the old (late 20th century) days too, but releasing on tape/disk had its advantages:
We shipped every year, or maybe every 6 months. What this meant in practice is that every team got their features to "shoddy" state, but one team would always take much longer to get there than the others, and because we were in a "feature freeze" near ship dates, the teams who were waiting for integration could use that time to clean up their features, applying craftsmanship to make things "shipshape"; products could ship with one or two wobbly new features, and most of them pretty solid.
Now, with internet updates, as soon as a team gets to "shoddy", the feature ships; no one polishes; and all new features tend towards wobbly. (for the business this is an advantage, as features are stabilised only in response to demonstrated customer demand, but I think the enforced opportunities for cleanup on the old schedule, however disadvantageous for the business, were advantageous for engineering, as well as those customers who hadn't explicitly signed up for the beta program)
We shipped every year, or maybe every 6 months. What this meant in practice is that every team got their features to "shoddy" state, but one team would always take much longer to get there than the others, and because we were in a "feature freeze" near ship dates, the teams who were waiting for integration could use that time to clean up their features, applying craftsmanship to make things "shipshape"; products could ship with one or two wobbly new features, and most of them pretty solid.
Now, with internet updates, as soon as a team gets to "shoddy", the feature ships; no one polishes; and all new features tend towards wobbly. (for the business this is an advantage, as features are stabilised only in response to demonstrated customer demand, but I think the enforced opportunities for cleanup on the old schedule, however disadvantageous for the business, were advantageous for engineering, as well as those customers who hadn't explicitly signed up for the beta program)
On the other hand, that also meant that if the team/company didn't give a toss about the quality they'd release the product unfinished, and there'd be no way to improve it even if someone wanted to. So while in some cases you'd have time to polish things up, in a lot of cases you still had the "eh, it's good enough, ship it" attitude without the benefit of being able to fix anything later.
I’m not sure you can characterise all software with such a sweeping generalisation. There are plenty of use cases outside commerce - outside optimising distribution and trade of goods and services - which software contributes to immensely.
Also, maybe there’s an ironic counterfactual in payment and financial reporting networks, like Visa and Peppol. Peppol is not for profit, and Visa infamously facilitates five-way transactions with a huge degree of backwards compatibility.
As an aside, I wonder what code quality means to the author of TFA, and what craftsmanship looks like. Would be fantastic to get clear examples of this from the history of the industry.
Also, maybe there’s an ironic counterfactual in payment and financial reporting networks, like Visa and Peppol. Peppol is not for profit, and Visa infamously facilitates five-way transactions with a huge degree of backwards compatibility.
As an aside, I wonder what code quality means to the author of TFA, and what craftsmanship looks like. Would be fantastic to get clear examples of this from the history of the industry.
Who are the 5 parties in the transaction though? I can only think of 4 (acquiring bank, user bank, user, merchant), who is the 5th party?
Craftsmanship in the old sense, such as carpentry, was also usually commercial. Perhaps the word you're looking for is "alienation", in the M sense; you don't have a personal relationship with the customer/user.
Imperfect carpentry is easily reproducible in the sense that when you spot a flaw, it's trivial to point it out to others. This motivates the maker.
Imperfect software is different. Your use case is (probably) not my use case so why would I get upset about the flaws that affect your use case ? Does this make me alienated from the user base as a whole ? Quite possibly.
Imperfect software is different. Your use case is (probably) not my use case so why would I get upset about the flaws that affect your use case ? Does this make me alienated from the user base as a whole ? Quite possibly.
I always tell new developers, take pride in your work, and take the time to do it right, because your manager will not care at all about the quality, but it'll bite you in the ass if you don't. The only person who is going to care about craftsmanship of your code is you.
Agile (TM) is orthogonal here. If you have too much work, you have too much work.
If anything, agile at least makes it possible to revisit parts of the software and fix them at anytime. No need to be in the same requirements/design/implement/test phase to work on something.
If anything, agile at least makes it possible to revisit parts of the software and fix them at anytime. No need to be in the same requirements/design/implement/test phase to work on something.
According to my job title I'm a software engineer; but I've always regarded myself as a craftsman, writing bespoke handcrafted software. I am also (as the OP suggests) a few years from retirement.
[deleted]
Scrum feature factories
Not sufficient craftsmanship to keep the site up under HN load, sadly.
Atlassian is responsible for putting the guns in the hands of incompetent micromanagers, for whom the likes are Jira are the wet dreams they can only fantasise about in their alternate reality.
Project management would be perfectly capable of being dysfunctional even when limited to typewritten memos. Jira is merely the place where the symptoms are collated. The lymph node of development.
Yeah, but Jira lets them be dysfunctional much faster and more thoroughly, and with greater confidence. That does not improve the situation.
Agile (TM)
Github deserves a lot of the blame here.
Teletubby-candy-crush blingware UI blended with manipulative social upvote features does not engender Göedel-level greatness.
Teletubby-candy-crush blingware UI blended with manipulative social upvote features does not engender Göedel-level greatness.
I can relate with the OP, not in terms of craftsmanship but criticality of things during the development process and operations.
When I worked in finance (algo trading), I was amazed at how we had so many controls and guard rails we had to ship something, that during the time was almost impossible to go south because if we knew that if something went off our company would be burst more or less like Knight Capital.
Fast forward to working in some SaaS, I always struggled to understand how things were so low impact/critical/important and how those things happened almost every second. Was hard to believe that some folks on the fund using VBA and excel spreadsheets were way more careful coding than allegedly professional software engineers. After some point in time, I just felt sloppy the whole process.
With time comes maturity and you start to realize that each company has some kind of meta-culture around how things are done and you end up adapting yourself; but is not an easy process. Another thing that I noticed is that your early experience in SWE or experiences that you're operating in hyper critical industry or a world-class environment creates a massive imprinting that you carry along your entire career.