One Way Smart Developers Make Bad Strategic Decisions(earthly.dev)
earthly.dev
One Way Smart Developers Make Bad Strategic Decisions
https://earthly.dev/blog/see-state/
80 comments
> That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.
Past a certain level of complexity, that's no longer true.
_Seeing Like a State_ is a great introduction to this, but I think Carol Sanford's work goes much more into detail. The main thing with the high-modernist view that James Scott was critiquing is that it comes from what Sanford would call the Machine World View. This is where the entire system can be understood by how all of its parts interact. This view breaks down at a certain level of complexity, of which James Scott's book is rife with examples.
Sanford then proposes a worldview she calls the Living Systems World View. Such a system is capable of self-healing and regenerating (such as ecologies, watersheds, communities, polities), and changing on its own. In such a system, you don't affect changes by using direct actions like you do with machines. You use indirect actions.
Kubernetes is a great example. If you're trying to map how everything work together, it can become very complex. I've met smart people who have trouble grasping just how Horizontal Pod Autoscaling works, let alone understand its operational characteristics in live environments. Furthermore, it can be disconcerting to be troubleshooting something and then have the HPA reverse changes you are trying to make ... if you are viewing this through the Machine World View. But viewed through Living Systems World View, it bears many similarities to cultivating a garden. Every living thing is going to grow on its own, and you cannot control for every single variable or conditions.
For similar ideas (which I won't go into detail), there is Christopher Alexander's ideas on Living Architecture. He is a building architect that greatly influenced how people think about Object Oriented Programming (http://www.patternlanguage.com/archive/ieee.html) and Human-Computer Interface design (what the startup world uses to great affect in product design).
Another is the Cynefin framework (https://en.wikipedia.org/wiki/Cynefin_framework). Cynefin identifies different domains -- Simple, Complicated, Complex, and Chaos. Engineers are used to working in the Complicated domain, but when the level of complexity phase-shifts into the Complex domain, the strategies and ways of problem-solving that engineers are used to, will no longer work. This includes clinging to the idea that for any given problem, there is a global solution which will satisfy all local conditions.
Past a certain level of complexity, that's no longer true.
_Seeing Like a State_ is a great introduction to this, but I think Carol Sanford's work goes much more into detail. The main thing with the high-modernist view that James Scott was critiquing is that it comes from what Sanford would call the Machine World View. This is where the entire system can be understood by how all of its parts interact. This view breaks down at a certain level of complexity, of which James Scott's book is rife with examples.
Sanford then proposes a worldview she calls the Living Systems World View. Such a system is capable of self-healing and regenerating (such as ecologies, watersheds, communities, polities), and changing on its own. In such a system, you don't affect changes by using direct actions like you do with machines. You use indirect actions.
Kubernetes is a great example. If you're trying to map how everything work together, it can become very complex. I've met smart people who have trouble grasping just how Horizontal Pod Autoscaling works, let alone understand its operational characteristics in live environments. Furthermore, it can be disconcerting to be troubleshooting something and then have the HPA reverse changes you are trying to make ... if you are viewing this through the Machine World View. But viewed through Living Systems World View, it bears many similarities to cultivating a garden. Every living thing is going to grow on its own, and you cannot control for every single variable or conditions.
For similar ideas (which I won't go into detail), there is Christopher Alexander's ideas on Living Architecture. He is a building architect that greatly influenced how people think about Object Oriented Programming (http://www.patternlanguage.com/archive/ieee.html) and Human-Computer Interface design (what the startup world uses to great affect in product design).
Another is the Cynefin framework (https://en.wikipedia.org/wiki/Cynefin_framework). Cynefin identifies different domains -- Simple, Complicated, Complex, and Chaos. Engineers are used to working in the Complicated domain, but when the level of complexity phase-shifts into the Complex domain, the strategies and ways of problem-solving that engineers are used to, will no longer work. This includes clinging to the idea that for any given problem, there is a global solution which will satisfy all local conditions.
> He is a building architect that greatly influenced how people think about Object Oriented Programming (http://www.patternlanguage.com/archive/ieee.html)
The funny part about this speech is that he's just telling everyone they did it wrong, and Richard Gabriel agrees:
https://dreamsongs.com/Files/DoingItWrong.pdf
The point of his pattern language is to enable people to create their own architecture for their own needs. The point of OOP design patterns is to lock you in a prison of enterprise Java programming in the Kingdom of Nouns. Of course, I think everyone realized this like a decade ago.
The funny part about this speech is that he's just telling everyone they did it wrong, and Richard Gabriel agrees:
https://dreamsongs.com/Files/DoingItWrong.pdf
The point of his pattern language is to enable people to create their own architecture for their own needs. The point of OOP design patterns is to lock you in a prison of enterprise Java programming in the Kingdom of Nouns. Of course, I think everyone realized this like a decade ago.
> This includes clinging to the idea that for any given problem, there is a global solution which will satisfy all local conditions.
The parent comment wasn't stating this. It was stating that there could be a partial global solution that would benefit all microservices, a solution which teams would have to adapt for covering local conditions as well. A middle ground per se.
Thanks for sharing the "Living Systems World View" btw, very interesting!
The parent comment wasn't stating this. It was stating that there could be a partial global solution that would benefit all microservices, a solution which teams would have to adapt for covering local conditions as well. A middle ground per se.
Thanks for sharing the "Living Systems World View" btw, very interesting!
Truly, a fascinating perspective, thank you.
Just for the context: I would say I'm a natural intuitive bottom-upper, except that I can't help but reconsider everything my intuitive self learns from a strongly analytical top-down way.
From that perspective and 30+ years of experience (where I like to think I'm at least open to being completely wrong about anything and everything), I think top-down, prescriptive solutions can be useful and effective, but need to understand and carve out the holes (and conservatively large ones at that) for "local" concerns - BTW, "local" often typically just means lower, where there the lower level itself can have "global" and "local" concerns.
Now, I know this often doesn't happen, so let's lay out how it can work:
- there's a top-down person -- "Tim" in the article -- who has responsibility for for developing a solution
- there are the separate teams, who are responsible for communicating feedback on potential solutions.
Also, I wish I didn't need to point this out, but "responsibility for" === "authority/control over".
(If that's not the case, then never mind: you essentially have a "free-for-all" organization, and just better hope someone who's not too crazy wins the cage-match and can hang on long enough to be a net positive.)
Just for the context: I would say I'm a natural intuitive bottom-upper, except that I can't help but reconsider everything my intuitive self learns from a strongly analytical top-down way.
From that perspective and 30+ years of experience (where I like to think I'm at least open to being completely wrong about anything and everything), I think top-down, prescriptive solutions can be useful and effective, but need to understand and carve out the holes (and conservatively large ones at that) for "local" concerns - BTW, "local" often typically just means lower, where there the lower level itself can have "global" and "local" concerns.
Now, I know this often doesn't happen, so let's lay out how it can work:
- there's a top-down person -- "Tim" in the article -- who has responsibility for for developing a solution
- there are the separate teams, who are responsible for communicating feedback on potential solutions.
Also, I wish I didn't need to point this out, but "responsibility for" === "authority/control over".
(If that's not the case, then never mind: you essentially have a "free-for-all" organization, and just better hope someone who's not too crazy wins the cage-match and can hang on long enough to be a net positive.)
A point made that I think you are missing is that unless all of the separate teams fully understand the potential solution, then they can't provide useful feedback.
If team X doesn't know Kafka, then they can't tell you the ways in which it's not as good as their existing (potentially ad-hoc, but definitely working!) message system. There may be things that their system does that the team just automatically assumes all message-brokers will do because it's "obvious" that it's needed.
If, on the other hand, someone on team X organically considers Kafka as a local solution, learns it, tries it out, all of this stuff becomes obvious immediately.
So the pure top-down approach has two possible solutions:
1. It gets useless feedback "meh, seems fine"
2. All N organizations actually take the time to try out the solution before giving feedback, which means you spend a lot of resources evaluating each top-down hypotheses
The suggested solution from TFA is to have a top-down person embed in one team, find some improvements that work locally, then embed in a second team and do the same. Only then should one try to generalize from the shared experiences of the team. It recognizes that good feedback is expensive and bad feedback is likely, so just cut out the whole "give feedback" stage and have the top-down person learn from actually being embedded in a couple of teams.
If team X doesn't know Kafka, then they can't tell you the ways in which it's not as good as their existing (potentially ad-hoc, but definitely working!) message system. There may be things that their system does that the team just automatically assumes all message-brokers will do because it's "obvious" that it's needed.
If, on the other hand, someone on team X organically considers Kafka as a local solution, learns it, tries it out, all of this stuff becomes obvious immediately.
So the pure top-down approach has two possible solutions:
1. It gets useless feedback "meh, seems fine"
2. All N organizations actually take the time to try out the solution before giving feedback, which means you spend a lot of resources evaluating each top-down hypotheses
The suggested solution from TFA is to have a top-down person embed in one team, find some improvements that work locally, then embed in a second team and do the same. Only then should one try to generalize from the shared experiences of the team. It recognizes that good feedback is expensive and bad feedback is likely, so just cut out the whole "give feedback" stage and have the top-down person learn from actually being embedded in a couple of teams.
Thanks for the reading recommendation! Learning about the Cynefin framework and thinking about those kinds of problems led me to James Scott and to Hayek, but I haven't come across Sanford's work before.
Oh yeah, and I just remembered -- Go. It's a great way of training strategic analysis and making decisions. After moving past the basis, what one explores are global vs. local, influence vs. territory, discerning urgent vs. big moves, direction of play, and so forth. It is theoretically a perfect-information game, but it is sufficiently complex enough for humans that it simulates the fog of war and having to make decisions in face of uncertainty.
>> > A global solution, by necessity, has to ignore local conditions.
>> That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.
So lets rephrase that. A global solution that ignores local conditions will have problems and will likely fail.
>> That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.
So lets rephrase that. A global solution that ignores local conditions will have problems and will likely fail.
Makes sense. In my work I've seen this when trying to get developers on my team using certain patterns, styles, types, conventions, or tools (or the inverse - deprecating them).
Suggestions are usually well grounded (e.g., "let's migrate to this `std` class instead of this old home-rolled wrapper), but sometimes there's some nuance to how something is currently done and deep discussion of the proposal can work through these bits.
Suggestions are usually well grounded (e.g., "let's migrate to this `std` class instead of this old home-rolled wrapper), but sometimes there's some nuance to how something is currently done and deep discussion of the proposal can work through these bits.
> That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.
In theory, yes. But in practice, no (and this is the author's point, I think). In theory, the more "local conditions" you have to account for, the more exponentially complex your "global solution" becomes. (This is the "state" metaphor.) In practice, you can't build that impossibly complex system (and it might not be desirable, anyways!) - so you're likely to try to change local practices in service of a more streamlined global solution. The more you do that, the farther away you move from respecting local conditions.
In theory, yes. But in practice, no (and this is the author's point, I think). In theory, the more "local conditions" you have to account for, the more exponentially complex your "global solution" becomes. (This is the "state" metaphor.) In practice, you can't build that impossibly complex system (and it might not be desirable, anyways!) - so you're likely to try to change local practices in service of a more streamlined global solution. The more you do that, the farther away you move from respecting local conditions.
That depends a lot on the cardinality of the set of Tims.
If there’s one Tim per team, you’ll have 100 Tims proposing different global improvements and 100 teams needing to respond intelligently to those suggestions.
If there’s one Tim per team, you’ll have 100 Tims proposing different global improvements and 100 teams needing to respond intelligently to those suggestions.
That's when the business needs to assign one Tim, or a team of Tims as dictator(s). Everyone else can provide feedback but isn't the decision maker.
A Timtactorship
Through enough iteration, all problems can be solved. But, how many iterations will be required to reach a solution that works for everyone? At that point, is there a solid business case for the project?
The scenario reminds me of this story:
https://mwl.io/archives/605
https://mwl.io/archives/605
> A global solution can solve global concerns and also allow for local conditions.
Not if standardization is the priority.
Not if standardization is the priority.
I think the key difference is whether the local teams have the choice to opt out or not, and my belief is that they should. If they can, they can solve their own problem if the global solution doesn't work. If the global solution wants to keep them as consumers, they must adapt. If they can't leave, the global team will almost certainly stop responding to their needs over time. Like communism, a global solution is terrific in theory, but human behavior causes it to break down in practice.
Caveat, for small enough problems, good enough solutions, and charismatic enough leaders, global solutions can work. But they all break eventually.
Caveat, for small enough problems, good enough solutions, and charismatic enough leaders, global solutions can work. But they all break eventually.
For anyone interested in social systems that help avoid this top-down, centralized failure mode, I cannot recommend RFC 7282 enough:
https://datatracker.ietf.org/doc/html/rfc7282
A whole lot of wisdom is captured in that document, including a deep understanding of the differences between unanimity, majority rule, and consensus.
If you're involved in standardization efforts in any way, whether it's deciding where your team will put braces in source code or running software architecture for a Fortune 100, it will well repay your reading time.
https://datatracker.ietf.org/doc/html/rfc7282
A whole lot of wisdom is captured in that document, including a deep understanding of the differences between unanimity, majority rule, and consensus.
If you're involved in standardization efforts in any way, whether it's deciding where your team will put braces in source code or running software architecture for a Fortune 100, it will well repay your reading time.
interesting. For long time i've found that negative logic is more powerful/overarching than positive one - #ifndef NOT_THIS is more powerful than #if THIS .. and this article applies that even to agreeing vs not-disagreeing.
This seems to be hallmark of a “Middle” developer. Not so junior that they couldn’t build a working solution that they assume everyone should use, but not senior enough to think twice about whether they should be building it.
The “we should make a common framework” for this line is the dominant thought at this level. Never even a library. A framework. Everyone must do it this way.
The more senior people share concepts and maybe libraries, and allow the team to use them if they see fit.
The “we should make a common framework” for this line is the dominant thought at this level. Never even a library. A framework. Everyone must do it this way.
The more senior people share concepts and maybe libraries, and allow the team to use them if they see fit.
It's the large-scale version of taking "DRY" too literally.
Junior devs just repeat themselves because they don't know better.
Middle devs rush into an incomplete abstraction by overzealously not-repeating-themselves.
Senior devs just repeat themselves because they know they don't understand what it would take to abstract out the solution.
Like everything... "It Depends". Don't Repeat Yourself Too Much.
Junior devs just repeat themselves because they don't know better.
Middle devs rush into an incomplete abstraction by overzealously not-repeating-themselves.
Senior devs just repeat themselves because they know they don't understand what it would take to abstract out the solution.
Like everything... "It Depends". Don't Repeat Yourself Too Much.
One thing to remember here is that a senior dev might be at the beginner stage for org wide changes.
Master devs avoid writing code at all costs. They've read Sun Tzu and understand that writing new code is just admitting defeat.
Part of the motivation behind DRY was to avoid the mess of repeating yourself within a task by updating 17 levels of classes and factories to add a field somewhere. This is mainly solved by using a sane language and not having useless intermediary code.
But you are right: applying DRY between tasks requires good judgement, and sometimes it's best to just copy some similar code around than to prematurely invent an abstraction.
But you are right: applying DRY between tasks requires good judgement, and sometimes it's best to just copy some similar code around than to prematurely invent an abstraction.
[deleted]
I've worked at bigger companies and there are plenty of folks much higher than 'middle dev' forcing these types of things down the organization's throat.
This kind of consequential decision can happen at high levels. Obviously less often when a truly brilliant developer ends up in a small organization (but that has its own risks.)
In the example, it was determined that they could not afford to let each service solve its individual bottlenecks ad hoc. So a corresponding strategic error was also made/forced at the senior business level.
It's easy to speculate in hindsight, but in this case I could imagine a globally enforced throughput mandate supported by a widely visible and frequently reviewed dashboard, new tools/libraries as needed, and an optional central queue service required to 'compete for business' of the individual service teams.
I can see potential problems with that too, though. In a sense, failure has already happened when growth management is deemed to be too important to be left to capable individuals on decentralized teams.
Enjoyed the article.
In the example, it was determined that they could not afford to let each service solve its individual bottlenecks ad hoc. So a corresponding strategic error was also made/forced at the senior business level.
It's easy to speculate in hindsight, but in this case I could imagine a globally enforced throughput mandate supported by a widely visible and frequently reviewed dashboard, new tools/libraries as needed, and an optional central queue service required to 'compete for business' of the individual service teams.
I can see potential problems with that too, though. In a sense, failure has already happened when growth management is deemed to be too important to be left to capable individuals on decentralized teams.
Enjoyed the article.
I agree.
People use the most practical things at their disposal. If Tim had opted to publish a repository of easy and _simple_ recipes for managing kafka and postgres integrations, while retaining the ability to use original libraries, then I see no reason why it would not have gained traction.
People use the most practical things at their disposal. If Tim had opted to publish a repository of easy and _simple_ recipes for managing kafka and postgres integrations, while retaining the ability to use original libraries, then I see no reason why it would not have gained traction.
This article does a good job describing one failure mode that's not understood well, but the opposite failure mode is much more common in my experience- having lots of ways to do the same thing can be very inefficient and brittle, even at small companies. The right answer is not "never unify systems" or "always unify systems", but develop judgement about when things should be unified.
Agree, I too have seen the lack of unification more often then not, because business projects are always local. This client wants feature Y, why build it for all clients right now if only one client wants it, I only want to pay for getting the feature out to the client as cheaply and quickly as possible. And now you've got a single use feature. Then next client comes over, and you can't reuse the feature, so you build it again in a slightly different way by different people, maybe even in a different team, rince and repeat. I see that all the time. And that's just one example of how people get their velocity down to a crawl over time. The only solution then is to hire more and more engineers until you're a huge engineering department maintaining a single product.
Of course, this is such a rampant problem in the software industry that a whole market for reusable standard generic solutions was created. That's why we got the cloud, and the array of SaaS, PaaS, IaaS, etc. And don't forget the entire open source is about standards, being able to reuse existing components and frameworks.
What I think the article doesn't mention is that unifying and creating a standard solution is a harder task then creating custom solutions one after the other for each use case/local context. In practice I've seen people try and fail, but often it's not the person with most experience trying, or the business isn't truly willing to put in the effort to succeed, both of these can sabotage things. And again, because it is hard, you have to be willing to fail the first time, but use those learning to try again, and again, until you crack it. And doing that is often worth it long term, cause when you crack it the efficiency and scale will go through the roof, if your business is smart, you might even realize what you have is more valuable than your current business, and pivot to being a SaaS vendor haha. Or you can keep it secret as a competitive advantage.
Of course, this is such a rampant problem in the software industry that a whole market for reusable standard generic solutions was created. That's why we got the cloud, and the array of SaaS, PaaS, IaaS, etc. And don't forget the entire open source is about standards, being able to reuse existing components and frameworks.
What I think the article doesn't mention is that unifying and creating a standard solution is a harder task then creating custom solutions one after the other for each use case/local context. In practice I've seen people try and fail, but often it's not the person with most experience trying, or the business isn't truly willing to put in the effort to succeed, both of these can sabotage things. And again, because it is hard, you have to be willing to fail the first time, but use those learning to try again, and again, until you crack it. And doing that is often worth it long term, cause when you crack it the efficiency and scale will go through the roof, if your business is smart, you might even realize what you have is more valuable than your current business, and pivot to being a SaaS vendor haha. Or you can keep it secret as a competitive advantage.
Lots of resonant points here. It’s worth making it to the end.
I work at a company where there’s a number of different little less-than-one-man projects, and there’s a lot of variety, and so a couple of non-tech types, frustrated with resource allocation (having the right kind of skills at the right place at the right time in the right amount) wants to standardize and simplify.
What I’ve observed though is that when you tell your house painters they can only work with black paint, they can only give your customers black walls, and when your customer wants wood panel, or textured fuschia, then you can’t earn revenue from that market demand.
I work at a company where there’s a number of different little less-than-one-man projects, and there’s a lot of variety, and so a couple of non-tech types, frustrated with resource allocation (having the right kind of skills at the right place at the right time in the right amount) wants to standardize and simplify.
What I’ve observed though is that when you tell your house painters they can only work with black paint, they can only give your customers black walls, and when your customer wants wood panel, or textured fuschia, then you can’t earn revenue from that market demand.
In general, "unity" is something software developers routinely pursue just for the sake of unity itself, failing to understand that unity comes with significant tradeoffs. It is much harder to build a unified solution than a localized, one-off solution. Divide-and-conquer is often a much better engineering strategy: DAC might create more work than unity, but the work is more likely to succeed instead of falling apart because we failed to anticipate all the use cases within the unified framework, especially when we lack experience in the domain.
Also refer to Jeff Atwood's Rule of Threes (which he borrowed from someone else) here.
Also refer to Jeff Atwood's Rule of Threes (which he borrowed from someone else) here.
I've noticed that ALL beginners seem to have a reinvented global solution phase.
Everyone who does electronics might say "Oh I'm going to use this one connector for everything". And it's either ok, if it's a standard connector, or a giant pile of crap that means they can't use a lot of existing stuff because they insisted on this insane DIY grand scheme.
Usually such things have an element of "I want to do Y, so I'll build a modular kit X and use that to Y". And then X becomes the real project and Y is never finished.
The insidious part is how the new product is often a tiny bit better than what's out there. But it doesn't matter. The mediocre standard solution is still way less trouble than the beautiful perfect custom thing. I'd rather have Just Works tech than tech that's Just Right. Anything that seems perfect and beautiful and simple, I don't trust, because it was probably made for one specific task, not to be a general standard you don't have to think about.
I think of the failures with global solutions are because someone did them on a small scale, or because they have to do with natural systems.
Fully top down planning of manmade things by a giant industry consortium is most of why tech is great. Otherwise we would have no USB C, and 12 different CPU architectures.
Sometimes design by comittee protocols suck, but usually because they didn't have enough control, and instead of a protocol, they deliver a description language for companies to make their own protocol, with every feature optional so that compliance does not necessarily mean compatibility.
When you do it internally it can suck because it's more effort than it's worth to replace all your existing stuff.
Everyone who does electronics might say "Oh I'm going to use this one connector for everything". And it's either ok, if it's a standard connector, or a giant pile of crap that means they can't use a lot of existing stuff because they insisted on this insane DIY grand scheme.
Usually such things have an element of "I want to do Y, so I'll build a modular kit X and use that to Y". And then X becomes the real project and Y is never finished.
The insidious part is how the new product is often a tiny bit better than what's out there. But it doesn't matter. The mediocre standard solution is still way less trouble than the beautiful perfect custom thing. I'd rather have Just Works tech than tech that's Just Right. Anything that seems perfect and beautiful and simple, I don't trust, because it was probably made for one specific task, not to be a general standard you don't have to think about.
I think of the failures with global solutions are because someone did them on a small scale, or because they have to do with natural systems.
Fully top down planning of manmade things by a giant industry consortium is most of why tech is great. Otherwise we would have no USB C, and 12 different CPU architectures.
Sometimes design by comittee protocols suck, but usually because they didn't have enough control, and instead of a protocol, they deliver a description language for companies to make their own protocol, with every feature optional so that compliance does not necessarily mean compatibility.
When you do it internally it can suck because it's more effort than it's worth to replace all your existing stuff.
Counter example: Tom standardized a bunch of services... and it worked! Everything is easier and more efficient now.
I agree with the thrust of this post: Changing something that is not understood is a dubious undertaking. But the author fails to make a compelling connection between the above and software development. A poor solution may be a result of not understanding enough of the system as a whole, or it may not. We simply can't tell.
Standardization (i.e. simplification) is generally a good thing in software development. How would Tim's system look if they had opted for his approach from the start? How does the 3rd iteration of the system compare to the 1st iteration? Maybe Tim's solution is stepping-stone to something better. Impossible to tell.
I agree with the thrust of this post: Changing something that is not understood is a dubious undertaking. But the author fails to make a compelling connection between the above and software development. A poor solution may be a result of not understanding enough of the system as a whole, or it may not. We simply can't tell.
Standardization (i.e. simplification) is generally a good thing in software development. How would Tim's system look if they had opted for his approach from the start? How does the 3rd iteration of the system compare to the 1st iteration? Maybe Tim's solution is stepping-stone to something better. Impossible to tell.
> Counter example: Tom standardized a bunch of services... and it worked! Everything is easier and more efficient now.
I’m sorry, but that isn’t really a counter point unless you have some cases to back it up.
In my completely anecdotal experience standardisation never really works. I say this as someone who’s worked on enterprise architecture at the national level in Denmark and has co-written standardisations and principles on how to define things from common building blocks.
The idea was that something like a journal of your health can be defined as a model that can be used by everyone who ever needs to define a journal for health data. And for some cases it works well, it lets thousands of companies define what a “person” is as an example and which parts are the person and which parts are the employee and so on, and it lets them exchange data between systems.
Until it doesn’t. Because all of the sudden an employee is two different things depending on what time of the day it is, because a Nurse has different responsibilities while patients are awake, in some hospitals, and not in others. But because the “standardisation” doesn’t account for this, 50 years of enterprise architecture in the Danish public sector is yet to really pay off.
Some of our best and most successful public sector projects are the ones that didn’t do fanatical standardisation but build things with single responsibilities so that they could easily be chained together to fit a myriad of unique needs.
Now, I’m not against standardisation in any way, but sometimes it just doesn’t make sense and sometimes it does. The issue is that the standardisation approach tends to begin before anyone knows which situation you are actually in.
I’m sorry, but that isn’t really a counter point unless you have some cases to back it up.
In my completely anecdotal experience standardisation never really works. I say this as someone who’s worked on enterprise architecture at the national level in Denmark and has co-written standardisations and principles on how to define things from common building blocks.
The idea was that something like a journal of your health can be defined as a model that can be used by everyone who ever needs to define a journal for health data. And for some cases it works well, it lets thousands of companies define what a “person” is as an example and which parts are the person and which parts are the employee and so on, and it lets them exchange data between systems.
Until it doesn’t. Because all of the sudden an employee is two different things depending on what time of the day it is, because a Nurse has different responsibilities while patients are awake, in some hospitals, and not in others. But because the “standardisation” doesn’t account for this, 50 years of enterprise architecture in the Danish public sector is yet to really pay off.
Some of our best and most successful public sector projects are the ones that didn’t do fanatical standardisation but build things with single responsibilities so that they could easily be chained together to fit a myriad of unique needs.
Now, I’m not against standardisation in any way, but sometimes it just doesn’t make sense and sometimes it does. The issue is that the standardisation approach tends to begin before anyone knows which situation you are actually in.
> I’m sorry, but that isn’t really a counter point unless you have some cases to back it up.
My counter example is about exactly as detailed as the author's example. Of course I was being tongue-and-cheek, but clearly standardization has worked in software.
You can toss your example right on top of all of the other failed attempts at standardization. It in no-way supports the conclusion that "standardization" is a problem. Like I said, I agree with the author's argument, but their conclusion is not supported by that argument. There are many failure modes to large projects.
My counter example is about exactly as detailed as the author's example. Of course I was being tongue-and-cheek, but clearly standardization has worked in software.
You can toss your example right on top of all of the other failed attempts at standardization. It in no-way supports the conclusion that "standardization" is a problem. Like I said, I agree with the author's argument, but their conclusion is not supported by that argument. There are many failure modes to large projects.
Thank you for bringing that up better than I could.
Author makes no claim to why Tim's approach couldn't work. Instead they bring up other scenarios about different things in different contexts and somehow try to say, see, attempting standardization never works. Except in many cases where it did work, like the internet, like USB, like HDMI, like all SaaS and cloud vendors, like Ruby on Rails, like Scikit, like GitHub, etc.
Author makes no claim to why Tim's approach couldn't work. Instead they bring up other scenarios about different things in different contexts and somehow try to say, see, attempting standardization never works. Except in many cases where it did work, like the internet, like USB, like HDMI, like all SaaS and cloud vendors, like Ruby on Rails, like Scikit, like GitHub, etc.
As far as I can tell how well standardization works depends on "how close to the humans" it is.
HTTP; TCP; json; xml: all standardize pretty well. Want to standardize your micro services on nginx with data in json? It will work swimmingly and save time because it is one less decision to be made, and overtime everyone will become familiar with how nginx is setup. Standardizing on what json libs to use so that everyone can dig into the json marshalling code without a lot of head scratching would be another big win.
Trying to standardize people never works because they want to do things their own way and view whatever standard you try to impose as wrong.
HTTP; TCP; json; xml: all standardize pretty well. Want to standardize your micro services on nginx with data in json? It will work swimmingly and save time because it is one less decision to be made, and overtime everyone will become familiar with how nginx is setup. Standardizing on what json libs to use so that everyone can dig into the json marshalling code without a lot of head scratching would be another big win.
Trying to standardize people never works because they want to do things their own way and view whatever standard you try to impose as wrong.
So the answer is, standardize communication/protocols but leave procedure to the free market?
> How would Tim's system look if they had opted for his approach from the start? How does the 3rd iteration of the system compare to the 1st iteration? Maybe Tim's solution is stepping-stone to something better. Impossible to tell.
Reminds of something a senior developer once told me about rewriting systems: the first iteration is ad-hoc and messy; the second iteration is well thought out but completely over-engineered and the third iteration gets it right because the developers have done extremes and know where the correct middle ground is.
Reminds of something a senior developer once told me about rewriting systems: the first iteration is ad-hoc and messy; the second iteration is well thought out but completely over-engineered and the third iteration gets it right because the developers have done extremes and know where the correct middle ground is.
This has been my experience as well! This is why, if I have enough time to do it, I'd normally go through at least 2 throw-away prototypes before settling on a design to implement.
A well publicized example is EA mandating all its studios to use the Frostbite engine
I used to work on the Frostbite team at EA and it was quite the train-wreck. Many game teams spent considerable time moving to Frostbite only to fail and go back to the old game engine, the game teams which managed to move to the frostbite engine were unable to keep up with engine updates and got stuck on old arcane engine versions, the frostbite engine team was split between multiple geolocations and the teams in different locations didn’t get along well and ended up developing silos, and finally there were about a million layers of management in the frostbite team — I heard from old timers that the team used to be much more engineering focused.
Counter example would be all games made using the Unity engine and the Unreal engine no? Or how well RE engine is working out for Capcom. Or how the Decima engine works out for Sony and Kojima, etc.
I have a bunch of friends who work for an EA studio (or who used to work for an EA studio) and they all _hate_ Frostbite. With a passion.
People are similar to water: they will often the path of least resistance.
The trick is to find a solution, document its “shape”, make it easy to integrate and market the hell out of it internally. Then you let the market decide.
Building a big shared common library can be a mistake, but that’s not because it’s intrinsically the wrong choice, it’s impact is partly a function of how many resources you can dedicate to effectively designing and maintaining it. At a certain scale the economics of this suddenly flips.
The problem in the post seems to scream infrastructure rather than code. Identify the different types of queues services need, pick some off the shelf and preferably managed solutions, then make it take 5 minutes to get started.
The trick is to find a solution, document its “shape”, make it easy to integrate and market the hell out of it internally. Then you let the market decide.
Building a big shared common library can be a mistake, but that’s not because it’s intrinsically the wrong choice, it’s impact is partly a function of how many resources you can dedicate to effectively designing and maintaining it. At a certain scale the economics of this suddenly flips.
The problem in the post seems to scream infrastructure rather than code. Identify the different types of queues services need, pick some off the shelf and preferably managed solutions, then make it take 5 minutes to get started.
As a whole the strategy of "Let's see what's common in all these systems" is a good start to understanding the systems. There is a limit to the complexity any single person can understand. Unification is simplification. It helps understanding. But I agree it is no good trying to make the landscape fit a simple map when reality is much more complex. There's no Silver Bullet in trying to combat complexity.
But rather than trying to unify everything think about micro-services. Each service can be its own isolated solution. Of course it needs to be optimized in terms of how well it works when all the other services are running as well. But I think isolation is the key to independently optimizing everything.
I like this sentence from the article: "Lots of it doesn’t matter, but some of it matters a lot".
But rather than trying to unify everything think about micro-services. Each service can be its own isolated solution. Of course it needs to be optimized in terms of how well it works when all the other services are running as well. But I think isolation is the key to independently optimizing everything.
I like this sentence from the article: "Lots of it doesn’t matter, but some of it matters a lot".
Micro Services need a lot of unified standards to talk to each other.
Maybe everyone understands "unification" differently, but for me making something simpler means making things that are small, independent, and where if they change internally they break nothing externally.
But for this to work, you need common patterns, properties, interfaces, otherwise you can't combine your modules into a greater more complex system.
If you do it right, the system as a whole might be complex, but each piece is simple to understand and reason about and making changes to them is not a risk to breaking the whole system.
But it also means reuse, those small simple independent parts can be reused for more than one thing, that's why the system grows complex, so each micro-service is very much a unified standard on its own, just of a small enough scope to successfully build and maintain for multiple clients to leverage.
But now, if you go to the next level up, you have a problem with the complex set of micro-services you now have, and that complex arrangement gets hard to reason about. That is where people added the idea of Supervisor, to have systems that watch over the subsystems, it serves as a way to unify a set of micro-services into a more reliable and understandable hole.
Maybe everyone understands "unification" differently, but for me making something simpler means making things that are small, independent, and where if they change internally they break nothing externally.
But for this to work, you need common patterns, properties, interfaces, otherwise you can't combine your modules into a greater more complex system.
If you do it right, the system as a whole might be complex, but each piece is simple to understand and reason about and making changes to them is not a risk to breaking the whole system.
But it also means reuse, those small simple independent parts can be reused for more than one thing, that's why the system grows complex, so each micro-service is very much a unified standard on its own, just of a small enough scope to successfully build and maintain for multiple clients to leverage.
But now, if you go to the next level up, you have a problem with the complex set of micro-services you now have, and that complex arrangement gets hard to reason about. That is where people added the idea of Supervisor, to have systems that watch over the subsystems, it serves as a way to unify a set of micro-services into a more reliable and understandable hole.
With a microservices architecture, it's still possible to locally optimize each microservice but then end up with a global architecture that is far from ideal. It seems to me that Tim's starting point was exactly that.
Except when Tim decides to standardize how each micro-service is built, using a custom framework...
Yes Tim is known to have done that :-)
But seriously I think there is something about micro-services that helps. It is that they unify things not on the application level but on a lower level by forcing a common communications protocol (http) to be used.
The services themselves can vary widely there is no need to make them all behave the same or even use the same database. All that is needed is for all of them to speak the same language, the same protocol with their clients.
We get the best of both worlds, unification and simplicity yet isolation at the same time. Micro-services are the greatest thing since sliced bread I think.
But seriously I think there is something about micro-services that helps. It is that they unify things not on the application level but on a lower level by forcing a common communications protocol (http) to be used.
The services themselves can vary widely there is no need to make them all behave the same or even use the same database. All that is needed is for all of them to speak the same language, the same protocol with their clients.
We get the best of both worlds, unification and simplicity yet isolation at the same time. Micro-services are the greatest thing since sliced bread I think.
The article seems a bit defeatist to me. When I create an Interface and have a few implementations for it, I've created a standardization, most people would agree it's a big improvement in code reuse and maintainability to leverage interfaces over just having a bunch of one time use concrete classes.
If you think of the United States, you might argue having a central government was better than each state being its own country and maybe that's the edge the US has over Europe.
Deciding to build roads everywhere top down definitely helped with overall car transportation.
Having HDMI as a common format for streaming video and audio sources is a big improvement over each TV using its own format.
There's plenty of counterexample to what the article talks about. So what gives? And on the example mentioned, how do you know if that's a great use of standardization or not?
I've been in many companies and I think they often fail to invest enough in frameworks and standards across the tech side. AWS was born out of Amazon's own effort to do top down standardization for example. And now it's their biggest cash cow and the entire industry is standardizing over its services.
Way too often I see people stuck on contract like work. Each locally scoped small problem needs its own locally scoped big project to be handled. That just simply doesn't scale.
The beauty of most engineering is exactly finding these top-down mechanism that do scale, even if it involves changing the business processes themselves. Think of a dishwasher for example, a top-down design that can wash most things but not all, eventually things that are not dishwasher safe became less and less popular and almost extinct, because people want to scale their efficiency.
Can top-down standards fail, ya sometimes, but when they succeed they take things to the next level of scale.
Don't shoehorn every problem in the same solution, but also, don't solve every solution independently of one another and reinvent the same wheel, or your engineering team of 10 will quickly become thousands while your business product will have barely grown.
If you think of the United States, you might argue having a central government was better than each state being its own country and maybe that's the edge the US has over Europe.
Deciding to build roads everywhere top down definitely helped with overall car transportation.
Having HDMI as a common format for streaming video and audio sources is a big improvement over each TV using its own format.
There's plenty of counterexample to what the article talks about. So what gives? And on the example mentioned, how do you know if that's a great use of standardization or not?
I've been in many companies and I think they often fail to invest enough in frameworks and standards across the tech side. AWS was born out of Amazon's own effort to do top down standardization for example. And now it's their biggest cash cow and the entire industry is standardizing over its services.
Way too often I see people stuck on contract like work. Each locally scoped small problem needs its own locally scoped big project to be handled. That just simply doesn't scale.
The beauty of most engineering is exactly finding these top-down mechanism that do scale, even if it involves changing the business processes themselves. Think of a dishwasher for example, a top-down design that can wash most things but not all, eventually things that are not dishwasher safe became less and less popular and almost extinct, because people want to scale their efficiency.
Can top-down standards fail, ya sometimes, but when they succeed they take things to the next level of scale.
Don't shoehorn every problem in the same solution, but also, don't solve every solution independently of one another and reinvent the same wheel, or your engineering team of 10 will quickly become thousands while your business product will have barely grown.
I haven't read Seeing Like A State. I have to say, i am extremely skeptical about the author's fable about forestry. Commercial foresters today still mostly use monocultures with evenly-spaced planting patterns, and i simply don't believe that they would be doing that if there was a straightforwardly better way to grow trees, even it was less "legible". This has a powerful scent of Gladwell-esque insight porn - the sort of story we love because it's counter-intuitive and makes us feel cleverer than people who haven't heard it.
I don't suppose we have any foresters on this board who can comment?
I don't suppose we have any foresters on this board who can comment?
Some good points. At the same time, sometimes standardization is good and necessary. Imagine if everyone had to reimplement TCP/IP analogues to communicate over a network; we'd never get anything done!
And those types of universal standards arrive by a process not unlike evolution - everyone who wants to has a crack at the problem and the solutions compete for a painful decade or two. Eventually a winner emerges from the top few frontrunners, and by 2040 only networking historians remember ALOHAnet or token ring networks.
However, you can't top-down design a universal standard.
With sufficient skill, patience, and wisdom, you may be able to design a standard that's good enough to be widely applicable.
Humans have free will, though (in practice, at least, no matter what you think about the philosophical question).
A standard only becomes universal when everyone chooses to adopt it.
However, you can't top-down design a universal standard.
With sufficient skill, patience, and wisdom, you may be able to design a standard that's good enough to be widely applicable.
Humans have free will, though (in practice, at least, no matter what you think about the philosophical question).
A standard only becomes universal when everyone chooses to adopt it.
The first part of the article describes the common situation where we see similarities across projects and effort being duplicated, apparently unnecessarily.
To improve understanding and efficiency, we come up with proposals that involve some kind of standardization, such as a shared library abstracting a service.
My team has recently embarked on such an effort, and I was really hoping for a new take on how to avoid the various pitfalls involved in attempting this kind of standardization.
Obviously, central planning works quite well for certain things and, just as obviously, it hasn't worked well for other things. In most situations, it seems that a combination of planning + flexibility works best.
Further, what works or doesn't work for nation-states isn't particularly applicable to software architecture. The analogy feels quite tortured to me.
As to the example given in the article regarding trees, one could just as easily choose an example from modern agriculture where "central planning" seems to work quite well.
In the end, I feel like the article just boils down to: "Make sure you understand the problem domain and the cost of large scale change before you spend a lot of time and effort making said changes."
To improve understanding and efficiency, we come up with proposals that involve some kind of standardization, such as a shared library abstracting a service.
My team has recently embarked on such an effort, and I was really hoping for a new take on how to avoid the various pitfalls involved in attempting this kind of standardization.
Obviously, central planning works quite well for certain things and, just as obviously, it hasn't worked well for other things. In most situations, it seems that a combination of planning + flexibility works best.
Further, what works or doesn't work for nation-states isn't particularly applicable to software architecture. The analogy feels quite tortured to me.
As to the example given in the article regarding trees, one could just as easily choose an example from modern agriculture where "central planning" seems to work quite well.
In the end, I feel like the article just boils down to: "Make sure you understand the problem domain and the cost of large scale change before you spend a lot of time and effort making said changes."
My reading of Seeing Like A State was not "central planning bad". The author explicitly acknowledges that there are many benefits to what they refer to as "high modernist" projects which we all enjoy on a daily basis. My reading was more that large scale projects designed to entirely change the way something is done in order to make it legible to a central authority necessarily throws away an enormous amount of local information and this leads to a tradeoff. You end up being able to build much bigger systems, but those systems are not as flexible.
In the context of microservices that seems like a pertinent point because the purpose of a microservice architecture in the first place is often to allow an organisation to be more flexible and accommodate local knowledge.
In the context of microservices that seems like a pertinent point because the purpose of a microservice architecture in the first place is often to allow an organisation to be more flexible and accommodate local knowledge.
I disagree with the conclusion although not necessarily with the situation described.
A programmer only has so many hours in the day; if you want to be a more efficient programmer you either have to learn to type/think faster or you need to build frameworks, write libraries, and codify common practices.
There are situations where that doesn't work but if your job is to pump out code for an organization there's a good chance that most of your applications will have the same authentication, the same look and feel, etc. Putting effort into that core will pay dividends later. But you can't be a slave to your own code; if it doesn't fit in the box then don't force it.
A programmer only has so many hours in the day; if you want to be a more efficient programmer you either have to learn to type/think faster or you need to build frameworks, write libraries, and codify common practices.
There are situations where that doesn't work but if your job is to pump out code for an organization there's a good chance that most of your applications will have the same authentication, the same look and feel, etc. Putting effort into that core will pay dividends later. But you can't be a slave to your own code; if it doesn't fit in the box then don't force it.
> to be a more efficient programmer you either have to learn to type/think faster or you need to build frameworks, write libraries, and codify common practices.
That’s focused on the writing/creating side of the equation. In my side projects, I became a lot more efficient when I decided to put effort into using frameworks, adopting libraries, and copying common practices.
That’s focused on the writing/creating side of the equation. In my side projects, I became a lot more efficient when I decided to put effort into using frameworks, adopting libraries, and copying common practices.
You should absolutely be using existing frameworks, libraries, and common practices. But a popular framework will allow you make any application imaginable. If you're not making radically different applications every day, you can extend and constrain existing frameworks with your own code to make them specific to your needs.
For example, all my applications have the same look and feel, corporate branding, and authentication features. We use a popular framework but extended it to handle all of this stuff so it's never repeated between projects.
For example, all my applications have the same look and feel, corporate branding, and authentication features. We use a popular framework but extended it to handle all of this stuff so it's never repeated between projects.
While there is much to worry about over centralization, there are benefits to at least trying if you can consolidate n-m efforts when n is the total number efforts using a queue and m are the snowflakes. It all depends on organization size and whether some costs can be amortized.
Everyone having specialized everything is exceptionally expensive at scale. Here, the key is tower of babel. What Tim should have done is kick start the effort with one or two teams, then organically grown it by reducing operational burden for new teams.
The mistake here is trying to globally optimize rather than seeking different vertical consolidation.
By the way, this is why Amazon has so many services.
Everyone having specialized everything is exceptionally expensive at scale. Here, the key is tower of babel. What Tim should have done is kick start the effort with one or two teams, then organically grown it by reducing operational burden for new teams.
The mistake here is trying to globally optimize rather than seeking different vertical consolidation.
By the way, this is why Amazon has so many services.
[deleted]
I expect "Seeing Like a State" goes into it, but top-down global abstraction and standardization often does work. e.g. mass production, mass media, and States themselves are unbelievably, fantastically successful.
The point is more than it doesn't always work, as standardization is a poor substitute for actual understanding.
The point is more than it doesn't always work, as standardization is a poor substitute for actual understanding.
Very interesting article! I was left wondering though in what ways the queue abstraction solution failed?
for this specific example, I think the shared library is not the correct approach. Queues work for simple fifo behavior, but in this case they also need fairness (and perhaps other logic, like rate limiting per client, different priority for certain clients etc)
For example, "Customer-A is clogging up the work queue and starving other customers out". The solution to this could look something like linux's completely fair scheduler, where every client is allocated some amount of messages per interval of time. This means messages need to be processed in a different order then they are enqueued, and queues are not good at reordering messages.
I would suggest implementing the queue abstraction as a rest or grpc service, backed by a database like postgres, which holds message payloads and everything related to message state (in progress, retry after times, etc). Now we can implement all the necessary scheduling logic within our queue service.
For example, "Customer-A is clogging up the work queue and starving other customers out". The solution to this could look something like linux's completely fair scheduler, where every client is allocated some amount of messages per interval of time. This means messages need to be processed in a different order then they are enqueued, and queues are not good at reordering messages.
I would suggest implementing the queue abstraction as a rest or grpc service, backed by a database like postgres, which holds message payloads and everything related to message state (in progress, retry after times, etc). Now we can implement all the necessary scheduling logic within our queue service.
They hinted at the fact that kafka is not actually a queue, and it especially has problems with fairness if you try to use it as a queue for unequally sized "jobs" and/or unequally sized consumers. Kafka is for the exceptional case; actual message queues are for the default/general case.
That makes sense, but it also doesn't really support the article's overall argument about the perils of "standardization without understanding". If what you're saying is true then it could very well be that the approach to standardize the queuing would have worked perfectly well had the senior dev chosen a different technology to underlay his library (ie not Kafka), right?
Yes technically the Kafka matter is a secondary to the author's main point, which is understanding each problem first. So yeah I would not waltz in and say, "Oh, no need for details, obviously you need to standardize on RabbitMQ not Kafka," without a second thought, even though it's likely that's roughly where we're gonna end up.
But even then, I am not going to write a custom library when the drivers for either product are perfectly adequate. Standardization works best when standards are flexible, and trying to force my homemade library on every team is inflexible (usually it's just arrogance).
But even then, I am not going to write a custom library when the drivers for either product are perfectly adequate. Standardization works best when standards are flexible, and trying to force my homemade library on every team is inflexible (usually it's just arrogance).
This article is amazing and is perfectly timed for the news about Sri Lanka's mass move to organic farming - https://news.ycombinator.com/item?id=28443798
I thought it was interesting that an article that talks about the dangers of generalizing (aka standardizing) based on a few data points is, itself, generalizing based on a few data points.
Great short read and book recc.
I find myself often trying to articulate this idea (sometimes to myself, often to a friend) - starting with the map and building a territory from it --> generally bad.
While reading this article I wasn't even thinking about software. I think of a lot of progressive and/or socialist politics. To assert that the world should look as you one sees fit is flawed, both because one is not representative but mostly because the complexity of a system beyond toy example cannot be accurately modeled enough when the risk is catastrophe.
I find myself often trying to articulate this idea (sometimes to myself, often to a friend) - starting with the map and building a territory from it --> generally bad.
While reading this article I wasn't even thinking about software. I think of a lot of progressive and/or socialist politics. To assert that the world should look as you one sees fit is flawed, both because one is not representative but mostly because the complexity of a system beyond toy example cannot be accurately modeled enough when the risk is catastrophe.
But I would stress less that "Seeing Like a State" -- that is a top-down, global solution -- was not the problem.
The problem was that "Tim" didn't really understand the problem he was trying to solve (well, none of us truly understand very much at all, but he didn't understand it better than many of the teams associated with the individual services).
"Tim"'s proposal probably solved some problems but created various other problems.
The best solution, though, (IMO) isn't that Tim should be smarted and better informed than everyone else combined, nor that every team should continue to create an independent solution. Instead "Tim" could propose a solution, and the 100 micro service teams would be tasked with responding constructively. Iterations would ensue. You still really, really need "Tim", though, because multiple teams, even sincere and proficient ones, will not arrive at a coherent solution without leadership/direction.
> A global solution, by necessity, has to ignore local conditions.
That's just flat wrong. A global solution can solve global concerns and also allow for local conditions.