Clever vs. Insightful Code(hillelwayne.com)
hillelwayne.com
Clever vs. Insightful Code
https://www.hillelwayne.com/post/cleverness/
118 comments
It's a nice sentiment... for running a military. Or managing a company with very high turnover. I don't think it maps well to software development.
The soldiers on the front lines spent years being trained to obey orders first, ask questions later. This is necessary because they're risking their lives, and explaining to each and every one of your grunts why it's important they do the thing that's likely to kill them, does not scale. You won't achieve your policy goals if your soldiers each decide the war isn't worth fighting, the orders aren't worth risking lives for.
Software developers spend years training (or self-training) to use their brains to solve complex problems, which involves continuous learning, constantly making "tactical" decisions, and quickly gaining understanding of the big picture. It's a shame to first filter people by how good they're at complex technical work, and then make them stop using these skills, for the sake of shortening onboarding time.
The soldiers on the front lines spent years being trained to obey orders first, ask questions later. This is necessary because they're risking their lives, and explaining to each and every one of your grunts why it's important they do the thing that's likely to kill them, does not scale. You won't achieve your policy goals if your soldiers each decide the war isn't worth fighting, the orders aren't worth risking lives for.
Software developers spend years training (or self-training) to use their brains to solve complex problems, which involves continuous learning, constantly making "tactical" decisions, and quickly gaining understanding of the big picture. It's a shame to first filter people by how good they're at complex technical work, and then make them stop using these skills, for the sake of shortening onboarding time.
I think you may be misunderstanding "simple". Simple works great for complex software because simple functions and features are assembled together into the fully complex solution. As a matter of fact, that is what makes a great dev - someone who can envision a complex solution, but then break it down into simple programming tasks.
Ideally solutions should be as simple as possible as well.
The ultimate core algorithm they're wrapped around might be particularly clever / insightful. It might even be mildly awful to wrap your head around -- but necessarily so.
But then all the stuff you wrap around that to produce libraries to build into production code / shippable product should be pretty brain dead stupid as possible. That way people who sit down with your code find things all the places they expect and other than possibly the little nightmare in the middle of it, it is all straightforward and readable.
The ultimate core algorithm they're wrapped around might be particularly clever / insightful. It might even be mildly awful to wrap your head around -- but necessarily so.
But then all the stuff you wrap around that to produce libraries to build into production code / shippable product should be pretty brain dead stupid as possible. That way people who sit down with your code find things all the places they expect and other than possibly the little nightmare in the middle of it, it is all straightforward and readable.
Just because you're smart and in the front lines, that doesn't mean you _need_ to take the complex route every time. It takes skill to do things the simple way, to know when you just need to get shit done and when you can/need to be fancy.
Let's say I want a fence for my yard.
The newbie carpenter wants to show off their skills and makes the most beautiful delicate lattice work with intricate fibonacci patterns.
An experienced carpenter sees that I have 4 huge dogs and determines that I need a sturdy fence, not a fancy one.
This is the way I do things at work:
First you make the simple and boring solution, the one that works. Then you see if it needs to be faster, most times it doesn't. After that, if you have time, you can spend time making the simple code prettier, more elegant and expressive.
In 99% of the cases I don't need to go past step 1, the simple solution works just fine.
Let's say I want a fence for my yard.
The newbie carpenter wants to show off their skills and makes the most beautiful delicate lattice work with intricate fibonacci patterns.
An experienced carpenter sees that I have 4 huge dogs and determines that I need a sturdy fence, not a fancy one.
This is the way I do things at work:
1) Make it work
2) Make it fast
3) Make it pretty
First you make the simple and boring solution, the one that works. Then you see if it needs to be faster, most times it doesn't. After that, if you have time, you can spend time making the simple code prettier, more elegant and expressive.
In 99% of the cases I don't need to go past step 1, the simple solution works just fine.
Of course. But I disagree with the general sentiment that "clever = fancy", or "a junior may have not seen this technique before" = "fancy" = "bad".
A lot of "advanced" techniques and features of tooling exists because they enable simpler code. Simpler, in the sense of better expressing intent, more readable, less bug-prone. For example, a map-reduce on a sequence will typically be simpler than an explicit loop - because it expresses only the things that matter, whereas an explicit for loop also emphasizes irrelevant details and provide opportunities to introduce bugs. And yet, a random junior is going to consider the latter to be "simpler" - but that's only because they're unfamiliar with more advanced programming concepts. The solution to that isn't to abandon advanced concepts - it's to teach the junior to work with them.
> The newbie carpenter wants to show off their skills and makes the most beautiful delicate lattice work with intricate fibonacci patterns.
Maybe. Regardless, the newbie carpenter will be identified by doing more work than necessary, using less sophisticated tools and techniques. The experienced carpenter will show up with a richer set of tools, and perhaps apply some non-standard technique to put it up and secure against your dogs. He'll be done faster, and tire themselves less.
> Make it work / Make it fast / Make it pretty
Yes, this is a good approach, and I'd say programming experience manifests itself in the following ways:
- A richer set of tools allows an experienced programmer to get the thing to work faster.
- Their experience allows them to simultaneously also make the code somewhat fast and somewhat pretty, because quite often, making things fast and pretty requires the same amount of effort as slow and ugly, if you have some experience to help you make the right choices.
- An experienced professional will not habitually stop after the "make it work" phase, unless made so by high deadline pressure.
A lot of "advanced" techniques and features of tooling exists because they enable simpler code. Simpler, in the sense of better expressing intent, more readable, less bug-prone. For example, a map-reduce on a sequence will typically be simpler than an explicit loop - because it expresses only the things that matter, whereas an explicit for loop also emphasizes irrelevant details and provide opportunities to introduce bugs. And yet, a random junior is going to consider the latter to be "simpler" - but that's only because they're unfamiliar with more advanced programming concepts. The solution to that isn't to abandon advanced concepts - it's to teach the junior to work with them.
> The newbie carpenter wants to show off their skills and makes the most beautiful delicate lattice work with intricate fibonacci patterns.
Maybe. Regardless, the newbie carpenter will be identified by doing more work than necessary, using less sophisticated tools and techniques. The experienced carpenter will show up with a richer set of tools, and perhaps apply some non-standard technique to put it up and secure against your dogs. He'll be done faster, and tire themselves less.
> Make it work / Make it fast / Make it pretty
Yes, this is a good approach, and I'd say programming experience manifests itself in the following ways:
- A richer set of tools allows an experienced programmer to get the thing to work faster.
- Their experience allows them to simultaneously also make the code somewhat fast and somewhat pretty, because quite often, making things fast and pretty requires the same amount of effort as slow and ugly, if you have some experience to help you make the right choices.
- An experienced professional will not habitually stop after the "make it work" phase, unless made so by high deadline pressure.
>> The soldiers on the front lines spent years being trained to obey orders first, ask questions later.
I think you have the wrong impression of the army. Decentralized command is one of the central tenants of American military doctrine. Trying to route everything through layers of leadership leads to an incompetent fighting unit. Unfortunately, this stereotype is hard to shake.
As a result the soldier must understand the broader objective and actively question the tactics used to achieve it. If the ground situation deviates from the plan, waiting for orders will get you killed.
I would argue software development is surprisingly similar. The difference between a decent developer and a great developer is the latter understand how their work fits into the larger picture. As a result they're able to quickly and effectively pivot to suit the dynamic needs of the business.
I think you have the wrong impression of the army. Decentralized command is one of the central tenants of American military doctrine. Trying to route everything through layers of leadership leads to an incompetent fighting unit. Unfortunately, this stereotype is hard to shake.
As a result the soldier must understand the broader objective and actively question the tactics used to achieve it. If the ground situation deviates from the plan, waiting for orders will get you killed.
I would argue software development is surprisingly similar. The difference between a decent developer and a great developer is the latter understand how their work fits into the larger picture. As a result they're able to quickly and effectively pivot to suit the dynamic needs of the business.
I am familiar with US decentralized command doctrine, which pushed a lot of decision making to the very front lines. But that's a Cold War-era American invention, and 'theshrike79 quoted a Finnish lieutenant general, who fought before and led through World War II. Back then, soldiers had much less autonomy.
And even if somebody ported the US doctrine 100 years into the past, so that LtGen Airo could've learned it in France, it probably wouldn't hold, because there was only so much a front line soldier could do with their decision making capacity. Tanks were a fresh invention, most of the world probably haven't seen them. Cavalry was the pinnacle of agility. Artillery was big, bulky, and hard to reposition. Planes were still laughed at. Portable radios weren't a thing yet. Where your modern American grunt is entrusted with the capability to direct howitzers and call down air strikes, the soldiers of Airo's era were just discovering automated weapons, and modes of transport other than "by foot" and "by horse". There was little they could really decide about.
In other words: soldiers gained greater autonomy and more sophisticated training when their tools became more sophisticated and powerful. As it should be - as it is, in any sane industry. If there was any officer that said things like, "those Stingers and Javelins are too complex, they'll confuse our new privates", or "why waste your time learning to use night vision gear when you could run with a flashlight and shoot the enemy", or "giving front line soldiers keys to artillery strikes, that's crazy!" - that officer was ignored, as others recognized that people can be trained in doctrine, tactics and operating complex equipment.
> The difference between a decent developer and a great developer is the latter understand how their work fits into the larger picture.
Strongly agreed. But 'theshrike79 wasn't talking about that. They were talking about managing men who weren't given the insight into larger picture, because there was nothing useful they could do with it anyway. That was true for the military then. It isn't now. It also isn't true for software development - but there's this persistent trend to make it true, to turn us from modern fighting force into a 17th century line infantry.
And even if somebody ported the US doctrine 100 years into the past, so that LtGen Airo could've learned it in France, it probably wouldn't hold, because there was only so much a front line soldier could do with their decision making capacity. Tanks were a fresh invention, most of the world probably haven't seen them. Cavalry was the pinnacle of agility. Artillery was big, bulky, and hard to reposition. Planes were still laughed at. Portable radios weren't a thing yet. Where your modern American grunt is entrusted with the capability to direct howitzers and call down air strikes, the soldiers of Airo's era were just discovering automated weapons, and modes of transport other than "by foot" and "by horse". There was little they could really decide about.
In other words: soldiers gained greater autonomy and more sophisticated training when their tools became more sophisticated and powerful. As it should be - as it is, in any sane industry. If there was any officer that said things like, "those Stingers and Javelins are too complex, they'll confuse our new privates", or "why waste your time learning to use night vision gear when you could run with a flashlight and shoot the enemy", or "giving front line soldiers keys to artillery strikes, that's crazy!" - that officer was ignored, as others recognized that people can be trained in doctrine, tactics and operating complex equipment.
> The difference between a decent developer and a great developer is the latter understand how their work fits into the larger picture.
Strongly agreed. But 'theshrike79 wasn't talking about that. They were talking about managing men who weren't given the insight into larger picture, because there was nothing useful they could do with it anyway. That was true for the military then. It isn't now. It also isn't true for software development - but there's this persistent trend to make it true, to turn us from modern fighting force into a 17th century line infantry.
This reminded me a lot of Jessitron's latest blog post.[0]
[0]:https://jessitron.com/2021/05/26/a-high-resilience-org-chart...
[0]:https://jessitron.com/2021/05/26/a-high-resilience-org-chart...
In my example the coder would be the general and the soldiers are the code :)
The easy way is to use all of the latest fancy gizmos, you'll have microservices on a kubernetes cluster and a sharded globally distributed schemaless database - all done with $hype_language and 4200 different npm packages. Because you don't have the knowledge or experience to cut through all the hype and masses of different options.
You need to have experience, education and/or training to come up with the simplest possible solution.
The easy way is to use all of the latest fancy gizmos, you'll have microservices on a kubernetes cluster and a sharded globally distributed schemaless database - all done with $hype_language and 4200 different npm packages. Because you don't have the knowledge or experience to cut through all the hype and masses of different options.
You need to have experience, education and/or training to come up with the simplest possible solution.
> Software developers spend years training (or self-training) to use their brains to solve complex problems, which involves continuous learning, constantly making "tactical" decisions, and quickly gaining understanding of the big picture.
I'm just not sure that applies to all people working in software engineering (or any job). I know too many that have no desire to actually understand the/a problem or make tactical decisions but just try out ten code snippets from SO until something works, use any plugin that kinda fits their need (without looking at the size of its code base, if its maintained, how flexible its configuration is etc) or are happy to use any framework as long as it tells them exactly what to do (and not because they actually came to the conclusion that's the way to go, but because they are not responsible for anything but the framework is).
I'm just not sure that applies to all people working in software engineering (or any job). I know too many that have no desire to actually understand the/a problem or make tactical decisions but just try out ten code snippets from SO until something works, use any plugin that kinda fits their need (without looking at the size of its code base, if its maintained, how flexible its configuration is etc) or are happy to use any framework as long as it tells them exactly what to do (and not because they actually came to the conclusion that's the way to go, but because they are not responsible for anything but the framework is).
If a problem can be reasonably solved with in either a dull or clever way then there is no advantage to choosing the clever way. It is pretty much all downside.
> It's a shame to first filter people by how good they're at complex technical work, and then make them stop using these skills, for the sake of shortening onboarding time.
Onboarding time isn't exactly the reason, the reason is that the costs of clever code are higher than the benefits. Longer onboarding time is a cost. One of many that come with not using basic techniques. It is a point of professionalism to target the best outcome for a situation considering factors other than how fun it would be to show off the code.
If the benefits of complex code are higher than the costs then it gets used. But basic code is generally more effective.
> It's a shame to first filter people by how good they're at complex technical work, and then make them stop using these skills, for the sake of shortening onboarding time.
Onboarding time isn't exactly the reason, the reason is that the costs of clever code are higher than the benefits. Longer onboarding time is a cost. One of many that come with not using basic techniques. It is a point of professionalism to target the best outcome for a situation considering factors other than how fun it would be to show off the code.
If the benefits of complex code are higher than the costs then it gets used. But basic code is generally more effective.
Maybe not so much to software development, but exactly right to software lifecycle.
This is so true.
We need to stop perpetuating the leadership lie in white collar jobs.
We need to stop perpetuating the leadership lie in white collar jobs.
When I advise people to write simpler code I do mean "not clever." Nor do I mean "insightful." In fact, please make it so obvious that in reading it I don't gain/need/curse/applaud any insights.
Make it simple enough that a brand new completely green junior member of the team immediately understands it on first reading.
And if you can't make the code completely obvious (e.g. the sorting example), just add a comment. A simple one.
None of this is about the creator's cleverness or deep understanding or speed or what tooling can do. It's not about the creator or the machine at all. Simple code is about clarity for all the people who will read and (hopefully not) debug it later.
Make it simple enough that a brand new completely green junior member of the team immediately understands it on first reading.
And if you can't make the code completely obvious (e.g. the sorting example), just add a comment. A simple one.
None of this is about the creator's cleverness or deep understanding or speed or what tooling can do. It's not about the creator or the machine at all. Simple code is about clarity for all the people who will read and (hopefully not) debug it later.
> Make it simple enough that a brand new completely green junior member of the team immediately understands it on first reading.
This is fine for writing business logic, but this is not good for domains that require insightful code in order to execute within certain time constraints. Some problems require domain knowledge and expertise, which means that you cannot as easily hire and treat programmers as replaceable commodities of junior level people.
This is fine for writing business logic, but this is not good for domains that require insightful code in order to execute within certain time constraints. Some problems require domain knowledge and expertise, which means that you cannot as easily hire and treat programmers as replaceable commodities of junior level people.
Code that's under time constraints especially needs good documentation. Do you still know the constraints of code you last touched 5 years ago? I don't.
As for your comment about the commodities. Yes, it makes it easier to onboard people, but by this, employers also make it easier for people to switch jobs and negotiate better terms. So I don't agree that this necessarily creates a worse situation for employees.
As for your comment about the commodities. Yes, it makes it easier to onboard people, but by this, employers also make it easier for people to switch jobs and negotiate better terms. So I don't agree that this necessarily creates a worse situation for employees.
Do keep in mind that there's a difference between reading and writing code.
You might very well need experience and domain-level knowledge to write the code, but often you can still make it readable to beginners.
Of course, 'often' doesn't mean always. See eg https://en.wikipedia.org/wiki/Fast_inverse_square_root for an example of code that falls in this gap.
You might very well need experience and domain-level knowledge to write the code, but often you can still make it readable to beginners.
Of course, 'often' doesn't mean always. See eg https://en.wikipedia.org/wiki/Fast_inverse_square_root for an example of code that falls in this gap.
Yes but this can be harder than writing good documentation along side the code. For example I work with software that automates CAD manufacturing. Knowing what side of the part is side A is hard to communicate. But we have drawings that help explain it all. I've yet to find a way to have this embedded into the code. Asci art comments are as good as I've gotten.
> Asci art comments are as good as I've gotten.
This is good, and people should use it more. But, to go further than that, I recommend:
- Setting an official, designated place for rich documentation related to code. A wiki, or a folder in a repo, or a separate repo - either works, as long as every developer is made aware the thing exists and is important.
- Put references to documentation in that place in your code comments.
Having done that, you can now put in your comments things like:
--
On the ASCII diagrams, it might be worth checking if your specific case allows richer content to be embedded in code comments directly. For example, in the past, I worked in a Java shop, and realized I can embed images in the comments - the IDE I used would parse data URLs (like: "data:image/png;base64,iVBORw...") in one JavaDoc tag (don't remember which) and render the image in the tooltip! I only ever did embed a cat picture like this, because I soon realized not all Java IDEs will display the images - but if your team standardized on tooling that can handle it, it's an easy way to sneak in proper diagrams.
It would be great if IDEs came with PlantUML support in code comments - there's a bunch of diagrams I have now that I'd love to embed in the source code, but can't, because nobody else can make their tool handle them automatically.
(In general, it's worth investigating team/tooling-specific options you have for quality-of-life enhancements. For example, even though everyone in my current team uses different tooling for C++, they all use the same debugger - the one in Visual Studio. So I started producing Natvis definitions for some of the complex types we use, as well as error codes from proprietary APIs. While not exactly documentation, they remove some type noise and save you from searching for error codes.)
This is good, and people should use it more. But, to go further than that, I recommend:
- Setting an official, designated place for rich documentation related to code. A wiki, or a folder in a repo, or a separate repo - either works, as long as every developer is made aware the thing exists and is important.
- Put references to documentation in that place in your code comments.
Having done that, you can now put in your comments things like:
// Which side is 'A' and which is 'B' is determined by
// Some Method. The code below implements this logic.
// The exact explanation (with diagrams) of the method can
// be found in docs:/Algorithms/Some Method.docx
--
On the ASCII diagrams, it might be worth checking if your specific case allows richer content to be embedded in code comments directly. For example, in the past, I worked in a Java shop, and realized I can embed images in the comments - the IDE I used would parse data URLs (like: "data:image/png;base64,iVBORw...") in one JavaDoc tag (don't remember which) and render the image in the tooltip! I only ever did embed a cat picture like this, because I soon realized not all Java IDEs will display the images - but if your team standardized on tooling that can handle it, it's an easy way to sneak in proper diagrams.
It would be great if IDEs came with PlantUML support in code comments - there's a bunch of diagrams I have now that I'd love to embed in the source code, but can't, because nobody else can make their tool handle them automatically.
(In general, it's worth investigating team/tooling-specific options you have for quality-of-life enhancements. For example, even though everyone in my current team uses different tooling for C++, they all use the same debugger - the one in Visual Studio. So I started producing Natvis definitions for some of the complex types we use, as well as error codes from proprietary APIs. While not exactly documentation, they remove some type noise and save you from searching for error codes.)
Not sure which language you use, but At least in java you can embed images into the javadoc comment and they get rendered by the IDE when you hover over a method or class. Comes in handy when you need to describe Boolean mesh operations for example.
Donald Knuth thought about this problem before:
https://en.wikipedia.org/wiki/Literate_programming (and https://wiki.c2.com/?LiterateProgramming)
https://en.wikipedia.org/wiki/Literate_programming (and https://wiki.c2.com/?LiterateProgramming)
Make code obvious such that someone else tackling the same problem could write the same obvious structure and solution. Writing the naive code first, then optimizing, makes it more obvious how to accomplish this, including non-obvious comments and references.
So make code obvious first to reduce WTF's later.
So make code obvious first to reduce WTF's later.
This never works. You cannot iteratively progress from a wrong algorithm to a good one in general. You can maybe optimize things by cutting off problem space, but that does not actually work in many problems.
For example, quicksort and radix sort have nothing in common besides ordering things. Likewise treating a tree like a general graph won't get you to a B-tree. Trying to simulate behavior of an array with a single linked list requires an expensive and smart memory allocator. Etc.
It gets even more complex if we're talking real-time software, especially multithreaded. Single thread solution does not appear even remotely similar to a performant, concurrent, ordering tolerant code.
For example, quicksort and radix sort have nothing in common besides ordering things. Likewise treating a tree like a general graph won't get you to a B-tree. Trying to simulate behavior of an array with a single linked list requires an expensive and smart memory allocator. Etc.
It gets even more complex if we're talking real-time software, especially multithreaded. Single thread solution does not appear even remotely similar to a performant, concurrent, ordering tolerant code.
It is as you say problem dependent. In a high level language, you could start off with a generic std lib sort. Or you could start off with the intended sort algo and improve on it. The point is to make it more or less straight-forward and readable first. Then you find optimizations, inlining, etc., keeping intentions in comments and the like.
For multithreaded there are design and tactics that can be documented separately, as comments would probably be too spread out anyways to be too useful.
This is hard when working iteratively, as you have to work out a working solution before you know it. The problem is after tough work to end up with something working, but unreadable and less maintainable. The ideal would be to have the time to rewrite and retest, and not etch that magic brilliant code in stone. Ie. the linux kernel is a good example of code being iteratively refactored and rewritten many times. Though as you say, it'd require quite some time diving deep into such code anyway. Doing what works well should make it accessible to more than one dev.
So the point is just so that others can access and modify the code a bit easier, even for tough sections. This goes very much against the grain, because of exhaustion and not wanting to take on further risks and rework.
For multithreaded there are design and tactics that can be documented separately, as comments would probably be too spread out anyways to be too useful.
This is hard when working iteratively, as you have to work out a working solution before you know it. The problem is after tough work to end up with something working, but unreadable and less maintainable. The ideal would be to have the time to rewrite and retest, and not etch that magic brilliant code in stone. Ie. the linux kernel is a good example of code being iteratively refactored and rewritten many times. Though as you say, it'd require quite some time diving deep into such code anyway. Doing what works well should make it accessible to more than one dev.
So the point is just so that others can access and modify the code a bit easier, even for tough sections. This goes very much against the grain, because of exhaustion and not wanting to take on further risks and rework.
[deleted]
Thankfully, that represents only a fraction of the software engineering domain.
Also, just because it's insightful doesn't mean it should be illegible. Yeah it may go down a level of abstraction and use assembly and SIMD code, but you can then write it in that language, at that level, instead of trying to be even more clever. And of course wrap it in a function name that describes what it does, accompanied by documentation that explains why it's implemented the way it is.
Also, just because it's insightful doesn't mean it should be illegible. Yeah it may go down a level of abstraction and use assembly and SIMD code, but you can then write it in that language, at that level, instead of trying to be even more clever. And of course wrap it in a function name that describes what it does, accompanied by documentation that explains why it's implemented the way it is.
True, but if the code is that subtle then you still need to write it as simply as possible, otherwise even your ever-so-clever senior programmers will struggle to understand it fully.
And unfortunately, only a tiny percentage of code in even a really interesting project will be anywhere near this tricky.
And unfortunately, only a tiny percentage of code in even a really interesting project will be anywhere near this tricky.
Make it simple enough that a brand new completely green junior member of the team immediately understands it on first reading.
This is why the software industry continues to decline: it's self-inflicted dumbing-down. I really don't understand where this rampant support for antiintellectualism comes from in the software industry, besides managers who want to make employees easily replaceable "cogs in the machine"; and dread what the world would look like if everyone was encouraged to do the same in other (natural) languages.
This is why the software industry continues to decline: it's self-inflicted dumbing-down. I really don't understand where this rampant support for antiintellectualism comes from in the software industry, besides managers who want to make employees easily replaceable "cogs in the machine"; and dread what the world would look like if everyone was encouraged to do the same in other (natural) languages.
It's because the vast, vast majority of code is not and need not be groundbreaking computer science, it's boring business logic and glue code, and it's built in teams of regular people. We aren't trying to save the world with efficiency and cleverness, we are trying to be robust and understood.
There is a time and a place for innovation of course, but you will be in a unique position when it happens. Perhaps you have a very interesting problem to solve, an opportunity to automate some part of the building, maybe you are architecting the system, or you are writing code for fun and not in a team. But you will know it when you see it, and if you can't tell then "write simply" is actually fantastic advice for you because you might be over-engineering some really boring stuff.
It is genuinely good, pragmatic advice to execute most code as simply as possible because that is all it needs to be, and your code will probably be worked on by others in the team at some point and you are wasting their time if they have to pontificate about the meaning of life before understanding your code.
There is a time and a place for innovation of course, but you will be in a unique position when it happens. Perhaps you have a very interesting problem to solve, an opportunity to automate some part of the building, maybe you are architecting the system, or you are writing code for fun and not in a team. But you will know it when you see it, and if you can't tell then "write simply" is actually fantastic advice for you because you might be over-engineering some really boring stuff.
It is genuinely good, pragmatic advice to execute most code as simply as possible because that is all it needs to be, and your code will probably be worked on by others in the team at some point and you are wasting their time if they have to pontificate about the meaning of life before understanding your code.
I think you're right, but also the developers you speak of are on inflated salaries, brought about by their proximity and similarity to people who do need to value efficiency and cleverness. Run of the mill software developer won't be a good job in a few years once employers realise this.
> Run of the mill software developer won't be a good job in a few years once employers realise this.
It's interesting you bring this up, as one of my long term strands of thinking is that software needs to be more accessible to the greater community, which involves removing the hurdle that is inflated wages. It's my thinking that there's much useful software that doesn't get written because it's not affordable to do so for regular companies with regular profit margins.
So I think you're right, but I cautiously welcome the sea change. I can't tell the future so maybe it won't turn out beneficial at all.
It's interesting you bring this up, as one of my long term strands of thinking is that software needs to be more accessible to the greater community, which involves removing the hurdle that is inflated wages. It's my thinking that there's much useful software that doesn't get written because it's not affordable to do so for regular companies with regular profit margins.
So I think you're right, but I cautiously welcome the sea change. I can't tell the future so maybe it won't turn out beneficial at all.
I assumed since I got involved in computers (about 1982), and for quite a few years thereafter, that the trade of programming would become obsolete. The arrival of the Lisa, mouse, Mac, GUI encouraged me in the view that soon, anyone would be able to program if they wanted.
What I missed is that the main work of a developer is to understand the problem (maybe that's what he means by "insight"? - seems a bit grandiose to me).
What I missed is that the main work of a developer is to understand the problem (maybe that's what he means by "insight"? - seems a bit grandiose to me).
If the software is not economical to write then it can't be all that important to the business, right? And if wages are really the problem then wouldn't we expect a lot of such software to pop up outside of silicon valley, where wages for software developers tend to be considerably lower already?
I think that's a hard question to answer outright, it's going to depend, but you're probably right for a lot of cases.
A recent example, a local school needed a new stock management tool for the school invetory, pretty simple stuff, but at a fairly reasonable contractor rate it was far and away too expensive for them.
The economic trick we use is to turn it into a SaaS and get a bunch of people on board, but sometimes the problem is too unique or you don't want to take on a SaaS. You just want a small piece of software embedded in the business.
There's lots of those small software opportunities in the community, I just think it would be cool to action them.
A recent example, a local school needed a new stock management tool for the school invetory, pretty simple stuff, but at a fairly reasonable contractor rate it was far and away too expensive for them.
The economic trick we use is to turn it into a SaaS and get a bunch of people on board, but sometimes the problem is too unique or you don't want to take on a SaaS. You just want a small piece of software embedded in the business.
There's lots of those small software opportunities in the community, I just think it would be cool to action them.
I have no experience in that area myself, but I assume most businesses (and schools!) track inventory. I would expect there to be plenty of commercial off-the-shelf inventory tracking solutions by now. Isn't that the case?
Sort of, it's true there are lots of inventory tracking tools available but this school needed particular requirements that they couldn't find. Some of their requirments were that it needed to:
- Run on their intranet
- Run on a server they own
- Not use user accounts but rather individual pins, anyone can use any pin, they need to match an already used pin system at the school
- Work like a kiosk for ordering, enter pin, add items to a perpetual order
- Track internal budgeting and generate reports, nonone actually pays or has limits but they wanted to see howmuch budget Pin X was using etc.
- Purchase once, no subscriptions
So that ruled out SaaS and since almost every modern software company is SaaS or just a desktop application they weren't able to find a reasonable replacement.
- Run on their intranet
- Run on a server they own
- Not use user accounts but rather individual pins, anyone can use any pin, they need to match an already used pin system at the school
- Work like a kiosk for ordering, enter pin, add items to a perpetual order
- Track internal budgeting and generate reports, nonone actually pays or has limits but they wanted to see howmuch budget Pin X was using etc.
- Purchase once, no subscriptions
So that ruled out SaaS and since almost every modern software company is SaaS or just a desktop application they weren't able to find a reasonable replacement.
Fair enough. I don't really see the advantage of pins over user accounts, and wonder how long it will take them to recoup the costs of developing a custom solution over just paying for a subscription. But if their requirements are not negotiable then it's understandable that custom software will be required.
Software engineering is broad; the glue code developers are more focused on the domain, the broad overview, whereas the nitty gritty ones will look at the smaller fragments of a bigger problem. Don't undervalue developers just because they're not writing leetcode as their day job.
This hasn't been my experience at all. Plumber-developers are generally terrible at architecting systems, whereas confident low-level/algorithms programmers tend to also be very good at designing architectures.
Personally, I think it's because development is more fractal than people give it credit for, but I think it's also that plumber-developers tend to be intellectually lazy, which is why they specialised in code-plumbing.
Personally, I think it's because development is more fractal than people give it credit for, but I think it's also that plumber-developers tend to be intellectually lazy, which is why they specialised in code-plumbing.
This sounds like elitism in disguise to me.
Sure, there are some cases when you want quick and dirty and just glue some system together, but most production code out there has some more important business requirement than "understandable by the cheapest engineers out there".
For instance, if you're writing an account management system for a regional bank, you'll care most about the accuracy and longevity (including easy maintainability) of the system.
If you're writing a microservice for a fancy web app with global distribution you might care about latency (high latency drives down CVR), reliability (errors drive down sales and ads too) and sustained agility (you need to develop features fast to keep ahead of competition).
I think the second example covers most of what web and mobile developers do. I've definitely seen cases of over-engineered systems with many layers of leaky abstractions, but also many cases of under-engineered systems. Here are some well-documented maladies:
1. RYE (Repeat Yourself Everywhere) - You have the same business logic repeated in multiple places, because originally it didn't seem common enough or large enough to warrant DRYing up. This is obviously easier to read, since you don't need to dive deeper into more functions, but in practice the shared logic quickly diverges between the different cases, until it's very hard to specify what your system does.
2. "Let's just add an if branch here for this special case" - quick and dirty, as unclever as it can be, until you realize you need to deal with the combined permutations of 20 different branches. This is readable only in the very surface sense.
3. "Our junior engineers understand for loops better than map-filter-reduce chains, so let's use for loops instead of spending a few days teaching them": You can replace "for loops" with anything else that your junior engineers happen to know. The end result is not avoiding an over-engineered or "too clever" solution, but rather just avoiding a solution that is often simpler and easier to understand but just happens to be unfamiliar to your engineer. See also "blub paradox".
4. "This 500-line function is brain-dead simple and uses no fancy tricks". Likewise, it's easier for a junior engineer to write long-winded code and avoid thinking about even the simplest abstractions. And the code works! This doesn't make the code more maintainable or reliable though.
5. "Let's add ad-hoc retry with for loops and branches when necessary instead of creating an abstraction based on closures or let alone monads - this is just too clever". End result: reliability is added only after somebody complains about a certain functionality instead of being designed and baked into the service, and your service suffers accordingly.
There are many more examples, but the general gist is that in almost every corner of our industry there are important concerns that require us to take our engineering practice seriously. I think statements like "It's all just glue code", "It's all just plumbing" or "It's just a CRUD app" are not constructive for quality.
I wholly agree with you about simplifying things. I just think we should be mindful of the difference between "simple" and "easy" (as Rich Hickey famously put it[1]). What is easy for a junior developer to understand (because it doesn't contain any concept they haven't learned yet) may increase the absolute complexity. If you choose the "easy" solution here, you make the code seem _subjectively_ simpler to junior developers, at the price of _objectively_ (and measurably!) increasing cyclomatic, cognitive or state complexity.
Unless the "clever solution" requires understanding that is beyond what we can expect the reasonable average engineer to learn quickly, I prefer to always err on the side of reducing absolute complexity, and code that is simpler to read _for senior engineers_.
[1] https://www.infoq.com/presentations/Simple-Made-Easy/
Sure, there are some cases when you want quick and dirty and just glue some system together, but most production code out there has some more important business requirement than "understandable by the cheapest engineers out there".
For instance, if you're writing an account management system for a regional bank, you'll care most about the accuracy and longevity (including easy maintainability) of the system.
If you're writing a microservice for a fancy web app with global distribution you might care about latency (high latency drives down CVR), reliability (errors drive down sales and ads too) and sustained agility (you need to develop features fast to keep ahead of competition).
I think the second example covers most of what web and mobile developers do. I've definitely seen cases of over-engineered systems with many layers of leaky abstractions, but also many cases of under-engineered systems. Here are some well-documented maladies:
1. RYE (Repeat Yourself Everywhere) - You have the same business logic repeated in multiple places, because originally it didn't seem common enough or large enough to warrant DRYing up. This is obviously easier to read, since you don't need to dive deeper into more functions, but in practice the shared logic quickly diverges between the different cases, until it's very hard to specify what your system does.
2. "Let's just add an if branch here for this special case" - quick and dirty, as unclever as it can be, until you realize you need to deal with the combined permutations of 20 different branches. This is readable only in the very surface sense.
3. "Our junior engineers understand for loops better than map-filter-reduce chains, so let's use for loops instead of spending a few days teaching them": You can replace "for loops" with anything else that your junior engineers happen to know. The end result is not avoiding an over-engineered or "too clever" solution, but rather just avoiding a solution that is often simpler and easier to understand but just happens to be unfamiliar to your engineer. See also "blub paradox".
4. "This 500-line function is brain-dead simple and uses no fancy tricks". Likewise, it's easier for a junior engineer to write long-winded code and avoid thinking about even the simplest abstractions. And the code works! This doesn't make the code more maintainable or reliable though.
5. "Let's add ad-hoc retry with for loops and branches when necessary instead of creating an abstraction based on closures or let alone monads - this is just too clever". End result: reliability is added only after somebody complains about a certain functionality instead of being designed and baked into the service, and your service suffers accordingly.
There are many more examples, but the general gist is that in almost every corner of our industry there are important concerns that require us to take our engineering practice seriously. I think statements like "It's all just glue code", "It's all just plumbing" or "It's just a CRUD app" are not constructive for quality.
I wholly agree with you about simplifying things. I just think we should be mindful of the difference between "simple" and "easy" (as Rich Hickey famously put it[1]). What is easy for a junior developer to understand (because it doesn't contain any concept they haven't learned yet) may increase the absolute complexity. If you choose the "easy" solution here, you make the code seem _subjectively_ simpler to junior developers, at the price of _objectively_ (and measurably!) increasing cyclomatic, cognitive or state complexity.
Unless the "clever solution" requires understanding that is beyond what we can expect the reasonable average engineer to learn quickly, I prefer to always err on the side of reducing absolute complexity, and code that is simpler to read _for senior engineers_.
[1] https://www.infoq.com/presentations/Simple-Made-Easy/
You bring up some great examples, I think we do agree in principle. I must point out that I didn't really specify what simple was to me but it's quite a nuanced topic as your examples show. I just mean keep it reasonable given the context.
Why I think we do agree is that what I really love to see is actually idiomatic code. Does the software you're working in already use array map/filter? Then go right ahead. Could the types of developers working on the project be reasonably expected to learn it? Then also, go right ahead!
I prefer to use frameworks for commercial projects because of this reason. I can hire more talented developers because I can test to see if they know the idioms well, and I can also be happy investing in their learning of the framework knowing that it's going to be repeatably useful for them while working on the project, and can upskill the whole team over time. You're essentially defining a body of knowledge expected by picking a framework.
Similarly when you're architecting software you're essentially choosing what level of developer you're going to need to hire or train, and so if you're in that position it's smart to be cautious of increasing the burden of knowledge too much unless you're happy hiring for that.
Why I think we do agree is that what I really love to see is actually idiomatic code. Does the software you're working in already use array map/filter? Then go right ahead. Could the types of developers working on the project be reasonably expected to learn it? Then also, go right ahead!
I prefer to use frameworks for commercial projects because of this reason. I can hire more talented developers because I can test to see if they know the idioms well, and I can also be happy investing in their learning of the framework knowing that it's going to be repeatably useful for them while working on the project, and can upskill the whole team over time. You're essentially defining a body of knowledge expected by picking a framework.
Similarly when you're architecting software you're essentially choosing what level of developer you're going to need to hire or train, and so if you're in that position it's smart to be cautious of increasing the burden of knowledge too much unless you're happy hiring for that.
I disagree, and agree with Mark Seemann's take[0]. If something is better, switch to it even if it causes inconsistencies. Over time you can refactor the old code.
You've limited your projects progression to if a billion dollar company has created a framework that is more productive such that they can be taught it in University. You'll never get hackers like pg who can code LISP if you go that route. You should hire good talent and then train them such that they understand, and hopefully keep them and they stay long after they work on your one project.
[0]: https://blog.ploeh.dk/2021/05/17/against-consistency/
You've limited your projects progression to if a billion dollar company has created a framework that is more productive such that they can be taught it in University. You'll never get hackers like pg who can code LISP if you go that route. You should hire good talent and then train them such that they understand, and hopefully keep them and they stay long after they work on your one project.
[0]: https://blog.ploeh.dk/2021/05/17/against-consistency/
I think that's the crux of the disagreement, I explicitly don't want hackers, I want professional software developers to execute a very reasonable software project effectively, which is not everyone's cup of tea.
There's plenty of room for hackers in the industry, to go make really cool stuff and make endless cash, and I love the architecture part more than the rest of it so for personal projects that's what I'm all about. But if I want to achieve delivery on time and in an expected format, I want zero hackers, I want career engineers. That is definitely a boring take for some, but that's just part of a maturing industry. Not every project needs to be an R&D project at the same time.
There's plenty of room for hackers in the industry, to go make really cool stuff and make endless cash, and I love the architecture part more than the rest of it so for personal projects that's what I'm all about. But if I want to achieve delivery on time and in an expected format, I want zero hackers, I want career engineers. That is definitely a boring take for some, but that's just part of a maturing industry. Not every project needs to be an R&D project at the same time.
Assuming that you will be ever allowed to refactor properly rather than be put under immense pressure to fix other things right now.
The problem 5 is sometimes caused by the management not allowing time for a proper redesign of a prototype. (Even if TDD is employed, which often is not because tests are not user deliverables.)
Hence why my statement "iterative development never works" as the managers will see the initial prototype mostly/partly working and start pushing features on it.
In such an environment, you need an experienced designer from day one - at worst you'll get slightly less performant but maintainable code due to extra abstractions.
If you get a design mistake... Well...
Hence why my statement "iterative development never works" as the managers will see the initial prototype mostly/partly working and start pushing features on it.
In such an environment, you need an experienced designer from day one - at worst you'll get slightly less performant but maintainable code due to extra abstractions.
If you get a design mistake... Well...
> it's self-inflicted dumbing-down. I really don't understand where this rampant support for antiintellectualism comes from
I don't think it is about dumbing-down or antiintellectualism. It is about knowledge. No matter how clever or smart you are, you do not know everything. So any special knowledge (about the OS, the language, etc) you have to be aware of to understand code is a barrier.
I specifically don't mean domain knowledge. You got to know stuff about the problem you're trying to solve, sure. I mean knowledge about the tool you use to solve the problem. Its just a tool. Nothing else. A tool should be practical, the less you need to know to use it, the better. If you want to write a manual for a machine, you wouldn't use poetry, right?
Of course some stuff is worth it. If there's a language construct that makes lots of things "better" (whatever that means) - cool, we should use that. But it should be some generic thing that can be applied to many problems (so its worth learning about it), not some specific thing that solves specific problems more clever.
I don't think it is about dumbing-down or antiintellectualism. It is about knowledge. No matter how clever or smart you are, you do not know everything. So any special knowledge (about the OS, the language, etc) you have to be aware of to understand code is a barrier.
I specifically don't mean domain knowledge. You got to know stuff about the problem you're trying to solve, sure. I mean knowledge about the tool you use to solve the problem. Its just a tool. Nothing else. A tool should be practical, the less you need to know to use it, the better. If you want to write a manual for a machine, you wouldn't use poetry, right?
Of course some stuff is worth it. If there's a language construct that makes lots of things "better" (whatever that means) - cool, we should use that. But it should be some generic thing that can be applied to many problems (so its worth learning about it), not some specific thing that solves specific problems more clever.
'unscaled gave a good example elsewhere in the thread[0]. "Our junior engineers understand for loops better than map-filter-reduce chains, so let's use for loops instead of spending a few days teaching them". This stuff really happens.
Back in the day, in a company that just transitioned from Java 7 to Java 8 across its main projects, I've been told by the boss to maybe not use lambdas too much, because certain co-workers like John Doe[1] aren't so experienced and might not understand them. Thankfully, I was just starting another project with (initially) just a single "less experienced" dev, so I taught him these and related things quickly, and we proceeded to write simpler code, making full use of lambdas and stream APIs where appropriate. And aforementioned John Doe? Half a year later, he discovered on his own that Java 8 had lambdas, grokked them in like 2 hours, and went around the company, evangelizing their use for everyone.
> I mean knowledge about the tool you use to solve the problem. Its just a tool. Nothing else. A tool should be practical, the less you need to know to use it, the better.
A tool does not exist in isolation from the process. It's a part of it. Everywhere else than software industry, this seems to be understood. If you don't know how to operate the tools you work with, you're considered incompetent, not trained for the work you're performing. The solution to that isn't to dumb down tools - it's to train the operators. You have to pay a little bit up front to upskill your workers, and you then get to continuously reap the benefits of improved productivity due to better use of better tools.
Should modern construction be done only with hand shovels, sticks and ropes, because junior construction workers will be confused by excavators and welders? That's how all this talk about junior-friendly code reads to me: an extreme case of being penny wise, pound foolish. Software development is a profession, the job of a junior is first and foremost to learn their trade[2].
--
[0] - https://news.ycombinator.com/item?id=27420543
[1] - Name intentionally omitted; the guy has been working in the company for many more years than I have - and was way better than our boss gave him credit for.
[2] - Well, unless your company is structured around the idea that juniors do all the work, and seniors are "using their experience to multiply productivity of everyone" aka. just managers without the authority. Which is another trend in the industry that's worrying to me, because it pins the proficiency of the work to whatever junior hires learned before getting the job. And then we're surprised that tech debt is at all-times high everywhere.
Back in the day, in a company that just transitioned from Java 7 to Java 8 across its main projects, I've been told by the boss to maybe not use lambdas too much, because certain co-workers like John Doe[1] aren't so experienced and might not understand them. Thankfully, I was just starting another project with (initially) just a single "less experienced" dev, so I taught him these and related things quickly, and we proceeded to write simpler code, making full use of lambdas and stream APIs where appropriate. And aforementioned John Doe? Half a year later, he discovered on his own that Java 8 had lambdas, grokked them in like 2 hours, and went around the company, evangelizing their use for everyone.
> I mean knowledge about the tool you use to solve the problem. Its just a tool. Nothing else. A tool should be practical, the less you need to know to use it, the better.
A tool does not exist in isolation from the process. It's a part of it. Everywhere else than software industry, this seems to be understood. If you don't know how to operate the tools you work with, you're considered incompetent, not trained for the work you're performing. The solution to that isn't to dumb down tools - it's to train the operators. You have to pay a little bit up front to upskill your workers, and you then get to continuously reap the benefits of improved productivity due to better use of better tools.
Should modern construction be done only with hand shovels, sticks and ropes, because junior construction workers will be confused by excavators and welders? That's how all this talk about junior-friendly code reads to me: an extreme case of being penny wise, pound foolish. Software development is a profession, the job of a junior is first and foremost to learn their trade[2].
--
[0] - https://news.ycombinator.com/item?id=27420543
[1] - Name intentionally omitted; the guy has been working in the company for many more years than I have - and was way better than our boss gave him credit for.
[2] - Well, unless your company is structured around the idea that juniors do all the work, and seniors are "using their experience to multiply productivity of everyone" aka. just managers without the authority. Which is another trend in the industry that's worrying to me, because it pins the proficiency of the work to whatever junior hires learned before getting the job. And then we're surprised that tech debt is at all-times high everywhere.
The examples you gave are exactly the ones I mean with generic solutions. Stuff you can apply to many things, like lambdas. Such things exist in many programming languages. Yes, that's stuff people should learn and keep up with.
But using some weird quirk or very uncommon language construct just to save a couple of lines of code (and show how well you know the language) - tempting, but just don't.
And use long, understandable variable names! Don't be a mathematician, you're an engineer! (totally unrelated but its just really annoying)
But using some weird quirk or very uncommon language construct just to save a couple of lines of code (and show how well you know the language) - tempting, but just don't.
And use long, understandable variable names! Don't be a mathematician, you're an engineer! (totally unrelated but its just really annoying)
[deleted]
The OP didn’t advocate for dumbing down the project, it advocated for writing understandable code and avoiding “clever” code where possible. More often than not “clever” code is all about “local cleverness”, and it’s entirely avoidable. Interesting projects have inherent global complexity (in addition to some unavoidable local highly complex places), and there is no need to introduce unnecessary complexity by using smart tricks. In these cases you won’t go very far if the developers are replaceable cogs, you’ll still need senior intelligent, autonomous people.
They are the same picture.
I don't think it is black and white. Disentangling a heavily optimized "clever" codebase took me four weeks once. Refactoring did not only make it more understandable but also faster.
Seems like often "clever" is used with those quotes around it and really stand for "(accidentally) complicated".
Edit: From the article: 'I’m drawn to this distinction because I feel like our discussions of “clever code” are too constrained.'
Seems like often "clever" is used with those quotes around it and really stand for "(accidentally) complicated".
Edit: From the article: 'I’m drawn to this distinction because I feel like our discussions of “clever code” are too constrained.'
This made me think about how we use the term 'over engineered' in our industry. In most industries it seems to be a good thing (i.e. space shuttle). In our industry it's synonymous with bad, unnecessarily complex code. We seem to have a knack of taking positive attributes such as clever, and flipping them!
> In most industries it seems to be a good thing (i.e. space shuttle)
It really isn’t the case in most industries. There are a couple of industries with inherent complexity and where the cost of failure justifies some kind of over-engineering (you could add particle accelerators, medical imagery devices, aircrafts, and things like that). In these cases, you throw a lot of money and very good engineers at the problem. Most industries are not willing to do that, because dit is expensive. So they produce unreliable crap that breaks all the time.
In the vast majority of cases, over engineering means more parts that can fail in more “interesting” ways, and that are more difficult to diagnose because the thing is more complex than it needs to be. “Over-engineered” is really not a compliment applied to a car or a house, for example.
Engineers are not gods. Sometimes they are stupid just like any other human. They also need to stay humble and recognise that a system needs to be as complex as needed to fulfil its purpose, but not more.
> We seem to have a knack of taking positive attributes such as clever, and flipping them!
“Clever” is a compliment. But you can have too much of a good thing, as in “too clever by half”. If you depend on something, you don’t want it to be clever, you want it to be reliable first.
It really isn’t the case in most industries. There are a couple of industries with inherent complexity and where the cost of failure justifies some kind of over-engineering (you could add particle accelerators, medical imagery devices, aircrafts, and things like that). In these cases, you throw a lot of money and very good engineers at the problem. Most industries are not willing to do that, because dit is expensive. So they produce unreliable crap that breaks all the time.
In the vast majority of cases, over engineering means more parts that can fail in more “interesting” ways, and that are more difficult to diagnose because the thing is more complex than it needs to be. “Over-engineered” is really not a compliment applied to a car or a house, for example.
Engineers are not gods. Sometimes they are stupid just like any other human. They also need to stay humble and recognise that a system needs to be as complex as needed to fulfil its purpose, but not more.
> We seem to have a knack of taking positive attributes such as clever, and flipping them!
“Clever” is a compliment. But you can have too much of a good thing, as in “too clever by half”. If you depend on something, you don’t want it to be clever, you want it to be reliable first.
I would say that the space shuttle is the perfect example of the risks of over-engineering.
It's not. Space shuttle was pretty well engineered to do the things it was meant to. (If anything, launcher and ops were underengineered hence the failures.)
The problem was that it was meant to do too many things at the same time. Some of them were: being able to return to US territory from anywhere. (Hence semi-aerodynamic.) Carrying people and a lot of cargo. (Hence big.) Deploying any kind of cargo rather than defined operations. (Hence Canadaarm.) Being reusable. (but not the launcher...) Etc.
The chief problem of it is that it was first in a class research vehicle. And you cannot refactor rockets. (At least not cheaply. Prove me wrong Starship.)
The problem was that it was meant to do too many things at the same time. Some of them were: being able to return to US territory from anywhere. (Hence semi-aerodynamic.) Carrying people and a lot of cargo. (Hence big.) Deploying any kind of cargo rather than defined operations. (Hence Canadaarm.) Being reusable. (but not the launcher...) Etc.
The chief problem of it is that it was first in a class research vehicle. And you cannot refactor rockets. (At least not cheaply. Prove me wrong Starship.)
It’s not exclusively used for clever code.
I think a more common case for the term is to describe a program/system that covers way more ground than it needs to in terms of abstraction, tooling, features, scalability and performance.
Example: I handed off a website a year ago that didn’t have any sort of remarkable engineering/usage requirements on the backend side. Internal operations of our client however insisted on deploying it on a kube cluster, so we had to accommodate for that. All they actually needed was maybe some caching, if at all.
Mild case, but I think of these types of stories when thinking of “over engineering”.
I think a more common case for the term is to describe a program/system that covers way more ground than it needs to in terms of abstraction, tooling, features, scalability and performance.
Example: I handed off a website a year ago that didn’t have any sort of remarkable engineering/usage requirements on the backend side. Internal operations of our client however insisted on deploying it on a kube cluster, so we had to accommodate for that. All they actually needed was maybe some caching, if at all.
Mild case, but I think of these types of stories when thinking of “over engineering”.
[deleted]
and potentially the dumbing down may increase the turn over rate which increases the need for dumbed-down code since everybody is a newb all the time
> Make it simple enough that a brand new completely green junior member of the team immediately understands it on first reading.
That can be a goal, but it's not necessarily the best one. We could of course write code so that it is most easily understood by someone who never programmed before. But that then reduces productivity for everyone with lots of experience and familiarity with that project/style/code.
Sometimes you can write code that achieves both goals, but often you cannot and in that case the "simpler" version is not always the best.
That can be a goal, but it's not necessarily the best one. We could of course write code so that it is most easily understood by someone who never programmed before. But that then reduces productivity for everyone with lots of experience and familiarity with that project/style/code.
Sometimes you can write code that achieves both goals, but often you cannot and in that case the "simpler" version is not always the best.
If we could make our domain specific code understandable to non coders we would have an easier time working with domain experts.
That said it's near impossible to do that.
That said it's near impossible to do that.
The mistake lies in the assumption that technical details are not part of the domain - but they are. Sure, in some languages like C you cannot properly separate irrelevant technical details from the domain specific code. But for the more advanced programming languages that is certainly possible.
The problem is just that things like the possibilities of errors, eventual consistency, concurrency and so on are relevant for the business domain as well. You simply cannot abstract over them in general. For domain experts who can actually deal with these issues as well, you can pick a language like Scala or Clojure which excel at creating embedded DSLs and then have the domain exports use them.
The problem is just that things like the possibilities of errors, eventual consistency, concurrency and so on are relevant for the business domain as well. You simply cannot abstract over them in general. For domain experts who can actually deal with these issues as well, you can pick a language like Scala or Clojure which excel at creating embedded DSLs and then have the domain exports use them.
Excellent insight.
Jane Street does this, traders contribute to code reviews by developers. Though at Jane Street a trader probably has a STEM degree from MIT.
>Make it simple enough that a brand new completely green junior member of the team immediately understands it on first reading.
But what if the junior member can't understand OOP, functional programming, has no idea of abstraction and can only produce duplication and duplication of code,should we also try to adjust to that level?
I do see the need to keep code simple, but sometimes an additional layer of abstraction and complexity is needed because code maintenance task requires it, junior people most likely can't appreciate it until they are no longer junior.
But what if the junior member can't understand OOP, functional programming, has no idea of abstraction and can only produce duplication and duplication of code,should we also try to adjust to that level?
I do see the need to keep code simple, but sometimes an additional layer of abstraction and complexity is needed because code maintenance task requires it, junior people most likely can't appreciate it until they are no longer junior.
> But what if the junior member can't understand OOP, functional programming, has no idea of abstraction and can only produce duplication and duplication of code
... then this person obviously did not pay attention in class and should probably not have been hired.
... then this person obviously did not pay attention in class and should probably not have been hired.
Then probably it’s too junior, and not ready to be a contributor to the project. He should be brought up to speed with the basic technologies and tools used. Project quality should not be sacrificed.
"But what if the junior member can't read english and mutiply 5 by 5"
>In fact, please make it so obvious that in reading it I don't gain/need/curse/applaud any insights.
at some point clarity requires succinctness, at some point obviousness becomes stupidity. The above requirement has not actual measurable parts and as such we can never know if we have made things too stupid in our attempt to make our code understandable by all parties.
Finally as languages evolve and gain new syntactical structures it means that things come into the language that will not be known by everyone, if you want everything understandable by the least knowledgeable that means that you cannot improve your code base. Your biggest legacy problem becomes the least advanced programmer on your team.
at some point clarity requires succinctness, at some point obviousness becomes stupidity. The above requirement has not actual measurable parts and as such we can never know if we have made things too stupid in our attempt to make our code understandable by all parties.
Finally as languages evolve and gain new syntactical structures it means that things come into the language that will not be known by everyone, if you want everything understandable by the least knowledgeable that means that you cannot improve your code base. Your biggest legacy problem becomes the least advanced programmer on your team.
I'm writing an UTF-8 decoder (just for fun) and there's quite a lot of information to understand even before starting that. And as I started I found that it naturally codes itself into a state machine with 12 byte types, 8 decoding states and 11 moves. If I rewrite my notes to condense them I expect to get maybe 4 screenfuls of comments :) Not complex but not something easily digestible by a completely green junior member perhaps.
> Make it simple enough that a brand new completely green junior member of the team immediately understands it on first reading.
This is pretty close to Tony Hoare's quote:
> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature.
This is pretty close to Tony Hoare's quote:
> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature.
Weakest Link Oriented Programming
A good junior is an expert at writing clever code. I think this desire is squashed out of people once they realize there are many more stakeholders in the code you write than just you. Being clever is often a waste of everyone's time.
Read over your code, and wherever you meet with a section which you think is particularly fine, strike it out.
This is a paraphrase of Samuel Johnson's "Read over your compositions and, wherever you meet with a passage which you think is particularly fine, strike it out.”
I learnt the truth in this when I spent years writing orchestral music. I think it's very similar to avoiding premature optimization. Each piece would take months, first sketching in the broad features - melody, harmony in a few voices, gradually adding more until I reached what seemed the end, then revising until no improvements came to mind on listening through, and lastly adding the large-scale dynamics, pauses, tempo changes, percussion. I learnt to always write/add just what I heard in my head, to try whatever it occurred to me to try, keep whatever changes sounded better, but never to plan anything.
Sometimes, early on, a particularly nice bit would emerge fairly fully formed, of which I was proud, and I'd polish that moment immediately to a near-complete state. But then for months it was fixed, couldn't flex and evolve with the rest of the piece, being already "perfect"—or it was originally, not in its final context—and usually I'd regret it was there, all dead and shiny. They felt like fossilized vanity.
I learnt the truth in this when I spent years writing orchestral music. I think it's very similar to avoiding premature optimization. Each piece would take months, first sketching in the broad features - melody, harmony in a few voices, gradually adding more until I reached what seemed the end, then revising until no improvements came to mind on listening through, and lastly adding the large-scale dynamics, pauses, tempo changes, percussion. I learnt to always write/add just what I heard in my head, to try whatever it occurred to me to try, keep whatever changes sounded better, but never to plan anything.
Sometimes, early on, a particularly nice bit would emerge fairly fully formed, of which I was proud, and I'd polish that moment immediately to a near-complete state. But then for months it was fixed, couldn't flex and evolve with the rest of the piece, being already "perfect"—or it was originally, not in its final context—and usually I'd regret it was there, all dead and shiny. They felt like fossilized vanity.
While the author is suggesting some good techniques, I suspect that if your code is being criticised for being “clever”, you’re unlikely to improve the situation by arguing that in fact it’s “insightful”.
In my view, the most important aspect of what we mean by “clever code” is that the “cleverness” primarily serves to demonstrate how clever the programmer is. In other words, in some sense it obfuscates what the code is actually doing or why certain choices have been made, without delivering sufficient benefit to outweigh the cost to readability and maintainability.
To use their bucket sort example, probably you’d want to stick a quick comment in the code that bucket sort has been chosen (and why). That helps with readability. But there’s nothing “clever” about choosing a standard algorithm to solve a problem. Absolutely, making the right choice demonstrates insight - but it also demonstrates genuine cleverness.
Really it boils down to how it doesn’t matter whether the code is being called “clever”, “insightful”, or “amazing”. It matters whether a) the tone is dripping with sarcasm, or b) the tone is respectful but the feedback also contains a warning to format and comment for readability.
In my view, the most important aspect of what we mean by “clever code” is that the “cleverness” primarily serves to demonstrate how clever the programmer is. In other words, in some sense it obfuscates what the code is actually doing or why certain choices have been made, without delivering sufficient benefit to outweigh the cost to readability and maintainability.
To use their bucket sort example, probably you’d want to stick a quick comment in the code that bucket sort has been chosen (and why). That helps with readability. But there’s nothing “clever” about choosing a standard algorithm to solve a problem. Absolutely, making the right choice demonstrates insight - but it also demonstrates genuine cleverness.
Really it boils down to how it doesn’t matter whether the code is being called “clever”, “insightful”, or “amazing”. It matters whether a) the tone is dripping with sarcasm, or b) the tone is respectful but the feedback also contains a warning to format and comment for readability.
I think there's another class of "clever/amazing" code that wraps up esoteric language features in easy to use APIs/libraries. So while the actual implementation can be quite hard to follow/understand (especially for novices) consuming the service is incredibly easy and makes an otherwise arduous task simple and requiring far less code.
I've found myself writing these kind of modules often over the years. They often use RTTI/reflection and generics and make a task declarative rather than procedural. A couple that spring to mind are a messaging/subscription framework in C++ (heavy use of templates) and a reporting API in C# (lots of reflection).
Always do my best to make the implementation comprehensible but there's only so much you can do when they require a deep understanding of the language.
I've found myself writing these kind of modules often over the years. They often use RTTI/reflection and generics and make a task declarative rather than procedural. A couple that spring to mind are a messaging/subscription framework in C++ (heavy use of templates) and a reporting API in C# (lots of reflection).
Always do my best to make the implementation comprehensible but there's only so much you can do when they require a deep understanding of the language.
I've written some (in my head) really clever code, changing a 15 line loop to a single line functional thing. But it didn't become clearer, and learning the functional language like map, filter, reduce etc takes some time.
I read an article the other day that also pointed out how functional constructs in an otherwise imperative language are painful. Not completely useless, but not the be-all and end-all. It was in the context of Go, where if you were to naively write some algorithms into functional constructs (like the underscore.js utilities), performance would take a massive nosedive. It's not a functional language, and shouldn't be used as such.
I read an article the other day that also pointed out how functional constructs in an otherwise imperative language are painful. Not completely useless, but not the be-all and end-all. It was in the context of Go, where if you were to naively write some algorithms into functional constructs (like the underscore.js utilities), performance would take a massive nosedive. It's not a functional language, and shouldn't be used as such.
We don't write code for computers to run, but for other programmers to read.
Best make it so that they understand it. Who "they" is might be dependent on the situation.
Best make it so that they understand it. Who "they" is might be dependent on the situation.
If you have either high performance real time needs, and it’s impactful to do so, or tight deliverables where performance is some component, the code is often written for a computer over a human.
The more specialized or niche certain project areas are, the less likely anyone will ever be reading it directly as a programmer (and those that need to, are often able to disentangle the obscurity). The compromise is a clean interface to otherwise isolated or pure functions.
Often times, this is unfortunately poorly executed or inappropriately assumed. Usually the people who should write clever code are the ones who have experience writing clever code, but almost certainly anyone starting to write clever code is writing terrible code.
Inexperienced folks often write clever code to solve a business need at that time. They create tremendous tech and culture debt. It’s easy to not understand when you’re an IC or line manager, though, that the business could have been severely affected otherwise. I’ll take millions in tech debt opportunity cost and the future morale hit over a failed startup/vertical any day.
Just my 2c in high performance or hardware constrained projects both as an IC, and as various levels of management who’s both inherited mountains of “clever” shit and and built my own directly or indirectly.
The more specialized or niche certain project areas are, the less likely anyone will ever be reading it directly as a programmer (and those that need to, are often able to disentangle the obscurity). The compromise is a clean interface to otherwise isolated or pure functions.
Often times, this is unfortunately poorly executed or inappropriately assumed. Usually the people who should write clever code are the ones who have experience writing clever code, but almost certainly anyone starting to write clever code is writing terrible code.
Inexperienced folks often write clever code to solve a business need at that time. They create tremendous tech and culture debt. It’s easy to not understand when you’re an IC or line manager, though, that the business could have been severely affected otherwise. I’ll take millions in tech debt opportunity cost and the future morale hit over a failed startup/vertical any day.
Just my 2c in high performance or hardware constrained projects both as an IC, and as various levels of management who’s both inherited mountains of “clever” shit and and built my own directly or indirectly.
True, but you'd be operating on a different level entirely - and clarity of the code is still important. You can have both, it's not a dichotomy. I see going "down" to low level, high performance code as crossing into a different domain, so you write the code aimed at that domain. It's all Klatchian to me personally, but I've seen high performance code that I could still decypher. decode. That.
You can use naming, formatting and comments where needed to ensure that you and the next person can read and understand it without affecting performance.
You can use naming, formatting and comments where needed to ensure that you and the next person can read and understand it without affecting performance.
We don't write code for computers to run, but for other programmers to read.
Well, maybe sometimes we shouldn’t.
Let’s pick a topical example. Suppose we have a tool used by JS developers, itself written in straightforward JS. Its code is nice and easy to read, and it does the job. However, it takes 5s to process a source tree containing a few thousand lines of code.
Now suppose we could have an alternative version of that tool, written by a skilled developer using a more performance-oriented language, and placing greater emphasis on runtime performance even if it means sacrificing some readability in the source code. This might do the same job in well under a second. As esbuild¹ has shown, this scale of performance improvement is realistic.
Maybe this tool gets run 50 times per day. Then a single developer using it could justify spending about a week writing the faster alternative tool to break even on time saved over five years. (There’s an xkcd² about this that is worth a look if you haven’t seen it before.)
But maybe this tool doesn’t have just one user, it has one million. Then those frequent 5s inefficiencies add up to several hundred users’ entire careers in lost productivity over that same five-year span.
That’s a high price to pay for writing software in a way that is more convenient for developers but less good for users.
¹ https://esbuild.github.io/
² https://xkcd.com/1205/
Well, maybe sometimes we shouldn’t.
Let’s pick a topical example. Suppose we have a tool used by JS developers, itself written in straightforward JS. Its code is nice and easy to read, and it does the job. However, it takes 5s to process a source tree containing a few thousand lines of code.
Now suppose we could have an alternative version of that tool, written by a skilled developer using a more performance-oriented language, and placing greater emphasis on runtime performance even if it means sacrificing some readability in the source code. This might do the same job in well under a second. As esbuild¹ has shown, this scale of performance improvement is realistic.
Maybe this tool gets run 50 times per day. Then a single developer using it could justify spending about a week writing the faster alternative tool to break even on time saved over five years. (There’s an xkcd² about this that is worth a look if you haven’t seen it before.)
But maybe this tool doesn’t have just one user, it has one million. Then those frequent 5s inefficiencies add up to several hundred users’ entire careers in lost productivity over that same five-year span.
That’s a high price to pay for writing software in a way that is more convenient for developers but less good for users.
¹ https://esbuild.github.io/
² https://xkcd.com/1205/
I found a practical example of this last week.
For years (literally!) I've been on the lookout for an efficient Javascript blur algorithm. I've attempted to code them myself following various online tutorials, or lift the code from others who seemed to have better insights into various algorithmic approaches than me. Because I struggle with complex mathematical concepts, I've tended to steer clear of code that looks nothing like Javascript.
Then last week I discovered a blisteringly fast gaussian blur algorithm in a GitHub repository[1] which translated work done by some very clever Intel developers[2] into Javascript. I copy-pasted the code on blind trust into my library code, then tweaked more in hope than knowledge to get it to bed in and play nicely[3] ... and it works! 40px radius gaussian blurs over 800x800px images at 60fps (in a 2D canvas - no webgl gets hurt by this algorithm).
I consider myself a well-versed JS coder, yet I've stared at this code until my eyes water: none of it makes sense to me. All I know is that it works by magic, and that makes me happy!
[1] - https://github.com/nodeca/glur/blob/master/index.js
[2] - https://software.intel.com/en-us/articles/iir-gaussian-blur-... - page redirects if you're not logged in
[3] - my tweaked code - https://scrawl-v8.rikweb.org.uk/docs/source/factory/filterEn...
For years (literally!) I've been on the lookout for an efficient Javascript blur algorithm. I've attempted to code them myself following various online tutorials, or lift the code from others who seemed to have better insights into various algorithmic approaches than me. Because I struggle with complex mathematical concepts, I've tended to steer clear of code that looks nothing like Javascript.
Then last week I discovered a blisteringly fast gaussian blur algorithm in a GitHub repository[1] which translated work done by some very clever Intel developers[2] into Javascript. I copy-pasted the code on blind trust into my library code, then tweaked more in hope than knowledge to get it to bed in and play nicely[3] ... and it works! 40px radius gaussian blurs over 800x800px images at 60fps (in a 2D canvas - no webgl gets hurt by this algorithm).
I consider myself a well-versed JS coder, yet I've stared at this code until my eyes water: none of it makes sense to me. All I know is that it works by magic, and that makes me happy!
[1] - https://github.com/nodeca/glur/blob/master/index.js
[2] - https://software.intel.com/en-us/articles/iir-gaussian-blur-... - page redirects if you're not logged in
[3] - my tweaked code - https://scrawl-v8.rikweb.org.uk/docs/source/factory/filterEn...
That is cool!
I think I may be able to explain the gist (though not the full details).
It uses an IIR filter, or Infinite Impulse Response filter. This is a digital filter (could be either software or hardware) where there’s a feedback loop from the output to the input. The advantage of an IIR is that it can produce a long output (potentially infinite, hence the name) from a single input. The disadvantage is that IIRs can easily get unstable and hard to work with; hence many (most?) applications use FIRs, Finite Impulse Response, where there’s no feedback and the output is strictly bounded.
It looks like they’re using an IIR tuned to produce a Gaussian bell curve as the impulse response. By feeding each pixel through the filter in sequence, each pixel produces a bell curve and those get blended together, exactly what you what.
So, that tight inner loop consists of reading in the next pixel and blending it with the intermediate results of earlier pixels.
Very efficient on a CPU where you can stream the data through, but it wouldn’t work in an OpenGL shader as that expects to work on each pixel in parallel, not sequentially.
I think I may be able to explain the gist (though not the full details).
It uses an IIR filter, or Infinite Impulse Response filter. This is a digital filter (could be either software or hardware) where there’s a feedback loop from the output to the input. The advantage of an IIR is that it can produce a long output (potentially infinite, hence the name) from a single input. The disadvantage is that IIRs can easily get unstable and hard to work with; hence many (most?) applications use FIRs, Finite Impulse Response, where there’s no feedback and the output is strictly bounded.
It looks like they’re using an IIR tuned to produce a Gaussian bell curve as the impulse response. By feeding each pixel through the filter in sequence, each pixel produces a bell curve and those get blended together, exactly what you what.
So, that tight inner loop consists of reading in the next pixel and blending it with the intermediate results of earlier pixels.
Very efficient on a CPU where you can stream the data through, but it wouldn’t work in an OpenGL shader as that expects to work on each pixel in parallel, not sequentially.
Thanks - that explanation helps a bit. But now I'm wondering if the code could be adapted to handle various convolution matrix filters and my brain is threatening to divorce me.
> Now suppose we could have an alternative version of that tool, written by a skilled developer using a more performance-oriented language, and placing greater emphasis on runtime performance even if it means sacrificing some readability in the source code.
I'm just not sure that's a decision that we often have to make. Is more performant code most of the time also less readable? What about still using JavaScript, still having readable code and getting the execution time to 1-2s?
Also, using another language is probably never the issue except its some weird rarely used language.
But given your scenario: what if changing to the other language and the less readable code keeps other developers from extending the tool, writing plugins and stuff? So what if you have to give up all those contributions from other people?
I'm just not sure that's a decision that we often have to make. Is more performant code most of the time also less readable? What about still using JavaScript, still having readable code and getting the execution time to 1-2s?
Also, using another language is probably never the issue except its some weird rarely used language.
But given your scenario: what if changing to the other language and the less readable code keeps other developers from extending the tool, writing plugins and stuff? So what if you have to give up all those contributions from other people?
Is more performant code most of the time also less readable?
I don’t think there is a simple relationship between performance and readability, but they aren’t completely independent. Many choices in programming can improve readability but also put a bound on achievable performance.
For example, suppose you choose a programming language that offers a lot of dynamic features at runtime. You might be able to express your ideas more easily using those features. On the other hand, you also need a runtime system to implement those features and that system may have a performance cost.
If you choose a programming language with a high level of abstraction, again you might be able to express your ideas more easily, but you might not be able to control the generated code to ensure that it fully exploits hardware features like parallelism and caching. In some cases, your compiler or interpreter might still do that anyway, depending on the semantics of the language and what the tools can safely infer about your program’s intended behaviour, but this is the classic “sufficiently smart compiler” argument.
An example not involving choice of programming language would be limiting your program to simple, well-understood data structures and algorithms. This makes your program more accessible to developers who are only familiar with the common tools. On the other hand, what if there is a more sophisticated alternative that is faster or requires less space, but the algorithm relies on building and navigating more complicated data structures? What if that alternative isn’t a standard technique that new developers can look up in a textbook, but rather something created after months of work by your own subject matter experts, who have since left the company? What if it is a standard technique, but one that is usually used to solve a different kind of problem, and it happens to be applicable in your current situation because some specific insight about your problem lets you reduce it to the standard one?
What about still using JavaScript, still having readable code and getting the execution time to 1-2s?
These things always depend on context.
With my UI hat on, a 1–2s delay can still be very noticeable to users. For a tool that runs on every save and where the developer is likely to check the results immediately — say, running a test suite, or building and then hot-reloading in a browser — that could still have an impact on each developer’s productivity. If each developer has several tools like that, and there are millions of developers, those little delays still add up to a significant overall drag on the industry.
On the other hand, for a background job that runs only under specific conditions that don’t happen very often, a few extra seconds is probably irrelevant.
But given your scenario: what if changing to the other language and the less readable code keeps other developers from extending the tool, writing plugins and stuff? So what if you have to give up all those contributions from other people?
Well, in this particular thought experiment, you could afford (in the productivity sense) to have a few thousand developers drop whatever they're doing and work full-time on developing your original tool instead for years and you’d still come out ahead. You can teach a lot of people a new programming language and explain a lot of correct but less readable code in thousands of years.
The real point here is that there is often a balance between making things easier for developers and making things better for users. Popular wisdom in our industry says that making things easier for developers is usually the right answer because developer time is expensive, but this is a conceit.
If there are few developers but many users, a little extra cost for developers may create a much bigger gain for users, taken over the groups as a whole. We’ve already been talking about a good example in this discussion.
This isn’t the only situation where developer costs aren’t necessarily the most important factor. What if the development environment is cheap but the runtime environment is expensive? If you have cloud infrastructure and usage-based pricing, maybe improved software performance would mean you could serve results twice as quickly or only need half as much storage. You can just throw more virtualised resources at an inefficient system in the cloud, but your infrastructure bill won’t thank you for it.
Related considerations arise if your software will run on a specialised platform like a supercomputer or a satellite or a tiny embedded medical device. Sometimes you have access to highly specialised but strictly limited resources, and you might not be the only one wanting to make the best use of them. Obviously each of these is a niche application, but there’s lots of software that isn’t web and mobile apps, and those developers also have to make trade-offs between readability and performance sometimes.
I don’t think there is a simple relationship between performance and readability, but they aren’t completely independent. Many choices in programming can improve readability but also put a bound on achievable performance.
For example, suppose you choose a programming language that offers a lot of dynamic features at runtime. You might be able to express your ideas more easily using those features. On the other hand, you also need a runtime system to implement those features and that system may have a performance cost.
If you choose a programming language with a high level of abstraction, again you might be able to express your ideas more easily, but you might not be able to control the generated code to ensure that it fully exploits hardware features like parallelism and caching. In some cases, your compiler or interpreter might still do that anyway, depending on the semantics of the language and what the tools can safely infer about your program’s intended behaviour, but this is the classic “sufficiently smart compiler” argument.
An example not involving choice of programming language would be limiting your program to simple, well-understood data structures and algorithms. This makes your program more accessible to developers who are only familiar with the common tools. On the other hand, what if there is a more sophisticated alternative that is faster or requires less space, but the algorithm relies on building and navigating more complicated data structures? What if that alternative isn’t a standard technique that new developers can look up in a textbook, but rather something created after months of work by your own subject matter experts, who have since left the company? What if it is a standard technique, but one that is usually used to solve a different kind of problem, and it happens to be applicable in your current situation because some specific insight about your problem lets you reduce it to the standard one?
What about still using JavaScript, still having readable code and getting the execution time to 1-2s?
These things always depend on context.
With my UI hat on, a 1–2s delay can still be very noticeable to users. For a tool that runs on every save and where the developer is likely to check the results immediately — say, running a test suite, or building and then hot-reloading in a browser — that could still have an impact on each developer’s productivity. If each developer has several tools like that, and there are millions of developers, those little delays still add up to a significant overall drag on the industry.
On the other hand, for a background job that runs only under specific conditions that don’t happen very often, a few extra seconds is probably irrelevant.
But given your scenario: what if changing to the other language and the less readable code keeps other developers from extending the tool, writing plugins and stuff? So what if you have to give up all those contributions from other people?
Well, in this particular thought experiment, you could afford (in the productivity sense) to have a few thousand developers drop whatever they're doing and work full-time on developing your original tool instead for years and you’d still come out ahead. You can teach a lot of people a new programming language and explain a lot of correct but less readable code in thousands of years.
The real point here is that there is often a balance between making things easier for developers and making things better for users. Popular wisdom in our industry says that making things easier for developers is usually the right answer because developer time is expensive, but this is a conceit.
If there are few developers but many users, a little extra cost for developers may create a much bigger gain for users, taken over the groups as a whole. We’ve already been talking about a good example in this discussion.
This isn’t the only situation where developer costs aren’t necessarily the most important factor. What if the development environment is cheap but the runtime environment is expensive? If you have cloud infrastructure and usage-based pricing, maybe improved software performance would mean you could serve results twice as quickly or only need half as much storage. You can just throw more virtualised resources at an inefficient system in the cloud, but your infrastructure bill won’t thank you for it.
Related considerations arise if your software will run on a specialised platform like a supercomputer or a satellite or a tiny embedded medical device. Sometimes you have access to highly specialised but strictly limited resources, and you might not be the only one wanting to make the best use of them. Obviously each of these is a niche application, but there’s lots of software that isn’t web and mobile apps, and those developers also have to make trade-offs between readability and performance sometimes.
One practical lesson I’ve learned about how to keep code simple is that possibly the most important factor to simplicity is staying on the same abstraction layer, ie. not accidentally slipping up or down the layers within the same code “stanza.”
If I’m trying to read a function that is ostensibly about, say, user auth, but what I’m actually reading is about database error handling and bitwise operations on a user role bitfield, something has gone wrong in terms of simplicity. A dumb example is “How do you drive a car?” The right answer is something like “Gas goes, Brakes slows, Wheel turns.” The wrong answer is something like “Ok well, you see there’s a drivetrain... oh and a spark plug, and then...”
In my experience very few code blocks need to be legit complex, eg. the core algorithm of some novel processing you’re doing on the data that came from a paper in the relevant field. Those unusual blocks can be sequestered and documented straightforwardly, with clear variable names and a comprehensive test harness around it.
If I’m trying to read a function that is ostensibly about, say, user auth, but what I’m actually reading is about database error handling and bitwise operations on a user role bitfield, something has gone wrong in terms of simplicity. A dumb example is “How do you drive a car?” The right answer is something like “Gas goes, Brakes slows, Wheel turns.” The wrong answer is something like “Ok well, you see there’s a drivetrain... oh and a spark plug, and then...”
In my experience very few code blocks need to be legit complex, eg. the core algorithm of some novel processing you’re doing on the data that came from a paper in the relevant field. Those unusual blocks can be sequestered and documented straightforwardly, with clear variable names and a comprehensive test harness around it.
We'll never get those things right at first. That's why - specifically the more complex code you talk about - stuff should be isolated with clearly defined interfaces, known or no side-effects etc. Only then its possible to reorganize and refactor the code when you figure out how it actually should be.
For example, the car driving example: what about the drivers license you need? All the traffic regulations you have to follow? Also gotta buy or rent a car to be able to drive one. What do you do if your car breaks down? Maintenance?
For example, the car driving example: what about the drivers license you need? All the traffic regulations you have to follow? Also gotta buy or rent a car to be able to drive one. What do you do if your car breaks down? Maintenance?
All those are great examples of things on different abstraction layers.
I agree with you insofar as you can't know what you're building ahead of time so writing software is continuous process of revision, but it's still the case that when I'm writing a particular piece of code I can ask myself what is actually on this layer vs not on this layer, and write my code accordingly.
I agree with you insofar as you can't know what you're building ahead of time so writing software is continuous process of revision, but it's still the case that when I'm writing a particular piece of code I can ask myself what is actually on this layer vs not on this layer, and write my code accordingly.
instead of classifying it as 'clever' and 'insight', i would classify it as knowledge in either the problem domain, or the solution stack/tech.
Duff's device uses knowledge of the solution tech (C language features). The sorting bucket example uses knowledge of the problem space - that you are sorting birth years of today's population.
Some code would have both incorporated - e.g., the fast inverse square root of Quake fame. It requires knowledge of how floating point gets represented, how longs get represented and what happens when you cast between them. It requires knowledge of logarithms and how exponentiation is related to logarithms, and how newton's method, with a good starting guess is quite accurate.
As for clever code - the more knowledge (of either kind) you need to know, the "worse" it is for someone to maintain it. As a corollary, if you comment on the code, comment should include both any and all domain knowledge, as well as tech stack knowledge, and reduce the burden on the reader's mind.
Duff's device uses knowledge of the solution tech (C language features). The sorting bucket example uses knowledge of the problem space - that you are sorting birth years of today's population.
Some code would have both incorporated - e.g., the fast inverse square root of Quake fame. It requires knowledge of how floating point gets represented, how longs get represented and what happens when you cast between them. It requires knowledge of logarithms and how exponentiation is related to logarithms, and how newton's method, with a good starting guess is quite accurate.
As for clever code - the more knowledge (of either kind) you need to know, the "worse" it is for someone to maintain it. As a corollary, if you comment on the code, comment should include both any and all domain knowledge, as well as tech stack knowledge, and reduce the burden on the reader's mind.
I still stand by this: simplicity is a function on familiarity more than anything else.
Twenty lines of procedural go code isn’t simpler than a one-liner of Closure. It might be, to some people.
Twenty lines of procedural go code isn’t simpler than a one-liner of Closure. It might be, to some people.
It depends on how much knowledge you need to read each line. The other factor is how much work the code does.
When it comes to Clojure or any kind of functional language or construct, those describe the "what"; what is the author trying to do, what should the code do.
If you go for imperative code, it quickly becomes more about the "how". Loops are a great example; in functional languages and constructs, you say "filter, then map, then reduce". In an iterative language, you say "Create a variable I, then start a loop, check if it's below length of list, increment, get element I from list X", etc, you get the idea.
The functional approach has a definite advantage here because - as long as you understand the language and constructs - it hides the implementation details and the exact approach to execution. A functional approach under water can transparently parallelize the request, even going as far as it becoming a distributed hadoop map/reduce job. Can't do that with a for loop.
Anyway, I forgot where I was going with this. The 20 lines can be fine if the one-liner is complicated.
When it comes to Clojure or any kind of functional language or construct, those describe the "what"; what is the author trying to do, what should the code do.
If you go for imperative code, it quickly becomes more about the "how". Loops are a great example; in functional languages and constructs, you say "filter, then map, then reduce". In an iterative language, you say "Create a variable I, then start a loop, check if it's below length of list, increment, get element I from list X", etc, you get the idea.
The functional approach has a definite advantage here because - as long as you understand the language and constructs - it hides the implementation details and the exact approach to execution. A functional approach under water can transparently parallelize the request, even going as far as it becoming a distributed hadoop map/reduce job. Can't do that with a for loop.
Anyway, I forgot where I was going with this. The 20 lines can be fine if the one-liner is complicated.
We can define the number of concepts a program uses, "Program | Concepts", and likewise,
Needless complexity arises as a distance from TokenSimplicity, relative to some choice of concepts. That can arise because the programmer uses concepts outside their audiences competence, or fails to use the minimum number of tokens.
(1) is cleverness; (2) is insightfulness. And "surprising" is all about the audience having an familiarity which differs from the programmer,
Where concepts are surprising, they can only rarely be used in programming, and hence drive up the number of tokens/lines written. However, where the code is getting excessive, the programmer is encouraged to surprise the audience.
The weighting of these objectives is of course team/problem/person specific. Typical webdev/bizapps perhaps extremely weights `min Suprise`, (perhaps routinely at the expense of properly understanding problem concepts).
In algorithmic R&D areas (occasionally, eg., data science) we expect the audience to have nearly full relevant ProblemConcepts.
TokenSimplicity(Program | LanguageConcepts, ProblemConcepts)
Needless complexity arises as a distance from TokenSimplicity, relative to some choice of concepts. That can arise because the programmer uses concepts outside their audiences competence, or fails to use the minimum number of tokens.
NeedlessComplexity = abs(TokenCount(Program) - TokenSimplicity(Program | Concepts, ...))
(1) is cleverness; (2) is insightfulness. And "surprising" is all about the audience having an familiarity which differs from the programmer,
Suprise(ProgrammerConcepts, AudienceConcepts) = len(Audience - Programmer)
Where concepts are surprising, they can only rarely be used in programming, and hence drive up the number of tokens/lines written. However, where the code is getting excessive, the programmer is encouraged to surprise the audience.
The weighting of these objectives is of course team/problem/person specific. Typical webdev/bizapps perhaps extremely weights `min Suprise`, (perhaps routinely at the expense of properly understanding problem concepts).
In algorithmic R&D areas (occasionally, eg., data science) we expect the audience to have nearly full relevant ProblemConcepts.
I really hate that saying "its simple until you make it complicated".
No! I really like the distinction between "simple" and "easy".
Writing a little script that does some job is easy. Then you add lots of extra conditions and edge cases and exceptions because damn, its not as easy as you thought, so the script becomes complicated. After a while you figure out what the problem domain is, think of some smarter rules making the thing complex. Then you dumb down those rules, throw out unnecessary stuff, streamline things, realize that multiple specifically covered edge cases can actually be covered by changing some structure in your code etc - and finally, finally that all makes the thing simple.
Simple isn't dumb. Simple is so elegant and smart that it looks dumb.
No! I really like the distinction between "simple" and "easy".
Writing a little script that does some job is easy. Then you add lots of extra conditions and edge cases and exceptions because damn, its not as easy as you thought, so the script becomes complicated. After a while you figure out what the problem domain is, think of some smarter rules making the thing complex. Then you dumb down those rules, throw out unnecessary stuff, streamline things, realize that multiple specifically covered edge cases can actually be covered by changing some structure in your code etc - and finally, finally that all makes the thing simple.
Simple isn't dumb. Simple is so elegant and smart that it looks dumb.
> Simple is so elegant and smart that it looks dumb.
It should look so dumb that when someone reads it they think "well, yeah, this is obviously how it should be done."
Of course, had they not read the finished product it's unlikely that they would have landed on that "obvious" solution on their first try.
It should look so dumb that when someone reads it they think "well, yeah, this is obviously how it should be done."
Of course, had they not read the finished product it's unlikely that they would have landed on that "obvious" solution on their first try.
I am the only one to actually like reading clever code?
When I see something like Duff's Device, my first reaction is a big WTF, like everyone else, but then, I sit down, try to understand, and when I get it, I am happy because I learned something. In fact, seeing Duff's Device is really what help me understand switch/case in C. And Duff's Device is certainly outdated, but it is by no mean "bad". If you need to unroll a loop by hand, that a good way to do it, it is just that you don't need to unroll loops anymore.
So if you write code I have to read later, please don't hold back, use every clever trick you have. A small comment would be appreciated but not necessary. This way, I can read your code and think "I learned something today".
Does it slows down comprehension? Of course it does... once. That's the cost of learning, but it is an investment, next time, I'll be faster.
I am really talking about the cleverness the article disapproves of. I also value insight, but here, I agree with the article.
I know a common saying is "newbies should understand your code at first glance". I only agree when by newbies, it means newbies on the project with regards to domain-specific knowledge. But when it comes to general principles, including knowledge of the programming language, code should be written for the highest level of expertise you can reach (but no further, you have to understand what you a doing!). I mean, if you are not using the full power of your tools, not only you are dragging yourself down, but the newbie you are trying to be kind with will not learn anything new.
When I see something like Duff's Device, my first reaction is a big WTF, like everyone else, but then, I sit down, try to understand, and when I get it, I am happy because I learned something. In fact, seeing Duff's Device is really what help me understand switch/case in C. And Duff's Device is certainly outdated, but it is by no mean "bad". If you need to unroll a loop by hand, that a good way to do it, it is just that you don't need to unroll loops anymore.
So if you write code I have to read later, please don't hold back, use every clever trick you have. A small comment would be appreciated but not necessary. This way, I can read your code and think "I learned something today".
Does it slows down comprehension? Of course it does... once. That's the cost of learning, but it is an investment, next time, I'll be faster.
I am really talking about the cleverness the article disapproves of. I also value insight, but here, I agree with the article.
I know a common saying is "newbies should understand your code at first glance". I only agree when by newbies, it means newbies on the project with regards to domain-specific knowledge. But when it comes to general principles, including knowledge of the programming language, code should be written for the highest level of expertise you can reach (but no further, you have to understand what you a doing!). I mean, if you are not using the full power of your tools, not only you are dragging yourself down, but the newbie you are trying to be kind with will not learn anything new.
Newbie can also be yourself looking at your own code a year later. You will have forgotten a lot and will have made it harder for yourself to re-understand what you were doing.
There's a difference between learning stuff and applying stuff.
Its about not making the code more complicated and obscure than necessary. Using generic language constructs like switch/case - sure. I don't think that falls in the category of "clever".
There's a difference between learning stuff and applying stuff.
Its about not making the code more complicated and obscure than necessary. Using generic language constructs like switch/case - sure. I don't think that falls in the category of "clever".
> Newbie can also be yourself looking at your own code a year later. You will have forgotten a lot and will have made it harder for yourself to re-understand what you were doing.
I hear this argument a lot but maybe that's just me but I don't forget much. I don't have perfect memory, far from it, but remembering my thought process when faced with my old code, even decades old is not really a problem for me, especially if a lot of though was put into it, as it tends to be the case with "clever" code.
And it can also an important part of the learning process, reading old code is a good way to take a step back and seeing what really are the good ideas and the ones that are terrible.
Also, I don't think there is too much of a difference between learning and applying, especially not in programming. That's one thing I love with this job, there is always more to learn, and while theory is also important, it happens by actually doing things. Boring, repetitive tasks are for the computer, not the developer.
And I totally agree that one should not make the code more complicated than necessary, but what it necessary is maybe the hardest skill to master.
As for Duff's Device is definitely "clever", few people realize that switch/case in C are that closely related to goto, it is not how they are usually taught and I certainly didn't before seeing Duff's Device. And there are more straightforwards ways of manually unrolling loops that don't have you pause and think.
I hear this argument a lot but maybe that's just me but I don't forget much. I don't have perfect memory, far from it, but remembering my thought process when faced with my old code, even decades old is not really a problem for me, especially if a lot of though was put into it, as it tends to be the case with "clever" code.
And it can also an important part of the learning process, reading old code is a good way to take a step back and seeing what really are the good ideas and the ones that are terrible.
Also, I don't think there is too much of a difference between learning and applying, especially not in programming. That's one thing I love with this job, there is always more to learn, and while theory is also important, it happens by actually doing things. Boring, repetitive tasks are for the computer, not the developer.
And I totally agree that one should not make the code more complicated than necessary, but what it necessary is maybe the hardest skill to master.
As for Duff's Device is definitely "clever", few people realize that switch/case in C are that closely related to goto, it is not how they are usually taught and I certainly didn't before seeing Duff's Device. And there are more straightforwards ways of manually unrolling loops that don't have you pause and think.
> I hear this argument a lot but maybe that's just me but I don't forget much. I don't have perfect memory, far from it, but remembering my thought process when faced with my old code, even decades old is not really a problem for me
That might be true for you. But I have met too many people that say this about themselves, but then I see them "understanding" their old code (i.e. looking around and not having a clue). In their cases, its just self-deception.
Its the same people that say they don't need comments, but then they look around their own files with confused faces.
But yeah there are definitely also people with good memory. Those are also the people that happen to, for example, put more thought in structuring their code.
> Boring, repetitive tasks are for the computer, not the developer.
Unfortunately we have to do a lot of boring, repetitive tasks when developing that could be done by the computer ;-(
That might be true for you. But I have met too many people that say this about themselves, but then I see them "understanding" their old code (i.e. looking around and not having a clue). In their cases, its just self-deception.
Its the same people that say they don't need comments, but then they look around their own files with confused faces.
But yeah there are definitely also people with good memory. Those are also the people that happen to, for example, put more thought in structuring their code.
> Boring, repetitive tasks are for the computer, not the developer.
Unfortunately we have to do a lot of boring, repetitive tasks when developing that could be done by the computer ;-(
A great source of both clever and insightful code can be found at https://codegolf.stackexchange.com/. Code golf is the sport of writing the smallest amount of code that solves a problem. It requires both clever use of syntax to write short code, but also insight in how to reduce the problem to something much simpler than it at first appears. For example, take this problem that checks if the given color name is one of the colors of the rainbow, where the insight is that you can create a perfect hash function to solve it: https://codegolf.stackexchange.com/questions/214678/is-it-a-...
I think "clever" is being applied at one level here and "insightful" is being applied at another.
You can write code at the equivalent of an 8th grade reading level like Hemingway wrote.
Each sentence can can be basic using a small vocabulary. Put together the whole novel can have depth and be insightful.
Its the difference between Hemingway and an actual random 8th grader.
You can write code at the equivalent of an 8th grade reading level like Hemingway wrote.
Each sentence can can be basic using a small vocabulary. Put together the whole novel can have depth and be insightful.
Its the difference between Hemingway and an actual random 8th grader.
I don't think "good" vs "bad" has anything at all to do with "platform" vs "problem".
It's really a question of "essential or extremely useful to the project goals" vs "fun/satisfying for the person being clever". That's the distinction we should focus on.
It's really a question of "essential or extremely useful to the project goals" vs "fun/satisfying for the person being clever". That's the distinction we should focus on.
If you change it slightly you can make it more readable:
Adding a docstring or comment can help too.
Is that still "clever code" after these few changes? I would say no.
def is_unique(elements):
unique_elements = set(elements)
has_only_unique_elements = len(unique_elements) == len(elements)
return has_only_unique_elements
Adding a docstring or comment can help too.
Is that still "clever code" after these few changes? I would say no.
Anyone who's written a fair share of Python can immediately see "len(set(l)) == len(l)" as "no duplicate elements", this is such a standard way to check for duplicates in Python that the chances are you've see this expression in this exact form dozens if not hundreds of times already.
Agreed with "l" identifier and docstring/comment. You'd probably write it like this if you want it to look nice:
Agreed with "l" identifier and docstring/comment. You'd probably write it like this if you want it to look nice:
from typing import Iterable, Any
def is_unique(elements: Iterable[Any]) -> bool:
"Returns True if the argument has no duplicate elements."
elements = list(elements)
return len(elements) == len(set(elements))I know Python but my Python is not the most Pythonic.
The typing does make it more readable, but the "anyone who's written a fair share of Python can immediately see..." argument doesn't resonate with me.
Naming boolean expressions saves time.
The typing does make it more readable, but the "anyone who's written a fair share of Python can immediately see..." argument doesn't resonate with me.
Naming boolean expressions saves time.
They did name the boolean expression; they named it `is_unique`. It's a poor name, but that's a different argument.
It blows my mind that anyone would consider your refactoring more readable. I don't think the example in OP suffers from readability issues at all, nor did the article suggest that. If anything it was just good, simple, 'clever' code.
Named boolean expressions, when used properly, make the effort of reading the actual expression optional.
The example given about the 120 years old assumption will very likely be broken in the next 10 to 20 years... Perhaps it is better and safer to just stick to simple code.
Is it simpler, though? It has fewer assumptions, but writing quicksort from scratch is probably a lot more challenging than putting items into buckets in a single pass.
[deleted]
It's not easy to do things the simple way, being clever is the easy path. Knowing when you need to be clever and when simple is enough is what separates experience from greenhorns.