Surgical Programming(macoy.me)
macoy.me
Surgical Programming
https://macoy.me/blog/programming/SurgicalProgramming
28 comments
I'm sure that you can get away with a less thorough "pre-op", but how many times have we all made changes to code and then realized our changes didn't work after-the-fact? Even in very complex systems I've worked with, I actually find that not spending enough time understanding the code tends to result in small errors rather than big ones: by the time I feel confident enough to write code, I can usually write something that's basically (but not entirely) correct. As I read it, the author is imploring us to practice putting in the time upfront to avoid these kinds of mistakes.
You can even apply this idea in a literal sense around pre-op preparation too.
I've often talked about and practice this for certain types of code changes. For example let's say you have a big chunk of code in 1 function or file (how you got into this situation isn't important).
When it comes to real life surgery often times the area being operated on will get isolated. Maybe there's a blanket over everything with a cut out near your stomach or whatever is being operated on.
You can do the same thing with large chunks of code. If you plan to modify something or add something new you can add a bunch of line breaks before and after the area giving you some form of isolation. If you don't do this you can easily get lost in the details of the things around it, especially if it's someone else's code. You'll be surprised at how useful this ends up being to keep you focused.
I've often talked about and practice this for certain types of code changes. For example let's say you have a big chunk of code in 1 function or file (how you got into this situation isn't important).
When it comes to real life surgery often times the area being operated on will get isolated. Maybe there's a blanket over everything with a cut out near your stomach or whatever is being operated on.
You can do the same thing with large chunks of code. If you plan to modify something or add something new you can add a bunch of line breaks before and after the area giving you some form of isolation. If you don't do this you can easily get lost in the details of the things around it, especially if it's someone else's code. You'll be surprised at how useful this ends up being to keep you focused.
I do something similar except I just stub in comments with a description of the functionality I think may need to exist there. It really does feel like surgery and allows for more high level planning before committing to write a line of code.
Reading code is so hard for years I thought it just wasn't possible to get good at it.
I could probably write code good code about 10 years ago, but only felt comfortable reading about 2 years ago.
Like the author, I now prefer to read the code I'm calling than docs, tutorials, function signatures, etc.
Getting comfortable reaching for a debugger helped immensely.
If you prefer Googling for docs to debugging the tools you're using... there's another level to this game!
I could probably write code good code about 10 years ago, but only felt comfortable reading about 2 years ago.
Like the author, I now prefer to read the code I'm calling than docs, tutorials, function signatures, etc.
Getting comfortable reaching for a debugger helped immensely.
If you prefer Googling for docs to debugging the tools you're using... there's another level to this game!
I remember being amazed at people who would just read the source to figure things out. Somewhere along the way, I became one of them.
Turns out, it takes a really long time. If any junior devs are reading this, be patient and enjoy the ride :)
Turns out, it takes a really long time. If any junior devs are reading this, be patient and enjoy the ride :)
My work in the past consisted mostly of dissecting and refactoring large legacy codebases. Usually after anybody who actually understood it already left and where the codebase is too large and rotten for anybody new to be able or willing to learn it.
I have my own process I apply for any change to code. The basic goal is to introduce zero defects to the code that I do not control/know. Which is crucial when I want to make thousands of changes to a huge application where if I make a mistake it can be very costly to the client, can cost me a huge amount of effort to fix and can completely undermine my effort.
Usually there is no good way to test the changes. I make hundreds or thousands of individual refactors, bundle them up and kind of hope I haven't broken anything in the process.
Some of these applications are just too large for one person to be able to comprehend. They either have vast user interfaces or vast network of integrations. That and no ability to test it in automated way.
This is the reasons why these applications usually aren't taken good care of -- new developers (new, not necessarily junior) are just too scared and unprepared to do anything than smallest changes and that with huge amount of effort.
From my experience, the programming language and environment plays enormous role in making this possible.
The most basic questions I may have when analysing the code I want to change are:
1) What is the thing I am looking at? For example, the exact type of variable, parameter, code for method, etc.
2) Where is the thing I am looking at referenced from? Usually when I want to modify a piece of badly written code, I have to be able to find and analyse every single usage of it.
3) When is the thing I am looking at being executed?
It is not just the ability to answer these questions -- I need to be able to answer them with absolute precision.
Since I program mostly Java, all that information is usually available to me. #3 is sometimes a little bit more tricky but if you can hunt all usages you can typically get to it with some effort.
But that is not the case for most programming languages these days. For example, I tried this with a large Python application and I have utterly failed and had to tell my client I can't help them. Which tells me that most programming languages are not suited to building large, complicated systems that somebody might want to be able to analyse and refactor in the future.
I have my own process I apply for any change to code. The basic goal is to introduce zero defects to the code that I do not control/know. Which is crucial when I want to make thousands of changes to a huge application where if I make a mistake it can be very costly to the client, can cost me a huge amount of effort to fix and can completely undermine my effort.
Usually there is no good way to test the changes. I make hundreds or thousands of individual refactors, bundle them up and kind of hope I haven't broken anything in the process.
Some of these applications are just too large for one person to be able to comprehend. They either have vast user interfaces or vast network of integrations. That and no ability to test it in automated way.
This is the reasons why these applications usually aren't taken good care of -- new developers (new, not necessarily junior) are just too scared and unprepared to do anything than smallest changes and that with huge amount of effort.
From my experience, the programming language and environment plays enormous role in making this possible.
The most basic questions I may have when analysing the code I want to change are:
1) What is the thing I am looking at? For example, the exact type of variable, parameter, code for method, etc.
2) Where is the thing I am looking at referenced from? Usually when I want to modify a piece of badly written code, I have to be able to find and analyse every single usage of it.
3) When is the thing I am looking at being executed?
It is not just the ability to answer these questions -- I need to be able to answer them with absolute precision.
Since I program mostly Java, all that information is usually available to me. #3 is sometimes a little bit more tricky but if you can hunt all usages you can typically get to it with some effort.
But that is not the case for most programming languages these days. For example, I tried this with a large Python application and I have utterly failed and had to tell my client I can't help them. Which tells me that most programming languages are not suited to building large, complicated systems that somebody might want to be able to analyse and refactor in the future.
#2 and #3 are very difficult in Java when working with frameworks like Spring. Spring obscures much of its internal mechanisms, and it takes a long time to build the knowledge needed to understand all of the nuances in it (and to even debug it). Jonathan Blow's arguments against frameworks in general apply strongly to the Spring Framework.
#1 is usually solved by using strongly typed languages, like Java or plenty of others
#1 is usually solved by using strongly typed languages, like Java or plenty of others
In general case you would be right -- Spring does complicate things by hiding some of the actual references.
But reality is that most projects use very simple dependency injection rules. Most use just singleton beans and occasionally profiles for different environments (mostly to access external components like databases or APIs) and that is practically the extent of the problem.
Spring is flexible on paper. In reality (and very fortunately for me), most projects follow the examples and conventions you can find on the Internet. Which makes pretty much every Spring application look like any other Spring app, even if they don't have to.
By giving defined, recognisable structure to the application, Spring actually makes part of work easier. When I see controllers, repositories, services, etc., I usually know what I can expect.
Problems start when you find a project written by "smart" developers. These people are smart enough to write very complex structures and "customise" their Spring experience, but not smart enough to figure out they should separate app development from satisfying their intellectual curiosity.
But reality is that most projects use very simple dependency injection rules. Most use just singleton beans and occasionally profiles for different environments (mostly to access external components like databases or APIs) and that is practically the extent of the problem.
Spring is flexible on paper. In reality (and very fortunately for me), most projects follow the examples and conventions you can find on the Internet. Which makes pretty much every Spring application look like any other Spring app, even if they don't have to.
By giving defined, recognisable structure to the application, Spring actually makes part of work easier. When I see controllers, repositories, services, etc., I usually know what I can expect.
Problems start when you find a project written by "smart" developers. These people are smart enough to write very complex structures and "customise" their Spring experience, but not smart enough to figure out they should separate app development from satisfying their intellectual curiosity.
I feel the reason most Spring applications look similar is because the Developers have been copy-pasting from the same Baeldung-articles.
Copy-paste driven development is something you get a lot of when you are working with frameworks. Since the framework is itself inscrutable, it's nearly impossible to properly understand. So developers copy.
Copy-paste driven development is something you get a lot of when you are working with frameworks. Since the framework is itself inscrutable, it's nearly impossible to properly understand. So developers copy.
I wouldn't say that is necessarily fault of Spring (though scope creep might be an important contributor).
I think the basic causes are:
1) Because most developers almost never need more than what you can find on Baeldung/Stack*. And even if they need, they need it for very small pieces of the application and the rest is just boilerplate.
2) Because you are penalised for customising the structure of your Spring application. You do that and you expose yourself to various complicated problems.
3) Because, frankly, Java is a bad programming language from the point of view of building abstractions. I would personally prefer to program applications in Common Lisp or even Clojure (or any other Lisp for that matter) if only for some reason these projects did not typically end in unmaintainable mess even well before Java projects do. It is truly tragic that Lisp and strong typing are fundamentally incompatible with each other.
There are different types of unmaintainable mess -- mess caused by copying and pasting stuff typically committed by novice developers who don't know any better tends to be infinitely easier to clean up than mess caused by "advanced" developers running amok with macros, continuations, generators and async callbacks.
4) Because developers are not taught how to structure their code. Nowadays getting something working is all that matters. And this is usually best accomplished by copying/pasting code blocks Lego-style from a dozen different tutorials. How anybody is going to learn to structure their large codebase if they never learned to structure a smaller one?
People who advance faster are the ones that are more efficient at copying pasting, not the ones that are better at writing better code.
I think the basic causes are:
1) Because most developers almost never need more than what you can find on Baeldung/Stack*. And even if they need, they need it for very small pieces of the application and the rest is just boilerplate.
2) Because you are penalised for customising the structure of your Spring application. You do that and you expose yourself to various complicated problems.
3) Because, frankly, Java is a bad programming language from the point of view of building abstractions. I would personally prefer to program applications in Common Lisp or even Clojure (or any other Lisp for that matter) if only for some reason these projects did not typically end in unmaintainable mess even well before Java projects do. It is truly tragic that Lisp and strong typing are fundamentally incompatible with each other.
There are different types of unmaintainable mess -- mess caused by copying and pasting stuff typically committed by novice developers who don't know any better tends to be infinitely easier to clean up than mess caused by "advanced" developers running amok with macros, continuations, generators and async callbacks.
4) Because developers are not taught how to structure their code. Nowadays getting something working is all that matters. And this is usually best accomplished by copying/pasting code blocks Lego-style from a dozen different tutorials. How anybody is going to learn to structure their large codebase if they never learned to structure a smaller one?
People who advance faster are the ones that are more efficient at copying pasting, not the ones that are better at writing better code.
Why are Lisp and typing incompatible with eachother? Sounds like an interesting problem.
[deleted]
Writing complex application in Python or any other dynamic languages is a madness anyway. I often witnessed cases when developers even reused the same variable for different purposes and types just because the language allows it.
> Now, I rarely ever read online documentation. The code is the ground truth, and additional context can be gleaned from reading the version control system logs for the file. I try to read much more functions in their entirety before using them. When I have a question about some functionality, I try to read the code before asking the original developers for help.
I agree to an extent, but to me this quote also says something about the type and quality of the libraries that he has ended up interacting with.
I invite him to use the same technique on the plurality of academic code (version control logs? hah! context? hah! functions??? hah!). Most academic codebases one inherits are typically 'cram everything in one file, no functions, obscure comments, no variable names other than letters of the alphabet, huge sections of commented out code, and the only documentation is the accompanying research paper'.
Not to mention, half of that is typically higher level wraps around obscure 80's fortran code that is worshipped with reverence and largely treated as a magical black box that should not be tampered with.
Confirm something in the source, you say? Guaranteed yak shaving for days!
I agree to an extent, but to me this quote also says something about the type and quality of the libraries that he has ended up interacting with.
I invite him to use the same technique on the plurality of academic code (version control logs? hah! context? hah! functions??? hah!). Most academic codebases one inherits are typically 'cram everything in one file, no functions, obscure comments, no variable names other than letters of the alphabet, huge sections of commented out code, and the only documentation is the accompanying research paper'.
Not to mention, half of that is typically higher level wraps around obscure 80's fortran code that is worshipped with reverence and largely treated as a magical black box that should not be tampered with.
Confirm something in the source, you say? Guaranteed yak shaving for days!
Good ability to read code is what separates agood developer from not that good.
A good developer can work on any project. A not so good developer can only work in smaller not so complex projects written in their favorite framework with certain patterns.
A good developer can solve any task. A not so good developer says that the task (or the whole project) is wrong if is too difficult to do in their favorite framework or with their favorite pattern.
A good developer can work on any project. A not so good developer can only work in smaller not so complex projects written in their favorite framework with certain patterns.
A good developer can solve any task. A not so good developer says that the task (or the whole project) is wrong if is too difficult to do in their favorite framework or with their favorite pattern.
> When I have a question about some functionality, I try to read the code before asking the original developers for help.
If I were the author of such a piece, I would appreciate if you asked first.
First and foremost because it's faster that way. Second, should an implementation have bugs which still produce an ostensibly correct result, you won't catch them unless you know the intent behind that piece of code. Reading alone is not guaranteed to provide enough context.
This is part of the reason why legacy code is so hard to maintain - over a certain point it's hard to tell whether something was put there by accident, or is there a profound reason for it being that way.
Bottom line is: you can't assume that what you read is perfectly correct, so it's more valuable to understand the underlying intent rather than the inner workings of something.
If I were the author of such a piece, I would appreciate if you asked first.
First and foremost because it's faster that way. Second, should an implementation have bugs which still produce an ostensibly correct result, you won't catch them unless you know the intent behind that piece of code. Reading alone is not guaranteed to provide enough context.
This is part of the reason why legacy code is so hard to maintain - over a certain point it's hard to tell whether something was put there by accident, or is there a profound reason for it being that way.
Bottom line is: you can't assume that what you read is perfectly correct, so it's more valuable to understand the underlying intent rather than the inner workings of something.
> First and foremost because it's faster that way.
That seems unlikely to scale, and all but guaranteed to age poorly. (The original developers will not always be available; eventually, they'll be busy or no longer with the company)
> Reading alone is not guaranteed to provide enough context.
That's what comments are for.
That seems unlikely to scale, and all but guaranteed to age poorly. (The original developers will not always be available; eventually, they'll be busy or no longer with the company)
> Reading alone is not guaranteed to provide enough context.
That's what comments are for.
> First and foremost because it's faster that way
No it is often not faster, most developers don't remember what they wrote or why they wrote it for code they wrote a while ago. They would have to read the code themselves and get up to speed again to understand it and then come up with a good explanation for you, and once you involve two persons trying to understand one problem it often takes more person time than before.
And since you just found an issue related to that piece of code, chances are you actually got better context than the other developer, and getting him up to speed about the issue would take most of the time.
No it is often not faster, most developers don't remember what they wrote or why they wrote it for code they wrote a while ago. They would have to read the code themselves and get up to speed again to understand it and then come up with a good explanation for you, and once you involve two persons trying to understand one problem it often takes more person time than before.
And since you just found an issue related to that piece of code, chances are you actually got better context than the other developer, and getting him up to speed about the issue would take most of the time.
This works in some cases, perhaps even most cases,
but it does not scale.
I reasonable sized monolith, a limited executable.
Looking at things that dont fit - Linux: the code base i vast, even for one sub system. Some sub systems are complex If you dont know enough about architecture and operating systems, computer chips etc etc it will be difficult.
- Distributed system: Here we have some parts in AWS Lambda, Some parts in different microservices, written in different languages. Some parts are extensions to the database to add native features.
I always prefer to start with am architecture document. A high level of all the parts, how they fit together what they do and so on.
Distributed systems are a pain in the ass to debug and get into.
I reasonable sized monolith, a limited executable.
Looking at things that dont fit - Linux: the code base i vast, even for one sub system. Some sub systems are complex If you dont know enough about architecture and operating systems, computer chips etc etc it will be difficult.
- Distributed system: Here we have some parts in AWS Lambda, Some parts in different microservices, written in different languages. Some parts are extensions to the database to add native features.
I always prefer to start with am architecture document. A high level of all the parts, how they fit together what they do and so on.
Distributed systems are a pain in the ass to debug and get into.
> I rarely ever read online documentation. The code is the ground truth
This can create problems. Relying on the current behavior instead of a documented contract makes the code brittler. It's sometimes worth it, but it's not something that makes me go "yeah, this is the way" without mentioning any downside.
This can create problems. Relying on the current behavior instead of a documented contract makes the code brittler. It's sometimes worth it, but it's not something that makes me go "yeah, this is the way" without mentioning any downside.
Relying on the documented contract when it doesn't match the current behavior makes the application broken.
And you typically find out much sooner. That cost is more obvious.
True, but relying on the current behavior when it is only one of many behaviors allowed by the contract prevents the author from changing the implementation, or may make porting the function to other platforms more difficult.
Nice post. Thanks!
Also, I am very drawn to the style of the page. :)
Also, I am very drawn to the style of the page. :)
Bhlkk
> You're no longer "feeling around" while at the same time fighting compiler errors.
> Off-by-one, inverted conditionals, and error-handling are usually very obvious when stepping through code, but difficult to spot when only testing.
These problems feel way more basic than whatever problem you set out to solve initially. Like, several layers away.
If you have to make a fix in an environment that is so complex that you need to fully read many functions and take notes on what they do in order to get the logic right, you are pretty doomed if "will the compiler accept my code" is something that's even on your mind.
Don't get me wrong, it's completely true what the author wrote - it's just strange that these are the problems discussed in a scenario where the code/design is complex enough to warrant an explicit "pre-op" phase. Change my view?
Edit: Sorry, just now read the Conclusion section and it mentions how this article is targeting somewhat junior devs - in that case, I fully concur! ("Cakelisp and GameLib development still continues" primed me for something else.) And yes, the steps do become automatic, even as you deal with more complex problems (i.e. when "will it compile" is no longer conscious thought).