The Grug Brained Developer (2022)(grugbrain.dev)
grugbrain.dev
The Grug Brained Developer (2022)
https://grugbrain.dev/
599 comments
i'm not crying your crying
“Good debugger worth weight in shiny rocks, in fact also more”
I’ve spent time at small startups and on “elite” big tech teams, and I’m usually the only one on my team using a debugger. Almost everyone in the real world (at least in web tech) seems to do print statement debugging. I have tried and failed to get others interested in using my workflow.
I generally agree that it’s the best way to start understanding a system. Breaking on an interesting line of code during a test run and studying the call stack that got me there is infinitely easier than trying to run the code forwards in my head.
Young grugs: learning this skill is a minor superpower. Take the time to get it working on your codebase, if you can.
I’ve spent time at small startups and on “elite” big tech teams, and I’m usually the only one on my team using a debugger. Almost everyone in the real world (at least in web tech) seems to do print statement debugging. I have tried and failed to get others interested in using my workflow.
I generally agree that it’s the best way to start understanding a system. Breaking on an interesting line of code during a test run and studying the call stack that got me there is infinitely easier than trying to run the code forwards in my head.
Young grugs: learning this skill is a minor superpower. Take the time to get it working on your codebase, if you can.
Using a debugger on my own code is easy and I love it.
The second the debugger steps deep into one of the libs or frameworks I'm using, I'm lost and I hate it.
That framework / lib easily has many ten thousands of person-hours under it's belly, and I'm way out of my league.
But you can just use the "step out" feature to get back out when you realise you've gone into a library function. Or "step over" when you can see you're about to go into one.
While that's true, it doesn't necessarily help me understand why my parameters for said lib or whatever lead to things blowing up.
IDEs tend to have a "just my code" option.
"Step out" is how to get out of the lower level frameworks, and or "step over" to avoid diving into them in the first place. I can't speak for other IDEs, but all of the JetBrains products have these.
I'd love to use a real debugger but as someone who has only ever worked at large companies, this was just never an option. In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger. Print debugging is all you have. If there's a problem with the logging system itself or something that crashes the program before the logs can flush, then not even that.
Same, this isn't my choice, debuggers don't work here. And we don't even have microservices.
> debuggers don't work here
It’s impossible? Or would take engineering work to enable
It’s impossible? Or would take engineering work to enable
It's not a realtime system kind of thing where the debugger would change the behavior too much... It's possible with enough engineering work, but nobody has put that work in, in fact they had a debugger for some staging envs that they deleted. Lately they keep adding more red tape making it hard to even run something locally, let alone attach a debugger.
I guess you can attach a debugger for unit tests, but that's not very useful.
I guess you can attach a debugger for unit tests, but that's not very useful.
> I guess you can attach a debugger for unit tests, but that's not very useful.
That is in fact incredibly useful
That is in fact incredibly useful
Eh, it's there for those who want it, but nobody uses it
See this just sounds like you do not have an engineering culture or learning, enabling, and using debuggers.
[deleted]
Not sure what an engineering culture is, but I don't want it. I just want to attach a debugger to our stuff (not the unit tests).
*engineering culture of
It’s a similar muscle to exercise as using a profiler.
It’s a similar muscle to exercise as using a profiler.
At my company our system is composed of 2 dozen different services and all of them can run locally in minikube and easily be debugged in jetbrains.
Yes, it's perfectly doable even if you're doing microservices. Not being able to debug your application is an engineering failure.
can you say more? how do you do it?
We mostly have dotnet services in k8s, using Rider (IDE) and Telepresence for remote debugging. Having observability (OpenTelemetry) is also really useful.
See ninkendo’s comment. They are doing it with the same tools we are.
A multi-process application isn't the same as microservices. Microservices is a team organization technique, seeing individual teams operate in isolation, as if they were individual businesses. You can't debug other team's services any more than you can debug what happens when you make a call to OpenAI. That is the level of separation you are up against in a microservices context. If you can, you're on the same team, and thus don't have microservices, just a single service.
What? We have dozens of microservices owned by multiple teams, but nothing stops you from cloning the git repository of another team's microservice and debug it the same way you would debug your own.
[deleted]
Service is provided by people. You, for example, discover a problem with OpenAI's system that you integrate with and the only way you can address it is to employ the services of the people who work for OpenAI. While that is an example of a macroservice (or what we usually just call a service), it playing out in the macro economy, microservice is the same concept except applied in the micro scale.
But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case?
Or are you just struggling to say that your service utilizes multiple applications?
But you checking out the code and debugging it means that you are providing the service. Where, exactly, do you find the service (micro or otherwise) boundary in this case?
Or are you just struggling to say that your service utilizes multiple applications?
Sorry but I fail to see how your comments are relevant in this discussion.
Where do configs and secrets come from? Also, big company = hundreds of microservices.
In my experience you just slap minikube or k3s on your dev machine, and treat it as any other environment. Argo, helm, kustomize, whatever can all work against a local single-node cluster just fine. It takes some effort to make sure your configs are overridable per-environment, but it’s worth doing. (And something you’re hopefully doing anyway if you’re doing any kind of integration/test environment.)
It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.
It also requires that each of your services can scale down as well as they can scale up… none of them should be so huge that your whole cluster can’t fit on a single machine, if you’re just simulating one “request” at a time. (Single instances of everything, don’t reserve memory, etc.) There’s use cases where this isn’t practical, but in most cases it’s very doable.
Yes all of this.
I certainly know how to debug each of the services in my environment, but how do you step-through debug a single request across services? Like, if service A make a gRPC call to service B, are you saying you can “step into” the call from A and your debugger is able to break on the corresponding call in B? And frames from the call in A are there in a logical “stack” from the breakpoint in B?
(Honest question… if such a workflow is possible I’d love to hear about it. Debugging just a single service at a time in isolation is boring and obvious, but if there’s something I’m missing I’d be really curious.)
(Honest question… if such a workflow is possible I’d love to hear about it. Debugging just a single service at a time in isolation is boring and obvious, but if there’s something I’m missing I’d be really curious.)
No I can’t debug multiple services at once, unfortunately. But I will switch between them as I track the request over multiple runs. Also extensive logging in available in grafana helps me know which service is having the issue before I start debugging.
This is usually enough for me, too. Use tracing, figure out where things fell apart in the traces, isolate those service(s), and debug from there. It's definitely more work. When we start new projects, I encourage people not to use services until proven necessary because this added layer of friction is a real drag. For a lot of us that isn't a choice, though.
This is basically it. When I started programming in C, I used a debugger all the time. Even a bit later doing Java monoliths I could spin up the whole app on my local and debug in the IDE. But nowadays running a dozen processes and containers and whatnot, it's just hopeless. The individual developer experience has gone very much backwards in the microservice era so the best thing to do is embrace observability, feature toggles etc and test in prod or a staging environment somewhere outside of your local machine.
> In a microservices mesh architecture, you can't really run anything locally at all, and the test environment is often not configured to allow hooking up a stepping debugger.
I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)
I don't often use a debugger, and I still feel the need to point out Visual Studio could step into DCOM RPCs across machines in the first release of DCOM, ca. 1995. (The COM specification has a description of how that is accomplished.)
Curious to learn more about why it is difficult to debug. I'm not familiar with service mesh. I also work at a large corp, but we use gateways and most things are event driven with kafka across domain boundaries. I spend most of my time debugging each service locally by injecting mock messages or objects. I do this one at a time if the problem is upstream. Usually, our logging helps us pinpoint the exact service at to target for debugging. It's really easy. Our devops infrastructure has built out patterns and libraries when teams need to mint a new service. Everything is standardized with terraform. Everything has the same standard swagger pages, everything is using okta, etc.. Seems a bit boring (which is good)
It's easy if Someone(tm) has already set up a system where you can whip up the relevant bits with something like localstack.
But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.
But if there is no support from anyone and you'll be starting from scratch, you've print-debugged and fixed the issue before you get the debugger attached to anything relevant.
I suppose in the description of "large corp" and service mesh, Someone would already exist and this would already have been worked out. It would be a nightmare dealing with hundreds of microservices without this kind of game plan.
Well, what's your workflow? Is there a particular debugger that you love?
I’ve learned not to go against the grain with tools, at least at big companies. Probably some dev productivity team has already done all the annoying work needed to make the company’s codebase work with some debugger and IDE, so I use that: currently, it’s VS Code and LLDB, which is fine. IntelliJ and jdb at my last job was probably better overall.
My workflow is usually:
1. insert a breakpoint on some code that I’m trying to understand
2. attach the debugger and run any tests that I expect to exercise that code
3. walk up and down the call stack, stepping occasionally, reading the code and inspecting the local variables at each level to understand how the hell this thing works and why it’s gone horribly wrong this time.
4. use my new understanding to set new, more relevant breakpoints; repeat 2-4.
Sometimes I fiddle with local variables to force different states and see what happens, but I consider this advanced usage, and anyway it often doesn’t work too well on my current codebase.
My workflow is usually:
1. insert a breakpoint on some code that I’m trying to understand
2. attach the debugger and run any tests that I expect to exercise that code
3. walk up and down the call stack, stepping occasionally, reading the code and inspecting the local variables at each level to understand how the hell this thing works and why it’s gone horribly wrong this time.
4. use my new understanding to set new, more relevant breakpoints; repeat 2-4.
Sometimes I fiddle with local variables to force different states and see what happens, but I consider this advanced usage, and anyway it often doesn’t work too well on my current codebase.
I've been doing this professionally for over a decade and have basically never used a debugger
I've often felt that I should, but never enough to actually learn how
I've often felt that I should, but never enough to actually learn how
There was a good discussion on this topic years ago [0]. The top comment shares this quote from Brian Kernighan and Rob Pike, neither of whom I'd call a young grug:
> As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.
I tend to agree with them on this. For almost all of the work that I do, this hypothesis-logs-exec loop gets me to the answer substantially faster. I'm not "trying to run the code forwards in my head". I already have a working model for the way that the code runs, I know what output I expect to see if the program is behaving according to that model, and I can usually quickly intuit what is actually happening based on the incorrect output from the prints.
[0] The unreasonable effectiveness of print debugging (349 points, 354 comments) April 2021 https://news.ycombinator.com/item?id=26925570
> As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.
I tend to agree with them on this. For almost all of the work that I do, this hypothesis-logs-exec loop gets me to the answer substantially faster. I'm not "trying to run the code forwards in my head". I already have a working model for the way that the code runs, I know what output I expect to see if the program is behaving according to that model, and I can usually quickly intuit what is actually happening based on the incorrect output from the prints.
[0] The unreasonable effectiveness of print debugging (349 points, 354 comments) April 2021 https://news.ycombinator.com/item?id=26925570
Adding these print statements is one of my favorite LLM use cases.
Hard to get wrong, tedious to type and a huge speed increase to visually scan the output.
Hard to get wrong, tedious to type and a huge speed increase to visually scan the output.
Agreed. Typically my debugger use case is when I'm exploring a potentially unknown range of values at a specific point in time, where I also might not know how to log it out. Having the LLM manage all of that for me and get it 95% correct is the real minor superpower.
I do a lot of print statements as well. I think the greatest value of debuggers comes when I’m working on a codebase where I don’t already have a strong mental model, because it lets me read the code as a living artifact with states and stack traces. Like Rob Pike, I also find single-stepping tedious.
> Brian Kernighan and Rob Pike
Most of us aren't Brian Kernighan or Rob Pike.
I am very happy for people who are, but I am firmly at a grug level.
Most of us aren't Brian Kernighan or Rob Pike.
I am very happy for people who are, but I am firmly at a grug level.
This! Also my guess would be Kernighan or Pike aren't (weren't?) deployed into some random codebase every now and then, while most grugs are. When you build something from scratch then you can get by without debuggers, sure, but foreign codebase, a stupid grug like I can do much better with tools.
I think a lot of “naturals” find visual debuggers pointless, but for people who don’t naturally intuit how a computer works it can be invaluable in building that intuition.
I insist that my students learn a visual debugger in my classes for this reason: what the "stack" really is, how a loop really executes, etc.
It doesn't replace thinking & print debugging, but it complements them both when done properly.
I insist that my students learn a visual debugger in my classes for this reason: what the "stack" really is, how a loop really executes, etc.
It doesn't replace thinking & print debugging, but it complements them both when done properly.
What do you mean “visual debugger?”
Presumably an IDE rather then dealing the gdb CLI.
I think it depends on the debugger and the language semantics as well. Debugging in Swift/Kotlin, so so. The Smalltalk debugger was one of the best learning tools I ever used. “Our killer app is our debugger” doesn’t win your language mad props though.
I don't often need gdb, but I appreciate the emacs mode that wraps it every time.
In vscode when you step to the next statement it highlights in the left pane the variables that change. Something like that.
It's useful for a beginner e.g in a for loop to see how `i` changes at the end of the loop. And similarly with return values of functions and so on.
It's useful for a beginner e.g in a for loop to see how `i` changes at the end of the loop. And similarly with return values of functions and so on.
the intellij debugger, for example, as opposed to a command line debugger
Agreed, I spent a lot more time using debuggers when I was getting started
Might be something to this. I relied heavily on IDE debugger and was integral part of my workflow for a while as a yungin and but very rare that I bother these days (not counting quick breaks in random webapps).
Perhaps been underappreciating the gap in mental intuition of the runtime between then and now and how much the debugger helped to bridge.
Perhaps been underappreciating the gap in mental intuition of the runtime between then and now and how much the debugger helped to bridge.
I tend not to use a debugger for breakpoints but I use it a lot for watchpoints because I can adjust my print statements without restarting the program
Exactly, these judiciously placed print statements help me locate the site of the error much faster than using a debugger. Then, I could switch to using a debugger once I narrow things down if I am still unsure about the cause of the problem.
There's another story I heard once from Rob Pike about debugging. (And this was many years ago - I hope I get the details right).
He said that him and Brian K would pair while debugging. As Rob Pike told it, he would often drive the computer, putting in print statements, rerunning the program and so on. Brian Kernighan would stand behind him and quietly just think about the bug and the output the program was generating. Apparently Brian K would often just - after being silent for awhile - say "oh, I think the bug is in this function, on this line" and sure enough, there it was. Apparently it happened so often enough that he thought Brian might have figured out more bugs than Rob did, even without his hands touching the keyboard.
Personally I love a good debugger. But I still think about that from time to time. There's a good chance I should step away from the computer more often and just contemplate it.
He said that him and Brian K would pair while debugging. As Rob Pike told it, he would often drive the computer, putting in print statements, rerunning the program and so on. Brian Kernighan would stand behind him and quietly just think about the bug and the output the program was generating. Apparently Brian K would often just - after being silent for awhile - say "oh, I think the bug is in this function, on this line" and sure enough, there it was. Apparently it happened so often enough that he thought Brian might have figured out more bugs than Rob did, even without his hands touching the keyboard.
Personally I love a good debugger. But I still think about that from time to time. There's a good chance I should step away from the computer more often and just contemplate it.
some of my best work as a programmer is done walking my dog or sitting in the forest
[deleted]
Sounds like Rob does use a debugger and it's name is Brian.
I think you're misremembering here, the other party is Ken Thompson not Brian K
I use single-stepping very rarely in practice when using a debugger, except when following through a "value of a variable or two". Yet it's more convenient than pprint.pprint() for that because structured display of values, eval expression, and ability to inspect callers up the stack.
On the other hand, John Carmack loves debuggers - he talks about the importance of knowing your debugging tools and using them to step through a complex system in his interview with Lex Friedman. I think it's fair to say that there's some nuance to the conversation.
My guess is that:
- Debuggers are most useful when you have a very poor understanding of the problem domain. Maybe you just joined a new company or are exploring an area of the code for the first time. In that case you can pick up a lot of information quickly with a debugger.
- Print debugging is most useful when you understand the code quite well, and are pretty sure you've got an idea of where the problem lies. In that case, a few judicious print statements can quickly illuminate things and get you back to what you were doing.
My guess is that:
- Debuggers are most useful when you have a very poor understanding of the problem domain. Maybe you just joined a new company or are exploring an area of the code for the first time. In that case you can pick up a lot of information quickly with a debugger.
- Print debugging is most useful when you understand the code quite well, and are pretty sure you've got an idea of where the problem lies. In that case, a few judicious print statements can quickly illuminate things and get you back to what you were doing.
It depends on the domain. Any complex long lived mutable state, precise memory management or visual rendering probably benefits from debuggers.
Most people who work on crud services do not see any benefit from it, as there is practically nothing going on. Observing input, outputs and databases is usually enough, and when it's not a well placed log will suffice. Debuggers will also not help you in distributed environments, which are quite common with microservices.
Most people who work on crud services do not see any benefit from it, as there is practically nothing going on. Observing input, outputs and databases is usually enough, and when it's not a well placed log will suffice. Debuggers will also not help you in distributed environments, which are quite common with microservices.
Debuggers absolutely help in distributed environments, in the exact same way that they help with multithreaded debugging of a single process. It is certainly requires a little bit more setup, but there isn't some essential aspect of a distributed environment that precludes the techniques of a debugger.
The only real issue in debugging a distributed/multithreaded environment is that frequently there is a timeout somewhere that is going to kill one of the threads that you may have wanted to continue stepping through after debugging a different thread.
The only real issue in debugging a distributed/multithreaded environment is that frequently there is a timeout somewhere that is going to kill one of the threads that you may have wanted to continue stepping through after debugging a different thread.
Is there a name for an approach to debugging that requires neither debuggers nor print calls? It works like this:
1. When you get a stack trace from a failure, without knowing anything else find the right level and make sure it has sensible error handling and reporting.
1a. If the bug is reproduced but the program experiences no failure & associated stack trace, change the logic such that if this bug occurs then there is a failure and a stack trace.
1b. If the failure is already appropriately handled but too high up or relevant details are missing, fix that by adding handling at a lower level, etc.
That is the first step, you make the program fail politely to the user and allow through some debug option recover/report enough state to explain what happened (likely with a combination of logging, stack trace, possibly app-specific state).
Often it can also be the last step, because you can now dogfood that very error handling to solve this issue along with any other future issue that may bubble up to that level.
If it is not enough you may have to resort to debugging anyway, but the goal is to make changes that long-term make the use of either debuggers or print statements unnecessary in the first place, ideally even before the actual fix.
1. When you get a stack trace from a failure, without knowing anything else find the right level and make sure it has sensible error handling and reporting.
1a. If the bug is reproduced but the program experiences no failure & associated stack trace, change the logic such that if this bug occurs then there is a failure and a stack trace.
1b. If the failure is already appropriately handled but too high up or relevant details are missing, fix that by adding handling at a lower level, etc.
That is the first step, you make the program fail politely to the user and allow through some debug option recover/report enough state to explain what happened (likely with a combination of logging, stack trace, possibly app-specific state).
Often it can also be the last step, because you can now dogfood that very error handling to solve this issue along with any other future issue that may bubble up to that level.
If it is not enough you may have to resort to debugging anyway, but the goal is to make changes that long-term make the use of either debuggers or print statements unnecessary in the first place, ideally even before the actual fix.
In order for this to cover enough space, I assume you‘d have to really pin down assumptions with asserts and so on in a design by contract style.
A different domain where debuggers are less useful: audio/video applications that sit in a tight, hardware driven loop.
In my case (a digital audio workstation), a debugger can be handy for figuring out stuff in the GUI code, but the "backend" code is essentially a single calltree that is executed up to a thousands times a second. The debugger just isn't much use there; debug print statements tend to be more valuable, especially if a problem would require two breakpoints to understand. For audio stuff, the first breakpoint will often break your connection to the hardware because of the time delay.
Being able to print stacktraces from inside the code is also immensely valuable, and when I am debugging, I use this a lot.
In my case (a digital audio workstation), a debugger can be handy for figuring out stuff in the GUI code, but the "backend" code is essentially a single calltree that is executed up to a thousands times a second. The debugger just isn't much use there; debug print statements tend to be more valuable, especially if a problem would require two breakpoints to understand. For audio stuff, the first breakpoint will often break your connection to the hardware because of the time delay.
Being able to print stacktraces from inside the code is also immensely valuable, and when I am debugging, I use this a lot.
Seems to be the opposite for me. Usually I can pretty quickly figure out how to garden hose a bunch of print statements everywhere in a completely unfamiliar domain and language.
But the debugger is essential for all the hard stuff. I’ll take heap snapshots and live inside lldb for days tracking down memory alignment issues. And print statements can be either counterproductive at best, or completely nonexistent in embedded or GPU bound compute.
But the debugger is essential for all the hard stuff. I’ll take heap snapshots and live inside lldb for days tracking down memory alignment issues. And print statements can be either counterproductive at best, or completely nonexistent in embedded or GPU bound compute.
If using step-by-step debugging is a minor superpower, conditional break-points make you an Omega level programming superthreat.
I'm sorry, but this juxtaposition is very funny to me:
- John Carmack loves debuggers
- Debuggers are most useful when you have a very poor understanding of the problem domain
- John Carmack loves debuggers
- Debuggers are most useful when you have a very poor understanding of the problem domain
I think you should substitute “code” for “domain” in the last paragraph.
John Carmack knows his domain very well. He knows what he expects to see. The debugger gives him insight into what “other” developers are doing without having to modify their code.
For Carmack, managing the code of others the debug environment is their safe space. For Kernighan et al in the role of progenitorous developer it is the code itself that is the safe space.
John Carmack knows his domain very well. He knows what he expects to see. The debugger gives him insight into what “other” developers are doing without having to modify their code.
For Carmack, managing the code of others the debug environment is their safe space. For Kernighan et al in the role of progenitorous developer it is the code itself that is the safe space.
If you're doing cutting edge work, then by definition you're in an area you don't fully understand.
This hasn't been my experience.
When I'm unfamiliar with a codebase, or unfamiliar with a particular corner of the code, I find myself reaching for console debugging. Its a bit of a scattershot approach, I don't know what I'm looking for so I console log variables in the vicinity.
Once I know a codebase I want to debug line by line, walking through to see where the execution deviates from what I expected. I very frequently lean on conditional breakpoints - I know I can skip breaks until a certain condition is met, at which point I need to see exactly what goes wrong.
When I'm unfamiliar with a codebase, or unfamiliar with a particular corner of the code, I find myself reaching for console debugging. Its a bit of a scattershot approach, I don't know what I'm looking for so I console log variables in the vicinity.
Once I know a codebase I want to debug line by line, walking through to see where the execution deviates from what I expected. I very frequently lean on conditional breakpoints - I know I can skip breaks until a certain condition is met, at which point I need to see exactly what goes wrong.
Adding print statements sucks when you are working on native apps and you have to wait for the compiler and linker every time you add one. Debuggers hands down if you are working on something like C++ or Rust. You can add tracepoints in your debugger if you want to do print debugging in native code.
In scripting languages print debugging makes sense especially when debugging a distributed system.
Also logging works better than breaking when debugging multithreading issues imo.
I use both methods.
In scripting languages print debugging makes sense especially when debugging a distributed system.
Also logging works better than breaking when debugging multithreading issues imo.
I use both methods.
How long are you realistically "waiting for the compiler and linker"? 3 seconds? You're not recompiling the whole project after all, just one source file typically
If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes. And then I'll have to hope that I can reproduce the issue without optimizations.
If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes. And then I'll have to hope that I can reproduce the issue without optimizations.
> How long are you realistically "waiting for the compiler and linker"? 3 seconds?
I've worked on projects where incremental linking after touching the wrong .cpp file will take several minutes... and this is after I've optimized link times from e.g. switching from BFD to Gold, whereas before it might take 30 minutes.
A full build (from e.g. touching the wrong header) is measured in hours.
And that's for a single configuration x platform combination, and I'm often the sort to work on the multi-platform abstractions, meaning I have it even worse than that.
> If I wanna use a debugger though, now that means a full recompile to build the project without optimizations
You can use debuggers on optimized builds, and in fact I use debuggers on optimized builds more frequently than I do unoptimized builds. Granted, to make sense of some of the undefined behavior heisenbugs you'll have to understand disassembly, and dig into the lower level details when the high level ones are unavailable or confused by optimization, but those are learnable skills. It's also possible to turn off optimizations on a per-module basis with pragmas, although then you're back into incremental build territory.
I've worked on projects where incremental linking after touching the wrong .cpp file will take several minutes... and this is after I've optimized link times from e.g. switching from BFD to Gold, whereas before it might take 30 minutes.
A full build (from e.g. touching the wrong header) is measured in hours.
And that's for a single configuration x platform combination, and I'm often the sort to work on the multi-platform abstractions, meaning I have it even worse than that.
> If I wanna use a debugger though, now that means a full recompile to build the project without optimizations
You can use debuggers on optimized builds, and in fact I use debuggers on optimized builds more frequently than I do unoptimized builds. Granted, to make sense of some of the undefined behavior heisenbugs you'll have to understand disassembly, and dig into the lower level details when the high level ones are unavailable or confused by optimization, but those are learnable skills. It's also possible to turn off optimizations on a per-module basis with pragmas, although then you're back into incremental build territory.
I understand you are likely doing this for shiny rocks, but life is too short to spend on abominable code bases like this.
Even in my open source income-generating code base (ardour.org) a complete build takes at least 3 minutes on current Apple ARM hardware, and up to 9mins on my 16 core Ryzen. It's quite a nice code base, according to most people who see it.
Sometimes, you just really do need hundreds of thousands of lines of C++ (or equivalent) ...
Sometimes, you just really do need hundreds of thousands of lines of C++ (or equivalent) ...
I work on Scala projects. Adding a log means stopping the service, recompiling and restarting. For projects using Play (the biggest one is one of them) that means waiting for the hot-reload to complete. In both cases it easily takes at least 30s on the smallest projects, with a fast machine. With my previous machine Play's hot reload on our biggest app could take 1 to 3mn.
I use the debugger in Intellij, using breakpoints which only log some context and do not stop the current thread. I can add / remove them without recompiling. There's no interruption in my flow (thus no excuse to check HN because "compiling...")
When I show this to colleagues they think it's really cool. Then they go back to using print statements anyway /shrug
I use the debugger in Intellij, using breakpoints which only log some context and do not stop the current thread. I can add / remove them without recompiling. There's no interruption in my flow (thus no excuse to check HN because "compiling...")
When I show this to colleagues they think it's really cool. Then they go back to using print statements anyway /shrug
This reminds me why I abandoned scala. That being said, even a small sbt project can cold boot in under 10 seconds on a six year old laptop. I shudder to think of the bloat in play if 30 seconds is the norm for a hot reload.
>How long are you realistically "waiting for the compiler and linker"? 3 seconds? You're not recompiling the whole project after all, just one source file typically
10 minutes.
>If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes.
Typically always compile with debug support. You can debug an optimized build as well. Full recompile takes up to 45 minutes.
The largest reason to use a debugger is the time to recompile. Kinda, I actually like rr a lot and would prefer that to print debugging.
10 minutes.
>If I wanna use a debugger though, now that means a full recompile to build the project without optimizations, which probably takes many minutes.
Typically always compile with debug support. You can debug an optimized build as well. Full recompile takes up to 45 minutes.
The largest reason to use a debugger is the time to recompile. Kinda, I actually like rr a lot and would prefer that to print debugging.
10 minutes for linking? The only projects I've touched which have had those kinds of link times have been behemoths like Chromium. That must absolutely suck to work with.
Have you tried out the Mold linker? It might speed it up significantly.
> You can debug an optimized build as well.
Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
Have you tried out the Mold linker? It might speed it up significantly.
> You can debug an optimized build as well.
Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
>10 minutes for linking? The only projects I've touched which have had those kinds of link times have been behemoths like Chromium. That must absolutely suck to work with.
I don't know the exact amounts of time per phase, but you might change a header file and that will of course hurt you a lot more than 1 translation unit.
> Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
Yeah, but sometimes that's life. Reading the assembly and what not to figure things out.
>That must absolutely suck to work with.
Well, you know, I also get to work on something approximately as exciting as Chromium.
I don't know the exact amounts of time per phase, but you might change a header file and that will of course hurt you a lot more than 1 translation unit.
> Eh, not really. Working with a binary where all variables are optimized out and all operators are inlined is hell.
Yeah, but sometimes that's life. Reading the assembly and what not to figure things out.
>That must absolutely suck to work with.
Well, you know, I also get to work on something approximately as exciting as Chromium.
> I don't know the exact amounts of time per phase, but you might change a header file and that will of course hurt you a lot more than 1 translation unit.
Yeah, which is why I was talking about source files. I was surprised that changing 1 source file (meaning re-compiling that one source file and then re-linking the project) takes 10 minutes. If you're changing header files then yeah it's gonna take longer.
Yeah, which is why I was talking about source files. I was surprised that changing 1 source file (meaning re-compiling that one source file and then re-linking the project) takes 10 minutes. If you're changing header files then yeah it's gonna take longer.
FWIW, I've heard from people who know this stuff that linking is actually super slow for us :). I also wanted to try out mold, but I couldn't manage to get it to work.
> How long are you realistically “waiting for the compiler and linker”
Debugging kernel module issues on AL2 on bare metal ec2 with kprint. Issue does not reproduce in qemu. It happens
Debugging kernel module issues on AL2 on bare metal ec2 with kprint. Issue does not reproduce in qemu. It happens
> How long are you realistically "waiting for the compiler and linker"? 3 seconds?
This is the "it's one banana Michael, how much could it cost, ten dollars?" of tech. I don't think I've ever worked on a nontrivial C++ project that compiled in three seconds. I've worked in plenty of embedded environments where simply the download-and-reboot cycle took over a minute. Those are the places where an interactive debugger is most useful .. and also sometimes most difficult.
(at my 2000s era job a full build took over an hour, so we had a machine room of ccache+distcc to bring it down to a single digit number of minutes. Then if you needed to do a full place and route and timing analysis run that took anything up to twelve hours. We're deep into "uphill both ways" territory now though)
This is the "it's one banana Michael, how much could it cost, ten dollars?" of tech. I don't think I've ever worked on a nontrivial C++ project that compiled in three seconds. I've worked in plenty of embedded environments where simply the download-and-reboot cycle took over a minute. Those are the places where an interactive debugger is most useful .. and also sometimes most difficult.
(at my 2000s era job a full build took over an hour, so we had a machine room of ccache+distcc to bring it down to a single digit number of minutes. Then if you needed to do a full place and route and timing analysis run that took anything up to twelve hours. We're deep into "uphill both ways" territory now though)
> I don't think I've ever worked on a nontrivial C++ project that compiled in three seconds.
No C++ project compiles in 3 seconds, but your "change a single source file and compile+link" time is often on the order of a couple of seconds. As an example, I'm working on a project right now where a clean build takes roughly 30 seconds (thanks to recent efforts to improve header include hygiene and move stuff from headers into source files using PIMPL; it was over twice that before). However, when I changed a single source file and ran 'time ninja -C build' just now, the time to compile that one file and re-link the project took just 1.5 seconds.
I know that there are some projects which are much slower to link, I've had to deal with Chromium now and then and that takes minutes just to link. But most projects I've worked with aren't that bad.
No C++ project compiles in 3 seconds, but your "change a single source file and compile+link" time is often on the order of a couple of seconds. As an example, I'm working on a project right now where a clean build takes roughly 30 seconds (thanks to recent efforts to improve header include hygiene and move stuff from headers into source files using PIMPL; it was over twice that before). However, when I changed a single source file and ran 'time ninja -C build' just now, the time to compile that one file and re-link the project took just 1.5 seconds.
I know that there are some projects which are much slower to link, I've had to deal with Chromium now and then and that takes minutes just to link. But most projects I've worked with aren't that bad.
Intoruding logging can actually hide concurrency bugs.
Yeah this is true, it can change the timing. But setting breakpoints or even just running in a debugger or even running a debug build at all without optimizations can also hide concurrency bugs. Literally anything can hide concurrency bugs.
Concurrency bugs just suck to debug sometimes.
Concurrency bugs just suck to debug sometimes.
You also have to remember the context.
First that visual debugging was still small and niche, probably not suited to the environment at Bell Labs at the time, given they were working with simpler hardware that might not provide an acceptable graphical environment (which can be seen as a lot of the UNIX system is oriented around the manipulation of lines of text). This is different from the workplace where most game developers, including J. Carmack were, with access to powerful graphical workstations and development tools.
Secondly there’s also a difference on the kind of work achieved: the work on UNIX systems mostly was about writing tools than big systems, favoring composition of these utilities. And indeed, I often find people working on batch tools not using visual debuggers since the integration of tools pretty much is a problem of data structure visualization (the flow being pretty linear), which is still cumbersome to do in graphical debuggers. The trend often is inverted when working on interactive systems where the main problem actually is understanding the control flow than visualizing data structures: I see a lot more of debuggers used.
Also to keep in mind that a lot of engineers today work on Linux boxes, which has yet to have acceptable graphical debuggers compared to what is offered in Visual Studio or XCode.
First that visual debugging was still small and niche, probably not suited to the environment at Bell Labs at the time, given they were working with simpler hardware that might not provide an acceptable graphical environment (which can be seen as a lot of the UNIX system is oriented around the manipulation of lines of text). This is different from the workplace where most game developers, including J. Carmack were, with access to powerful graphical workstations and development tools.
Secondly there’s also a difference on the kind of work achieved: the work on UNIX systems mostly was about writing tools than big systems, favoring composition of these utilities. And indeed, I often find people working on batch tools not using visual debuggers since the integration of tools pretty much is a problem of data structure visualization (the flow being pretty linear), which is still cumbersome to do in graphical debuggers. The trend often is inverted when working on interactive systems where the main problem actually is understanding the control flow than visualizing data structures: I see a lot more of debuggers used.
Also to keep in mind that a lot of engineers today work on Linux boxes, which has yet to have acceptable graphical debuggers compared to what is offered in Visual Studio or XCode.
Why the emphasis on the use of cartoons (graphical debuggers) for analyzing problems with the text of computer code?
I think graphical debuggers are a big help: 1. It separates the meta-information of the debugger into the graphical domain. 2. It's easier to browse code and set/clear breakpoints using the mouse than the keyboard.
I prefer a debugger first workflow, I'm ideally always running in the debugger except in production, so I'm always ready to inspect a bug that depends on some obscure state corruption.
It seems unlikely that John Carmack doesn't understand his problem domain. Rather it is more likely the problem domain itself, i.e., game dev vs web dev. Game dev is highly stateful and runs in a single process. This class of program can logically be extended to any complex single computer program (or perhaps even a tightly coupled multi-computer program using MPI / related). Web dev effectively runs on a cluster of machines and tends to offload state to 3rd parties (like databases that, on their own look more like game dev) and I/O is loosely coupled / event driven. There is no debugger that can pause all services in web dev such that one can inspect the overall state of the system (and you probably don't want that). So, logging is the best approach to understand what is going on.
In any case, my suggestion is to understand both approaches and boldly use them in the right circumstance. If the need arises, be a rebel and break out the debugger or be a rebel and add some printfs - just don't weakly follow some tribal ritual.
In any case, my suggestion is to understand both approaches and boldly use them in the right circumstance. If the need arises, be a rebel and break out the debugger or be a rebel and add some printfs - just don't weakly follow some tribal ritual.
I posted this elsewhere in the thread, but if you listen to Carmack in the interview, it's quite interesting. He would occasionally use one to step through an entire frame of gameplay to get an idea of performance and see if there were any redundancies. This is what I mean by "doesn't understand the problem domain". He's a smart guy, but no one could immediately understand all the code added in by everyone else on the team and how it all interacts.
Thankfully, we live in an era where entire AAA games can be written almost completely from scratch by one person. Not sarcasm. If I wrote the code myself, I know where almost everything is that could go wrong. It should come as no surprise that I do not use a debugger.
Yeah I coded a AAA game yesterday.
AAA games are not even close to being write-able by one person, what are you talking about. You couldn't even write AAA games from 20 years ago.
Find me a bank that will give me a 150k collateralized loan and after 2 years I will give you the best AAA game you've ever played. You choose all the features. Vulkan/PC only. If you respond back with further features and constraints, I will explain in great detail how to implement them.
I suspect you're trolling, but if not then this is the kind of thing that kickstarter or indiegogo are designed to solve: give me money on my word, in 2 years you get license keys to this thing, assuming it materializes. I was going to also offer peer-to-peer platforms like Prosper but I think they top out at $50k
I agree with you, but I would prefer to not socialize the risks of the project among thousands of individuals, because that lessens their ability to collect against me legally.
By keep just one party to the loan, and most important, by me offering collateral to the loan in the event I do not deliver, then it keeps enforcement more honest and possible.
Furthermore, the loan contract should be written in such a way that the game is judged by the ONE TIME sales performance of the game (no microtransactions) and not qualitative milestones like features or reviews. Lastly, I would add a piece of the contract that says two years after the game is released, it becomes fully open source, similar to the terms of the BSL.
This is the fairest thing to the players, the bank, and the developer, and it lets the focus be absolutely rock solid on shipping something fun ASAP.
By keep just one party to the loan, and most important, by me offering collateral to the loan in the event I do not deliver, then it keeps enforcement more honest and possible.
Furthermore, the loan contract should be written in such a way that the game is judged by the ONE TIME sales performance of the game (no microtransactions) and not qualitative milestones like features or reviews. Lastly, I would add a piece of the contract that says two years after the game is released, it becomes fully open source, similar to the terms of the BSL.
This is the fairest thing to the players, the bank, and the developer, and it lets the focus be absolutely rock solid on shipping something fun ASAP.
Imagine being this delusional, sheesh.
My offer still stands!
So you’re aware it’s delusional? That’s something I suppose
It seems to me you don't quite understand how much needless complexity and functionality is in AAA games these days. Watch how I remove the complexity:
1) We only support Linux. 2) We only support Vulkan (or OpenGL, but I prefer Vulkan) 3) Singleplayer only. 4) Engine features: RGBA Diffuse, specular, shadow, normal/bump, emission maps for materials only. A simple triangle rasterizer with forward rendering. Support for audio, dae animations/rigs/geometry. Physics collisions through an octree and capsule and basic linear algebra. Basic first person/third person cameras. 4) As much content as possible from copyleft/free sources. Mixamo, Cgtrader, poliigon, textures.com etc. A programmer's job is to use whats out there. But, given an extra 6 months, I could also model and texture everything myself as well. I'll do all level design. 5) Dialogue written by me, voice acting through AI or vocaloid. 6) I would estimate the final code to be around 50,000 lines for everything. I can give you a breakdown if you want. All code is in C99, minspec is a ryzen 5600g, which is like $150 max these days.
Something I've missed? I've described a classic first person adventure shooter like Halo for you there. Simply let me know if I should do something else.
1) We only support Linux. 2) We only support Vulkan (or OpenGL, but I prefer Vulkan) 3) Singleplayer only. 4) Engine features: RGBA Diffuse, specular, shadow, normal/bump, emission maps for materials only. A simple triangle rasterizer with forward rendering. Support for audio, dae animations/rigs/geometry. Physics collisions through an octree and capsule and basic linear algebra. Basic first person/third person cameras. 4) As much content as possible from copyleft/free sources. Mixamo, Cgtrader, poliigon, textures.com etc. A programmer's job is to use whats out there. But, given an extra 6 months, I could also model and texture everything myself as well. I'll do all level design. 5) Dialogue written by me, voice acting through AI or vocaloid. 6) I would estimate the final code to be around 50,000 lines for everything. I can give you a breakdown if you want. All code is in C99, minspec is a ryzen 5600g, which is like $150 max these days.
Something I've missed? I've described a classic first person adventure shooter like Halo for you there. Simply let me know if I should do something else.
Depends what you mean by "writable". Triangle-based (software) 3D rendering engines aren't difficult but most people used COTS like Unity or actual 3D hw APIs at least, but it's all the shit that goes around them, the assets, physics, and game logic that sucks.
Back in college, we had to write a constructive geometry object renderer that used GLUT simply as a 2D canvas for a scanline-oriented triangle->trapezoid engine that roughly mirrored the capabilities of OpenGL with Phong and Gouraud shading, texture mapping, and bump mapping. It wasn't hard when each piece was broken down and spoon fed (quaternions and transformation matrices). The hardest part was creating a scene of complex object parts to model the real world programmatically.
Back in college, we had to write a constructive geometry object renderer that used GLUT simply as a 2D canvas for a scanline-oriented triangle->trapezoid engine that roughly mirrored the capabilities of OpenGL with Phong and Gouraud shading, texture mapping, and bump mapping. It wasn't hard when each piece was broken down and spoon fed (quaternions and transformation matrices). The hardest part was creating a scene of complex object parts to model the real world programmatically.
None of the so called "shit" that you mentioned needs to be any more difficult than the 3d things you mentioned. You even seem to say that creating a believable scene is harder than creating say a scalable ragdoll physics engine, but I entirely disagree-- content is incredibly easy to come by, load in, and modify these days entirely for free. The longest amount of time would be spent reimplementing complex systems, say for bone animations or efficient texture atlasing (if performance requirements demand it), rather than trying to find believable content or writing a camera system. And let's please not say anything like OOP or ECS ;)
Waste of time. Flamegraphs do that (as a result of instrumentation and/or profiling), and that is the domain of profiling rather than bug hunting.
Many people seem to have this overly reductive take on performance in which you 1) wait until someone complains and 2) if someone does complain the problem will be readily identified by a hotspot and will be easy to fix. The idea is why spent time optimizing things that no one cares about? Usually there are some ROI and "root of all evil" arguments go along with this and perhaps some other unexamined phrases from 00s agile.
The problem is, sometimes profilers don't identity anything in particular or unwinding things to a point where they can be fixed is actually very har. A more realistic ROI argument should include this as it is a real problem.
I think code should be reasonably fast out of the box. I'm not suggesting vectorizing everything or even taking extreme steps to avoid allocations, etc. Rather, if an algorithm can easily be O(N), don't dumbly implement an O(N^2) or O(N^24) version. If O(1) is possible, do that unless you know the numbers are going to be very small. Don't make 500 DB calls when one will do and finally, don't program by unexamined phrases - particularly those than include "evil" or "harmful" or other such adjectives.
The problem is, sometimes profilers don't identity anything in particular or unwinding things to a point where they can be fixed is actually very har. A more realistic ROI argument should include this as it is a real problem.
I think code should be reasonably fast out of the box. I'm not suggesting vectorizing everything or even taking extreme steps to avoid allocations, etc. Rather, if an algorithm can easily be O(N), don't dumbly implement an O(N^2) or O(N^24) version. If O(1) is possible, do that unless you know the numbers are going to be very small. Don't make 500 DB calls when one will do and finally, don't program by unexamined phrases - particularly those than include "evil" or "harmful" or other such adjectives.
There are many methodologies and tools for profiling but single-stepping assembly stack frames mythology ain't one of them. Identifying hot sections requires gathering data, not wasting time tinkering manually without a plan.
This statement explain his position very clearly. Anyone who did any serious DOS programming understands it well.
"A debugger is how you get a view into a system that's too complicated to understand. I mean, anybody that thinks just read the code and think about it, that's an insane statement, you can't even read all the code on a big system. You have to do experiments on the system. And doing that by adding log statements, recompiling and rerunning it, is an incredibly inefficient way of doing it. I mean, yes, you can always get things done, even if you're working with stone knives and bare skins."
"A debugger is how you get a view into a system that's too complicated to understand. I mean, anybody that thinks just read the code and think about it, that's an insane statement, you can't even read all the code on a big system. You have to do experiments on the system. And doing that by adding log statements, recompiling and rerunning it, is an incredibly inefficient way of doing it. I mean, yes, you can always get things done, even if you're working with stone knives and bare skins."
I still don't understand how, with a properly configured debugger, manually typing print statements is better than clicking a breakpoint at the spot you were going to print. Context overload might be an issue, but just add a 'watch' to the things you care about and focus there.
Two situations immediately come to mind, though the second is admittedly domain specific:
1. If I actually pause execution, the thing I'm trying to debug will time out some network service, at which point trying to step forward is only going to hit sad paths
2. The device I'm debugging doesn't *have* a real debugger. (Common on embedded, really common for video games. Ever triggered a breakpoint in a graphics shader?) Here I might substitute "print" for "display anything at all" but it's the same idea really.
1. If I actually pause execution, the thing I'm trying to debug will time out some network service, at which point trying to step forward is only going to hit sad paths
2. The device I'm debugging doesn't *have* a real debugger. (Common on embedded, really common for video games. Ever triggered a breakpoint in a graphics shader?) Here I might substitute "print" for "display anything at all" but it's the same idea really.
while inspecting some code inside loop, i prefer to put print and see all iterations at once in my screen, instead of countless clicking "continue" in debugger.
> Print debugging is most useful when you understand the code quite well
Every debugger I've ever worked with has logpoints along with breakpoints that allow you to _dynamically_ insert "print statements" into running code without having to recompile or pollute the code with a line of code you're going to have to remember to remove. So I still think debuggers win.
Every debugger I've ever worked with has logpoints along with breakpoints that allow you to _dynamically_ insert "print statements" into running code without having to recompile or pollute the code with a line of code you're going to have to remember to remove. So I still think debuggers win.
The thing about print statement debugging is that it's trivial to replicate in a debugger - just put break points in those few areas where you're curious. It's a tiny bit faster than writing the print statements.
> time to decide where to put print statements
But... that's where you put breakpoints and then you don't need to "single-step" through code. Takes less time to put a breakpoint then to add (and later remove) temporary print statements.
(Now if you're putting in permanent logging that makes sense, do that anyway. But that probably won't coincide with debugging print statements...)
But... that's where you put breakpoints and then you don't need to "single-step" through code. Takes less time to put a breakpoint then to add (and later remove) temporary print statements.
(Now if you're putting in permanent logging that makes sense, do that anyway. But that probably won't coincide with debugging print statements...)
True, but then you're still left stepping through your breakpoints one by one.
Printf debugging gives you the full picture of an entire execution at a glance, allowing you to see time as it happened. The debugger restricts you to step through time and hold the evolution of state in your memory in exchange for giving you free access to explore the state at each point.
Occasionally that arbitrary access is useful, but more often than not it's the evolution of state that you're interested in, and printf gives you that for free.
Printf debugging gives you the full picture of an entire execution at a glance, allowing you to see time as it happened. The debugger restricts you to step through time and hold the evolution of state in your memory in exchange for giving you free access to explore the state at each point.
Occasionally that arbitrary access is useful, but more often than not it's the evolution of state that you're interested in, and printf gives you that for free.
You can use tracepoints instead of breakpoints, or (easier, at least for me), set up breakpoints to execute "print stack frame, continue" when hit - giving you the equivalent of printf debugging, but one you can add/remove without recompiling (or even at runtime) and can give you more information for less typing. And, should this help you spot the problem, you can easily add another breakpoint or convert one of the "printing" ones so it stops instead of continuing.
And of course, should the problem you're debugging be throwing exceptions or crashing the app, the debugger can pause the world at that moment for you, and you get the benefit of debugging and having a "printf log" of the same execution already available.
And of course, should the problem you're debugging be throwing exceptions or crashing the app, the debugger can pause the world at that moment for you, and you get the benefit of debugging and having a "printf log" of the same execution already available.
I set a break point, look at the variables in play and then start looking up the call stack.
But that is still slow compared to print debugging if there is a lot happening. Print debugging you can just print out everything that happens and then scan for the issue and you have a nice timeline of what happens after and before in the print statements.
I don't think you can achieve the same result using debuggers, they just stop you there and you have no context how you got there or what happens after.
Maybe some people just aren't good at print debugging, but usually it finds the issue faster for me, since it helps pinpointing where the issue started by giving you a timeline of events.
Edit: And you can see the result of debugger use in this article, under "Expression Complexity" he rewrote the code to be easier to see in a debugger because he wanted to see the past values. That makes the code worse just to fit a debugger, so it also has such problems. When I use a debugger I do the same, it makes the code harder to read but easier to see in a debugger.
I don't think you can achieve the same result using debuggers, they just stop you there and you have no context how you got there or what happens after.
Maybe some people just aren't good at print debugging, but usually it finds the issue faster for me, since it helps pinpointing where the issue started by giving you a timeline of events.
Edit: And you can see the result of debugger use in this article, under "Expression Complexity" he rewrote the code to be easier to see in a debugger because he wanted to see the past values. That makes the code worse just to fit a debugger, so it also has such problems. When I use a debugger I do the same, it makes the code harder to read but easier to see in a debugger.
Yeah I think it's really addressing different bug issues.
One is finding a needle in a haystack - you have no idea when or where the bug occurred. Presumably your logging / error report didn't spit out anything useful, so you're starting from scratch. That and race conditions. Then print statements can be lovely and get you started.
Most of my debugging is a different case where I know about where in code it happened, but not why it happened, and need to know values / state. A few breakpoints before/during/after my suspected code block, add a few watches, and I get all the information I need quite quickly.
One is finding a needle in a haystack - you have no idea when or where the bug occurred. Presumably your logging / error report didn't spit out anything useful, so you're starting from scratch. That and race conditions. Then print statements can be lovely and get you started.
Most of my debugging is a different case where I know about where in code it happened, but not why it happened, and need to know values / state. A few breakpoints before/during/after my suspected code block, add a few watches, and I get all the information I need quite quickly.
> Takes less time to put a breakpoint then to add (and later remove) temporary print statements
Temporary? Nah, you leave them in as debug or trace log statements. It's an investment. Shipping breakpoints to a teammate for a problem you solved three months ago is a tedious and time-consuming task.
Anyway, breakpoints are themselves quite tedious to interact with iterative problem solving. At least if you're familiar with the codebase.
Temporary? Nah, you leave them in as debug or trace log statements. It's an investment. Shipping breakpoints to a teammate for a problem you solved three months ago is a tedious and time-consuming task.
Anyway, breakpoints are themselves quite tedious to interact with iterative problem solving. At least if you're familiar with the codebase.
There's this idea that the way you use a debugger is by stepping over line after line during execution.
That's not usually the case.
Setting conditional breakpoints, especially for things like break on all exceptions, or when the video buffer has a certain pattern, etc, is usually where the value starts to come in.
That's not usually the case.
Setting conditional breakpoints, especially for things like break on all exceptions, or when the video buffer has a certain pattern, etc, is usually where the value starts to come in.
I wonder if Brian Kerninghan was using modern tooling or that comment was using quote from 70’s.
It's from The Practice of Programming, published 1999. Not a lot has changed in debuggers since then from what I can see.
Debugging in Visual Studio by Microsoft has changed a lot in last 5 years, JetBrains IDE debugging a lot as well.
I can debug .NET application and change code live, change variable states if needed. Watch variables and all kinds of helpers like stack navigation was immensely improved since I started 15 years ago.
I can say debugging Java/.NET applications is totally different experience than using debugger from 1999.
Multi threaded apps debugging and all kinds of helpers i visual debuggers.
I just fail to see why someone would waste time putting in debug statements when they can configure debug session with conditional break points.
I can debug .NET application and change code live, change variable states if needed. Watch variables and all kinds of helpers like stack navigation was immensely improved since I started 15 years ago.
I can say debugging Java/.NET applications is totally different experience than using debugger from 1999.
Multi threaded apps debugging and all kinds of helpers i visual debuggers.
I just fail to see why someone would waste time putting in debug statements when they can configure debug session with conditional break points.
But can't you instead just set a breakpoint next to wherever you are gonna put that print stmt and inspect value once code hits? print stmt seems like extra overhead
That doesn’t necessarily give you a clean log to review
While also avoiding having to re-run cases to get new introspection when you forgot to add a print statement.
I tend to do both, print statements when I don't feel I want to be stepping through some cumbersome interplay of calls but diving into the debugger to step through the nitty gritty, even better when I can evaluate code in the debugger to understand the state of the data at precise points.
I don't think there's a better or worse version of doing it, you use the tool that is best for what introspection you need.
I tend to do both, print statements when I don't feel I want to be stepping through some cumbersome interplay of calls but diving into the debugger to step through the nitty gritty, even better when I can evaluate code in the debugger to understand the state of the data at precise points.
I don't think there's a better or worse version of doing it, you use the tool that is best for what introspection you need.
Debuggers allow you inspect stuff forward in time, while print statements allow you to debug backwards. (There was a lot of academic work on reversible debuggers at one point; to be honest I haven’t kept up on how that turned out.)
If you can detect a problematic condition and you want to know what will happen next, a debugger is a great tool.
If you can detect a problematic condition and you need to find out what caused it, it’s printf all the way.
My theory is that different types of programming encounter these two types of problems at different relative rates, and that this explains why many people strongly prefer one over the other but don’t agree on which.
If you can detect a problematic condition and you want to know what will happen next, a debugger is a great tool.
If you can detect a problematic condition and you need to find out what caused it, it’s printf all the way.
My theory is that different types of programming encounter these two types of problems at different relative rates, and that this explains why many people strongly prefer one over the other but don’t agree on which.
The tools are not mutually exclusive. I also do quite a lot with print debugging, but some of the most pernicious problems often require a debugger.
> It takes less time to decide where to put print statements than to single-step to the critical section of code
Why would you ever be single-stepping? Put a break point (conditional if necessary) where you would put the print statement. The difference between a single break point and a print statement is that the break point will allow you to inspect the local variables associated with all calls in the stack trace and evaluate further expressions.
So when do you debug instead of using print statements? When you know that no matter what the outcome of your hypothesis is, that you will need to iteratively inspect details from other points up the stack. That is, when you know, from experience, that you are going to need further print statements but you don't know where they will be.
> It takes less time to decide where to put print statements than to single-step to the critical section of code
Why would you ever be single-stepping? Put a break point (conditional if necessary) where you would put the print statement. The difference between a single break point and a print statement is that the break point will allow you to inspect the local variables associated with all calls in the stack trace and evaluate further expressions.
So when do you debug instead of using print statements? When you know that no matter what the outcome of your hypothesis is, that you will need to iteratively inspect details from other points up the stack. That is, when you know, from experience, that you are going to need further print statements but you don't know where they will be.
This depends on a lot of things.
For example, one thing you wrote that jumps out at me:
> I already have a working model for the way that the code runs [...]
This is not always true. It's only true for code that I wrote or know very well. E.g. as a consultant, I often work on codebases that are new to me, and I do tend to use debuggers there more often than I use print debugging.
Although lots of other variables affect this - how much complicated state there is to get things running, how fast the "system" starts up, what language it's written in and if there are alternatives (in some situations I'll use a Jupyter Notebook for exploring the code, Clojure has its own repl-based way of doing things, etc).
For example, one thing you wrote that jumps out at me:
> I already have a working model for the way that the code runs [...]
This is not always true. It's only true for code that I wrote or know very well. E.g. as a consultant, I often work on codebases that are new to me, and I do tend to use debuggers there more often than I use print debugging.
Although lots of other variables affect this - how much complicated state there is to get things running, how fast the "system" starts up, what language it's written in and if there are alternatives (in some situations I'll use a Jupyter Notebook for exploring the code, Clojure has its own repl-based way of doing things, etc).
There's a time and place for everything, and an escalation of tooling/environmental context for me.
1. Printing vars in unit tests may be the fastest first approach. If i know where the bug may be.
2. When that fails i usually bring in debuggers to unit tests.
3. When these aren't helping, you need debuggers on the entire binary.
4. Still stuck? Use a debugger in production.
1. Printing vars in unit tests may be the fastest first approach. If i know where the bug may be.
2. When that fails i usually bring in debuggers to unit tests.
3. When these aren't helping, you need debuggers on the entire binary.
4. Still stuck? Use a debugger in production.
Isn't 4 very unsafe? I wouldn't trust my code to pause in places that it doesn't usually pause in.
3. What if the binary interacts with other networked computers, you gonna debug all of them? Do you end up instrumeting the whole internet? You scope out, you spiral out of control until someone puts a limit.
3. What if the binary interacts with other networked computers, you gonna debug all of them? Do you end up instrumeting the whole internet? You scope out, you spiral out of control until someone puts a limit.
You probably just don't know how to use conditional breakpoints effectively. This is faster than adding print statements.
I feel like you need to know when to use what. Debugger is so much faster and easier for me when looking for errors in the use of (my own) abstractions.
But when looking for errors in the bowels of abstracted or very small partitioned code print logs are far easier to see the devil in the detail.
I personally use both, and I'm not sure I find the argument about needing to step through convincing. I put the debugger breakpoint at the same place I might put a print. I hardly ever step through, but I do often continue to reach this line again. The real advantage is that you can inspect the current state live and make calls with the data.
However, I use prints a lot more because you can, as you say, usually get to the answer faster.
However, I use prints a lot more because you can, as you say, usually get to the answer faster.
I disagree - using an interactive debugger can give insights that just looking at the code can't (tbf it might be different for different people). But the number of times I have found pathological behaviour from just stepping through the code is many. Think "holy f**, this bit of code is running 100 times??" type stuff. With complex event-driven code written by many teams, it's not obvious what is happening at runtime by just perusing the code and stroking one's long wizard beard.
> I disagree - using an interactive debugger can give insights that just looking at the code can't
This in no way disagrees with the quote. Both can be true. The quote isn’t saying debuggers can’t provide unique insights, just that for the majority of debugging the print statement is a faster way to get what you need.
This in no way disagrees with the quote. Both can be true. The quote isn’t saying debuggers can’t provide unique insights, just that for the majority of debugging the print statement is a faster way to get what you need.
That is the difference between complex state and simple state.
I use a debugger when I've constructed a complex process that has a large cardinally of states it could end up in. There is no possibility that I can write logic checks (tests) for all source inputs to that state.
I don't use one when I could simply increase test situations to find my logical error.
Consider the difference between a game engine and a simple state machine. The former can be complex enough to replicate many features of the real world while a simple state machine/lexer probably just needs more tests of each individual state to spot the issue.
I use a debugger when I've constructed a complex process that has a large cardinally of states it could end up in. There is no possibility that I can write logic checks (tests) for all source inputs to that state.
I don't use one when I could simply increase test situations to find my logical error.
Consider the difference between a game engine and a simple state machine. The former can be complex enough to replicate many features of the real world while a simple state machine/lexer probably just needs more tests of each individual state to spot the issue.
Their comment conflates debugging with logging.
Professional debuggers such as the one in IntelliJ IDEA are invaluable regardless of one's familiarity with a given codebase, to say otherwise is utter ignorance. Outside of logging, unless attaching a debugger is impractical, using print statements is at best wasting time.
Professional debuggers such as the one in IntelliJ IDEA are invaluable regardless of one's familiarity with a given codebase, to say otherwise is utter ignorance. Outside of logging, unless attaching a debugger is impractical, using print statements is at best wasting time.
Perhaps consider that your experience is not universal and that others have good reasons for their decisions that are not simply ignorance.
I didn't say there aren't acceptable reasons to reach for the print statement, there are. But for the vast majority of codebases out there, if good debugger tooling is available, it's a crime not to use it as a primary diagnostics tool. Claiming otherwise is indeed ignorant if not irresponsible.
Debugging is for the developer; logging is for all, and especially those who need to support the code without the skills/setup/bandwidth to drop into a debugger. Once you start paying attention to what/where/how of logging, you (and others) can spot things faster then you can step through the debugger. Plus logs provide history and and searchable.
This feels a little like "I don't use a cordless drill because my screw driver works so well and is faster in most cases" grug brain says use best tool, not just tool grug used last.
> substantially faster
Than what? In languages with good debugger support (see JVM/Java) it can be far quicker to click a single line to set a breakpoint, hit Debug, the inspect the values or evaluate expressions to get the runtime context you cant get from purely reading code. Print statements require rebuilding code and backing them out, so its hard to imagine that technique being faster.
I do use print debugging for languages with poor IDE/debugger support, but it is one big thing I miss when outside of Java.
Than what? In languages with good debugger support (see JVM/Java) it can be far quicker to click a single line to set a breakpoint, hit Debug, the inspect the values or evaluate expressions to get the runtime context you cant get from purely reading code. Print statements require rebuilding code and backing them out, so its hard to imagine that technique being faster.
I do use print debugging for languages with poor IDE/debugger support, but it is one big thing I miss when outside of Java.
One thing this quote doesn't touch is that speed of fixing the bug isn't the only variable. The learning along the way is at least as important, if not more so. Reading and understanding the code serves the developer better long term if they are regularly working on it. On the other hand debuggers really shine when jumping into a project to help out and don't have or need a good understanding of the code base.
I would tend to say that printf debugging is widespread in the Linux-adjacent world because you can't trust a visual debugger to actually be working there because of the general brokenness of GUIs in the Linux world.
I didn't really get into debuggers until (1) I was firmly in Windows, where you expect the GUI to work and the LI to be busted, and (2) I'd been burned too many times by adding debugging printfs() that got checked into version control and caused trouble.
Since then I've had some adventures with CLI debuggers, such as using gdb to debug another gdb, using both jdb and gdb on the same process at the same time to debug a Java/C++ system, automating gdb, etc. But there is the thing, as you say, is that there is usually some investment required to get the debugger working for a particular system.
With a good IDE I think Junit + debugging gives an experience in Java similar to using the REPL in a language like Python in that you can write some code that is experimental and experiment it, but in this case the code doesn't just scroll out of the terminal but ultimately gets checked in as a unit test.
I didn't really get into debuggers until (1) I was firmly in Windows, where you expect the GUI to work and the LI to be busted, and (2) I'd been burned too many times by adding debugging printfs() that got checked into version control and caused trouble.
Since then I've had some adventures with CLI debuggers, such as using gdb to debug another gdb, using both jdb and gdb on the same process at the same time to debug a Java/C++ system, automating gdb, etc. But there is the thing, as you say, is that there is usually some investment required to get the debugger working for a particular system.
With a good IDE I think Junit + debugging gives an experience in Java similar to using the REPL in a language like Python in that you can write some code that is experimental and experiment it, but in this case the code doesn't just scroll out of the terminal but ultimately gets checked in as a unit test.
Debuggers exist in the terminal, in vim, and in emacs.
Why would you want a GUI debugger?
Can see all your source code while you're debugging. And it's not like emacs where your termcap is 99.99% right which means it is 0.01% wrong. (Mac-ers get made when something is 1px out of place, in Linux culture they'll close a bug report if the window is 15000 px to the left of the screen and invisible because it's just some little fit and finish thing)
Honestly, I consider myself pretty comfortable with the terminal and vim and whatnot, but I've never been able to get into using GDB. For me I feel like a debugger is one of those things that's just so much better as a proper GUI.
Huh, this must really be a personal taste things, because I only want the debug info I specifically request to be printed on the next line. But I can imagine wanting a different interface to it.
I can't do it either. Something about typing commands that have some kind of grammar and pressing Return to submit each one for consideration just throws me off. Don't make me think. Thinking is typically what got the code into this mess in the first place - whether too much of it or too little, it doesn't really matter. Time to try some other approach.
Having the code, the callstack, locals, your watched variables and expressions, the threads, memory, breakpoints and machine code and registers if needed available at a glance? As well as being able to dig deeper into data structures just by clicking on them. Why wouldn't you want that? A good GUI debugger is a dashboard showing the state of your program in a manner that is impossible to replicate in a CLI or a TUI interface.
You get all of that in the terminal debugger. That’s why dwarf files exist.
All the information is there, but the presentation isn't. You have to keep querying it while you're debugging. Sure, there are TUI debuggers that are more like GUI debuggers. Except that they are worse at everything compared to a GUI debugger.
I don’t know what debugger you’ve used but the entire query command is `f v` in lldb for the current stack frame
Yes, but in a GUI debugger the stack and everything else is available all the time and you don't have to enter commands to see what the state is. It even highlights changes as you step. It's just so plainly superior to any terminal solution.
I don’t disagree that a visual debugger made with a proper GUI toolkit is better than a TUI. However, nvim-dap-ui[0] does a pretty good job.
[0] https://github.com/rcarriga/nvim-dap-ui
[0] https://github.com/rcarriga/nvim-dap-ui
[deleted]
> and I’m usually the only one on my team using a debugger. Almost everyone in the real world (at least in web tech) seems to do print statement debugging.
One of the first things I do in a codebase is get some working IDE/editor up where I can quickly run the program under a debugger, even if I'm not immediately troubleshooting something. It's never long before I need to use it.
I was baffled when I too encountered this. Even working collaboratively with people they'd have no concept of how to use a debugger.
"No, set a breakpoint there"
"yeah now step into the function and inspect the state of those variables"
"step over that"
: blank stares at each instance :
One of the first things I do in a codebase is get some working IDE/editor up where I can quickly run the program under a debugger, even if I'm not immediately troubleshooting something. It's never long before I need to use it.
I was baffled when I too encountered this. Even working collaboratively with people they'd have no concept of how to use a debugger.
"No, set a breakpoint there"
"yeah now step into the function and inspect the state of those variables"
"step over that"
: blank stares at each instance :
For desktop GUI development I can't imagine not using breakpoints and stepping. Especially when you have several ways that some callback might be triggered. It's super helpful to break on a UI element signal (or slot) and then follow along to see why things aren't working.
I don't use debuggers as often in Python, probably because it's eaiser to throw code in a notebook and run line by line to inspect variables, change/inject state and re-run. That's possible but a lot harder to do in C++.
Also for embedded work, using a debugger and memory viewer is pretty powerful. It's not something people think about for Arduino but almost every commodity micro supports some sort of debugwire-like interface (which is usually simpler than JTAG).
I don't use debuggers as often in Python, probably because it's eaiser to throw code in a notebook and run line by line to inspect variables, change/inject state and re-run. That's possible but a lot harder to do in C++.
Also for embedded work, using a debugger and memory viewer is pretty powerful. It's not something people think about for Arduino but almost every commodity micro supports some sort of debugwire-like interface (which is usually simpler than JTAG).
Personally, I think that debugger is very helpful in understanding what is going on, but once you are familiar with the code and data structures, I am very often pretty close in my assessment, so scanning code and inserting multiple print lines is both faster and more productive.
I only used debugger recently in C# and C, when I was learning and practicing them.
I only used debugger recently in C# and C, when I was learning and practicing them.
A lot of people think that. That's why it's important to read the essay.
I am also in the camp that has very little use for debuggers.
A point that may be pedantic: I don't add (and then remove) "print" statements. I add logging code, that stays forever. For a major interface, I'll usually start with INFO level debugging, to document function entry/exit, with param values. I add more detailed logging as I start to use the system and find out what needs extra scrutiny. This approach is very easy to get started with and maintain, and provides powerful insight into problems as they arise.
I also put a lot of work into formatting log statements. I once worked on a distributed system, and getting the prefix of each log statement exactly right was very useful -- node id, pid, timestamp, all of it fixed width. I could download logs from across the cluster, sort, and have a single file that interleaved actions from across the cluster.
A point that may be pedantic: I don't add (and then remove) "print" statements. I add logging code, that stays forever. For a major interface, I'll usually start with INFO level debugging, to document function entry/exit, with param values. I add more detailed logging as I start to use the system and find out what needs extra scrutiny. This approach is very easy to get started with and maintain, and provides powerful insight into problems as they arise.
I also put a lot of work into formatting log statements. I once worked on a distributed system, and getting the prefix of each log statement exactly right was very useful -- node id, pid, timestamp, all of it fixed width. I could download logs from across the cluster, sort, and have a single file that interleaved actions from across the cluster.
A log is very different than a debugger though, one tells you what happened, one shows you the entire state and doesn't make you assemble it in your head.
Your framing makes it sound like the log is worse in some way, but what the log gives you that the debugger makes you assemble in your head is a timeline of when things happen. Being able to see time is a pretty big benefit for most types of software.
I can always drop an entire state object into the log if I need it, but the only way for a debugger to approximate what a log can give me is for me to step through a bunch of break points and hold the time stream in my head.
The one place where a debugger is straight up better is if I know exactly which unit of code is failing and that unit has complicated logic that is worth stepping through line by line. That's what they were designed for, and they're very useful for that, but it's also not the most common kind of troubleshooting I run into.
I can always drop an entire state object into the log if I need it, but the only way for a debugger to approximate what a log can give me is for me to step through a bunch of break points and hold the time stream in my head.
The one place where a debugger is straight up better is if I know exactly which unit of code is failing and that unit has complicated logic that is worth stepping through line by line. That's what they were designed for, and they're very useful for that, but it's also not the most common kind of troubleshooting I run into.
It's not worse or better, but its not really comparable is all I am really saying, I would not use them for the same things.
In the early 2000’s I whipped up a tool to convert log statements into visual swim lanes like the Chrome profiler does. That thing was a godsend for reasoning about complex parallelism.
Would you be willing to share this tool?
Oh that's long lost to the trashbin of history. As I recall it was a combination of:
- Log4j + MDC's so we could get a timestamp and thread ID easily
- some python scripting
I can't remember if it was PDF or SVG, but I do remember having to choose something that would allow for huge graphs and lots of scrolling!
These days there's better options on github, but nice to see I was on a decent path.
- Log4j + MDC's so we could get a timestamp and thread ID easily
- some python scripting
I can't remember if it was PDF or SVG, but I do remember having to choose something that would allow for huge graphs and lots of scrolling!
These days there's better options on github, but nice to see I was on a decent path.
All these print debugging advocates are blowing my mind. Are most people unaware that both lldb and
gdb have conditional pass throughout breakpoints with function hooks? In other words, you can create a breakpoint that just prints its location and doesn’t pause execution.
You can script this so all function entry/exists, or whatever, are logged without touching the code or needing to recompile. You can then bulk toggle these breakpoints, at runtime, so you only see a particular subset when things get interesting.
Modifying the code to print stuff will feel barbaric after driving around a fine tuned debugging experience.
You can script this so all function entry/exists, or whatever, are logged without touching the code or needing to recompile. You can then bulk toggle these breakpoints, at runtime, so you only see a particular subset when things get interesting.
Modifying the code to print stuff will feel barbaric after driving around a fine tuned debugging experience.
I can't tell you how many times lldb has failed to hit breakpoints or has dumped me in some library without symbols. This was in Xcode while writing an iOS app, maybe it's better in other environments.
Print debugging, while not clever or powerful, has never once failed me.
Print debugging, while not clever or powerful, has never once failed me.
Sometimes the debugserver is flakey, I’ll give you that. But some that also sounds like UI quirks such as ambiguous breakpoints on a function definition with default initialized default values.
You can attach lldb without Xcode. Or you can open the lldb terminal in Xcode, pause execution, and inspect the breakpoints manually
You can attach lldb without Xcode. Or you can open the lldb terminal in Xcode, pause execution, and inspect the breakpoints manually
I've never had a debugger show me the entire state. I'm not even sure I want to know the entire state, but GDB has a lot of issue with anything but the most basic data structures most of the time, and I always need to explicitly ask for what I want to see by calling things like operator[] (and then hope that the particular operator[] wasn't optimized out of the final binary). It's not exactly a great experience.
What I find annoying is how these async toolkits screw up the stack trace, so I have little what the real program flow looks like. That reduces much of the benefit off the top.
Some IDEs promise to solve that, but I’ve not been impressed thus far.
YMMV based on language/runtime/toolkit of course. This might get added to my wishlist for my next language of choice.
Some IDEs promise to solve that, but I’ve not been impressed thus far.
YMMV based on language/runtime/toolkit of course. This might get added to my wishlist for my next language of choice.
> A point that may be pedantic: I don't add (and then remove) "print" statements. I add logging code, that stays forever. For a major interface, I'll usually start with INFO level debugging, to document function entry/exit, with param values.
This is an anti-pattern which results in voluminous log "noise" when the system operates as expected. To the degree that I have personally seen gigabytes per day produced by employing it. It also can litter the solution with transient concerns once thought important and are no longer relevant.
If detailed method invocation history is a requirement, consider using the Writer Monad[0] and only emitting log entries when either an error is detected or in an "unconditionally emit trace logs" environment (such as local unit/integration tests).
0 - https://williamyaoh.com/posts/2020-07-26-deriving-writer-mon...
This is an anti-pattern which results in voluminous log "noise" when the system operates as expected. To the degree that I have personally seen gigabytes per day produced by employing it. It also can litter the solution with transient concerns once thought important and are no longer relevant.
If detailed method invocation history is a requirement, consider using the Writer Monad[0] and only emitting log entries when either an error is detected or in an "unconditionally emit trace logs" environment (such as local unit/integration tests).
0 - https://williamyaoh.com/posts/2020-07-26-deriving-writer-mon...
It's absolutely not an anti-pattern if you have appropriate tools to handle different levels of logging, and especially not if you can filter debug output by area. You touch on this, but it's a bit strange to me that the default case is assumed to be "all logs all the time".
I usually roll my own wrapper around an existing logging package, but https://www.npmjs.com/package/debug is a good example of what life can be like if you're using JS. Want to debug your rate limiter? Write `DEBUG=app:middleware:rate-limiter npm start` and off you go.
I usually roll my own wrapper around an existing logging package, but https://www.npmjs.com/package/debug is a good example of what life can be like if you're using JS. Want to debug your rate limiter? Write `DEBUG=app:middleware:rate-limiter npm start` and off you go.
> It's absolutely not an anti-pattern if you have appropriate tools to handle different levels of logging, and especially not if you can filter debug output by area.
It is an anti-pattern due to what was originally espoused:
Also, you did not address the non-trivial cost implications of voluminous log output.
> You touch on this, but it's a bit strange to me that the default case is assumed to be "all logs all the time".
Regarding the above, production-ready logging libraries such as Logback[0], log4net[1], log4cpp[2], et al, allow for run-time configuration to determine what "areas" will have their entries emitted. So "all logs all the time" is a non sequitur in this context.
What is relevant is the technique identified of emitting execution context when it matters and not when it doesn't. As to your `npm` example, I believe this falls under the scenario I explicitly identified thusly:
1 - https://logging.apache.org/log4net/index.html
2 - https://log4cpp.sourceforge.net/
It is an anti-pattern due to what was originally espoused:
I add logging code, that stays forever. For a major
interface, I'll usually start with INFO level debugging, to
document function entry/exit, with param values.
Also, you did not address the non-trivial cost implications of voluminous log output.
> You touch on this, but it's a bit strange to me that the default case is assumed to be "all logs all the time".
Regarding the above, production-ready logging libraries such as Logback[0], log4net[1], log4cpp[2], et al, allow for run-time configuration to determine what "areas" will have their entries emitted. So "all logs all the time" is a non sequitur in this context.
What is relevant is the technique identified of emitting execution context when it matters and not when it doesn't. As to your `npm` example, I believe this falls under the scenario I explicitly identified thusly:
... an "unconditionally emit trace logs" environment
(such as local unit/integration tests).
1 - https://logging.apache.org/log4net/index.html
2 - https://log4cpp.sourceforge.net/
I understand that you explained some exceptions to the rule, but I disagree with two things: the assumption of incompetence on the part of geophile to not make logging conditional in some way, and adding the label of "anti-pattern" to something that's evidently got so much nuance to it.
> the non-trivial cost implications of voluminous log output
If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
> the non-trivial cost implications of voluminous log output
If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
> ... I disagree with two things: the assumption of incompetence on the part of geophile to not make logging conditional in some way ...
I assumed nothing of the sort. What I did was identify an anti-pattern and describe an alternative which experience has shown to be a better approach.
"Incompetence" is your word, not mine.
> ... and adding the label of "anti-pattern" to something that's evidently got so much nuance to it.
I fail to see the nuance you apparently can see.
>> the non-trivial cost implications of voluminous log output
> If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
Cloud deployment requires transmission of log entries to one or more log aggregators in order to be known.
By definition, this involves network I/O.
Network communication is orders of magnitude slower than local I/O.
Useless logging of "function entry/exit, with param values" increases pressure on network I/O.
Unless logging is allowed to be lossy, which it never is, transmission must be completed when log buffers are near full capacity.
Provisioning production systems having excessive logging can often require more resources than those which do not excessively log.
Thus disproving:
> ... even at runtime the costs are often trivial.
When considering the implication of voluminous log output in a production environment.
I assumed nothing of the sort. What I did was identify an anti-pattern and describe an alternative which experience has shown to be a better approach.
"Incompetence" is your word, not mine.
> ... and adding the label of "anti-pattern" to something that's evidently got so much nuance to it.
I fail to see the nuance you apparently can see.
>> the non-trivial cost implications of voluminous log output
> If log output is conditional at compile time there are no non-trivial cost implications, and even at runtime the costs are often trivial.
Cloud deployment requires transmission of log entries to one or more log aggregators in order to be known.
By definition, this involves network I/O.
Network communication is orders of magnitude slower than local I/O.
Useless logging of "function entry/exit, with param values" increases pressure on network I/O.
Unless logging is allowed to be lossy, which it never is, transmission must be completed when log buffers are near full capacity.
Provisioning production systems having excessive logging can often require more resources than those which do not excessively log.
Thus disproving:
> ... even at runtime the costs are often trivial.
When considering the implication of voluminous log output in a production environment.
As I said, conditional. As in, you add logging to your code but you either remove it at compile time or you check your config at run time. By definition, work you don't do is not done.
Conditionals aren't free either, and conditionals - especially compile-time - on logging code are considered by some a bugprone anti-pattern as well.
The code that computes data for and assembles your log message may end up executing logic that affects the system elsewhere. If you put that code under conditional, your program will behave differently depending on the logging configuration; if you put it outside, you end up wasting potentially substantial amount of work building log messages that never get used.
The code that computes data for and assembles your log message may end up executing logic that affects the system elsewhere. If you put that code under conditional, your program will behave differently depending on the logging configuration; if you put it outside, you end up wasting potentially substantial amount of work building log messages that never get used.
This is getting a bit far into the weeds, but I've found that debug output which is disabled by default in all environments is quite safe. I agree that it would be a problem to leave it turned on in development, testing, or staging environments.
You are very attached to this "voluminous" point. What do you mean by it?
As I said, responding to another comment of yours, a distributed system I worked on produced a few GB a day. The logs were rotated daily. They were never transmitted anywhere, during normal operation. When things go wrong, sure, we look at them, and generate even more logging. But that was rare. I cannot stress enough how much of a non-issue log volume was in practice.
So I ask you to quantify: What counts (to you) as voluminous, as in daily log file sizes, and how many times they are sent over the network?
As I said, responding to another comment of yours, a distributed system I worked on produced a few GB a day. The logs were rotated daily. They were never transmitted anywhere, during normal operation. When things go wrong, sure, we look at them, and generate even more logging. But that was rare. I cannot stress enough how much of a non-issue log volume was in practice.
So I ask you to quantify: What counts (to you) as voluminous, as in daily log file sizes, and how many times they are sent over the network?
> You are very attached to this "voluminous" point. What do you mean by it?
I mean "a lot" or more specifically; "a whole lot."
Here is an exercise which illustrates this. For the purposes here, assume ASCII characters are used for log entries to make the math a bit easier.
Suppose the following:
The quantification is above and regarding log entries being sent over a network - in many production systems, log entries are unconditionally sent to a log aggregator and never stored in a local file system.
I mean "a lot" or more specifically; "a whole lot."
Here is an exercise which illustrates this. For the purposes here, assume ASCII characters are used for log entries to make the math a bit easier.
Suppose the following:
Each log statement is 100 characters.
Each service invocation emits 50 log statements.
Average transactions per second during high usage is 200tps.
High usage is on average 2 hours per day.
100 x 50 x 200 x 60 x 60 x 2 = 7_200_000_000 = 7.2GB / day
The quantification is above and regarding log entries being sent over a network - in many production systems, log entries are unconditionally sent to a log aggregator and never stored in a local file system.
7.2G/day doesn't sound terrible. And I'd reduce it by a factor of 25, since in normal operation (i.e., not looking into a problem) I would have either 2 log statements per call (entry, exit), or none at all. It might be more than 2, if I needed detailed logging.
But in normal usage, even in the scenario you describe your argument about log volume is not convincing.
But in normal usage, even in the scenario you describe your argument about log volume is not convincing.
BTW, your username is a bit too on-the-nose, given the way you are arguing, using "anti-pattern" as a way to end all discussion.
> BTW, your username is a bit too on-the-nose ...
Resorting to an ad hominem and non sequiturs are we? Really?
C'mon, you are better than that.
Resorting to an ad hominem and non sequiturs are we? Really?
C'mon, you are better than that.
The whole concept of an “anti-pattern” is a discussion ender. It’s basically a signal that one party isn’t willing to consider the specific advantages and disadvantages of a particular approach in a given context.
> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.
Well, I agree completely, but those conditions are a tall order. The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended. If I have a failure, and suspect a major subsystem, I sure do want to see all calls and param values leading up to a failure.
In addition to this point, you have constructed a strawman in which logging is on all the time. Have you ever looked at syslog? On my desktop Linux system, output there counts as voluminous. It isn't so much space, or so CPU-intensive that I would consider disabling syslog output (even if I could).
The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue. And for the rare times that something did fail, we could turn up logging with precision and get useful information.
Well, I agree completely, but those conditions are a tall order. The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended. If I have a failure, and suspect a major subsystem, I sure do want to see all calls and param values leading up to a failure.
In addition to this point, you have constructed a strawman in which logging is on all the time. Have you ever looked at syslog? On my desktop Linux system, output there counts as voluminous. It isn't so much space, or so CPU-intensive that I would consider disabling syslog output (even if I could).
The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue. And for the rare times that something did fail, we could turn up logging with precision and get useful information.
>> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.
> Well, I agree completely, but those conditions are a tall order.
Every successful service invocation satisfies "all collaborations succeed and the system operates as intended." Another way to state this is every HTTP `1xx`, `2xx`, and `3xx` response code produced by an HTTP service qualifies as such.
> The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended.
Providing sufficient context in the presence of errors, or "situations in which things don't succeed or operate as intended", was addressed thusly:
See above.
> In addition to this point, you have constructed a strawman in which logging is on all the time.
No, I addressed your original premise in the context of a production web application, where logging is configured during deployment.
See also your own contradiction by previously asserting, "I sure do want to see all calls and param values leading up to a failure."
So which is it?
Did I construct a strawman "in which logging is on all the time"?
Or do you "want to see all calls and param values leading up to a failure", which requires "logging is on all the time"?
> Have you ever looked at syslog?
This is a strawman. Syslog is a component for logging and has nothing to do with the programs which use it.
> The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue.
If this is the same system you described in a different comment also in this thread, I identified a standard industry practice of log entries produced by application nodes in a production environment being unconditionally sent to a log aggregator and not stored in a local file system. The reasons for this are well documented.
> Well, I agree completely, but those conditions are a tall order.
Every successful service invocation satisfies "all collaborations succeed and the system operates as intended." Another way to state this is every HTTP `1xx`, `2xx`, and `3xx` response code produced by an HTTP service qualifies as such.
> The whole point of debugging (by whatever means you prefer) is for those situations in which things don't succeed or operate as intended.
Providing sufficient context in the presence of errors, or "situations in which things don't succeed or operate as intended", was addressed thusly:
If detailed method invocation history is a requirement,
consider using the Writer Monad and only emitting log
entries when either an error is detected or in an
"unconditionally emit trace logs" environment (such as
local unit/integration tests).
See above.
> In addition to this point, you have constructed a strawman in which logging is on all the time.
No, I addressed your original premise in the context of a production web application, where logging is configured during deployment.
See also your own contradiction by previously asserting, "I sure do want to see all calls and param values leading up to a failure."
So which is it?
Did I construct a strawman "in which logging is on all the time"?
Or do you "want to see all calls and param values leading up to a failure", which requires "logging is on all the time"?
> Have you ever looked at syslog?
This is a strawman. Syslog is a component for logging and has nothing to do with the programs which use it.
> The large distributed system I worked on would produce a few GB per day, and the logs were rotated. A complete non-issue.
If this is the same system you described in a different comment also in this thread, I identified a standard industry practice of log entries produced by application nodes in a production environment being unconditionally sent to a log aggregator and not stored in a local file system. The reasons for this are well documented.
I'm not sure what point you are making with your scenario involving HTTP response codes. What if the HTTP server crashes, and doesn't send a response at all?
I don't know from Writer Monads. But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory? What about log volume? Does this writer monad implement log rotation? It sounds like you are sweeping a lot of the things you object to under this writer monad rug.
Let me be real clear about all calls and param values leading up to a failure.
- In normal operation, turn logging off completely, or turn on some level that produces tolerable log volume, (it seems like your threshold is much lower than mine).
- When a failure occurs: Restart the service with more logging enabled, (hence the all calls an param values), so that you have logging when the failure occurs again.
About local logs vs a log aggregator: The system I worked on was a shared nothing archive. To add storage and metadata capacity, you add nodes. Each node also stored its own log files. I get that this may not be the answer all the time, and that a log aggregator is useful in some scenarios. However, even in that case, your concerns about log volume seem overblown to me.
I don't know from Writer Monads. But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory? What about log volume? Does this writer monad implement log rotation? It sounds like you are sweeping a lot of the things you object to under this writer monad rug.
Let me be real clear about all calls and param values leading up to a failure.
- In normal operation, turn logging off completely, or turn on some level that produces tolerable log volume, (it seems like your threshold is much lower than mine).
- When a failure occurs: Restart the service with more logging enabled, (hence the all calls an param values), so that you have logging when the failure occurs again.
About local logs vs a log aggregator: The system I worked on was a shared nothing archive. To add storage and metadata capacity, you add nodes. Each node also stored its own log files. I get that this may not be the answer all the time, and that a log aggregator is useful in some scenarios. However, even in that case, your concerns about log volume seem overblown to me.
> I'm not sure what point you are making with your scenario involving HTTP response codes.
My point was to identify how common the "happy path" scenario is and was in direct response to:
>> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.
> Well, I agree completely, but those conditions are a tall order.
Regarding your question:
> What if the HTTP server crashes, and doesn't send a response at all?
Again, the HTTP status codes were used to illustrate the frequency of successful invocations. But to your point, if an HTTP server crashes then log entries for in-flight workflows would likely not be emitted. A similar possibility also holds for local file system logging as buffering exists (sometimes on multiple levels).
> I don't know from Writer Monads.
No worries. All it is is a formal definition of a type having specific capabilities called "laws" in the functional programming world.
> But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory?
What is meant by "future failure" is a failure potentially encountered during the evaluation of a single invocation. In the context of a HTTP server, this would be the handling of each submission to a specific HTTP endpoint and verb. This is often defined within an IO Monad[0], but does not have to be, and is out of scope for this discussion.
The answer to the rest of your questions is the deferred log entry definitions are held in memory for the duration of the single service invocation, with any log emissions produced transmitted to a network-accesible log aggregator via the logging component used.
> Let me be real clear about all calls and param values leading up to a failure.
The scenario you kindly shared is understandable, yet is one which has been unacceptable in teams I have worked with. Losing the "first error" is not an option in those environments.
0 - https://en.wikibooks.org/wiki/Haskell/Understanding_monads/I...
My point was to identify how common the "happy path" scenario is and was in direct response to:
>> There is no value for logging "function entry/exit, with param values" when all collaborations succeed and the system operates as intended.
> Well, I agree completely, but those conditions are a tall order.
Regarding your question:
> What if the HTTP server crashes, and doesn't send a response at all?
Again, the HTTP status codes were used to illustrate the frequency of successful invocations. But to your point, if an HTTP server crashes then log entries for in-flight workflows would likely not be emitted. A similar possibility also holds for local file system logging as buffering exists (sometimes on multiple levels).
> I don't know from Writer Monads.
No worries. All it is is a formal definition of a type having specific capabilities called "laws" in the functional programming world.
> But if you only emitting log entries on some future failure or request, then that's potentially a lot of logging to keep somewhere. Where? Log aggregator? Local files? Memory?
What is meant by "future failure" is a failure potentially encountered during the evaluation of a single invocation. In the context of a HTTP server, this would be the handling of each submission to a specific HTTP endpoint and verb. This is often defined within an IO Monad[0], but does not have to be, and is out of scope for this discussion.
The answer to the rest of your questions is the deferred log entry definitions are held in memory for the duration of the single service invocation, with any log emissions produced transmitted to a network-accesible log aggregator via the logging component used.
> Let me be real clear about all calls and param values leading up to a failure.
The scenario you kindly shared is understandable, yet is one which has been unacceptable in teams I have worked with. Losing the "first error" is not an option in those environments.
0 - https://en.wikibooks.org/wiki/Haskell/Understanding_monads/I...
I know a lot of people do that in all kinds of software (especially enterprise), still, I can't help but notice this is getting close to Greenspunning[0] territory.
What you describe is leaving around hand-rolled instrumentation code that conditionally executes expensive reporting actions, which you can toggle on demand between executions. Thing is, this is already all done automatically for you[1] - all you need is the right build flag to prevent optimizing away information about function boundaries, and then you can easily add and remove such instrumentation code on the fly with a debugger.
I mean, tracing function entry and exit with params is pretty much the main task of a debugger. In some way, it's silly that we end up duplicating this by hand in our own projects. But it goes beyond that; a lot of logging and tracing I see is basically hand-rolling an ad hoc, informally-specified, bug-ridden, slow implementation of 5% of GDB.
Why not accept you need instrumentation in production too, and run everything in a lightweight, non-interactive debugging session? It's literally the same thing, just done better, and couple layers of abstraction below your own code, so it's more efficient too.
--
[0] - https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
[1] - Well, at least in most languages used on the backend, it is. I'm not sure how debugging works in Node at the JS level, if it exists at all.
What you describe is leaving around hand-rolled instrumentation code that conditionally executes expensive reporting actions, which you can toggle on demand between executions. Thing is, this is already all done automatically for you[1] - all you need is the right build flag to prevent optimizing away information about function boundaries, and then you can easily add and remove such instrumentation code on the fly with a debugger.
I mean, tracing function entry and exit with params is pretty much the main task of a debugger. In some way, it's silly that we end up duplicating this by hand in our own projects. But it goes beyond that; a lot of logging and tracing I see is basically hand-rolling an ad hoc, informally-specified, bug-ridden, slow implementation of 5% of GDB.
Why not accept you need instrumentation in production too, and run everything in a lightweight, non-interactive debugging session? It's literally the same thing, just done better, and couple layers of abstraction below your own code, so it's more efficient too.
--
[0] - https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule
[1] - Well, at least in most languages used on the backend, it is. I'm not sure how debugging works in Node at the JS level, if it exists at all.
I agree that logging all functions is reinventing the wheel.
I think there's still value in adding toggleable debug output to major interfaces. It tells you exactly what and where the important events are happening, so that you don't need to work out where to stick your breakpoints.
I think there's still value in adding toggleable debug output to major interfaces. It tells you exactly what and where the important events are happening, so that you don't need to work out where to stick your breakpoints.
A logging library is very, very far from a Turing complete language, so no Greenspunning. (Yes, I know about that Java logger fiasco from a few years ago. Not my idea.)
I don't want logging done automatically for me, what I want is too idiosyncratic. While I will log every call on major interfaces, I do want to control exactly what is printed. Maybe some parameter values are not of interest. Maybe I want special formatting. Maybe I want the same log line to include something computed inside the function. Also, most of my logging is not on entry/exit. It's deeper down, to look at very specific things.
Look, I do not want a debugger, except for tiny programs, or debugging unit tests. In a system with lots of processes, running on lots of nodes, if a debugger is even possible to use, it is just too much of a PITA, and provides far too miniscule a view of things. I don't want to deal with running to just before the failure, repeatedly, resetting the environment on each attempt, blah, blah, blah. It's a ridiculous way to debug a large and complex system.
What a debugger can do, that is harder with logging, is to explore arbitrary code. If I chase a problem into a part of my system that doesn't have logging, okay, I add some logging, and keep it there. That's a good investment in the future. (This new logging is probably at a detailed level like DEBUG, and therefore only used on demand. Obvious, but it seems like a necessary thing to point out in this conversation.)
I don't want logging done automatically for me, what I want is too idiosyncratic. While I will log every call on major interfaces, I do want to control exactly what is printed. Maybe some parameter values are not of interest. Maybe I want special formatting. Maybe I want the same log line to include something computed inside the function. Also, most of my logging is not on entry/exit. It's deeper down, to look at very specific things.
Look, I do not want a debugger, except for tiny programs, or debugging unit tests. In a system with lots of processes, running on lots of nodes, if a debugger is even possible to use, it is just too much of a PITA, and provides far too miniscule a view of things. I don't want to deal with running to just before the failure, repeatedly, resetting the environment on each attempt, blah, blah, blah. It's a ridiculous way to debug a large and complex system.
What a debugger can do, that is harder with logging, is to explore arbitrary code. If I chase a problem into a part of my system that doesn't have logging, okay, I add some logging, and keep it there. That's a good investment in the future. (This new logging is probably at a detailed level like DEBUG, and therefore only used on demand. Obvious, but it seems like a necessary thing to point out in this conversation.)
FWIW, it seems like poor man's tracing. You'd get that and a lot more having opentelemetry setup (using Jaeger for UI locally)
I don't quite like littering the code with logs, but I understand there's a value to it.
The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.
This is useful when you encounter unknown problem and need to find unusual stuff in the logs. This unusual stuff is not always an error state, it might be some aggregate problem (something is called too many times, something is happening in problematic order, etc.)
The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.
This is useful when you encounter unknown problem and need to find unusual stuff in the logs. This unusual stuff is not always an error state, it might be some aggregate problem (something is called too many times, something is happening in problematic order, etc.)
I don't know what's "important" at the beginning. In my work, logging grows as I work on the system. More logging in more complex or fragile parts. Sometimes I remove logging where it provides no value.
> The problem is that if you only log problems or "important" things, then you have a selection bias in the log and don't have a reference of how the log looks like when the system operates normally.
A case can be made for only logging the steps performed up to and including an error. This even excludes logging "important things" other than those satisfying system/functional requirements (such as request/response audit logs).
It is reminiscent of "the Unix philosophy", but different in important ways, and is essentially:
While using a functional programming language and employing referentially transparent[0] abstractions (such as the Writer Monad) usually makes implementing this pattern much simpler than when using an imperative language, it can be successfully done with the latter given sufficient discipline and the workflow implementation being referentially transparent[0].
An important complement to the above is to employ other industry standard verification activities, such as unit/feature/integration tests.
0 - https://en.wikipedia.org/wiki/Referential_transparency
A case can be made for only logging the steps performed up to and including an error. This even excludes logging "important things" other than those satisfying system/functional requirements (such as request/response audit logs).
It is reminiscent of "the Unix philosophy", but different in important ways, and is essentially:
Capture what would be log entries if a there is a
future unrecoverable error.
If an error is encountered, emit *all* log entries starting
from the earliest point (such as receiving an external
event or a REST endpoint request) up to and including the
information detailing the unrecoverable error.
If the workflow succeeds, including producing an expected
failed workflow response (such as a validation error),
discard the deferred log entries.
While using a functional programming language and employing referentially transparent[0] abstractions (such as the Writer Monad) usually makes implementing this pattern much simpler than when using an imperative language, it can be successfully done with the latter given sufficient discipline and the workflow implementation being referentially transparent[0].
An important complement to the above is to employ other industry standard verification activities, such as unit/feature/integration tests.
0 - https://en.wikipedia.org/wiki/Referential_transparency
This sounds like TRACE level information to me.
I find that debuggers solve a very specific class of bugs of intense branching complexity in a self contained system. But the moment there's stuff going in and out of DBs, other services, multithreading, integrations, etc, the debugger becomes more of a liability without a really advanced tooling team.
I think of "don't break the debugger" as a top important design heuristic.
It's one point in favor of async, and language that make async debugging easy.
And at the very least, you can encapsulate things so they can be debugged separately from the multi threading stuff.
If you're always using the debugger, you learn to build around it as one of your primary tools for interacting with code, and you notice right away if the debugger can't help with something.
It's one point in favor of async, and language that make async debugging easy.
And at the very least, you can encapsulate things so they can be debugged separately from the multi threading stuff.
If you're always using the debugger, you learn to build around it as one of your primary tools for interacting with code, and you notice right away if the debugger can't help with something.
I've had more sr devs than I tell me they don't see a benefit to using a debugger, because they have a type system.
Wtaf?
Wtaf?
I don't use debuggers in general development but use them a lot when writing and running automated tests, much faster and easier to see stuff than with print statements.
It has gotten to the point where when somebody wants to add a DSL to our architecture one of my first questions is "where is your specification for integrating it to the existing debuggers?"
If there isn't one, I'd rather use a language with a debugger and write a thousand lines of code than 100 lines of code in a language I'm going to have to black box.
If there isn't one, I'd rather use a language with a debugger and write a thousand lines of code than 100 lines of code in a language I'm going to have to black box.
I am young grug who didn't use debuggers much until last year or so.
What sold me on debugger is the things you can do with it
What sold me on debugger is the things you can do with it
* See values and eval expressions in calling frames.
* Modify the course of execution by eval'ing a mutating expression.
* Set exception breakpoints which stop deep where the exception is raised.
The rise of virtualization, containers, microservices, etc has I think contributed to this being more difficult. Even local dev-test loops often have something other than you launching the executable, which can make it challenging to get the debugger attached to it.
Not any excuse, but another factor to be considered when adding infra layers between the developer and the application.
Not any excuse, but another factor to be considered when adding infra layers between the developer and the application.
Debuggers are also brittle when working with asynchronous code
Debuggers actually can hide entire categories of bugs caused by race conditions when breakpoints cause async functions to resolve in a different order than they would when running in realtime
Debuggers actually can hide entire categories of bugs caused by race conditions when breakpoints cause async functions to resolve in a different order than they would when running in realtime
Another underutilized debugging superpower is debug-level logging.
I've never worked somewhere where logging is taken seriously. Like, our AWS systems produce logs and they get collected somewhere, but none of our code ever does any serious logging.
If people like print-statement debugging so much, then double down on it and do it right, with a proper logging framework and putting quality debug statements into all code.
I've never worked somewhere where logging is taken seriously. Like, our AWS systems produce logs and they get collected somewhere, but none of our code ever does any serious logging.
If people like print-statement debugging so much, then double down on it and do it right, with a proper logging framework and putting quality debug statements into all code.
If you want to double-down on logging and do it right: make your logs fit for computer consumption and the source of truth.
That's all event-sourcing is.
That's all event-sourcing is.
Are you more productive with a debugger? For all bugs? How so?
Debuggers are great until you have to work in a context where you can't attach a debugger. Good old printf works in every context.
By all means, learn to use a debugger well but don't become overly dependent on them.
By all means, learn to use a debugger well but don't become overly dependent on them.
[deleted]
I'll introduce you to our bespoke tool that automatically restarts processes when they exit... and redirects stdout/err by default to /dev/null :D
I use a repl instead of a debugger. White box vs black box.
They can be used in conjunction. You can drop into debugger from REPL and vice versa.
I’m a debugger guy too, but print statements can be very powerful. I keep this bookmarked https://tenderlovemaking.com/2016/02/05/i-am-a-puts-debugger...
My hypothesis is that because we generally don't/can't use debuggers in production but rather rely on logging and tracing, that extends to local dev.
Agreed!
If you usually aren't able/allowed to use a debugger in production and must rely on observability tools, it's helpful to know how to utilize those tools locally as effectively as possible when debugging.
If you usually aren't able/allowed to use a debugger in production and must rely on observability tools, it's helpful to know how to utilize those tools locally as effectively as possible when debugging.
> Breaking on an interesting line of code during a test run and studying the call stack that got me there is infinitely easier than trying to run the code forwards in my head.
I really don't get this at all. To me it is infinitely easier to iterate and narrow the problem rather than trying to identify sight-unseen where the problem is—it's incredibly rare that the bug immediately crashes the program. And you can fit a far higher density of relevant information through print statements over execution of a reproduced bug than you can reproduce at any single point in the call stack. And 99% of the information you can access at any single point in the call stack will be irrelevant.
To be sure, a debugger is an incredibly useful and irreplaceable tool.... but it's far too slow and buggy to rely on for daily debugging (unless, as you indicate, you don't know the codebase well enough to reason about it by reading the code).
Things that complicate this:
* highly mutable state
* extremely complex control or data flow
* not being able to access logs
* the compiler lying to you or being incorrect
* subtle instruction ordering issues
I really don't get this at all. To me it is infinitely easier to iterate and narrow the problem rather than trying to identify sight-unseen where the problem is—it's incredibly rare that the bug immediately crashes the program. And you can fit a far higher density of relevant information through print statements over execution of a reproduced bug than you can reproduce at any single point in the call stack. And 99% of the information you can access at any single point in the call stack will be irrelevant.
To be sure, a debugger is an incredibly useful and irreplaceable tool.... but it's far too slow and buggy to rely on for daily debugging (unless, as you indicate, you don't know the codebase well enough to reason about it by reading the code).
Things that complicate this:
* highly mutable state
* extremely complex control or data flow
* not being able to access logs
* the compiler lying to you or being incorrect
* subtle instruction ordering issues
debugging is useful when your codebase is bad: imperative style, mutable state, weak type system, spaghetti code, state and control variables intermixed.
i'd rather never use debugger, so that my coding style is enforced to be clean code, strong type system, immutable variables, explicit error control, explicit control flow etc
i'd rather never use debugger, so that my coding style is enforced to be clean code, strong type system, immutable variables, explicit error control, explicit control flow etc
You've gotten downvoted but I think you're correct - if there were no debuggers, the developers would be forced to write (better) code that didn't need them.
Thank you for defending the common sense opinion. Some people here are too reactionary
Having worked in many languages and debuggers across many kinds of backend and front end systems, I think what some folks miss here is that some debuggers are great and fast, and some suck and are extremely slow. For example, using LLDB with Swift is hot garbage. It lies to you and frequently takes 30 seconds or more to evaluate statements or show you local variable values. But e.g. JavaScript debuggers tend to be fantastic and very fast. In addition, some kinds of systems are very easy to exercise in a debugger, and some are very difficult. Some bugs resist debugging, and must be printf’d.
In short, which is better? It depends, and varies wildly by domain.
In short, which is better? It depends, and varies wildly by domain.
What's insane is that debuggers could support at least 80-90% of print debugging workflow with a good UI.
The only thing that's really missing is history of the values of the watches you added, all in one log. With filters because why not.
For some reason I never seen this option in any debugger I tried.
The only thing that's really missing is history of the values of the watches you added, all in one log. With filters because why not.
For some reason I never seen this option in any debugger I tried.
UPDATE:
Heh, who knew, debuggers already have this feature under the name tracepoints (VS) or logpoints (Viscose).
https://code.visualstudio.com/blogs/2018/07/12/introducing-l...
It should be way more well known among print debug crowd.
Heh, who knew, debuggers already have this feature under the name tracepoints (VS) or logpoints (Viscose).
https://code.visualstudio.com/blogs/2018/07/12/introducing-l...
It should be way more well known among print debug crowd.
>Young grugs: learning this skill is a minor superpower. Take the time to get it working on your codebase, if you can.
TIL Linus Torvald is a young grug.
TIL Linus Torvald is a young grug.
I used to use debugger when I was young - disk space was small, disks were slow and logging was expensive.
Now, some 35 years later, I greatly prefer logs. This way I can compare execution paths of different use cases, I can compare outcomes of my changes, etc. I am not confined to a single point of time with tricky manual control as with debugger, I can see them all.
To old grugs: learning to execute and rewind code in your head is a major superpower. And it works on any codebase.
Now, some 35 years later, I greatly prefer logs. This way I can compare execution paths of different use cases, I can compare outcomes of my changes, etc. I am not confined to a single point of time with tricky manual control as with debugger, I can see them all.
To old grugs: learning to execute and rewind code in your head is a major superpower. And it works on any codebase.
I thought debugging was table stakes. It isn't always the answer. If a lot is going on logs can be excellent (plus grep or an observability tool)
However debugging is an essential tool in the arsenal. If something is behaving oddly even the best REPL can't match debugging as a dev loop (maybe lisp excepted).
I even miss the ability to move the current execution point back in .NET now I use Go and JS. That is a killer feature. Edit and continue even more so!
Then next level is debugging unit tests. Saved me hours.
However debugging is an essential tool in the arsenal. If something is behaving oddly even the best REPL can't match debugging as a dev loop (maybe lisp excepted).
I even miss the ability to move the current execution point back in .NET now I use Go and JS. That is a killer feature. Edit and continue even more so!
Then next level is debugging unit tests. Saved me hours.
It's been years since I last used a debugger, but then, it's been years since I last worked on code that was complicated enough to warrant it.
Which is a good thing! Easily comprehended code that you can reason about without stepping through it is good for grug brain.
Which is a good thing! Easily comprehended code that you can reason about without stepping through it is good for grug brain.
There might be another factor at not using the debugger beyond the pure cluelessness: often you can’t really run it in production. Back when I started with coding (it was Turbo Pascal 3.0, so you get the idea :-), I enjoyed the use of the debugger quite a lot.
But in 2000 I started working in a role which required understanding the misbehavior of embedded systems that were forwarding the live traffic, there was a technical possibility to do “target remote …” but almost never an option to stop a box that is forwarding the traffic.
So you end up being dependent on debugs - and very occasional debug images with enhanced artisanal diagnostics code (the most fun was using gcc’s -finstrument-function to catch a memory corruption of an IPSec field by an unrelated IKE code in a use-after free scenario)
Where the GDB shined though is the analysis of the crash dumps.
Implementing a “fake” gdb stub in Perl, which was sucking in the crash dump data and allow to leisurely explore it with debugger rather than decoding hex by hand, was a huge productivity boon.
So I would say - it’s better to have more than one tool in the toolbox and use the most appropriate one.
But in 2000 I started working in a role which required understanding the misbehavior of embedded systems that were forwarding the live traffic, there was a technical possibility to do “target remote …” but almost never an option to stop a box that is forwarding the traffic.
So you end up being dependent on debugs - and very occasional debug images with enhanced artisanal diagnostics code (the most fun was using gcc’s -finstrument-function to catch a memory corruption of an IPSec field by an unrelated IKE code in a use-after free scenario)
Where the GDB shined though is the analysis of the crash dumps.
Implementing a “fake” gdb stub in Perl, which was sucking in the crash dump data and allow to leisurely explore it with debugger rather than decoding hex by hand, was a huge productivity boon.
So I would say - it’s better to have more than one tool in the toolbox and use the most appropriate one.
Wholeheartedly agree. There’s often good performance or security reasons why it’s hard to get a debugger running in prod, but it’s still worth figuring out how to do it IMO.
Your experience sounds more sophisticated than mine, but the one time I was able to get even basic debugger support into a production Ruby app, it made fixing certain classes of bug absolutely trivial compared to what it would have been.
The main challenge was getting this considered as a requirement up front rather than after the fact.
Your experience sounds more sophisticated than mine, but the one time I was able to get even basic debugger support into a production Ruby app, it made fixing certain classes of bug absolutely trivial compared to what it would have been.
The main challenge was getting this considered as a requirement up front rather than after the fact.
Indeed. Let the machine do the work!
[deleted]
I work with some pretty niche tech where it's usually, ironically, easier to use a debugger than to add print statements. Unfortunately the debugger is pretty primitive, it can't really show the call stack for example. But even just stopping at a line of code and printing variables or poking around in memory is pretty powerful.
> Almost everyone in the real world (at least in web tech) seems to do print statement debugging. I have tried and failed to get others interested in using my workflow.
Sigh. Same. To a large extent, this is caused by debuggers just sucking for async/await code. And just sucking in general for webdev.
Sigh. Same. To a large extent, this is caused by debuggers just sucking for async/await code. And just sucking in general for webdev.
I try all the time, but I always end up having to wrestle a trillion calls into some library code that has 0 relevance to me, and if the issue is happening at some undetermined point in the chain, you basically have to step through it all to get an idea for where things are going wrong.
On the other hand, the humble console.log() just works without requiring insanely tedious and frustrating debugger steps.
On the other hand, the humble console.log() just works without requiring insanely tedious and frustrating debugger steps.
I mean, "step over" and "go to line" exist...
My current position has implemented a toolchain that essentially makes debugging either impossible or extremely unwieldy for any backend projects and nobody seems to think it's a problem.
Running a debugger on test failure is a ridiculously effective workflow. Instead of getting a wall of text, you drop right into the call stack where the failure/error happened. `pytest --pdb` in python, worth its weight in shiny rocks for sure!
I find the differences between printf debugging and line debuggers (or whatever you call them) unimportant in most circumstances.
Line debuggers usually have some nice conveniences, but the major bottlenecks are between the ears, not in the tool.
Line debuggers usually have some nice conveniences, but the major bottlenecks are between the ears, not in the tool.
I usually use a normal debugger to find a problem when I can see its symptoms but not the original caus. That way I can break on the line that is causing the symptom, check what the variables are like and go back up the call stack to find the origin of the incorrect state. I can do all that in one shot (maybe a couple if I need to break somewhere else instead) rather than putting prints everywhere to try and work out what the call stack is, and a load of prints to list off all the local variables
Sometimes I need a debugger because there's a ton of variables or I just have no idea what's wrong and that's the easiest way to see everything that's going on. It's really frustrating to feel like I need a debugger and don't have a good way to add the IDEs visual debugger (because I'm using a CLI on a remote session or something). It's also really frustrating to be inside a debugging session and wish you knew what the previous value for something was but you can't because you can't go backwards in time. That happens so often to me, in fact, that print debugging is actually more effective for me in the vast majority of cases.
I love debuggers, but unfortunately at my current job I've found that certain things we do to make our application more performant (mainly using giant structs full of fixed size arrays allocated at the start of the application) cause LLDB to slow to a crawl when `this` points to them. It really really doesn't like trying to read the state of a nearly 1GB struct...
This is one of those reasons why you really really need to get other people on board with your workflows. If you're the only one who works like that and someone does something insane, but it technically works, but it blows your workflow up... that's your problem. "You should just develop how I'm developing. Putting in a print statement for every line then waiting 5 minutes for the application to compile."
So long as no one sees your workflow as valuable, they will happily destroy it if it means getting the ticket done.
So long as no one sees your workflow as valuable, they will happily destroy it if it means getting the ticket done.
Some people are wizards with the debugger. Some people prefer printfs.
I used to debug occasionally but haven't touched a debugger in years. I'm not sure exactly why this is, but I'm generally not concerned with exactly what's in the stack on a particular run, but more with some specific state and where something changes time after time, and it's easier to collect that data programatically in the program than to figure out how to do it in the debugger that's integrated with whatever IDE I'm using that day.
And the codebase I deal with at work is Spring Boot. 90% of the stack is meaningless to me anyway. The debugger has been handy for finding out "what the f*$% is the caller doing to the transaction context" but that kind of thing is frustrating no matter what.
Anyway, I think they're both valid ways to explore the runtime characteristics of your program. Depends on what you're comfortable with, how you think of the problem, and the type of codebase you're working on. I could see living in the debugger being more useful to me if I'm in some C++ codebase where every line potentially has implicit destructor calls or something.
I used to debug occasionally but haven't touched a debugger in years. I'm not sure exactly why this is, but I'm generally not concerned with exactly what's in the stack on a particular run, but more with some specific state and where something changes time after time, and it's easier to collect that data programatically in the program than to figure out how to do it in the debugger that's integrated with whatever IDE I'm using that day.
And the codebase I deal with at work is Spring Boot. 90% of the stack is meaningless to me anyway. The debugger has been handy for finding out "what the f*$% is the caller doing to the transaction context" but that kind of thing is frustrating no matter what.
Anyway, I think they're both valid ways to explore the runtime characteristics of your program. Depends on what you're comfortable with, how you think of the problem, and the type of codebase you're working on. I could see living in the debugger being more useful to me if I'm in some C++ codebase where every line potentially has implicit destructor calls or something.
I want to master JS/React and Python debugging. Am an "advanced beginner" in both. What tools do you recommend?
For JavaScript, you're actually able to debug fairly easily by default by adding a `debugger()` call in your code. Browsers will stop at that call, and start the debugger.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
Another way (and probably a better idea) is creating a launch definition for VS Code in launch.json which attaches the IDE's debugger to Chrome. Here is an article describing how that works: https://profy.dev/article/debug-react-vscode
Breakpoints are nice because they can't accidentally get shipped to production like debugger calls can.
For Python I essentially do the same thing, minus involving Chrome. I run the entry point to my application from launch.json, and breakpoints in the IDE 'just work'. From there, you can experiment with how the debugger tools work, observe how state change as the application runs, etc.
If you don't use VS Code, these conventions are similar in other IDEs as well.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
Another way (and probably a better idea) is creating a launch definition for VS Code in launch.json which attaches the IDE's debugger to Chrome. Here is an article describing how that works: https://profy.dev/article/debug-react-vscode
Breakpoints are nice because they can't accidentally get shipped to production like debugger calls can.
For Python I essentially do the same thing, minus involving Chrome. I run the entry point to my application from launch.json, and breakpoints in the IDE 'just work'. From there, you can experiment with how the debugger tools work, observe how state change as the application runs, etc.
If you don't use VS Code, these conventions are similar in other IDEs as well.
I'm looking around and I don't see anyone mentioning what I think is one of the biggest advantages of debugging: not having to recompile.
Even assuming print statements and debuggers are equally effective (they're not), debuggers are better simply because they are faster. With print statements, you might need to recompile a dozen times before you find whatever it is you're looking for. Even with quick builds this is infuriating and a ton of wasted time.
Even assuming print statements and debuggers are equally effective (they're not), debuggers are better simply because they are faster. With print statements, you might need to recompile a dozen times before you find whatever it is you're looking for. Even with quick builds this is infuriating and a ton of wasted time.
Here is my circular argument against debuggers: if I learn to use a debugger, I will spend much, possibly most, of my time debugging. I'd rather learn how to write useful programs that don't have bugs. Most people believe this is impossible.
The trouble of course is that there is always money to be made debugging. There is almost no incentive in industry to truly eliminate bugs and, indeed, I would argue that the incentives in the industry actively encourage bugs because they lead to lucrative support contracts and large dev teams that spend half their time chasing down bugs in a never-ending cycle. If a company actually shipped perfect software, how could they keep extracting more money from their customer?
The trouble of course is that there is always money to be made debugging. There is almost no incentive in industry to truly eliminate bugs and, indeed, I would argue that the incentives in the industry actively encourage bugs because they lead to lucrative support contracts and large dev teams that spend half their time chasing down bugs in a never-ending cycle. If a company actually shipped perfect software, how could they keep extracting more money from their customer?
[deleted]
So you think that not knowing how to operate a debugger will exempt your code from having bugs, or you from having to debug them?
I loved Chrome's debugger for years, then build tools and React ruined debugging for me.
Built code largely works with source maps, but it fails often enough, and in bizarre ways, that my workflow has simply gone back to console logs.
React's frequent re-renders have also made breakpoints very unpleasant - I'd rather just look at the results of console logs.
Are there ways I can learn to continue enjoying the debugger with TS+React? It is still occasionally useful and I'm glad its there, but I have reverted to defaulting to console logs.
Built code largely works with source maps, but it fails often enough, and in bizarre ways, that my workflow has simply gone back to console logs.
React's frequent re-renders have also made breakpoints very unpleasant - I'd rather just look at the results of console logs.
Are there ways I can learn to continue enjoying the debugger with TS+React? It is still occasionally useful and I'm glad its there, but I have reverted to defaulting to console logs.
Console logs in the browser have some unique advantages. You can assign the output to a variable, play with it etc.
But yes, any code that is inside jsx generally sucks to debug with standard tooling. There are browser plugins that help you inspect the react ui tree though.
But yes, any code that is inside jsx generally sucks to debug with standard tooling. There are browser plugins that help you inspect the react ui tree though.
Agreed, debugging tools for the browser are almost comically incapable, to the point of not even useful in most cases.
I find myself doing a mix of both. Source maps are good enough most of the time, I haven't seen the bizarre failures you're seeing - maybe your bundling configuration needs some tweaking? But yes, the frequent re-renders are obnoxious. In those cases logging is generally better.
Conditional breakpoints help alleviate the pain when there are frequent re-renders. Generally you can pinpoint a specific value that you're looking for and only pause when that condition is satisfied. Setting watch expressions helps a lot too.
Conditional breakpoints help alleviate the pain when there are frequent re-renders. Generally you can pinpoint a specific value that you're looking for and only pause when that condition is satisfied. Setting watch expressions helps a lot too.
Nobody has time for that. Need time for prompting AI!
I refuse to believe there are professional software developers who don't use debuggers. What are you all working on? How do you get your bearings in a new code-base? Do you read it like a book, and keep the whole thing in your mind? How do you specify all of the relevant state in your print statements? How do you verify your logic?
Debuggers are great. I too have tried and failed to spread their use.
But the reason is simple: they are always 100x harder to set up and keep working than just type a print statement and be done with it.
Especially when you're in a typical situation where an app has a big convoluted docker based setup, and with umpteen packers & compilers & transpilers etc.
This is why all software companies should have at least one person tasked with making sure there are working debuggers everywhere, and that everyone's been trained how to use them.
There should also be some kind of automated testing that catches any failures in debugging tooling.
But that effort's hard to link directly to shipped features, so if you started doing that, management would home in on that dev like a laser guided missile and task him to something with "business value", thereby wasting far more dev time and losing far more business value.
But the reason is simple: they are always 100x harder to set up and keep working than just type a print statement and be done with it.
Especially when you're in a typical situation where an app has a big convoluted docker based setup, and with umpteen packers & compilers & transpilers etc.
This is why all software companies should have at least one person tasked with making sure there are working debuggers everywhere, and that everyone's been trained how to use them.
There should also be some kind of automated testing that catches any failures in debugging tooling.
But that effort's hard to link directly to shipped features, so if you started doing that, management would home in on that dev like a laser guided missile and task him to something with "business value", thereby wasting far more dev time and losing far more business value.
So many gems in here but this one about microservices is my favorite:
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
I keep trying to explain this to tiny dev teams (1-2 people) that will cheerfully take a trivial web app with maybe five forms and split it up into “microservices” that share a database, an API Management layer, a queue for batch jobs to process “huge” volumes (megabytes) of data, an email notification system, an observablity platform (bespoke!) and then… and then… turn the trivial web forms into a SPA app because “that’s easier”.
Now I understand that “architecture” and “patterns” is a jobs program for useless developers. It’s this, or they’d be on the streets holding a sign saying “will write JavaScript for a sandwich”.
Now I understand that “architecture” and “patterns” is a jobs program for useless developers. It’s this, or they’d be on the streets holding a sign saying “will write JavaScript for a sandwich”.
> Now I understand that “architecture” and “patterns” is a jobs program for useless developers.
Yet, developers are always using patterns and are thinking about architecture.
Here you are doing so too, a pattern, "form submission" and an architecture, "request-response".
Yet, developers are always using patterns and are thinking about architecture.
Here you are doing so too, a pattern, "form submission" and an architecture, "request-response".
A word and its meaning are usually antonyms.
>I keep trying to explain this to tiny dev teams
I'm curious what role you have where you're doing this repeatedly
I'm curious what role you have where you're doing this repeatedly
The customer is a government department formed by the merger of a bunch of only vaguely related agencies. They have “inherited” dozens of developers from these mergers, maybe over a hundred if you count the random foreign outsourcers. As you can imagine there’s no consistency or organisational structure because it wasn’t built up as a cohesive team from the beginning.
The agencies are similarly uncoordinated and will pick up their metaphorical credit card and just throw it at random small dev teams, internally, external, or a mix.
Those people will happily take the credit! The money just… disappears. It’s like a magic trick, or one of those street urchins that rips you off when you’re on holiday in some backwards part of the world like Paris.
I get brought in as “the cloud consultant” for a week or two at the end to deploy the latest ball of mud with live wires sticking out of it to production.
This invariably becomes an argument because the ball of mud the street urchins have sold to the customer is not fit for… anything… certainly not for handling PII or money, but they spent the budget and the status reports were all green ticks for years.
Fundamentally, the issue is that they're "going into the cloud" with platform as a service, IaC, and everything, but at some level they don't fully grok what that means and the type of oversight required to make that work at a reasonable cost.
"But the nice sales person from Microsoft assured me the cloud is cheaper!"
The agencies are similarly uncoordinated and will pick up their metaphorical credit card and just throw it at random small dev teams, internally, external, or a mix.
Those people will happily take the credit! The money just… disappears. It’s like a magic trick, or one of those street urchins that rips you off when you’re on holiday in some backwards part of the world like Paris.
I get brought in as “the cloud consultant” for a week or two at the end to deploy the latest ball of mud with live wires sticking out of it to production.
This invariably becomes an argument because the ball of mud the street urchins have sold to the customer is not fit for… anything… certainly not for handling PII or money, but they spent the budget and the status reports were all green ticks for years.
Fundamentally, the issue is that they're "going into the cloud" with platform as a service, IaC, and everything, but at some level they don't fully grok what that means and the type of oversight required to make that work at a reasonable cost.
"But the nice sales person from Microsoft assured me the cloud is cheaper!"
Omg this is something I have experienced too many times, and constantly warring with the other side of the coin: people who never want to make any change unless it is blessed by a consultant from Microsoft/VMWare/SAP and then it becomes the only possible course of action, and they get the CIO to sign off on some idiocy that will never work and say "CIO has decreed Project Falcon MUST SUCCEED" when CIO can't even tie his shoelaces. Giant enterprise integration will happen!
In fact we're going through one of these SAP HANA migrations at present and it's very broken, because the prime contractor has delivered a big ball of mud with lots of internal microservices.
In fact we're going through one of these SAP HANA migrations at present and it's very broken, because the prime contractor has delivered a big ball of mud with lots of internal microservices.
[deleted]
Is this DCS in NSW? If so that would explain so much about my own work interactions with them.
No… but you’re close with your guess!
These new superdepartments all have the same issues to the point that I sometimes confuse them.
These new superdepartments all have the same issues to the point that I sometimes confuse them.
> some backwards part of the world like Paris.
This alone earns my upvote.
This alone earns my upvote.
It's all they've seen. They don't get why they're doing it, because they're junior devs masquerading as architects. There's so many 'senior' or 'architect' level devs in our industry who are utterly useless.
One app I got brought in late on the architect had done some complicated mediator pattern for saving data with a micro service architecture. They'd also semi-implemented DDD.
It was a ten page form. Literally that was what it was supposed to replace. An existing paper, 10 page, form. One of those "domains" was a list of the 1,000 schools in the country. That needed to be updated once a year.
A government spent millions on this thing.
I could have done it on my todd in 3 months. It just needed to use simple forms, with some simple client side logic for hiding sections, and save the data with an ORM.
The funniest bit was when I said that it couldn't handle the load because the architecture had obvious bottlenecks. The load was known and fairly trivial (100k form submissions in one month).
The architect claimed that it wasn't possible as the architecture was all checked and approved by one of the big 5.
So I brought the test server down during the call by making 10 requests at once.
One app I got brought in late on the architect had done some complicated mediator pattern for saving data with a micro service architecture. They'd also semi-implemented DDD.
It was a ten page form. Literally that was what it was supposed to replace. An existing paper, 10 page, form. One of those "domains" was a list of the 1,000 schools in the country. That needed to be updated once a year.
A government spent millions on this thing.
I could have done it on my todd in 3 months. It just needed to use simple forms, with some simple client side logic for hiding sections, and save the data with an ORM.
The funniest bit was when I said that it couldn't handle the load because the architecture had obvious bottlenecks. The load was known and fairly trivial (100k form submissions in one month).
The architect claimed that it wasn't possible as the architecture was all checked and approved by one of the big 5.
So I brought the test server down during the call by making 10 requests at once.
> So I brought the test server down during the call by making 10 requests at once.
Back in the very early 2000s I got sent to "tune IIS performance" at a 100-developer ISV working on a huge government project.
They showed me that pressing the form submit button on just two PCs at once had "bad performance".
No, not it didn't. One was fast[1], the other took 60 seconds almost exactly. "That's a timeout on a lock or something similar", I told them.
They then showed me their 16-socket database server that must have cost them millions and with a straight face asked me if I thought that they needed to upgrade it to get more capacity. Upgrade to what!? That was the biggest machine I have ever seen! I've never in the quarter century since then seen anything that size with my own two eyes. I don't believe bigger Wintel boxes have ever been made.
I then asked their database developers how they're doing transactions and whether they're using stored procedures or not.
One "senior" database developer asked me what a stored procedure is.
The other "senior" database developer asked me what a transaction is.
"Oh boy..."
[1] Well no, not really, it took about a second, which was long enough for a human button press to to "overlap" the two transactions in time. That was a whole other horror story of ODBC connection pooling left off and one-second sleeps in loops to "fix" concurrency issues.
Back in the very early 2000s I got sent to "tune IIS performance" at a 100-developer ISV working on a huge government project.
They showed me that pressing the form submit button on just two PCs at once had "bad performance".
No, not it didn't. One was fast[1], the other took 60 seconds almost exactly. "That's a timeout on a lock or something similar", I told them.
They then showed me their 16-socket database server that must have cost them millions and with a straight face asked me if I thought that they needed to upgrade it to get more capacity. Upgrade to what!? That was the biggest machine I have ever seen! I've never in the quarter century since then seen anything that size with my own two eyes. I don't believe bigger Wintel boxes have ever been made.
I then asked their database developers how they're doing transactions and whether they're using stored procedures or not.
One "senior" database developer asked me what a stored procedure is.
The other "senior" database developer asked me what a transaction is.
"Oh boy..."
[1] Well no, not really, it took about a second, which was long enough for a human button press to to "overlap" the two transactions in time. That was a whole other horror story of ODBC connection pooling left off and one-second sleeps in loops to "fix" concurrency issues.
> It's all they've seen. They don't get why they're doing it, because they're junior devs masquerading as architects. There's so many 'senior' or 'architect' level devs in our industry who are utterly useless.
This is the real, actual conversation to be had about "AI taking jobs."
I've seen similar things a lot in the private sector.
There's just loads of people just flailing around doing stuff without really having any expertise other than some vague proxy of years of experience.
It's really not even exactly their fault (people have lives that don't revolve around messing about with software systems design, sure, and there's no good exposure to anything outside of these messes in their workplaces).
But, outside of major software firms (think banks, and other non-"tech" F500s; speaking from experience here) there's loads of people that are "Enterprise Architects" or something that basically spend 5 hours a day in meetings and write 11 lines of C# or something a quarter and then just adopt ideas they heard from someone else a few years back.
Software is really an utterly bizarre field where there's really nothing that even acts as valuable credentials or experience without complete understanding of what that "experience" is actually comprised of. I think about this a lot.
This is the real, actual conversation to be had about "AI taking jobs."
I've seen similar things a lot in the private sector.
There's just loads of people just flailing around doing stuff without really having any expertise other than some vague proxy of years of experience.
It's really not even exactly their fault (people have lives that don't revolve around messing about with software systems design, sure, and there's no good exposure to anything outside of these messes in their workplaces).
But, outside of major software firms (think banks, and other non-"tech" F500s; speaking from experience here) there's loads of people that are "Enterprise Architects" or something that basically spend 5 hours a day in meetings and write 11 lines of C# or something a quarter and then just adopt ideas they heard from someone else a few years back.
Software is really an utterly bizarre field where there's really nothing that even acts as valuable credentials or experience without complete understanding of what that "experience" is actually comprised of. I think about this a lot.
>Software is really an utterly bizarre field where there's really nothing that even acts as valuable credentials or experience without complete understanding of what that "experience" is actually comprised of. I think about this a lot.
Most other fields are similar, frankly.
Most other fields are similar, frankly.
This is in no way unique to software. Virtually all fields I have insight in are the same and managment is one of the worst.
> They'd also semi-implemented DDD.
One of my pet-peeves. "We're doing DDD, hexagonal architecture, CQRS". So, when was the last time your dev team had a conversation with your domain experts? You have access to domain experts don't you? What does your ubiquitous language look like?
So no, some "senior" read a blog post (and usually just diagonally) and ran with it and now monkey see monkey does is in full effect.
And you get the same shit with everything. How many "manager" read one of the books about the method they tell you they're implementing (or any book about management) ? How many TDD shop where QA and dev are still separate silos? How many CI/CD with no test suite at all? Kanban with no physical board, no agreed upon WIP limits, no queue replenishing system but we use the Kanban board in JIRA.
One of my pet-peeves. "We're doing DDD, hexagonal architecture, CQRS". So, when was the last time your dev team had a conversation with your domain experts? You have access to domain experts don't you? What does your ubiquitous language look like?
So no, some "senior" read a blog post (and usually just diagonally) and ran with it and now monkey see monkey does is in full effect.
And you get the same shit with everything. How many "manager" read one of the books about the method they tell you they're implementing (or any book about management) ? How many TDD shop where QA and dev are still separate silos? How many CI/CD with no test suite at all? Kanban with no physical board, no agreed upon WIP limits, no queue replenishing system but we use the Kanban board in JIRA.
Yeah, exactly, and you see it all over the place. It's not even cargo-culting, it's more half-arsed than that.
"We're all-in on using Kanban here"
"Ah, great. What's your current WIP limit?"
"Um, what's a whip limit?"
As a consultant, I don't actually mind finding myself in the midst of that sort of situation - at the very least, it means I'm going to be able to have a positive impact on the team just by putting in a bit of thought and consistent effort.
"We're all-in on using Kanban here"
"Ah, great. What's your current WIP limit?"
"Um, what's a whip limit?"
As a consultant, I don't actually mind finding myself in the midst of that sort of situation - at the very least, it means I'm going to be able to have a positive impact on the team just by putting in a bit of thought and consistent effort.
On the other side of the coin, once some part of government contacted us about a project they wanted done. I don't even remember what it was, but it was something very simple that we estimated (with huge margins) to be 3 months of work end-to-end. What we heard back was that they need it to take at least two years to make. I suspect some shady incentives are in play and that exceedingly inefficient solutions are a plus for someone up the chain.
The only useful definition of a "service" I've ever heard is that it's a database. Doesn't matter what the jobs and network calls are. One job with two DBs is two services, one DB shared by two jobs is one service. We once had 10 teams sharing one DB, and for all intents and purposes, that was one huge service (a disaster too).
Yes! My viewpoint also. I've been very alone in that view over the last few decades. Nice to know there's someone else out there.
At least it's widely considered bad practice for two services to share a database. But that's different from orienting your entire view of services around databases. The case of one job with two DBs matters too, mostly because there's no strong consistency between them.
Great point. I've only worked at a couple places that architected the system in this manner, where the data layer defines the service boundary. It really helps keep the management of separate services sane IMO, vs. different "services" that all share the same database.
A more precise view is that there are boundaries inside of which certain operations are atomic. To make this more precise: the difference between a “dedicated schema” and a “database” is that the latter is the boundary for transactions and disaster recovery rollbacks.
If you mix in two services into a single database - no matter how good the logical and security isolation is — they will roll back their transactions together if the DBA presses the restore button.
Similarly they have the option (but not the obligation!) to participate in truly atomic transactions instead of distributed transactions. If this is externally observable then this tight coupling means they can no longer be treated as separate apps.
Many architects will just draw directed arrows on diagrams, not realising that any time two icons point at the same icon it often joins them into a single system where none of the parts are functional without all of the others.
If you mix in two services into a single database - no matter how good the logical and security isolation is — they will roll back their transactions together if the DBA presses the restore button.
Similarly they have the option (but not the obligation!) to participate in truly atomic transactions instead of distributed transactions. If this is externally observable then this tight coupling means they can no longer be treated as separate apps.
Many architects will just draw directed arrows on diagrams, not realising that any time two icons point at the same icon it often joins them into a single system where none of the parts are functional without all of the others.
The atomicity is the main reason, but it's also about API boundaries. Sometimes someone makes a "service" that's more like a database exposing a full suite of transactions and CRUD operations, then other teams call themselves "stateless" interacting with it. But really they're sharing a DB, and thus building one giant service, because the API they're using is so broad that it creates an inseparable coupling.
I'm convinced that some people don't know any other way to break down a system into smaller parts. To these people, if it's not exposed as a API call it's just some opaque blob of code that cannot be understood or reused.
That's what I've observed empirically over my last half-dozen jobs. Many developers treat decomposition and contract design between services seriously, and work until they get it right. I've seen very few developers who put the same effort into decomposing the modules of a monolith and designing the interfaces between them, and never enough in the same team to stop a monolith from turning into a highly coupled amorphous blob.
My grug brain conclusion: Grug see good microservice in many valley. Grug see grug tribe carry good microservice home and roast on spit. Grug taste good microservice, many time. Shaman tell of good monolith in vision. Grug also dream of good monolith. Maybe grug taste good monolith after die. Grug go hunt good microservice now.
My grug brain conclusion: Grug see good microservice in many valley. Grug see grug tribe carry good microservice home and roast on spit. Grug taste good microservice, many time. Shaman tell of good monolith in vision. Grug also dream of good monolith. Maybe grug taste good monolith after die. Grug go hunt good microservice now.
We've solved this problem by making the modules in the monolith only able to call each other from well-defined APIs, otherwise CI fails.
In the Java world both Spring and Guice are meant to do this, and if you have an ISomething you've got the possibility of making an ILocalSomething and a IDistributedSomething and swap one for the other.
This is generally a bad idea imo. You fundamentally will have a hard time if your api is opaquely network-dependent or not. I suppose, you’ll be ok if you assume there is a network call, but that means your client will need to pay that cost every time, even if using the ILocal.
It depends on what the API is. For instance you might use something like JDBC or SQLAlchemy to access either a sqlite database or a postgres database.
But you are right that the remote procedure call is a fraught concept for more reasons than one. On one hand there is the fundamental difference between a local procedure call that takes a few ns and a remote call which might take 1,000,000 longer. There's also the fact that most RPC mechanisms that call themselves RPC mechanisms are terribly complicated, like DCOM or the old Sun RPC. In some sense RPC became mainstream once people started pretending it was REST. People say it is not RPC but often you have a function in your front end Javascript like fetch_data(75) and that becomes GET /data/75 and your back end JAXB looks like
But you are right that the remote procedure call is a fraught concept for more reasons than one. On one hand there is the fundamental difference between a local procedure call that takes a few ns and a remote call which might take 1,000,000 longer. There's also the fact that most RPC mechanisms that call themselves RPC mechanisms are terribly complicated, like DCOM or the old Sun RPC. In some sense RPC became mainstream once people started pretending it was REST. People say it is not RPC but often you have a function in your front end Javascript like fetch_data(75) and that becomes GET /data/75 and your back end JAXB looks like
@GET
@Path("/{id}")
public List<Data> fetchData(@PathParam("id") int id) { ... }I honestly think it's the only way outside of one-person projects (and even then...), you need _some_ design pressure.
Maybe the friction imposed to mess up the well-factored microservice arch is sufficiently marginally higher than in the monolith that the perception of value in the factoring is higher, whereas the implicit expectation of factoring the monolith is that you’ll look away for five seconds and someone will ruin it.
I think monoliths are not such a good idea anymore. Particularly with the direction development is going w.r.t the usage of LLMs, I think it's best to break things down. Ofcourse, it shouldn't be overdone.
> grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
> I think it's best to break things down
Factoring system = break things down.
> I think it's best to break things down
Factoring system = break things down.
Put the modules in different git repos and interfaces will get super clean eventually.
> To these people, if it's not exposed as a API call it's just some opaque blob of code that cannot be understood or reused.
I think this is correct as an explanation for the phenomenon, but it's not just a false perception on their part: for a lot of organizations it is actually true that the only way to preserve boundaries between systems over the course of years is to stick the network in between. Without a network layer enforcing module boundaries code does, in fact, tend to morph into a big ball of mud.
I blame a few things for this:
1. Developers almost universally lack discipline.
2. Most programming languages are not designed to sufficiently account for #1.
It's not a coincidence that microservices became popular shortly after Node.js and Python became the dominant web backend languages. A strong static type system is generally necessary (but not sufficient) to create clear boundaries between modules, and both Python and JavaScript have historically been even worse than usual for dynamic languages when it comes to having a strong modularity story.
And while Python and JS have it worse than most, even most of our popular static languages are pretty lousy at giving developers the tools needed to clearly delineate module boundaries. Rust has a pretty decent starting point but it too could stand to be improved.
I think this is correct as an explanation for the phenomenon, but it's not just a false perception on their part: for a lot of organizations it is actually true that the only way to preserve boundaries between systems over the course of years is to stick the network in between. Without a network layer enforcing module boundaries code does, in fact, tend to morph into a big ball of mud.
I blame a few things for this:
1. Developers almost universally lack discipline.
2. Most programming languages are not designed to sufficiently account for #1.
It's not a coincidence that microservices became popular shortly after Node.js and Python became the dominant web backend languages. A strong static type system is generally necessary (but not sufficient) to create clear boundaries between modules, and both Python and JavaScript have historically been even worse than usual for dynamic languages when it comes to having a strong modularity story.
And while Python and JS have it worse than most, even most of our popular static languages are pretty lousy at giving developers the tools needed to clearly delineate module boundaries. Rust has a pretty decent starting point but it too could stand to be improved.
3. Company structure poorly supports cross-team or department code ownership
Many companies don't seem to do a good job coordinating between teams. Different teams have different incentives and priorities. If group A needs fixes/work from group B and B has been given some other priority, group A is stuck.
By putting a network between modules different groups can limit blast damage from other teams' modules and more clearly show ownership when things go wrong. If group A's project fails because of B's module it still looks like A's code has the problem.
Upper management rarely cares about nuance. They want to assign blame, especially if it's in another team or department. So teams under them always want clear boundaries of responsibility so they don't get thrown under the bus.
The root cause of a lot of software problems is the organization that produces it more than any individual or even team working on it.
Many companies don't seem to do a good job coordinating between teams. Different teams have different incentives and priorities. If group A needs fixes/work from group B and B has been given some other priority, group A is stuck.
By putting a network between modules different groups can limit blast damage from other teams' modules and more clearly show ownership when things go wrong. If group A's project fails because of B's module it still looks like A's code has the problem.
Upper management rarely cares about nuance. They want to assign blame, especially if it's in another team or department. So teams under them always want clear boundaries of responsibility so they don't get thrown under the bus.
The root cause of a lot of software problems is the organization that produces it more than any individual or even team working on it.
[O]rganizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations.
— Melvin E. Conway, How Do Committees Invent?
— Melvin E. Conway, How Do Committees Invent?
> Developers almost universally lack discipline.
Or developers are given a deadline and no slack to learn the code base. So developers will tactically take the fastest route to closing their ticket.
Or developers are given a deadline and no slack to learn the code base. So developers will tactically take the fastest route to closing their ticket.
This. You'll take "too long", you'll be told you're overthinking/overengineering, people will preach iterating, that done is better than perfect, etc.
It's not developers that lack discipline. It's CTOs, VPs, etc.
It's not developers that lack discipline. It's CTOs, VPs, etc.
I think languages without proper support for modules are worse off than Python. Python actually has pretty good support for modules and defining their boundaries (via __init__.py).
Well, if people are really that stupid maybe they should just not be developers.
I swear I'm not making this up; a guy at my current client needed to join two CSV files. A one off thing for some business request. He wrote a REST api in Java, where you get the merged csv after POSTing your inputs.
I must scream but I'm in a vacuum. Everyone is fine with this.
(Also it takes a few seconds to process a 500 line test file and runs for ten minutes on the real 20k line input.)
I must scream but I'm in a vacuum. Everyone is fine with this.
(Also it takes a few seconds to process a 500 line test file and runs for ten minutes on the real 20k line input.)
I mean, it would be faster to just import them into an in-memory sqlite database, run a `union all` query and then dump it to a csv...
That's still probably the wrong way to do it, but 10 minutes for a 20k line file? That seems like poor engineering in the most basic sense.
That's still probably the wrong way to do it, but 10 minutes for a 20k line file? That seems like poor engineering in the most basic sense.
I'd probably think of xsv, go to its github repo, remember it's unmaintained and got replaced by qsv, and then use qsv.
It's a twenty line bash script. Pipe some shit into sqlite, done.
But the guy 'is known to get the job done' apparently.
But the guy 'is known to get the job done' apparently.
Maybe he’s recognized something brilliant. Management doesn’t know that the program he wrote was just a reimplementation of the Unix “cut” and “paste” commands, so he might as well get rewarded for their ignorance.
And to be fair, if folks didn’t get paid for reinventing basic Unix utilities with extra steps, the economy would probably collapse.
And to be fair, if folks didn’t get paid for reinventing basic Unix utilities with extra steps, the economy would probably collapse.
Clearly I'm the dumbass in this story, as we're all paid by the hour...
Clearly! He’s found a magic portal to the good old days when the fruit was all low hanging, and you keep showing up with a ladder.
csvkit and duckdb would also be good options. Any llm will spit out a one-liner for any type of join you can describe.
I'm really dumb, genuinely asking the question—when people do such things, where are they generally running the actual code? Would it be in a VM on generally available infra that their company provides...? Or like... On a spare laptop under their desk? I have use cases for similar things (more valid use cases than this one, at least my smooth brain likes to think) but I literally don't know how to deploy it once it's written. I've never been shown or done it before.
Typically you run both the client program and the server program on your computer during development. Even though they're running on the same machine they can talk with one another using http as if they were both on the world wide web.
Then you deploy the server program, and then you deploy the client program, to another machine, or machines, where they continue to talk to one another over http, maybe over the public Internet or maybe not.
Deploying can mean any one of umpteen possible things. In general, you (use automations that) copy your programs over to dedicated machines that then run your programs.
Then you deploy the server program, and then you deploy the client program, to another machine, or machines, where they continue to talk to one another over http, maybe over the public Internet or maybe not.
Deploying can mean any one of umpteen possible things. In general, you (use automations that) copy your programs over to dedicated machines that then run your programs.
The worst part of stories like this is how much potential there is in gaslighting you, the negative person, on just how professional and wonderful this solution is:
* Information hiding by exposing a closed interface via the API
* Isolated, scalable, fault tolerant service
* Iterable, understandable and super agile
Was it joining on some columns or just concatenating the files?
I'm going to laugh pretty hard if it could just be done with: cat file1.csv file2.csv > combined.csv
I'm going to laugh pretty hard if it could just be done with: cat file1.csv file2.csv > combined.csv
You need to account for the headers, which many (most?) csv files I've encountered have.
So I guess something like this to skip the headers in the second file (this also assumes that headers don't have line breaks):
So I guess something like this to skip the headers in the second file (this also assumes that headers don't have line breaks):
cp file1.csv combined.csv && tail -n+2 file2.csv >> combined.csvThere are also a lot of command line options for joining by column like csvkit
To be fair, microservices is about breaking people down into smaller parts, with the idea of mirroring services found in the macro economy, but within the microcosm of a single business. In other words, a business is broken down into different teams that operate in isolation from each other, just as individual businesses do at the macro scale. Any technical outcomes from that are merely a result of Conway's Law.
I have a conspiracy theory that it’s a pattern pushed by cloud to get people to build applications that:
- Cannot be run without an orchestrator like K8S, which is a bear to install and maintain, which helps sell managed cloud.
- Uses more network bandwidth, which they bill for, and CPU, which they bill for.
- Makes it hard to share and maintain complex or large state within the application, encouraging the use of more managed database and event queue services as a substitute, which they bill for. (Example: a monolith can use a queue or a channel, while for microservices you’re going to want Kafka or some other beast.)
- Can’t be run locally easily, meaning you need dev environments in cloud, which means more cloud costs. You might even need multiple dev and test environments. That’s even more cloud cost.
- Tends to become dependent on the peculiarities of a given cloud host, such as how they do networking, increasing cloud lock in.
Anyone else remember how cloud was pitched as saving money on IT? That was hilarious. Knew it was BS way back in the 2000s and that it would eventually end up making everything cost more.
- Cannot be run without an orchestrator like K8S, which is a bear to install and maintain, which helps sell managed cloud.
- Uses more network bandwidth, which they bill for, and CPU, which they bill for.
- Makes it hard to share and maintain complex or large state within the application, encouraging the use of more managed database and event queue services as a substitute, which they bill for. (Example: a monolith can use a queue or a channel, while for microservices you’re going to want Kafka or some other beast.)
- Can’t be run locally easily, meaning you need dev environments in cloud, which means more cloud costs. You might even need multiple dev and test environments. That’s even more cloud cost.
- Tends to become dependent on the peculiarities of a given cloud host, such as how they do networking, increasing cloud lock in.
Anyone else remember how cloud was pitched as saving money on IT? That was hilarious. Knew it was BS way back in the 2000s and that it would eventually end up making everything cost more.
Those are all good points, but missing the most important one, the "Gospel of Scalability". Every other startup wants to be the next Google and therefore thinks they need to design service boundaries that can scale infinitely...
It's 100% this; you're right on the money (pun intended).
Don't forget various pipelines, IaC, pipelines for deploying IaC, test/dev/staging/whatever environments, organization permissions strategies etc etc...
When I worked at a large, uh, cloud company as a consultant, solutions were often tailored towards "best practices"--this meant, in reality, large complex serverless/containerized things with all sorts of integrations for monitoring, logging, NoSQL, queues etc, often for dinky little things that an RPI running RoR or NodeJS could serve without breaking a sweat.
With rare exceptions, we'd never be able to say, deploy a simple go server on a VM with server-side rendered templates behind a load balancer with some auto-scaling and a managed database. Far too pedestrian.
Sure, it's "best practices" for "high-availability" but was almost always overkill and a nightmare to troubleshoot.
Don't forget various pipelines, IaC, pipelines for deploying IaC, test/dev/staging/whatever environments, organization permissions strategies etc etc...
When I worked at a large, uh, cloud company as a consultant, solutions were often tailored towards "best practices"--this meant, in reality, large complex serverless/containerized things with all sorts of integrations for monitoring, logging, NoSQL, queues etc, often for dinky little things that an RPI running RoR or NodeJS could serve without breaking a sweat.
With rare exceptions, we'd never be able to say, deploy a simple go server on a VM with server-side rendered templates behind a load balancer with some auto-scaling and a managed database. Far too pedestrian.
Sure, it's "best practices" for "high-availability" but was almost always overkill and a nightmare to troubleshoot.
There is now an entire generation of developers steeped in SaaS who literally don’t know how to do anything else, and have this insanely distorted picture of how much power is needed to do simple things.
It’s hard to hire people to do anything else. People don’t know how to admin machines so forget bare metal even though it can be thousands of times cheaper for some work loads (especially bandwidth).
You’re not exaggerating with a raspberry pi. Not at all.
It’s hard to hire people to do anything else. People don’t know how to admin machines so forget bare metal even though it can be thousands of times cheaper for some work loads (especially bandwidth).
You’re not exaggerating with a raspberry pi. Not at all.
Thanks for making me feel less alone in this perspective--it's always been kind of verboten to say such a thing in those kinds of workplaces, but all my software type friends agree completely.
The "entire generation of developers" paradigm is all over in different domains too--web programmers that seem to honestly think web development is only React/Angular and seem to have no idea that you can just write JS, python programmers that have no idea a large portion of the "performant codebases" are piles of native dependencies etc
The "entire generation of developers" paradigm is all over in different domains too--web programmers that seem to honestly think web development is only React/Angular and seem to have no idea that you can just write JS, python programmers that have no idea a large portion of the "performant codebases" are piles of native dependencies etc
One way of looking at it is that there are about the same number of programmers today with a deep understanding of the machine as there were in the 90s. There are just 3-4X more programmers who are just after a career and learn only the skills necessary and follow what seem to be the most employable trends.
Same goes for users. There are about the same number of computer literate users as there were back then. There’s just a new class of iPad/iPhone user who is only a casual user of computers and the net and barely knows what a file is.
Same goes for users. There are about the same number of computer literate users as there were back then. There’s just a new class of iPad/iPhone user who is only a casual user of computers and the net and barely knows what a file is.
Unfortunately it is useful to do this for many other reasons!
Unfortunately indeed. I lament the necessity of microservices at my current job. It’s just such a silver bullet for so many scaling problems.
The scaling problems we face could probably be solved by other solutions, but the company is primed and ready to chuck functionality into new microservices. That’s what all our infrastructure is set up to do, and it’s what inevitably happens every time
The scaling problems we face could probably be solved by other solutions, but the company is primed and ready to chuck functionality into new microservices. That’s what all our infrastructure is set up to do, and it’s what inevitably happens every time
The frequency that you use the term "re-factor" over the term "factor" is often very telling about how you develop your systems. I worked a job one time where the guys didn't even know what factoring was.
Probably many people don't pick up on the word "to factor" something these days. They do not make the connection between the thing that mathematicians do and what that could relate to in terms of writing code. At the same time everyone picks up the buzzword "to refactor". It all depends on what ecosystems you expose yourself to. I think I first heard the term "to factor" something in math obviously, but in software when I looked at some Forth. Most people will never do that, because it is so far off the mainstream, that they have never even heard of it.
[deleted]
The network boundary gives you a factoring tool that most language module systems don't: the ability for a collection of packages to cooperate internally but expose only a small API to the rest of the codebase. The fact that it's network further disciplines the modules to exchange only data (not callbacks or behaviors) which simplifies programming, and to evolve their interfaces in backwards compatible ways, which makes it possible to "hot reload" different modules at different times without blowing up.
You could probably get most of this without the literal network hop, but I haven't seen a serious attempt.
You could probably get most of this without the literal network hop, but I haven't seen a serious attempt.
Any language that offers a mechanism for libraries has formal or informal support for defining modules with public APIs?
Or maybe I’m missing what you mean - can you explain with an example an API boundary you can’t define by interfaces in Go, Java, C# etc? Or by Protocols in Python?
Or maybe I’m missing what you mean - can you explain with an example an API boundary you can’t define by interfaces in Go, Java, C# etc? Or by Protocols in Python?
The service I'm working on right now has about 25 packages. From the language's perspective, each package is a "module" with a "public" API. But from the microservices architecture's perspective, the whole thing is one module with only a few methods.
But why would users of the module care about the dependency packages? You could still have a module with only a few methods and that's the interface.
i’ve seen devs do stuff like this (heavily simplified example)
microservices putting a network call in on the factoring is a feature in this case, not a bug. it’s a physical blocker stopping devs doing stuff like that. it’s the one thing i don’t agree with grug on.
HOWEVER — it’s only a useful club if you use it well. and most of the time it’s used because of expectations of shiny rocks, putting statements about microservices in the company website, big brain dev making more big brain resume.
from submodule import pandas
microservices putting a network call in on the factoring is a feature in this case, not a bug. it’s a physical blocker stopping devs doing stuff like that. it’s the one thing i don’t agree with grug on.
HOWEVER — it’s only a useful club if you use it well. and most of the time it’s used because of expectations of shiny rocks, putting statements about microservices in the company website, big brain dev making more big brain resume.
True - but most languages make it much easier than Python to disallow this kind of accidental public API creation. Python inverts the public API thing - in most (all?) other mainstream languages I can think of you need to explicitly export the parts of your module you want to be public API.
You can do this in Python as well, but it does involve a bit of care; I like the pattern of a module named “internal” that has the bulk of the modules code in it, and a small public api.py or similar that explicitly exposes the public bits, like an informal version of the compiler-enforced pattern for this in Go
You can do this in Python as well, but it does involve a bit of care; I like the pattern of a module named “internal” that has the bulk of the modules code in it, and a small public api.py or similar that explicitly exposes the public bits, like an informal version of the compiler-enforced pattern for this in Go
If “everyone would just” restrict themselves to import only the package you meant to public API, sure, it would work. But everyone will not just.
I'm not sure why you would bother, though. If you need the package, just import it directly, no? (besides, in many languages you can't even do that kind of thing)
grug hears microservice shaman talk about smol api but then grug see single database, shared queue, microservice smol but depend on huge central piece, big nest of complexity demon waiting to mock grug
I think mostly this is to brake down the system between teams. This is easier to manage this way. Nothing to do with technical decision - more the way of development.
What is the alternative? Mono-repo? IMHO it is even worse.
microservices and mono repo are not mutually exclusive. Monolith, is. Important distinction imo, Micro services in mono repo definitely works and ime is >>> multi repo.
Of course the best is mono repo and monolith :3
Of course the best is mono repo and monolith :3
I did not stated that they can be used together. I just meant that they are mainly, for me, work as a tool to allow multiple teams work on a single big project. So the tool is more for work organisation purpose not for architectural one.
If you prefer working on mono repo it s fine though I think micro services are more popular one now.
If you prefer working on mono repo it s fine though I think micro services are more popular one now.
Am I reading it wrong ? I understand both your original comment and this one to paint mono repo as an alternative to micro services. E.g.: “ What is the alternative [to micro services]? Mono-repo?” and “ If you prefer working on mono repo it s fine though I think micro services are more popular one now.” What I understand here is that “mono repo is an alternative to micro services”. Is that what you’re saying?
If so: mono repo is not an alternative to micro services. You can have both. You can have micro services in a mono repo. One single repository with many micro services. That is possible.
What is an alternative to micro services is “monolith”.
Curious how I’m misunderstanding this because I looked at it for a while and I really don’t see it.
If so: mono repo is not an alternative to micro services. You can have both. You can have micro services in a mono repo. One single repository with many micro services. That is possible.
What is an alternative to micro services is “monolith”.
Curious how I’m misunderstanding this because I looked at it for a while and I really don’t see it.
> given choice between complexity or one on one against t-rex, grug take t-rex: at least grug see t-rex
I think about this line at least once a week
I think about this line at least once a week
grug obviously never took on invisible t-rex
this grug keeps one on one invisible t-rex, grug cursed
this grug keeps one on one invisible t-rex, grug cursed
"...even as he fell, Leyster realized that he was still carrying the shovel. In his confusion, he’d forgotten to drop the thing. So, desperately, he swung it around with all his strength at the juvenile’s legs.
Tyrannosaurs were built for speed. Their leg bones were hollow, like a bird’s. If he could break a femur …
The shovel connected, but not solidly. It hit without breaking anything. But, still, it got tangled up in those powerful legs. With enormous force, it was wrenched out of his hands. Leyster was sent tumbling on the ground.
Somebody was screaming. Dazed, Leyster raised himself up on his arms to see Patrick, hysterically slamming the juvenile, over and over, with the butt of the shotgun. He didn’t seem to be having much effect. Scarface was clumsily trying to struggle to its feet. It seemed not so much angry as bewildered by what was happening to it.
Then, out of nowhere, Tamara was standing in front of the monster. She looked like a warrior goddess, all rage and purpose. Her spear was raised up high above Scarface, gripped tightly in both hands. Her knuckles were white.
With all her strength, she drove the spear down through the center of the tyrannosaur’s face. It spasmed, and died. Suddenly everything was very still."
Tyrannosaurs were built for speed. Their leg bones were hollow, like a bird’s. If he could break a femur …
The shovel connected, but not solidly. It hit without breaking anything. But, still, it got tangled up in those powerful legs. With enormous force, it was wrenched out of his hands. Leyster was sent tumbling on the ground.
Somebody was screaming. Dazed, Leyster raised himself up on his arms to see Patrick, hysterically slamming the juvenile, over and over, with the butt of the shotgun. He didn’t seem to be having much effect. Scarface was clumsily trying to struggle to its feet. It seemed not so much angry as bewildered by what was happening to it.
Then, out of nowhere, Tamara was standing in front of the monster. She looked like a warrior goddess, all rage and purpose. Her spear was raised up high above Scarface, gripped tightly in both hands. Her knuckles were white.
With all her strength, she drove the spear down through the center of the tyrannosaur’s face. It spasmed, and died. Suddenly everything was very still."
I know this is fiction because everyone is focused on the same problem.
I felt like my third eye has been opened after reading that. Truly inspiring.
One thing to appreciate is that this article comes from someone who can do the more sophisticated (complex) thing, but tries not to based on experience.
There is of course a time and place for sophistication, pushing for higher levels of abstraction and so on. But this grug philosophy is saying that there isn't any inherent value in doing this sort of thing and I think that is very sound advice.
Also I noticed AI assistance is more effective with consistent, mundane and data driven code. YMMV
There is of course a time and place for sophistication, pushing for higher levels of abstraction and so on. But this grug philosophy is saying that there isn't any inherent value in doing this sort of thing and I think that is very sound advice.
Also I noticed AI assistance is more effective with consistent, mundane and data driven code. YMMV
> Everything should be made as simple as possible, but not simpler
The time and place for sophistication and abstraction is when and where they make the code easier to understand without first needing a special course to explain why it's easier to understand. (It varies by situation which courses can be taken for granted.)
I feel like this would fit the bell curve meme -
Novice dev writes simple code
Intermediate dev writes complex code
Expert dev writes simple code
Novice dev writes simple code
Intermediate dev writes complex code
Expert dev writes simple code
I gave this advice to an intermediate dev at my company a couple of years ago
Something along the lines of "Hey, you're a great developer, really smart, you really know your stuff. But you have to stop reaching for the most complicated answer to everything"
He took it to heart and got promoted at the start of this year. Was nice to see. :)
Something along the lines of "Hey, you're a great developer, really smart, you really know your stuff. But you have to stop reaching for the most complicated answer to everything"
He took it to heart and got promoted at the start of this year. Was nice to see. :)
One of the many ironies of modern software development is that we sometimes introduce complexity because we think it will "save time in the end". Sometimes we're right and it does save time--but not always and maybe not often.
Three examples:
DRY (Don't Repeat Yourself) sometimes leads to premature abstraction. We think, "hey, I bet this pattern will get used elsewhere, so we need to abstract out the common parts of the pattern and then..." And that's when the Complexity Demon enters.
We want as many bugs as possible caught at compile-time. But that means the compiler needs to know more and more about what we're actually trying to do, so we come up with increasingly complex types which tax your ability to understand.
To avoid boilerplate we create complex macros or entire DSLs to reduce typing. Unfortunately, the Law of Leaky Abstractions means that when we actually need to know the underlying implementation, our head explodes.
Our challenge is that each of these examples is sometimes a good idea. But not always. Being able to decide when to introduce complexity to simplify things is, IMHO, the mark of a good software engineer.
Three examples:
DRY (Don't Repeat Yourself) sometimes leads to premature abstraction. We think, "hey, I bet this pattern will get used elsewhere, so we need to abstract out the common parts of the pattern and then..." And that's when the Complexity Demon enters.
We want as many bugs as possible caught at compile-time. But that means the compiler needs to know more and more about what we're actually trying to do, so we come up with increasingly complex types which tax your ability to understand.
To avoid boilerplate we create complex macros or entire DSLs to reduce typing. Unfortunately, the Law of Leaky Abstractions means that when we actually need to know the underlying implementation, our head explodes.
Our challenge is that each of these examples is sometimes a good idea. But not always. Being able to decide when to introduce complexity to simplify things is, IMHO, the mark of a good software engineer.
For folks who seek a rule of thumb, I’ve found SPoT (single point of truth) a better maxim than DRY: there should be ideally one place where business logic is defined. Other stuff can be duplicated as needed and it isn’t inherently a bad thing.
To modulate DRY, I try to emphasize the “rule of three”: up to three duplicates of some copy/paste code is fine, and after that we should think about abstracting.
Of course no rule of thumb applies in all cases, and the sense for that is hard to teach.
To modulate DRY, I try to emphasize the “rule of three”: up to three duplicates of some copy/paste code is fine, and after that we should think about abstracting.
Of course no rule of thumb applies in all cases, and the sense for that is hard to teach.
100% agree. Duplication is far cheaper than the wrong abstraction.
Student: I notice that you duplicated code here rather than creating an abstraction for both.
Master: That is correct.
Student: But what if you need to change the code in the future?
Master: Then I will change it in the future.
At that point the student became enlightened.
Student: I notice that you duplicated code here rather than creating an abstraction for both.
Master: That is correct.
Student: But what if you need to change the code in the future?
Master: Then I will change it in the future.
At that point the student became enlightened.
[deleted]
> To modulate DRY, I try to emphasize the “rule of three”: up to three duplicates of some copy/paste code is fine, and after that we should think about abstracting
Just for fun, this more or less already exists as another acronym: WET. Write Everything Twice
It basically just means exactly what you said. Don't bother DRYing your code until you find yourself writing it for the third time.
Just for fun, this more or less already exists as another acronym: WET. Write Everything Twice
It basically just means exactly what you said. Don't bother DRYing your code until you find yourself writing it for the third time.
And to those who feel the OCD and fear of forgetting coming over by writing twice, put TODOs on both spots; so that when the third time comes, you can find the other two easily. If you are the backlogging type, put JIRA reference with the TODOs to make finding even easier.
> I’ve found SPoT (single point of truth) a better maxim than DRY
I totally agree. For example having 5 variables that are all the same value but mean very different things is good. Combining them to one variable would be "DRY" but would defeat separations of concern. With variables its obvious but the same applies to more complex concepts like functions, classes, programs to a degree.
It's fine to share code across abstractions but you gotta make sure that it doesn't end up tying these things too much together just for the cause of DRY.
I totally agree. For example having 5 variables that are all the same value but mean very different things is good. Combining them to one variable would be "DRY" but would defeat separations of concern. With variables its obvious but the same applies to more complex concepts like functions, classes, programs to a degree.
It's fine to share code across abstractions but you gotta make sure that it doesn't end up tying these things too much together just for the cause of DRY.
I still believe that most code, on average, is not DRY enough, but for projects I do on my own account I've recently developed a doctrine of "there are no applications, only screens" and funny enough this has been using HTMX which I think the author of that blog wrote.
Usually I make web applications using Sinatra-like frameworks like Flask or JAXB where I write a function that answers URLs that match a pattern and a "screen" is one or more of those functions that work together and maybe some HTML templates that go with them. For instance there might be a URL for a web page that shows data about a user, and another URL that HTMX calls when you flip a <select> to change the status of that user.
Assuming the "application" has the stuff to configure the database connection and file locations and draw HTML headers and footers and such, there is otherwise little coupling between the screens so if you want to make a new screen you can cut and paste an old screen and modify it, or you can ask an LLM to make you a screen or endpoint and if it "vibe coded" you a bad screen you can just try again to make another screen. It can make sense to use inheritance or composition to make a screen that can be specialized, or to write screens that are standalone (other than fetching the db connection and such.)
The origin story was that I was working on a framework for making ML training sets called "Themis" that was using microservices, React, Docker and such. The real requirement was that we were (1) always adding new tasks, and (2) having to create simple but always optimized "screens" for those tasks because if you are making 20,000 judgements it is bad enough to click 20,000 times, if you have to click 4x for each one and it adds up to 80,000 you will probably give up. As it was written you had to write a bunch of API endpoints as part of a JAXB application and React components that were part of a monolithic React app and wait 20 minutes for typescript and Docker and javac to do their things and if you are lucky it boots up otherwise you have to start over.
I wrote up a criticism of Themis and designed "Nemesis" that was designed for rapid development of new tasks and it was a path not taken at the old job, but Nemesis and I have been chewing through millions of instances of tasks ever since.
Usually I make web applications using Sinatra-like frameworks like Flask or JAXB where I write a function that answers URLs that match a pattern and a "screen" is one or more of those functions that work together and maybe some HTML templates that go with them. For instance there might be a URL for a web page that shows data about a user, and another URL that HTMX calls when you flip a <select> to change the status of that user.
Assuming the "application" has the stuff to configure the database connection and file locations and draw HTML headers and footers and such, there is otherwise little coupling between the screens so if you want to make a new screen you can cut and paste an old screen and modify it, or you can ask an LLM to make you a screen or endpoint and if it "vibe coded" you a bad screen you can just try again to make another screen. It can make sense to use inheritance or composition to make a screen that can be specialized, or to write screens that are standalone (other than fetching the db connection and such.)
The origin story was that I was working on a framework for making ML training sets called "Themis" that was using microservices, React, Docker and such. The real requirement was that we were (1) always adding new tasks, and (2) having to create simple but always optimized "screens" for those tasks because if you are making 20,000 judgements it is bad enough to click 20,000 times, if you have to click 4x for each one and it adds up to 80,000 you will probably give up. As it was written you had to write a bunch of API endpoints as part of a JAXB application and React components that were part of a monolithic React app and wait 20 minutes for typescript and Docker and javac to do their things and if you are lucky it boots up otherwise you have to start over.
I wrote up a criticism of Themis and designed "Nemesis" that was designed for rapid development of new tasks and it was a path not taken at the old job, but Nemesis and I have been chewing through millions of instances of tasks ever since.
Fascinating!
I also recoiled at the complexity of React, Docker, etc. and went a different path: I basically moved all the code to the server and added a way to "project" the UI to the browser. From the program's perspective, you think you're just showing GUI controls on a local screen. There is no client/server split. Under the covers, the platform talks to some JavaScript on the browser to render the controls.
This works well for me since I grew up programming on Windows PCs, where you have full control over the machine. Check it out if you're interested: https://gridwhale.com.
I think pushing code to the server via HTMX and treating the browser like a dumb terminal has the same kind of advantage: you only have to worry about one system.
Fundamentally, IMHO, the client/server split is where all the complexity happens. If you're writing two programs, on the client and one on the server, you're basically creating a distributed system, which we know is very hard.
I also recoiled at the complexity of React, Docker, etc. and went a different path: I basically moved all the code to the server and added a way to "project" the UI to the browser. From the program's perspective, you think you're just showing GUI controls on a local screen. There is no client/server split. Under the covers, the platform talks to some JavaScript on the browser to render the controls.
This works well for me since I grew up programming on Windows PCs, where you have full control over the machine. Check it out if you're interested: https://gridwhale.com.
I think pushing code to the server via HTMX and treating the browser like a dumb terminal has the same kind of advantage: you only have to worry about one system.
Fundamentally, IMHO, the client/server split is where all the complexity happens. If you're writing two programs, on the client and one on the server, you're basically creating a distributed system, which we know is very hard.
On my laptop I have a yin-yang sticker with the yin labeled DRY and the yang labeled YAGNI.
I love it!
sometime grug spend 100 hours building machine to complete work, but manual work takes 1 hour. or spend 1 hour to make machine, lead to 100 hours fixing stupid machine.
dont matter if complex or simple, if result not add value. focus on add more value than detract, worry complexity after
dont matter if complex or simple, if result not add value. focus on add more value than detract, worry complexity after
DRY isn't very dangerous. It's not telling you to spin off a helper that's only used in one place. If a ton of logic is in one function/class/file/whatever, it's still DRY as long as it's not copied.
Premature abstraction is a thing. Doesn't help that every CS course kinda tells you to do this. Give a new grad a MySQL database and the first thing they might try to do is abstract away MySQL.
Premature abstraction is a thing. Doesn't help that every CS course kinda tells you to do this. Give a new grad a MySQL database and the first thing they might try to do is abstract away MySQL.
One of my favorite LLM uses is to feed it this essay, then ask it to assume the persona of the grug-brained developer and comment on $ISSUE_IM_CURRENTLY_DEALING_WITH. Good stress relief.
I am not very proficient with LLMs yet, but this sounds awesome! How do you do that, to "feed it this essay"? Do you just start the prompt with something like "Act like the Grug Brained Developer from this essay <url>"?
Could put it in a ChatGPT project description or Cursor rules to avoid copy pasting every time.
> complexity very bad
Oh boy, this is so true. In all my years of software engineering this is one of those ideas that has proved consistently true in every single situation. Some problems are inherently complex, yes, but even then you'd be much, much better off spending time to think things through to arrive at the simplest way to solve it. Again and again my most effective work has been after I questioned my prior approaches and radically simplified things. You might lose some potential flexibility, but in most case you don't even need all that you think you need.
Some examples:
- Now that reasonably good (and agentic) LLMs are a thing, I started avoiding overly complex TypeScript types that are brittle and hard to debug, in favor of writing spec-like code and asking the LLM to statically generate other code based on it.
- The ESLint dependency in my projects kept breaking after version updates, many rules were not sophisticated enough to avoid false positives, and keeping it working properly with TypeScript and VSCode was getting complicated. I switched to Biome.js, and it was simpler and just as effective. However, I'm recently having bugs with it (not sure if Biome itself or the VSCode extension is to blame). But whatever, I realized that linting is a nice-to-have, not something I should be spending inordinate amount of times babying. So I removed it from the build tool-chain, and neither do I even need have it enabled all the time in VSCode. I run Biome every now and then to check the code style and formatting , and that's it, simple.
- Working on custom data migration tooling for my projects, I realized forward migrations are necessary to implement, but backwards migrations are not worth the time and complexity to implement. In case a database with data needs to be rolled back, just restore the backup. If there was no data, or it is not a production database, just run the versioned initialization script(s) to start from a clean state. Simple.
Oh boy, this is so true. In all my years of software engineering this is one of those ideas that has proved consistently true in every single situation. Some problems are inherently complex, yes, but even then you'd be much, much better off spending time to think things through to arrive at the simplest way to solve it. Again and again my most effective work has been after I questioned my prior approaches and radically simplified things. You might lose some potential flexibility, but in most case you don't even need all that you think you need.
Some examples:
- Now that reasonably good (and agentic) LLMs are a thing, I started avoiding overly complex TypeScript types that are brittle and hard to debug, in favor of writing spec-like code and asking the LLM to statically generate other code based on it.
- The ESLint dependency in my projects kept breaking after version updates, many rules were not sophisticated enough to avoid false positives, and keeping it working properly with TypeScript and VSCode was getting complicated. I switched to Biome.js, and it was simpler and just as effective. However, I'm recently having bugs with it (not sure if Biome itself or the VSCode extension is to blame). But whatever, I realized that linting is a nice-to-have, not something I should be spending inordinate amount of times babying. So I removed it from the build tool-chain, and neither do I even need have it enabled all the time in VSCode. I run Biome every now and then to check the code style and formatting , and that's it, simple.
- Working on custom data migration tooling for my projects, I realized forward migrations are necessary to implement, but backwards migrations are not worth the time and complexity to implement. In case a database with data needs to be rolled back, just restore the backup. If there was no data, or it is not a production database, just run the versioned initialization script(s) to start from a clean state. Simple.
Your two first examples, you just hide the complexity by using another tool, no ?
And I don’t see how number 3 is simpler. In my maths head I can easily create bijective spaces. Emulating backward migration through others means might be harder (depending on details of course thats not a general rule)
And I don’t see how number 3 is simpler. In my maths head I can easily create bijective spaces. Emulating backward migration through others means might be harder (depending on details of course thats not a general rule)
I'm not sure where the complexity is hiding in my examples.
For the code generation, note that some types are almost impossible to express properly, but code can be generated using simpler types that capture all the same constraints that you wanted. And, of course I only use this approach for cases where it is not that complicated to generate the code, and so I can be sure that each time I need to (re)generate it, it will be done correctly (ie., the abstraction is not leaky). Also, I don't use this approach for generating large amounts of code, which would hide the inherent structure of the code when reading it.
For the eslint example, I simply made do without depending on linting as a hard dependency that is always active. That is one of my points: sometimes simply some "niceties" would simplify thing a lot. As another example in this vein, I avoid too much complex configuration and modding of my dev environment; that allows me to focus on what matters.
In the migration example, the complexity with backward migration is that you then need to write a reverse migration script for every forward migration script. Keeping this up and managing and applying them properly can become complex. If you have a better way of doing it I'd like to hear it.
For the code generation, note that some types are almost impossible to express properly, but code can be generated using simpler types that capture all the same constraints that you wanted. And, of course I only use this approach for cases where it is not that complicated to generate the code, and so I can be sure that each time I need to (re)generate it, it will be done correctly (ie., the abstraction is not leaky). Also, I don't use this approach for generating large amounts of code, which would hide the inherent structure of the code when reading it.
For the eslint example, I simply made do without depending on linting as a hard dependency that is always active. That is one of my points: sometimes simply some "niceties" would simplify thing a lot. As another example in this vein, I avoid too much complex configuration and modding of my dev environment; that allows me to focus on what matters.
In the migration example, the complexity with backward migration is that you then need to write a reverse migration script for every forward migration script. Keeping this up and managing and applying them properly can become complex. If you have a better way of doing it I'd like to hear it.
Ah yeah in that case it all makes sense
> Your two first examples, you just hide the complexity by using another tool, no ?
The article says that the best way to manage complexity is to find good cut-points to contain complexity. Another tool is an excellent cut-point, probably the best one there is. (Think about how much complexity a compiler manages for you without you ever having to worry about it.)
The article says that the best way to manage complexity is to find good cut-points to contain complexity. Another tool is an excellent cut-point, probably the best one there is. (Think about how much complexity a compiler manages for you without you ever having to worry about it.)
Many talk complexity. Few say what mean complexity. Big brain Rich say complect is tie together. Me agree. Big brain Rich say complexity bad. Me disagree. Tie things necessary. If things not connected things not solve problem.
Haha. But I thought Rich Hickey was making the simple point that don't intertwine things than can be kept separate!
P.S: For those wondering what this refers to, here is his talk: https://youtu.be/SxdOUGdseq4?t=1896
P.S: For those wondering what this refers to, here is his talk: https://youtu.be/SxdOUGdseq4?t=1896
I can't believe this is (2022). I would have confidently told you I read this 10 years ago and guessed that it was already a classic then.
Same. I think it is inspired by an earlier work.
sad but true: learn "yes" then learn blame other grugs when fail, ideal career advice
This is, I think, my favorite essay about building software. The style is charming (I can see why some might not like it) and the content is always relevant.
I know, I get it, but I've realised that I'm not actually grug-brained. The way my brain works, I remember things pretty well; I like to get into the details of systems. So if more complexity in the code means the app can do more or a task is automated away I'll make the change and know I'll be able to remember how it works in the future.
This doesn't mean OP is bad advice, just make a conscious decision about what to do with complexity and understand the implications.
This doesn't mean OP is bad advice, just make a conscious decision about what to do with complexity and understand the implications.
The knowing how it works in the future should really just be comments, right? And if it’s a bit more complex, perhaps a markdown file in a docs folder or stuffed in a README? When working with a large enough organization, tribal knowledge is an invisible t-rex
I don't think comments can capture the complexity of everything - there's too much interaction between systems to explain it all. I'm probably unique here in that my tribe is just one person: I wouldn't recommend adopting a pet t-rex in a team.
What about the rest of your team?
I am the rest of my team. So yeah, not applicable to most people.
big brain dev say, "me add complexity. no problem."
grug whisper: “problem come later.”
grug see lone dev make clever code.
grug light torch for future archaeologist.
grug whisper: “problem come later.”
grug see lone dev make clever code.
grug light torch for future archaeologist.
While I agree that complexity is bad the fact that we don't really have a shared understanding of what complexity is doesn't help. At worst, it can be just another synonym for "bad" that passes through the mental firewall without detection. For instance is having multiple files in a project "complex"? If I am unfamiliar with a codebase is it "complex" and I therefore have to re-write it?
Part of the problem with complexity is that it is very easy for engineers to justify. Yes, there is an important distinction between necessary and accidental complexity, but, to take a point from the essay, even necessary complexity can be reduced by saying "no" to features.
This is why I treat "complexity bad" as a mantra to keep me in the right mindset when programming. Complexity bad. Even necessary complexity. We may have to deal with it, but, like fire, it's still dangerous.
This is why I treat "complexity bad" as a mantra to keep me in the right mindset when programming. Complexity bad. Even necessary complexity. We may have to deal with it, but, like fire, it's still dangerous.
What exactly is complexity?
How do I recognise it?
When do I tell whether something is complex or I am just not very familiar with it?
How do I recognise it?
When do I tell whether something is complex or I am just not very familiar with it?
really hard questions
but complexity bad
but complexity bad
If Grug sees new code base, sometime Grug get anxiety about learning new code. What in it for Grug? Says Grug. Rather start new project and say "complexity bad" to other Grugs. If other Grug or big brain disagree, create Grug tribe - show bible to other Grugs and big brains. Must convert to Grug way or leave team. Complexity bad.
Grog is not Grug. Grog talk like Grug, walk like Grug, obey rituals of simplicity worship like Grug. But Grog secretly just mean: anything he didn't invent, not worth learning, anything not making his resume-weave, not worth doing. So Grog is always saying, boo! complexity here! rewrite time!!
Grug like say - No True Grugsman. Only True Grug pure Grug.
Grug hurt itself in its confusion!
I think you hit the nail on the head. This article is definitely biased against modern front-end development for example and recommends HTMX as less "complex", but from what I've seen, using HTMX just trades one form of complexity for another.
This part:
> back end better more boring because all bad ideas have tried at this point maybe (still retry some!)
I entered a Spring Boot codebase recently, and it was anything but boring or "simple" -- everything is wrapped by convention in classes/abstract classes/extending layers deep of interfaces, static classes for single methods. Classic OO design that I thankfully moved away from after college.
I think the author makes good points, but I don't think the author is any different than your average developer who accuses the thing they are not familiar with to be "complex".
This part:
> back end better more boring because all bad ideas have tried at this point maybe (still retry some!)
I entered a Spring Boot codebase recently, and it was anything but boring or "simple" -- everything is wrapped by convention in classes/abstract classes/extending layers deep of interfaces, static classes for single methods. Classic OO design that I thankfully moved away from after college.
I think the author makes good points, but I don't think the author is any different than your average developer who accuses the thing they are not familiar with to be "complex".
Seems consistent to me. Does your example fit within his caveat “(still retry some!)”.
The main thing with modern web applications is that everyone wants to move complexity away from the backend into the front end. The problem with this is that nobody asks if it can be done, they just start doing it. The result is that you still end up with a complex backend, but now you also have a monstrous front end. And now you need to reconcile state between two different places - good luck! That problem right there - reconciling state from different places - is one of the hardest ones in software engineering.
Yeah, everyone says complexity is bad
Related. Others?
The Grug Brained Developer (2022) - https://news.ycombinator.com/item?id=38076886 - Oct 2023 (192 comments)
The Grug Brained Developer - https://news.ycombinator.com/item?id=31840331 - June 2022 (374 comments)
The Grug Brained Developer (2022) - https://news.ycombinator.com/item?id=38076886 - Oct 2023 (192 comments)
The Grug Brained Developer - https://news.ycombinator.com/item?id=31840331 - June 2022 (374 comments)
The anecdote about rob pike and logging made me chuckle.
Fun fact about Google: logging is like 95% of the job, easily... From tracking everything every service is doing all the time to wrangling the incoming raw crawl data, it's all going through some kind of logging infrastructure.
I was there when they actually ran themselves out of integers; one of their core pieces of logging infrastructure used a protocol buffer to track datatypes of logged information. Since each field in a protocol buffer message is tagged with an integer key, they hit the problem when their top-level message bumped up against the (if memory serves) int16 implementation limit on maximum tag ID and had to scramble to fix it.
Fun fact about Google: logging is like 95% of the job, easily... From tracking everything every service is doing all the time to wrangling the incoming raw crawl data, it's all going through some kind of logging infrastructure.
I was there when they actually ran themselves out of integers; one of their core pieces of logging infrastructure used a protocol buffer to track datatypes of logged information. Since each field in a protocol buffer message is tagged with an integer key, they hit the problem when their top-level message bumped up against the (if memory serves) int16 implementation limit on maximum tag ID and had to scramble to fix it.
Me no like grug perpetuate complexity myth. Little grugs no understand complexity is not monolith. Me want fix that.
Complexity not bad. Complexity just mean "thing have many consideration". Some thing always have many consideration. Not bad if useful and necessary.
Complexity still difficult and raise problem. So try avoid complexity when unnecessary and no add value. But shun complexity bad when it detract value or raise new problem.
Little grug benefit from not over-simplify. Not everything just good or bad, most thing both, it depend.
Complexity not bad. Complexity just mean "thing have many consideration". Some thing always have many consideration. Not bad if useful and necessary.
Complexity still difficult and raise problem. So try avoid complexity when unnecessary and no add value. But shun complexity bad when it detract value or raise new problem.
Little grug benefit from not over-simplify. Not everything just good or bad, most thing both, it depend.
that's a good point, but complexity bad
if complexity bad, then houses bad, air conditioning bad, roads bad, clothing bad, farming bad, computers bad, ....
simpler to live naked in wood, use big rock hunt food. but grug really like pizza. use computer order pizza, leave tip on app. grug no have to social interact, make grug happy.
manage complexity well and it make good result, worth occasional pain. soon complexity become normal.
simpler to live naked in wood, use big rock hunt food. but grug really like pizza. use computer order pizza, leave tip on app. grug no have to social interact, make grug happy.
manage complexity well and it make good result, worth occasional pain. soon complexity become normal.
it's a good point, well worth considering, but also complexity bad
This concept is really interesting when you think about statically typed, pure functional languages. I like working in them because I'm too pretty and stupid to think about side effects in every function. My hair is too shiny and my muscles are too large to have to deal with "what if this input is null?" everywhere. Can't do it. Need to wrap that bad boy up in a Maybe or some such and have the computer tell me what I'm forgetting to handle with a red squiggly.
Formal proof languages are pretty neat, but boy are they tedious to use in practice. It is unsurprising that effectively no code is written in them. How do you overcome that?
Where the type system isn't that expressive, you still have to fall back to testing to fill in the places where the type system isn't sufficient, and your tests are also going to 'deal with "what if this input is null?" everywhere' cases and whatnot by virtue of execution constraints anyway.
Where the type system isn't that expressive, you still have to fall back to testing to fill in the places where the type system isn't sufficient, and your tests are also going to 'deal with "what if this input is null?" everywhere' cases and whatnot by virtue of execution constraints anyway.
I'm just talking null-less FP languages such as Haskell and Elm, not a full proof system such Lean and Agda or a formal specification language such as TLA+.
I'm not sure I agree with your prior that "your tests are also going to 'deal with "what if this input is null?" everywhere' cases and whatnot." Invalid input is at the very edge of the program where it goes through a parser. If I parse a string value into a type with no room for null, that eliminates a whole class of errors throughout the program. I do need to test that my parser does what I assume it will, sure. But once I have a type that cannot represent illegal data, I can focus my testing away from playing defense on every function.
I'm not sure I agree with your prior that "your tests are also going to 'deal with "what if this input is null?" everywhere' cases and whatnot." Invalid input is at the very edge of the program where it goes through a parser. If I parse a string value into a type with no room for null, that eliminates a whole class of errors throughout the program. I do need to test that my parser does what I assume it will, sure. But once I have a type that cannot represent illegal data, I can focus my testing away from playing defense on every function.
Assume that there was room for null. What is your concrete example of where null becomes a problem in production, but goes unnoticed by your tests?
In a world where I am writing a language with null and have half-implemented a grown up type system by checking for null everywhere and writing tests that try to call functions with null (EDIT: and I remembered to do all of that), I guess we could say that I'm at the same place I am right now. But right now I don't have to write defensive tests that include null.
You're asking about a circumstance that's just very very different from the one I'm in.
You're asking about a circumstance that's just very very different from the one I'm in.
> But right now I don't have to write defensive tests that include null.
You seem to misunderstand. The question was centred around the fact that unexpected null cases end up being tested by virtue of you covering normal test cases due to the constraints on execution. Explicitly testing for null is something else. Something else I suggest unnecessary — at least where null is not actually part of the contract, which for the purposes of our discussion is the case. Again, we're specifically talking about the testing that is necessary to the extent of covering what is missing in the type system when you don't have a formal proof language in hand.
> You're asking about a circumstance that's just very very different from the one I'm in.
But one you've clearly had trouble with in the past given your claims that you weren't able to deal with it. Otherwise, how would you know?
Perhaps I can phrase my request in another way: If null isn't expected in your codebase, and your testing over where the type system is lacking has covered your bases to know that the behaviour is as documented, where are the errant nulls going to magically appear from?
You seem to misunderstand. The question was centred around the fact that unexpected null cases end up being tested by virtue of you covering normal test cases due to the constraints on execution. Explicitly testing for null is something else. Something else I suggest unnecessary — at least where null is not actually part of the contract, which for the purposes of our discussion is the case. Again, we're specifically talking about the testing that is necessary to the extent of covering what is missing in the type system when you don't have a formal proof language in hand.
> You're asking about a circumstance that's just very very different from the one I'm in.
But one you've clearly had trouble with in the past given your claims that you weren't able to deal with it. Otherwise, how would you know?
Perhaps I can phrase my request in another way: If null isn't expected in your codebase, and your testing over where the type system is lacking has covered your bases to know that the behaviour is as documented, where are the errant nulls going to magically appear from?
Oh the errant nulls only ever came up when I was writing dynamically typed languages. Is that what you're asking about?
Not specifically, but if that is where your example comes from, that’s fine. It’s all the same. What have you got?
On a PHP project I ran into cases where I changed the shape of an object (associative array) I was passing around and forgot about one of the places I was using it. Didn’t turn into a production bug but still was the kind of thing I would rather be a squiggly line rather than remembering to rerun all the paths through the code. Didn’t help that we were testing by hand.
Same thing on the front end in JS: change the shape of some record that has more places using it than I could remember. Better tests would have caught these. A compiler would be even better.
FWIW I’ve written a lot of tests of code written in all of the languages I like. You absolutely need tests, you just don’t need them to be as paranoid when writing them.
Same thing on the front end in JS: change the shape of some record that has more places using it than I could remember. Better tests would have caught these. A compiler would be even better.
FWIW I’ve written a lot of tests of code written in all of the languages I like. You absolutely need tests, you just don’t need them to be as paranoid when writing them.
> A compiler would be even better.
Right, but the condition here is that the languages that are expressive enough to negate the need for testing are, shall we say, unusable. In the real world people are going to be using, at best, languages with gimped type systems that still require testing to fill in the gaps.
Given that, we're trying to understand your rejection of the premise that the tests you will write to fill in those gaps will also catch things like null exceptions in the due course of execution. It remains unclear where you think these errant nulls are magically coming from.
I'm not convinced "Didn’t help that we were testing by hand.", "Better tests would have caught these." is that rejection. Those assertions, while no doubt applicable to your circumstances, is kind of like saying that static type systems don't help either because you can write something like this:
Right, but the condition here is that the languages that are expressive enough to negate the need for testing are, shall we say, unusable. In the real world people are going to be using, at best, languages with gimped type systems that still require testing to fill in the gaps.
Given that, we're trying to understand your rejection of the premise that the tests you will write to fill in those gaps will also catch things like null exceptions in the due course of execution. It remains unclear where you think these errant nulls are magically coming from.
I'm not convinced "Didn’t help that we were testing by hand.", "Better tests would have caught these." is that rejection. Those assertions, while no doubt applicable to your circumstances, is kind of like saying that static type systems don't help either because you can write something like this:
data MaybeString = Null | Str String
len :: MaybeString -> Int
len (Str s) = length s
main :: IO ()
main = do
print (len Null) -- exception thrown
In the example you gave you have an incomplete implementation of len. We had either a language extension or a compiler flag to disallow incomplete implementations in Haskell (pretty sure it's the flag -Werror), and Elm has no way of allowing them in the first place. I should have specified that that was the case, because "Haskell" is a rather broad term since you can turn on/off language extensions on a per file basis as well as at the project level.
To head off (hah) discussion of taking the head of [], we used a prelude where head returned a Maybe. As far as I know, there were no incomplete functions in the prelude. https://hackage.haskell.org/package/nri-prelude
To head off (hah) discussion of taking the head of [], we used a prelude where head returned a Maybe. As far as I know, there were no incomplete functions in the prelude. https://hackage.haskell.org/package/nri-prelude
> We had either a language extension or a compiler flag to disallow incomplete implementations in Haskell
"Better flag choices would have caught it" is a poor take given what came before. Of course that's true, but the same as your "better tests would have caught it". However, that really has nothing to do with our discussion.
Again, the premise here is that you are writing tests to close the gaps where the type system is insufficient to describe the full program. Of course you are as the astute observation of "My hair is too shiny and my muscles are too large to have to deal with [error prone things] everywhere. Can't do it." is true and valid. Null checks alone, or even all type checks alone, are not sufficient to satisfy all cases of [error prone things]. That is at least outside of formal proof languages, but we already established that we aren't talking about those.
So... Given the tests you are writing to fill in those gaps (again, not tests specifically looking for null pointer cases, but the normal tests you are writing), how would null pointer cases slip through, even if the compiler didn't notice? What is the concrete example that demonstrates how that is possible?
Because frankly I have no idea how it could be possible and I am starting to think your rejection was just made up on the spot and thrown out there without you giving any thought, or perhaps reiterating some made up nonsense you read elsewhere without considering if it were valid? It is seemingly telling that every attempt to dismiss that idea has devolved into bizarre statements along the lines of "well, if you don't write tests then null pointer exceptions might make it into production" even though it is clear that's not what we are talking about.
"Better flag choices would have caught it" is a poor take given what came before. Of course that's true, but the same as your "better tests would have caught it". However, that really has nothing to do with our discussion.
Again, the premise here is that you are writing tests to close the gaps where the type system is insufficient to describe the full program. Of course you are as the astute observation of "My hair is too shiny and my muscles are too large to have to deal with [error prone things] everywhere. Can't do it." is true and valid. Null checks alone, or even all type checks alone, are not sufficient to satisfy all cases of [error prone things]. That is at least outside of formal proof languages, but we already established that we aren't talking about those.
So... Given the tests you are writing to fill in those gaps (again, not tests specifically looking for null pointer cases, but the normal tests you are writing), how would null pointer cases slip through, even if the compiler didn't notice? What is the concrete example that demonstrates how that is possible?
Because frankly I have no idea how it could be possible and I am starting to think your rejection was just made up on the spot and thrown out there without you giving any thought, or perhaps reiterating some made up nonsense you read elsewhere without considering if it were valid? It is seemingly telling that every attempt to dismiss that idea has devolved into bizarre statements along the lines of "well, if you don't write tests then null pointer exceptions might make it into production" even though it is clear that's not what we are talking about.
How many people actually write exhaustive tests for everything that could possibly be null? No one I've ever met in my mostly-C# career.
I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions. I insist on writing new C# code with null-checking enabled which mostly solves the problem, but there's still plenty of code in the wild crashing on null bugs all the time.
I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions. I insist on writing new C# code with null-checking enabled which mostly solves the problem, but there's still plenty of code in the wild crashing on null bugs all the time.
> How many people actually write exhaustive tests for everything that could possibly be null?
Of those who are concerned about type theory? 99%. With a delusional 1% thinking that a gimped type system (read: insufficient for formal proofs) is some kind of magic that negates the need to write tests, somehow not noticing that many of the lessons on how to write good tests come from those language ecosystems (e.g. Haskell).
> I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions.
I don't think I've ever seen a null exception (or closest language analog) occur in production, and I spent a lot of years involved in projects that used dynamically typed languages even. I'd still love for someone to show actual code and associated tests to see how they ended up in that situation. The other commenter, despite being adamant, has become avoidant when it comes down to it.
Of those who are concerned about type theory? 99%. With a delusional 1% thinking that a gimped type system (read: insufficient for formal proofs) is some kind of magic that negates the need to write tests, somehow not noticing that many of the lessons on how to write good tests come from those language ecosystems (e.g. Haskell).
> I can confirm that at least 30% of the prod alerts I've seen come from NullReferenceExceptions.
I don't think I've ever seen a null exception (or closest language analog) occur in production, and I spent a lot of years involved in projects that used dynamically typed languages even. I'd still love for someone to show actual code and associated tests to see how they ended up in that situation. The other commenter, despite being adamant, has become avoidant when it comes down to it.
> to see how they ended up in that situation
The "how" is almost always lack of discipline (or as I sometimes couch it, "imagination") but usually shit like https://github.com/microsoft/SynapseML/issues/405#:~:text=cl...
I've had to learn in my career not to open other people's projects in a real IDE because of all the screaming it does about "value can be null"
The "how" is almost always lack of discipline (or as I sometimes couch it, "imagination") but usually shit like https://github.com/microsoft/SynapseML/issues/405#:~:text=cl...
I've had to learn in my career not to open other people's projects in a real IDE because of all the screaming it does about "value can be null"
That's a tough bouncing ball to follow as it appears that the resolution was to upgrade to a newer version of a dependency, but if we look at that dependency, the fix seems to be found somewhere in this https://github.com/microsoft/LightGBM/compare/v2.2.1...v2.2....
There is admittedly a lot in the update and I may have simply missed it, but I don't see any modifications, even additions, to tests to denote recognition of the problem in the earlier version. Which, while not knowing much about the project, makes me think that there really isn't any meaningful testing going on. That maybe be interesting for what it is, I suppose, but not really in the vein of the discussion here about where one is using type systems and testing to overcome their personal limitations.
There is admittedly a lot in the update and I may have simply missed it, but I don't see any modifications, even additions, to tests to denote recognition of the problem in the earlier version. Which, while not knowing much about the project, makes me think that there really isn't any meaningful testing going on. That maybe be interesting for what it is, I suppose, but not really in the vein of the discussion here about where one is using type systems and testing to overcome their personal limitations.
> in a real IDE because of all the screaming it does about "value can be null"
Yeah, when you have extra tools like that it can certainly help. The thing is that you can ignore any warning! I like it to be a compiler error because then there's no way to put that on the tech-debt credit card and still pass CI. If you are able to put those warnings into your CI so a PR with them cannot merge, then that's like 99% of what I like in my code: make the computer think of the low-hanging-fruit things that can go wrong.
With all of that said, solving for null does not get you everything a tagged union type does, but that's a different story.
Yeah, when you have extra tools like that it can certainly help. The thing is that you can ignore any warning! I like it to be a compiler error because then there's no way to put that on the tech-debt credit card and still pass CI. If you are able to put those warnings into your CI so a PR with them cannot merge, then that's like 99% of what I like in my code: make the computer think of the low-hanging-fruit things that can go wrong.
With all of that said, solving for null does not get you everything a tagged union type does, but that's a different story.
Relevant: Google Cloud Incident Report – 2025-06-13 - https://news.ycombinator.com/item?id=44274563 - June, 2025 (220 comments)
Not exactly as, in that case, they chose not to test the error condition for correct behaviour at all. The problem wasn't just the null pointer condition, but also that the coded behaviour under that state was just plain wrong from top to bottom.
More careful use of the type system might have caught the null pointer case specifically, but the compiler still wouldn't have caught that they were doing the wrong thing beyond the null pointer error. In other words, the failure would have still occurred due to the next problem down the line from a problem that is impossible to catch with a partial type system.
While a developer full of hubris who thinks they can do no wrong may let that slide, our discussion is specifically about a developer who fully understands his personal limitations. He recognizes that he needs the machine to hold his hand. While that includes leveraging types, he understands that the type system in any language he will use in the real world isn't expressive enough to cover all of the conditions necessary. Thus he will also write tests to fill in the gaps.
Now, the premise proposed was that once you write the tests to cover that behaviour in order to fill in the gaps where the type system isn't expressive enough, you naturally also ensure that you don't end up with things like null pointer cases by way of the constraints of test execution. The parent rejected that notion, but it remains unclear why, and we don't yet have a concrete example showing how errant nulls "magically" appear under those conditions.
"I don't need testing" isn't the same thing, and has nothing to do with the discussion taking place here.
More careful use of the type system might have caught the null pointer case specifically, but the compiler still wouldn't have caught that they were doing the wrong thing beyond the null pointer error. In other words, the failure would have still occurred due to the next problem down the line from a problem that is impossible to catch with a partial type system.
While a developer full of hubris who thinks they can do no wrong may let that slide, our discussion is specifically about a developer who fully understands his personal limitations. He recognizes that he needs the machine to hold his hand. While that includes leveraging types, he understands that the type system in any language he will use in the real world isn't expressive enough to cover all of the conditions necessary. Thus he will also write tests to fill in the gaps.
Now, the premise proposed was that once you write the tests to cover that behaviour in order to fill in the gaps where the type system isn't expressive enough, you naturally also ensure that you don't end up with things like null pointer cases by way of the constraints of test execution. The parent rejected that notion, but it remains unclear why, and we don't yet have a concrete example showing how errant nulls "magically" appear under those conditions.
"I don't need testing" isn't the same thing, and has nothing to do with the discussion taking place here.
This has by far the best discussion of the visitor pattern I've yet to come across.
I went to look for that bit. It said:
"Bad"
lol
"Bad"
lol
I don't work in typical OO codebases, so I wasn't aware of what the visitor pattern even is. But there's an _excellent_ book about building an interpreter (and vm) "crafting interpreters". It has a section where it uses the visitor pattern.
https://craftinginterpreters.com/representing-code.html#the-...
I remember reading through it and not understanding why it had to be this complicated and then just used a tagged union instead.
Maybe I'm too stupid for OO. But I think that's kind of the point of the grug article as well. Why burden ourselves with indirection and complexity when there's a more straight forward way?
https://craftinginterpreters.com/representing-code.html#the-...
I remember reading through it and not understanding why it had to be this complicated and then just used a tagged union instead.
Maybe I'm too stupid for OO. But I think that's kind of the point of the grug article as well. Why burden ourselves with indirection and complexity when there's a more straight forward way?
It's an engineering tradeoff.
https://prog2.de/book/sec-java-expr-problem.html - Not the writeup I was looking for but seems to cover it well.
> Why burden ourselves with indirection and complexity when there's a more straight forward way?
Because each way has its own tradeoffs that make it more or less difficult to use in particular circumstances.
https://homepages.inf.ed.ac.uk/wadler/papers/expression/expr... - Wadler's description of the expression problem.
https://prog2.de/book/sec-java-expr-problem.html - Not the writeup I was looking for but seems to cover it well.
> Why burden ourselves with indirection and complexity when there's a more straight forward way?
Because each way has its own tradeoffs that make it more or less difficult to use in particular circumstances.
https://homepages.inf.ed.ac.uk/wadler/papers/expression/expr... - Wadler's description of the expression problem.
Thank you for those links. The first one is especially clear.
However, this is just not something that I typically perceive as a problem. For example in the book that I mentioned above, I didn't feel the need to use it at all. I just added the fields or the functions that were required.
In the first link you provided, the OCaml code seems to use unions as well (I don't know the language). I assume OCaml checks for exhaustive matching, so it seems extremely straight forward to extend this code.
On the other hand I have absolutely no issues with a big switch case in a more simple language. I just had a look at the code I wrote quite a while ago and it looks fine.
However, this is just not something that I typically perceive as a problem. For example in the book that I mentioned above, I didn't feel the need to use it at all. I just added the fields or the functions that were required.
In the first link you provided, the OCaml code seems to use unions as well (I don't know the language). I assume OCaml checks for exhaustive matching, so it seems extremely straight forward to extend this code.
On the other hand I have absolutely no issues with a big switch case in a more simple language. I just had a look at the code I wrote quite a while ago and it looks fine.
I love crafting interpreters and mention it on grugbrain:
https://grugbrain.dev/#grug-on-parsing
but the visitor pattern is nearly always a bad idea IMO: you should just encode the operation in the tree if you control it or create a recursive function that manually dispatches on the argument type if you don't
https://grugbrain.dev/#grug-on-parsing
but the visitor pattern is nearly always a bad idea IMO: you should just encode the operation in the tree if you control it or create a recursive function that manually dispatches on the argument type if you don't
An implementor of a data structure might take precautions for users of the data structure to perform such visiting operations by passing in a visitor-like thing.
I just don't think it's a significantly better way of dealing w/the problem than a recursive function that dispatches on the arg type (or whatever) using an if statement or pattern matching or whatever.
The additional complexity doesn't add significant value IMO. I admit that's a subjective claim.
The additional complexity doesn't add significant value IMO. I admit that's a subjective claim.
I mean, at some point you can also make that recursive function take an argument, that decides what to do depending on the type of the item, to make that recursive function reusable, if one has multiple use-cases ... but that's basically the same as the visitor pattern. There really isn't much to it, other than programming language limitations, that necessitate a special implementation, "making it a thing". Like when Java didn't have lambdas and one needed to make visitors objects, ergo had to write a class for them.
As far as I understand it, the limited circumstances when you absolutely need the visitor pattern are when you have type erasure, i.e., can't use a tagged union or its equivalent? In that case visitors are AIUI a very clever trick to use vtables or whatever to get back to your concrete types! but ... clever tricks make grug angry.
even when you have tagged unions, visitors are a useful way to abstract a heterogenous tree traversal from code that processes specific nodes in the tree. e.g. if you have an ast with an `if` node and subnodes `condition`, `if_body`, and `else_body` you could either have the `if node == "if" then call f(subnode) for subnode in [node.condition, node.if_body, node.else_body]` and repeat that for every function `f` that walks the tree, or define a visitor that takes `f` as an argument and keep the knowledge of which subnodes every node has in a single place.
What do you mean by tagged union? And how does it make the visitor pattern not needed?
See https://en.wikipedia.org/wiki/Tagged_union
In languages influenced by ML (like contemporary Java!) it is common in compiler work in that you might have an AST or similar kind of structure and you end up writing a lot of functions that use pattern matching like
In languages influenced by ML (like contemporary Java!) it is common in compiler work in that you might have an AST or similar kind of structure and you end up writing a lot of functions that use pattern matching like
switch(node) {
type1(a,b) -> whatever(a,b)
type2(c) -> process(c)
}
OK yeah I see, that's natural to do with like rust enums
Java doesn't support this though I thought?
Java doesn't support this though I thought?
Currently Java supports records (finalized in JDK 16) and sealed classes (JDK 17) which together work as algebraic data types; pattern matching in switch was finalized in JDK 21. The syntax is pretty sweet
https://blog.scottlogic.com/2025/01/20/algebraic-data-types-...
https://blog.scottlogic.com/2025/01/20/algebraic-data-types-...
oh thats cool. thanks for sharing!
> that's natural to do with like rust enums
Stands to reason. Rust "enums" are tagged unions (a.k.a. sum types, discriminated unions).
In implementation, the tag, unless otherwise specified, is produced by an enum, which I guess is why it got that somewhat confusing keyword.
Stands to reason. Rust "enums" are tagged unions (a.k.a. sum types, discriminated unions).
In implementation, the tag, unless otherwise specified, is produced by an enum, which I guess is why it got that somewhat confusing keyword.
You're right. It's an example of changing best practices to fit whatever the language designers released.
If you work in a language that can pass closures as arguments, then you don't need a special visitor pattern. It is one of those patterns, that exists because of limitations of languages. Or, if you want to call passing a closure visitor pattern in some cases, it becomes so natural, that it does not deserve special mention as something out of the ordinary. You may be too smart for it.
I care about naming, and I find the name of the visitor pattern infuriatingly bad. Very clubbable. I think I have never created one called "Visitor" in my life.
Given the syntax tree example from Wikipedia, I think I'd call it AstWalker, AstItem::dispatch(AstWalker) and AstWalker::process(AstItem) instead of Visitor, AstItem::accept(AstVisitor) and AstVisitor::visit(AstItem).
"The walker walks the AST, each items sends it to the next ones, and the walker processes them". That means something. "The visitor visits the AST items, which accept it" means basically nothing. It's more general, but also contains very little useful information. So the visitor might need different names in different situations. Fine. Just add a comment "visitor pattern" for recognizability.
I remember a situation where I needed to walk two object trees for a data comparison and import operation. I created an AbstractImporter that walked the two trees in lockstep in a guaranteed order and invoked virtual methods for each difference. It had a non-virtual doImport() for the ordered data walk, and doImport() called virtual methods like importUserAccount(), importUserAccountGrouMemberships() etc. There were two subclasses of AbstractImporter: ImportAnalyzer collected differences to display them, then there was a selection step implemented by a kind of list model + a bit of controller logic, then an ImportWorker to make the selected changes. All rather specific terminology and not exactly the visitor pattern.
Given the syntax tree example from Wikipedia, I think I'd call it AstWalker, AstItem::dispatch(AstWalker) and AstWalker::process(AstItem) instead of Visitor, AstItem::accept(AstVisitor) and AstVisitor::visit(AstItem).
"The walker walks the AST, each items sends it to the next ones, and the walker processes them". That means something. "The visitor visits the AST items, which accept it" means basically nothing. It's more general, but also contains very little useful information. So the visitor might need different names in different situations. Fine. Just add a comment "visitor pattern" for recognizability.
I remember a situation where I needed to walk two object trees for a data comparison and import operation. I created an AbstractImporter that walked the two trees in lockstep in a guaranteed order and invoked virtual methods for each difference. It had a non-virtual doImport() for the ordered data walk, and doImport() called virtual methods like importUserAccount(), importUserAccountGrouMemberships() etc. There were two subclasses of AbstractImporter: ImportAnalyzer collected differences to display them, then there was a selection step implemented by a kind of list model + a bit of controller logic, then an ImportWorker to make the selected changes. All rather specific terminology and not exactly the visitor pattern.
But the idea of the visitor pattern is not, that it itself walks a tree. The idea is, that it doesn't know how to walk the tree and will be passed in elements from the tree, to its visit method. It does only need to know what to do with one element at a time. The walking is implemented elsewhere.
One may pretend that it's not the case, but in practice, the visitor/walker does traverse the tree in a particular, systematic order. Walker implies a somewhat systematic traversal (I think), which is what it does. As the implementor of its visit() methods, it doesn't even matter whether the generic Visitor class chooses the path or the nodes do. Visitor is also super vague. Does it just say "Hi" to the whole tree and then leave? Does it only visit the museums and ignore the parks?
From the creator of HTMX.
Massive grug.
grug guy THINK big brain and make up hypermedia api and so complicated words like revealed religion but only in his imagination.
more important app is like how programmer intended. web only for docs like unix files anyway no good to think too much.
more important app is like how programmer intended. web only for docs like unix files anyway no good to think too much.
i've seen things you wouldn't believe...
A classic.
And it’s inspired many more grugs like grug ceo https://www.sam-rodriques.com/post/the-grugbrained-ceo
And it’s inspired many more grugs like grug ceo https://www.sam-rodriques.com/post/the-grugbrained-ceo
It took me decades to learn these lessons on my own.
many, many shiney rock lost to agile shaman!
many, many shiney rock lost to agile shaman!
I avoided this page since it was a terrible programmer at work who shared it to me a couple years back. The title and the writing style made me think of him in new depths. Perhaps there’s something here for it to trend in HN, but I’m sure it resonates to all skill levels of programmers in wildly different ways—the comments are only an illusion of consensus and wisdom.
The distain for shamans in palpable and obviously born out of similar work experience. *shudder*
Shamans suck for the same reason architects suck -- take "getting stuff done" out of someone's responsibilities, and they lose the plot pretty quickly.
Shamans suck for the same reason architects suck -- take "getting stuff done" out of someone's responsibilities, and they lose the plot pretty quickly.
I used to be against complexity and worried about such narratives making fun of people who tried to avoid it but now I'm grateful. If software developers didn't have such strong biases in favor of complexity, LLMs would probably be producing really high quality code and have replaced us all by now... Instead, because the average code online is over-engineered, un-reusable junk, their training set is a mess and hence they can only produce overengineered junk code. Also, this may provide long term job safety since now LLMs are producing more and more code online, further soiling the training set.
while it's #1 I might as well say, there's a book:
https://www.lulu.com/shop/carson-gross/the-grug-brained-deve...
it's the same content + an index, so not worth buying unless you want the hard copy, but maybe a good convo-starter for the ol'dev team
https://www.lulu.com/shop/carson-gross/the-grug-brained-deve...
it's the same content + an index, so not worth buying unless you want the hard copy, but maybe a good convo-starter for the ol'dev team
Probably my single favourite programming article.
Very entertaining (and enlightening) read. I love the 'reach for club' visuals, made me laugh out loud a few times.
This was shared with me years ago by another developer I worked with. I still reference it today as I continue my external battle with the complexity demon.
trap in crystal
OMG is that the technical name for my development style? I'm not like super deep in technobabble since there are so many coined names and references that it is nearly impossible to assign the correct one.
Grug brained dev I am I guess.
Grug brained dev I am I guess.
Grug user want keyword research tool.
Grug user find program that does that and more.
Grug user confused by menu.
Grug user wish tool only did keyword research.
Grug user find program that does that and more.
Grug user confused by menu.
Grug user wish tool only did keyword research.
grug read words and move head up down lot
grug make other work grugs read this after yellow circle arrive next
grug thank clever grug
grug make other work grugs read this after yellow circle arrive next
grug thank clever grug
this very cool
I smell a formal grammar behind dumbiffied grug english.
nonetheless, I think that when it says:
> so grug say again and say often: complexity very, very bad
at the end of that section, it shoulud say instead:
> so grug say again and say often: complexity very, very, very bad
this disambiguates 3 instances of same concept/idea AND, even better, showcases 3 values of increasing strength like for warning, error, critical use. most compact.
end of groog
I smell a formal grammar behind dumbiffied grug english.
nonetheless, I think that when it says:
> so grug say again and say often: complexity very, very bad
at the end of that section, it shoulud say instead:
> so grug say again and say often: complexity very, very, very bad
this disambiguates 3 instances of same concept/idea AND, even better, showcases 3 values of increasing strength like for warning, error, critical use. most compact.
end of groog
This will be discussed in the next standup and everyone has to have an opinion. We'll need approval from legal and that takes at least a week so we want to minimise ping pong emails.
But it should be fairly quick, expect an updated version around end of summer or just after.
But it should be fairly quick, expect an updated version around end of summer or just after.
I've been fooling around with applying grugspeak to famous essays on tech.
https://docs.google.com/document/d/1emldq9MovfYshOSkM9rhRUcl...
not as good as the original, i know!
https://docs.google.com/document/d/1emldq9MovfYshOSkM9rhRUcl...
not as good as the original, i know!
I argue that complexity is good. Complicated things though, those are bad.
The world is complex, and we must deal with it. But as long as we organize our work and don't let it spiral into being complicated, we'll be fine.
very hard for most developers (maybe even you) to tell the difference
"be organized" is, of course, good advice
"be organized" is, of course, good advice
I think even the simplest of programming tools is still unfathomably powerful, enough to where you could eliminate data structures and still be where we are today. BitTorrent had a great amount of complexity, and look what streaming did to it.
This is one of my favorite pieces of non-fiction. No sarcasm.
Grug quickly tire of verbal gimmick that go on way too long.
i grant you a full refund
Some solid nuggets here. Totally agree on keeping it simple and not rushing. I’ve rushed things before to meet unrealistic deadlines, resulting in bad first impression. Took a step back, simplified, and let the design emerge. Ended up with something users actually loved. Thanks for sharing.
So this is how a troll from The Witcher 3 would explain software engineering
I've always wondered at the best way of doing integration tests. There is a lot of material on unit tests, but not so much on integration tests. Does anyone know of a good book on the subject?
Share this gem on a team chat, everyone had a good laugh and now everytime we discuss some weird situation, we start speaking as grug brains… Don’t know how I lived without it so far
I send this article as part of onboarding for all new devs we hire. It is super great to keep a fast growing team from falling into the typical cycle of more people, more complexity.
> big brain developer Bob Nystrom
Lies and slander, munificent is quite open about his grugness and his grug-brained approach is why his book is so good.
Lies and slander, munificent is quite open about his grugness and his grug-brained approach is why his book is so good.
> visitor pattern
mmmm
mmmm
This just became my favorite article of all time.
This has to be adapted to the LLM era as well.
In my experience, LLM agents are pretty Grug-brained.
Grug has common sense. LLMs don't even have that.
LLMs actually reinforce grug principles - they work best with simple, consistent patterns and struggle with the same complexity demons that confuse humans.
I read it back then and then forgot the word and couldn't find it with search. LOL. Thanks for reposting!
This is why I like embedded.
The bigger the CPU, the more room the complexity spirit daemon has to thrive.
Wasn’t HTMX just a meme? I can’t really tell if it’s serious because of Poe’s Law.
htmx sucks:
https://htmx.org/essays/htmx-sucks/
https://htmx.org/essays/htmx-sucks/
well, at least he is (you are?) consistent in this style of criticizing others' ideas with satirical sarcasm fueled prose focused on tearing down straw men.
So it is bad?
Definitely for some stuff!
https://htmx.org/essays/when-to-use-hypermedia/
https://htmx.org/essays/#on-the-other-hand
https://htmx.org/essays/when-to-use-hypermedia/
https://htmx.org/essays/#on-the-other-hand
If you read this and concluded that it's bad, then you probably shouldn't use it.
Solopreneur making use of it in my bootstrapped B2B SaaS business. Clients don't need or want anything flashy. There are islands of interactivity, and some HTMX sprinkled there has been a great fit.
Wish I had your clients, instead of ones that say a page needs more “pizazz!”
I’m getting zombo.com vibes from this client request.
you can do anything at Zombocom!
The pizazz clients want sites for their customers, the no-frills clients want sites for them to use themselves.
Using HTMX in my B2B SaaS allowed me to enjoy coding for the web again. I had lost that joy somewhere along the way.
And our customers seem very content with the way our product works.
(For the record: yes, it’s a stable, profitable, long-established product.)
And our customers seem very content with the way our product works.
(For the record: yes, it’s a stable, profitable, long-established product.)
I started using htmx relatively early on, because its a more elegant version of what I've been doing anyways for a series of projects.
It's very effective, simple and expressive to work this way, as long as you keep in mind that some client side rendering is fine.
There are a few bits I don't like about it, like defaulting to swap innerHTML instead of outerHTML, not swapping HTML when the status code isn't 200-299 by default and it has some features that I avoid, like inline JSON on buttons instead of just using forms.
Other than that, it's great. I can also recommend reading the book https://hypermedia.systems/.
It's very effective, simple and expressive to work this way, as long as you keep in mind that some client side rendering is fine.
There are a few bits I don't like about it, like defaulting to swap innerHTML instead of outerHTML, not swapping HTML when the status code isn't 200-299 by default and it has some features that I avoid, like inline JSON on buttons instead of just using forms.
Other than that, it's great. I can also recommend reading the book https://hypermedia.systems/.
might be some good points in here but it's sooo hard to read.
It isn't as skimmable as some other writing styles, but if you read it one word at a time (either aloud or "in your head"), it's not too bad.
Sometimes if I'm reading something and having trouble with the words or sentences, I'll slow down and focus on the individual letters. Usually helps a tremendous amount.
It actually might be a psychological trick to make readers slow down and try to comprehend fully what is written. So, making something hard to read on purpose to get better comprehension
https://reidjs.github.io/grug-dev-translation/ :)
(no affiliation, I enjoy the original and wish for it to reach as many people as possible)
(no affiliation, I enjoy the original and wish for it to reach as many people as possible)
Good one....I put it into Claude with a prompt to not change the meaning but make it more normal
I like it. Grug is grammatically incorrect but concise, which forces my Big Brain to step back, allowing my Grug Brain to slowly absorb the meaning of each word.
Grug is an obvious misspelling of the name Greg, similar to how HTMX is a misspelling of HTML
Since you're being downvoted I just wanted to say I agree. I'm sure it was cathartic to write but it's not a good way to actually communicate.
Also like a lot of programming advice it isn't actually that useful. Advice like "avoid complexity" sounds like it is good advice, but it isn't good advice. Of course you should avoid complexity. Telling people to do that is about as useful as telling people to "be more confident".
We mostly learn to avoid complexity through trial and error - working on complex and simple systems, seeing the pitfalls, specific techniques to avoid complexity, what specific complexity is bad, etc. Because not all complexity is bad. You want simplicity? Better trade in you Zen 4 and buy a Cortex M0. And I hope you aren't running a modern OS on it.
Ok "avoid unnecessary complexity"? Great how exactly do you know what's unnecessary? Years of experience that's how. Nothing you can distill to a gimmicky essay.
Also like a lot of programming advice it isn't actually that useful. Advice like "avoid complexity" sounds like it is good advice, but it isn't good advice. Of course you should avoid complexity. Telling people to do that is about as useful as telling people to "be more confident".
We mostly learn to avoid complexity through trial and error - working on complex and simple systems, seeing the pitfalls, specific techniques to avoid complexity, what specific complexity is bad, etc. Because not all complexity is bad. You want simplicity? Better trade in you Zen 4 and buy a Cortex M0. And I hope you aren't running a modern OS on it.
Ok "avoid unnecessary complexity"? Great how exactly do you know what's unnecessary? Years of experience that's how. Nothing you can distill to a gimmicky essay.
Yeah that's the point, to communicate the idea that some complexity is unnecessary, and we should beware of it, instead of just accepting wholesale whatever complexity is handed to us, like many in this industry do.
I raise you "premature optimization is the root of all evil". Great advice, so good in fact that it's true for literally anything: "premature X is the root of all evil".
If it's "unnecessary" or "premature", then of course it's bad. I don't need to be told that. What I do need is advice on telling apart the unnecessary and premature from the necessary and timely.
If it's "unnecessary" or "premature", then of course it's bad. I don't need to be told that. What I do need is advice on telling apart the unnecessary and premature from the necessary and timely.
Totally agree! I think that one is actually a net negative because it's mostly used as an excuse not to think about performance at all.
"Oh we don't need to think about performance when deciding the fundamental architecture of our system because everyone knows premature optimization is the root of all evil."
"Oh we don't need to think about performance when deciding the fundamental architecture of our system because everyone knows premature optimization is the root of all evil."
The medium is the message though.
big fan of HTMX (pinnacle of grug)
i have started to use repl debugging … just insert a repl statement anywhere in your raku code and you can interact with the code state
i have started to use repl debugging … just insert a repl statement anywhere in your raku code and you can interact with the code state
Pinnacle of grug is PHP and plain forms. Maybe a validate on submit in JS.
In what way would you say they are related? I've read both but don't see it
Introducing a needless us/them dynamic. Don't do it the way those elites do it, do it like us real grugs.
That's why it's so easy to rush in and agree with everything. In another article, you might have to read 3 points for microservices, and 4 against. But here? Nope.
Factoring? Just something you do with your gut. Know it when ya see it. Grug isn't going to offer up the why or how, because that's something a reader could disagree with.
That's why it's so easy to rush in and agree with everything. In another article, you might have to read 3 points for microservices, and 4 against. But here? Nope.
Factoring? Just something you do with your gut. Know it when ya see it. Grug isn't going to offer up the why or how, because that's something a reader could disagree with.
I don't see it as an us/them, he even says:
> note: grug once think big brained but learn hard way
It's basically a lot of K.I.S.S. advice. Of course you should use the best tool for the job, but complexity often sneaks up on you.
> Factoring? Just something you do with your gut. Know it when ya see it. Grug isn't going to offer up the why or how, because that's something a reader could disagree with.
I would dispute this characterization. Grug says it's difficult, and it is, but gives specific advice, to the extent one can:
- grug try not to factor in early part of project and then, at some point, good cut-points emerge from code base
- good cut point has narrow interface with rest of system: small number of functions or abstractions
- grug try watch patiently as cut points emerge from code and slowly refactor
- working demo especially good trick: force big brain make something to actually work to talk about and code to look at that do thing, will help big brain see reality on ground more quickly
> note: grug once think big brained but learn hard way
It's basically a lot of K.I.S.S. advice. Of course you should use the best tool for the job, but complexity often sneaks up on you.
> Factoring? Just something you do with your gut. Know it when ya see it. Grug isn't going to offer up the why or how, because that's something a reader could disagree with.
I would dispute this characterization. Grug says it's difficult, and it is, but gives specific advice, to the extent one can:
- grug try not to factor in early part of project and then, at some point, good cut-points emerge from code base
- good cut point has narrow interface with rest of system: small number of functions or abstractions
- grug try watch patiently as cut points emerge from code and slowly refactor
- working demo especially good trick: force big brain make something to actually work to talk about and code to look at that do thing, will help big brain see reality on ground more quickly
i'm a professor
I still call coffee "black think juice"
I am grug, and I endorse this message.
Oh FFS why is print-vs-debug being debated as either/or? Both can be valuable, depending on the circumstances. It's clearly both/and.
it's not presented as either-or in the article: i talk about how debuggers AND logging are both important tools
The guy that wrote this is a raging schizophrenic that argues with his own alt on X
complete lunatic, hate him!
IKR. He's at it again
https://x.com/intercoolerjs/status/1935190940903424241?t=WuM...
https://x.com/intercoolerjs/status/1935190940903424241?t=WuM...
this is a masterpiece of writing -- well done
impressive :)
htmx.
and clojure.
mmmmm.
and clojure.
mmmmm.
Content 1, Style 0
Thinking you are too smart leads to all sorts of trouble, like using C++ and being proud of it.
If you think your intelligence is a limited resource however you'll conserve it and not waste it on tools, process and the wrong sort of design.
Thinking you are too smart leads to all sorts of trouble, like using C++ and being proud of it.
If you think your intelligence is a limited resource however you'll conserve it and not waste it on tools, process and the wrong sort of design.
It would be really embarrassing to use one of the most popular, time-tested languages.
Even if we decided to use Zig for everything, hiring for less popular languages like Zig, lua, or Rust is significantly harder. There are no developers with 20 years experience in Zig
Even if we decided to use Zig for everything, hiring for less popular languages like Zig, lua, or Rust is significantly harder. There are no developers with 20 years experience in Zig
You don't need developers with 20 years of experience in a specific language.
Any decent engineer must be able to work with other languages and tools. What you're looking for is someone with experience building systems in your area of expertise.
And even then, experience is often a poor substitute for competence.
Any decent engineer must be able to work with other languages and tools. What you're looking for is someone with experience building systems in your area of expertise.
And even then, experience is often a poor substitute for competence.
> You don't need developers with 20 years of experience in a specific language.
You may in trivia quiz languages that have more features than anyone can learn in a lifetime
You may in trivia quiz languages that have more features than anyone can learn in a lifetime
I assume you're referring to C++ with all of its... features.
And yeah, its a mess. Even when you think you have a good understanding of at least one part of the language, you'll stumble across some small section of cppreference that makes you question your entire career.
And yeah, its a mess. Even when you think you have a good understanding of at least one part of the language, you'll stumble across some small section of cppreference that makes you question your entire career.
Being at a firm where the decision to use C++ was made, the thought process went something like this:
"We're going to need to fit parts of this into very constrained architectures."
"Right, so we need a language that compiles directly to machine code with no runtime interpretation."
"Which one should we use?"
"What about Rust?"
"I know zero Rust developers."
"What about C++?"
"I know twenty C++ developers and am confident we can hire three of them tomorrow."
The calculus at the corporate level really isn't more complicated than that. And the thing about twenty C++ developers is that they're very good at using the tools to stamp the undefined behavior out of the system because they've been doing it their entire careers.
"We're going to need to fit parts of this into very constrained architectures."
"Right, so we need a language that compiles directly to machine code with no runtime interpretation."
"Which one should we use?"
"What about Rust?"
"I know zero Rust developers."
"What about C++?"
"I know twenty C++ developers and am confident we can hire three of them tomorrow."
The calculus at the corporate level really isn't more complicated than that. And the thing about twenty C++ developers is that they're very good at using the tools to stamp the undefined behavior out of the system because they've been doing it their entire careers.
How does someone know twenty C++ developers and zero C developers though?
Born in the 80s.
That wouldn't stop someone from knowing any C developers. It's still a common language today, and was more common when those 80s kids would have become adults and entered the industry.
Maybe they use only Microsoft Windows?
As a kid in the 1980s I thought something was a bit off about K&R, kind of a discontinuity. Notably C succeeded where PL/I failed but by 1990 or so you started to see actual specs written by adults such as Common Lisp and Java where you really can start at the beginning and work to the end and not have to skip forward or read the spec twice. That discontinuity is structural though to C and also C++ and you find it in most books about C++ and in little weird anomalies like the way typedefs force the parser to have access to the symbol table.
Sure C was a huge advance in portability but C and C++ represent a transitional form between an age where you could cleanly spec a special purpose language like COBOL or FORTRAN but not quite spec a general systems programming language and one in which you could. C++, thus, piles a huge amount of complexity on top of a foundation which is almost but not quite right.
Sure C was a huge advance in portability but C and C++ represent a transitional form between an age where you could cleanly spec a special purpose language like COBOL or FORTRAN but not quite spec a general systems programming language and one in which you could. C++, thus, piles a huge amount of complexity on top of a foundation which is almost but not quite right.
Probably they know C but the project is complex enough to warrant something else. Personally I'd rather C++ not exist and it's just C and Rust, but I don't have a magic wand.
A lot of fintech. Bloomberg is real into C++.
People sometimes forget we're not just trying to use the shiniest tool for fun, we're trying to build something with deadlines that must be profitable. If you want to hire for large teams or do hard things that require software support, you often have to use a popular language like C++.
And none of those 20 C++ developers can learn rust? What’s wrong with them?
They can, but why pay 20 people to learn Rust?
Personally I think Rust is better thought out than C++ but that I've got better things to do than fight with the borrow checker and I appreciate that the garbage collector in Java can handle complexity so I don't have to.
I think it's still little appreciated how revolutionary garbage collection is. You don't have maven or cargo for C because you can't really smack together arbitrary C libraries together unless the libraries have an impoverished API when it comes to memory management. In general if you care about performance you would want to pass a library a buffer from the application in some cases, or you might want to pass the library custom malloc and free functions. If your API is not impoverished the library can never really know if the application is done with the buffer and the application can't know if the library is done. But the garbage collector knows!
It is painful to see Rustifarians pushing bubbles around under the rug when the real message the borrow checker is trying to tell them is that their application has a garbage-collector shaped hole in it. "RC all the things" is an answer to reuse but if you are going to do that why not just "GC all the things?" There's nothing more painful than watching people struggle with async Rust because async is all about going off the stack and onto the heap and once you do that you go from a borrowing model that is simple and correct to one that is fraught and structurally unstable -- but people are so proud of their ability to fight complexity they can't see it.
I think it's still little appreciated how revolutionary garbage collection is. You don't have maven or cargo for C because you can't really smack together arbitrary C libraries together unless the libraries have an impoverished API when it comes to memory management. In general if you care about performance you would want to pass a library a buffer from the application in some cases, or you might want to pass the library custom malloc and free functions. If your API is not impoverished the library can never really know if the application is done with the buffer and the application can't know if the library is done. But the garbage collector knows!
It is painful to see Rustifarians pushing bubbles around under the rug when the real message the borrow checker is trying to tell them is that their application has a garbage-collector shaped hole in it. "RC all the things" is an answer to reuse but if you are going to do that why not just "GC all the things?" There's nothing more painful than watching people struggle with async Rust because async is all about going off the stack and onto the heap and once you do that you go from a borrowing model that is simple and correct to one that is fraught and structurally unstable -- but people are so proud of their ability to fight complexity they can't see it.
In our case, a garbage collector is a non-starter because it can't make enough guarantees about either constraining space or time to make the suits happy (embedded architecture with some pretty strict constraints on memory and time to execute for safety).
I do think that there are a lot of circumstances a garbage collector is the right answer where people, for whatever reason, decide they want to manage memory themselves instead.
I do think that there are a lot of circumstances a garbage collector is the right answer where people, for whatever reason, decide they want to manage memory themselves instead.
Usually the main message is that they haven't thought about the organisation of their datastructures sufficiently. In my experience this is also very important with a GC, but you don't get much help from the compiler, instead you wind up chasing strange issues at runtime because object lifetimes don't match what you expected (not to mention, object lifetime is more than just memory allocation, and GC languages tend to have less effective means of managing lifetimes tightly). I agree async Rust is very painful still. It's very much something I don't use unless I have to, and when I do I keep things very simple.
(Also, the lack of a package manager for C is much more due to historical cruft around build systems than it is difficulty getting C libraries to work together. Most of them can interoperate perfectly well, though there is a lot more faff with memory book-keeping)
(Also, the lack of a package manager for C is much more due to historical cruft around build systems than it is difficulty getting C libraries to work together. Most of them can interoperate perfectly well, though there is a lot more faff with memory book-keeping)
Flip the question around: what is the benefit when they already know C++? Most of the safety promises one could make with Rust they can already give through proper application of sanitizers and tooling. At least they believe they can, and management believes them. Grug not ask too many questions when the working prototype is already sitting on Grug's desk because someone hacked it together last night instead of spending that time learning a new language.
I suspect that in a generation or so Rust will probably be where C++ is now: the language business uses because they can quickly find 20 developers who have a career in it.
I suspect that in a generation or so Rust will probably be where C++ is now: the language business uses because they can quickly find 20 developers who have a career in it.
They're probably busy writing code for a living.
The correct answer was almost surely lua + native modules for hotspots. I'm not surprised they couldn't see that though.
There's a layer that you could turn your head and squint and call Lua, but it's far more constrained than Lua is.
They never wanted to be in a situation in the embedded architecture where performance was dependent upon GC pauses (even incremental GC pauses). Their higher-level abstraction has tightly constrained lifecycles and is amenable to static analysis of maximum memory consumption in a way Lua is not.
They never wanted to be in a situation in the embedded architecture where performance was dependent upon GC pauses (even incremental GC pauses). Their higher-level abstraction has tightly constrained lifecycles and is amenable to static analysis of maximum memory consumption in a way Lua is not.
C++ called and filed a complaint about receiving a haymaker of a suckerpunch out of nowhere.
Honestly, burn.
[deleted]
[deleted]
“grug once again catch grug slowly reaching for club, but grug stay calm”
Relateable.
Relateable.
cjcenizal(1)
Coding too hard, this grug use AI.
[deleted]
I had Claude rewrite this in readable English and add links to other perspectives: https://gist.github.com/hbmartin/c169c55d3cffeed0ca4a66f0f2f...
The original reads easier in the voice of the cookie monster:
https://www.youtube.com/watch?v=shbgRyColvE
https://www.youtube.com/watch?v=shbgRyColvE
I've seen your work in Hotwire in my role as a Staff Ruby on Rails Engineer. It's the coolest thing to see you pop up in Hacker News every now and then and also see you talking with the Hotwire devs in GitHub.
Thanks for being a light in the programming community. You're greatly respected and appreciated.