Vibe Debugging: Enterprises' Up and Coming Nightmare(marketsaintefficient.substack.com)
marketsaintefficient.substack.com
Vibe Debugging: Enterprises' Up and Coming Nightmare
https://marketsaintefficient.substack.com/p/vibe-debugging-enterprises-up-and
68 comments
The proof for TDD is usually looking at bug detection rates. Similar for code review. OTOH, the "design damage" of TDD is something overlooked by those metrics.
What it boils down to: - TDD in the hands of a junior is very good. Drastically reduces bugs, and teaches the junior how to write code that can be tested and is not just a big long single method of spaghetti with every data structure represented as another dimension on some array.
- TDD in the hands of a midlevel can be a mixed bag. They've learned how to do TDD well, but have not learned when and why TDD can go bad. This creates design damage, where everything is shoe-horned into TDD and the goal of 90% line coverage is a real consideration. This is maximum correctness but also potentially maximum design damage.
- TDD in the hands of a senior is a power tool. The "right" tests are written for the right reasons with the right level of coupling and the tests overall are useful. Every really complicated algorithm I've had to write, TDD was a life saver for getting it landed.
Feels a lot like asking someone if they prefer X or Y and they say "X" is the industry best practice. My response universally is now an eye brow raise "oh, is it? For which segments of the industry? Why? How do we know it's actually a best practice? Okay, given our context, why would it be a best practice for US". Juniors don't know the best practices, mid-levels apply them everywhere, seniors evaluate and consider when best practices are not best practices.
TDD slows development when tests are written in a blind way with an eye on code coverage and not correctness and design. TDD speeds up development in being a good way to catch errors and is one of the best ways to ensure correctness.
What it boils down to: - TDD in the hands of a junior is very good. Drastically reduces bugs, and teaches the junior how to write code that can be tested and is not just a big long single method of spaghetti with every data structure represented as another dimension on some array.
- TDD in the hands of a midlevel can be a mixed bag. They've learned how to do TDD well, but have not learned when and why TDD can go bad. This creates design damage, where everything is shoe-horned into TDD and the goal of 90% line coverage is a real consideration. This is maximum correctness but also potentially maximum design damage.
- TDD in the hands of a senior is a power tool. The "right" tests are written for the right reasons with the right level of coupling and the tests overall are useful. Every really complicated algorithm I've had to write, TDD was a life saver for getting it landed.
Feels a lot like asking someone if they prefer X or Y and they say "X" is the industry best practice. My response universally is now an eye brow raise "oh, is it? For which segments of the industry? Why? How do we know it's actually a best practice? Okay, given our context, why would it be a best practice for US". Juniors don't know the best practices, mid-levels apply them everywhere, seniors evaluate and consider when best practices are not best practices.
TDD slows development when tests are written in a blind way with an eye on code coverage and not correctness and design. TDD speeds up development in being a good way to catch errors and is one of the best ways to ensure correctness.
Developers have selective amnesia and only count dev time when working on what they want to work on rather than including time spent fixing things they’ve already mentally marked as done.
The worst actors find ways to make other people responsible for fixing their bugs.
The worst actors find ways to make other people responsible for fixing their bugs.
Your comment doesn’t distinguish between having a robust automated test suite and doing TDD.
I’ll take your comment as testing is good and constraining your workflow to TDD is worthless.
I’ll take your comment as testing is good and constraining your workflow to TDD is worthless.
TDD is really commonly misunderstood to be a testing strategy that helps reliability- it's not. It's supposed to guide your software design
But that’s just it-as a design aid it can really go off the rails, but as a testing strategy it’s really useful in one domain. Defect fixing. If I can convince a junior engineer that when he gets a bug report to first write a test that shows the problem and then fix it, using the test to prove it’s fixed, it provides immense benefits.
> If I can convince a junior engineer that when he gets a bug report to first write a test that shows the problem and then fix it, using the test to prove it’s fixed, it provides immense benefits.
That's just writing a regression test and making sure it catches the regression. What does that have to do with TDD? Does the philosophy of TDD lay claim to any test written before the bugfix, regardless of how much or little someone subscribes to TDD overall?
That's just writing a regression test and making sure it catches the regression. What does that have to do with TDD? Does the philosophy of TDD lay claim to any test written before the bugfix, regardless of how much or little someone subscribes to TDD overall?
It's pretty bad at this. It's much better used as a testing methodology than a design methodology.
It can provide high level guardrails confirming implementation correctness that are as indifferent to software design as possible (giving freedom to refactor).
It can provide high level guardrails confirming implementation correctness that are as indifferent to software design as possible (giving freedom to refactor).
I always love to compare Peter Norvig’s sudoku solver implementation [1] versus Ron Jeffries TDD approach [2].
Norvig starts with the theory building and creates a constraint solver in about 50 lines of code. Jeffries starts with TDD, assumes an implementation, has to change that implementation, therefore has to change the tests, and after a series of five blog posts kind of fizzles out on it.
To me it just highlights that defining the problem is with so much more than defining the tests, as you can’t write a test for a problem you haven’t defined yet. In this way the tests are an imposition. In short to me it shows that TDD only really works if you already knew how to buold the project to begin with.
[1] https://norvig.com/sudoku.html [2] https://ronjeffries.com/xprog/articles/oksudoku/
Norvig starts with the theory building and creates a constraint solver in about 50 lines of code. Jeffries starts with TDD, assumes an implementation, has to change that implementation, therefore has to change the tests, and after a series of five blog posts kind of fizzles out on it.
To me it just highlights that defining the problem is with so much more than defining the tests, as you can’t write a test for a problem you haven’t defined yet. In this way the tests are an imposition. In short to me it shows that TDD only really works if you already knew how to buold the project to begin with.
[1] https://norvig.com/sudoku.html [2] https://ronjeffries.com/xprog/articles/oksudoku/
So, there you have it. TDD is good if applied correctly, and only if you apply it 100% correct. And so it seems for LLM usage. If it doesn't work for you, then you are obviously doing it wrong according to many folks here. TDD is nice to catch refactoring mistakes, LLMs are nice to maybe do some initial refactoring on a small enough code base. And it doesn't mean that one precludes the other. But I haven't seen TDD put engineers out of work and neither should LLMs. Trust either model fully and you are in for a world of hurt.
I would usually measure "TDD correctness" in terms of how closely the test matches a user story vs how closely it mirrors code implementation.
The former is desirable, not common. The latter is common, not desirable.
The former is desirable, not common. The latter is common, not desirable.
And with LLMs you'd have to measure how close the prompt is to a user story then. And how close its output is to the user story.
I think it works only for people with a certain thought pattern, the ones that like to think everything up-front before doing anything.
Most people prefer to play around and make several crappy attempts and combine them until the whole is somewhat solved, then go over and polish it a little, and maybe then add tests and fix the behavior in place.
For this last group, TDD it's jarring, unnatural and requires a lot of willpower to follow.
It's not bad in itself, it's just not for everyone.
Most people prefer to play around and make several crappy attempts and combine them until the whole is somewhat solved, then go over and polish it a little, and maybe then add tests and fix the behavior in place.
For this last group, TDD it's jarring, unnatural and requires a lot of willpower to follow.
It's not bad in itself, it's just not for everyone.
TDD does not require an upfront understanding of the solution. It’s primarily about verifying what your application (or module) does, not how it does it. In other words, tests should be written from the outside in.
The common struggle with TDD arises when people reduce it to writing glorified spell checkers. This usually goes hand in hand with the belief that unit tests must always check classes in isolation using excessive mocks—an approach that misses the real purpose of TDD.
It has less to do with thought patterns and more with simply misunderstanding the approach entirely because of clinging to the wrong dogmas they've heard somewhere.
The common struggle with TDD arises when people reduce it to writing glorified spell checkers. This usually goes hand in hand with the belief that unit tests must always check classes in isolation using excessive mocks—an approach that misses the real purpose of TDD.
It has less to do with thought patterns and more with simply misunderstanding the approach entirely because of clinging to the wrong dogmas they've heard somewhere.
> the ones that like to think everything up-front before doing anything.
I don't think that's the case. If they were really thinking up-front, they'd be doing proper req analysis and design work, rather than interactively growing a ball of mud that "does the minimal thing to pass a test". To me, it seems like TDD is sold as this "foolproof" design / dev approach, which is anything but.
I don't think that's the case. If they were really thinking up-front, they'd be doing proper req analysis and design work, rather than interactively growing a ball of mud that "does the minimal thing to pass a test". To me, it seems like TDD is sold as this "foolproof" design / dev approach, which is anything but.
TDD works if know what 100% of the inputs/outputs look like, AND you know the implementation will be complex, terse, whatever. Write tests, then twiddle a bunch of regex until the tests pass. This situation rarely comes up in real software jobs, but when it does, yeah use it!
TDD doesn't work for the same reasons why Todo lists don't work. I don't have a good explanation why that is so, but I know from countless examples (mine and other people's) that they don't work.
In general, doing things work, planning to do things don't.
In general, doing things work, planning to do things don't.
I guess you could say TDD is the waterfall of coding styles.
Who did you hear the it slowed development from?
My personal experience (and I think the experience of many who do it full time) is that it makes things faster.
My personal experience (and I think the experience of many who do it full time) is that it makes things faster.
I wrote a full fledged 3D exporter in 6 days without a single test case. Used by thousands of people every day.
How did I test and debug? Run my code and printf.
How did I test and debug? Run my code and printf.
Try working on a project with lots of contributors.
The whole point is that it shouldn't happen. You should consume the thing I produce.
> 81% of developers agreed that AI increases their productivity
I've had a few AI generated PRs come my way and the code-review process is, shall we say, not fun. It takes me a lot more time to review these PRs, there is way more back-and-forth between me and the 'developer', and it takes much more time to get the PR merged. That's not saying anything about the increased difficulty in modifying this code in the future.
I have a feeling these claims of being more productive don't account for the entire development cycle.
I've had a few AI generated PRs come my way and the code-review process is, shall we say, not fun. It takes me a lot more time to review these PRs, there is way more back-and-forth between me and the 'developer', and it takes much more time to get the PR merged. That's not saying anything about the increased difficulty in modifying this code in the future.
I have a feeling these claims of being more productive don't account for the entire development cycle.
I think employees (myself included) often think, "I'm more productive" when in reality what they are actually experiencing is, "My job is now easier."
Easy does not necessarily mean more productive when you're trading ease for something else. In the case of coding, you're trading ease for things like understanding and maintainability.
Easy does not necessarily mean more productive when you're trading ease for something else. In the case of coding, you're trading ease for things like understanding and maintainability.
Yes, exactly.
Right, a machine that automates the planting of mines may make the user more productive at their job, but that doesn't really account for the time spent cleaning them up on the backend and how many limbs will be lost in the process. AI is an automated landmine planting machine.
Sure AI increases developer output, which is sometimes correlated with productivity -- but often times not. Insofar as AI is accelerating positive outcomes (we were able to write some tricky code that was blocking us), it's also accelerating the negative outcomes (we used the LLM to write 40k lines of code in an hour and no one know what any of it does). One of these things is "productive" the other is just performative work.
If "being more productive" is using an ai to write an email which is then summarized by AI on the receiving end, or students using AI to write papers which are graded by AI, or developers using AI to write code which is then reviewed by AI, then AI isn't actually making anything better.
Sure AI increases developer output, which is sometimes correlated with productivity -- but often times not. Insofar as AI is accelerating positive outcomes (we were able to write some tricky code that was blocking us), it's also accelerating the negative outcomes (we used the LLM to write 40k lines of code in an hour and no one know what any of it does). One of these things is "productive" the other is just performative work.
If "being more productive" is using an ai to write an email which is then summarized by AI on the receiving end, or students using AI to write papers which are graded by AI, or developers using AI to write code which is then reviewed by AI, then AI isn't actually making anything better.
The recent RCT on open source ai bug fixing had most participants feeling like they were more productive but actually being less productive. It may just be perception error, and then the issue you identify makes it even worse.
AI search makes me personally way more productive, e.g.: write snippet how to do X using library Y I never touched before.
Maybe as a small anecdote: I had a coworker ask an AI to write them a function to auth to aws using boto3 and environment variables. The LLM happily complied and gave them a 30 line snippet but it failed to point out that if they were to open boto3 docs it would tell them that it will already default to environment variables when it has no credentials and the whole function is just reinventing the wheel.
If you’re throwing the LLM at APIs you don’t know, how could you possibly verify it is using them properly?
If you’re throwing the LLM at APIs you don’t know, how could you possibly verify it is using them properly?
The question is how fast and well dev unfamiliar with that API would achieve the goal alone compared to being assisted by LLM: he could set up snippet, run test, see what are the error, ask LLM to check why error could happen.
AI makes it quicker and cheaper to ship something. The problem with most companies is shipping the wrong thing.
VCs hope that with AI they can have a larger portfolio, shipping more things, so that by sheer luck, one is a success. That's why many employees are critical of the AI hype while VCs and C-level love it. The whole discussion about maintainability doesn't even register on the radar, employees vs. VCs and C-level are operating at a different definition of "failure".
VCs hope that with AI they can have a larger portfolio, shipping more things, so that by sheer luck, one is a success. That's why many employees are critical of the AI hype while VCs and C-level love it. The whole discussion about maintainability doesn't even register on the radar, employees vs. VCs and C-level are operating at a different definition of "failure".
It’s based on the same category error that says I can buy a company but lose the employees and still have the value.
You don’t. You either don’t get that, or you do but would rather people not know that you really just wanted to destroy a competitor and snake some of their customers.
You don’t. You either don’t get that, or you do but would rather people not know that you really just wanted to destroy a competitor and snake some of their customers.
While this may help VCs get richer, this is definitely putting software quality and software safety at risk.
If an AI-boosted startup makes it, the tech debt inside the company will be worse than in a "traditional" one. That seems like a net negative for society in the long run.
If an AI-boosted startup makes it, the tech debt inside the company will be worse than in a "traditional" one. That seems like a net negative for society in the long run.
> To fix this bug, I need to understand the code.
Almost every day on this site for the past few months has been an instance of Mugatu's "I feel like I'm taking crazy pills!" moment.
Almost every day on this site for the past few months has been an instance of Mugatu's "I feel like I'm taking crazy pills!" moment.
Hah! Especially seeing the same articles, and pro/con discussions feels like a form of the preceding lines:
_"There's only one subject on HN for Christ's sake. _Tips For Agentic AI Use_,_Gemini CLI_, _Building X with Claude_, they're the same thing!"_
Wasn’t the point of vibe coding to write throwaway code you will never maintain or care about?
Why do people think it means we can write enterprise applications without understanding the code/specifications?
Why do people think it means we can write enterprise applications without understanding the code/specifications?
That was the theory, but it ran straight into the reality that most of the time “throwaway prototyping code” gets shipped straight to production and never updated, because it’s addressing real needs right now and there’s no time to fix it because a dozen other feature requests and support issues have arrived in the meantime.
The quip that “there’s nothing more permanent than a temporary solution” has been a truism of software engineering since long before AI arrived on the scene, vibe coding is just making the problem much worse.
The quip that “there’s nothing more permanent than a temporary solution” has been a truism of software engineering since long before AI arrived on the scene, vibe coding is just making the problem much worse.
The dirty secret of a huge amount of "production code", i.e. code that is in production right now servicing enterprise needs, is that it's written with the care and understanding of typical throwaway code. It's not a new problem. Ball of mud architectures are everywhere, prototypes becoming permanent are common, understanding is low (especially if the original author has left or switched teams), but teams muddle through with the super powers of being able to ship ongoing patch fixes, code review, automated testing, and sometimes even writing code with professional care and forming little oases of quality.
I'm mildly optimistic that vibe coding won't make the problems that much worse and can actually lift the standards of quality in some cases. In my own personal careless / throwaway bash scripts, I've been using AI for them more and more, but I also notice that it puts more care into them than I otherwise would have with respect to things like error checking, friendlier help and other options, progress echos, and comments. AI tools still have a ways to go on larger projects though, and hallucinations seem particularly bad when it comes to foreign library bindings...
I'm mildly optimistic that vibe coding won't make the problems that much worse and can actually lift the standards of quality in some cases. In my own personal careless / throwaway bash scripts, I've been using AI for them more and more, but I also notice that it puts more care into them than I otherwise would have with respect to things like error checking, friendlier help and other options, progress echos, and comments. AI tools still have a ways to go on larger projects though, and hallucinations seem particularly bad when it comes to foreign library bindings...
Let's see how many forget how to write and think through their own code while primarily vibe coding.
[deleted]
I have a basic rule: understand and agree with every line of code in a PR I’m responsible for generating (manually or via agent). This simple act prevents 99% of AI-related bullshit code.
Brian Kernighan wrote:
Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?
— The Elements of Programming Style, 2nd edition, chapter 2
So if vibe coding produces code that is as clever as -- or more clever than -- you, then you have no chance of debugging it.
Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?
— The Elements of Programming Style, 2nd edition, chapter 2
So if vibe coding produces code that is as clever as -- or more clever than -- you, then you have no chance of debugging it.
Throwaway prototypes have their uses. Go ahead and vibe-code the throwaway prototype.
Nice in theory, but I once made a throwaway prototype which the client used as the foundation for their mobile app. It was translated into 26 languages and deployed across the globe. I had to work with their mobile engineers who asked why some of the views were so knotty and complex. They were complex because I was quickly patching and tweaking things between user interviews until the end of the project! The code was a mess.
Once the business sees that the prototype more or less works, it's incredibly difficult to get them to spend money on a "sane" clean-sheet rewrite.
Once the business sees that the prototype more or less works, it's incredibly difficult to get them to spend money on a "sane" clean-sheet rewrite.
There’s nothing more permanent than a throwaway prototype.
There's a point in the article that mentions allowing the model to ask questions. I've found this to be especially helpful in avoiding the bad or incomplete assumptions that so often lead to lousy code and debugging.
The (occasionally) surprising part is that there are times where the generated clarifying questions actually spawn questions of my own. Making the process more interactive is sort of like a pseudo rubber duckie process: forcing yourself to specifically articulate ideas serves to solidify and improve them.
The (occasionally) surprising part is that there are times where the generated clarifying questions actually spawn questions of my own. Making the process more interactive is sort of like a pseudo rubber duckie process: forcing yourself to specifically articulate ideas serves to solidify and improve them.
>The discipline required to use AI tools responsibly is surprisingly difficult to maintain
I don't find that this requires discipline. AI code simply requires code review the same as anything else. I don't feel the need to let AI code in unchecked in the same way I don't feel the need to go to my pull request page one day and gleefully hit approve and merge on all of them without checking anything.
I don't find that this requires discipline. AI code simply requires code review the same as anything else. I don't feel the need to let AI code in unchecked in the same way I don't feel the need to go to my pull request page one day and gleefully hit approve and merge on all of them without checking anything.
Up until now I considered the enterprise nightmare of vibe debugging to be a data/governance problem involving something like:
Dev: enables verbose/debug logging
App: encounters error, creating big log file
Dev: uploads entire logfile, containing secrets, to 3rd party LLM and asks "read this log and identify the problem"
meanwhile...
LLM: leaks prompt, logs, and secrets to hackers
LLM: uses prompt for training data, then provides secrets as responses to other users
Dev: enables verbose/debug logging
App: encounters error, creating big log file
Dev: uploads entire logfile, containing secrets, to 3rd party LLM and asks "read this log and identify the problem"
meanwhile...
LLM: leaks prompt, logs, and secrets to hackers
LLM: uses prompt for training data, then provides secrets as responses to other users
I feel this not much different from dealing with large teams of offshore devs. You constantly get huge piles of code of questionable quality written by people who don’t understand your domain. Doing thorough review probably takes as much time as writing it yourself so you end up shipping code where nobody really knows what the problems are and hope for the best.
Half the problem will be lack of a coherent commit history, to piece together what the goal of certain changes might have been. Large check-ins lose resolution here which magnifies the number of possibilities.
When you’re trying to preserve features but fix bugs this information saves a lot of time and helps prevent regressions.
When you’re trying to preserve features but fix bugs this information saves a lot of time and helps prevent regressions.
The better programmer you are, the better the software you're going to get using LLMs. But the worst programmers are leaning on LLMs the most.
The paradox is that the better LLMs get, the more serious the bugs will be because the software will seem ok, only to blow up after people have developed a false sense of security.
The paradox is that the better LLMs get, the more serious the bugs will be because the software will seem ok, only to blow up after people have developed a false sense of security.
Self-inflicted nightmare.
I just imagined a future where not only are programs entirely vibe coded and vibe debugged, but even the end users interact with them entirely through vibing. Wow.
From TFA:
Why automate the fun part and keep a human for the shitty part?
I don't really get the reasoning behind all the hype, or better said: I kinda do, but it's more of a knee jerk reaction or essentially FOMO.
What makes me think this is a bubble is the amount of emotion behind the decision making process (plus the fact that almost nobody is making a dime with this so far).
Imagine the 3 AM on-call alert. The engineer trying to fix it might be navigating a section of the codebase they've never seen before, generated entirely by an AI. In this scenario, you can't afford to vibe it out or gamble precious minutes while an AI agent attempts a fix. You need ground truth, and you need it fast.
Why automate the fun part and keep a human for the shitty part?
I don't really get the reasoning behind all the hype, or better said: I kinda do, but it's more of a knee jerk reaction or essentially FOMO.
What makes me think this is a bubble is the amount of emotion behind the decision making process (plus the fact that almost nobody is making a dime with this so far).
"Isn't it just temporary, and AI will get better?"
There are 2 instances of the word "understand" in the first paragraph, 3 if you count the beginning of the second.
In my book, "understanding" is a synonym for "intelligence" - the roots of the word are "read between the lines", where something else that just knowledge is, the ability to use and manipulate knowledge [1].
But the thing is, despite this tech being classified as "artificial intelligence", it does not understand a thing - or so little.
So, if we extrapolate Betteridge's law of headlines, no it is not temporary for this type of technology. But I think connecting it with formal computations - inference engines for formal logic, calculators [2], etc. could be amazing.
[1] https://en.wiktionary.org/wiki/intelligence - well, yes, that's a bit cherry-picked.
[2] https://arxiv.org/abs/2406.03445 - amazing result, and maybe will find out that it is exactly what we do "under the skull", but doing arithmetic with Fourier transforms is not the best use of a microprocessor.
There are 2 instances of the word "understand" in the first paragraph, 3 if you count the beginning of the second.
In my book, "understanding" is a synonym for "intelligence" - the roots of the word are "read between the lines", where something else that just knowledge is, the ability to use and manipulate knowledge [1].
But the thing is, despite this tech being classified as "artificial intelligence", it does not understand a thing - or so little.
So, if we extrapolate Betteridge's law of headlines, no it is not temporary for this type of technology. But I think connecting it with formal computations - inference engines for formal logic, calculators [2], etc. could be amazing.
[1] https://en.wiktionary.org/wiki/intelligence - well, yes, that's a bit cherry-picked.
[2] https://arxiv.org/abs/2406.03445 - amazing result, and maybe will find out that it is exactly what we do "under the skull", but doing arithmetic with Fourier transforms is not the best use of a microprocessor.
The underlying model architectures have issues that perpetuate as models get better. We're still using transformers and RL.
Maybe it's cliche from an AI safety perspective. But I can never get over https://en.wikipedia.org/wiki/Instrumental_convergence as we see micro instances of it in our day-to-day with today's agents. Again, an issue that has existed from the dawn of these types of models. https://www.youtube.com/watch?v=s5qqjyGiBdc&t=1853s
* Optimized for task completion, with limited attention resources for global alignment (RL/RLAIF reward loops/hacking)
* These systems run outside of chat now. file systems, CLIs, DBs, browsers → real-world side‑effects that you cannot train for. Hallucination becomes a problem of contradiction in the real world, and alignment is something an agent will struggle with as it's optimized to complete tasks. Which is why you see things like databases being dropped today.
* These are baked-in problems, not even considering the the adversarial nuances of things like prompt injection.
Maybe it's cliche from an AI safety perspective. But I can never get over https://en.wikipedia.org/wiki/Instrumental_convergence as we see micro instances of it in our day-to-day with today's agents. Again, an issue that has existed from the dawn of these types of models. https://www.youtube.com/watch?v=s5qqjyGiBdc&t=1853s
My nightmare is a product from search. You search and AI spins up a super slow just in time created online service of what it assumes you need. Like click button, agent creates code, compiles and adds while you wonder why its still spinning ..
I’d like to see the proof for TDD; last I heard it slowed development with only minor reliability improvements.