As I was reading this article, a similar thought occurred to me: "I wonder if that's better or worse than a human?" Unfortunately, there was no human baseline in this study. That said, there are studies that compare LLM to human performance. Usually, humans perform much better (like 5-7x better) at long-running tasks.
In other words, a human would probably do better than an LLM on this task.
Humans lose to LLMs in narrow, well-specified text/symbolic reasoning tasks where the model can exploit breadth, speed, and search. Usually, the LLM performed ~15% better than humans, but I saw studies that were as high as 80%. To my surprise, these studies were usually about "soft skills" like creativity and persuasion.
> Given that you've at minimum doubled the code (and doubled the bugs), it seems like a really bad long-term trade off.
I'd say you've at maximum doubled the code. The test ensures you write only what you need to get the test pass. Without them, devs get distracted and wander until the feature works. Usually distracting themselves with tons of YAGNI violations along the way. In my experience, untested code bases have a ton of unnecessary code.
I don't understand how it would double the bugs. The article has references saying it reduces them. But, even thinking about it, I don't see why you'd say that.
Maybe it's only difficult to test end-to-end? I would assume there's code in there that's algorithm-like. Give it these inputs, it should return these outputs. If so, it should be possible to isolate that code and test it.
But, I dunno, I haven't looked at the source code. It might be very difficult to maintain.
I've used tons of programs that have problems and crash for various reasons. Is this an argument against the language? I don't think there'd be any left to use with this line of thinking.
Besides, there's too many variables in your anecdote. Is it a laptop from 1995? Is the OOM from a bug that could/should be fixed?
> Why does this matter? It matters for the exact same reason why memory leaks are bad in general: They consume more memory than necessary.
Thing is, this doesn't usually matter. I have never gotten an out of memory error from a leak in Java. Now compare that to all the development time I've saved by not having to deal with pointer arithmetic. I consider it a huge win. It's all about the type of apps you're making.
> Place your test files next to the implementation.
Java background is probably biasing me, but I don't like this. It interferes with my ability to find code because the file list in every directory is twice as large.
Everyone is interpreting this as, "write 10 tests then try to get them all to pass at once". That is not how you TDD. You write one, then get it to pass, then write another.
Maybe you mean, "write the test before the code", but when you say "write all tests before the code", it's not interpreted the same way.
> Software, as an industry, generally profits the most when it can identify an existing need that is currently solved without computers, and then make it 10x+ more efficient by applying computers. In this situation, the software doesn't need to be bug-free, it doesn't need to do everything, it just needs to work better than a human can.
Unfortunately, the project isn't "finished" after you achieve initial success. You have to maintain it for years after that. Without good tests, you get yourself into a state where one change breaks two things. You need teams of people manually checking it still works. It takes them an order of magnitude longer to check than it would take a computer. Your feedback loop that answers the question, "does it still work?" is in weeks instead of minutes.
You stop changing the bad code because it's too scary and dangerous. The bad code reduces the frequency at which you release new features. You slow to a crawl. The team of people testing gets expensive and eats into your profits. Some competitor eats your lunch.
In my experience, these problems occur earlier than you expect. Usually before you even attain success. I'm too lazy to comb over my app checking for bugs every time I develop a new feature. I get bored too easily to check that the same bug doesn't crop up over and over again. I'm also too impatient to wait for another person to do it. In these conditions, TDD speeds you up.
I've never worked on a manually tested project that avoided these problems. It usually happens around month 2 and gets even worse as time goes on.
Here's why I think TDD fails: The consultants say, "it's so easy! Just red-green-refactor!" No, it's extremely complex and difficult, actually. There's a ton of depth that nobody talks about because they're trying to get you to try it. The typical dev learns about "Just red-green-refactor" and fails. Now "TDD did not live up to expectations".
For example, did you know there's different styles of TDD? The author of this article doesn't seem to (I may be wrong, but he gives no indication). He's talking about Detroit style TDD (also known as classicist). This is the style Ron Jeffries, Uncle Bob, Martin Fowler, Kent Beck, etc. practice and advocate for. There's another style called London style (AKA mockist) that Gary Bernhardt, etc. practice. And then there's Discovery Testing from Justin Searls. There's probably others I haven't heard of yet.
They all have different pros and cons and a very common pitfall is to use a style in a way it's not really meant for. e.g., Using mockist to gain confidence your high level features work. Expecting classicist to help you drive out good design for your code.
To wrap this up, there's so much unknown unknowns about TDD it's usually premature to judge the technique. If you haven't tried all these styles, one might click with you. A lot of people are concluding they hate all sports when they've only tried baseball.
Could it be that people who are non-agreeable often switch careers when they butt heads with others and this switching gives them higher income than those who stay in one place and wait for promotions?
> It seems that a simple yeoman/grunt setup that just preprocesses some HTML and JS takes... a very long time.
I have not experienced this. I built a 15k+ line CoffeeScript app. Every time a file changed it would compile it to JS and run thousands of unit tests in ~1 second.
As for the clean NPM... I never deleted that directory. I believe I read an article somewhere that you should actually commit your module dir. Maybe that's no longer in vogue though?