I cannot disagree, but many who should know better do.
I have seen people argue with a straight face that there are no copyright concerns simply because of the sheer volume of the data that LLMs are trained on.
This makes less than zero sense. If someone has seen code, or heard music, and creates something too similar, it is a copyright violation, even though that person has seen much code or heard much music before. This is why the concept of "clean room" implementation exists, and why the concept of the abstraction-filtration-comparison legal text exists.
LLM proponents will point to the fact that courts have ruled that using copyrighted material for training has been ruled fair use.
This actually makes sense. Just as you can read a book, so can an LLM.
The thing that, AFAIK, hasn't been ruled on yet, is when the LLM regurgitates something that is too close to the book. If a human were to do that it is a clear copyright violation.
To pretend that "dilution is the solution to pollution" in terms of LLM training data, and that anything the LLM produces is original material, is to give LLMs more rights than humans have.
It's kind of a useful distinction, but the categories are really blurry.
For example, the layout software vendors provide a solution that both uses non-determinism in their final product, and also checks the results against whatever criteria you give it.
If it tells you it will work, it's probably not lying, and if it tells you it won't work, it's almost certainly not lying.
So in your own creative process of building a board, you can rely on that.
Many LLMs don't have this property at all, and honestly, it really doesn't matter whether you are using the LLM to craft a legal document or to craft code; if you don't check the output yourself, you're possibly in for a world of hurt.
> Now if you are "running" your "generated code" that's where people will have strong opinions because it conflates the two completely different processes in the worst way possible.
I think a primary driver of the attractive nuisance here is that, because the execution of the resultant code is itself a deterministic process (even if the process that produced that code it isn't) people think that they can tame the process, by some combination of automated testing and looped automated reprompting.
At some level, they may not be wrong -- computer chips themselves are built on top of stochastic atomic processes, and google famously proved in its early days that it could have reliable processes on top of shitty commodity hardware.
But one key difference is that the nondeterminism in the atomic processes or in the non-ECC memory of the white box computers is uncorrelated.
And another key difference is that, unlike the layout software vendors, so far the LLM vendors seem unable or unwilling to properly self-check their own outputs.
To me, this is kind of the canary in the coal mine. You would think that the LLM vendors have every incentive to weed out bad results, that they, more than anybody else, have the understanding of root causes and probabilities that a particular output might be bad, and have more than sufficient resources to fix this, if it's not an intractable problem.
So Occam's Razor says that (currently) this is still an intractable problem.
> I would say [whether nondeterminism is bad] is unambiguously defined by the problem you're solving.
Sure. You gave an example where it can work. Another example is something like a PCB or chip layout engine.
That particular domain (layout) is NP-complete. You'll never have an exhaustive brute force search for the optimal layout, but...
You can subsequently easily check whether the produced layout meets all your acceptance criteria or not.
Another example that successfully utilizes non-determinism for good outcomes is the application of genetic algorithms to things like antenna design.
This works because (a) you have a relatively cheap fitness test; and (b) as with real evolution, the mix of combining working designs and randomly introducing mutations often eventually produces outstanding results.
Presumably, if you applied genetic algorithm techniques to, e.g., creating your LLM prompts, you could also get good results, but that would probably quickly get expensive in terms of tokens.
So we're left with people just semi-randomly modifying prompts in order to try to tweak results.
When it works, it can be amazing. When it doesn't work, it's like a brick wall.
I like your "embarrassingly nondeterministic" term, but I somewhat disagree with:
> A "bug" then would be considered "weak accuracy" not "crash" or "incorrect behavior",
When a lawyer asks an LLM for citations of cases that support his position, he is arguably doing something stupid, because embedding an assertion such as "Show me cases which support X" is just asking for hallucinatory trouble with many current LLMs.
Nonetheless, I submit that hallucinations are, by definition, "incorrect behavior" and not merely "weak accuracy."
Now, nondeterminism could certainly be useful to the lawyer, in that it could help an LLM make connections that LexisNexis might not have in their database. So asking an LLM for help with legal issues is theoretically not an insane thing to do, but the results need to be checked very carefully.
As others have discussed, people argue for many reasons, ego being one of them.
I didn't really understand this. I grew up before the internet, and I have ADHD, which essentially means I have limited working memory.
One of my compensatory strategies for this is to have a fairly comprehensive world model at the ready in long-term memory.
If you told me something that contradicts my mental model, I might argue, in order to figure out whether I need to update my model or not.
The argument between someone ego-driven operating on a motte-and-bailey basis, and someone who just truly wants to understand, but won't let it go because they feel they need to understand, gets ugly quickly.
Fortunately, I'm older, my model doesn't need to change as often, I'm better at discriminating about things I care about or that are irrelevant, and, of course, I can always disengage with "that's interesting; I'll have to research it" and go down a rabbit hole on the internet if what they are saying doesn't seem to make a lick of sense.
I will say that the need to be right -- not the need to lord it over others, or the need to prove I'm right -- has probably helped my programming career immensely.
The burning desire to be right can be completely orthogonal to giving a shit about whether others think you're right or not, or giving a shit about others when they're wrong and it doesn't adversely affect you.
> The fact that short iterations adding features incrementally leads to better outcomes for software project is something professionals have known and argued for since the 1960s.
And practiced by software professionals since the 1960s. For maintenance. Even for aviation and nuclear. But in those industries, you're going to have a clear case, better documentation, and not be trying to "sprint."
Again, many truly best practices well predated XP/agile, and were subsumed into it. The real problem with XP/agile is the dogmatic straw-manning and demonization of other good practices that have their place, such as waterfall and exploratory programming. YAGNI, in particular, is more often used as a cudgel to shut down useful learning and exploration than anything else.
On the one hand, maybe it matters, because most people won't go to the trouble to sue, and suing the store doesn't really address the root cause anyway.
On the other hand, maybe it doesn't matter, because if this were really an issue, I'd expect to see class action suits.
On the other hand, like a lot of other good ideas, the agile community has claimed this. A quick google will show that many claim it is a "core agile idea."
> You've typically got to be a big customer, of course.
Yes, if you're a big enough customer, you might essentially be part of the design team.
> Sharing simulations and prototypes and engineering samples can and does happen.
Simulations aren't the thing. They don't go fast enough to solve anybody's problem. To your point, if a customer is part of the design team, then yes, they can, at that point, help to debug, or possibly even get started on their own dependent designs. (Part of the shift-left I talked about in another comment.)
I'm not sure what you mean by "prototypes" but "engineering samples" are essentially the finished product, done after all the work I described.
Yes, they may have bugs (or they might just not have passed validation and ESD testing yet), but that doesn't alter the fact that a waterfall effort happened before they were delivered.
> But yes, insights for an industry with relatively small costs for change don't apply easily to an industry with large costs for change, and often vice versa.
The problem with indiscriminate use of agile is that, while, yes, the software industry has relatively small costs for change, it has traditionally had huge costs for the initial delivery, and many agile proponents don't properly segregate those two cases.
If LLMs live up to their apparent promise, then, of course, the equations around the huge costs for the initial delivery could change dramatically.
Of course, the same LLM promise means that the strict definition of TDD (tests written first) is also irrelevant, and perhaps even counterproductive.
> Well he makes software and writes about software development doesn’t he? Hardware has some hard limitations.
Right.
> The reason software was even invented at all was precisely to escape those limitations.
But the methods which are useful for hardware are also often useful for software. Most of the useful parts of agile were already practiced well before that was a name for anything. And the demonization of the straw-man version of waterfall in order to sell more agile consulting has led to some serious misconceptions of what waterfall really is and what it is really capable of and useful for.
The initial impetus for what became known as TDD was software maintenance, and it makes sense there.
But most TDD practitioners are nowhere near as good at real testing as the waterfall test practitioners who understand that a single missed testcase could delay a $10 million project by six months.
And this is why, even in the realm of software you still see serious efforts for aviation and nuclear power plants, and other things with real-world consequences, using more traditional methods.
(Some) chip companies have jumped on the agile bandwagon for (some) tasks.
It's always interesting to read about some chip company or another making some agile move, when the reality is that they were already doing about as many agile things as possible before agile was a thing. (For example, a management commitment to "shift left" when they have always been about significant up-front testing and feedback.)
In many, if not most, cases, the testing software is so huge that at least some of it needs to be tested itself. That can certainly benefit from agile.
But the overall process more resembles traditional waterfall. You have several definite final endpoints, and although you can make subsequent changes, those are expensive. Also, you have a silicon budget, and a pin budget, and a heat and power budget. At the end of the day, you are producing something physical with real-world physical constraints that (a) cost real money, and (b) can't be altered by just telling your customer to add more RAM or a bigger processor.
Also, in general, although designers will write their own little unit tests for a few things, it is best practice to insure that the real tests are performed by internal organizations that are different than the organization the designer is in.
I think that subconsciously, he truism that it is easier to work with and reason about a system that is already working, and to keep it working, than to get it working at all to begin with, drives a lot of the methodology.
The designer might focus on tests to insure that things work well enough to see some results, so things can be hooked up and system tests performed earlier. In one sense, this is a shift left -- the validation people and the people writing software for the chip can get started sooner than they would have otherwise, even if it's a bit frustrating because not everything works off the bat.
But the real torture tests are typically written by the dedicated verification and validation teams. Those are really different skills than design.
Nothing I have read by Kent Beck has ever suggested that he would be useful in a chip company, where lots of people toil for a long period of time in order to produce something that no customer can possibly see until it's finished, and that must be sold in quantities of millions in order to make money.
> There's something incredibly peaceful about being in the hands of an expert you trust.
I want to know if this is a religious thing, or is related to never having had multiple doctors so bad it seemed like they were actively trying to kill you, or both. I've never had this peaceful experience personally within the realm of healthcare.
> AI can absolutely shatter that feeling in an uncomfortable way
Good. Reality is always good.
> but I don't know if I can fully trust AI either.
WTF??!? Why on earth would anybody ever think they could fully trust LLMs? Even their most vocal proponents concede they aren't infallible panaceas.
Yeah, the distributed responsibility might make this difficult, but maybe not impossible.
Is the disc defective because it doesn't play in a labeled player, or is the player defective because it doesn't play a labeled disc?
Can the licensing body be held responsible?
In point of fact, you'd probably get your money back in small claims court just by suing the store, with evidence that your player plays other discs, just not this particular disc.
Unfortunately, that doesn't really fix the problem, so much as show that an angry-enough consumer with time, energy, and money, can usually get a token of recompense.
> the biggest question for me is how robust are these designs.
Maybe it doesn't matter?
I mean, of course it matters. But most of this sort of design space is effectively NP-complete, where the creation starts with a blank schematic page and has an impossibly large search space, but where the checking of the design is much simpler.
> also, obligatory mention: "genetic antennas"
Exactly. How does this work? When confronted with the question, of course, everybody gets all excited about the constrained randomness of the GA, but if you think about it, what really makes it work is that there is a comparatively cheap test for fitness for purpose.
I have seen people argue with a straight face that there are no copyright concerns simply because of the sheer volume of the data that LLMs are trained on.
This makes less than zero sense. If someone has seen code, or heard music, and creates something too similar, it is a copyright violation, even though that person has seen much code or heard much music before. This is why the concept of "clean room" implementation exists, and why the concept of the abstraction-filtration-comparison legal text exists.
LLM proponents will point to the fact that courts have ruled that using copyrighted material for training has been ruled fair use.
This actually makes sense. Just as you can read a book, so can an LLM.
The thing that, AFAIK, hasn't been ruled on yet, is when the LLM regurgitates something that is too close to the book. If a human were to do that it is a clear copyright violation.
To pretend that "dilution is the solution to pollution" in terms of LLM training data, and that anything the LLM produces is original material, is to give LLMs more rights than humans have.