This is a well-referenced essay, drawing the on writing of David Parnas [1], Peter Naur [2], and Zach Tellman [3].
As software developers we’re intimately familiar with these ideas. But the industry still treats it as “folk knowledge”, despite decades of academic work and systemization attempts like the original Agile.
We really need more connective work, relating the theoretical ideas to the observed behavior of real-life software projects, and to the subsequent damage and dysfunction. I liked this essay because it scratches that itch for me. But we need this work to go beyond personal blogs/newsletters/dev.to articles. It needs to be recognized & accepted as formal “scientific” knowledge, and to be seen and grokked by industry and corporate leadership.
I’m a violinist (amateur but play regularly). When I have an important note, which is held for a while and needs vibrato, I frequently decide to shift my left hand position so that my middle finger is responsible, rather than the index finger. It feels stronger, easier to nail the intonation (pitch) with precision, and freer to perform the desired type of vibrato. (String players do vibrato by wiggling the left hand finger, which affects the pitch and overtones / oscillation modes of the string.) In fact, I tend to avoid using the index finger on notes that require vibrato.
That preference might be explained here, by the precision/strength combination. I tried holding a hammer as described in the author’s hammer exercise, and there’s similarity, though it requires much more weight-holding. The left hand doesn’t hold the weight of the violin (consider a cello or a guitar with shoulder strap), but a little grip strength is required to securely hold down the string, especially with vibrato.
Overall, fascinating article. I feel quite motivated to read more on hand anatomy and biomechanics.
Agreed. I recall being taught in college physics labs: there is no such thing as “human error”. Instead, think about the causes and mechanisms of each source of error, which helps both quantifying and mitigating them.
Same energy here. “Be more careful” is extraordinarily hand-wavy for a profession that calls itself engineering.
A while back I prototyped (very roughly) an auditory equivalent to “syntax highlighting”, using ambient tones and white noise, rather than discrete beeps/sound effects. [1]
I’m actually revisiting this project right now! I’m reimplementing it in Rust and also exploring different ways to communicate parser state and other contextual information through sound.
I’m glad I saw your comment!! I’ve experienced this exact phenomenon when playing Minecraft while listening to an audiobook or podcast. Returning to that area will immediately remind me of the topic or narrative that I heard. Presumably it’s related to the “memory palace” technique, but otherwise, I can’t make heads or tails of it. It’s immediate, as if the location is a hash key mapping to the information. Or as if they’re stored in literally the same place, and fetching one implies fetching the other.
Similarly to you and the article’s author, this doesn’t happen with whatever thoughts I may think while at a location. But in that situation the brain is engaged in generating those thoughts, and not with the task of learning new information. So I don’t find it surprising that it works differently.
I haven’t thought about it in relation to “consciousness” yet. Will have to chew on this article a bit.
The museum was also a generous community space; I remember attending a Seattle Indies game jam and other events before the pandemic. It was very special to be surrounded by reminders of early-computing exploratory spirit.
*If* such a thing could be done under typical corporate incentives/behavior, then I suspect the “high impact” part would need to be scrapped. Because when something is important to a corporation, it turns its eyes that thing (so to speak), which disrupts the other properties.
Or, “high impact” could be spread over the long term. So, unknown-payoff R&D. It would need to be an “invest and ignore” strategy and require a lot of institutional trust.
Question for ML/AI people: is it still common to use the term “overfitting” for these cases, where a model is overtrained on one thing, to the detriment of another? Or is that term only used for literal curve fitting?
I do think it’s an interesting line of inquiry… but not robust enough.
E.g. this paper would be much more interesting if it measured the threshold at which the LLM starts to become good at X, and linked that threshold to the number and character of training examples of X. Then, maybe, we can begin to think about comparing the LLM to a human.
Alas, it requires access to the training data to do that study, and it requires a vast amount of compute to do it robustly.
Good point - I saw the FLAN anomaly and this didn’t occur to me!
A good follow up question would be: why didn’t the other models do better on the 2nd-order question? Especially BLOOM and davinci-003, which were middling on the 1st-order question.
I agree on your overall criticism of the experimental protocol, though.
Skimming through studies like this, it strikes me that LLM inquiry is in its infancy. I’m not sure that the typical tools & heuristics of quantitative science are powerful enough.
For instance, some questions on this particular study:
- Measurements and other quantities are cited here with anywhere between 2 and 5 significant figures. Is this enough? Can these say anything meaningful about a set of objects which differ by literally billions (if not trillions) of internal parameters?
- One of prompts in second set of experiments replaces the word “person” (from the first experiment) with the word “burglar”. This is a major change, and one that was unnecessary as far as I can tell. I don’t see any discussion of why that change was included. How should experiments control for things like this?
- We know that LLMs can generate fiction. How do we detect the “usage” of the capability and control for that in studies of deception?
A lot of my concerns are similar to those I have with studies in the “soft” sciences. (Psychology, sociology, etc.) However, because an LLM is a “thing” - an artifact that can be measured, copied, tweaked, poked and prodded without ethical concern - we could do more with them, scientifically and quantitatively. And because it’s a “thing”, casual readers might implicitly expect a higher level of certainty when they see these paper titles.
(I don’t give this level of attention to all papers I come across, and I don’t follow this area in general, so maybe I’ve missed relevant research that answers some of these questions.)