We all know this isn’t replacing doctors. We do know that when applied correctly, this will assist doctors and other clinical specialists in their work. With doctor burnout at all time highs, I think stuff like this is amazing.
I could see that, and I wouldn't want LLMs just generating tests willy-nilly with no human oversight. I don't have any trust in the reasoning ability of LLMs at all.
Rather, I prefer to view LLMs as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.
You're right. I've edited my wording to be more realistic about the value of the test. I believe you're right that the test is not an outlier in terms of value provided.
Some of my comments within the article are more aspirational than realistic in this case, and I've made edits to reflect that.
I want to clarify that I view this LLM as a junior dev that submits PRs that pass presubmits and other verifiable, programmatic checks. A human dev then reviews the PR manually. In this case, the LLM + its processing is used to make sure that no BS is sent out of review - only potential improvements.
In no scenario should it's auto-generated code be auto-submitted into the codebase. That becomes a nightmare really fast.
Good point! I used exponential to emphasize the nature of the value of a certain test compared to the rest of the generated tests, but you're right it's not the right word. I updated the article to remove the usage. :)
Ouch - half of the article (the "Actionable Takeaways" section) was my own commentary. The summary was for those who didn't want to necessarily parse through the entire paper's PDF.
Happy to listen to any constructive feedback if you have any, though!
Hey! Thanks for your comment - I'm the one who wrote this article. I wasn't trying to say that the paper authors talked about "unexpected edge cases" or "thinking outside the box." I edited the post to be more clear that some of these takeaways are my own opinions.
This article is less of a summary of a paper and rather commentary on what the results of the paper entails. After all, Hacker News is meant for discussion :)
I will say though that I do believe that I still stand by the "exponentially more valuable" portion. I think the fact that LLMs can fluke their way into "hitting a jackpot" in terms of test coverage is exactly why they're so valuable. When you have something constantly trying out different combinations, if it hits even one jackpot, like in the paper, it's extremely valuable to the team. It's a case that could have been either non-obvious or simply too tedious to write a test for manually. I think there's tremendous value in that, especially speaking as someone who has spend way too much time simply figuring out how to test something within a Big Tech codebase (F/G) when I already knew what to test.
I didn’t “slap” any date on top of the article. 2 Oct 2023 is the date I published my article. (This is how blogs on the Internet work.)
The first line of the article says that this all happened in 2012. I’m not sure I could get any clearer there.
The “GPT distillation” phrase is pretty rude. Anyone reading the article can see the difference between it and the presentation. Every “hand-drawn” image in the article is created by me in Excalidraw, and any images from the presentation are sourced with a very clear link to the presentation.
If you have any constructive feedback, I am happy to listen and take it into account.
Otherwise, I’m not really a fan of the rudeness. Thanks.
I’m the writer of this. I linked to this presentation very clearly throughout the article as a source.
This was an article meant to resurface learnings from something that happened a decade ago, with added images and a clear distillation so that somebody doesn’t have to watch the 45 minute presentation to understand what happened.
I’m sorry you didn’t like it.
But I hope you can see the value I’m trying to provide here, as someone myself who doesn’t really have the time to sit through hour-long presentations to learn something.
A lot more founders and employees are heavily diluted than you think. I always just assumed massive exits meant people were rich.
But when I calculated out how much an exec would make for a diluted $1B exit compared to working at FAANG for the same amount of time... FAANG turned out better many times (unfortunately).
I think preventing dilution should be one of the highest priorities for founders for themselves and for retention. Every single dilution event sets the outcome bar higher and, thus, harder to achieve.
While X hasn't been dying just yet, I think this would probably kill it. The value of X comes from both the creators and the free users and lurkers. If it's just full of creators and die-hard lurkers, the value decreases a lot for both.
I barely use Bard, but I do use the Search Generative Experience and the Bard-in-Google Docs quite a lot. I find both quite useful as they integrate quite well into my daily workflow.