Something Big Is Happening(twitter.com)
twitter.com
Something Big Is Happening
https://twitter.com/mattshumer_/status/2021256989876109403
29 comments
I went down a bit of a rabbit-hole trying to figure out exactly who Matt Shumer is and why anyone should care what he thinks. The best information I found came from this article, which was from before he pivoted to being an AI startup bro:
https://www.newsweek.com/i-couldnt-play-rules-so-i-became-en...
It's kind of a sad read. He would benefit a lot from getting outside the startup bubble and talking to some people who do useful work for a living instead of riding internet fads and growthmaxxing via viral social media posts.
https://www.newsweek.com/i-couldnt-play-rules-so-i-became-en...
It's kind of a sad read. He would benefit a lot from getting outside the startup bubble and talking to some people who do useful work for a living instead of riding internet fads and growthmaxxing via viral social media posts.
I put it through its paces and 4.6 is actually a downgrade from 4.5 in terms of communication and steerability. I went back to 4.5.
> I use Opus 4.6 all day long and this is not my experience at all.
It's not the experience of anyone who uses AI. People have put these AI through freshman level college CS courses and the results weren't impressive at all.
> these sort of technically-light puff pieces are pretty tiresome and reflect poorly on the people who author and promote them.
These people tend to be invested heavily in AI. That's the commonality tying them all together - AI grift. They'll say anything for more money.
So for AI appears to be a useful tool that requires a bit of hand holding. Nothing more. Maybe things will change and AGI is right around the corner.
It's not the experience of anyone who uses AI. People have put these AI through freshman level college CS courses and the results weren't impressive at all.
> these sort of technically-light puff pieces are pretty tiresome and reflect poorly on the people who author and promote them.
These people tend to be invested heavily in AI. That's the commonality tying them all together - AI grift. They'll say anything for more money.
So for AI appears to be a useful tool that requires a bit of hand holding. Nothing more. Maybe things will change and AGI is right around the corner.
I can’t stand all this hype anymore. They’re pushing it so hard that I’m starting to develop an AI phobia. If it were really that good, they’d already be making big money off it. But it’s not — it just needs your data (codebase?), supposedly to make it better. The fact is, at least they’ve gotten halfway there, which from their perspective, is sweet. All the AI labs are actually losing a ton of $$$ to this AI race, but the data, ohh yeah! Let's go, baby!
I am strictly talking about the coding capabilities of the LLM, and not their core LLM capabilities, which they genuinely excel at.
I am strictly talking about the coding capabilities of the LLM, and not their core LLM capabilities, which they genuinely excel at.
I'm pretty pragmatic and resist AI hype. However, I am in the camp that the tools we've built do not maximize the potential of the LLMs we have today.
I see 2 things happening in parallel.
1) Tools on existing LLMs continue to improve (cursor -> claude code).
2) The LLMs themselves improve which makes existing tooling better and results in new tooling to take advantage of the improvements.
I'm not sure when I see either of these slowing down and they've been accelerating at a very rapid pace. Perhaps when the funding dries up.
I think that's the question but I believe if we don't have any more LLM improvements that we still have a couple years of tooling improvements using what's there today.
I'm sort of surprised that coding is a leading use case but do not see any reason it would not spread to other industries (what the OP is saying).
I see 2 things happening in parallel.
1) Tools on existing LLMs continue to improve (cursor -> claude code).
2) The LLMs themselves improve which makes existing tooling better and results in new tooling to take advantage of the improvements.
I'm not sure when I see either of these slowing down and they've been accelerating at a very rapid pace. Perhaps when the funding dries up.
I think that's the question but I believe if we don't have any more LLM improvements that we still have a couple years of tooling improvements using what's there today.
I'm sort of surprised that coding is a leading use case but do not see any reason it would not spread to other industries (what the OP is saying).
> Your dreams just got a lot closer.
Sure, if your dream is to see a bunch of text with your name on the front. The reason people might dream of writing a book is because writing a book is hard. They want the feeling of accomplishing something hard. If it becomes easy, then it's no longer a dream.
We do things not because they are easy, but because they are hard. So unless you can explain how AI is going to give me more time in the garden, to play music or to paint, then I am not closer to any of my dreams.
Sure, if your dream is to see a bunch of text with your name on the front. The reason people might dream of writing a book is because writing a book is hard. They want the feeling of accomplishing something hard. If it becomes easy, then it's no longer a dream.
We do things not because they are easy, but because they are hard. So unless you can explain how AI is going to give me more time in the garden, to play music or to paint, then I am not closer to any of my dreams.
> We do things not because they are easy, but because they are hard.
https://www.rice.edu/jfk-speech
https://www.rice.edu/jfk-speech
Like many others, I can vouch for “AI is good at programming now,” at least for the generic web programming that I do. But that doesn’t imply that it generalizes to other fields, and this article, at least, doesn’t show that it does.
I would like to read more from people who have other jobs about what they see when they use AI. Did they see a similar change?
I would like to read more from people who have other jobs about what they see when they use AI. Did they see a similar change?
I’m a mech engineer and project manager.
It translates PDFs for me and gives me a good enough text dump in the console to understand what I’m being told to do. If the PDF is simple enough (a letter, for example). It doesn’t give me a structured English recreation of the PDF.
I’ll give it credit that it’s probably underpinning improved translation in e.g. google translate when I dump a paragraph of English and then copy the Chinese into an email. But that’s not really in the same ballpark.
The only other professional interaction I’ve had with it was when a colleague saw an industry-slang term and asked AI what it meant. The answer, predictably, was incredibly wrong but to his completely naive eyes seemed plausible enough to put in an email. As in, it was a term relating to a metallurgical phenomena observed by a fault and AI found an unrelated industry widget that contained the same term and suggested it was due to the use of said widget.
I don’t even really see the telltale AI writing signs of people using it to summarise documents or whatnot. Nor could I think how I could take what I do and use it to do it faster or more efficiently. So I don’t even think it’s being used to ingest and summarise stuff either.
It translates PDFs for me and gives me a good enough text dump in the console to understand what I’m being told to do. If the PDF is simple enough (a letter, for example). It doesn’t give me a structured English recreation of the PDF.
I’ll give it credit that it’s probably underpinning improved translation in e.g. google translate when I dump a paragraph of English and then copy the Chinese into an email. But that’s not really in the same ballpark.
The only other professional interaction I’ve had with it was when a colleague saw an industry-slang term and asked AI what it meant. The answer, predictably, was incredibly wrong but to his completely naive eyes seemed plausible enough to put in an email. As in, it was a term relating to a metallurgical phenomena observed by a fault and AI found an unrelated industry widget that contained the same term and suggested it was due to the use of said widget.
I don’t even really see the telltale AI writing signs of people using it to summarise documents or whatnot. Nor could I think how I could take what I do and use it to do it faster or more efficiently. So I don’t even think it’s being used to ingest and summarise stuff either.
Blake Lemoine thought that 2022 already: https://www.criticalopalescence.com/p/is-blake-lemoine-reall...
Thought this name sounded familiar... Matt Shumer was one of the people responsible for the "Reflection 70b" hoax a few years ago. There is no reason to take anything he writes seriously, he has a history of flat-out lying to go viral.
Edit: Summary for anyone who didn't follow this saga at the time: https://www.ignorance.ai/p/the-fable-of-reflection-70b
Shumer is at best a fool and at worst a con artist.
Edit: Summary for anyone who didn't follow this saga at the time: https://www.ignorance.ai/p/the-fable-of-reflection-70b
Shumer is at best a fool and at worst a con artist.
It's big, but it's controlled by humans and how society uses it, and to whom the benefits accrue are ultimately political choices.
I think this should end with "write a letter to your congressperson and senators because the companies building this and current large employers already have lobbyists."
I also think we can't be too breathlessly optimistic about where this can go. Producing feasible text was apparently easy. Producing code that meets a prose specification took longer, but we're basically there. But you can know something about the quality of that output by reading it, or reading and running it. But some stuff will be limited by slow real-world processes.
> The upside, if we get it right, is staggering. AI could compress a century of medical research into a decade. Cancer, Alzheimer's, infectious disease, aging itself...
You can't really do medical research on aging dramatically faster than people age. Maybe there's a metabolic process which, if you started intervening before people hit 60, would help more of them live to 100+ -- but more and smarter AI may still require 40+ years to run the experiment.
I also think we can't be too breathlessly optimistic about where this can go. Producing feasible text was apparently easy. Producing code that meets a prose specification took longer, but we're basically there. But you can know something about the quality of that output by reading it, or reading and running it. But some stuff will be limited by slow real-world processes.
> The upside, if we get it right, is staggering. AI could compress a century of medical research into a decade. Cancer, Alzheimer's, infectious disease, aging itself...
You can't really do medical research on aging dramatically faster than people age. Maybe there's a metabolic process which, if you started intervening before people hit 60, would help more of them live to 100+ -- but more and smarter AI may still require 40+ years to run the experiment.
My hot take: nerds think AI is transformative because nerds build AI to be really good at their niche tech activities.
In my experience, it’s far less useful outside of that. To the point where if AI disappeared tomorrow, it’d make approximately 0 difference to my overall life. I simply don’t find it useful, neither in my professional life nor my personal.
The only repeated use case I’ve found is throwing a PDF at it and asking it to translate the PDF. To its credit, it’s able to now OCR handwriting prior to translating which is nice.
It still doesn’t make a translated PDF. Yes, I know PDF is a shitty proprietary mess of undocumented functions. I don’t care - this is the vaunted AI, it’s apparently eating the entire jobs of programmers. Have it go create and A/B test an entire clean room implementation of the PDF format then.
Now it may be an underlying shim in a feature pipeline with which i interact but thats chasms apart from this “AI is about to eat all of our jobs”. It’s a tuned feature, such as improved translation, in that instance.
My experience from very sporadic use and observation of my colleagues is that, outside of tech, AI is much more of a “go and find the info, summarise it and give me the results” layer to the internet. It’s a slightly more convenient search engine. That’s it.
In my experience, it’s far less useful outside of that. To the point where if AI disappeared tomorrow, it’d make approximately 0 difference to my overall life. I simply don’t find it useful, neither in my professional life nor my personal.
The only repeated use case I’ve found is throwing a PDF at it and asking it to translate the PDF. To its credit, it’s able to now OCR handwriting prior to translating which is nice.
It still doesn’t make a translated PDF. Yes, I know PDF is a shitty proprietary mess of undocumented functions. I don’t care - this is the vaunted AI, it’s apparently eating the entire jobs of programmers. Have it go create and A/B test an entire clean room implementation of the PDF format then.
Now it may be an underlying shim in a feature pipeline with which i interact but thats chasms apart from this “AI is about to eat all of our jobs”. It’s a tuned feature, such as improved translation, in that instance.
My experience from very sporadic use and observation of my colleagues is that, outside of tech, AI is much more of a “go and find the info, summarise it and give me the results” layer to the internet. It’s a slightly more convenient search engine. That’s it.
I love how the apparent solution being peddled for the same automation assisted unemployment is apparently for everyone to adopt more AI.
Back in the day, when the Jacquard machine entered the industry, my advice to weavers would have been to either begin tapping into their network and move to artisanal sales, or to learn how to operate/maintain mechanical looms. The writing was on the wall with the punch card system. In retrospect, it didn't take too long to go from the mechanical loom to diffusion models and world models for robots.
Of course, there was a longer timeline of incremental improvements on the loom over years, so the industry was not transformed overnight. But that's what's worrying people this time. We might not have time to reskill or retrain before financial hardship strikes. Especially in the current political climate. The middle of WWIII is one of the last periods I'd have asked for this automation revolution to take place.
Of course, there was a longer timeline of incremental improvements on the loom over years, so the industry was not transformed overnight. But that's what's worrying people this time. We might not have time to reskill or retrain before financial hardship strikes. Especially in the current political climate. The middle of WWIII is one of the last periods I'd have asked for this automation revolution to take place.
Unpopular opinion but I kind of agree. I've been using Codex extensively these past few months. Using the latest paid models. Something is changing. It was a nice assistant before. In the past few weeks, I realized it became better than me in making architectural decisions on low-level performance-critical code. For the first time, I realize that I prefer his decisions rather than mine. He is just better now. I have 30+ years of programming experience. I agree with the author that we should get ready to adapt quickly to a world that may soon become radically different.
As far as I could see there is nothing new here. I want to hear realistic solutions for a post-AI society/economy, and how we will solve alignment; the two big problems.
I’m literally fed up with half-minded techbros throwing around such overhyped writings. They’re absolutely detached from reality and talk about something that doesn't pass any fact checks.
After reading this writing, I was really curious "are we really got a breakthrough" so I subscribed to a Claude to personally check the state of the matter. As my personal test, I'm using a task that must be relatively simple for an experienced programmer: write an SSH agent that manages keys. The protocol is very simple, and the agents exist for decades now. I already tested that a year ago with Claude, and I was curious how it would do this time.
I installed up the Claude Code CLI tool, created CLAUDE.md with instructions that sounded like "this project is to build a fully functional SSH agent application that listens on a UNIX socket and implements the SSH agent protocol according to RFC specification <link> and all functionality must be covered with tests, create a plan, track progress, etc. The ultimate goal is to have a fully functional application with all SSH agent features." There were more details, of course, but you got it.
Opus 4.6 model worked autonomously for 30 minutes or so. During the process, it had been producing non-compilable code, which it had to fix. Then it stopped at a half-baked project that wasn't working as an SSH agent even closely and had zero tests pretending the job was done.
The truly intelligent technology should finish the project. Why did it stop? How is it intelligent if it makes mistakes that it needs to fix? Why can’t it finish the project while it understands a goal? How much must I push it to finish a project? How someone can claim that "this" is going to replace us?
I can't buy someone's claims that “we are one step before replacing everyone’s jobs" when the facts are saying that nothing has changed for the last year and LLMs are literally not capable of finishing things or concentrating on small important details. They always miss something important. What are we talking about here? Snake oil. This is not intelligence; the technology is a world's blurred snapshot compressed into a couple of gigabytes blob and acts as a dumb parrot.
After reading this writing, I was really curious "are we really got a breakthrough" so I subscribed to a Claude to personally check the state of the matter. As my personal test, I'm using a task that must be relatively simple for an experienced programmer: write an SSH agent that manages keys. The protocol is very simple, and the agents exist for decades now. I already tested that a year ago with Claude, and I was curious how it would do this time.
I installed up the Claude Code CLI tool, created CLAUDE.md with instructions that sounded like "this project is to build a fully functional SSH agent application that listens on a UNIX socket and implements the SSH agent protocol according to RFC specification <link> and all functionality must be covered with tests, create a plan, track progress, etc. The ultimate goal is to have a fully functional application with all SSH agent features." There were more details, of course, but you got it.
Opus 4.6 model worked autonomously for 30 minutes or so. During the process, it had been producing non-compilable code, which it had to fix. Then it stopped at a half-baked project that wasn't working as an SSH agent even closely and had zero tests pretending the job was done.
The truly intelligent technology should finish the project. Why did it stop? How is it intelligent if it makes mistakes that it needs to fix? Why can’t it finish the project while it understands a goal? How much must I push it to finish a project? How someone can claim that "this" is going to replace us?
I can't buy someone's claims that “we are one step before replacing everyone’s jobs" when the facts are saying that nothing has changed for the last year and LLMs are literally not capable of finishing things or concentrating on small important details. They always miss something important. What are we talking about here? Snake oil. This is not intelligence; the technology is a world's blurred snapshot compressed into a couple of gigabytes blob and acts as a dumb parrot.
"This page is not supported.
Please visit the author’s profile on the latest version of X to view this content.
twitter is useless haha
https://shumer.dev/something-big-is-happening
Its a pretty good article but tldr is that AI is getting pretty good according to the author, and he believes it will impact employment etc because a couple of the latest models are really good. I have to agree and the thing is that many folks are writing that these AIs have to replace or automate a really significant number of jobs for the financials to make sense.
Personally I think these giant models are the wrong tech for the wrong time but there's no denying these things are getting very good at what they do.
twitter is useless haha
https://shumer.dev/something-big-is-happening
Its a pretty good article but tldr is that AI is getting pretty good according to the author, and he believes it will impact employment etc because a couple of the latest models are really good. I have to agree and the thing is that many folks are writing that these AIs have to replace or automate a really significant number of jobs for the financials to make sense.
Personally I think these giant models are the wrong tech for the wrong time but there's no denying these things are getting very good at what they do.
How do people read this drivel and take it seriously? I read the first 5 paragraphs of complete fluff, and then started skimming, and then realized it went on and on, and just closed it. The author had not gotten to any cogent point either within the first 5 paragraphs or the subsequent ones I skimmmed.
AI is just the modern Tower of Babel. Useful? Absolutely. Valuable? Quite. But ffs please stop with the fearmongering. People are already struggling, the last thing they need is to be beaten over the head with is "and it's coming for everything!"
I think there's a gap in the market: it's people who know how to just interact with AI in a healthy way and avoid circles that are obsessed with warning people about how they're ushering in a dystopian, broken world (purposefully) and why you should celebrate their prognostication and efforts to destroy you.
I think there's a gap in the market: it's people who know how to just interact with AI in a healthy way and avoid circles that are obsessed with warning people about how they're ushering in a dystopian, broken world (purposefully) and why you should celebrate their prognostication and efforts to destroy you.
These models are great at legacy code but for the love of god please do not let it run wild on a new project for 4 hours and assume it’s “done better than I ever could have”.
I was thoroughly impressed as well by Claude but quickly disenchanted. It often creates bloated, slightly missing the mark code.
Don’t get me wrong, this is all insanely impressive considering 5 years ago LLMs were basically nonexistent in our workflow. Regardless of the overhyped sentiment in this tweet, I really do think something big is happening as well…
I was thoroughly impressed as well by Claude but quickly disenchanted. It often creates bloated, slightly missing the mark code.
Don’t get me wrong, this is all insanely impressive considering 5 years ago LLMs were basically nonexistent in our workflow. Regardless of the overhyped sentiment in this tweet, I really do think something big is happening as well…
> I know this isn't a fad. The technology works, it improves predictably, and the richest institutions in history are committing trillions to it.
I know this isn't a fad. The tulip breeding works, it improves predictably, and the richest institutions in history are committing billions to it.
I know this isn't a fad. The tulip breeding works, it improves predictably, and the richest institutions in history are committing billions to it.
I use Opus 4.6 all day long and this is not my experience at all. Maybe if you're writing standard CRUD apps or other projects well-represented in the training data. Anyone who has written "real" software knows that it's lots of iterating, ambiguity and shifting/opposing requirements.
The article seems to be written in order to feed into some combination of hype/anxiety. If the author wants to make a more compelling case for their stance I would suggest they build and deploy some of this software they're supposedly getting the LLM to perfectly create.
Yes, it's a very useful tool, but these sort of technically-light puff pieces are pretty tiresome and reflect poorly on the people who author and promote them. Also, didn't this guy previously make up some benchmarks that turned out to be bogus? https://www.reddit.com/r/LocalLLaMA/comments/1fd75nm/out_of_...