Ask HN: Does it feel like AI progress is slowing down, or being hidden?
14 comments
Having worked on developing a GPT-4 based product from the day we got access, which was quite early as a YC company, I would say it's the following:
- GPT-4 is too slow, and GPT-3 is not powerful enough to be 'good enough'
- Therefore, a lot of 'the work' is coming up with UX magic to overcome this
- It just is not reliable enough for business applications (yet) without hardcore QA that is currently being invented and improved materially on a weekly basis
- Therefore, the progress is in the last mile stuff like productionizing, QA, UX idiom invention, and/or coaxing GPT-3 to be good enough and go fast enough
- All of what you're mentioning are LLMs, and GPT-4 seems likely the theoretical maximum LLM based on currently available knowledge
- No doubt some more powerful models are in the works, but then the same process repeats (e.g. assess capabilities, invent UX idioms, QA and wait)
So a conspiracy that incredible things are happening but no one is talking about it to me is not accurate. More like productionizing ChatGPT/GPT-4 is very early and putting new technology into use is always harder than it says on the label.
- GPT-4 is too slow, and GPT-3 is not powerful enough to be 'good enough'
- Therefore, a lot of 'the work' is coming up with UX magic to overcome this
- It just is not reliable enough for business applications (yet) without hardcore QA that is currently being invented and improved materially on a weekly basis
- Therefore, the progress is in the last mile stuff like productionizing, QA, UX idiom invention, and/or coaxing GPT-3 to be good enough and go fast enough
- All of what you're mentioning are LLMs, and GPT-4 seems likely the theoretical maximum LLM based on currently available knowledge
- No doubt some more powerful models are in the works, but then the same process repeats (e.g. assess capabilities, invent UX idioms, QA and wait)
So a conspiracy that incredible things are happening but no one is talking about it to me is not accurate. More like productionizing ChatGPT/GPT-4 is very early and putting new technology into use is always harder than it says on the label.
Conspiracies are right
...and most of what's watchable from afar of the actual internal systems are just artificially, purposedly downgraded models.
Like anyone playing with public (even paid), APIs, you're accessing only already deprecated software.
These things are around from 2018, having reached production level circa 2020. So we - you - are seeing the past, 2020-2021 level technology, and internally they're working with 2023 level technology and playing with future AIs technologies.
It's nothing new, this was already happening from 2018 onwards.
...and most of what's watchable from afar of the actual internal systems are just artificially, purposedly downgraded models.
Like anyone playing with public (even paid), APIs, you're accessing only already deprecated software.
These things are around from 2018, having reached production level circa 2020. So we - you - are seeing the past, 2020-2021 level technology, and internally they're working with 2023 level technology and playing with future AIs technologies.
It's nothing new, this was already happening from 2018 onwards.
Reminds me of the famous quote that's often attributed to William Gibson: "The future is already here -- it's just not very evenly distributed."
I've never thought that GPT4 was 'slow', just prompt it all at once and let it do it's thing.
The problem with GPT4 is scalability and cost, even chatGPT plus users are still limited and it's still fairly pricey for a chatbot.
The problem with GPT4 is scalability and cost, even chatGPT plus users are still limited and it's still fairly pricey for a chatbot.
Well, Google decided to stop publishing their research after losing the battle to ClosedAI, which Google themselves started (the transformer paper that Google researchers wrote in 2017). So it's reasonable to think that companies are not going to compete with ClosedAI by being open about their latest achievements until they have something comparable to show. They might have achieved incredible success in terms of inference speed, but not in the quality of responses.
It's also possible that they're waiting to see how the regulation landscape changes. Big tech like Google and Meta are under much more scrutiny than a small company like ClosedAI, so they might need to solve legal issues before they can unleash their models. For example, Meta didn't release LlaMA for commercial use partly due to legal reasons. I've heard they're working on the next gen LlaMA w/ commercial license. I hope that happens so that ClosedAI realizes they can't enjoy their monopoly for long unless they become more open about their products.
It's also possible that they're waiting to see how the regulation landscape changes. Big tech like Google and Meta are under much more scrutiny than a small company like ClosedAI, so they might need to solve legal issues before they can unleash their models. For example, Meta didn't release LlaMA for commercial use partly due to legal reasons. I've heard they're working on the next gen LlaMA w/ commercial license. I hope that happens so that ClosedAI realizes they can't enjoy their monopoly for long unless they become more open about their products.
Even though most people here probably understand what you are referring to by "ClosedAI" I think that trying to coin neologisms or new designations for existing entities with the intent of conveying a specific opinion or agenda ultimately isn't helpful.
It can serve a purpose as an ironic figure of speech, but attempting to redefine the entire meaning of an entity or concept by stubbornly referring to it by a different name (e.g., consistently using "service as a software substitute" for the SaaS acronym) seems petty and doesn't add anything beyond the original argument.
It can serve a purpose as an ironic figure of speech, but attempting to redefine the entire meaning of an entity or concept by stubbornly referring to it by a different name (e.g., consistently using "service as a software substitute" for the SaaS acronym) seems petty and doesn't add anything beyond the original argument.
I agree that Google and other companies will not share as much. Facebook seems to be an exception since they are sharing models.
I expect lots of good work using LLMs under the surface of large existing systems. Google Docs, Microsoft Office 365, etc. I think this is a mixed bag. I like to see popular tools get significantly better, but I also like to see open models, open data sources, public scrutiny of open models, etc. My hope is that self hoisting useful LLMs will get easier.
I expect lots of good work using LLMs under the surface of large existing systems. Google Docs, Microsoft Office 365, etc. I think this is a mixed bag. I like to see popular tools get significantly better, but I also like to see open models, open data sources, public scrutiny of open models, etc. My hope is that self hoisting useful LLMs will get easier.
It will continue to slow down. We just don't have the know how to move to the next level. Faster hardware might bring about more gains but there's a limit there too.
I see it as a plus. The current AIs will change the way society functions so it's a good thing we have a bit of time for society to catch up.
The current AI naysayers though have a point to be very careful with AI breakthroughs. There's a non-zero chance that eventually we will hit on the AGI formula and create something. Maybe not in our lifetimes but eventually. It makes sense to start preparing now.
I see it as a plus. The current AIs will change the way society functions so it's a good thing we have a bit of time for society to catch up.
The current AI naysayers though have a point to be very careful with AI breakthroughs. There's a non-zero chance that eventually we will hit on the AGI formula and create something. Maybe not in our lifetimes but eventually. It makes sense to start preparing now.
[deleted]
I think the AI hype was a little overblown, a lot of things by people not understanding the field or just out right wrong. Don't get me wrong, AI will come and it will do most of what people are talking about, if not all and more, but there have been a lot of unrealistic time frames thrown about.
Look at self driving cars, I'm pretty sure ten years ago people were talking about it being a huge thing and being almost here and here we are in 2023 and it feels like there is far less buzz around it now and any system in public use seems highly problematic and limited, despite many more years of development.
It's the same in the AI world, although it keeps ticking away, one of the first breakthroughs that got public attention was image generation and in particular midjourney and they have been adding more cool stuff lately, it's all continuing at pace but some people have unrealistic expectations.
People saw ChatGPT and suddenly that was AI for them and it was everywhere and then ChatGPT/ GPT 4 came along and since that was only a few months afterwards, everyone is expecting GPT 5 soon, which I believe OpenAI have said hasn't even started training yet. The reality is that there was a bigger gap between GPT 3 and GPT 4 then people realised as it was less public and potentially the gaps will get bigger as it needs more training/development as it's harder as the low hanging fruit has already been picked.
There is also a bunch of tech floating around at companies that don't know what to do with it yet, facebook for example have the audio generation thing they don't want to release to the public as they said it was too 'dangerious'
Look at self driving cars, I'm pretty sure ten years ago people were talking about it being a huge thing and being almost here and here we are in 2023 and it feels like there is far less buzz around it now and any system in public use seems highly problematic and limited, despite many more years of development.
It's the same in the AI world, although it keeps ticking away, one of the first breakthroughs that got public attention was image generation and in particular midjourney and they have been adding more cool stuff lately, it's all continuing at pace but some people have unrealistic expectations.
People saw ChatGPT and suddenly that was AI for them and it was everywhere and then ChatGPT/ GPT 4 came along and since that was only a few months afterwards, everyone is expecting GPT 5 soon, which I believe OpenAI have said hasn't even started training yet. The reality is that there was a bigger gap between GPT 3 and GPT 4 then people realised as it was less public and potentially the gaps will get bigger as it needs more training/development as it's harder as the low hanging fruit has already been picked.
There is also a bunch of tech floating around at companies that don't know what to do with it yet, facebook for example have the audio generation thing they don't want to release to the public as they said it was too 'dangerious'
It's progressing, but maybe not in the way you think. 1 model will not rule them all, they'll get overtaken by vertical LLM's specialized with niche, high quality data sets.
Also prompt templates are rapidly progressing. "Tree of Thought" is the latest state of the art, but we're not applying predictive probabilistic tree traversal yet.
There's still a ton of low hanging fruit everywhere. Why are we inputting with language instead of code? Seems really backwards to use imprecise freeform language instead of intent templates.
Also prompt templates are rapidly progressing. "Tree of Thought" is the latest state of the art, but we're not applying predictive probabilistic tree traversal yet.
There's still a ton of low hanging fruit everywhere. Why are we inputting with language instead of code? Seems really backwards to use imprecise freeform language instead of intent templates.
I wonder if there's something like the Doppler effect going on here. The field continues to advance at a constant rate, but it takes time for news to reach us, proportional to the distance between us and the frontier.
To a stationary observer, it would appear as if the time between developments is increasing, and that the field is slowing.
I hope I'm wrong to use this analogy, though. I'd like to continue thinking of myself as relatively informed about technology.
To a stationary observer, it would appear as if the time between developments is increasing, and that the field is slowing.
I hope I'm wrong to use this analogy, though. I'd like to continue thinking of myself as relatively informed about technology.
Does the Gartner Hype Cycle not exist in your world?
No not really. It also felt like an AI "winter" after the fairly transformative Talk-To-Transformer released, but we eventually got better stuff in the end.
1. GPT-3 -> ChatGPT
2. ChatGPT -> GPT-4
3. GPT-4 -> now
1, discussion about AI was big in the tech field but ignored in the mainstream. During 2, it was all the buzz everywhere and new developments were emerging at a rapid pace (stable diffusion, LLama, etc.) Now however during period 3 it feels like there's a big dampener on all the buzz, either because things are slowing down or that progress isn't being shared.
Currently, while there is still a huge buzz in the field, it feels quieter than period 2. I imagine it's because companies are hiding their progress more to ensure their competitive advantage.
My question is, has the open hype over AI during period 2 moved "underground" to private discussions between tech workers and researchers? Is all the accelerating change happening behind NDA's?