What makes GPT-4 so much better than GPT-3.5 and competitors?
Any ideas? Obviously it's not a simple answer. Size, training set matter, but don't explain it. I suspect there is a set of models, algorithms which filter wrong and abusive responses. Humans are involved somehow. Reviewing the interactions, probably updating some databases. There should be significant difference in architecture..?
7 comments
It's the size. Specifically the number of parameters, the amount of training data, and the off-the-charts amount of computer power. Probably the architecture has changed in some way since GPT-2 but that's not the most important thing. The important thing is how this architecture class benefits from such massive scaling.
I remember an interview with Ilya Sutskever where he said something that was very interesting to me. He said, the GPT architecture is the first one where you can just keep scaling it and it keeps unlocking new cognitive abilities. The point that he was making in the interview was that it's not going to be the last one. Probably there are much better architectures that we will be discovered and GPT will become obsolete. But the point that I thought was important, was that GPT was the first one.
I should put some caveat that possibly ancient experimental architectures could have allowed it but GPT was the first one to actually show such amazing emergent abilities because such powerful GPGPU systems were brought to bear on it. Maybe some academic architecture from the 70s would have worked too, if they had been allowed to use such exaflops of computer power, but GPT was the first to demonstrate it.
I remember an interview with Ilya Sutskever where he said something that was very interesting to me. He said, the GPT architecture is the first one where you can just keep scaling it and it keeps unlocking new cognitive abilities. The point that he was making in the interview was that it's not going to be the last one. Probably there are much better architectures that we will be discovered and GPT will become obsolete. But the point that I thought was important, was that GPT was the first one.
I should put some caveat that possibly ancient experimental architectures could have allowed it but GPT was the first one to actually show such amazing emergent abilities because such powerful GPGPU systems were brought to bear on it. Maybe some academic architecture from the 70s would have worked too, if they had been allowed to use such exaflops of computer power, but GPT was the first to demonstrate it.
If size was the major factor, then Amazon with it's GPU farms would be the king. Google also would have nothing to worry about. So, just the size is not enough to explain. I think
a) it is really complex thing with many components
b) multimodal adds different components. But, I think, it's not limited to text and images.
c) they found the way to use 'addons', algorithms, or external models, to improve the results. this would explain why it is good at some tasks, but not the others.
d) using 'addons' requires model (re)training. however, some addons can be updated separately.
e) they put quite a lot of work in design and implementation, besides the training itself.
a) it is really complex thing with many components
b) multimodal adds different components. But, I think, it's not limited to text and images.
c) they found the way to use 'addons', algorithms, or external models, to improve the results. this would explain why it is good at some tasks, but not the others.
d) using 'addons' requires model (re)training. however, some addons can be updated separately.
e) they put quite a lot of work in design and implementation, besides the training itself.
> If size was the major factor, then Amazon with it's GPU farms would be the king. Google also would have nothing to worry about.
It's size (gpt architecture, params, training set, gpus). But there are two complicating factors: first that expertise in training a hundred million dollar transformer LLM model hasn't yet diffused throughout the industry and most of them work at OpenAI, and second is that there is nothing close to equilibrium reached yet. It's the middle of a race right now.
You say that Google would have nothing to worry about. Well, maybe you don't know it but Google already has PaLM which is competitive with ChatGPT (although maybe its most recently published version isn't as good as GPT-4). But it's not public. They also have LaMDA the one that Blake Lemoine flagged as having a kind of sentience that is worthy of some rights. Maybe he had a point or maybe he was crazy but in any case LaMDA isn't public either. Their public one is Bard which everyone knows sucks. It's not like Google can't make good AI, it's that they have already made it and they are using it for something else. Presumably they decided it's not worth exposing it publicly. Amazon is joining the race too, and is undoubtedly making some bank by letting others train LLMs on their cloud.
It's size (gpt architecture, params, training set, gpus). But there are two complicating factors: first that expertise in training a hundred million dollar transformer LLM model hasn't yet diffused throughout the industry and most of them work at OpenAI, and second is that there is nothing close to equilibrium reached yet. It's the middle of a race right now.
You say that Google would have nothing to worry about. Well, maybe you don't know it but Google already has PaLM which is competitive with ChatGPT (although maybe its most recently published version isn't as good as GPT-4). But it's not public. They also have LaMDA the one that Blake Lemoine flagged as having a kind of sentience that is worthy of some rights. Maybe he had a point or maybe he was crazy but in any case LaMDA isn't public either. Their public one is Bard which everyone knows sucks. It's not like Google can't make good AI, it's that they have already made it and they are using it for something else. Presumably they decided it's not worth exposing it publicly. Amazon is joining the race too, and is undoubtedly making some bank by letting others train LLMs on their cloud.
I've heard that Bard has become very good at coding with a recent update and beats GPT4.
I'm sure Google is sitting on a lot, it's about making it cost effective and scalable to the masses.
OpenAI have massive investment and are not one of the big tech companies, so in order to get into the field, they have an advantage were they are happy to burn through cash to gain fame, it's working for them currently, but I wouldn't be surprised for some of the big players to pop up and be more relevant when the tech really starts to take off.
I'm sure Google is sitting on a lot, it's about making it cost effective and scalable to the masses.
OpenAI have massive investment and are not one of the big tech companies, so in order to get into the field, they have an advantage were they are happy to burn through cash to gain fame, it's working for them currently, but I wouldn't be surprised for some of the big players to pop up and be more relevant when the tech really starts to take off.
I'm not saying Google isn't doing, or has nothing. So far it looks like it has nothing comparable to GPT-4. That's why they are panicking, reorganizing AI groups. They already had 2 months, and didn't catch up. Meta doesn't have either. I played with LLaMA. Only 65B model can tell how old Joe Biden will be in 2024. They all know his birth date. GPT-4 does it easily. Meta is in better position then Google, they have no competitors with comparable AI or AI team. Twitter is not even thinking about it, under the rule of notorious robophobe.
Yes, we are in a race, just a beginning. The winner will be the one with better scientists and business model, IMHO. I'm not taking sides here. Just learning, looking for new ideas. Everything is accelerating, so I expect it to blow up one way or another. Subhuman (near)AGI will be probably first, with some superhuman abilities. That will be a big bang. Soon.
And some people still believe there is nothing new since 80-th...
Yes, we are in a race, just a beginning. The winner will be the one with better scientists and business model, IMHO. I'm not taking sides here. Just learning, looking for new ideas. Everything is accelerating, so I expect it to blow up one way or another. Subhuman (near)AGI will be probably first, with some superhuman abilities. That will be a big bang. Soon.
And some people still believe there is nothing new since 80-th...
This is a total guess as I don’t work for OpenAI. But GPT-4 was the first GPT trained using a multimodal dataset of both images and text. Some have speculated that it’s this additional sensory grounding that allows the model to attain a much more accurate understanding of the universe.
Encourage you to look into Google PALM-E if you want more examples of the astounding capabilities of multimodal models.
Encourage you to look into Google PALM-E if you want more examples of the astounding capabilities of multimodal models.
The ScienceQA paper from amazon supports this. In a task where images and text were used for prediction tasks (and often the images showed physical things like pulling a door while the text was a question about physics etc), even the items with only text and no images see higher accuracy.