We are experimenting with ways to use ChatGPT to get better answers more reliably, remove hallucinations, etc.
This little library will generate multiple draft responses and then use a second model to judge the answers and pick a winner, which is then returned to the user. Google's Bard uses this same approach.
With this library you can apply the pattern to gpt-3.5 and gpt-4.
Drafts are generated in parallel and all drafts are evaluated with a single prompt.
This will use a lot of tokens. For example to generate 3 drafts, you are at 3x + you need to feed those drafts into another prompt + get that response, so >7x.
What all these tools need to adopt is sending 10-20 requests out and finding the "best" response. I think it's incorrect that we try to get the tool to work right the first time. Auto-GPT has JSON parse errors 20-50% of the time. Instead, with enough parallel responses we can increase the likelihood one of them is "really good". The next challenge is figuring out which answer is really good and continuing with that.
The limitation is because of the word position embedding matrix size. This isn't a config issue, or an API limitation. This is a limitation on the size of a matrix that is part of the model and is decided on before training. You can't change it.
What does that mean?
For each token in your input or inference output it requires the model to have some understanding of what the position of the word means.
So there is the word position embedding matrix that contains a vector per position. The matrix has "only" 1024 entries in it for GPT2 or 4096 for GPT3. The size of each entry varies as well, containing a vector from 768 for GPT2 small and up to 12,288 for GPT3.
So the WPE (word position embeddings) for GPT2 is (1024x768) and for GPT3 (4096x12288)
Inference requires info from this vector to be added to the word tokens embedding for each token in the original prompt + each generated token.
This is scientifically verified and yet nobody does anything about it.