Increase GPT4's context size by asking it to compress your prompts(twitter.com)
twitter.com
Increase GPT4's context size by asking it to compress your prompts
https://twitter.com/VictorTaelin/status/1642664054912155648
14 comments
People are smashing the duplos together too hard.
Sessions are getting treated as more valuable than necessary.
This should look more like functions.
Needs lower prices and greater availability for that.
But until then… smashing duplos together.
Sessions are getting treated as more valuable than necessary.
This should look more like functions.
Needs lower prices and greater availability for that.
But until then… smashing duplos together.
I spent some time on this, here is a better version: https://mobile.twitter.com/gfodor/status/1643415357615640577
A more accurate title would be "decrease GPT4's context size by asking it to obfuscate your prompts". You're getting a prompt that consumes more tokens and isn't particularly faithful to the original.
One reply points out that even though the text length is 1/4 of the original, the “compressed” output is actually LONGER in token count!
https://twitter.com/dogeofcoin/status/1642918892602290179
https://twitter.com/dogeofcoin/status/1642918892602290179
A promising and potentially useful approach for adding additional context to your prompts.
How likely is it that context size will greatly increase in the coming year or two? Are there fundamental limits, or could we reasonably expect greatly increased context size in the future?
How likely is it that context size will greatly increase in the coming year or two? Are there fundamental limits, or could we reasonably expect greatly increased context size in the future?
ChatGPT 3.5 in response to a brief analysis of the community gave out:
"It seems like the conversation you provided is discussing the idea of compressing prompts or summaries of larger documents in order to fit them into a smaller context. While the idea has potential, it appears that people have had difficulty reproducing it and that lookup tables and external storage of memory may be more effective in the long term. Additionally, there is discussion about the possibility of greatly increasing context size in the future, with the example given that there was an almost 10x jump between GPT-3.5-turbo and GPT-4 in terms of token capacity."
The compressed one is shorter number of characters but does it have fewer tokens? https://platform.openai.com/tokenizer
The context length includes all of GPT4's replies, is that correct? If I want to minimize the context length, should I be asking GPT4 to limit its replies to just the fix that is requested?
GPT4 seems to prefer long-winded replies, i.e., when I ask for what amounts to a one-line fix, it repeats the entire block of code with the one line correction. In contrast, Replit's Ghostwriter often gives a concise reply showing only the one-line fix, and I have to ask for the entire block when it isn't clear where the fix is applied.
GPT4 seems to prefer long-winded replies, i.e., when I ask for what amounts to a one-line fix, it repeats the entire block of code with the one line correction. In contrast, Replit's Ghostwriter often gives a concise reply showing only the one-line fix, and I have to ask for the entire block when it isn't clear where the fix is applied.
This post from @gfodor is also a really interesting read on the topic of compressing prompts, not sure what the main use case would be though!
https://twitter.com/gfodor/status/1643444605332099072?s=20
https://twitter.com/gfodor/status/1643444605332099072?s=20
ZipGPT is a novel, LLM-based non-deterministic compression algorithm. Give our beta version a try. The results may surprise you!
This is the most disappointing part about the frenzy.
Obvious ideas being held up as something more than that.
If you think this is incredible, give yourself some time to go play with LLMs. Try things. Clever things. Stupid things.
It helps set a more reasonable scale for these sensational sounding snippets.
Obvious ideas being held up as something more than that.
If you think this is incredible, give yourself some time to go play with LLMs. Try things. Clever things. Stupid things.
It helps set a more reasonable scale for these sensational sounding snippets.
A) people have had a hard time reproducing it, and
B) more damning, the "compressed" version uses more tokens than the original (https://gist.github.com/VictorTaelin/d293328f75291b23e203e9d...)
> In practice, plain, well-designed summaries should be optimal to fit larger documents in the context.
> This concept has potential, though; building lookup tables seems to outperform long text summarization.
It's a clever idea, and I agree that lookup tables and external storage of memory is likely going to be important at some point, but I suspect that's going to come out of giving LLM more ability to externally reference "long-term" memory rather than compressing everything into immediate context.