I wonder how much this depends on the quality and consistency of the context?
For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental.
Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?
I run readlang.com as one person. I started it back in 2012 and it currently makes about 14K euros / month, with expenses of about 1.5K, so it's mostly profit.
Interesting point but I don't think it's that clear cut. Twitter/X seemed to increase the pace of product changes directly after laying of the majority of its employees after Elon Musk took over. Also, when Steve Jobs returned to lead apple in 1997 he fired a significant fraction of the company before starting an incredible period of innovation. So I think a lot depends on the leadership and incentive structures.
For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental.
Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?