I wonder how much this depends on the quality and consistency of the context?
For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental.
Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?
I run readlang.com as one person. I started it back in 2012 and it currently makes about 14K euros / month, with expenses of about 1.5K, so it's mostly profit.
Interesting point but I don't think it's that clear cut. Twitter/X seemed to increase the pace of product changes directly after laying of the majority of its employees after Elon Musk took over. Also, when Steve Jobs returned to lead apple in 1997 he fired a significant fraction of the company before starting an incredible period of innovation. So I think a lot depends on the leadership and incentive structures.
I thought the same thing. Surely it must be a function of the prevalence of these fonts though. In a hypothetical world where 90% of text you read is in sans serif I'd have to imagine that this would tilt the readability study results in sans serif's favor. I wonder if the studies attempt to control for this somehow?
For example, it may be the case that a long context full of useful information relevant to the task is completely fine, perhaps even beneficial. And if the context contains a bunch of unrelated tangents and conflicting instructions, then it will be detrimental.
Have there been studies on what makes models get dumber? To what extent is context length to blame vs context quality?