The blog's title can be misleading here, "we" in this context refers to the Cognition team. I don't work at Cognition, just thought this was interesting.
> On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected.
Interesting, this implies that the 1M context servers performs worst at low context.
Perhaps this is due to some KV cache compression, eviction or sparse attention scheme being applied on these 1M context servers?
They sounded a tinge strange, like they’ve almost crossed the uncanny valley, only to succumb at the final 3% stretch.
I was suspicious, but their ability to understand my complex request and the relatively low latency make an LLM -> TTS or e2e voice model unlikely.
This post finally solved the mystery.