If you are building on top of multiple LLM APIs or even a single one amongst OpenAI, Claude, Gemini, etc. what do you do when the API starts degrading (slow TTFT, elevated error rates, timeouts). Or even worse, when there are responses but the model is drifting. How do you find this out? I'm trying to understand if this is a widespread pain or just something I've been unlucky with.
Four specific questions:
1. When an LLM API starts silently degrading, how do you currently find out? (Your own monitoring? User complaints? Checking the status page? Reddit?)
2. How long does it typically take you to confirm "this is the provider, not my code"?
3. If something told you before you noticed, that Claude API was showing elevated TTFT on Sonnet right now, would that change anything about how you operate? Or would you just retry and move on regardless?
4. Would you pay for an independent alert service that tells you when an LLM's behaviour has drifted, before your users notice?
If this isn't actually a problem for you, I think that also would be the most useful answer I can get.
Four specific questions:
1. When an LLM API starts silently degrading, how do you currently find out? (Your own monitoring? User complaints? Checking the status page? Reddit?)
2. How long does it typically take you to confirm "this is the provider, not my code"?
3. If something told you before you noticed, that Claude API was showing elevated TTFT on Sonnet right now, would that change anything about how you operate? Or would you just retry and move on regardless?
4. Would you pay for an independent alert service that tells you when an LLM's behaviour has drifted, before your users notice?
If this isn't actually a problem for you, I think that also would be the most useful answer I can get.