I wrote this after building an AI gateway layer for ModelRiver to handle streaming responses, retries, and provider failover across LLM APIs. Phoenix supervision trees ended up being especially useful for isolating failures between model calls. Curious how others are structuring gateway layers in front of LLM providers.
I’ve worked on and shipped a few AI systems that reached real users.
This post isn’t about models or prompts. It’s about the things that kept breaking once AI moved off the happy path: async jobs, retries, silent failures, provider outages, cost blowups, and debugging without visibility.
I wrote this mostly as a way to document the mistakes I made and what I wish I had known earlier. Happy to answer questions or dig deeper into any of the failure modes.
This is one of the harder problems, and there isn’t a perfect answer.
The main thing we try to avoid is pretending mid-stream retries are the same as pre-request retries. Once a stream has started, we treat it as a sequence of events with checkpoints rather than a single opaque response. Retries are scoped to known safe boundaries, and anything ambiguous is surfaced explicitly instead of silently re-emitting tokens.
In other words, correctness is prioritized over pretending the stream is seamless. If we can’t guarantee no duplication, we make that visible rather than hide it.
This was something we were careful about. The request and event models are intentionally close to what most providers already expose, rather than introducing a completely new abstraction.
Teams usually integrate it incrementally in front of existing calls. If you remove it, you’re mostly deleting the orchestration layer and keeping your provider integrations and client logic. You lose centralized retries and observability, but you’re not stuck rewriting your entire request model.
If adopting it requires a full rewrite, that’s usually a sign it’s being applied too broadly.
I’m one of the builders. Once AI requests moved beyond simple sync calls, we kept running into the same problems in production: retries hiding failures, async flows that were hard to reason about, frontend state drifting, and providers timing out mid-request.
This page breaks down the three request patterns we see teams actually using in production (sync, async, and event-driven async), how data flows in each case, and why we ended up favoring an event-driven approach for interactive, streaming apps.
Happy to answer questions or go deeper on any part of the architecture.
Totally agree, that “what actually hit the wire?” view is critical once things go async.
ModelRiver already has this covered via request logs. Every request captures the full lifecycle, the exact payload sent to the provider, streaming chunks as they arrive, partial responses, errors, retries, and the final outcome. Even if a stream fails midway, you can still inspect what was sent and what came back before the failure.
So you can clearly tell whether the issue is payload shape, provider throttling, or a mid stream failure, before any retry or failover logic kicks in. That wire level visibility is core to how we approach debugging async AI requests.
If streaming behavior is still product-specific and changing fast, this adds friction. It only pays off once failure handling stabilizes and starts repeating across the system.
Queues work well before or after a request, but they’re awkward once a response is already streaming. This layer exists mainly to handle failures during a stream without spreading that logic across handlers, workers, and client code.
While building AI features that rely on real-time streaming responses, I kept running into failures that were hard to reason about once things went async.
Requests would partially stream, providers would throttle or fail mid-stream, and retry logic ended up scattered across background jobs, webhooks, and request handlers.
I built ModelRiver as a thin API layer that sits between an app and AI providers and centralizes streaming, retries, failover, and request-level debugging in one place.
It’s early and opinionated, and there are tradeoffs. Happy to answer technical questions or hear how others are handling streaming reliability in production AI apps.
That’s fair, Rails + Hotwire is excellent for building fast MVPs. What really set Phoenix apart for me was the built-in real-time layer (via channels), the BEAM’s fault tolerance and concurrency, and how effortlessly it scales with background jobs, PubSub, and LiveView without needing extra services. Rails can absolutely do all that, but Phoenix gives you those capabilities natively with far less setup and overhead. I appreciate your feedback, I’ll definitely try to add more details in the next one. Also, I’m not implying Phoenix is better than Rails or Laravel.
Absolutely, any frontend, LiveView or React, can get messy if not carefully maintained. As the app grows and more developers contribute, regular code reviews and removing unused logic are essential, otherwise DX suffers just like in any other framework. I also agree there is still so much room for improvement in this space.
True, I just liked the gradient for branding, and I love how it looks. From your perspective, you’re right though. At least I can assure you there’s no AI-generated code on the landing page. I’ll keep working to improve it as much as possible. Thanks mate :)
Author here: Not sure why everyone’s taking this as anti-rails or anti-laravel. It’s not. I just shared what worked best for my use case. Real-time updates are built into phoenix through channels and liveview, while in rails it’s handled through Action Cable and Turbo Streams. Both work great, but phoenix’s setup felt more integrated for what I was building.
true, there’s always more to learn. i just shared what worked best for my use case with liveview. rails is also a great framework and i enjoy using it, but liveview’s built-in real-time updates and state management fit my project needs better.
Thanks for your feedback! Yes, Oban is a third-party package, but it’s built for Phoenix, which is what I meant in the article. Also, the article is not AI-generated. I’ve mentioned the app I built with Phoenix at the bottom to give context from real experience.