I wrote a deep dive on how modern reverse proxies handle thousands of concurrent requests.
The article walks through what actually happens inside a proxy when multiple requests arrive at the same time: connection handling, request queues, event loops vs thread pools, and how different implementations approach the problem.
It also compares design patterns used in systems like Apache Traffic Server, HAProxy, and Envoy, and explains why these architectural choices matter for latency, throughput, and failure behavior.
The goal was to help zoom in what happens between the client and backend when traffic spikes.
Would love feedback from folks running proxies at scale.
Resolver-level resilience is often manageable centrally. The harder part is application-level recovery; especially in larger orgs where DNS behavior spans multiple teams.
Even with low TTLs or cache flush automation, apps may: resolve once at startup or hold long-lived gRPC/TCP connections, or even worse -> ignore TTL semantics entirely
So infra assumes "DNS healed," but the app never re-resolves.
Nice. Running Unbound locally with intelligent upstream selection and caching definitely reduces blast radius from edge resolver outages.
I haven't tried Unbound but I’m curious though, how do you handle recovery behavior when the failure isn’t just recursive resolver unavailability, but scenarios like stale IPs after control plane failover, or long-lived gRPC connections that never re-resolve, or bootstrap loops where the system that needs to reconfigure DNS itself depends on DNS?
In my experience, local recursive resolvers solve availability pretty well, but recovery semantics still depend heavily on client behavior and connection lifecycle management.
Do you rely on aggressive re-resolution policies at the application layer? Or force connection churn after TTL expiry?
Would love to understand how you think about resolver-level resilience vs application-level recovery.
DNS keeps showing up in outage postmortems, but what's often missing is discussion about recovery, not just prevention.
In this post, I break down common DNS failure patterns (TTL propagation, resolver overload, control plane dependency loops) and why recovery can deadlock when your tooling itself depends on DNS.
I'd love to hear how others design around this:
Do you use DNS-independent fallbacks?
Static seed lists?
Separate control plane resolution?
Aggressive caching vs short TTLs?
Curious what patterns have worked (or failed) in real systems for folks.
DNS works well in certain scenarios (multi-region failover, coarse traffic steering), but I've seen it misapplied in cases where faster failure detection or finer-grained routing was required.
This piece tries to outline where DNS fits well architecturally and where it starts to show limits.
For client-side LB, moving active healthcheck outside into dedicated service, wouldn't it create more reliability issues with one more service to worry about? Are there any examples of this approach being used in the industry?
> when a single endpoint in a service begins having high latency
Yes, have seen this first hand. Tracking the latency per endpoint in a sliding window helped in some way, but it created other problems for low qps services.
Agree - sliding window error rates plus client-side circuit breakers (with half-open probes and ramp-up) work really well in practice, and the recovery-speed point is especially important.
The only nuance I was trying to call out is what happens at very large scale. These mechanisms operate per client instance, so each client needs a few failures before it trips its breaker and then runs its own probes and ramp-up. That's perfectly reasonable locally, but when you have hundreds or thousands of clients, the aggregate "learning traffic" can still be noticeable. Each client might only send a little bad traffic before reacting, but multiplied across the fleet it can still add up. Similarly, recovery can still produce smaller synchronized ramps as many clients independently notice improvement around the same time.
So I tend to think of client-side circuit breakers as necessary but not always sufficient at scale. They're great for fast local containment and tail-latency protection, but they work best when paired with some shared signal (LB, mesh control plane, or similar) that can dampen the aggregate effect and smooth recovery globally.
The idea is attractive (especially for draining), but once you try to map arbitrary inbound client connections onto backend-initiated "reverse" pipes, you end up needing standardized semantics for multiplexing, backpressure, failure recovery, identity propagation, and streaming! So, you're no longer just standardizing "reverse HTTP", you’re standardizing a full proxy transport + control plane. In practice, the ecosystem standardized draining/health via readiness + LB control-plane APIs and (for HTTP/2/3) graceful shutdown signals, which solves the draining problem without flipping the fundamental accept/connect roles.
I wrote this after seeing cases where instances were technically “up” but clearly not serving traffic correctly.
The article explores how client-side and server-side load balancing differ in failure detection speed, consistency, and operational complexity.
I’d love input from people who’ve operated service meshes, Envoy/HAProxy setups, or large distributed fleets — particularly around edge cases and scaling tradeoffs.