How are people debugging multi-agent AI workflows in production?(agentsentinelai.com)
agentsentinelai.com
How are people debugging multi-agent AI workflows in production?
https://www.agentsentinelai.com/
13 comments
[deleted]
OTEL and LGTM, the same open source o11y stack I use for everything
Interesting — are you instrumenting the agent workflows themselves with OpenTelemetry spans?
I was wondering how well the standard o11y stack works once agents start running multi-step workflows (agent → tools → other agents → APIs). Tracing probably helps visualize the steps, but I'm curious how people handle operational things like retries, replaying failed workflows, or containing cascading failures across agents.
Those reliability aspects are what I've been exploring.
I was wondering how well the standard o11y stack works once agents start running multi-step workflows (agent → tools → other agents → APIs). Tracing probably helps visualize the steps, but I'm curious how people handle operational things like retries, replaying failed workflows, or containing cascading failures across agents.
Those reliability aspects are what I've been exploring.
I use ADK, a well thought through framework by a company that leads in both AI and engineering. They are the only company in existence with the experience in both to build a solid, production worthy agent framework. It does nearly everything you talk about, with what remains being at the agent design level and not the purview of any framework imo, i.e. wiggum loop level.
People too focused on AI and what's popular are missing the forest for a tree. Most of the things I see being talked about as problems & solutions have already been taken care of with mature tooling choices and engineering practices. History and archeology are underrated skills in software.
People too focused on AI and what's popular are missing the forest for a tree. Most of the things I see being talked about as problems & solutions have already been taken care of with mature tooling choices and engineering practices. History and archeology are underrated skills in software.
OpenTelemetry and standard observability stacks are great for seeing the latency and token counts of individual LLM calls, but they break down when you try to debug the coordination between agents.The hardest failure mode we've had to debug isn't a single agent hallucinating; it's Agent A correctly doing its job, but passing slightly malformed state to Agent B, which then confidently executes a destructive action based on that bad state. By the time you see the error, the root cause is three steps up the chain.Tracing doesn't solve this because it just shows you the execution path, not the authority boundary. What you actually need is a way to enforce contracts between agents—an execution layer that says "Agent B cannot accept this payload from Agent A unless it meets X criteria, and if it fails, rollback Agent A's last action." Until we treat multi-agent systems as concurrent state machines rather than just chained API calls, debugging them is going to remain a nightmare.
The “authority boundary” framing is really helpful — tracing explains what happened, but not whether a transition between agents should have been allowed.
Curious how teams are handling this today — are those contracts usually defined explicitly (schemas / validators), or are they mostly implicit in the agent code and discovered only after failures?
Curious how teams are handling this today — are those contracts usually defined explicitly (schemas / validators), or are they mostly implicit in the agent code and discovered only after failures?
If you can't trace across agents (like services), then you haven't set up OTEL completely
What your hard fail is, that's at a different layer of control, separate from OP questions about just seeing it so you can design those control systems. That's more guards, validators, and the like (more subagents)
I stay more human in the loop because these things are not ready for prime time the way you describe using them. That's burning tokens on average imo.
What your hard fail is, that's at a different layer of control, separate from OP questions about just seeing it so you can design those control systems. That's more guards, validators, and the like (more subagents)
I stay more human in the loop because these things are not ready for prime time the way you describe using them. That's burning tokens on average imo.
That makes sense — sounds like a lot of this is handled at the framework + design level in your setup.
In practice, when something does go wrong in a multi-step workflow, do you typically rely on tracing + manual debugging, or do you have built-in mechanisms for partial replay / recovery?
In practice, when something does go wrong in a multi-step workflow, do you typically rely on tracing + manual debugging, or do you have built-in mechanisms for partial replay / recovery?
The #1 production failure I've seen in multi-agent systems: state collision. Two agents read shared context at nearly the same time, process independently, then one overwrites the other. Zero errors, plausible output, wrong result.
It looks like a "model quality" issue but it's actually a concurrency bug. Adding more agents makes it worse.
The fix that worked for us: atomic state coordination — propose/validate/commit cycle for every shared state mutation. Open-sourced it: https://github.com/Jovancoding/Network-AI
It looks like a "model quality" issue but it's actually a concurrency bug. Adding more agents makes it worse.
The fix that worked for us: atomic state coordination — propose/validate/commit cycle for every shared state mutation. Open-sourced it: https://github.com/Jovancoding/Network-AI
This is a great example — feels very similar to classic lost update problems in distributed systems. The propose/validate/commit cycle makes a lot of sense.
Curious how you're handling this in practice — are all shared state mutations going through that flow, or only critical paths? And does the coordination overhead become a bottleneck as workflows scale?
Curious how you're handling this in practice — are all shared state mutations going through that flow, or only critical paths? And does the coordination overhead become a bottleneck as workflows scale?
Once agents start calling tools, APIs, and other agents in a chain, debugging failures becomes surprisingly hard. A single task can involve multiple steps—LLM calls, tool invocations, retries—and when something breaks it's often difficult to understand exactly what happened or where the failure originated.
In traditional distributed systems we eventually built things like tracing, circuit breakers, retry policies, SLOs, and other reliability primitives to operate systems safely in production.
I'm curious how people building agent systems today are handling this.
Some questions I'm particularly interested in: - How do you debug agent failures? - Do you have visibility into multi-agent workflows? - How do you replay or reproduce failures?
I've been exploring this problem space and built a small prototype to experiment with reliability tooling for agent systems. The link above shows the demo, but I'm mainly interested in learning how others are approaching this problem.