The debugging part at this scale is harder than you would expect - behavioral drift between parallel agent instances is nearly invisible without something aggregating what they are actually doing across runs. We hit this ourselves: two agents completing the same task successfully via completely different paths, one of which quietly broke edge cases in prod. The only thing that caught it was treating the conversation traces as a dataset, not just logs.
I think one of the very few who actually support ebpf & xdp, which you do need when you're building low level stuff. + the bare metal setup is like out of the world lol.
Runtime policies as an actual gate rather than prompt instructions is the right model. Most frameworks just bolt governance on as a wrapper and hope the model obeys. What I'd want on top of this: observability into why agents are hitting policy blocks, not just that they were blocked.
The oracle problem is tractable when the output is code: you can compile it, run tests, diff the output. For conversational AI it's much harder. We've seen teams use LLM-as-judge as their validation layer and it works until the judge starts missing the same failure modes as the generator.
The MoE point matters here ie sparse activation means you're not reading all 2TB per forward pass, but the access pattern flips from sequential to random which is exactly the worst case for NVMe. Been thinking about this a lot for agent inference workloads where you want consistent latency more than peak throughput.
We've seen this exact pattern. Most devtools assume a human will eventually log in and contextualize the data. When the 'user' is an agent, you need the surface to be machine-readable by default, not as an afterthought. The adapter approach mostly doesn't work ie you end up with a translation layer that loses exactly the signal you needed.
4.4 tok/s with reliable structured output is a solid local benchmark altho the question is whether SSD streaming introduces per-token latency variance that messes up tool call parsing downstream. The gap between 400 GB/s unified memory bandwidth and 17.5 GB/s SSD reads means you're in the hot path pretty much every time an expert isn't cached.
we've started to document any a/b decision we take in terms of tech and store in our engineering internal docs! have gone back to it once in a whilw but that usually helps keep us grounded
IMO the under-discussed risk here is that sites will start serving different content to verified crawlers vs real users. You're already seeing it with known search bots getting sanitized views. If your agent's context comes from a crawl the site knows is going to an AI, you have no guarantee it matches what a human sees, and that data quality problem won't surface until your agent starts acting on selectively curated information.
Hard limits are a good first layer but they don't tell you why the agent is looping. Retrying because it's confused, retrying because a dependency is flaky, and genuine planning loops are three different problems with different fixes. What helped us was logging the agent's intent at each step, and if it's asking the same underlying question three times in different syntax, that's the signal to bail early rather than burning through your iteration budget.
The BCG framing makes it sound like a cognitive load problem but I think it is more unreliability fatigue. When your AI does 8 things right and then confidently does the 9th wrong, you spend mental energy second-guessing everything. Supervising an unreliable system is more exhausting than just doing the task yourself.
Same mental model problem comes up in AI agent observability. Two conversation flows can produce identical user outcomes and look totally different at the message level, or vice versa. The normalization step that actually captures 'did behavior change' is the hard part in both domains.
The latency point matters more than it looks imo like the GPU work isn't just async CPU work at a different speed, the cost model is completely different. In LLM inference, the hard scheduling problem is batching non-uniform requests where prompt lengths and generation lengths vary, and treating that like normal thread scheduling leads to terrible utilization. Would be curious if Eyot has anything to say about non-uniform work units.
Lol the joke works because it's halfway serious. Agents already choose tools, vote with API calls, and quietly churn off platforms that create friction. "What do agents want" is actually a real product question and imo most teams shipping agent products have zero visibility into how their agents are actually behaving in production. What do you think?
The pattern is always the same: product team has a new AI feature, someone says put it everywhere, nobody asks users if they want it, and then power users revolt while the feature languishes unused anyway. The right move was opt-in, but that would've made the adoption numbers look bad. Progressive disclosure exists for a reason.
The "trust our logs" problem is real ie regulators and security teams don't care about your dashboard. Curious about the semantic layer though: once you can verify a log is intact, the next hard question is why the agent made the specific decision that caused an incident. Integrity proves the what, but you still need the interpretability layer for the why.
I'd like to believe that vast knowledge is still more than what I have in my mind?