44% cache hit rate is low. Over half the expert loads are cold reads off SSD, so at 1.4 GB/s effective bandwidth and ~1.8GB I/O per token, 4.74 tok/s checks out — but it'll drop with longer context or heavier reasoning.
Running 397B on consumer hardware is genuinely impressive for a proof of concept. A year ago this wasn't a thing. But I keep wondering whether a well-quantized 70B that fits entirely in RAM would just be faster in practice. No I/O bottleneck, consistent throughput, smaller model but actually usable.
Dynamic tool registration matters more than it sounds. I've hit the context limit multiple times just from MCP servers loading 50+ tools at startup, most of which the agent never touches.
The Trello example is fine but I keep coming back to the security piece. If agents can spin up tools with credentials baked in, what's stopping a tool from dumping those credentials through its output? "Encrypted at rest" handles storage, sure. But at runtime those credentials are sitting in memory and passing through the response chain. Nobody seems to talk about that part.
Other thing — do dynamically created tools survive a restart? Or am I recreating them every session?
Running 397B on consumer hardware is genuinely impressive for a proof of concept. A year ago this wasn't a thing. But I keep wondering whether a well-quantized 70B that fits entirely in RAM would just be faster in practice. No I/O bottleneck, consistent throughput, smaller model but actually usable.