One thing I’m still unclear on: in real production workloads, what ended up being the main bottleneck first — memory bandwidth, KV cache management, or scheduler overhead?
Curious how much of this showed up only under sustained load versus benchmarks.
Standard VMs are definitely overkill for per-agent instances due to the resource overhead.
If you need strict isolation for untrusted code but want container-like speed, look into Firecracker (MicroVMs) or gVisor (userspace kernel).
Firecracker is what AWS Lambda uses. It strips down the kernel to the bare minimum, so you get VM-level isolation with millisecond boot times and a tiny memory footprint. It’s essentially the sweet spot between "insecure" Docker and "heavy" full VMs.
One thing I’m still unclear on: in real production workloads, what ended up being the main bottleneck first — memory bandwidth, KV cache management, or scheduler overhead?
Curious how much of this showed up only under sustained load versus benchmarks.