Scaling to 1M concurrent sandboxes in seconds(modal.com)
modal.com
Scaling to 1M concurrent sandboxes in seconds
https://modal.com/blog/scaling-to-1-million-concurrent-sandboxes-in-seconds
15 comments
I'm a huge scheduling nerd, and the container scheduling system in this post is probably the most impactful system I've worked on. It's quite different than existing solutions, and I personally feel it's at an interesting point in the design space -- very distributed, no strong consistency anywhere, and oriented towards massive scales. Would love to hear feedback and thoughts!
> Rather than a single, serialized scheduler, we run a fleet of scheduling servers which handle sandbox creation requests concurrently. To handle a creation request, a scheduling server runs a fast scheduling algorithm against in-memory cached data. The result is that scheduling scales horizontally, and looks more like load balancing than traditional container scheduling.
What does this mean? You bucket requests on some attribute and use that to route the request (or create an ordered list of routes to try)?
What does this mean? You bucket requests on some attribute and use that to route the request (or create an ordered list of routes to try)?
We make a probabilistic routing decision based on worker load and attributes of the sandbox request. I compare to a load balancer because it's essentially just forwarding an HTTP request.
Definitely interesting to read about. It seems like it has some similarities to the approach Fly.io blogged about here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchest...
Sounds like the lesson learned is using the right tool for the job -- reusing Kubernetes in an existing cluster to spin up sandboxes is a fair initial path to start offering the service. But Kubernetes likely isn't meant for rapid churn of workloads, here sandboxes.
The architecture to me seemed very similar to SeaweedFS [1] (Facebook Haystack [2]) except with an extra layer for sandbox-hosting nodes. Like requests go into a master, or the global load balancer, then to a volume server, which in turn knows where the files/sandboxes should go. There is no need for sandboxes to be managed with the Kubernetes overhead since the the nodes/bare metal servers probably have scheduling taints on them to preserve the memory/cpu for the sandboxes.
[1] https://github.com/seaweedfs/seaweedfs
[2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...
The architecture to me seemed very similar to SeaweedFS [1] (Facebook Haystack [2]) except with an extra layer for sandbox-hosting nodes. Like requests go into a master, or the global load balancer, then to a volume server, which in turn knows where the files/sandboxes should go. There is no need for sandboxes to be managed with the Kubernetes overhead since the the nodes/bare metal servers probably have scheduling taints on them to preserve the memory/cpu for the sandboxes.
[1] https://github.com/seaweedfs/seaweedfs
[2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...
They never used K8s for this.
They did reference it as an example for how a non-specialized solution would fall over.
> Modal’s original sandbox architecture has similar issues. Like Kubernetes, we rely on strong consistency throughout our backend, so creating and scheduling sandboxes requires global coordination, and O(sandboxes) writes to Postgres, which we cannot trivially shard.
They did reference it as an example for how a non-specialized solution would fall over.
> Modal’s original sandbox architecture has similar issues. Like Kubernetes, we rely on strong consistency throughout our backend, so creating and scheduling sandboxes requires global coordination, and O(sandboxes) writes to Postgres, which we cannot trivially shard.
Every scheduler node has cached view of whole cluster and optimistically makes a scheduling decision, retrying on conflict?
Any tricks you did to reduce conflict rate? Is there a certain cluster saturation threshold (little free capacity) where conflict rates would get too high?
Any tricks you did to reduce conflict rate? Is there a certain cluster saturation threshold (little free capacity) where conflict rates would get too high?
We try to spread out sandboxes evenly across the cluster (at least, across the workers which are available to take new sandboxes) to minimize conflict. But in general we don't get close to saturation thresholds so high that conflict becomes a problem, except during massive load tests. I suspect we'd see issues around 90% effective utilization.
Makes sense thanks!
You can scale firecracker vms like really fast. In our shop we have a simple go service that maintain the state in a sqlite database. The only requirement is bare-metal servers.
> While we do need to write sandbox metadata and results to durable storage, we do so largely asynchronously.
How do you guarantee durability of sandbox task metadata if it’s written to durable storage async? What if the node it’s scheduled on goes down right after scheduling completes - what service durably knows about the intended state of the sandbox and retries scheduling?
How do you guarantee durability of sandbox task metadata if it’s written to durable storage async? What if the node it’s scheduled on goes down right after scheduling completes - what service durably knows about the intended state of the sandbox and retries scheduling?
Do you binpack containers such that you overcommit cpu/ram on the machines to drive up utilization?
Did you do any simulations to see if this optimistic distributed scheduling approach maintains on-par utilization and low preemption rates to a non-distributed scheduler?
Did you do any simulations to see if this optimistic distributed scheduling approach maintains on-par utilization and low preemption rates to a non-distributed scheduler?
What was the hardest part/most unexpected design challenge in getting this to work?