We hit a Cloud NAT bill of ~$4,500/month (3.2 TiB/day at $0.045/GiB) on a project where we'd "enabled Private Google Access" on the subnet. The traffic was inference workloads pulling data from GCS — exactly what PGA is supposed to put on the private path for free.
The host is a multi-tenant LXC setup (Containarium — open source, many isolated user containers on one VM behind a shared NAT IP). The cost-saving reason we run it that way is exactly the same reason this incident was sneaky: ~20 workloads sharing one egress IP means VPC flow logs and Cloud NAT metrics all point at the same place, with no native GCP way to attribute traffic per workload.
Where the platform earned its keep: it tracks per-container resource stats (CPU/memory/disk/network via veth counters) and exposes them in a web UI. We sorted containers by lifetime tx bytes, and the offender jumped out — xxx-container, 1.5 TB rx since boot. Two orders of magnitude above any other container.
Without per-container traffic accounting, we'd have been left correlating VPC flow log timestamps against ps and lsof on the host — the kind of investigation that takes a day, not five minutes.
Then we looked at the destination IP from VPC flow logs: 192.178.163.207 → tt-in-f207.1e100.net. Google-owned. That misled us at first: it looked like PGA was working, just to a different Google service. It wasn't.
The actual problem: PGA has two halves and GCP only surfaces one of them in the subnet UI.
1. Subnet flag — --enable-private-ip-google-access. We had this on.
2. DNS — *.googleapis.com has to resolve to the private VIP range 199.36.153.8/30 (or 199.36.153.4/30 for restricted). Without that, storage.googleapis.com resolves to a normal public Google IP, the route to PGA never gets used, and Cloud NAT processes every byte.
The fix is a Cloud DNS private zone for googleapis.com attached to the VPC, with A and AAAA records pointing at the private VIP range (don't forget IPv6 — we hit that on attempt one and saw traffic go right back out the public path). Once that's in place, dig storage.googleapis.com from inside the VPC returns 199.36.153.11, traffic uses the private path, NAT bytes drop ~95%.
Verification one-liner from a VM in the VPC:
$ dig +short storage.googleapis.com
199.36.153.11 # private VIP — good
# vs.
142.251.x.x # public IP — your "PGA-enabled" subnet is doing nothing
The annoying part is that everything else looks correct. Subnet flag set, no external IP on the VM, destination is a Google IP, NAT gateway is healthy. There's no warning anywhere that DNS is the missing piece.
Worth checking on every VPC where you assumed PGA was doing its job.
I agree CI should catch as much as possible — image scanning and dependency checks at build time are table stakes.
But in practice, CI is only a point-in-time guarantee. A build can pass all checks and still become vulnerable later as new CVEs are disclosed.
So the goal isn’t to rely on runtime to “catch mistakes”, but to add a second layer of defense — continuous monitoring and probing for already-deployed services.
If anything, this incident showed us that CI alone isn’t sufficient once systems are long-lived.
I think both points are true in practice. Reviewing AI-generated code can require more experience than generating it, but at the same time some basic checks (dependency versions, release notes, etc.) are still worth doing.
One thing this incident reminded us of is that review is only a snapshot in time. Even if everything looks fine when a PR is merged, new CVEs can appear later and suddenly make previously safe dependencies vulnerable.
That’s why we started treating monitoring and vulnerability checks as part of the platform itself, not just the review process.
That’s a great point about fresh-context reviews — the same session that generated or assembled the code often won’t catch its own mistakes.
What worried us in this incident is that even if you catch everything during review, new CVEs can appear later and suddenly make previously “safe” code vulnerable.
That’s why we started treating monitoring as part of the platform itself — every service runs with centralized checks so unusual behavior (CPU spikes, unexpected processes, endpoint exposure) gets flagged quickly.
Totally agree — AI scaffolding automates work, but best practices like CI/CD and pentesting are still essential. Continuous monitoring is necessary for all commits, and combining it with a dev-like centralized platform ensures every service and endpoint stays safe.
CVEs are time-dependent. Even if npm audit guarantees no known vulnerabilities at the moment you merge a PR, new CVEs can emerge later, silently impacting your system without anyone realizing it.
That’s why I think continuous monitoring and centralized pentesting are essential — not just at merge time, but throughout the lifecycle of AI-generated projects.
This incident showed how AI-generated code can inadvertently introduce vulnerabilities. The cryptominer ran because a dependency version chosen by an AI coding agent had a known CVE.
Containarium now runs centralized pentests and vulnerability checks for all applications on the platform to prevent similar attacks.
Curious if others have similar workflows or lessons learned with AI-generated projects.
Good questions — yes, Containarium relies heavily on *user namespaces*. Here’s how it works:
- We enable `security.nesting=true` on unprivileged LXC containers, so Docker can run inside (rootless).
- *User namespace isolation* ensures that even if a user is “root” inside the container, they are mapped to an unprivileged UID on the host (e.g., UID 100000), preventing access to host files or devices.
This setup allows developers to run Docker and do almost anything inside their sandbox, while keeping the host safe.
Containarium does indeed build on LXC/Incus and isn’t trying to reinvent the wheel. If you’ve run multi-tenant sandboxes at scale, we’d love to hear what pitfalls or limitations you’ve seen.
Sorry, we want to understand your use case better. Did you provision *one VM via Proxmox* and then run *multiple users via Incus* inside it?
We’re curious how you handled provisioning, isolation, and resource limits in your setup. More importantly, what’s the maximum scale you’ve been able to push?
Thanks for sharing! We’re definitely aware that Incus + Proxmox are very mature and full-featured.
Containarium is more of a "purpose-built, single-VM, SSH-first dev environment" approach:
- Lightweight: 1 VM can host 50–100+ LXC containers
- Quick provisioning: seconds instead of minutes per environment
- Focused on SSH workflows and dev sandboxing, not full datacenter management
- Minimal infra overhead: no GUI, no HA cluster required
Tradeoffs we’re aware of:
- Shared kernel (not VM-level isolation)
- Linux-only
- Less built-in tooling compared to Proxmox
We designed it to *optimize for cost efficiency and rapid dev onboarding*, rather than full-featured virtualization.
Would love to hear if you see any pitfalls with this approach compared to using Proxmox/Incus in a single-host scenario!
- Did you use LXC containers, or full VMs for each sandbox?
- How did you handle SSH / network isolation?
- Any tips on making provisioning faster or keeping resources efficient?
We’re using unprivileged LXC + SSH jump hosts on a single VM for cost efficiency.
I’d love to hear what tradeoffs you found using the Proxmox API.
The idea is simple:
- One cloud VM
- Many unprivileged LXC system containers
- Each user gets their own isolated Linux environment via SSH (ProxyJump)
- Persistent storage survives VM restarts
This is NOT Kubernetes, Docker app containers, or a web IDE.
Each container behaves like a lightweight VM (full OS, users, SSH access).
Why we built it:
We kept seeing teams pay for dozens of mostly-idle VMs just to give people
a place to SSH into. Using LXC, we can host tens or hundreds of environments
on a single VM and cut infra costs significantly.
What we’re looking for:
- Feedback from people who’ve run multi-tenant Linux systems at scale
- Security concerns we might be underestimating
- Where this approach breaks down in real-world usage
- Alternatives we should be considering (LXD, Proxmox, something else?)
Tradeoffs we’re aware of:
- Shared kernel (not VM-level isolation)
- Not suitable for untrusted workloads
- Linux-only
- Requires infra discipline (limits, monitoring, backups)
This is early-stage and open source. APIs and workflows will evolve.
We’re not trying to “replace Kubernetes” — just trying to do one thing well:
cheap, fast, SSH-based dev environments.
Would love blunt feedback from folks who’ve been down this road before.