KV cache is, well, a cache that can fill up and trigger eviction. You require enough space to execute at least 1 fwd pass of 1 request at your context length. KV cache hits reduce TTFT by avoiding prefill. You don’t get to skip decode.
MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.
“It depends”: what’s your prior experience, what kind of roles interest you, how big is the gap between what you have + a little ML knowledge/side projects?
I’d argue there’s a big need for people with solid fundamental CS, sysadmin, infra skills who can bridge the gap into ML practitioner/researcher understanding. Applications or inference generally are probably easiest to break into, especially if you already have service knowledge. If you want to work on distributed training or kernel/model optimization, you probably need to prove your chops there.
Neoclouds, startups in the AI space, maybe hw vendors are probably good places to look.
as someone in the space this ticks a lot of boxes: kubernetes-native, strong isolation, python sdk (ideal for ML scenarios). devmapper is a nice ootb approach.
Glancing at the readme, is your business model technical support? Or what's your plan with this?
Anything interesting to share around startup time for large artifacts, scaling, passing through persistent storage (or GPUs) to these sandboxes?
Curious what things like 'Multi-node cluster capabilities for distributed workloads' mean exactly? inter-VM networking?
Oh nice! I just had an excuse to try mojo via max inference, it was pretty impressive. Basically on par with vllm for some small benchmarks, bit of variance in ttft and tpot. Very cool!
Larger memory, weaker comms. You can optimize for this by doing things like increasing batch size/data parallelism vs sharding schemes with more comms.
At scale training won’t be able to avoid comms entirely, while many models can fit in a single MI300 for serving.
can't speak to GCP specifically but usually the issue is they are host-attached and can't be migrated, so need to be wiped on VM termination or migration -- that's when you lose data.
Reboots typically don't otherwise do anything special unless they also trigger a host migration. GCP live migration has some mention of support though
note that stop/terminate via cloud APIs usually releases host capacity for other customers and would trigger data wipe, a guest initiated reboot typically will not.
yeah, the system/application distinction feels somewhat superficial. The “multiple user space” inside a container thing sounds interesting (not sure what that means exactly), but maybe more similar to a Kubernetes pod, except maybe instead of different rootfs there’s another isolation mechanism?
> Let's say our solution causes a net decrease in developer productivity of 1%
This is an extremely aggressive assumption, and affects the entire equation. What happens when you achieve parity in 1 month, because actually, docker isn't that important? nerdctl + containerd basically eliminate my need for docker in a work context. nerdctl only for my local development.
Tech companies with XX thousand employees already have dedicated infrastructure teams of all sorts. This math doesn't feel like it reflects reality of the marginal costs and payoff time.
This point comes up so often when discussing Orwell/1984 it's almost meta in itself. 1984 seems more scary to us for some reason, perhaps because it's more foreign/further from our reality than Huxley's fears/Brave New World?
Meant to reply to you but replied to parent by accident. $200k total comp seems plausible. $200k salary seems like a bit of a stretch to me. Companies would much rather give you a fat cash bonus or stock than raise the salary so high.
$200k for salary for a new grad strikes me as being on the (unrealistically?) high side. You'd need to be quite talented and have negotiating leverage for this. FAANG generally won't offer this out the gate. That kind of salary would buy senior devs no problem, even in big tech cities.
$200k total comp, absolutely. A standard new grad offer with no negotiation might be something like $120k salary, $25k cash, $120k options vesting over 4 years with 6 month periods. So in the first year, no negotiation, you're taking home $177k pre-tax. I know people who have gotten up to $60k cash bonuses (split over two years) out of college, simply by having another offer.
I really didn't mean for the comment to come across that way. I fully agreed with "it sounds great because it is great". I just wanted to provide another perspective too.
You have a great point, but let's balance this out with a few of the author's other comments.
- OP uses the phrasing senior engineer.
- Never worked for FAANG. This is relevant because $120k + bonus/benefits is basically a FAANG new grad. Fairly normal for SV tech companies.
- Those numbers likely provide a solid standard of living, but as a senior engineer you are likely underpaid.
- Esoteric and proprietary knowledge means if you choose to go elsewhere, you will be at a competitive disadvantage compared to using industry standard tools. There are of course tradeoffs and lots of general learning that comes with experience, but all other things equal it's a disadvantage.
> I guess those numbers also explain why the author can recommend maxing out the 401k
Yes, but again fairly typical for the target audience I think. There are even startups trying to target this market to optimize the flow of money from salary -> 401k -> post tax contributions/megabackdoor -> other investments, brokerage accounts, etc., e.g. https://www.helloplaybook.com/
Just as a sanity check, BLS suggests the median SWE wages work out to ~$110k.
Any chance you have more details on GP's question about the tech basis of the router (ebpf, dpdk)? I didn't find this component among the OSS in the superfly org.
MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.