To address the earlier comment: I didn’t go down the “implement the OS-level stuff required to get k8s to compile to wasm” path due to bundle size, honestly and truly. My testing indicated it could be over 10Mb, and I didn’t want that if I could avoid it. Turns out, I could avoid it.
Whether it’s a bad idea remains to be seen. It could be! The project will never have full parity with k8s, and as k8s moves forward I probably won’t keep up. But I think the core of k8s is stable enough that that’s okay, and I can make great content about the bits that don’t change.
I’d guess that about 90% of the webernetes code is ported directly from the Kubernetes codebase. In that sense it is a port of Kubernetes to the browser. But the CRI and CNI are facsimiles created to work in the browser and for the purpose of making educational content.
The thing I think is most misleading about the title is that I haven’t ported _all_ of Kubernetes. It’s just the slice I needed. I’m hoping to gradually port more of it, though.
I’m 100% confident it would be possible to do a worker-per-pod runtime. If you’re serious about trying to make it then slide into my DMs, would love to talk about it. I also think a CRI that supports WASM would be fairly easy.
Volumes are one of those things that I’m quite sure won’t be that difficult to implement but I don’t have a need for them in the near future, so likely won’t support them for a good while.
This is it. If you’re building something novel, and you want to rely on LLMs, you need to invest heavily in making tests. You don’t have the luxury of a reference implementation like I did here.
For some projects I think only reading the tests is probably fine. In this project I didn’t think it was enough purely because it’s a port of existing code, so there was a need to validate the port was as exact as it could be.
Many projects would be just fine if you created a comprehensive-enough set of tests that you understood to be enough.
Web workers were on my mind from the start but I never found myself needing them. They were always
my ace-in-the-hole if this ended up being too CPU hungry on the main thread but it never happened, so I didn’t bother.
One of the fun things is it shouldn’t be too difficult to create a new RuntimeService that uses web workers and slots in alongside my existing CRI. I’d love a PR along those lines!
Sadly Katacoda got paywalled (totally get why they did it, these things have costs). I think some other similar platforms disappeared because they ran out of people willing to fund it. It’s a shame.
I’m hoping this offers an alternative. It has the risk of becoming out of date with reality, but at least even in that case the core should ~always be relevant.
“Zero point” is how I saw it referred to in the literature, so that’s what I went with. I personally prefer to think of it as an offset, but I try to stick with terms folks are likely to see in the wild.
Definitely could be, but in the time I spent talking to the 4-bit models in comparison to the 16-bit original it seemed surprisingly capable still. I do recommend benchmarking quantized models at the specific tasks you care about.
Thank you! I was really surprised how robust models are to losing information. It seems wrong that they can be compressed so much and still function at all, never mind function quite closely to the original size.
Think we're only going to keep seeing more progress in this area on the research side, too.
I wrote a tool called llmwalk (https://github.com/samwho/llmwalk) that’ll deterministically show you the likelihood the top N answers are for a given open model and prompt. No help on frontier models, but maybe helpful if you want to run a similar analysis more quickly on open models!