Indeed - definitely not cost effective to run it on this laptop LOL. It makes me wonder how fast we could run the model if we could fit the weights entirely within CPU cache (assuming a whole ton of CPUs with low latency & high speed IO of course).
I ran it on my laptop, which is a Lenovo Legion 5i (think 32 GB RAM, 4060 w/ 8 GB VRAM, you get the picture). It was a quantized model (otherwise it would not fit on my NVMe 1TB drive) at 4 bits per weight - UD_Q4_K_XL. It ran at about 12 seconds per token (not tokens per second). A fun project, but not worth it. I used 4096 tokens of context cache, and I ran it with llama.cpp - as it supports memory mapping. Because the whole thing could obviously not fit in RAM, I was curious how much it would need to stream from SSD. The answer? For a simple 4 sentence description of who it was, about 1.5 TiB was streamed from disk.
OpenZL is nice, but it's often less useful than you think - it requires that you know the structure of your data, and don't care about inspecting that data outside of your program. I've extracted one too many png files from a word document (by renaming .docx to .zip) to desire OpenZL everywhere... It might be better as a short-term "data in transit" compression than for long term storage.
"open source" means that the code itself (for LLMs - this is training code) is available to the general public. "open weights" means that the weights (trained over time) are available publicly, rather than locked behind a paywalled chat. I do not know of an open source LLM that is not also open weights (unless they never bothered training it). Models like Claude and Gemini are neither open source, nor are they open weights.
Attemping to write my own CDCL SAT solver right now. I've experimented in the past with a DP & DPLL SAT solver. I'm currently somewhat mentally stuck on how to create the derived clause after a conflict, but I'll get there :)
I hope that this brings out a bunch more real study about the qualitative metrics of these models, both to increase the confidence and accessibility of local LLMs, but also to reduce the blind worship that seems to be propagating about their miracle work in all domains.
No questions, but I really enjoyed the article - thank you for sharing. It amazes me how few vacuum tubes these early computers use, compared to the billions and trillions of mosfet transistors used in modern devices.
I'm very pleased with the performance of the largest gemma4 model (which I tested through ollama). My singular data point on whether an LLM remembers things well is whether it can translate toki pona to (and from) English. I find it easy to evaluate because I know the language. This local LLM marks the first version that 1) doesn't hallucinate words - at least, for the largest model - and 2) uses common word-phrases that other toki pona speakers use, and most importantly 3) can actually run on my laptop.
There's solid reason for the translation here; the Zig core team is aiming to eliminate duplicated code and C functions, and avoid the need to track libc from multiple sources. In the future, LLMs could serve as part of this, but they are currently quite terrible at Zig (as far as I understand it, it's not a lack of Zig code samples, it's an imbalance of OLD Zig to NEW Zig, as Zig changes quite frequently).
You would need to consider if it is even worth it translating your C code. If the paradigm is identical and the entire purpose would be "haha it is now one language," surely you could just compile and link the C code with libzigc... In my opinion, it's not worth translating code if the benefit of "hey look one language" requires the cost of "let's pray the LLM didn't hallucinate or make a mistake while translating the code."
In the theoretical world where a subset of English could be formalized and proven and compiled, the complexity of the language would reduce my willingness to use it. I find that the draw of AI comes from it's "simplicity," and removing that (in favor of correct programs) would be pointless - because such a compiler would surely take forever to compile "English" code, and would not be too different from current high level languages, imo.