Do you see much of a difference between started Chromium instances with the same configuration in terms of the contents of allocated memory? Are they deterministic?
If not, could you template the memory and apply runtime patches (like timers or other initialized values) before releasing the process to run?
Would forcing the isolates to allocate memory better help at all, such as reducing fragmentation making your 2MB page sizes more effective?
I've been working on a mesh network for private AI models running remotely, controlled by mobile devices (smartphones, tablets, etc.). The mesh is constructed like a piconet, a few devices controlled by a single individual, layered on top of the internet.
How does it support semi-connected devices, intermittent connection failures, etc?
Same. I switched my efforts to a larger Gemma 4 MoE model (26B-A4B) and llama.cpp and started getting meaningful results. I also implemented subagents for querying, determining which object/action to execute, and composing a short title. This is all running on an M4 in approximately 16 gb of ram. Also using Google's native tool calling channels.
It's basically restricting what logits are allowed when sampling the model to conform with the JSON (or whatever) shape. It can also cause the model to get "confused" though and doesn't always result in the output you want.
Doing the same for Apple M-series with fused wgsl shaders specifically targeting Qwen3/3.5.
My effort is called shady-thinker and is on github at github.com/tmzt/shady-thinker.
This was inspired in part by Antirez's earlier work with C kernels as well as other efforts to support in-browser LLMs. I've adapted them to Rust and the wgpu library.
Gemma 4 is also the next likely target (with the MTP work) as I'm experimenting with local AI agents.
I'd love to see what you've done to improve prefill and decode even if its not directly applicable.
One difference, I'm using MLX and GPTQ 4bit quants including AutoRound with safetensors as my shader pipeline is pretty much fixed for each model, ggml just adds unnecessary complexity.
I'm working on this too. I'm building a distributed environment where compute and GPU resources can be on one system, display on another, organized into a type of piconet. So far, I'm working with offline/local AI though running into limitations with it. The goal is to allow for a user to customize the environment with personalized cards powered by open schema databases. The same UI works on mobile devices, tablets, desktops, even TVs and IoT devices (ESP).
While it's true that early Linux ARM devices where embedded and generally only supported a single configuration, they didn't actually use devicetree.
Originally, embedded Linux ARM devices used a board file with a platform bus and hard-coded device metadata. The bootloader had to pass a machine id which told the kernel which hardware you were running on and which board file to use.
You can see remenants of this in the kernel still, though it's quickly being removed. I'm actually working on a hybrid kernel with the goal of bringing modern Linux support (on an lts branch) to old MSM7x300 devices, like the Evo 4G Shift I intent to use a tmux console/cyberdeck.
On another note, ACPI/UEFI doesn't always give you a clean abstract surface to work with either. ACPI is notorious for building in OS checks into it's compiled bytecode to the point that Linux often lies to it about what OS is running.
Has anybody thought about encoding AST tokens as LLM tokens, similar to how different words can have different meanings and that's reflected in their embedding?
Also working on something similar and using a dual LLM architecture (small router, larger deep thinker) with offline models, as well as determinisitic skills encoded as TSX.
It's evolved into a mesh-based operating system, gained it's own GPU-based AI library/runtime, and even molted and extended itself to ESP nodes.
Getting closer to a full release sometime in May. For now, pieces are released on my github.
Same here. Then you see SOTA in a browser from Ex0byt, online 10x training (JIT-Lora), TurboQuant (Google), etc. Just saw KV prediction mentioned in this thread, so looking into that too.
I'm adapting all of this to Rust+WGPU with compute shaders if you want to follow along.