Mesh LLM: computação de IA distribuída no iroh(iroh.computer)
iroh.computer
Mesh LLM: distributed AI computing on iroh
https://www.iroh.computer/blog/mesh-llm
47 comments
I've been looking for similar distributed computing style LLM, and I found AI Horde and a few other smaller efforts like one from Aphrodite people and distributed training from Nous Research.
AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.
I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.
AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.
I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.
I'm more interested in running distributed inference for purpose built small language models than these coding LLMs.
Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.
Nicely done OP.
Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.
Nicely done OP.
I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
This was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN.
The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others.
I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others.
I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
What hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
The lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet.
We use a customized Q2 quantization that preserves sensitive tensors at Q8.
To reduce compute time per layer, we are developing a custom GLM DSA Metal graph.
While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2.
Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.
We use a customized Q2 quantization that preserves sensitive tensors at Q8.
To reduce compute time per layer, we are developing a custom GLM DSA Metal graph.
While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2.
Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.
Equivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each
That sounds cool, but it's still pretty meaningless without information about what your home lab looks like. A few DGX Sparks wired up with their fancy super fast network is much different than a few laptops on wifi.
That's about the speed I get on a AMD Ryzen AI 9 HX 370 (inside a Framework 13), with Qwen3.6-35B-A3B, so doing the same on that much larger model...
Perf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.
If you split sequential layers across GPUs, latency is often the main constraint, not bandwidth.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
What is the incentive for me to join the public mesh? Do you have any fairness guarantees, e.g. if I contribute 1/8th of the VRAM required to run a particular model, do I get at least 1/16th of the inference share, or anything similar to this?
I have never really delved into kv cache implementation, do they run effectively separate caches per layer?
If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.
Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.
Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
Each stage has its own KV for the layers it hosts. You are on the money there, when one stage is waiting it's free for more parallelism. I am planning on exploiting this for more token verification through ngram spec decoding.
I wonder how security is done in this engine, since it's accepting input from anyone. llama.cpp's RPC layer seems to says that you shouldn't run it in public (I assume because it is lower level and may result in RCE on your GPU)
This is super impressive, We have a lab with lots of different epycs and different models - to bring them together this way is amazing. Well done!
Thank you! AMD is a weak spot in our testing right now. If you’re willing to contribute or let us borrow some compute time, drop in on the Discord.
The obvious burning question is how performance looks over different network conditions on some standard models. Have you done much benchmarking? Is it mainly latency affected or is overall throughput less than the capacity of the GPUs due to being distributed?
Hey, this is a super cool project. It's great to see a lot of the IPFS stuff resurfacing again.
A few questions:
1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.
2.) Any safeguards against malicious actors poisoning model activations?
A few questions:
1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.
2.) Any safeguards against malicious actors poisoning model activations?
To be honest, both are very tough problems we don't have a good answer for yet. If that is something that concerns you, look into building a private mesh with trusted peers.
Thanks for answering, that makes sense. Also - your setup seems like it could greatly benefit from speculative decoding. Have you guys given any thought to how that might work in this system?
P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.
P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.
Curious about: does it have fault tolerance if one of the machines goes down mid-inference? Can it dynamically reroute, or does it just retry?
It can dynamically route. If a machine drops out of split, the topology is recalculated and the request is automatically retried.
Is skippy related to a certain beer can?
I’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow.
What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.
What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.
"Point any OpenAI client at [...] and stop caring where the work actually happens."
Sounds great but unfortunately in this here world these days I think you have to KYC (know your computing)
Sounds great but unfortunately in this here world these days I think you have to KYC (know your computing)
I thought about this too, but the throughput over a network is incredibly slow. It’s not usable for interactive use.
That isn’t true. llama RPC is incredibly slow but staged splits in skippy are orders of magnitude faster.
Does Mesh LLM encrypt the payload between nodes? Is it possible to read requests from other users?
I'm not affiliated, but yes – the main 'point' of iroh is that it's 'dial-a-key', QUIC with encryption based on the keys of the endpoints.
Just wondering, why do you care about encryption in this context?
If payloads to LLMs are being passed around to various nodes, even trusted ones (like friends and family), it gets awkward if you send something very personal. Think sending a medical question to medgemma:27b.
Even if transport is encrypted, the LLM computations will always be clear text, right?
The real test is throughput. I'd like to see tokens/sec at higher concurrency and with uneven hardware.
It sounds like iroh enables distributed compute without having to finangle custom hardware.
cocompute.ai is already doing this really well.
Is it? I don't see anything on the website about splitting a model across multiple devices, only about putting local models on the internet, a wholly orthogonal problem (which is already easy with existing tools, since models use an http API).
Good point. I know cocompute is working on splitting, but it’s not there yet; I was referring to the round-robin delegation within a trusted pool. Mesh LLM looks great too!
Cool, always good to have more in the ecosystem. I love Iroh and hope this continues to succeed.
Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.
difference between this and Exo?
I just wish I had the hardware to try it out!
As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.