This is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM
Yeah, this is one area we’re struggling with due to the sheer volume of variations and conditions, but I’ve been thinking of collecting some real-time statistics around latency, prefill/decode, and model distribution… that way we can update some kind of live + aggregated performance numbers for interested parties.
Great question! We’ve had a lot of discussions about the direction we want to take this, and how to best generate some kind of incentive / fairness reward.
And we’ve found ourselves hesitating on a direction because, at least for now, the primary use case in a useable env is private hosts you own in a mesh; which makes any sort of reward/ incentive structure somewhat unnecessary (for this setup).
When the public mesh becomes large enough and we get around to extending the existing “mesh governance” features is likely when we’ll add something like you’ve described.
For now, the public mesh is totally open with no restrictions or limitations.