Client-side GPU load balancing with Redis and Luagalileo.ai54 points·by lneiman·7 माह पहले·11 comments
lneiman·7 माह पहले·discussAuthor here. We were hitting tail latency and low GPU utilization issues serving SLMs via Triton.I built a scrappy client-side router using Redis and Lua to track real-time GPU load. It boosted utilization by ~40% and improved latencies.Happy to hear feedback on the implementation or thoughts on better ways to do this!