As LLMs continue to diversifying, model routers become increasingly critical for connecting various models and achieving the best performance-cost trade off. RouterArena is the first open platform & leaderboard for rigorous and comprehensive router evaluation, supporting both open-source and commercial routers. It provides principally-constructed datasets, extensive evaluation metrics, an automated evaluation framework, and a live leaderboard. We aim for RouterArena to serve as a foundation for the community to evaluate, understand, and advance the next generation of routing systems.
Sharing the big GPU cluster with non-latency critical load is one solution we also explored.
For this work, we are targeting more on the problem of smaller models running SOTA GPUs. Distilled/fine-tuned small models have shown comparable performance in vertial tasks.