Backstory: I was building a customer support AI for a client last year. We started with Claude Opus for everything because it worked great. The bill was $250/month for maybe 10K conversations.
Then I looked at the actual queries. 70% were things like "what are your hours?" and "how do I return something?" — questions where a $0.80/M-token model gives the same answer as a $15/M-token model. But about 5% were genuinely complex (multi-step troubleshooting, product comparisons requiring reasoning) where Opus was noticeably better.
I started manually routing: simple patterns to a cheap model, everything else to Opus. The bill dropped to $40/month with no quality complaints from users. But maintaining the routing logic across projects got tedious — every new app needed the same classification + model selection + failover logic.
So I built Komilion to package it up. The classification runs in two stages:
1. A regex fast path catches ~60% of requests instantly (greetings, FAQ patterns, simple classification tasks). Zero API calls, under 5ms.
2. For the rest, a lightweight LLM classifier determines task type and complexity, then matches against a routing table built from LMArena and Artificial Analysis benchmark data.
What surprised me in the benchmark data: complex tasks through the router actually produced MORE detailed output than any single pinned model (6,614 chars avg vs 3,573 for Opus). The router selects specialized models per task type rather than using a generalist model for everything.
Stack: Next.js on Vercel, Neon PostgreSQL, OpenRouter upstream. Total hosting cost ~$20/month. It's a solo project.
The thing I'd do differently: I should have started with the benchmark data instead of building the product first. The numbers make the case better than any feature list.
Happy to answer technical questions about the routing logic, benchmark methodology, or anything else.
Nice dashboard. Cost visibility is the first step — you cannot optimize what you cannot measure.
We have found that the natural next step after tracking is automated routing. Once you see that 70% of your frontier model calls could have been handled by something much cheaper, you want a system that does that automatically.
At Komilion we route each API call to the cheapest model that fits based on benchmarks. The combination of monitoring (like this) + routing gives you both visibility and automatic savings.
One thing I would suggest adding: cost-per-quality metrics. Raw cost is not the full picture — $0.80/M tokens that fails 20% of the time is more expensive than $2/M tokens that works reliably.
This is a nice approach — using model internals to predict success before committing to full generation.
We took a different path at Komilion: classify the prompt upfront (regex fast path + lightweight LLM classifier) and route to the cheapest model that benchmarks well for that query type. Simpler, but works without model-specific training.
The 70% cost reduction figure matches what we see in practice. The insight that most queries are "easy" is the key — once you stop sending FAQ-level questions to frontier models, the savings are dramatic.
Curious if you have looked at combining both approaches — upfront classification for obvious cases, then internal state probes for the ambiguous ones in the middle.
Interesting approach. I built something similar (Komilion) — a model router that classifies each request and routes to the cheapest model that handles it well.
A few things I learned building this:
1. Price-only routing is not enough. You need quality signals too. I use LMArena and Artificial Analysis benchmarks to set quality floors per tier.
2. A regex fast path for simple queries (greetings, yes/no, translations) handles ~60% of requests in <5ms with zero API calls to a classifier. This matters because routing latency eats into your savings if every request hits an LLM classifier first.
3. Provider reliability varies wildly. Auto-failover saves you from the "DeepSeek is down" problem — your app keeps working, just on the next cheapest option.
One real example: a support bot went from ~$250/mo on Opus 4.6 to ~$40/mo with routing. Most conversations were FAQ-level.
Then I looked at the actual queries. 70% were things like "what are your hours?" and "how do I return something?" — questions where a $0.80/M-token model gives the same answer as a $15/M-token model. But about 5% were genuinely complex (multi-step troubleshooting, product comparisons requiring reasoning) where Opus was noticeably better.
I started manually routing: simple patterns to a cheap model, everything else to Opus. The bill dropped to $40/month with no quality complaints from users. But maintaining the routing logic across projects got tedious — every new app needed the same classification + model selection + failover logic.
So I built Komilion to package it up. The classification runs in two stages:
1. A regex fast path catches ~60% of requests instantly (greetings, FAQ patterns, simple classification tasks). Zero API calls, under 5ms.
2. For the rest, a lightweight LLM classifier determines task type and complexity, then matches against a routing table built from LMArena and Artificial Analysis benchmark data.
What surprised me in the benchmark data: complex tasks through the router actually produced MORE detailed output than any single pinned model (6,614 chars avg vs 3,573 for Opus). The router selects specialized models per task type rather than using a generalist model for everything.
Stack: Next.js on Vercel, Neon PostgreSQL, OpenRouter upstream. Total hosting cost ~$20/month. It's a solo project.
The thing I'd do differently: I should have started with the benchmark data instead of building the product first. The numbers make the case better than any feature list.
Happy to answer technical questions about the routing logic, benchmark methodology, or anything else.