Slash LLM API Costs with This Open-Source Gateway(twitter.com)
twitter.com
Slash LLM API Costs with This Open-Source Gateway
https://twitter.com/realedgelab/status/2022351000989012015
2 comments
Just open-sourced Squirrel — an LLM API Gateway built to solve the nightmare of managing multiple models, providers, and prompts across different projects.
If you are building AI apps, managing agents, or running backend services, you have probably hit these walls:
Upgrading models is a grind. Updating hardcoded strings across 10+ repositories takes too much time.
Bleeding money blindly. Provider prices fluctuate, and tracking costs across multiple vendors is impossible manually.
Debugging is pure guesswork. Without full request/response logs, fixing broken prompts is a shot in the dark.
I built Squirrel to fix exactly this. Here is what it does out of the box:
Model Mapping (Change once, apply everywhere) Stop hardcoding specific models like gpt-4o or claude-3.5-sonnet. Map a virtual name (like my-smart) to a provider in the gateway. Want to upgrade all your apps at once? Just update the mapping in Squirrel. It takes effect instantly across all projects with zero code changes required.
Cost-Based Auto-Routing Set your provider pricing, and Squirrel automatically routes requests to the cheapest available option. It also supports priority, weight-based, and round-robin strategies.
Complete Observability Logs every single API call, including streaming responses. Check the admin dashboard to see the exact prompt sent, the model's output, token usage, and Time to First Byte (TTFB). This is an absolute lifesaver for debugging and fine-tuning.
Auto-Retry & Failover If a provider throws a 500 error or times out, Squirrel seamlessly switches to a backup provider. Your client-side code does not need to handle a thing.
Protocol Compatibility Works natively with OpenAI and Anthropic SDKs, and auto-translates protocols between them under the hood.
Tech Stack: Python (FastAPI) + Next.js + PostgreSQL/SQLite. One-click deployment via Docker Compose.
Fully open-source under the MIT license. It is still under active development, so feedback, issues, and PRs are incredibly welcome.
Check it out here: https://github.com/mylxsw/llm-gateway
If you are building AI apps, managing agents, or running backend services, you have probably hit these walls:
Upgrading models is a grind. Updating hardcoded strings across 10+ repositories takes too much time.
Bleeding money blindly. Provider prices fluctuate, and tracking costs across multiple vendors is impossible manually.
Debugging is pure guesswork. Without full request/response logs, fixing broken prompts is a shot in the dark.
I built Squirrel to fix exactly this. Here is what it does out of the box:
Model Mapping (Change once, apply everywhere) Stop hardcoding specific models like gpt-4o or claude-3.5-sonnet. Map a virtual name (like my-smart) to a provider in the gateway. Want to upgrade all your apps at once? Just update the mapping in Squirrel. It takes effect instantly across all projects with zero code changes required.
Cost-Based Auto-Routing Set your provider pricing, and Squirrel automatically routes requests to the cheapest available option. It also supports priority, weight-based, and round-robin strategies.
Complete Observability Logs every single API call, including streaming responses. Check the admin dashboard to see the exact prompt sent, the model's output, token usage, and Time to First Byte (TTFB). This is an absolute lifesaver for debugging and fine-tuning.
Auto-Retry & Failover If a provider throws a 500 error or times out, Squirrel seamlessly switches to a backup provider. Your client-side code does not need to handle a thing.
Protocol Compatibility Works natively with OpenAI and Anthropic SDKs, and auto-translates protocols between them under the hood.
Tech Stack: Python (FastAPI) + Next.js + PostgreSQL/SQLite. One-click deployment via Docker Compose.
Fully open-source under the MIT license. It is still under active development, so feedback, issues, and PRs are incredibly welcome.
Check it out here: https://github.com/mylxsw/llm-gateway
A few things I learned building this:
1. Price-only routing is not enough. You need quality signals too. I use LMArena and Artificial Analysis benchmarks to set quality floors per tier.
2. A regex fast path for simple queries (greetings, yes/no, translations) handles ~60% of requests in <5ms with zero API calls to a classifier. This matters because routing latency eats into your savings if every request hits an LLM classifier first.
3. Provider reliability varies wildly. Auto-failover saves you from the "DeepSeek is down" problem — your app keeps working, just on the next cheapest option.
One real example: a support bot went from ~$250/mo on Opus 4.6 to ~$40/mo with routing. Most conversations were FAQ-level.
https://www.komilion.com if anyone wants to try it (free tier, no card).