I built a simple (free) eval tool for my own uses (Github Gists + Model Outputs) after not being able to find a suitable one in the market.
The market's being split into
1. Longitudinal LLM observability tooling
Most eval startups have gone down the route of something more like being an observability platform for LLM inference. They want to be in your stack and running the inference to collect data on performance of it.
They collect things like how often a model returns JSON that's out of spec or returns values that aren't expected as well as general timing and cost info.
2. Safety Limiting / Pentesting
Say you're doing something in the medical field or that's sensitive in some way and you want to figure out what model has the best outputs for your task that won't fly off the guardrails.
3. Simple cost + performance + quality swapping
This is what my tool does, basically lets you test if you _really_ need to be running that frontier model in a loop across a million records or if you'd be better with an older model or something else.
Something that's improved my life has been buying a sticker sheet of those LED darkening dots. They're only a couple bucks and look much cleaner than other solutions I've tried while still allowing for _some_ light to come through.
A counterpoint to this is that we have some real different definitions of AI.
If you consider things like the machine learning filters in your smartphone camera and Google's AI Overviews for searches it's entirely plausible that the US is currently at 75%+ of AI usage.
I think it's your last point that's actually the strongest.
There's always gaps between theoretical and practical, but to see China investing so hard in the future while the US digs in it's heels is infuriating.
I thought it was more implied, but let me be more explicit:
- This is something I made for myself without a lot of commercial thought, so I still haven't thought through pricing + usage + limits + operational limits. In it's current wildly unoptimized state it's still very cheap to run.
- For the specific concern about API Key leakage there's not a lot I can do about that (that I'm aware of) as the logic of what gets sent is handled by the client AI. It is possible to pull down and audit both the tools + instructions that are published by the MCP server if there are concerns on that side.
This is all very fair criticisms. This thread asked: "What are tools you have made for yourself?" and that's genuinely what this is. I wanted it so I made it and then AI makes it so easy to just throw up a marketing page.
I've a a handful of dev friends that have started to use it as well and give their feedback and it's been slowly growing as I've added sharing/invites.
I would absolutely not recommend putting big production data into it currently.
My vision for it was something more like how the #1 use of spreadsheets is actually people making lists and not actually people doing lots of calculations.
Given the uptake today (thanks everybody!) and your feedback (thanks Mystery-Machine) I'm going to work at addressing your concerns.
But hands down the most useful thing I've made is HutchDB, which is a MCP service that you can call from any AI chat or Agent setup to store data for you.
Literally from your AI you just say "save that to Hutch" and then it figures out:
- The schema + fields
- Builds nice webviews (Kanban, Timeline, Grid, Calendar)
- Lets you share the output with people
So people use it for all kinds of things like time tracking ("every hour save a summary of my activities to Hutch"), for Agent to Human handoff ("Every day check social media for mentions of my company and save them to Hutch").
I use it for things like recording all of our marketing activities and then having my AI compare those to signups for rough attribution, etc.
It's not just comparing all the models, it's also comparing all the providers and configurations of those models.
If you're doing any kind of production AI work you'll end up with outages caused by calling a single provider, OpenRouter seamlessly switching between providers is a godsend for uptime.
But even more than that there's meaningful cost+speed differences.
Here's Sonnet 4.6 being served direct, via Amazon and via Google
WRT the native grammar, consider adjective order. Few native english speakers (me included) can off the cuff name the proper order, but everyone knows the "right" order.
We're already seeing other states struggling with soaring house insurance rates (California and Florida) and a pattern of spikes in rates leading government mandated caps on rates leading to insurers pulling out of the states.
I've no idea what the going rate for insurance is currently in New Orleans but it has to be crazy right?
I was curious just how much of a difference there was, so ran a quick eval comparing them and fwiw DeepSeek is considerably slower but much much ~5x cheaper than Haiku and fwiw ~35x cheaper than Claude Opus 4.7.
I was trying to get a better sense of the time cost quality matrix of these, so I threw together a quick eval of Sonnet 4.6, Mistral's dev model, and Opus 4.7 (figuring it's what you'd use if you were on Max).
The results for a function implementation and test of levenshtein distance in js are pretty similar but Mistral is 30x cheaper than Opus 4.7 and 4x faster than Sonnet 4.6.
My plan was to just see if anyone wanted to actually use it first. That if I couldn't give it away I'd not invest the time in selling or open sourcing it.
I'd sort of designed it for my own needs first and hadn't thought too far beyond that.
I'd be curious to know if you consider a Lego store a "toy store". There's one that opened in my city fairly recently and is in an area of smaller boutique shops (kind of like what you described).
The market's being split into
1. Longitudinal LLM observability tooling
Most eval startups have gone down the route of something more like being an observability platform for LLM inference. They want to be in your stack and running the inference to collect data on performance of it.
They collect things like how often a model returns JSON that's out of spec or returns values that aren't expected as well as general timing and cost info.
2. Safety Limiting / Pentesting
Say you're doing something in the medical field or that's sensitive in some way and you want to figure out what model has the best outputs for your task that won't fly off the guardrails.
3. Simple cost + performance + quality swapping
This is what my tool does, basically lets you test if you _really_ need to be running that frontier model in a loop across a million records or if you'd be better with an older model or something else.
https://evvl.ai/
Example eval: https://giyd8stidy.evvl.io