We're building something similar and found that no matter how good the agent loop is, you still need "canonical metrics" that are human-curated. Otherwise non-technical users (marketing, product managers) are playing a guessing game with high-stakes decisions, and they can't verify the SQL themselves.
Our approach:
1. We control the data pipeline and work with a discrete set of data sources where schemas are consistent across customers
2. We benchmark extensively so the agent uses a verified metric when one exists, falls back to raw SQL when it doesn't, and captures those gaps as "opportunities" for human review
Over time, most queries hit canonical metrics. The agent becomes less of a SQL generator and more of a smart router from user intent -> verified metric.
The "Moving fast without breaking trust" section resonates, their eval system with golden SQL is essentially the same insight: you need ground truth to catch drift.
My partner and I have been playing this almost every morning. We're really enjoying it!
Some feedback:
1) it would be great if the incomplete clues could move to the top. this would avoid having to scroll down towards the end of the puzzle.
2) better collission behavior; it would be nice if we could drag a chunk of words and it would just "move the other words" out of the way. Sometimes we have to spend time to make a path to move chunks of words around.
Hermes is our harness, and we run it in the sandbox.
Session history is tracked in a Postgres db (small monkey patch to do this)
We built a lightweight skills hub to manage/track skills.
And the file system is backed up on S3 (using the new S3 FS).
But everything else is just running in a k8 pod.
We haven’t ran into any issues yet, but our strategy here is to have the least invasive changes so upstream harness changes don’t get in the way.