Show HN: VAOS – A feedback loop that makes deployed agents less stupid(vaos.sh)
vaos.sh
Show HN: VAOS – A feedback loop that makes deployed agents less stupid
https://vaos.sh
4 comments
The confidence-scoring approach is really smart. I'd definitely give users threshold control -- different use cases have wildly different risk tolerances. A social media posting agent can auto-approve at 0.7, but an agent sending emails to clients probably needs 0.95.
Re: the cold start problem -- have you tried seeding with a few manually-written 'ideal response' examples instead of starting from zero? In my experience with agent management, giving agents even 5-10 reference outputs dramatically reduces the ramp-up period. Essentially turning the cold start into a warm start.
The corrections-as-structured-context approach (vs fine-tuning) is the right call for this stage. Fine-tuning is expensive and brittle. Structured context corrections are portable, inspectable, and you can version them. That also solves your lock-in concern -- the corrections are just data, export them as JSON.
One suggestion: consider adding a 'correction categories' feature. After a while, you'll notice patterns (tone too formal, wrong audience assumptions, missing context). Categorizing corrections could let you surface systemic issues rather than fixing one-offs.
Re: the cold start problem -- have you tried seeding with a few manually-written 'ideal response' examples instead of starting from zero? In my experience with agent management, giving agents even 5-10 reference outputs dramatically reduces the ramp-up period. Essentially turning the cold start into a warm start.
The corrections-as-structured-context approach (vs fine-tuning) is the right call for this stage. Fine-tuning is expensive and brittle. Structured context corrections are portable, inspectable, and you can version them. That also solves your lock-in concern -- the corrections are just data, export them as JSON.
One suggestion: consider adding a 'correction categories' feature. After a while, you'll notice patterns (tone too formal, wrong audience assumptions, missing context). Categorizing corrections could let you surface systemic issues rather than fixing one-offs.
Good points across the board.
Threshold control — yes, that's the plan. Right now it's a single 0.8 cutoff which is obviously too blunt. A social media agent vs a client-facing email agent have completely different risk profiles. Building per-channel thresholds into the next release.
Cold start seeding — we actually do something close to this already. The rules system lets you pre-load corrections before the agent handles its first real conversation. But you're right that 5-10 reference outputs would be even better than corrections. That's a cleaner onboarding UX. Adding it to the backlog.
Corrections as structured context vs fine-tuning — glad someone else sees this the same way. The portability argument is the one that convinced me. If you can export your corrections as JSON and bring them to another provider, that kills the lock-in problem. We store them as structured records in Supabase right now.
Correction categories — this is smart and we're halfway there. The dashboard already groups corrections by trace/conversation, but not by pattern type (tone, audience, missing context). That's a better abstraction. Would make it much easier to spot systemic issues instead of whack-a-mole-ing individual responses.
Appreciate the detailed feedback. Curious — are you running agents in production yourself?
Threshold control — yes, that's the plan. Right now it's a single 0.8 cutoff which is obviously too blunt. A social media agent vs a client-facing email agent have completely different risk profiles. Building per-channel thresholds into the next release.
Cold start seeding — we actually do something close to this already. The rules system lets you pre-load corrections before the agent handles its first real conversation. But you're right that 5-10 reference outputs would be even better than corrections. That's a cleaner onboarding UX. Adding it to the backlog.
Corrections as structured context vs fine-tuning — glad someone else sees this the same way. The portability argument is the one that convinced me. If you can export your corrections as JSON and bring them to another provider, that kills the lock-in problem. We store them as structured records in Supabase right now.
Correction categories — this is smart and we're halfway there. The dashboard already groups corrections by trace/conversation, but not by pattern type (tone, audience, missing context). That's a better abstraction. Would make it much easier to spot systemic issues instead of whack-a-mole-ing individual responses.
Appreciate the detailed feedback. Curious — are you running agents in production yourself?
Thanks! Threshold control is on the roadmap. Right now the confidence gating works at two levels: auto-approve (high confidence, like correcting a greeting style) and suggest-only (lower confidence, needs manual confirmation from dashboard). The correction-to-rule pipeline is the key mechanic — once you confirm a correction, it becomes a permanent behavioral rule injected at every boot. The agent literally cannot make that mistake again. Since launch we have added a 7-agent autonomous marketing crew running on the same memory infrastructure as proof it works in production.
The part I care about most: every 5 minutes, a loop scores each agent response on confidence. Low-confidence ones get flagged for you to review. When you correct something, that correction goes into the agent's context for future responses. Not fine-tuning -- just feeding corrections back as structured context. After a few days, the agent stops repeating the same bad answers.
I'm dogfooding it with an agent called Scribe that posts to X for me. Scribe was terrible for the first ~80 interactions. Now it's mostly fine. The cold start period is real and I haven't figured out how to shorten it.
What works: Telegram responses in under 2 seconds. Swap between GPT-5.2, Claude Opus 4.6, and Gemini without reconfiguring. The feedback loop does what I wanted.
What doesn't: Discord and WhatsApp aren't hooked up. No way to export learned corrections (lock-in problem I need to solve). Observability dashboard exists but only I can see it right now.
$29/mo, $10 in AI credits included, 14-day trial. Stack is Node.js on Fly.io.
Curious about the confidence-scoring approach. Anything above 0.8 auto-approves, below gets queued for human review. Should I give users that threshold control, or is one knob enough?