I'm building Veritrooper — it audits how accurately an LLM answers from a body of written rules (tax code, safety regs, drug labels, filings… anything written) and hands back a per-question record of where and why it's wrong.
It started as an AI Dungeon Master that kept confidently breaking D&D's own rules. That failure mode — sounding certain about something the source doesn't actually support — turned out to be the real problem with LLMs on regulated text, so I went and built the thing that catches it.
The part I care about: it isn't just an LLM grading an LLM. The clear-cut calls are deterministic; only the genuinely contested ones go to a model from a different vendor, and nothing ever grades its own vendor. Every number reproduces from the logs, and the run records are downloadable — so you can check my work.
Solo dev, first real software project. veritrooper.com · [email protected]
This was a fascinating trip down the rabbit hole of transistors, RAM types, and the structure of everything involved. It's written in engineer speak, as is necessary because of the content, I suppose. Great article.
Hi HN. I built this platform completely buy accident, at least at first. I am a disabled vet that was getting bored sitting around all of the time. I decided to try and make a D&D game. I learned Unity, installed python, and... off I went.
The problem that I ran into was that the Dungeon Master AI was not realizing when I told it to do something I could not do. It would just happily narrate as if the act I just took was completely fine. Spending gold I didn't have, Casting spells I didn't know, etc...
As I dove into that problem and started fixing it, a buddy of mine said something about switching rulesets to RIFTS. I said 'I think I can". It was then that I realized that the system I was developing might just work on any type of data, with any LLM.
I shifted my focus, and began developing. IIt's a system that reads any corpus, generates a set of questions and the correct answers straight from that source, then makes the MUT (the LLM or enterprise endpoint being tested) answer them. It runs the same questions through its pipeline too, and a different vendor's model handles the close calls. THen, it provides a complete report of the audit, including every question and answer pair, the answers given, and a question by question analysis of any incorrect answers, determining the reason the MUT was incorrect.
It compiles those collectively, looking for patterns, and then provides an extensive report which lists specific training goals for the MUT based upon the audit.
Although it provides a significant Boost in accurate answers, the pipeline is not all knowing. its limits are set by the model it is testing. I built this solo, on a gaming PC and rented GPU's.
I would really like people's opinions of the output, presentation, and anything else about it that you want to critique. Good or bad, I'll welcome it.
I'm happy to answer any questions if you have them.
I clicked on this, expecting to see something about the visual appearance of a doc. WHat you have presented here is so much more though. A Doc that is not 'beautiful' in the sense that you are talking about is usually harder to read, harder to understand, harder to derive the point from, or a combination of all three. Well said.
I absolutely agree. There are so many prongs to a complete education; mandates, curriculum changes, social and political injections, funding issues (particularly taxpayers needing to see a return on their educational investment), and so many more. One bad systemic decision can set off downward spirals in one, some, or all of these.
I love this. It's is an interesting take on 'Surround yourself with the people you want to become'. It is amazing when you meet someone who is so good at what they do, they are clearly in another league. Whether they're magicians, geniuses, 'out of the box' thinkers, or whatever label they are given, they are amazing people to surround yourself with.
With education, there's always someone trying to build a better mouse trap. When it comes down to it though, the basics still apply. In my opinion the single largest factor in education today (In this country) is the student's home environment. If the student isn't being set up for success at home, any attempts to re-vamp, redo, or otherwise modify the educational system that student learns from is doomed. If the student's family isn't reinforcing the importance of school, the authority and respect teachers should be given, and helping in meaningful ways, they are not setting up their child for success.