It was fun writing our book because I SAW you do that. And I had a different approach - I would outline obsessively and hold the whole chapter in my head at once before I started writing. Holding a whole chapter and cross referencing everything with everything else was O(N^2). You're approach for writing one instance of the chapter was linear O(N) but you did it M times... so O(M*N) ... maybe about the same :P
You're absolutely right. This was a post I tossed together quickly just to see what could be done without thinking too much. In retrospect, I think this would be better implemented using Elasticsearch sparse vector fields which allow you to specify the value of every token. Maybe I'l make an update post to try again.
Practical advice! So many good products are lost by people that become fixated on unnecessary evals too early. You need to build your eval muscle AS you release product and get real feedback.
RAG is a pain to set up, so I tried something different. Instead of dealing with vector DBs and all that complexity, just let the LLM navigate well-structured docs like a human—exploring outlines and diving into sections. It’s simple, and works great for stuff like technical manuals or llms.txt.
Feels like it's a dopaminergic response to hearing a word but not knowing what it is. It's a novelty seeking thing. But once the work seems to be well understood, the novelty wears off and the novelty seeking mechanisms in humans quit responding.