For a chatbot like claude code, since the instruction prompt part (including descriptions of tools) is relatively constant for multiple users and over a long time, a lot of optimizations can be made. Even basic prompt caching gives a lot of speed and cost reduction.
What is the strategy to deal with pay-as-you-go services used in vibe coded apps? I am always worried about the risk of some bad loop consuming thousands of dollars in a day.
The RAG was setup on a bunch of documents, most of them were manuals containing steps about measurements, troubleshooting, and replacing components of industrial machines.
The issue was that most of these steps were long (above 512 tokens). So the typical chunk window wouldn't capture the full steps. We added a tool calling capability by which LLM can request nearby chunks of a given chunk. This worked well in practice, but burned more $$.
optimal chunk size is strongly query-dependent - very true.
Faced similar issues. Ended up adding a agentic tool call layer on the top to retrieve the nearby chunks to handle a case where a relevant answer was only partially available in a chunk (like a 7 step instruction in which only 4 were available in a chunk). It worked ok.