Nice write up, one thing that stood out is the V2 to V3 jump. One of my clients is integrating personal wellness & AI, and we took a slightly different route. The health data and personal data live in separate dbs with an encrypted mapping layer between. This way the model only sees health context attached to a unique pseudo-user level session. Your problem almost seems harder, because the PII is the signal/context. One challenge we are facing is re-identification, e.g rich-health profiles being identifiable in themselves.
Curious if you have thought about that side of things with your V3 implementation?
That has been our finding, it seems that we are able to do larger sessions in a single context window because of the lessened bloat. Certainly needs wider spread testing though which is why we are excited to get more feedback.
Great question! We initially started with a proxy, and plan to support it for users that already have things like LiteLLM setup. However we chose the MCP route, and soon plugin route, because it was much lower lift for installation. Setting up a proxy and configuring it for Cursor especially can be a bit tricky. There is actually still some code in the MCP server for a LiteLLM integration, which we hope to support officially soon.
Another large part of it is without MCP, and only a proxy in a client like cursor, the agent doesn't reason about the budget. When it has it has to use it as a tool, it actively thinks about the cost of its actions.