Clerk has quite a few dark patterns in their free tier, eg: if your app is on Clerk free tier, all your users will be forced to log out and re-login every 7 days (and they try to obfuscate this fact until you're locked in). For this reason, I've recently had to migrate away from them - I'm really glad there are alternatives.
There are several issues that make the KV cache as-is unsuitable for caching across requests. First, it requires the cached tokens to be in the exact same position in the sentence, this means it's mainly only useful for autoregressive generation where the prefix is always the same. Second, it is extremely big, so without some sort of compression, the cost to store it between requests and the time required to transfer the data to the GPU will outweigh any compute savings.
Yeah, I had a similar experience with Chroma DB. On paper, it checked all my boxes. But yea, it's alpha software with the first non-prerelease version only coming out in July 2023 (so it's 3 months old).
I ran into some dumb issues during install like the SQLite version being incorrect, and there wasn't much guidance on how to fix these problems, so gave up after struggling for a few hours. Switched to PGVector which was much simpler to setup. I hope Chroma DB improves, but I wouldn't recommend it for now.
It would be great to see more innovation like AI in DAW tools, but there are some challenges. The main constraint is it needs to process in real time, allowing just a few ms to process a sample. Very few neural methods can work with that constraint, without it, they can't fit into the standard DAW workflow where you string together many plugins, each processing the signal in real time.
There are some AI tools that work outside the main workflow, like for mastering after you're done with the DAW. But it's quite difficult to improve and bring new ideas beyond the typical signal processing modules without completely revamping the current workflow.