> alignment tooling is fascinating, as we increasingly want to re-fit->embed over time as our envs change and compare, eg, day-over-day analysis. This area is not well-defined yet common for anyone operational so seems ripe for innovation
Training the parametric UMAP is a little more expensive, but the new landmarked based updating really does allow you to steadily update with new data and have new clusters appear as required. Happy to chat as always, so reach out if you haven't already looked at this and it seems interesting.
> And algorithms can only predict content that you've seen before. It'll never surprise you with something different. It keeps you in a little bubble.
This is not true at all, algorithms can predict things you haven't seen before, and can take you well outside your bubble. A lot of the existing recommendation algorithms on social media etc. do keep you in a bubble, but that's a very specific choice 'cause apparently that's where the money is at. There's enough work in multi-armed-bandit explore/exploit systems that we definitely could have excellent algorithms that do exactly the kind of curation the author would like. The issue is not algorithms, but rather incentives on media recommendation and consumption. People say they would like something new, but they keep going back to the places that feed them more of the comfortable same.
It is probably not all the things you want, but AlignedUMAP can do some of this right now: https://umap-learn.readthedocs.io/en/latest/aligned_umap_bas...
If you want to do better than that, I would suggest that the quite new landmarked parametric UMAP options are actually very good this: https://umap-learn.readthedocs.io/en/latest/transform_landma...
Training the parametric UMAP is a little more expensive, but the new landmarked based updating really does allow you to steadily update with new data and have new clusters appear as required. Happy to chat as always, so reach out if you haven't already looked at this and it seems interesting.