Rather superficial and obfuscating. The article keeps raising the question "why ignore the magnitude" and never answers it.
"The important part of an embedding is its direction, not its length. If two embeddings are pointing in the same direction, then according to the model they represent the same "meaning"."
This can't be quite right. Any LLM transformer model looks at the embedding of the token sequence, (without normalizing, i.e. including its magnitude) for deciding on the next token. Why would you throw away that information, equivalent to throwing away one embedding dimension?
If I had to guess why cosine similarity is the standard for comparing embeddings I suspect it's simply because the score is bounded in [-1, 1], which you may find more interpretable than the unbounded score obtained by the unnormalized dot product or Euclidean distance.
In my experience, choice of similarity metric doesn't affect embedding performance much, simply use the one the embedding model was trained with.
How big is the state space of the Markov chain corresponding to a LLM generating a sequence of tokens? Wouldn't it be (size of the vocabulary)^(size of the context window), i.e. ~ (100k)^(4k)? How useful is it to conceptualize LLMs as Markov chains at that point? For example, is there a result about Markov chains with interesting implications for LLMs?
Mapping corresponding text and image into the same vector space is exactly how CLIP and other contrastive learning setups work: Have text and computer vision networks embed input data, and teach the networks to embed related inputs closely and unrelated ones far away. You can train the models on data scraped from the internet (e.g. images and corresponding captions) in a self-supervised way. Like you say it has huge applications for search and is also used by the generative models (to guide a generated image towards your textual input).
BTW it seems that the data2vec is at its core actually just one model, only the input and output parts are different depending on the modality (text, image sound etc.) I wouldn't expect the learned representations to be similar for similar content across modalities. The point of data2vec is to use just one model with a very general self-supervised learning setup for tasks, with the different tasks hopefully benefitting from each other during training.
"The important part of an embedding is its direction, not its length. If two embeddings are pointing in the same direction, then according to the model they represent the same "meaning"."
This can't be quite right. Any LLM transformer model looks at the embedding of the token sequence, (without normalizing, i.e. including its magnitude) for deciding on the next token. Why would you throw away that information, equivalent to throwing away one embedding dimension?
If I had to guess why cosine similarity is the standard for comparing embeddings I suspect it's simply because the score is bounded in [-1, 1], which you may find more interpretable than the unbounded score obtained by the unnormalized dot product or Euclidean distance.
In my experience, choice of similarity metric doesn't affect embedding performance much, simply use the one the embedding model was trained with.