Word2vec: Fish + Music = Bass(graceavery.com)
graceavery.com
Word2vec: Fish + Music = Bass
https://graceavery.com/word2vec-fish-music-bass/
18 comments
The "trick" is that word vectors are just projections of sparse tensor.
For example the word "queen" can be represented by its attributes. Representing in triplet form from wikidata Q16511993 (queen, subclass ,monarch),(queen, subclass, female),... A king would be (king, subclass ,monarch),(king, subclass, male),
Transitioning from queen to king is just changing an attribute in this triplet space, i.e removing a triplet and adding another. All those operations are linear. And then you take the projections.
This is kind of similar to the kernel trick of SVMs. You do the calculations in a lower dimensional space (word vectors) but implicitly you are doing them in the higher dimensional space (graphs of relations, aka triplets lists).
For example the word "queen" can be represented by its attributes. Representing in triplet form from wikidata Q16511993 (queen, subclass ,monarch),(queen, subclass, female),... A king would be (king, subclass ,monarch),(king, subclass, male),
Transitioning from queen to king is just changing an attribute in this triplet space, i.e removing a triplet and adding another. All those operations are linear. And then you take the projections.
This is kind of similar to the kernel trick of SVMs. You do the calculations in a lower dimensional space (word vectors) but implicitly you are doing them in the higher dimensional space (graphs of relations, aka triplets lists).
The sparse tensor perspective intrigues me, could you perhaps provide me with some references?
that person's description is unnecessarily jargony. sparse tensor just means vector that's mostly zero and has some ones. take your corpus of concepts and number them 0 to n-1. then each word you want to turn into a vec turns into a vector that has ones for concepts it's associated with and 0 elsewhere. learning that association is what word2vec does.
Can you explain attention in this same way?
nah i don't know much about LSTM but this looks like a fine tutorial
https://towardsdatascience.com/intuitive-understanding-of-at...
https://towardsdatascience.com/intuitive-understanding-of-at...
Not specifically. You can look at the literature for ontology, relation networks, semantic databases, reasoning.
The video that made me clicked, was https://www.ted.com/talks/ajit_narayanan_a_word_game_to_comm... It depicts in a game to help autistic children better communicate, a simple order-independent language representation, which is isomorphic to graphs.
If you do the math, you see that some triplets query like (queen, subclass, ?) is just a point-wise multiplication with the knowledge database sparse tensor in this sparse boolean tensor representation (the ? is a column of 1). Then it's just chaining requests.
A graph of triplet is isomorphic to a boolean tensor of dimension (nbSubjects, nbRelations, nbObjects).
The video that made me clicked, was https://www.ted.com/talks/ajit_narayanan_a_word_game_to_comm... It depicts in a game to help autistic children better communicate, a simple order-independent language representation, which is isomorphic to graphs.
If you do the math, you see that some triplets query like (queen, subclass, ?) is just a point-wise multiplication with the knowledge database sparse tensor in this sparse boolean tensor representation (the ? is a column of 1). Then it's just chaining requests.
A graph of triplet is isomorphic to a boolean tensor of dimension (nbSubjects, nbRelations, nbObjects).
One contention I have with this description (if I understand it at all), is that the projection into the linear space does not preserve the properties of changes in a relation graph.
I mean I "think" I said pretty much the same in my reply, but one big issue with word2vec is that the relations, which are graphical, can be projected, but not all changes in the graph can be. Therefore, I think it is not correct that these operations are linear in the sense that they can be equivalent to linear operations in the embedding space.
But maybe I just don't understand your description.
I mean I "think" I said pretty much the same in my reply, but one big issue with word2vec is that the relations, which are graphical, can be projected, but not all changes in the graph can be. Therefore, I think it is not correct that these operations are linear in the sense that they can be equivalent to linear operations in the embedding space.
But maybe I just don't understand your description.
As you know the basic idea, which precedes word2vec, is to classify each word in certain meaningful vector dimensions, for example "furryness" for pets.
If you code this manually, it makes sense to manipulate directions. For example, removing furryness from the archetypical pet vector, leaves you with non-furry pets in proximity. It essentially works the same way for word2vec.
Word2Vec encodes words without manual coding of dimensions.
The distributional hypothesis is that if words occur in the same context of other words, they share similar positions on some dimensions, and those dimensions are defined by the contextual words.
Now take two words that appear in more than one "cluster" of words together. Since these clusters differ, their similarity is high in these two groups of dimensions, so we may remove one cluster of dimensions - say related to gender - and we are left with the similarity on the other dimension.
The analogy operation used in the examples works by finding the relevant dimensions from the second word. We pick a word, and then another word for which the semantics are clear to us - like "male". Removing "male" from our starting word, we expect to move the semantic dimensions of "maleness", setting them to zero. Any word now left in proximity, retains the other contextual bundles distinct from gender.
Here you can see that this is not cut and dry. It is not clear that male is situated in a distinct semantic category. It is probably also related to dimensions that we would not want to set to zero. Nevertheless, the approach works well.
For your last question: It should be clear that a linear space can not encode true relations between words - it being linear. Moving toward another cluster usually means moving away from another, which is not true to meaning as a concept. It has been shown, that more sophisticated embeddings encode a non-linear space where such transformations are possible.
If you code this manually, it makes sense to manipulate directions. For example, removing furryness from the archetypical pet vector, leaves you with non-furry pets in proximity. It essentially works the same way for word2vec.
Word2Vec encodes words without manual coding of dimensions.
The distributional hypothesis is that if words occur in the same context of other words, they share similar positions on some dimensions, and those dimensions are defined by the contextual words.
Now take two words that appear in more than one "cluster" of words together. Since these clusters differ, their similarity is high in these two groups of dimensions, so we may remove one cluster of dimensions - say related to gender - and we are left with the similarity on the other dimension.
The analogy operation used in the examples works by finding the relevant dimensions from the second word. We pick a word, and then another word for which the semantics are clear to us - like "male". Removing "male" from our starting word, we expect to move the semantic dimensions of "maleness", setting them to zero. Any word now left in proximity, retains the other contextual bundles distinct from gender.
Here you can see that this is not cut and dry. It is not clear that male is situated in a distinct semantic category. It is probably also related to dimensions that we would not want to set to zero. Nevertheless, the approach works well.
For your last question: It should be clear that a linear space can not encode true relations between words - it being linear. Moving toward another cluster usually means moving away from another, which is not true to meaning as a concept. It has been shown, that more sophisticated embeddings encode a non-linear space where such transformations are possible.
>It has been shown, that more sophisticated embeddings encode a non-linear space where such transformations are possible.
Main paper doi/arxiv of this please? It seems potentially powerful to combine with reference graphs for automatic inference of proper causality from text-based communication streams. Like twitters firehose, GDELT, HN, etc.
I'm trying to track such analysis techniques on the side for applications to auto-crawling/archiving of things likely being short-lived data, yet part of the causality chain leading up to big, global events.
It'd provide an archive for future historians to analyze how humanity fucked up.
I haven't seen an actually feasible way yet, though.
Main paper doi/arxiv of this please? It seems potentially powerful to combine with reference graphs for automatic inference of proper causality from text-based communication streams. Like twitters firehose, GDELT, HN, etc.
I'm trying to track such analysis techniques on the side for applications to auto-crawling/archiving of things likely being short-lived data, yet part of the causality chain leading up to big, global events.
It'd provide an archive for future historians to analyze how humanity fucked up.
I haven't seen an actually feasible way yet, though.
I am really interested in how these analogies would appear in different word embedding methods (Bert..), or different implementations of word2vec.
As to different implementations of word2vec, I am still not satisfied why the original code uses two separate embeddings for target and context words. It could use shared embedding layer.
Also for the layer above the sigmoid layer (Dot product), why is cosine similarity not considered for calculating the vector similarity.
I have searched these issues and not found detailed comments. I have created word2vec embeddings with shared embedding layer and cosine similarity. The vectors seems similar to the original code but deciding on which one is better, needs more work.
As to different implementations of word2vec, I am still not satisfied why the original code uses two separate embeddings for target and context words. It could use shared embedding layer.
Also for the layer above the sigmoid layer (Dot product), why is cosine similarity not considered for calculating the vector similarity.
I have searched these issues and not found detailed comments. I have created word2vec embeddings with shared embedding layer and cosine similarity. The vectors seems similar to the original code but deciding on which one is better, needs more work.
I think that it is joining the bass as the musical definition of the word and sea bass the fish
word2vec simply is a neural network with a hidden layer of n perceptrons. The number they use in the paper is 300 I believe. Thus, this is the dimensionality of each word.
To train the model, we generate feature-label pairs by taking a word and randomly choosing a word next to it. We feed this into our model. After training, we are left with a 300 long vector for each word (input words x hidden layer since it’s fully connected).
To train the model, we generate feature-label pairs by taking a word and randomly choosing a word next to it. We feed this into our model. After training, we are left with a 300 long vector for each word (input words x hidden layer since it’s fully connected).
These analogies need to be taken with a grain of salt. For instance, king - man + woman actually returns king (since woman - man ~= 0). It's only queen when you ban the words king, man, or woman (since the embeddings of queen and king are close).
See Ian's tweet https://twitter.com/goodfellow_ian/status/113352818965167718... or the linked paper https://arxiv.org/pdf/1905.09866.pdf for more information.
See Ian's tweet https://twitter.com/goodfellow_ian/status/113352818965167718... or the linked paper https://arxiv.org/pdf/1905.09866.pdf for more information.
The bass/bass confusion seems to have permeated Chinese machine translation, as a search for 低音魚 (literally "low frequency fish") will show.
Another amusing one there that shows a similar ambiguity is: pig : oink :: mouse : keypress
Another amusing one there that shows a similar ambiguity is: pig : oink :: mouse : keypress
The typo in "pig : oink :: Fred Flinstone : wassup" bothered me, but even "Flintstone" doesn't produce the "Yabba dabba doo" that Woody Woodpecker (!) produces: pig : oink :: Fred Flintstone : mumble mumble.
If you're wanting to try it, you need about 6GB of RAM free.
If you're wanting to try it, you need about 6GB of RAM free.
I'm trying to replicate this on my Macbook and it's causing a lot of lag, and taking a long time. Anyone else have this issue, or might be able to help me debug this? I'm assuming I just don't have enough RAM (I only have 8GB here)?
It took several minutes to load off my old SSD, and took 6GB just to load the data. Trying out a bunch of word combinations pushed it up to ~9GB I think. So yeah, it seems like you just don't have enough RAM.
My (very rough) intuition to what w2v does is that it associates each word with a point in a space, then moves the points around so that their distances roughly correspond to how often they are used together with similar other words. (I.e., two words that both are often preceded by "It happened in" would be put close to each other, while words occurring into very different sentences would be placed further apart.)
I can see how this could give some semantic meaning to clusters of words - however, I'm still baffled as to why it gives meaning to directions within the vector space.
I also wonder how consistent it is. Seems to me it would be interesting to investigate the directions further. E.g., are all of them meaningful? What happens if I slightly rotate or scale a direction? How much transformation can I apply before it changes its meaning?
Are there more complex shapes in the space that also have meaning?