en.wikipedia.org/wiki/Word2Vec
3 corrections found
continuously sliding skip-gram
The standard Word2Vec architecture is called the continuous skip-gram model, not “continuously sliding skip-gram.”
Full reasoning
The original Word2Vec paper names the second architecture Continuous Skip-gram Model. The follow-up NeurIPS paper also refers to the continuous Skip-gram model. I could not find this architecture referred to in the literature as “continuously sliding skip-gram,” so this is a naming error that could mislead readers searching for the model's standard name.
2 sources
- Efficient Estimation of Word Representations in Vector Space
3.2 Continuous Skip-gram Model
- Distributed Representations of Words and Phrases and their Compositionality
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations.
the one-hot encoding of the word is used as the input to the neural network.
This reverses CBOW’s direction. In CBOW, the input is the surrounding context words, and the model predicts the current target word.
Full reasoning
In the original Word2Vec paper, CBOW is defined as predicting the current (middle) word from its surrounding context words. The paper explicitly says the model uses history and future words at the input, and the training criterion is to classify the current word. Its figure caption also states: "The CBOW architecture predicts the current word based on the context." So saying that the current word's one-hot vector is the input to CBOW describes the opposite direction of prediction and confuses CBOW with skip-gram.
2 sources
- Efficient Estimation of Word Representations in Vector Space
we have obtained the best performance ... by building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word.
- Efficient Estimation of Word Representations in Vector Space
Figure 1: New model architectures. The CBOW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word.
If the Word2Vec model has not encountered a particular word before, it will be forced to use a random vector, which is generally far from its ideal representation.
Standard Word2Vec does not inherently assign a random vector to unseen words; it typically has no learned representation for out-of-vocabulary words at all.
Full reasoning
This statement misdescribes the OOV behavior of standard Word2Vec/CBOW/skip-gram models. In the fastText paper, which introduces a subword extension specifically to address this limitation, the authors state that standard cbow and skipgram baselines cannot obtain word representations for words that do not appear in the training data. For evaluation, they used null vectors for such words to make comparisons possible. In other words, the basic Word2Vec model does not naturally produce a random learned vector for an unseen word; without an added fallback mechanism, it simply lacks a representation for that OOV token.
2 sources
- Enriching Word Vectors with Subword Information
Some words from these datasets do not appear in our training data, and thus, we cannot obtain word representation for these words using the cbow and skipgram baselines. In order to provide comparable results, we propose by default to use null vectors for these words.
- Enriching Word Vectors with Subword Information
Our method ... allows us to compute word representations for words that did not appear in the training data.