Word2Vec
Because deep neural networks cannot understand words or sentences the way humans do, words and sentences must be represented in a form the network can understand.
Suppose there is a vocabulary $\mathbf{V}=\{w_1,...,w_{|v|}\}$. For each word $w_i$, there are two ways to represent it: one-hot encoding, and distributed representation.
one-hot representation
Each word is represented as a vector whose length equals the total number of words in the vocabulary. Each vector has a single 1, and the remaining values are all 0.
Advantages
- Highly interpretable: each word has a unique one-hot code, and the length of that one-hot vector equals the total number of words in the vocabulary.
- No information loss: the dimension of the word vector equals the number of words.
Disadvantages
- High-dimensional data makes computation complex: because the dimension of each word vector equals the total number of words in the vocabulary, each word vector can have thousands to tens of thousands of dimensions, which greatly increases computational complexity.
- Inefficient: each word vector contains only a single 1 and the rest are 0s, so from an information-theoretic point of view the utilization is very low.
- Cannot represent the similarity between any two words: because one-hot encoding is used, any two words are orthogonal.
distributed representation
Advantages
- Lower dimensionality than one-hot codes.
- Each element of the vector takes values in the real numbers.
- Can express similarity information between two words.
word2vec
Assumptions:
- Assume the window size is $2c$
- Assume that the distributions of any two word vectors are mutually independent.
For the vocabulary $\mathbf{V}=\{w_1,\dots,w_T\}$, the joint probability distribution is:
$$ P(w_{1:T})=P(w_t)*P(\text{context}(w_t)|w_t) $$Given $w_t$, the conditional likelihood is:
$$ P(w_{1:T})=P(w_t)P(w_{t-c : t-1}, w_{t+1:t+c}|w_t)=\prod_{t=1}^{T}P(w_{t-c:t-1},w_{t+1:t+c}|w_t) $$The average conditional likelihood can be written as:
$$ L = \frac{1}{T}\sum\limits_{t=1}^{T}\log P(w_{t-c:t-1},w_{t+1:t+c}|w_t)=\frac{1}{T}\sum\limits_{t=1}^{T}\sum\limits_{i\in [-c,c]}\log P(w_{t+1}|w_t) $$