Generally, word2vec is a language model to predict the words probability based on the context. When build the model, it create word embedding for each word, and word embedding is widely used in many NLP tasks.
Models
CBOW (Continuous Bag of Words)
Use the context to predict the probability of current word. (In the picture, the word is encoded with one-hot encoding, WV∗N is word embedding, and WV∗N′, the output weight matrix in hidden layer, is same as υ^ in following equations)
Context words’ vectors are υc−n...υc+m (m is the window size)
Context vector υ^=2mυc−m+υc−m+1+...+υc+m
Score vector zi=uiυ^, where ui is the output vector representation of word ωi
Turn scores into probabilities y^=softmax(z)
We desire probabilities y^ match the true probabilities y.
We use cross entropy H(y^,y) to measure the distance between these two distributions.
H(y^,y)=−j=1∑∣V∣yjlog(y^j)
y and y^ is accurate, so the loss simplifies to:
H(y^,y)=−yjlog(y^)
For perfect prediction, H(y^,y)=−1log(1)=0
According to this, we can create this loss function:
Minimize J is expensive, you need to calculate the probability of each word in vocabulary list. There are two ways to reduce the computation. Hierarchical Softmax and Negative Sampling.
Hierarchical Softmax
Encode words into a huffman tree, then each word has a Huffman code. The probability of it’s probability P(w∣Context(ω)) can change to choose the path from root to the leaf node, each node is a binary classification. Suppose code 0 is a positive label, 1 is negative label. If the probability of a positive classification is
σ(XωTθ)=1+e−XωT1
Then the probability of negative classification is
1−σ(XωTθ)
足球’s Huffman code is 1001, then it’s probability in each node are
The probability of the 足球 is the production of these equation.
Generally,
p(ω∣Context(ω))=j=2∏lωp(djω∣Xω,θj−1ω)
This reduce the calculation complexity to log(n) instead of n
Negative Sampling
This method will choose some negative sample, then add the probability of the negative word into loss function. The optimisation target becomes maximise the positive words’ probability and minimise the negative words’ probability.
Let P(D=0∣ω,c) be the probability that (ω,c) did not come from the corpus data. Then the objective function will be