Training Word2Vec live: negative sampling, α and subsampling

Pulls real (word, context) pairs together and pushes random pairs apart until meaning-groups form.

The idea in plain words

Party seating. Real neighbours are friends you pull closer; a few randomly chosen strangers are pushed away. You do not need to check every guest in the room each time, just the real friends plus a handful of random strangers.

Step by step

1. For each real (word, context) pair, nudge the two vectors together.
2. Pick a few random words (negatives) and nudge them apart from the word. This turns a huge multi-class problem into cheap yes/no questions.
3. Subsampling drops very common words like "the" so they do not dominate; α (the learning rate) sets the step size.

Play it on paper first

Stick 10 word cards on a table. For each sentence you read, slide the words that appear together a bit closer and slide a random word away. After 20 sentences, look at the clusters.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ