Skip-gram: one word, many training pairs

Uses the centre word to predict each neighbour, producing (target, context) training pairs.

The idea in plain words

The reverse game. Instead of guessing a hidden word from its neighbours, you are told one word and must name the words likely to stand next to it. One word yields many training pairs, so rare words get more practice than in CBOW.

Step by step

1. Choose a centre word and a window size (say 2 words each side).
2. Make a (centre, neighbour) pair for every neighbour inside the window.
3. Train the net to predict each neighbour from the centre. Computing a softmax over the whole vocabulary for every pair is the expensive part, which the next lab fixes.

Play it on paper first

Write a sentence, circle one word, and list every word within two places of it. Those are the pairs. Repeat for each word and count how fast the list grows.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ