CBOW: predict the missing word from its neighbours

Averages the context word vectors and predicts the centre word with a softmax.

The idea in plain words

Fill in the blank. Show a neural net "the ___ sat on the mat" and make it guess "cat". Its guess is built from the words around the blank, mixed together like paint colours into one average.

Step by step

1. Take the context words around a hidden centre word.
2. Average their vectors into one summary vector.
3. Use a softmax over the vocabulary to guess the centre word; nudge the vectors so the right word scores higher.

Play it on paper first

Cover one word in a printed sentence and let a friend guess it from the rest. CBOW is exactly this game, played millions of times.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ