Evaluating language models: perplexity and friends

Gives one number to compare two language models on held-out text.

The idea in plain words

A multiple-choice quiz. If a model is as unsure as someone guessing between 6 equally likely options each time, its perplexity is 6, like rolling a fair die. A perplexity of 1 would be a model that is never surprised; higher means more confused.

Step by step

1. Hold back some text the model never trained on.
2. Measure how surprised the model is by each real next word (low probability = high surprise).
3. Average the surprise and convert it into "effective number of choices": that is perplexity. Lower is better.

Play it on paper first

Shannon game: a friend hides a sentence and you guess it letter by letter. Count your guesses. Text you predict quickly has low perplexity for you; text full of surprises has high perplexity.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ