LLM Concept Lab

Smoothing: giving unseen N-grams a chance

Moves a little probability from seen word pairs to unseen ones: Laplace, add-k, interpolation, Good-Turing and Kneser-Ney.

The idea in plain words

A fishing trip. You catch 18 fish: 10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel. How likely is the next fish to be a species you have never caught? Saying 0 is silly: you only fished for one afternoon. Smoothing keeps a small "just in case" share of probability for things you have not seen yet.

Step by step

  1. Laplace / add-k: pretend every possible pair was seen k extra times. Simple, but it can steal too much from real counts.
  2. Good-Turing: the species caught exactly once (3 of 18) tell you how much of the next catch will be brand new, so the unseen share is 3/18. Seen counts shrink slightly to pay for it (the trout drops from 1/18 to 1/27).
  3. Interpolation / backoff: when the bigram is unseen, lean on the shorter unigram.
  4. Kneser-Ney: judge a word by how many different words it follows, not just how often it appears. "Francisco" is common but almost always follows "San", so it is a poor guess after "reading ___"; "glasses" fits many contexts.

Play it on paper first

Write 10 words on slips, drawing from a hat with replacement, and note which words appeared only once. The count of once-only words is your estimate for how much of the next draw will be a word you have never seen.

Other labs