WordPiece vs BPE: building a subword vocabulary

Learns reusable pieces so new words are split into known parts instead of becoming unknown.

The idea in plain words

Lego bricks. A shop cannot stock every possible model, but with a good set of small bricks you can build any word, even one you have never seen. A subword vocabulary is that brick box: whole words for the common ones, small pieces for the rest.

Step by step

1. Start with single characters as the smallest bricks.
2. BPE repeatedly merges the pair that occurs most often. WordPiece merges the pair that most improves how well the vocabulary explains the text (frequency of the pair relative to its parts).
3. To tokenize a new word, WordPiece takes the longest piece in the vocabulary from the left, marks later pieces with ##, and repeats (hugs → hug + ##s).

Play it on paper first

Write 10 words that share endings (play, playing, played, walking, walked ...). Circle the repeated chunks ("play", "walk", "ing", "ed"). Those chunks are your vocabulary; now spell a new word like "walked" with them.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ