Seq2Seq: encoder, decoder, attention and beam search

Reads a whole sentence into a vector, generates the translation word by word, and searches for the best output.

The idea in plain words

A human interpreter. The encoder reads the whole French sentence and forms a gist; the decoder then speaks English one word at a time. Plain seq2seq squeezes everything into one gist; attention lets the interpreter glance back at the source for each word. Beam search is trying a few promising sentence starts at once instead of betting everything on the first word.

Step by step

1. Encoder: read the source words into hidden states.
2. Decoder: produce one target word per step, feeding its last word back in. Teacher forcing during training feeds it the true previous word.
3. Attention: at each step, weigh the source words by relevance, and use that mix as extra context.
4. Beam search: keep the best k partial sentences at each step, instead of only the single best (greedy), so one unlucky early choice does not ruin the sentence.

Play it on paper first

Hike with a group of three: at each fork, each hiker takes the best-looking path but only the three best overall routes are kept. That is beam search with k = 3. Greedy is a lone hiker who never looks back.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ