Recurrent neural networks: memory that fades

Carries a hidden state from word to word, so the output depends on the whole history and on word order.

The idea in plain words

A game of telephone with notes. Each word, you update your one-line summary of everything so far and pass it on. The summary can carry meaning forward, but when it is copied and shrunk again and again, early words fade away.

Step by step

1. Keep a hidden state h. At each word: new h = a squashed mix of the old h and the new word.
2. The same weights are reused at every step, so the model handles any length and is sensitive to word order.
3. Learning sends the error signal backwards through every step, shrinking it each time (vanishing gradient). That is why plain RNNs forget the far past, and why LSTMs and GRUs add gates.

Play it on paper first

Whisper a sentence along a line of five people, each told to repeat it a bit more quietly and shorter. Compare the first and last versions: that gap is the vanishing memory.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ