Scaled dot-product attention and the √dₖ

Lets every token build a new vector from the tokens most relevant to it.

The idea in plain words

A library search. Your query is what you are looking for; each book has a key (its label) and a value (its contents). Compare your query with every key, turn the scores into percentages, and read a blend of the values, mostly from the best matches.

Step by step

1. Each token makes three vectors: a query Q, a key K and a value V.
2. Score = Q · K for every pair; high means relevant.
3. Divide by √dₖ. A dot product of d random numbers has spread about √d, so without the division, scores for big d get huge, softmax saturates to nearly one-hot, and gradients vanish.
4. Softmax the scores into weights, and output the weighted sum of the V vectors.

Play it on paper first

A room of 8 people each shouts a number; you add them up. With 64 people the total is much louder even if everyone is equally loud. Dividing by the square root of the head-count keeps the volume steady, which is what √dₖ does for the dot product.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ