Word2Vec: meaning as a position in space

Gives every word a dense vector so similar words sit close and relations become directions.

The idea in plain words

A map of meaning. Cities that are alike sit close on a map; words that are used alike sit close in vector space. And just as "go 500 km north" moves you the same way from any city, a relation like man → woman is a consistent direction.

Step by step

1. Give every word a short list of numbers (its vector), learned from the company it keeps.
2. Close vectors mean similar usage: measure with cosine similarity.
3. Relations become arrows. king − man + woman lands near queen, as long as you leave the input words out of the answer list.

Play it on paper first

Plot five foods and five vehicles on a sheet of paper so that similar things are close. Then draw one arrow from "car" to "cars". Does the same arrow turn "dog" into "dogs"? That repeatable arrow is the whole analogy trick.

Other labs

• The Markov assumption and the bigram model
• Smoothing: giving unseen N-grams a chance
• Evaluating language models: perplexity and friends
• WordNet: meaning written down by hand
• Word2Vec: meaning as a position in space
• CBOW: predict the missing word from its neighbours
• Skip-gram: one word, many training pairs
• Training Word2Vec live: negative sampling, α and subsampling
• WordPiece vs BPE: building a subword vocabulary
• Recurrent neural networks: memory that fades
• Seq2Seq: encoder, decoder, attention and beam search
• Scaled dot-product attention and the √dₖ