Spaces:
Running
Download content/components/misc.token_embedding.md from philippyt/ablation-lab: direct link, hf CLI and curl.
- Browser
- Download file 1.56 kB
-
https://huggingface.co/spaces/philippyt/ablation-lab/resolve/main/content/components/misc.token_embedding.md
- Command line
-
hf download hf://spaces/philippyt/ablation-lab/content/components/misc.token_embedding.md
-
curl -L -o misc.token_embedding.md https://huggingface.co/spaces/philippyt/ablation-lab/resolve/main/content/components/misc.token_embedding.md
id: misc.token_embedding
level: component
title: Token Embedding
short: A lookup table that turns each token ID into a learned d_model vector.
related_concepts: []
The first thing the model does to your input is turn each token ID into a vector. The token embedding is the table that does that:
tok_embed = nn.Embedding(vocab_size, d_model)
It is a vocab_size × d_model matrix of learned parameters. For an input ID i, the model reads row i of the table and emits a d_model-dimensional vector. That vector is the model's input representation for that token.
In this lab vocab_size = 72 (one row per character in the TinyStories corpus, since we use a char-level tokenizer) and d_model = 128. The table is 72 × 128 = 9,216 learned parameters.
Two things the token embedding does not do:
- It does not add positional information. Position is handled by RoPE inside attention, or by a separate learned absolute table, depending on
pos_embed. - It does not depend on context. Every occurrence of the same token starts from the same row. Context-aware representations are what the rest of the network produces.
Why it matters: The embedding rows are where the model stores per-token meaning. Tokens that behave similarly in the data end up with similar rows. When tie_embeddings is on, the same matrix is reused (transposed) as the LM head, so the input lookup and the output projection share weights. When tie_embeddings is off, the LM head is a separate matrix and the token embedding rows are only used at the input.