Spaces:
Running
Running
|
Download content/components/misc.token_embedding.md from philippyt/ablation-lab: direct link, hf CLI and curl.
- Browser
- Download file 1.56 kB
-
https://huggingface.co/spaces/philippyt/ablation-lab/resolve/main/content/components/misc.token_embedding.md
- Command line
-
hf download hf://spaces/philippyt/ablation-lab/content/components/misc.token_embedding.md
-
curl -L -o misc.token_embedding.md https://huggingface.co/spaces/philippyt/ablation-lab/resolve/main/content/components/misc.token_embedding.md
1.56 kB
| id: misc.token_embedding | |
| level: component | |
| title: Token Embedding | |
| short: A lookup table that turns each token ID into a learned d_model vector. | |
| related_concepts: [] | |
| The first thing the model does to your input is turn each token ID into a vector. The token embedding is the table that does that: | |
| `tok_embed = nn.Embedding(vocab_size, d_model)` | |
| It is a `vocab_size × d_model` matrix of learned parameters. For an input ID `i`, the model reads row `i` of the table and emits a `d_model`-dimensional vector. That vector is the model's input representation for that token. | |
| In this lab `vocab_size = 72` (one row per character in the TinyStories corpus, since we use a char-level tokenizer) and `d_model = 128`. The table is 72 × 128 = 9,216 learned parameters. | |
| Two things the token embedding does **not** do: | |
| 1. It does not add positional information. Position is handled by RoPE inside attention, or by a separate learned absolute table, depending on `pos_embed`. | |
| 2. It does not depend on context. Every occurrence of the same token starts from the same row. Context-aware representations are what the rest of the network produces. | |
| **Why it matters:** The embedding rows are where the model stores per-token meaning. Tokens that behave similarly in the data end up with similar rows. When `tie_embeddings` is on, the same matrix is reused (transposed) as the LM head, so the input lookup and the output projection share weights. When `tie_embeddings` is off, the LM head is a separate matrix and the token embedding rows are only used at the input. | |