ablation-lab / content /components /misc.token_embedding.md
philippyt's picture
more manifests, update user facing component descriptions, instrument generate in backend
227a396
|
Raw History Blame Contribute Delete
1.56 kB
---
id: misc.token_embedding
level: component
title: Token Embedding
short: A lookup table that turns each token ID into a learned d_model vector.
related_concepts: []
---
The first thing the model does to your input is turn each token ID into a vector. The token embedding is the table that does that:
`tok_embed = nn.Embedding(vocab_size, d_model)`
It is a `vocab_size × d_model` matrix of learned parameters. For an input ID `i`, the model reads row `i` of the table and emits a `d_model`-dimensional vector. That vector is the model's input representation for that token.
In this lab `vocab_size = 72` (one row per character in the TinyStories corpus, since we use a char-level tokenizer) and `d_model = 128`. The table is 72 × 128 = 9,216 learned parameters.
Two things the token embedding does **not** do:
1. It does not add positional information. Position is handled by RoPE inside attention, or by a separate learned absolute table, depending on `pos_embed`.
2. It does not depend on context. Every occurrence of the same token starts from the same row. Context-aware representations are what the rest of the network produces.
**Why it matters:** The embedding rows are where the model stores per-token meaning. Tokens that behave similarly in the data end up with similar rows. When `tie_embeddings` is on, the same matrix is reused (transposed) as the LM head, so the input lookup and the output projection share weights. When `tie_embeddings` is off, the LM head is a separate matrix and the token embedding rows are only used at the input.