ablation-lab / content /components /misc.token_embedding.md
philippyt's picture
more manifests, update user facing component descriptions, instrument generate in backend
227a396
|
Raw History Blame Contribute Delete
1.56 kB
metadata
id: misc.token_embedding
level: component
title: Token Embedding
short: A lookup table that turns each token ID into a learned d_model vector.
related_concepts: []

The first thing the model does to your input is turn each token ID into a vector. The token embedding is the table that does that:

tok_embed = nn.Embedding(vocab_size, d_model)

It is a vocab_size × d_model matrix of learned parameters. For an input ID i, the model reads row i of the table and emits a d_model-dimensional vector. That vector is the model's input representation for that token.

In this lab vocab_size = 72 (one row per character in the TinyStories corpus, since we use a char-level tokenizer) and d_model = 128. The table is 72 × 128 = 9,216 learned parameters.

Two things the token embedding does not do:

  1. It does not add positional information. Position is handled by RoPE inside attention, or by a separate learned absolute table, depending on pos_embed.
  2. It does not depend on context. Every occurrence of the same token starts from the same row. Context-aware representations are what the rest of the network produces.

Why it matters: The embedding rows are where the model stores per-token meaning. Tokens that behave similarly in the data end up with similar rows. When tie_embeddings is on, the same matrix is reused (transposed) as the LM head, so the input lookup and the output projection share weights. When tie_embeddings is off, the LM head is a separate matrix and the token embedding rows are only used at the input.