---
title: Inside GPT-2 Embeddings
emoji: 🔍
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.28.0
python_version: "3.12"
app_file: app.py
pinned: false
license: mit
short_description: Token embeddings, analogies and a shared prompt map
preload_from_hub:
  - openai-community/gpt2
  - sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
---

# Inside GPT-2: embeddings

An interactive teaching demo about how text becomes vectors. Runs on a free CPU Space. Companion Space: *Inside GPT-2: attention*.

| Tab | What it shows |
|---|---|
| **Prompt map** | A shared, live 2D map of 320 common English words in 8 groups. Anyone can type a word or a prompt in any language and it appears on the map next to the most similar words, with dotted lines to its 5 true nearest words. Switch between a UMAP and a PCA layout. |
| **Tokens → vectors** | How GPT-2's tokenizer splits text (with ids), and the 768-number token embedding, position embedding and their sum for every token. Emoji and rare characters show up as raw bytes. |
| **Neighbours & arithmetic** | Nearest tokens by cosine similarity in GPT-2's embedding table, and analogies like `king - man + woman`. A checkbox shows what happens without the usual "exclude the input words" trick. |
| **Compare your own lists** | Your own words or sentences (grouped with `# headers`) in 2D or 3D with PCA, plus a similarity matrix. Compares GPT-2 static embeddings, GPT-2 hidden states and the multilingual sentence-embedding model. |
| **Same word, different meaning** | The same word (e.g. *bank*, *light*) in different sentences, and how GPT-2's vectors for it drift apart layer by layer. |

## Prompt map

- **Model:** [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) (384 dimensions, mean pooling). German and English with the same meaning land close together ("Mach das Licht aus" vs. "Turn off the lights": cosine 0.86).
- **Reference words:** `reference_words.py`, 8 groups × 40 words. Edit that file to change the map.
- **Shared between visitors:** prompts are kept in memory only (max. 300, max. 200 characters each, one every 1.5 s per visitor) and disappear when the Space restarts. The app warns visitors that what they type is visible to everyone. A visitor's own prompts are recognised through browser storage, so they stay "theirs" after a reload.
- **Teacher controls:** add a Space secret called `ADMIN_KEY`. Whoever enters it can remove the most recent prompt or clear the map.
- **Optional:** a Space variable `COMPANION_URL` adds a link to the attention Space at the top.

### Why UMAP (and not t-SNE) next to PCA

The map is fitted once at startup on the 320 words. New prompts are then placed on the finished map without moving anything. I compared the three layouts on the 320 words (2 CPU threads):

| | PCA | t-SNE | UMAP |
|---|---|---|---|
| Trustworthiness (k = 10) | 0.76 | 0.94 | 0.93 |
| 10 nearest neighbours kept in 2D | 21 % | 55 % | 54 % |
| Group of the 5 nearest 2D points correct | 54 % | 89 % | 89 % |
| Held-out words placed in the right group (5-fold, 3 seeds) | 51 % | 76–79 % | 80–84 % |
| Held-out words: 10 nearest neighbours kept | 18 % | 46–49 % | 44–46 % |
| 18 test prompts (9 English, 9 German) in the right group | 10 / 18 | 17 / 18 | 18 / 18 |
| Fit time | < 0.1 s | ~1 s | ~0.4 s (+ ~9 s one-off compile at startup) |

t-SNE and UMAP make equally good maps of the words themselves. UMAP was chosen because it has a built-in way to place new points on an existing map (`transform`), while t-SNE needs a workaround (here: a weighted average of the 5 nearest words' positions). UMAP also put held-out words and test prompts into the right group slightly more often. Its costs are an extra dependency (`umap-learn` with `numba`) and about 10 seconds of extra startup. PCA stays in as the honest, linear comparison: it keeps only about 10 % of the variation in 2D, which is why its groups overlap.

## Notes on method

- Contextual similarities are computed on z-scored hidden states (mean and std per dimension from a fixed reference paragraph). Raw GPT-2 hidden states have a large shared offset and a few outlier dimensions that push every cosine similarity towards 1. The toggle shows the raw values.
- Words split into several tokens use the mean of their token vectors (neighbours, arithmetic, static map) or the last token (contextual tab). The UI says when this happens.
- Hidden state 0 is the input embedding, 12 is after GPT-2's final LayerNorm.

## Run locally

```bash
pip install -r requirements.txt gradio==6.28.0
python app.py
```