README.md

metadata

title: Inside GPT-2 Embeddings
emoji: 🔍
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.28.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: Token embeddings, analogies and a shared prompt map
preload_from_hub:
  - openai-community/gpt2
  - sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

Inside GPT-2: embeddings

An interactive teaching demo about how text becomes vectors. Runs on a free CPU Space. Companion Space: Inside GPT-2: attention.

Tab	What it shows
Prompt map	A shared, live 2D map of 320 common English words in 8 groups. Anyone can type a word or a prompt in any language and it appears on the map next to the most similar words, with dotted lines to its 5 true nearest words. Switch between a UMAP and a PCA layout.
Tokens → vectors	How GPT-2's tokenizer splits text (with ids), and the 768-number token embedding, position embedding and their sum for every token. Emoji and rare characters show up as raw bytes.
Neighbours & arithmetic	Nearest tokens by cosine similarity in GPT-2's embedding table, and analogies like king - man + woman. A checkbox shows what happens without the usual "exclude the input words" trick.
Compare your own lists	Your own words or sentences (grouped with # headers) in 2D or 3D with PCA, plus a similarity matrix. Compares GPT-2 static embeddings, GPT-2 hidden states and the multilingual sentence-embedding model.
Same word, different meaning	The same word (e.g. bank, light) in different sentences, and how GPT-2's vectors for it drift apart layer by layer.

Prompt map

• Model: paraphrase-multilingual-MiniLM-L12-v2 (384 dimensions, mean pooling). German and English with the same meaning land close together ("Mach das Licht aus" vs. "Turn off the lights": cosine 0.86).
• Reference words: reference_words.py, 8 groups × 40 words. Edit that file to change the map.
• Shared between visitors: prompts are kept in memory only (max. 300, max. 200 characters each, one every 1.5 s per visitor) and disappear when the Space restarts. The app warns visitors that what they type is visible to everyone. A visitor's own prompts are recognised through browser storage, so they stay "theirs" after a reload.
• Teacher controls: add a Space secret called ADMIN_KEY. Whoever enters it can remove the most recent prompt or clear the map.
• Optional: a Space variable COMPANION_URL adds a link to the attention Space at the top.

Why UMAP (and not t-SNE) next to PCA

The map is fitted once at startup on the 320 words. New prompts are then placed on the finished map without moving anything. I compared the three layouts on the 320 words (2 CPU threads):

	PCA	t-SNE	UMAP
Trustworthiness (k = 10)	0.76	0.94	0.93
10 nearest neighbours kept in 2D	21 %	55 %	54 %
Group of the 5 nearest 2D points correct	54 %	89 %	89 %
Held-out words placed in the right group (5-fold, 3 seeds)	51 %	76–79 %	80–84 %
Held-out words: 10 nearest neighbours kept	18 %	46–49 %	44–46 %
18 test prompts (9 English, 9 German) in the right group	10 / 18	17 / 18	18 / 18
Fit time	< 0.1 s	~1 s	~0.4 s (+ ~9 s one-off compile at startup)

t-SNE and UMAP make equally good maps of the words themselves. UMAP was chosen because it has a built-in way to place new points on an existing map (transform), while t-SNE needs a workaround (here: a weighted average of the 5 nearest words' positions). UMAP also put held-out words and test prompts into the right group slightly more often. Its costs are an extra dependency (umap-learn with numba) and about 10 seconds of extra startup. PCA stays in as the honest, linear comparison: it keeps only about 10 % of the variation in 2D, which is why its groups overlap.

Notes on method

• Contextual similarities are computed on z-scored hidden states (mean and std per dimension from a fixed reference paragraph). Raw GPT-2 hidden states have a large shared offset and a few outlier dimensions that push every cosine similarity towards 1. The toggle shows the raw values.
• Words split into several tokens use the mean of their token vectors (neighbours, arithmetic, static map) or the last token (contextual tab). The UI says when this happens.
• Hidden state 0 is the input embedding, 12 is after GPT-2's final LayerNorm.

Run locally

pip install -r requirements.txt gradio==6.28.0
python app.py