Blame for README.md

---
title: Token Visualiser
emoji: 🔬
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
short_description: Token embeddings, vector arithmetic and analogies
---

# Token Visualiser

Explore the numbers inside language-model tokens. Compare embeddings, find nearest tokens, calculate vectors and visualise relationships in 2D or 3D.

## Use this Space

1. Choose **Hugging Face**, pick a suggested model (SmolLM2, GPT-2, Gemma or Qwen) or paste a repository ID/model URL, select compatible weights, and click **Load into Space**. Weights download directly to the Space server; no local download/upload is needed. Public models need no token. Gated/private models require your own approved access and a read token.
2. Alternatively, choose **Upload files** to upload a GGUF file or safetensors weights, with an optional matching tokenizer JSON.
3. Select tokens, compare coloured curves, search neighbours, or use **Vector tools → Relationships** for arbitrary analogies and averaged example pairs.

The chart stays visible above a separately scrolling controls pane. **Clear model** unloads the model and returns to the loading screen.

### Hosted limits and data

- CPU-only; no GPU is needed. Shared CPU performance and available RAM limit very large models.
- Hub downloads: up to 16 GB per selected model by default, with a known download size and enough free disk space (2 GB reserved for the app). Complete cached files are reused. The owner can configure `TOKEN_VISUALISER_MAX_DOWNLOAD_GB`. Only one hosted model download runs at a time per app process. Embedding shards are selected automatically from recognised safetensors indexes.
- Uploads: up to 1 GB per file and 2 GB per session load. Uploaded copies are removed when the model is cleared.
- Tokens entered into the UI are sent to the Space server and Hugging Face for access. They are not saved as a login and are cleared after loading. The Space owner operates the server.
- Authenticated Hub downloads use separate session folders, removed on Clear model. Public downloads may be reused across visitors. Space restarts can clear all cached data.
- No server filesystem browsing, host path inputs, pickle models, or remote repository Python execution.

## Local version and feature documentation

# Token Visualiser

A local viewer for a language model's stored token embeddings. Select tokens and compare every embedding dimension on an interactive chart; inspect exact values and export CSV.

## Find similar tokens

Select a token on the chart, choose it under **Find neighbours of**, then click **Find closest tokens**. The app performs an exact cosine similarity search over every row of the selected embedding tensor in bounded batches. GGUF batches are dequantized as needed; no GPU or full floating-point model copy is required. A progress indicator tracks the scan. Results are cached for the current session and cleared when a model is loaded.

The ranked list excludes the source token and rows with zero or nonfinite embeddings. Change **Results** to show up to 50 neighbours immediately after a search. Each row has an **Add to chart** button to compare its vector in a distinct colour. Added neighbours stay plotted when the result count changes; remove them under **Added neighbours on chart**. The chart supports up to 20 tokens in total. Scores range from −1 to 1, with higher scores indicating closer directions. Similarity uses all embedding dimensions, with no projection. Nearby representations can reflect spelling, token boundaries or usage as well as semantic meaning. On quantized models, scores use the dequantized stored values.

## Run on Windows

Install Python 3.11 or newer, open PowerShell in this folder, and run:

```powershell
powershell -ExecutionPolicy Bypass -File .\start.ps1
```

The launcher creates a project-local virtual environment, installs dependencies, and opens Streamlit at http://localhost:8501. Package installation requires internet access. Model inspection stays local and needs no GPU. Stop the server with Ctrl+C.

## Load your model

1. The app starts on a full-page **Load a model** screen. Click **Open model…** to browse drives and folders and choose a `.gguf` or `.safetensors` file. You can also enter a path manually, including a folder containing safetensors shards.
2. Under **Tokenizer (optional)**, enter the matching `tokenizer.json` file or its folder if needed. The app also detects it beside the model. Click **Load model**.
3. After loading, the chart stays visible in the upper half of the page. The lower half scrolls independently and has four columns: token selection, nearest-token search, vector tools, and chart/model settings with statistics and export. Vector tools has **Calculations** and **Relationships** tabs. Check the embedding tensor selection under **Chart & model**. Common Gemma names are `token_embd.weight` (GGUF) and `model.embed_tokens.weight` (safetensors).
4. Type text and select its tokens. Without a tokenizer JSON, GGUF supports vocabulary substring search; any model supports manual token IDs.
5. Click **Clear model** at the top to unload the model, reset the chart and similarity results, and return to the loading screen.

The vocabulary and tokenizer must match the model. GGUF vocabulary search uses raw token spellings and is not equivalent to encoding text. The app does not reconstruct a tokenizer from GGUF metadata. Use a matching tokenizer JSON for exact text segmentation. Tokenizer special tokens are not automatically added.

## Load from Hugging Face

On the loading screen, open **Hugging Face** and **Choose from list** for SmolLM2 135M, GPT-2, Gemma 3 1B Q4 GGUF or Qwen3 0.6B. Alternatively choose **Paste URL or repository ID** and enter any supported model, then click **Find model files**. The suggestions download on first use; they do not limit custom models or require a local download and upload. Repository, tree, blob and resolve URLs are supported. You can also specify a branch, tag or commit under **Revision & access token (optional)**. Metadata and downloads are pinned to the resolved commit so all files come from the same version.

Choose the weights file, review the download size and file list, then click **Load into Space** (hosted) or **Download & load** (local). The hosted button downloads directly onto the Space server. The app supports single-file GGUF, standalone safetensors, and safetensors indexes. For indexed models it downloads only shards containing recognised input embedding names; if none are recognised, it downloads all shards from the index. A matching `tokenizer.json` is downloaded from the same repository when present. Split GGUF and PyTorch `.bin` files are not supported. Safetensors dtype restrictions below still apply.

Public models need no token. For private or gated models, enter a Hugging Face read token in the password field. Gated models require access to be accepted or approved on the model's Hugging Face page. The app passes the token directly to the Hub client, does not save it as a login, redacts it from displayed errors, and clears the input after loading. Token and metadata controls are scoped to the current session.

Files are downloaded into this project's git-ignored `.model-cache` folder and reused on subsequent loads. **Clear model** unloads the model without deleting cached downloads. Downloads require internet access; vector inspection and calculations run locally. No remote repository Python code is executed.

## Calculate vectors

In **Calculations**, choose **Vector A**, **Subtract (A − B)** or **Add (A + B)**, and **Vector B**. Click **Calculate & add to chart**. The operation applies to corresponding dimensions, producing a derived vector with a `calc:` label, distinct colour and dashed line. You can optionally name the result and use earlier calculations as operands.

Calculated curves are snapshots: they remain when their original tokens or earlier calculations are removed from the chart. Use their **Remove** buttons to delete them. They appear in statistics, exact values and CSV exports, and count towards the 20-curve limit. Clearing the model clears all calculations. These are derived vectors, rather than new vocabulary entries.

## Explore relationships and analogies

Add your input tokens or calculated vectors to the chart, then open **Vector tools → Relationships**. Choose one of three tasks:

- **Compare relationships:** select A, B, C and D. Compare `B − A` with `D − C`, including full-dimensional cosine, angle, length ratio, and the L2 error between `C + B − A` and D. Zero offsets have undefined direction metrics.
- **Complete analogy:** select A, B and C, optionally providing an expected D. The prediction is `C + B − A`.
- **Average example pairs:** add up to ten A → B pairs. Their equally weighted mean offset is applied to C; optional D provides a comparison or expected answer. Example pairs are stored as snapshots.

Click **Show relationship** to apply the inputs and switch the upper chart to **Relationship 3D**. The **Chart view** selector also offers **Relationship 2D** and the original **Dimensions** curves. One PCA is fitted to the input endpoints. The predicted endpoint and other chart vectors use the same axes. The title reports the fraction of input variance retained; scores and search always use the full embedding vectors. Axes use equal scales, and 3D supports rotation. Input points are labelled A/B/C/D (or A1/B1, A2/B2 for multiple examples), with full names on hover.

Click **Find analogy answers** to search the vocabulary around the last shown prediction. Source-pair tokens and C are excluded; a valid expected vocabulary token D is assigned a rank among all eligible tokens, even when outside the displayed top 50. Expected calculated vectors receive geometric comparisons but cannot have a vocabulary rank. A zero or nonfinite query reports an error. Each result can be added to the chart and is projected into the relationship view using the existing axes.

Use **Add prediction as calc** or **Add mean offset as calc** to keep a result in the dimension chart, exact values, statistics and CSV export. The nearest-token panel also accepts calculated curves as query vectors. Relationship snapshots, examples, calculations and search caches are reset by **Clear model**.

## Scope

- X is embedding dimension index (starting at zero); Y is the stored weight value.
- These are input embeddings, not all model parameters or context-dependent activations. Model-specific embedding scaling is not applied.
- GGUF is memory-mapped and only selected rows are dequantized. Quantization support depends on the installed gguf package; unsupported types show an error.
- Safetensors supports F64/F32/F16/BF16 matrix rows and sharded folders. Packed quantized safetensors and PyTorch pickle files are not supported.
- Local paths are read by the Python process; keep the server bound to localhost.

## Checks

The app can be hosted as a Docker Hugging Face Space. Run `python scripts/prepare_space.py` to create the explicit `space-build` source package with pinned direct dependencies. Publish those files using the Docker SDK on port 7860. Hosted mode replaces local-drive browsing with per-session uploads, defaults to 16 GB per Hub model download, and separates authenticated download folders by session. The owner can set the positive integer environment variable `TOKEN_VISUALISER_MAX_DOWNLOAD_GB` to change this limit. Before downloading, the app checks remaining bytes against free disk space, reserving 2 GB for the running app; complete cached files do not need additional space. Hosted model downloads are limited to one at a time per app process, with a fresh capacity check before starting. Larger models can still exceed available RAM or make full-vocabulary searches slow. Upload limits remain 1 GB per file and 2 GB total. Local Windows use keeps the file picker and path controls, with no app-imposed model download limit.

```powershell
.\.venv\Scripts\python.exe -m unittest discover -s tests -v
```