Sillage
A frozen language model that remembers what it reads — 4.2 MB, no gradients, no fine-tuning, no vector database.
sillage (n., French) — the trace left behind by something that has passed: a ship's wake, a scent in a room. What a model keeps of what it read.
1It has already read a paper
The memory below has read — the paper describing this very mechanism, which GPT-2 has never seen. Same prompt, same greedy decoding, one column with the memory and one without. Pick a beginning:
GPT-2, frozen
GPT-2 + Sillage memory
2What it does to a document it has never seen
An operations manual invented for this demo, so GPT-2 cannot have seen it. Read once, left to right: the first half builds the memory, the second half is measured with it — the papers' own dev/test split — and every token is scored before being written.
Predictions the memory corrected
The frozen model had no way of knowing these. They are facts that exist only in that document.
| context | what came next | what frozen GPT-2 said |
|---|
3How much better, exactly
36k tokens of technical text the model had never seen, frozen GPT-2 124M, every system tuned identically on a held-out prefix, 95 % bootstrap confidence intervals.
| system | perplexity | change | memory used |
|---|---|---|---|
| frozen GPT-2 | 31.2 | — | 0 |
| + RAG-style retrieve & rescore | 29.9 | −4 % | corpus + index |
| + kNN-LM, unbounded store | 23.6 | −24 % | 55 MB, grows forever |
| + this memory (fixed) | 19.2 | −38 % | 4.2 MB, constant |
| + memory and fast weights | 16.6 | −47 % | 7.4 MB, constant |
Paired block bootstrap P = 1.000 against the unbounded datastore, replicated over 5 random seeds and on a second model (Qwen3-0.6B).
4Where it does not work
On long, low-repetition narrative, an unbounded kNN-LM still wins (+0.048 vs +0.007 nats). This memory captures verbatim recurrence, not paraphrase. The boundary is measured and published rather than hidden.
Three results that did not work are published too. Hidden states make poor Hebbian keys — their geometry is too entangled. Surprise gating helps the memory and hurts the fast-weight adapter, because the delta rule already carries its own error term. And calibrating the readout on your own stream loses to a proper tuning (+0.109 against +0.120 nats), because the calibration window is read by a colder memory than the one it will govern.
A fixed matrix also saturates at long horizons, near 0.5 writes per parameter. Forgetting recovers ×2.3 of the gain and a 4× larger matrix recovers ×3.4 — both are one flag away.
5Run it yourself
Every output on this page came out of the command line below, on a laptop CPU. It works with any causal language model — a Hugging Face id or a local folder — not just the two the papers measured.
pip install sillage sillage index notes.md # instant: no model needed sillage ask "what did the report say?" sillage read notes.md # memorise it sillage complete "The report said" # generate WITH the memory sillage status # what it knows, tier by tier sillage read notes.md --model HuggingFaceTB/SmolLM2-135M # any LM