Sillage: the demo, its memory of paper 1, and the recorded runs

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+demo.gif filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,15 +1,87 @@
 ---
-title: Sillage
-emoji: 🐢
-colorFrom: purple
-colorTo: blue
+title: Sillage — a frozen LM that remembers what it reads
+emoji: 🌊
+colorFrom: blue
+colorTo: indigo
 sdk: gradio
-sdk_version: 6.26.0
-python_version: '3.12'
+sdk_version: 6.13.0
 app_file: app.py
 pinned: false
 license: mit
 short_description: A frozen GPT-2 given a 4 MB memory. No gradients, no index.
+models:
+  - openai-community/gpt2
+preload_from_hub:
+  - openai-community/gpt2
+tags:
+  - associative-memory
+  - test-time-learning
+  - continual-learning
+  - hebbian
+  - gradient-free
+  - knn-lm
+  - hyperdimensional-computing
+  - cpu
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Sillage — the demo
+
+A frozen language model reads a document once and gets better at it, with no
+gradients, no fine-tuning and no vector database. Three screens:
+
+1. **It has already read a paper.** A memory of paper 1 (8 969 tokens) ships
+   with this Space, so the first thing you see is a frozen GPT-2 and a
+   remembering GPT-2 completing the same sentence side by side. No wait.
+2. **Give it your own text.** Up to 1 600 tokens, read once. The first half
+   builds the memory, the second half is *measured* with it — the papers' own
+   dev/test split — and you get the list of predictions the memory corrected.
+3. **Where it does not work.** The regime boundary, in the authors' words.
+
+Code, four preprints with DOIs and the reproduction pipeline:
+<https://github.com/riscoss63/sillage> · `pip install sillage` (this Space
+runs the published 1.0.1)
+
+## Hardware: ZeroGPU, and why the loop stays on the CPU
+
+Hosting a Gradio Space on free `cpu-basic` now requires PRO, but a personal
+account in good standing (verified email, older than 30 days) may host **two
+ZeroGPU Spaces for free**. Pick ZeroGPU in the Space settings — the creation
+form on the website lets you choose it; `hf repos create` does not, which is
+why that command fails with a 402.
+
+ZeroGPU quota is charged to the **visitor** (2 minutes a day for an anonymous
+one, 5 for a signed-in free account) for the time spent inside a
+`@spaces.GPU` function. This demo is mostly *not* GPU work: the Hebbian
+mechanism is numpy on the CPU. So the app is split accordingly:
+
+* `forward_all` — every frozen forward pass of the document, in one decorated
+  call, a couple of seconds of GPU;
+* the read loop — the memory, the adapter, the abstention, all outside the
+  decorator, on the CPU, costing the visitor nothing;
+* `compare` / `compare_user` — short generations, decorated, a second or two.
+
+Reading a 1 600-token document therefore costs a visitor a few seconds of
+quota rather than half a minute.
+
+## What is in this repository
+
+| file | role |
+|---|---|
+| `app.py` | the three screens |
+| `sample.py` | two documents to click: an invented operations manual, and paper 1 |
+| `build_state.py` | rebuilds `state/`, the memory the demo starts with |
+| `state/` | 7.1 MB: a GPT-2 memory that has already read paper 1 |
+| `index.html`, `data.json`, `capture.py`, `build_page.py` | a static fallback page built from recorded runs — used only if the Space is switched to `sdk: static`, which is free for everyone with no eligibility rules |
+| `demo.gif` | the two-session terminal recording |
+
+## Running it locally
+
+```bash
+pip install sillage gradio
+python build_state.py path/to/a/document   # the memory the demo starts with
+python app.py                              # http://127.0.0.1:7860
+```
+
+There is no GPU-specific code path to test: `@spaces.GPU` is a no-op when the
+`spaces` package is absent, and the model goes to `cuda` only when there is
+one.

app.py

@@ -0,0 +1,454 @@
+"""Sillage — Hugging Face Space.
+
+Three screens, in the order that convinces:
+
+  1. "It has already read this paper."  A memory built at deploy time is
+     already loaded, so the very first thing a visitor sees is a frozen model
+     and a remembering model completing the same sentence side by side. No
+     waiting, no setup, no account.
+  2. "Give it your own text."  Half a minute, then the same comparison on
+     THEIR document, plus the list of predictions the memory actually
+     corrected -- the part that turns a number into a fact.
+  3. "Where it does not work."  The regime boundary, stated by the authors
+     before anyone else has to point it out.
+
+Everything here runs on `pip install sillage`; the read loop below is the
+library's own, instrumented to report which tokens changed.
+
+On ZeroGPU the visitor pays quota only for time spent inside a @spaces.GPU
+function, so the split matters: the frozen forward passes are batched into
+one decorated call, the Hebbian mechanism stays in numpy on the CPU, and the
+suggested prompts of screen 1 are served from runs recorded at build time.
+Looking at the page costs nothing; a click costs a second or two.
+"""
+
+import collections
+import io
+import json
+import os
+import re
+import shutil
+import tempfile
+
+import gradio as gr
+import numpy as np
+import torch
+
+from sillage import Sillage
+from sillage.core import CAP
+
+try:                                  # ZeroGPU: a real GPU is attached only
+    import spaces                     # inside a decorated function
+    GPU = spaces.GPU
+except ImportError:                   # anywhere else the decorator is a no-op
+    def GPU(*args, **kwargs):
+        if args and callable(args[0]):
+            return args[0]
+        return lambda fn: fn
+
+import sample
+
+STATE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "state")
+MAX_TOKENS = 1600                 # 160 MB of logits, ~30 s end to end
+MIN_TOKENS = 900                  # below this, the split has nothing to say
+WINDOW, STRIDE = 1024, 512
+PAPER = "Sillage (paper 1), 8 969 tokens"
+
+# Chosen by running them: each one makes the two columns visibly diverge,
+# and the right-hand column recites the paper rather than inventing.
+EXAMPLE_PROMPTS = [
+    "On a 36k-token stream of novel technical text, the memory",
+    "At 500k tokens the fixed matrix",
+    "surprise gating quadruples the gain of",
+    "Every write is gated by the model's own",
+]
+
+
+# --------------------------------------------------------------- backend ---
+
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+memory = Sillage(model="gpt2", state=STATE, device=DEVICE, quiet=True)
+frozen = Sillage(model="gpt2", state=tempfile.mkdtemp(), device=DEVICE,
+                 quiet=True)
+
+# Each visitor who reads a document leaves a 7.4 MB state behind. Keep the
+# most recent few so the follow-up completion still works, and delete the
+# rest: a public Space should not fill its disk with strangers' memories.
+SESSIONS = collections.deque(maxlen=8)
+
+
+def new_session_dir():
+    if len(SESSIONS) == SESSIONS.maxlen:
+        shutil.rmtree(SESSIONS[0], ignore_errors=True)
+    d = tempfile.mkdtemp(prefix="sillage-session-")
+    SESSIONS.append(d)
+    return d
+
+
+def models():
+    """Load GPT-2 once and let both assistants share the frozen weights."""
+    tok, model = memory.load_model()
+    frozen._tok, frozen._model = tok, model
+    frozen.device = memory.device
+    return tok, model
+
+
+# ZeroGPU asks for the weights to be placed at startup rather than inside a
+# decorated call, where a cold load would eat into the visitor's quota and
+# could outlast the function's duration budget.
+models()
+
+
+# Screen 1's suggested prompts were run once, at build time, into data.json.
+# Greedy decoding over a fixed memory is deterministic, so a cached answer is
+# the same answer -- and serving it costs the visitor no GPU quota at all.
+# Anything a visitor types instead is computed live.
+CACHED = {}
+try:
+    for _c in json.load(io.open(os.path.join(
+            os.path.dirname(os.path.abspath(__file__)), "data.json"),
+            encoding="utf-8"))["completions"]:
+        CACHED[_c["prompt"].strip()] = (
+            _c["prompt"] + _c["frozen"], _c["prompt"] + _c["memory"],
+            _c["same"])
+except Exception:                     # no capture yet: everything runs live
+    CACHED = {}
+
+
+def compare(prompt, n_tokens=24):
+    """Screen 1. Recorded prompts are served from cache, the rest is live."""
+    hit = CACHED.get((prompt or "").strip())
+    if hit:
+        return hit[0], hit[1], _verdict(hit[2])
+    return compare_live(prompt, n_tokens)
+
+
+@GPU(duration=30)
+def compare_live(prompt, n_tokens=24):
+    """The decorated path: generation is the only part that needs the model.
+
+    Twenty-four tokens twice is a second or two of GPU, which is what a
+    visitor's daily ZeroGPU quota can afford many times over.
+    """
+    return _compare(prompt, n_tokens)
+
+
+def _verdict(same):
+    return ("identical here — the memory abstains when it is not confident, "
+            "which is the point of the abstention threshold" if same else
+            "the two columns diverge: everything after the prompt on the "
+            "right comes from what was read, not from GPT-2's weights")
+
+
+@GPU(duration=60)
+@torch.no_grad()
+def forward_all(ids):
+    """Every frozen forward pass for a document, in one GPU visit.
+
+    The Hebbian loop that follows is numpy on the CPU: keeping it outside
+    this function is what stops a 30-second read from costing 30 seconds of
+    the visitor's GPU quota. Returned in fp16 -- 1600 positions of GPT-2
+    logits are 160 MB, and the loop only ever reads them.
+    """
+    tok, model = models()
+    x = torch.tensor(ids, device=memory.device)
+    n = len(ids) - 1
+    logits = np.empty((n + 1, model.config.vocab_size), dtype=np.float16)
+    hidden = np.empty((n + 1, model.config.n_embd), dtype=np.float16)
+    a = 0
+    while a < n:
+        w = min(WINDOW, len(ids) - a)
+        out = model(x[a:a + w].unsqueeze(0), output_hidden_states=True)
+        lo = 0 if a == 0 else WINDOW - STRIDE
+        logits[a + lo:a + w] = (
+            out.logits[0, lo:w].to(torch.float16).cpu().numpy())
+        hidden[a + lo:a + w] = (
+            out.hidden_states[-1][0, lo:w].to(torch.float16).cpu().numpy())
+        if a + w >= len(ids):
+            break
+        a += STRIDE
+    return logits, hidden
+
+
+def _compare(prompt, n_tokens=24):
+    """The same prompt, decoded greedily, with and without the memory."""
+    if not prompt.strip():
+        return "", "", ""
+    models()
+    a = frozen.complete(prompt, n=n_tokens)
+    b = memory.complete(prompt, n=n_tokens)
+    return prompt + a, prompt + b, _verdict(a.strip() == b.strip())
+
+
+def suggest_prompt(text, tail):
+    """Half of a sentence whose opening recurs — so the memory knows the rest.
+
+    A prompt taken at random usually lands where the memory has nothing to
+    say, and the demo falls flat through no fault of the mechanism. Picking a
+    sentence whose first words already appeared earlier is the honest way to
+    show the memory doing what it is for.
+    """
+    best = ""
+    for sentence in re.split(r"(?<=[.:])\s+|\n+", tail):
+        words = sentence.split()
+        if len(words) < 9:
+            continue
+        if text.count(" ".join(words[:5])) >= 2:
+            cut = " ".join(words[:max(5, len(words) // 2)])
+            if len(cut) > len(best):
+                best = cut
+    return best or " ".join(tail.split()[:10])
+
+
+def read_and_report(text, progress=gr.Progress()):
+    """Stream a document through a fresh memory, reporting what it changed.
+
+    This is `sillage.runtime.Sillage.read_text`, unrolled for two reasons:
+    the demo collects the positions where the memory turned a wrong
+    prediction into a right one, and the frozen forward passes are done in
+    one batch beforehand (on the GPU when there is one) so that this loop --
+    plain numpy, no model -- costs the visitor nothing. Strictly prequential
+    either way: every token is scored before it is written.
+    """
+    text = (text or "").strip()
+    tok, lm = models()
+    state_dir = new_session_dir()
+    fresh = Sillage(model="gpt2", state=state_dir, device=memory.device,
+                    quiet=True)
+    fresh._tok, fresh._model = tok, lm
+    mem = fresh.mem
+
+    ids = np.array(tok.encode(text), dtype=np.int64)[:MAX_TOKENS]
+    n = len(ids) - 1
+    if n < MIN_TOKENS:
+        return (None, f"That is {max(n, 0)} tokens. Below about "
+                f"{MIN_TOKENS} there is nothing to measure honestly: the "
+                f"first half of a document is what builds the memory, and "
+                f"the abstention threshold needs a few hundred observations "
+                f"before it will let the memory speak at all. Paste a longer "
+                f"one — or click an example below.", None, "", "")
+    dev_end = n // 2            # first half: the memory reads and calibrates
+    mem.new_stream()
+    thrG = thrS = np.inf        # silent until the dev half is behind us
+    nll_b = nll_f = nll_m = 0.0
+    cnt = active = 0
+    fixes = []
+    logits, hidden = forward_all(ids)      # the only GPU visit
+    for j in range(n):
+        if j == dev_end:        # the papers' protocol: dev decides, test tells
+            thrG, thrS = mem.thresholds()
+        truth = int(ids[j + 1])
+        lb = logits[j].astype(np.float32)
+        mx = lb.max()
+        lpb = lb - (mx + np.log(np.exp(lb - mx).sum()))
+        lp = float(lpb[truth])
+        la, phi = mem.adapt(lb, hidden[j].astype(np.float32))
+        m2 = la.max()
+        p_ad = np.exp(la - m2)
+        p_ad /= p_ad.sum()
+        lp_f = float(np.log(max(p_ad[truth], 1e-30)))
+        qG = mem.step_key(int(ids[j]))
+        uG, sG = mem.scores(mem.M, qG)
+        mem.res_G.append(float(sG.max()))
+        pc = mem.cold_lookup(truth)
+        p_true = mem.mix_true(np.exp(lp_f), sG, truth, None, pc, thrG, thrS)
+        speaks = float(sG.max()) >= thrG
+        if j >= dev_end:
+            active += speaks
+            nll_b += -lp
+            nll_f += -lp_f
+            nll_m += -np.log(max(p_true, 1e-30))
+            cnt += 1
+        if speaks and len(fixes) < 12:
+            # mix_true takes the probability of one token, mix_full takes the
+            # whole distribution -- so the cold tier is asked twice, once for
+            # each shape
+            full = mem.mix_full(p_ad.copy(), sG, None, mem.cold_lookup(),
+                                thrG, thrS)
+            said, now = int(np.argmax(p_ad)), int(np.argmax(full))
+            if now == truth != said:
+                ctx = tok.decode(ids[max(0, j - 9):j + 1])
+                fixes.append([" ".join(ctx.split())[-58:],
+                              repr(tok.decode([truth])),
+                              repr(tok.decode([said]))])
+        g = min(CAP, max(0.0, -lp))
+        mem.write_all(qG, uG, None, None, truth, g, phi, p_ad)
+        if j % 128 == 0:
+            progress(j / n, desc=f"reading {j}/{n} tokens")
+    fresh.index.add(text, "your document")
+    fresh.save()
+
+    ppl = [float(np.exp(v / cnt)) for v in (nll_b, nll_f, nll_m)]
+    plot = {"frozen GPT-2": ppl[0], "+ fast weights": ppl[1],
+            "+ memory": ppl[2]}
+    knn = n * 1536 * 4 / 1e6          # one 1536-float key+value per token
+    summary = (
+        f"**Read once, left to right: {dev_end} tokens built the memory, the "
+        f"next {cnt} were measured with it.** That split is the papers' own "
+        f"protocol — the numbers below are on text the memory had not seen "
+        f"when it scored them, and every token was scored *before* being "
+        f"written.\n\n"
+        f"Perplexity on that second half: {ppl[0]:.2f} frozen → "
+        f"{ppl[1]:.2f} with the rank-16 adapter → **{ppl[2]:.2f}** with the "
+        f"memory on top — **{100 * (1 - ppl[2] / ppl[0]):.0f}% lower**. The "
+        f"memory spoke on {100 * active / cnt:.0f}% of those positions and "
+        f"kept quiet on the rest; abstaining when it has nothing to say is "
+        f"what keeps it from doing harm.\n\n"
+        f"State on disk: **7.4 MB**, and it would still be 7.4 MB after a "
+        f"million tokens. A kNN-LM datastore over the same text would "
+        f"already hold about {knn:.1f} MB, and would keep growing.")
+    return (plot, summary, fixes or None, state_dir,
+            suggest_prompt(text, tok.decode(ids[dev_end:])))
+
+
+def read_then_compare(text, progress=gr.Progress()):
+    """Read, report, and hand back a prompt worth trying plus the new state."""
+    plot, summary, fixes, state_dir, suggestion = read_and_report(text,
+                                                                  progress)
+    return (plot, summary, fixes, state_dir,
+            gr.update(value=suggestion, visible=bool(state_dir)),
+            gr.update(visible=bool(state_dir)))
+
+
+@GPU(duration=30)
+def compare_user(prompt, state_dir, n_tokens=12):
+    """The same comparison, against the memory the visitor just built."""
+    if not state_dir or not os.path.exists(os.path.join(state_dir,
+                                                        "state.npz")):
+        return "", "", ("That session's memory has been cleared — read a "
+                        "document again.")
+    if not (prompt or "").strip():
+        return "", "", ""
+    tok, model = models()
+    theirs = Sillage(model="gpt2", state=state_dir, device=memory.device,
+                     quiet=True)
+    theirs._tok, theirs._model = tok, model
+    a = frozen.complete(prompt, n=n_tokens)
+    b = theirs.complete(prompt, n=n_tokens)
+    note = ("identical here — try a phrase that recurs in your document, "
+            "the memory only speaks where it is confident"
+            if a.strip() == b.strip() else
+            "the right-hand column comes from your document, not from "
+            "GPT-2's weights")
+    return prompt + a, prompt + b, note
+
+
+# ------------------------------------------------------------------ ui -----
+
+CSS = """
+.hero {text-align:center}
+.small {font-size:0.9em; opacity:0.75}
+footer {visibility:hidden}
+"""
+
+with gr.Blocks(title="Sillage: a frozen LM that remembers "
+                     "what it reads") as demo:
+    gr.Markdown(
+        "# Sillage\n"
+        "### A frozen language model that remembers what it reads — "
+        "4.2 MB, no gradients, no fine-tuning, no vector database.\n"
+        "[![PyPI](https://img.shields.io/pypi/v/sillage.svg)]"
+        "(https://pypi.org/project/sillage/) "
+        "[![GitHub](https://img.shields.io/badge/code-GitHub-181717?logo=github)]"
+        "(https://github.com/riscoss63/sillage) "
+        "[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.22079016.svg)]"
+        "(https://doi.org/10.5281/zenodo.22079016)",
+        elem_classes="hero")
+    gr.Markdown(
+        "> This Space runs **GPT-2 124M on a free CPU**. Its prose is weak in "
+        "absolute terms — that is not what is on display. What is on display "
+        "is the **difference** between the two columns, and it comes from a "
+        "4.2 MB matrix written while reading, with no gradient anywhere.",
+        elem_classes="small")
+
+    with gr.Tab("1 · It has already read a paper"):
+        gr.Markdown(
+            f"The memory loaded here has read **{PAPER}** — the paper that "
+            "describes this very mechanism. GPT-2 has never seen that text. "
+            "Complete a sentence from it and watch the right-hand column "
+            "recall what the left one cannot know.")
+        prompt = gr.Textbox(label="Beginning of a sentence",
+                            value=EXAMPLE_PROMPTS[0], lines=2)
+        gr.Examples(examples=[[p] for p in EXAMPLE_PROMPTS], inputs=prompt,
+                    label="Try one of these")
+        go = gr.Button("Complete both", variant="primary")
+        with gr.Row():
+            out_frozen = gr.Textbox(label="GPT-2, frozen", lines=6)
+            out_memory = gr.Textbox(label="GPT-2 + Sillage memory", lines=6)
+        verdict = gr.Markdown(elem_classes="small")
+        go.click(compare, prompt, [out_frozen, out_memory, verdict])
+        demo.load(compare, prompt, [out_frozen, out_memory, verdict])
+
+    with gr.Tab("2 · Give it your own text"):
+        gr.Markdown(
+            f"Paste something GPT-2 has never seen: your notes, an internal "
+            f"document, a README, a specification. It is read **once**, left "
+            f"to right, and every token is scored *before* it is memorised — "
+            f"so the numbers below are honest online measurements, not a "
+            f"replay. Capped at {MAX_TOKENS} tokens (about 90 seconds on this "
+            f"free CPU).")
+        user_text = gr.Textbox(label="Your document", lines=12,
+                               value=sample.MANUAL)
+        gr.Examples(
+            examples=[[sample.MANUAL], [sample.paper_excerpt()]],
+            inputs=user_text, example_labels=[
+                "An invented operations manual (GPT-2 cannot know it)",
+                "The opening of paper 1 (novel technical prose)"],
+            label="Or start from one of these")
+        read_btn = gr.Button("Read it", variant="primary")
+        chart = gr.Label(label="Perplexity, lower is better")
+        report = gr.Markdown()
+        fixed = gr.Dataframe(
+            headers=["context", "what came next", "what frozen GPT-2 said"],
+            label="Predictions the memory corrected — the frozen model had "
+                  "no way of knowing these", wrap=True)
+
+        gr.Markdown("### Now ask it to continue a sentence from *your* text")
+        session = gr.State("")
+        your_prompt = gr.Textbox(label="Beginning of a sentence", lines=2,
+                                 visible=False)
+        your_btn = gr.Button("Complete both", variant="primary",
+                             visible=False)
+        with gr.Row():
+            your_frozen = gr.Textbox(label="GPT-2, frozen", lines=5)
+            your_memory = gr.Textbox(label="GPT-2 + the memory you just "
+                                           "built", lines=5)
+        your_note = gr.Markdown(elem_classes="small")
+        read_btn.click(read_then_compare, user_text,
+                       [chart, report, fixed, session, your_prompt, your_btn])
+        your_btn.click(compare_user, [your_prompt, session],
+                       [your_frozen, your_memory, your_note])
+
+    with gr.Tab("3 · Where it does not work"):
+        gr.Markdown(
+            "**The regime matters, and it is measured.**\n\n"
+            "| system | perplexity | memory used |\n|---|---|---|\n"
+            "| frozen GPT-2 | 31.2 | 0 |\n"
+            "| + RAG-style retrieve & rescore | 29.9 | corpus + index |\n"
+            "| + kNN-LM, unbounded store | 23.6 | 55 MB, grows forever |\n"
+            "| + **this memory** (fixed) | **19.2** | **4.2 MB, constant** |\n"
+            "| + memory and fast weights | **16.6** | 7.4 MB, constant |\n\n"
+            "That is on 36k tokens of novel technical text. **On long, "
+            "low-repetition narrative, an unbounded kNN-LM still wins** "
+            "(+0.048 vs +0.007 nats): this memory captures verbatim "
+            "recurrence, not paraphrase. The boundary is mapped in the "
+            "papers rather than hidden.\n\n"
+            "Three results that did *not* work are published too: hidden "
+            "states make poor Hebbian keys; surprise gating helps the memory "
+            "and *hurts* the fast-weight adapter; and calibrating the readout "
+            "on your own stream loses to a proper tuning, because the "
+            "calibration window is read by a colder memory than the one it "
+            "will govern.\n\n"
+            "---\n\n"
+            "**Run it yourself, on any causal LM:**\n\n"
+            "```bash\npip install sillage\nsillage read notes.md\n"
+            "sillage ask \"what did the report say?\"\n"
+            "sillage complete \"The report said\"\n```\n\n"
+            "Four preprints with DOIs, the full reproduction pipeline and "
+            "every number as committed JSON: "
+            "[github.com/riscoss63/sillage](https://github.com/riscoss63/sillage)")
+
+if __name__ == "__main__":
+    # one reading at a time: the whole point is a single CPU
+    demo.queue(default_concurrency_limit=1).launch(
+        theme=gr.themes.Soft(), css=CSS)

build_page.py

@@ -0,0 +1,291 @@
+"""Build index.html for the static Space from the captured runs.
+
+A static Space has no backend, so the page carries its evidence with it: the
+outputs in data.json are inlined, and the page says where they came from.
+Nothing here is illustrative -- every string was produced by the tool.
+
+    python capture.py && python build_page.py
+"""
+
+import io
+import json
+import os
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+
+PAGE = """<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Sillage — a frozen LM that remembers what it reads</title>
+<style>
+:root {
+  --bg:#0d1117; --panel:#161b22; --line:#26303d; --ink:#c9d1d9;
+  --dim:#8b949e; --hl:#79c0ff; --warm:#ffc457; --ok:#3fb950;
+}
+* { box-sizing:border-box }
+body {
+  margin:0; background:var(--bg); color:var(--ink); line-height:1.6;
+  font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Roboto,
+              Helvetica,Arial,sans-serif;
+}
+.wrap { max-width:980px; margin:0 auto; padding:0 20px 80px }
+header { text-align:center; padding:56px 0 24px }
+h1 { font-size:2.6rem; margin:0 0 10px; letter-spacing:-.02em }
+.tag { font-size:1.15rem; color:var(--ink); max-width:640px; margin:0 auto }
+.etym { color:var(--dim); font-style:italic; font-size:.95rem;
+        max-width:560px; margin:18px auto 0 }
+.badges { margin:22px 0 8px; display:flex; gap:8px; justify-content:center;
+          flex-wrap:wrap }
+.badges img { height:20px }
+h2 { font-size:1.5rem; margin:52px 0 6px; letter-spacing:-.01em }
+h2 .num { color:var(--dim); font-weight:400; margin-right:8px }
+.lede { color:var(--dim); margin:0 0 20px }
+.gif { width:100%; border:1px solid var(--line); border-radius:10px;
+       display:block; margin:26px 0 }
+.chips { display:flex; gap:8px; flex-wrap:wrap; margin-bottom:16px }
+.chip {
+  background:var(--panel); border:1px solid var(--line); color:var(--ink);
+  border-radius:999px; padding:7px 14px; cursor:pointer; font-size:.9rem;
+  font-family:inherit; transition:border-color .15s, color .15s;
+}
+.chip:hover { border-color:var(--hl); color:var(--hl) }
+.chip[aria-pressed="true"] { border-color:var(--hl); color:var(--hl);
+                             background:#16273b }
+.cols { display:grid; grid-template-columns:1fr 1fr; gap:14px }
+@media (max-width:760px) { .cols { grid-template-columns:1fr } }
+.col { background:var(--panel); border:1px solid var(--line);
+       border-radius:10px; padding:14px 16px; min-height:170px }
+.col h3 { margin:0 0 10px; font-size:.85rem; text-transform:uppercase;
+          letter-spacing:.08em; color:var(--dim); font-weight:600 }
+.col.win h3 { color:var(--ok) }
+.out { font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+       font-size:.9rem; white-space:pre-wrap; word-break:break-word; margin:0 }
+.out .p { color:var(--dim) }
+.out .new { color:var(--warm) }
+.col.win .out .new { color:var(--hl) }
+.note { color:var(--dim); font-size:.9rem; margin:12px 0 0 }
+table { width:100%; border-collapse:collapse; margin:18px 0; font-size:.92rem }
+th,td { text-align:left; padding:9px 12px; border-bottom:1px solid var(--line) }
+th { color:var(--dim); font-weight:600; font-size:.82rem;
+     text-transform:uppercase; letter-spacing:.06em }
+td.mono,th.mono { font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+                  font-size:.86rem }
+tr.best td { color:var(--hl); font-weight:600 }
+.panel { background:var(--panel); border:1px solid var(--line);
+         border-radius:10px; padding:18px 20px; margin:18px 0 }
+.panel p:first-child { margin-top:0 }
+.panel p:last-child { margin-bottom:0 }
+pre.sh { background:var(--panel); border:1px solid var(--line);
+         border-radius:10px; padding:16px 18px; overflow-x:auto;
+         font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+         font-size:.9rem; color:var(--ink) }
+pre.sh .c { color:var(--dim) }
+a { color:var(--hl) }
+footer { color:var(--dim); font-size:.9rem; text-align:center;
+         margin-top:64px; border-top:1px solid var(--line); padding-top:24px }
+.kbd { font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+       background:#1c2432; border:1px solid var(--line); border-radius:5px;
+       padding:1px 6px; font-size:.85em }
+</style>
+</head>
+<body>
+<div class="wrap">
+
+<header>
+  <h1>Sillage</h1>
+  <p class="tag">A frozen language model that <b>remembers what it reads</b> —
+  4.2&nbsp;MB, no gradients, no fine-tuning, no vector database.</p>
+  <p class="etym">sillage (n., French) — the trace left behind by something
+  that has passed: a ship's wake, a scent in a room. What a model keeps of
+  what it read.</p>
+  <div class="badges">
+    <a href="https://pypi.org/project/sillage/"><img alt="PyPI"
+      src="https://img.shields.io/pypi/v/sillage.svg"></a>
+    <a href="https://github.com/riscoss63/sillage"><img alt="GitHub"
+      src="https://img.shields.io/badge/code-GitHub-181717?logo=github"></a>
+    <a href="https://doi.org/10.5281/zenodo.22079016"><img alt="DOI"
+      src="https://zenodo.org/badge/DOI/10.5281/zenodo.22079016.svg"></a>
+    <a href="https://github.com/riscoss63/sillage/blob/main/LICENSE"><img
+      alt="MIT" src="https://img.shields.io/badge/License-MIT-blue.svg"></a>
+  </div>
+</header>
+
+<img class="gif" src="demo.gif" alt="Two sessions a day apart: the model
+  reads a draft on Monday and halves the perplexity of the next draft on
+  Tuesday, then completes a sentence it can only know from what it read.">
+
+<h2><span class="num">1</span>It has already read a paper</h2>
+<p class="lede">The memory below has read
+<b id="read-what"></b> — the paper describing this very mechanism, which
+GPT-2 has never seen. Same prompt, same greedy decoding, one column with the
+memory and one without. Pick a beginning:</p>
+
+<div class="chips" id="chips"></div>
+<div class="cols">
+  <div class="col"><h3>GPT-2, frozen</h3><p class="out" id="out-frozen"></p></div>
+  <div class="col win"><h3>GPT-2 + Sillage memory</h3>
+    <p class="out" id="out-memory"></p></div>
+</div>
+<p class="note" id="note"></p>
+
+<h2><span class="num">2</span>What it does to a document it has never seen</h2>
+<p class="lede">An operations manual invented for this demo, so GPT-2 cannot
+have seen it. Read <b>once</b>, left to right: the first half builds the
+memory, the second half is measured with it — the papers' own dev/test split —
+and every token is scored <i>before</i> being written.</p>
+<div class="panel" id="manual-summary"></div>
+
+<h3 style="margin:26px 0 4px">Predictions the memory corrected</h3>
+<p class="lede">The frozen model had no way of knowing these. They are facts
+that exist only in that document.</p>
+<table id="fixes"><thead><tr>
+  <th class="mono">context</th><th class="mono">what came next</th>
+  <th class="mono">what frozen GPT-2 said</th>
+</tr></thead><tbody></tbody></table>
+
+<h2><span class="num">3</span>How much better, exactly</h2>
+<p class="lede">36k tokens of technical text the model had never seen, frozen
+GPT-2 124M, every system tuned identically on a held-out prefix,
+95&nbsp;% bootstrap confidence intervals.</p>
+<table>
+<thead><tr><th>system</th><th>perplexity</th><th>change</th>
+  <th>memory used</th></tr></thead>
+<tbody>
+<tr><td>frozen GPT-2</td><td>31.2</td><td>—</td><td>0</td></tr>
+<tr><td>+ RAG-style retrieve &amp; rescore</td><td>29.9</td><td>−4 %</td>
+  <td>corpus + index</td></tr>
+<tr><td>+ kNN-LM, <b>unbounded</b> store</td><td>23.6</td><td>−24 %</td>
+  <td>55 MB, grows forever</td></tr>
+<tr class="best"><td>+ this memory (fixed)</td><td>19.2</td><td>−38 %</td>
+  <td>4.2 MB, constant</td></tr>
+<tr class="best"><td>+ memory and fast weights</td><td>16.6</td><td>−47 %</td>
+  <td>7.4 MB, constant</td></tr>
+</tbody></table>
+<p class="note">Paired block bootstrap <b>P = 1.000</b> against the unbounded
+datastore, replicated over 5 random seeds and on a second model
+(Qwen3-0.6B).</p>
+
+<h2><span class="num">4</span>Where it does not work</h2>
+<div class="panel">
+<p><b>On long, low-repetition narrative, an unbounded kNN-LM still wins</b>
+(+0.048 vs +0.007 nats). This memory captures verbatim recurrence, not
+paraphrase. The boundary is measured and published rather than hidden.</p>
+<p>Three results that did <i>not</i> work are published too. Hidden states
+make poor Hebbian keys — their geometry is too entangled. Surprise gating
+helps the memory and <i>hurts</i> the fast-weight adapter, because the delta
+rule already carries its own error term. And calibrating the readout on your
+own stream loses to a proper tuning (+0.109 against +0.120 nats), because the
+calibration window is read by a colder memory than the one it will govern.</p>
+<p>A fixed matrix also saturates at long horizons, near 0.5 writes per
+parameter. Forgetting recovers ×2.3 of the gain and a 4× larger matrix
+recovers ×3.4 — both are one flag away.</p>
+</div>
+
+<h2><span class="num">5</span>Run it yourself</h2>
+<p class="lede">Every output on this page came out of the command line below,
+on a laptop CPU. It works with any causal language model — a Hugging Face id
+or a local folder — not just the two the papers measured.</p>
+<pre class="sh">pip install sillage
+
+sillage index notes.md              <span class="c"># instant: no model needed</span>
+sillage ask "what did the report say?"
+
+sillage read notes.md               <span class="c"># memorise it</span>
+sillage complete "The report said"  <span class="c"># generate WITH the memory</span>
+sillage status                      <span class="c"># what it knows, tier by tier</span>
+
+sillage read notes.md --model HuggingFaceTB/SmolLM2-135M   <span class="c"># any LM</span></pre>
+
+<footer>
+<p>Four preprints with permanent DOIs, the full reproduction pipeline, every
+number as committed JSON and 24 tests:
+<a href="https://github.com/riscoss63/sillage">github.com/riscoss63/sillage</a>
+· <a href="https://pypi.org/project/sillage/">pypi.org/project/sillage</a>
+· <a href="https://doi.org/10.5281/zenodo.22079016">doi:10.5281/zenodo.22079016</a></p>
+<p id="provenance"></p>
+</footer>
+
+</div>
+<script>
+const DATA = __DATA__;
+
+function split(prompt, text) {
+  // the prompt is echoed back by the tool; colour only what was generated
+  const rest = text.startsWith(prompt) ? text.slice(prompt.length) : text;
+  const head = text.startsWith(prompt) ? prompt : "";
+  return [head, rest];
+}
+function render(i) {
+  const c = DATA.completions[i];
+  for (const [id, key] of [["out-frozen", "frozen"], ["out-memory", "memory"]]) {
+    const [head, rest] = split(c.prompt, c.prompt + c[key]);
+    document.getElementById(id).innerHTML =
+      '<span class="p">' + esc(head) + '</span><span class="new">'
+      + esc(rest) + '</span>';
+  }
+  document.getElementById("note").textContent = c.same
+    ? "Identical here — the memory abstains when it is not confident, which "
+      + "is what keeps it from doing harm."
+    : "Everything after the prompt on the right comes from the paper the "
+      + "memory read, not from GPT-2's weights.";
+  document.querySelectorAll(".chip").forEach((b, k) =>
+    b.setAttribute("aria-pressed", k === i ? "true" : "false"));
+}
+function esc(s) {
+  return s.replace(/[&<>]/g, m => ({"&":"&amp;","<":"&lt;",">":"&gt;"}[m]));
+}
+const chips = document.getElementById("chips");
+DATA.completions.forEach((c, i) => {
+  const b = document.createElement("button");
+  b.className = "chip";
+  b.type = "button";
+  b.textContent = c.prompt.length > 52 ? c.prompt.slice(0, 52) + "…" : c.prompt;
+  b.title = c.prompt;
+  b.onclick = () => render(i);
+  chips.appendChild(b);
+});
+document.getElementById("read-what").textContent = DATA.read;
+document.getElementById("manual-summary").innerHTML = md(DATA.manual.summary);
+const body = document.querySelector("#fixes tbody");
+DATA.manual.fixes.forEach(row => {
+  const tr = document.createElement("tr");
+  row.forEach(cell => {
+    const td = document.createElement("td");
+    td.className = "mono";
+    td.textContent = cell;
+    tr.appendChild(td);
+  });
+  body.appendChild(tr);
+});
+document.getElementById("provenance").textContent =
+  "The completions and corrections on this page are recorded runs of "
+  + DATA.model + " on a laptop CPU, state " + DATA.state_mb
+  + " MB — reproduce them with the commands above.";
+function md(s) {
+  return "<p>" + esc(s)
+    .replace(/\\*\\*(.+?)\\*\\*/g, "<b>$1</b>")
+    .replace(/\\*(.+?)\\*/g, "<i>$1</i>")
+    .replace(/\\n\\n/g, "</p><p>") + "</p>";
+}
+render(0);
+</script>
+</body>
+</html>
+"""
+
+
+def main():
+    data = json.load(io.open(os.path.join(HERE, "data.json"),
+                             encoding="utf-8"))
+    page = PAGE.replace("__DATA__", json.dumps(data, ensure_ascii=False))
+    out = os.path.join(HERE, "index.html")
+    io.open(out, "w", encoding="utf-8", newline="\n").write(page)
+    print(f"wrote {out}: {len(page) / 1024:.0f} kB, "
+          f"{len(data['completions'])} completions, "
+          f"{len(data['manual']['fixes'])} corrections")
+
+
+if __name__ == "__main__":
+    main()

build_state.py

@@ -0,0 +1,37 @@
+"""Build the memory the Space ships with, so the first screen is instant.
+
+Reads paper 1 into a GPT-2 memory and leaves it in ./state (7.4 MB). Run it
+from a clone of the sillage repository, or point it at any document you want
+the Space to have already read.
+
+    python build_state.py [path/to/document.tex]
+"""
+
+import os
+import shutil
+import sys
+
+from sillage import Sillage
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+STATE = os.path.join(HERE, "state")
+DEFAULT = os.path.join(os.path.dirname(HERE), "llm_memory", "papers",
+                       "sillage", "sillage.tex")
+
+
+def main():
+    doc = sys.argv[1] if len(sys.argv) > 1 else DEFAULT
+    if not os.path.exists(doc):
+        raise SystemExit(f"no such document: {doc}")
+    shutil.rmtree(STATE, ignore_errors=True)
+    s = Sillage(model="gpt2", state=STATE)
+    rec = s.read(doc)[0]
+    size = sum(os.path.getsize(os.path.join(STATE, f))
+               for f in os.listdir(STATE)) / 1e6
+    print(f"{rec['file']}: {rec['tokens']} tokens, PPL {rec['ppl_frozen']} "
+          f"-> {rec['ppl_with_memory']}, state {size:.1f} MB in {STATE}")
+    print("The Space will load this memory at startup.")
+
+
+if __name__ == "__main__":
+    main()

capture.py

@@ -0,0 +1,69 @@
+"""Capture real outputs so the static page can show them without a server.
+
+A static Space has no backend: nothing can run GPT-2 for the visitor. What it
+can do is show what GPT-2 really produced, side by side, and say plainly that
+these are recorded runs anyone can reproduce with `pip install sillage`.
+
+Everything written to data.json here comes out of the tool, unedited.
+
+    python capture.py     ->  data.json
+"""
+
+import json
+import os
+import tempfile
+
+import sample
+from sillage import Sillage
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+STATE = os.path.join(HERE, "state")
+N = 20
+
+PROMPTS = [
+    "On a 36k-token stream of novel technical text, the memory",
+    "the memory improves GPT-2's test negative log-likelihood by",
+    "At 500k tokens the fixed matrix",
+    "surprise gating quadruples the gain of",
+    "Every write is gated by the model's own",
+    "a three-factor plasticity rule whose modulator is",
+]
+
+
+def main():
+    memory = Sillage(model="gpt2", state=STATE, quiet=False)
+    frozen = Sillage(model="gpt2", state=tempfile.mkdtemp(), quiet=True)
+    tok, model = memory.load_model()
+    frozen._tok, frozen._model = tok, model
+
+    out = {"model": "openai-community/gpt2",
+           "read": "papers/sillage/sillage.tex (8969 tokens)",
+           "state_mb": round(sum(
+               os.path.getsize(os.path.join(STATE, f))
+               for f in os.listdir(STATE)) / 1e6, 1),
+           "completions": []}
+    for p in PROMPTS:
+        a, b = frozen.complete(p, n=N), memory.complete(p, n=N)
+        out["completions"].append({"prompt": p, "frozen": a, "memory": b,
+                                   "same": a.strip() == b.strip()})
+        print(f"  {p[:44]:46s} {'=' if a == b else 'differs'}")
+
+    import app                      # reuse the demo's instrumented read loop
+    plot, summary, fixes, _, suggestion = app.read_and_report(sample.MANUAL)
+    out["manual"] = {"summary": summary, "fixes": fixes or [],
+                     "perplexity": plot, "suggestion": suggestion}
+    a, b = frozen.complete(suggestion, n=12), None
+    fresh = Sillage(model="gpt2", state=app.SESSIONS[-1], quiet=True)
+    fresh._tok, fresh._model = tok, model
+    b = fresh.complete(suggestion, n=12)
+    out["manual"]["completion"] = {"prompt": suggestion, "frozen": a,
+                                   "memory": b}
+
+    with open(os.path.join(HERE, "data.json"), "w", encoding="utf-8") as f:
+        json.dump(out, f, indent=1, ensure_ascii=False)
+    print(f"wrote data.json: {len(out['completions'])} completions, "
+          f"{len(out['manual']['fixes'])} corrections")
+
+
+if __name__ == "__main__":
+    main()

data.json

@@ -0,0 +1,119 @@
+{
+ "model": "openai-community/gpt2",
+ "read": "papers/sillage/sillage.tex (8969 tokens)",
+ "state_mb": 7.4,
+ "completions": [
+  {
+   "prompt": "On a 36k-token stream of novel technical text, the memory",
+   "frozen": " of the original text is not lost.\n\nThe original text is not lost.\n\nThe",
+   "memory": " improves GPT-2's test negative log-likelihood by +0.486 +/- 0.",
+   "same": false
+  },
+  {
+   "prompt": "the memory improves GPT-2's test negative log-likelihood by",
+   "frozen": " a factor of 1.5.\n\nThe results of the study are summarized in Table 1.",
+   "memory": " +0.486 +/- 0.005 nats (perplexity 31.2 -> 19.",
+   "same": false
+  },
+  {
+   "prompt": "At 500k tokens the fixed matrix",
+   "frozen": " is a bit more complex.\n\nThe first thing to note is that the fixed matrix is not",
+   "memory": " saturates (0.5 writes per parameter); leaky decay recovers 2.3k writes per",
+   "same": false
+  },
+  {
+   "prompt": "surprise gating quadruples the gain of",
+   "frozen": " the previous year.\n\nThe new study, published in the journal Nature Communications, found that the",
+   "memory": " uniform writes at equal plasticity budget (+0.203 vs +0.050 for the count variant",
+   "same": false
+  },
+  {
+   "prompt": "Every write is gated by the model's own",
+   "frozen": " rules.\n\nThe model's rules are:\n\nThe model must be able to write a",
+   "memory": " surprise\n\n\nThe model's surprise is the surprise of the augmented reality -- the surprise of the augmented",
+   "same": false
+  },
+  {
+   "prompt": "a three-factor plasticity rule whose modulator is",
+   "frozen": " a single-cell polyethylene (CPM) polyethylene (PPM) polyethyl",
+   "memory": " free at inference. We call the system . On the other hand, the system is free at inference",
+   "same": false
+  }
+ ],
+ "manual": {
+  "summary": "**Read once, left to right: 652 tokens built the memory, the next 653 were measured with it.** That split is the papers' own protocol — the numbers below are on text the memory had not seen when it scored them, and every token was scored *before* being written.\n\nPerplexity on that second half: 11.16 frozen → 9.84 with the rank-16 adapter → **9.80** with the memory on top — **12% lower**. The memory spoke on 36% of those positions and kept quiet on the rest; abstaining when it has nothing to say is what keeps it from doing harm.\n\nState on disk: **7.4 MB**, and it would still be 7.4 MB after a million tokens. A kNN-LM datastore over the same text would already hold about 8.0 MB, and would keep growing.",
+  "fixes": [
+   [
+    "wick valves or the ninth cabinet is reported immediately",
+    "','",
+    "' to'"
+   ],
+   [
+    "incoming duty officer. The morning routine begins at",
+    "' 05'",
+    "' 19'"
+   ],
+   [
+    " holds the amber cipher. The rotation changes every eleven",
+    "' days'",
+    "'\\n'"
+   ],
+   [
+    " incident, and is recorded in the gallery log",
+    "' beside'",
+    "'.'"
+   ],
+   [
+    ". Record the seal number in the gallery log.",
+    "'\\n'",
+    "' The'"
+   ],
+   [
+    "in manifold. Confirm the Kelbrin manifold",
+    "' reads'",
+    "' is'"
+   ],
+   [
+    ". Confirm the Kelbrin manifold reads between",
+    "' 4'",
+    "'\\n'"
+   ],
+   [
+    "4.6 bar. Open the Fenwick valves",
+    "' in'",
+    "'.'"
+   ],
+   [
+    "6 bar. Open the Fenwick valves in the",
+    "' order'",
+    "' west'"
+   ],
+   [
+    " bar. Open the Fenwick valves in the order",
+    "' three'",
+    "' in'"
+   ],
+   [
+    " one, two. Inspect the condensate",
+    "' trap'",
+    "'.'"
+   ],
+   [
+    " the seal number. Confirm the technician",
+    "' on'",
+    "\"'s\""
+   ]
+  ],
+  "perplexity": {
+   "frozen GPT-2": 11.161853381368227,
+   "+ fast weights": 9.837810800300112,
+   "+ memory": 9.795396904604075
+  },
+  "suggestion": "The Fenwick valves open three, one,",
+  "completion": {
+   "prompt": "The Fenwick valves open three, one,",
+   "frozen": " two, three, four, five, six, seven,",
+   "memory": " two. Inspect the\n\nfenwick valve and the"
+  }
+ }
+}

index.html

@@ -0,0 +1,260 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Sillage — a frozen LM that remembers what it reads</title>
+<style>
+:root {
+  --bg:#0d1117; --panel:#161b22; --line:#26303d; --ink:#c9d1d9;
+  --dim:#8b949e; --hl:#79c0ff; --warm:#ffc457; --ok:#3fb950;
+}
+* { box-sizing:border-box }
+body {
+  margin:0; background:var(--bg); color:var(--ink); line-height:1.6;
+  font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Roboto,
+              Helvetica,Arial,sans-serif;
+}
+.wrap { max-width:980px; margin:0 auto; padding:0 20px 80px }
+header { text-align:center; padding:56px 0 24px }
+h1 { font-size:2.6rem; margin:0 0 10px; letter-spacing:-.02em }
+.tag { font-size:1.15rem; color:var(--ink); max-width:640px; margin:0 auto }
+.etym { color:var(--dim); font-style:italic; font-size:.95rem;
+        max-width:560px; margin:18px auto 0 }
+.badges { margin:22px 0 8px; display:flex; gap:8px; justify-content:center;
+          flex-wrap:wrap }
+.badges img { height:20px }
+h2 { font-size:1.5rem; margin:52px 0 6px; letter-spacing:-.01em }
+h2 .num { color:var(--dim); font-weight:400; margin-right:8px }
+.lede { color:var(--dim); margin:0 0 20px }
+.gif { width:100%; border:1px solid var(--line); border-radius:10px;
+       display:block; margin:26px 0 }
+.chips { display:flex; gap:8px; flex-wrap:wrap; margin-bottom:16px }
+.chip {
+  background:var(--panel); border:1px solid var(--line); color:var(--ink);
+  border-radius:999px; padding:7px 14px; cursor:pointer; font-size:.9rem;
+  font-family:inherit; transition:border-color .15s, color .15s;
+}
+.chip:hover { border-color:var(--hl); color:var(--hl) }
+.chip[aria-pressed="true"] { border-color:var(--hl); color:var(--hl);
+                             background:#16273b }
+.cols { display:grid; grid-template-columns:1fr 1fr; gap:14px }
+@media (max-width:760px) { .cols { grid-template-columns:1fr } }
+.col { background:var(--panel); border:1px solid var(--line);
+       border-radius:10px; padding:14px 16px; min-height:170px }
+.col h3 { margin:0 0 10px; font-size:.85rem; text-transform:uppercase;
+          letter-spacing:.08em; color:var(--dim); font-weight:600 }
+.col.win h3 { color:var(--ok) }
+.out { font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+       font-size:.9rem; white-space:pre-wrap; word-break:break-word; margin:0 }
+.out .p { color:var(--dim) }
+.out .new { color:var(--warm) }
+.col.win .out .new { color:var(--hl) }
+.note { color:var(--dim); font-size:.9rem; margin:12px 0 0 }
+table { width:100%; border-collapse:collapse; margin:18px 0; font-size:.92rem }
+th,td { text-align:left; padding:9px 12px; border-bottom:1px solid var(--line) }
+th { color:var(--dim); font-weight:600; font-size:.82rem;
+     text-transform:uppercase; letter-spacing:.06em }
+td.mono,th.mono { font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+                  font-size:.86rem }
+tr.best td { color:var(--hl); font-weight:600 }
+.panel { background:var(--panel); border:1px solid var(--line);
+         border-radius:10px; padding:18px 20px; margin:18px 0 }
+.panel p:first-child { margin-top:0 }
+.panel p:last-child { margin-bottom:0 }
+pre.sh { background:var(--panel); border:1px solid var(--line);
+         border-radius:10px; padding:16px 18px; overflow-x:auto;
+         font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+         font-size:.9rem; color:var(--ink) }
+pre.sh .c { color:var(--dim) }
+a { color:var(--hl) }
+footer { color:var(--dim); font-size:.9rem; text-align:center;
+         margin-top:64px; border-top:1px solid var(--line); padding-top:24px }
+.kbd { font-family:"Cascadia Mono",Consolas,ui-monospace,monospace;
+       background:#1c2432; border:1px solid var(--line); border-radius:5px;
+       padding:1px 6px; font-size:.85em }
+</style>
+</head>
+<body>
+<div class="wrap">
+
+<header>
+  <h1>Sillage</h1>
+  <p class="tag">A frozen language model that <b>remembers what it reads</b> —
+  4.2&nbsp;MB, no gradients, no fine-tuning, no vector database.</p>
+  <p class="etym">sillage (n., French) — the trace left behind by something
+  that has passed: a ship's wake, a scent in a room. What a model keeps of
+  what it read.</p>
+  <div class="badges">
+    <a href="https://pypi.org/project/sillage/"><img alt="PyPI"
+      src="https://img.shields.io/pypi/v/sillage.svg"></a>
+    <a href="https://github.com/riscoss63/sillage"><img alt="GitHub"
+      src="https://img.shields.io/badge/code-GitHub-181717?logo=github"></a>
+    <a href="https://doi.org/10.5281/zenodo.22079016"><img alt="DOI"
+      src="https://zenodo.org/badge/DOI/10.5281/zenodo.22079016.svg"></a>
+    <a href="https://github.com/riscoss63/sillage/blob/main/LICENSE"><img
+      alt="MIT" src="https://img.shields.io/badge/License-MIT-blue.svg"></a>
+  </div>
+</header>
+
+<img class="gif" src="demo.gif" alt="Two sessions a day apart: the model
+  reads a draft on Monday and halves the perplexity of the next draft on
+  Tuesday, then completes a sentence it can only know from what it read.">
+
+<h2><span class="num">1</span>It has already read a paper</h2>
+<p class="lede">The memory below has read
+<b id="read-what"></b> — the paper describing this very mechanism, which
+GPT-2 has never seen. Same prompt, same greedy decoding, one column with the
+memory and one without. Pick a beginning:</p>
+
+<div class="chips" id="chips"></div>
+<div class="cols">
+  <div class="col"><h3>GPT-2, frozen</h3><p class="out" id="out-frozen"></p></div>
+  <div class="col win"><h3>GPT-2 + Sillage memory</h3>
+    <p class="out" id="out-memory"></p></div>
+</div>
+<p class="note" id="note"></p>
+
+<h2><span class="num">2</span>What it does to a document it has never seen</h2>
+<p class="lede">An operations manual invented for this demo, so GPT-2 cannot
+have seen it. Read <b>once</b>, left to right: the first half builds the
+memory, the second half is measured with it — the papers' own dev/test split —
+and every token is scored <i>before</i> being written.</p>
+<div class="panel" id="manual-summary"></div>
+
+<h3 style="margin:26px 0 4px">Predictions the memory corrected</h3>
+<p class="lede">The frozen model had no way of knowing these. They are facts
+that exist only in that document.</p>
+<table id="fixes"><thead><tr>
+  <th class="mono">context</th><th class="mono">what came next</th>
+  <th class="mono">what frozen GPT-2 said</th>
+</tr></thead><tbody></tbody></table>
+
+<h2><span class="num">3</span>How much better, exactly</h2>
+<p class="lede">36k tokens of technical text the model had never seen, frozen
+GPT-2 124M, every system tuned identically on a held-out prefix,
+95&nbsp;% bootstrap confidence intervals.</p>
+<table>
+<thead><tr><th>system</th><th>perplexity</th><th>change</th>
+  <th>memory used</th></tr></thead>
+<tbody>
+<tr><td>frozen GPT-2</td><td>31.2</td><td>—</td><td>0</td></tr>
+<tr><td>+ RAG-style retrieve &amp; rescore</td><td>29.9</td><td>−4 %</td>
+  <td>corpus + index</td></tr>
+<tr><td>+ kNN-LM, <b>unbounded</b> store</td><td>23.6</td><td>−24 %</td>
+  <td>55 MB, grows forever</td></tr>
+<tr class="best"><td>+ this memory (fixed)</td><td>19.2</td><td>−38 %</td>
+  <td>4.2 MB, constant</td></tr>
+<tr class="best"><td>+ memory and fast weights</td><td>16.6</td><td>−47 %</td>
+  <td>7.4 MB, constant</td></tr>
+</tbody></table>
+<p class="note">Paired block bootstrap <b>P = 1.000</b> against the unbounded
+datastore, replicated over 5 random seeds and on a second model
+(Qwen3-0.6B).</p>
+
+<h2><span class="num">4</span>Where it does not work</h2>
+<div class="panel">
+<p><b>On long, low-repetition narrative, an unbounded kNN-LM still wins</b>
+(+0.048 vs +0.007 nats). This memory captures verbatim recurrence, not
+paraphrase. The boundary is measured and published rather than hidden.</p>
+<p>Three results that did <i>not</i> work are published too. Hidden states
+make poor Hebbian keys — their geometry is too entangled. Surprise gating
+helps the memory and <i>hurts</i> the fast-weight adapter, because the delta
+rule already carries its own error term. And calibrating the readout on your
+own stream loses to a proper tuning (+0.109 against +0.120 nats), because the
+calibration window is read by a colder memory than the one it will govern.</p>
+<p>A fixed matrix also saturates at long horizons, near 0.5 writes per
+parameter. Forgetting recovers ×2.3 of the gain and a 4× larger matrix
+recovers ×3.4 — both are one flag away.</p>
+</div>
+
+<h2><span class="num">5</span>Run it yourself</h2>
+<p class="lede">Every output on this page came out of the command line below,
+on a laptop CPU. It works with any causal language model — a Hugging Face id
+or a local folder — not just the two the papers measured.</p>
+<pre class="sh">pip install sillage
+
+sillage index notes.md              <span class="c"># instant: no model needed</span>
+sillage ask "what did the report say?"
+
+sillage read notes.md               <span class="c"># memorise it</span>
+sillage complete "The report said"  <span class="c"># generate WITH the memory</span>
+sillage status                      <span class="c"># what it knows, tier by tier</span>
+
+sillage read notes.md --model HuggingFaceTB/SmolLM2-135M   <span class="c"># any LM</span></pre>
+
+<footer>
+<p>Four preprints with permanent DOIs, the full reproduction pipeline, every
+number as committed JSON and 24 tests:
+<a href="https://github.com/riscoss63/sillage">github.com/riscoss63/sillage</a>
+· <a href="https://pypi.org/project/sillage/">pypi.org/project/sillage</a>
+· <a href="https://doi.org/10.5281/zenodo.22079016">doi:10.5281/zenodo.22079016</a></p>
+<p id="provenance"></p>
+</footer>
+
+</div>
+<script>
+const DATA = {"model": "openai-community/gpt2", "read": "papers/sillage/sillage.tex (8969 tokens)", "state_mb": 7.4, "completions": [{"prompt": "On a 36k-token stream of novel technical text, the memory", "frozen": " of the original text is not lost.\n\nThe original text is not lost.\n\nThe", "memory": " improves GPT-2's test negative log-likelihood by +0.486 +/- 0.", "same": false}, {"prompt": "the memory improves GPT-2's test negative log-likelihood by", "frozen": " a factor of 1.5.\n\nThe results of the study are summarized in Table 1.", "memory": " +0.486 +/- 0.005 nats (perplexity 31.2 -> 19.", "same": false}, {"prompt": "At 500k tokens the fixed matrix", "frozen": " is a bit more complex.\n\nThe first thing to note is that the fixed matrix is not", "memory": " saturates (0.5 writes per parameter); leaky decay recovers 2.3k writes per", "same": false}, {"prompt": "surprise gating quadruples the gain of", "frozen": " the previous year.\n\nThe new study, published in the journal Nature Communications, found that the", "memory": " uniform writes at equal plasticity budget (+0.203 vs +0.050 for the count variant", "same": false}, {"prompt": "Every write is gated by the model's own", "frozen": " rules.\n\nThe model's rules are:\n\nThe model must be able to write a", "memory": " surprise\n\n\nThe model's surprise is the surprise of the augmented reality -- the surprise of the augmented", "same": false}, {"prompt": "a three-factor plasticity rule whose modulator is", "frozen": " a single-cell polyethylene (CPM) polyethylene (PPM) polyethyl", "memory": " free at inference. We call the system . On the other hand, the system is free at inference", "same": false}], "manual": {"summary": "**Read once, left to right: 652 tokens built the memory, the next 653 were measured with it.** That split is the papers' own protocol — the numbers below are on text the memory had not seen when it scored them, and every token was scored *before* being written.\n\nPerplexity on that second half: 11.16 frozen → 9.84 with the rank-16 adapter → **9.80** with the memory on top — **12% lower**. The memory spoke on 36% of those positions and kept quiet on the rest; abstaining when it has nothing to say is what keeps it from doing harm.\n\nState on disk: **7.4 MB**, and it would still be 7.4 MB after a million tokens. A kNN-LM datastore over the same text would already hold about 8.0 MB, and would keep growing.", "fixes": [["wick valves or the ninth cabinet is reported immediately", "','", "' to'"], ["incoming duty officer. The morning routine begins at", "' 05'", "' 19'"], [" holds the amber cipher. The rotation changes every eleven", "' days'", "'\\n'"], [" incident, and is recorded in the gallery log", "' beside'", "'.'"], [". Record the seal number in the gallery log.", "'\\n'", "' The'"], ["in manifold. Confirm the Kelbrin manifold", "' reads'", "' is'"], [". Confirm the Kelbrin manifold reads between", "' 4'", "'\\n'"], ["4.6 bar. Open the Fenwick valves", "' in'", "'.'"], ["6 bar. Open the Fenwick valves in the", "' order'", "' west'"], [" bar. Open the Fenwick valves in the order", "' three'", "' in'"], [" one, two. Inspect the condensate", "' trap'", "'.'"], [" the seal number. Confirm the technician", "' on'", "\"'s\""]], "perplexity": {"frozen GPT-2": 11.161853381368227, "+ fast weights": 9.837810800300112, "+ memory": 9.795396904604075}, "suggestion": "The Fenwick valves open three, one,", "completion": {"prompt": "The Fenwick valves open three, one,", "frozen": " two, three, four, five, six, seven,", "memory": " two. Inspect the\n\nfenwick valve and the"}}};
+
+function split(prompt, text) {
+  // the prompt is echoed back by the tool; colour only what was generated
+  const rest = text.startsWith(prompt) ? text.slice(prompt.length) : text;
+  const head = text.startsWith(prompt) ? prompt : "";
+  return [head, rest];
+}
+function render(i) {
+  const c = DATA.completions[i];
+  for (const [id, key] of [["out-frozen", "frozen"], ["out-memory", "memory"]]) {
+    const [head, rest] = split(c.prompt, c.prompt + c[key]);
+    document.getElementById(id).innerHTML =
+      '<span class="p">' + esc(head) + '</span><span class="new">'
+      + esc(rest) + '</span>';
+  }
+  document.getElementById("note").textContent = c.same
+    ? "Identical here — the memory abstains when it is not confident, which "
+      + "is what keeps it from doing harm."
+    : "Everything after the prompt on the right comes from the paper the "
+      + "memory read, not from GPT-2's weights.";
+  document.querySelectorAll(".chip").forEach((b, k) =>
+    b.setAttribute("aria-pressed", k === i ? "true" : "false"));
+}
+function esc(s) {
+  return s.replace(/[&<>]/g, m => ({"&":"&amp;","<":"&lt;",">":"&gt;"}[m]));
+}
+const chips = document.getElementById("chips");
+DATA.completions.forEach((c, i) => {
+  const b = document.createElement("button");
+  b.className = "chip";
+  b.type = "button";
+  b.textContent = c.prompt.length > 52 ? c.prompt.slice(0, 52) + "…" : c.prompt;
+  b.title = c.prompt;
+  b.onclick = () => render(i);
+  chips.appendChild(b);
+});
+document.getElementById("read-what").textContent = DATA.read;
+document.getElementById("manual-summary").innerHTML = md(DATA.manual.summary);
+const body = document.querySelector("#fixes tbody");
+DATA.manual.fixes.forEach(row => {
+  const tr = document.createElement("tr");
+  row.forEach(cell => {
+    const td = document.createElement("td");
+    td.className = "mono";
+    td.textContent = cell;
+    tr.appendChild(td);
+  });
+  body.appendChild(tr);
+});
+document.getElementById("provenance").textContent =
+  "The completions and corrections on this page are recorded runs of "
+  + DATA.model + " on a laptop CPU, state " + DATA.state_mb
+  + " MB — reproduce them with the commands above.";
+function md(s) {
+  return "<p>" + esc(s)
+    .replace(/\*\*(.+?)\*\*/g, "<b>$1</b>")
+    .replace(/\*(.+?)\*/g, "<i>$1</i>")
+    .replace(/\n\n/g, "</p><p>") + "</p>";
+}
+render(0);
+</script>
+</body>
+</html>

requirements.txt

@@ -0,0 +1,5 @@
+# The tool itself, straight from PyPI.
+sillage==1.0.1
+
+# ZeroGPU: provides the @spaces.GPU decorator. Harmless anywhere else.
+spaces

sample.py

@@ -0,0 +1,137 @@
+"""Texts the visitor can read into a fresh memory in one click.
+
+Two of them, chosen for what they demonstrate:
+
+  MANUAL   an internal-looking operations manual, invented for this demo, so
+           GPT-2 cannot possibly have seen it. Recurring names, recurring
+           procedures, the structure real internal documents actually have --
+           the regime where a fixed memory wins. Long enough (~1400 tokens)
+           that the abstention threshold has a distribution to work from.
+  PAPER    the opening of paper 1, i.e. genuinely novel technical prose.
+"""
+
+import os
+
+MANUAL = """Meridian Station — Standing Operations Manual, revision 12
+
+1. Purpose
+
+This manual governs every routine carried out at Meridian Station. It is
+issued by the Station Warden and supersedes revision 11. Where this manual
+conflicts with a local instruction, this manual prevails, and the Station
+Warden is notified within one shift.
+
+2. The morning routine
+
+The morning routine begins at 05:40 when the duty officer unseals the west
+gallery. The duty officer records the seal number in the gallery log, then
+verifies the pressure of the Kelbrin manifold. The Kelbrin manifold must read
+between 4.1 and 4.6 bar. If the Kelbrin manifold reads below 4.1 bar, the
+duty officer raises a yellow flag with the Station Warden and does not
+proceed to the second stage. If the Kelbrin manifold reads above 4.6 bar, the
+duty officer vents the auxiliary line and waits eleven minutes before reading
+it again.
+
+Once the Kelbrin manifold is within range, the duty officer opens the
+Fenwick valves in the order three, one, two. The order three, one, two is not
+negotiable: opening the Fenwick valves in any other order floods the
+condensate trap. The condensate trap is drained every second morning by the
+technician on rotation, who signs the gallery log beside the seal number.
+
+3. The evening routine
+
+The evening routine begins at 19:15 when the duty officer reseals the west
+gallery. Before resealing, the duty officer closes the Fenwick valves in the
+order two, one, three, which is the reverse of the morning order. The
+condensate trap is inspected, and the reading of the Kelbrin manifold is
+recorded a second time in the gallery log.
+
+If the Kelbrin manifold has drifted by more than 0.3 bar since morning, the
+duty officer files a drift note. A drift note names the shift, the two
+readings, and the technician on rotation. Three drift notes in one week
+trigger a manifold survey by the Station Warden.
+
+4. Personnel
+
+The Station Warden is Ilvane Corr. The Station Warden holds the only key to
+the ninth cabinet and is the sole authority for a manifold survey. In the
+absence of Ilvane Corr, authority passes to the senior technician, currently
+Bo Halloway, except for the ninth cabinet, which remains sealed.
+
+The technician on rotation changes every eleven days. The rotation list is
+posted in the west gallery and countersigned by Bo Halloway. A technician on
+rotation may not also serve as duty officer during the same shift.
+
+5. The ninth cabinet
+
+The ninth cabinet holds the amber cipher and the spare seals for the west
+gallery. Only Ilvane Corr may open the ninth cabinet. The amber cipher is
+required whenever the Kelbrin manifold is surveyed, and it is returned to the
+ninth cabinet before the evening routine begins. No copy of the amber cipher
+is kept anywhere at Meridian Station.
+
+6. Incidents
+
+An incident is any deviation from the morning routine, the evening routine,
+or the rotation list. Incidents are recorded in the gallery log beside the
+seal number, and reported to the Station Warden within one shift. An incident
+involving the Kelbrin manifold, the Fenwick valves or the ninth cabinet is
+reported immediately, not within one shift.
+
+If the west gallery cannot be sealed at 19:15, the duty officer declares a
+standing incident, remains at the west gallery, and notifies Ilvane Corr. A
+standing incident is closed only by the Station Warden, in person, with the
+amber cipher present.
+
+7. Handover
+
+At every handover the outgoing duty officer recites the fixed points to the
+incoming duty officer. The morning routine begins at 05:40. The evening
+routine begins at 19:15. The Kelbrin manifold reads between 4.1 and 4.6 bar.
+The Fenwick valves open three, one, two and close two, one, three. The
+Station Warden is Ilvane Corr. The senior technician is Bo Halloway. The
+ninth cabinet holds the amber cipher. The rotation changes every eleven days.
+
+The incoming duty officer repeats the fixed points back. A handover in which
+the fixed points are not recited and repeated back is an incident, and is
+recorded in the gallery log beside the seal number.
+
+8. Checklist, morning
+
+Unseal the west gallery at 05:40. Record the seal number in the gallery log.
+Read the Kelbrin manifold. Confirm the Kelbrin manifold reads between 4.1 and
+4.6 bar. Open the Fenwick valves in the order three, one, two. Inspect the
+condensate trap. Sign the gallery log beside the seal number. Confirm the
+technician on rotation against the rotation list posted in the west gallery.
+
+9. Checklist, evening
+
+Close the Fenwick valves in the order two, one, three. Read the Kelbrin
+manifold a second time. Record the reading in the gallery log. Compare it
+with the morning reading and file a drift note if the Kelbrin manifold has
+drifted by more than 0.3 bar. Confirm the amber cipher has been returned to
+the ninth cabinet. Reseal the west gallery at 19:15. Notify Ilvane Corr of
+any incident within one shift.
+
+10. Summary of the fixed points
+
+The morning routine begins at 05:40. The evening routine begins at 19:15. The
+Kelbrin manifold reads between 4.1 and 4.6 bar. The Fenwick valves open three,
+one, two and close two, one, three. The Station Warden is Ilvane Corr. The
+senior technician is Bo Halloway. The ninth cabinet holds the amber cipher.
+The rotation changes every eleven days. These fixed points are recited at
+every handover, and any deviation from them is an incident.
+"""
+
+_PAPER = os.path.join(os.path.dirname(os.path.dirname(
+    os.path.abspath(__file__))), "llm_memory", "papers", "sillage",
+    "sillage.tex")
+
+
+def paper_excerpt(limit=14000):
+    """The opening of paper 1 as plain text, if the repository is nearby."""
+    try:
+        from sillage.index import read_text
+        return read_text(_PAPER)[:limit]
+    except Exception:
+        return MANUAL

state/cold.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75e7a34ebc793b298e6bb671375edfff20a95404527152d0134e622d890c2484
+size 311125

state/index.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d90f319b3f5828a1cabd15886f1c143da0238b74f322a231be09a2a7a33b64e7
+size 101643

state/log.json

@@ -0,0 +1,12 @@
+{
+  "files": [
+    {
+      "file": "sillage.tex",
+      "tokens": 8969,
+      "date": "2026-08-25 03:02",
+      "ppl_frozen": 68.81,
+      "ppl_fastweights": 58.24,
+      "ppl_with_memory": 56.01
+    }
+  ]
+}

state/state.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2a11143fe1df59a9fa599336cff651ca3c6bd8fca37d362fb002c6edfc000a9
+size 6985621