{
 "model": "openai-community/gpt2",
 "read": "papers/sillage/sillage.tex (8969 tokens)",
 "state_mb": 7.4,
 "completions": [
  {
   "prompt": "On a 36k-token stream of novel technical text, the memory",
   "frozen": " of the original text is not lost.\n\nThe original text is not lost.\n\nThe",
   "memory": " improves GPT-2's test negative log-likelihood by +0.486 +/- 0.",
   "same": false
  },
  {
   "prompt": "the memory improves GPT-2's test negative log-likelihood by",
   "frozen": " a factor of 1.5.\n\nThe results of the study are summarized in Table 1.",
   "memory": " +0.486 +/- 0.005 nats (perplexity 31.2 -> 19.",
   "same": false
  },
  {
   "prompt": "At 500k tokens the fixed matrix",
   "frozen": " is a bit more complex.\n\nThe first thing to note is that the fixed matrix is not",
   "memory": " saturates (0.5 writes per parameter); leaky decay recovers 2.3k writes per",
   "same": false
  },
  {
   "prompt": "surprise gating quadruples the gain of",
   "frozen": " the previous year.\n\nThe new study, published in the journal Nature Communications, found that the",
   "memory": " uniform writes at equal plasticity budget (+0.203 vs +0.050 for the count variant",
   "same": false
  },
  {
   "prompt": "Every write is gated by the model's own",
   "frozen": " rules.\n\nThe model's rules are:\n\nThe model must be able to write a",
   "memory": " surprise\n\n\nThe model's surprise is the surprise of the augmented reality -- the surprise of the augmented",
   "same": false
  },
  {
   "prompt": "a three-factor plasticity rule whose modulator is",
   "frozen": " a single-cell polyethylene (CPM) polyethylene (PPM) polyethyl",
   "memory": " free at inference. We call the system . On the other hand, the system is free at inference",
   "same": false
  }
 ],
 "manual": {
  "summary": "**Read once, left to right: 652 tokens built the memory, the next 653 were measured with it.** That split is the papers' own protocol — the numbers below are on text the memory had not seen when it scored them, and every token was scored *before* being written.\n\nPerplexity on that second half: 11.16 frozen → 9.84 with the rank-16 adapter → **9.80** with the memory on top — **12% lower**. The memory spoke on 36% of those positions and kept quiet on the rest; abstaining when it has nothing to say is what keeps it from doing harm.\n\nState on disk: **7.4 MB**, and it would still be 7.4 MB after a million tokens. A kNN-LM datastore over the same text would already hold about 8.0 MB, and would keep growing.",
  "fixes": [
   [
    "wick valves or the ninth cabinet is reported immediately",
    "','",
    "' to'"
   ],
   [
    "incoming duty officer. The morning routine begins at",
    "' 05'",
    "' 19'"
   ],
   [
    " holds the amber cipher. The rotation changes every eleven",
    "' days'",
    "'\\n'"
   ],
   [
    " incident, and is recorded in the gallery log",
    "' beside'",
    "'.'"
   ],
   [
    ". Record the seal number in the gallery log.",
    "'\\n'",
    "' The'"
   ],
   [
    "in manifold. Confirm the Kelbrin manifold",
    "' reads'",
    "' is'"
   ],
   [
    ". Confirm the Kelbrin manifold reads between",
    "' 4'",
    "'\\n'"
   ],
   [
    "4.6 bar. Open the Fenwick valves",
    "' in'",
    "'.'"
   ],
   [
    "6 bar. Open the Fenwick valves in the",
    "' order'",
    "' west'"
   ],
   [
    " bar. Open the Fenwick valves in the order",
    "' three'",
    "' in'"
   ],
   [
    " one, two. Inspect the condensate",
    "' trap'",
    "'.'"
   ],
   [
    " the seal number. Confirm the technician",
    "' on'",
    "\"'s\""
   ]
  ],
  "perplexity": {
   "frozen GPT-2": 11.161853381368227,
   "+ fast weights": 9.837810800300112,
   "+ memory": 9.795396904604075
  },
  "suggestion": "The Fenwick valves open three, one,",
  "completion": {
   "prompt": "The Fenwick valves open three, one,",
   "frozen": " two, three, four, five, six, seven,",
   "memory": " two. Inspect the\n\nfenwick valve and the"
  }
 }
}