ATOM (atom-ai): code snapshot (100fae0 + batched-train), README with measured numbers, curated checkpoints
9f1076a verified | # Slow-mode readout + slow-memory loss — `--slow-mode-readout` (2026-09-29) | |
| English. Numbers. **No fluency claim: every arm, control included, still produces junk replies.** | |
| ## Why | |
| `--field-leak-spectrum 0.01 0.3 --field-leak-spectrum-gain sqrt --field-leak-spectrum-slow-frac 0.25` | |
| (docs/FIELD_LEAK_SPECTRUM.md) makes modes 5–11 slow. Those modes **do** hold the prompt: a linear probe | |
| reads which of 8 prompts was seen 16 bytes earlier at about 0.95, versus about 0.11 (chance 0.125) for the scalar | |
| control. However, the only readout was the shared `a_hat` fingerprint. There the slow energy was just noise | |
| (field-alone CE 1.93–2.35 vs 1.71–1.81; replies no more prompt-dependent). This lever gives the slow | |
| modes **their own readout path** and an **auxiliary objective that requires context memory**. | |
| ## Lever (opt-in, default OFF) | |
| ``` | |
| --slow-mode-readout # requires --field-leak-spectrum; off = exact current behaviour | |
| --slow-readout-rank R # default 64 | |
| --slow-readout-ahat {exclude,mix} # default exclude: slow modes are removed from the existing a_hat readout | |
| --slow-readout-lr LR # default = surface LR (used: 1e-3) | |
| --slow-memory-loss W # default 0 = no auxiliary loss | |
| --slow-memory-lags K [K ...] # default 16 32 64 | |
| --slow-memory-target {bag,byte} # default bag | |
| ``` | |
| Implementation (`src/atom_native.py`, class `SlowModeReadout`, and `AtomNativeModel`): | |
| * **Input.** The state of the slow modes (`slow_mode_indices(rates)` = modes with λ < LMAX, i.e. [5..11]) | |
| (7 modes × d 32 = 224 floats) is flattened, divided by its RMS, then **standardized per dimension** with running buffers (`in_mean`, | |
| `in_var`, `n_seen`; cumulative average for the first 2000 training ticks, then EMA; var floor 1e-4). Stats update | |
| only on training ticks where grad is enabled, never in eval or chat. Buffers are saved in the checkpoint. | |
| * **Head.** `z = tanh(trunk(x))` (rank 64). `logits += to_logits(z)`, with `to_logits` (rank→256, no bias) | |
| **zero-initialised**, so the logits at step 0 are identical to the loaded checkpoint (with `mix`; with `exclude` | |
| the only change is removing the slow modes from `a_hat`). The term remains active under `last_atom_force_off`, | |
| because it is a field term. | |
| * **Memory heads.** One `Linear(rank→256)` per lag K, used only by the auxiliary loss. | |
| * **Gradient routing.** Next-byte CE trains only `to_logits`; `z` is computed under no_grad for the CE path. | |
| The memory loss trains trunk + memory heads; the slow state is detached, so the auxiliary loss never touches | |
| the field dynamics or any control parameter (tested). | |
| * **Optimizer.** The head is a separate Adam param group, clipped separately (norm 1), so it never rescales | |
| the control's updates. With the head off the clip path is the old one. | |
| * **Init RNG.** The head is built under `torch.random.fork_rng` with its own generator (seed 20260929), | |
| so enabling it does not shift the global RNG stream (tested). | |
| * **Checkpoint.** Top-level keys `slow_readout` and `slow_readout_config` are written only when the head | |
| exists and restored after the spectrum. Absent keys → off. The CLI wins after `--resume`, same as | |
| `--field-leak`/`--field-leak-spectrum`. | |
| * **Cost.** A 224→64→256 MLP per tick (+ 64→256 per memory lag when W > 0). Throughput is unchanged within noise (below). | |
| ### Design choices, and the failures that forced them | |
| 1. **`exclude` rather than `mix` (frozen check, 4996k weights, fresh zero-output head).** `mix` keeps the slow | |
| modes inside `a_hat`. That makes `a_hat` prompt-aware (memory 0.48 at 16), but it raises the a_hat linear CE ceiling | |
| (1.756 vs 1.678 control) and the frozen field-alone shock (2.254). `exclude` keeps the a_hat ceiling at the | |
| control's level (1.691) and a smaller shock (2.182). The prompt memory then lives in the dedicated path | |
| (raw slow features 0.947/0.734/0.319 at 16/32/64; random fresh `z` 0.86/0.58/0.24). | |
| 2. **Stop-gradient from CE into the trunk.** In sweep v1, CE also trained the trunk. The trunk saturated | |
| (z constant) and the head degenerated into an output bias. The memory loss stayed at the prior (~3.27) and val got | |
| worse (1.085–1.107 vs A 1.032, in-run metric). | |
| 3. **Running standardization.** In v2 (CE blocked, no standardization), at lr 1e-3 z still saturated | |
| (std 0.0004), and the memory loss stayed at the prior at every lr. Cause: 92% of the slow-feature energy is a | |
| static mean. An offline online-learning sim showed that standardized inputs reach bag CE 2.76/3.01/3.15 | |
| (K = 16/32/64) against a prior of about 3.21. That led to v3. | |
| 4. **Bag-of-bytes target over byte@K.** Frozen linear decodability from slow features: byte@16 2.53 vs unigram 3.22, | |
| bag16 2.57 vs prior 3.21 (target entropy 1.88), bag32 2.90, bag64 3.13; byte@64 overfits. The bag of lags | |
| [K/2, K) asks "what was said in that window", which is the information a reply needs, and it is smoother than | |
| one exact byte. `byte` is kept as an option. | |
| 5. **W is almost an on/off switch.** CE never reaches the trunk, and Adam normalises per parameter, so the trunk's | |
| update size barely depends on W. Only whether W > 0 matters, so no finer W sweep was run. | |
| ## Frozen check (4996k fieldsat weights, eval only) | |
| | variant | a_hat lin CE | ring normal | ring f-alone | mem a_hat 16/32/64 | mem raw slow 16/32/64 | JS reply0 normal / f-alone | | |
| |---|--:|--:|--:|--|--|--| | |
| | control scalar 0.3 | 1.678 | 0.497 | 1.796 | .106/.113/.087 | .109/.116/.100 (modes 5–11) | .0035/.0040 | | |
| | spectrum sqrt, mix | 1.756 | 0.509 | 2.254 | .475/.219/.125 | .947/.734/.319 | .0120/.0644 | | |
| | **spectrum sqrt, exclude** | 1.691 | 0.522 | 2.182 | .109/.113/.081 | .947/.734/.319 | .0039/.0041 | | |
| | spectrum none, mix | 1.918 | 0.482 | 3.462 | .703/.422/.163 | .944/.716/.287 | — | | |
| | spectrum none, exclude | 1.691 | 0.521 | 2.185 | — | .944/.716/.287 | — | | |
| Linear next-byte CE from `a_hat + slow` features is 1.485 in the spectrum run. Caveat: the control's own modes 5–11 plus | |
| `a_hat` already give 1.42. Part of any next-byte gain from a second readout is therefore **readout capacity, not memory**. | |
| JSONs: `docs/artifacts/slow_mode_readout/frozen/`. | |
| ## Sweep (+8k from 4996k, same recipe/seed/skip, 2 threads each; `sweep/`, `sweep_table.md`) | |
| v3 (standardized, CE blocked from the trunk, exclude, sqrt): | |
| | run | val ring | f-alone ring | val 2048 | f-alone 2048 | mem z 16/32/64 | JS reply0 normal | seeded distinct (first word) uniq/27 | | |
| |---|--:|--:|--:|--:|--|--:|--| | |
| | W 0 (readout only), lr 1e-3 | 0.401 | 1.687 | 1.233 | 2.107 | .83/.50/.22 | .030 | 1.56 (1.00) 10 | | |
| | W 0.1 bag, lr 1e-3 | 0.474 | 1.959 | 1.270 | 2.266 | .59/.34/.16 | .017 | 1.78 (1.00) 13 | | |
| | W 0.1 bag, lr 3e-4 | 0.475 | 1.991 | 1.254 | 2.224 | .74/.45/.20 | .007 | 1.11 (1.00) 4 | | |
| | W 0.1 byte (lags 8/16/32), lr 1e-3 | 0.479 | 1.959 | 1.295 | 2.313 | .56/.36/.17 | .015 | 1.00 (1.00) 4 | | |
| | A (control) | 0.478 | 1.799 | 1.242 | 2.082 | — | .004 | 1.00 (1.00) 2 | | |
| Already after 8k, the memory auxiliary made `z` **less** prompt-discriminative (0.59 vs 0.83 at 16) and worsened field-alone CE. | |
| The bag loss learns the corpus's **generic** recent-byte statistics (training memory loss only falls from 3.12 to 2.89, | |
| prior about 3.2) rather than what distinguishes one context from another. Therefore the A/B ran the lever as specified (B, W 0.1 bag, | |
| lr 1e-3) **plus** the readout-only arm B0 (W 0) as an ablation. | |
| ## A/B from 4996k (+8k chunks, seed 20260913, stream-skip = START − 496000, identical recipe) | |
| * A = scalar control `--field-leak 0.3`. Rows reused from the leakspec experiment (same seed, skip, recipe | |
| and start checkpoint). Its 2048-val, a_hat-probe and output-dependence measurements were re-run here on its checkpoints. | |
| * B = A's recipe with `--field-leak 0.3` replaced by `--field-leak-spectrum 0.01 0.3 --field-leak-spectrum-gain sqrt | |
| --field-leak-spectrum-slow-frac 0.25 --slow-mode-readout --slow-readout-ahat exclude --slow-readout-lr 1e-3 | |
| --slow-memory-loss 0.1 --slow-memory-target bag --slow-memory-lags 16 32 64`. | |
| Checkpoints: `checkpoints/byte_tick_slowread_B/*_slowread.pt`. | |
| * B0 = same as B without `--slow-memory-loss` (readout only). Checkpoints: `checkpoints/byte_tick_slowread_B0/*_slowread0.pt`. | |
| * The first chunk (5004k) of B and B0 is the corresponding v3 sweep run (same recipe = identical chunk). | |
| * **B was stopped early after the 5028k measurement** (+32k): val 2048 +0.051 and field-alone 2048 +0.135 vs A, with | |
| seeded distinct below A. It was "clearly worse" under the task's stop rule. The 5028k→5036k chunk was killed mid-training; | |
| no checkpoint was written. B0 ran to 5036k. | |
| * Start (4996k, all arms): ring 0.497 / 1.796, 2048 1.247 / 2.103, seeded distinct 1.11 (1.00) 3, chats `Oui,` / `Peut--` / `Peut--`. | |
| Columns: val = teacher-forced byte CE (ring = fixed 256-window val ring at 4876k; 2048 = 2048-transition val windows); | |
| f-alone = same with `last_atom_force_off` (field path only; the slow readout stays on because it is a field term); | |
| a_hat lin = linear next-byte probe CE on `a_hat`; mem X k = 8-prompt identity probe accuracy k bytes after the prompt, | |
| on features X (slow = raw slow modes, z = readout trunk); JS reply0 = mean pairwise Jensen–Shannon divergence of the | |
| next-byte distribution at the reply start across 8 prompts; argmax distinct = distinct argmax first bytes across those prompts; | |
| seeded = seeds 1–3 × T 0.8/0.6/1.0 × 3 prompts, distinct replies per (seed, T) triple (max 3), first-word distinct, unique replies out of 27; | |
| cap = fraction of ticks at the field RMS cap (train / episode). | |
| ### Per-chunk table | |
| | step | arm | val ring | f-alone ring | val 2048 | f-alone 2048 | a_hat lin | mem slow 16/32/64 | mem z 16/32/64 | mem a_hat 16/32/64 | JS reply0 normal / f-alone | argmax distinct /8 | seeded distinct (first word) uniq/27 | cap | mem loss | steps/s | | |
| |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | |
| | 5004k | A | 0.478 | 1.799 | 1.242 | 2.082 | 1.673 | 0.109/0.106/0.103 | — | 0.087/0.116/0.091 | 0.00400 / 0.00580 | 1 | 1.00 (1.00) 2 | 0.000/0.000 | — | 18.6 | | |
| | 5004k | B | 0.474 | 1.959 | 1.270 | 2.266 | 1.689 | 0.938/0.775/0.287 | 0.588/0.344/0.163 | 0.103/0.113/0.100 | 0.01699 / 0.02725 | 2 | 1.78 (1.00) 13 | 0.000/0.000 | 3.1212 | 73.8 | | |
| | 5004k | B0 | 0.401 | 1.687 | 1.233 | 2.107 | 1.688 | 0.938/0.778/0.306 | 0.828/0.503/0.222 | 0.116/0.109/0.087 | 0.03029 / 0.03569 | 2 | 1.56 (1.00) 10 | 0.000/0.000 | — | 78.5 | | |
| | 5012k | A | 0.523 | 1.796 | 1.263 | 2.062 | 1.670 | 0.122/0.125/0.119 | — | 0.087/0.113/0.094 | 0.00458 / 0.00829 | 1 | 1.56 (1.56) 8 | 0.000/0.000 | — | 80.6 | | |
| | 5012k | B | 0.527 | 1.956 | 1.267 | 2.182 | 1.686 | 0.934/0.753/0.312 | 0.684/0.384/0.150 | 0.106/0.097/0.094 | 0.01855 / 0.03177 | 1 | 1.56 (1.00) 9 | 0.000/0.000 | 2.9716 | 77.2 | | |
| | 5012k | B0 | 0.476 | 1.731 | 1.227 | 2.018 | 1.685 | 0.938/0.759/0.303 | 0.847/0.525/0.206 | 0.116/0.103/0.087 | 0.01885 / 0.05105 | 2 | 2.89 (2.22) 18 | 0.000/0.000 | — | 81.2 | | |
| | 5020k | A | 0.517 | 1.725 | 1.278 | 2.056 | 1.680 | 0.106/0.125/0.113 | — | 0.100/0.106/0.091 | 0.00358 / 0.00687 | 1 | 2.00 (1.78) 10 | 0.000/0.000 | — | 78.7 | | |
| | 5020k | B | 0.463 | 1.777 | 1.272 | 2.209 | 1.689 | 0.953/0.775/0.319 | 0.625/0.350/0.153 | 0.106/0.106/0.091 | 0.02319 / 0.04516 | 1 | 2.44 (2.00) 19 | 0.000/0.000 | 2.9586 | 77.6 | | |
| | 5020k | B0 | 0.438 | 1.619 | 1.241 | 2.050 | 1.690 | 0.947/0.775/0.303 | 0.847/0.497/0.169 | 0.106/0.097/0.091 | 0.02857 / 0.05935 | 3 | 2.56 (2.11) 16 | 0.000/0.000 | — | 83.2 | | |
| | 5028k | A | 0.548 | 1.708 | 1.279 | 2.015 | 1.670 | 0.125/0.109/0.122 | — | 0.103/0.113/0.087 | 0.00441 / 0.00516 | 2 | 2.11 (2.11) 6 | 0.000/0.000 | — | 77.8 | | |
| | 5028k | B | 0.527 | 1.770 | 1.331 | 2.150 | 1.674 | 0.953/0.781/0.341 | 0.725/0.422/0.153 | 0.106/0.091/0.097 | 0.05833 / 0.05970 | 2 | 1.78 (1.22) 8 | 0.000/0.000 | 2.8881 | 74.7 | | |
| | 5028k | B0 | 0.519 | 1.732 | 1.288 | 2.093 | 1.680 | 0.956/0.787/0.338 | 0.859/0.522/0.191 | 0.106/0.097/0.100 | 0.04875 / 0.06295 | 3 | 2.56 (2.22) 10 | 0.000/0.000 | — | 81.6 | | |
| | 5036k | A | 0.565 | 1.812 | 1.310 | 2.072 | 1.674 | 0.131/0.097/0.116 | — | 0.109/0.106/0.094 | 0.00401 / 0.00664 | 2 | 2.00 (2.00) 7 | 0.000/0.000 | — | 79.9 | | |
| | 5036k | B0 | 0.566 | 1.702 | 1.350 | 2.121 | 1.678 | 0.950/0.784/0.344 | 0.838/0.512/0.200 | 0.100/0.094/0.100 | 0.03652 / 0.06724 | 3 | 2.89 (2.33) 24 | 0.000/0.000 | — | 81.2 | | |
| | step | arm | Bonjour, comment ça va ? | Qui es-tu ? | Il était une fois | | |
| |---|---|---|---|---| | |
| | 5004k | A | `Peut--` | `Peut--` | `Peut--` | | |
| | 5004k | B | `Peut--oonne reut--ourreeee` | `Peut--oonne ruurrecomprence` | `Peut--on reut--ourreeeux` | | |
| | 5004k | B0 | `Peut--être aparireee` | `Peut--être ruurra.` | `D'''''''` | | |
| | 5012k | A | `Oui,` | `Oui,` | `Oui,` | | |
| | 5012k | B | `Peut--en` | `Peut--en` | `Peut--en` | | |
| | 5012k | B0 | `Je ruuirra.iisaant:::::::::::::O` | `Oui,, rarirecommence` | `Oui,, rarirecommence` | | |
| | 5020k | A | `Oui,` | `Je` | `Oui,` | | |
| | 5020k | B | `Je end` | `Peut--on ??issoire` | `Bonne ??issoire` | | |
| | 5020k | B0 | `Oui,` | `Bonne` | `D'''''''` | | |
| | 5028k | A | `Peut--` | `D'''''''` | `Peut--` | | |
| | 5028k | B | `Oui,,` | `Oui,,` | `Oui,,` | | |
| | 5028k | B0 | `Peut--` | `Bonne` | `Peut--être` | | |
| | 5036k | A | `Peut--` | `D'''''''` | `D'''''''` | | |
| | 5036k | B0 | `Je` | `Peut--uu'on i?.` | `Je` | | |
| ### Deltas vs A at the same step (`delta_table.md`) | |
| | arm | step | d val ring | d f-alone ring | d val 2048 | d f-alone 2048 | JS normal (x A) | d seeded distinct | d uniq/27 | | |
| |---|---|---|---|---|---|---|---|---| | |
| | B | 5004k | -0.004 | +0.160 | +0.029 | +0.184 | 4.2x | +0.78 | +11.00 | | |
| | B | 5012k | +0.004 | +0.160 | +0.004 | +0.121 | 4.1x | +0.00 | +1.00 | | |
| | B | 5020k | -0.055 | +0.053 | -0.007 | +0.153 | 6.5x | +0.44 | +9.00 | | |
| | B | 5028k | -0.021 | +0.062 | +0.051 | +0.135 | 13.2x | -0.33 | +2.00 | | |
| | **B mean** | 4 chunks | -0.019 | +0.109 | +0.019 | +0.148 | 7.0x | +0.22 | +5.75 | | |
| | B0 | 5004k | -0.076 | -0.112 | -0.009 | +0.026 | 7.6x | +0.56 | +8.00 | | |
| | B0 | 5012k | -0.047 | -0.065 | -0.037 | -0.043 | 4.1x | +1.33 | +10.00 | | |
| | B0 | 5020k | -0.079 | -0.106 | -0.038 | -0.006 | 8.0x | +0.56 | +6.00 | | |
| | B0 | 5028k | -0.029 | +0.024 | +0.008 | +0.079 | 11.1x | +0.45 | +4.00 | | |
| | B0 | 5036k | +0.001 | -0.110 | +0.040 | +0.049 | 9.1x | +0.89 | +17.00 | | |
| | **B0 mean** | 5 chunks | -0.046 | -0.074 | -0.007 | +0.021 | 8.0x | +0.76 | +9.00 | | |
| Chunk-to-chunk noise in A itself: val 2048 moves 1.242 → 1.263 → 1.279 → 1.280 → 1.311 and field-alone 2048 moves 2.015–2.082. | |
| Deltas around ±0.03 are therefore within noise. | |
| ### Seeded examples (T 0.8, seeds 1–3; `arm_*/chat_seeds_5036000.txt`, trailing spaces trimmed) | |
| A @5036k (27 samples, 7 unique): | |
| ``` | |
| s1 Bonjour → 'Je' Qui → 'Oui,' Il était → 'Je' | |
| s2 Bonjour → 'Je' Qui → 'Oui,' Il était → 'Je' | |
| s3 Bonjour → 'Je' Qui → "D'''''''" Il était → 'Je' | |
| ``` | |
| B0 @5036k (27 samples, 24 unique): | |
| ``` | |
| s1 Bonjour → 'Oui, ?ireux quoirec' Qui → 'Oui, ?irecommation, un ruuissi' Il était → 'Peut--on ?ireux un quuissi' | |
| s2 Bonjour → 'Oui,, un' Qui → 'Oui,, un un' Il était → "Peuu''''''''''''''''''utec" | |
| s3 Bonjour → 'Oui,' Qui → 'On un' Il était → 'Parle un' | |
| ``` | |
| B @5020k: `Oui,, … ??issistant:::::::::` for all three prompts at most seeds. The extra "distinct" replies are mostly | |
| different amounts of role-token junk (`??issistant::::`, `Utilisateur::::`), not different answers. | |
| ## Throughput | |
| Both arms ran in parallel, 3 threads each (the measurement tools ran on 1 thread between chunks): B 74–78 steps/s, B0 78–83 steps/s, | |
| vs A 78–80 steps/s alone at 8 threads. The head costs nothing measurable. One 8k chunk ≈ 470 s of training + ≈ 9 min of measurements. | |
| The field never touched the RMS cap in any arm (cap fraction 0.000). | |
| ## Tests | |
| `test/test_slow_mode_readout.py` (15 tests, all pass): | |
| * Off is bit-identical to commit 2b44198 (model trace with no leak, with `--field-leak 0.3`, and with the spectrum; trainer run vs a | |
| `git archive` of the legacy trainer). | |
| * Enable-then-disable equals never enabled. The lever requires the spectrum. Slow indices are [5..11]. The global RNG is untouched. | |
| * Zero-init: `mix` gives identical logits at step 0; `exclude` differs only through the a_hat masking. | |
| * Gradient: the memory loss reaches trunk + memory heads; CE reaches `to_logits` only (never the trunk); the memory loss never reaches non-head params. | |
| * Running stats move only on training ticks. The history buffer and bag/byte targets are correct. | |
| * Checkpoint roundtrip (params + buffers + config); an off checkpoint loads as off. CLI default off, spectrum required, | |
| CLI wins after `--resume` in both directions (a trained head is kept when re-enabled). | |
| Full suite (`pytest_full.txt`): **137 passed, 1 skipped, 3 failed**. The 3 failures are the known pre-existing ones | |
| (`test_chat_merge_off` payload_copy; `test_speech_lock::test_dialogue_wrap`, `::test_speech_ok_rejects_attractor_soup`). | |
| ## Caveats | |
| * **Val is not held out.** The corpus is about 1 MB and was looped about 4.5 times by step 4.5M, so the val windows have been | |
| seen in training. CE gains can be partly memorization. | |
| * **Capacity vs memory.** A second readout adds capacity. Even the control's modes 5–11 give a linear next-byte gain. | |
| B0's early ring gains could be capacity and need not come from prompt memory. | |
| * **Prompt dependence ≠ quality.** JS divergence and distinct-reply counts go up, but the replies stay junk and | |
| often echo the dialogue frame (role tokens, colons, runs of spaces). | |
| * One seed, one data order, 5 chunks: the verdict below is about direction and size, not significance. | |
| ## Verdict | |
| * **The lever as specified (B: slow readout + bag memory loss W 0.1) fails the success criterion.** It raises prompt | |
| dependence (JS 4–13× A, a few more unique seeded replies), but field-alone CE is worse at every chunk (+0.12 to +0.18 | |
| on the 2048 val, well beyond noise), and at +32k normal val is worse too (+0.05). The auxiliary loss makes the readout | |
| **less** prompt-discriminative (z memory 0.59–0.73 vs 0.83–0.86 in B0). It learns generic recent-byte statistics and | |
| stays close to the prior. Stopped early at 5028k. | |
| * **The readout-only ablation (B0, W 0) is the better arm but not a clear win.** Prompt dependence goes up consistently | |
| (JS 4–11× A; seeded unique 10–24/27 vs A 2–10; seeded distinct +0.76 per triple on average; argmax distinct 2–3 vs 1–2), | |
| and the mean 2048 val / field-alone deltas are −0.007 / +0.021 (within noise). But the trend over the chunks is | |
| unfavourable: val 2048 −0.04 at 5012k/5020k becomes +0.04 at 5036k, and field-alone +0.05–0.08 in the last two chunks. | |
| The extra prompt dependence is visible only as different junk. **Nothing here is fluent, and no arm is closer to | |
| coherent French than A.** | |
| * Net: giving the slow modes a readout does make the output depend on context held by the field (the first lever in | |
| this project to move that metric by an order of magnitude). A "remember recent bytes" auxiliary is the wrong signal. | |
| The memory is there and readable; what is missing is a training signal that rewards using it **for the reply**. | |
| ## Next lever (one) | |
| **Train the slow readout on dialogue continuations, not generic stream bytes:** an opt-in auxiliary/weighted CE on the | |
| reply bytes of `Utilisateur: … / Assistant: …` pairs (or corpus sentence pairs wrapped in that frame), with `z` fed by the | |
| slow modes (B0 settings, W 0). Plain next-byte CE on the stream almost never needs information older than a few bytes, so | |
| the readout learns only a little from it. A reply-conditioned objective is the smallest change that rewards carrying the prompt across the 12-byte | |
| `\nAssistant: ` gap. Run it with a capacity control (same readout on modes 5–11 of the scalar arm) so memory and capacity | |
| can be told apart. | |
| ## Reproduce | |
| * Arms: `scripts/train_slowread_arm.sh` (env `ARM OUT SUF LEAK_FLAGS TSV START_CKPT START_STEP END_TARGET THREADS`). | |
| * Sweep: `docs/artifacts/slow_mode_readout/sweep.sh`. Measurements of one checkpoint: `measure_ckpt.sh CK OUTDIR TAG`. | |
| * Tables: `ab_table.py`, `delta_table.py`, `sweep_table.py` (all read `slowread_progress.tsv` / `sweep/`). | |
| * Frozen check: `tools/probe_leak_spectrum.py --slow-probes --slow-readout {exclude,mix}` on the 4996k fieldsat checkpoint. | |