docs/SLOW_MODE_READOUT.md

Slow-mode readout + slow-memory loss — --slow-mode-readout (2026-09-29)

English. Numbers. No fluency claim: every arm, control included, still produces junk replies.

Why

--field-leak-spectrum 0.01 0.3 --field-leak-spectrum-gain sqrt --field-leak-spectrum-slow-frac 0.25 (docs/FIELD_LEAK_SPECTRUM.md) makes modes 5–11 slow. Those modes do hold the prompt: a linear probe reads which of 8 prompts was seen 16 bytes earlier at about 0.95, versus about 0.11 (chance 0.125) for the scalar control. However, the only readout was the shared a_hat fingerprint. There the slow energy was just noise (field-alone CE 1.93–2.35 vs 1.71–1.81; replies no more prompt-dependent). This lever gives the slow modes their own readout path and an auxiliary objective that requires context memory.

Lever (opt-in, default OFF)

--slow-mode-readout                    # requires --field-leak-spectrum; off = exact current behaviour
--slow-readout-rank R                  # default 64
--slow-readout-ahat {exclude,mix}      # default exclude: slow modes are removed from the existing a_hat readout
--slow-readout-lr LR                   # default = surface LR (used: 1e-3)
--slow-memory-loss W                   # default 0 = no auxiliary loss
--slow-memory-lags K [K ...]           # default 16 32 64
--slow-memory-target {bag,byte}        # default bag

Implementation (src/atom_native.py, class SlowModeReadout, and AtomNativeModel):

• Input. The state of the slow modes (slow_mode_indices(rates) = modes with λ < LMAX, i.e. [5..11]) (7 modes × d 32 = 224 floats) is flattened, divided by its RMS, then standardized per dimension with running buffers (in_mean, in_var, n_seen; cumulative average for the first 2000 training ticks, then EMA; var floor 1e-4). Stats update only on training ticks where grad is enabled, never in eval or chat. Buffers are saved in the checkpoint.
• Head. z = tanh(trunk(x)) (rank 64). logits += to_logits(z), with to_logits (rank→256, no bias) zero-initialised, so the logits at step 0 are identical to the loaded checkpoint (with mix; with exclude the only change is removing the slow modes from a_hat). The term remains active under last_atom_force_off, because it is a field term.
• Memory heads. One Linear(rank→256) per lag K, used only by the auxiliary loss.
• Gradient routing. Next-byte CE trains only to_logits; z is computed under no_grad for the CE path. The memory loss trains trunk + memory heads; the slow state is detached, so the auxiliary loss never touches the field dynamics or any control parameter (tested).
• Optimizer. The head is a separate Adam param group, clipped separately (norm 1), so it never rescales the control's updates. With the head off the clip path is the old one.
• Init RNG. The head is built under torch.random.fork_rng with its own generator (seed 20260929), so enabling it does not shift the global RNG stream (tested).
• Checkpoint. Top-level keys slow_readout and slow_readout_config are written only when the head exists and restored after the spectrum. Absent keys → off. The CLI wins after --resume, same as --field-leak/--field-leak-spectrum.
• Cost. A 224→64→256 MLP per tick (+ 64→256 per memory lag when W > 0). Throughput is unchanged within noise (below).

Design choices, and the failures that forced them

1. exclude rather than mix (frozen check, 4996k weights, fresh zero-output head). mix keeps the slow modes inside a_hat. That makes a_hat prompt-aware (memory 0.48 at 16), but it raises the a_hat linear CE ceiling (1.756 vs 1.678 control) and the frozen field-alone shock (2.254). exclude keeps the a_hat ceiling at the control's level (1.691) and a smaller shock (2.182). The prompt memory then lives in the dedicated path (raw slow features 0.947/0.734/0.319 at 16/32/64; random fresh z 0.86/0.58/0.24).
2. Stop-gradient from CE into the trunk. In sweep v1, CE also trained the trunk. The trunk saturated (z constant) and the head degenerated into an output bias. The memory loss stayed at the prior (~3.27) and val got worse (1.085–1.107 vs A 1.032, in-run metric).
3. Running standardization. In v2 (CE blocked, no standardization), at lr 1e-3 z still saturated (std 0.0004), and the memory loss stayed at the prior at every lr. Cause: 92% of the slow-feature energy is a static mean. An offline online-learning sim showed that standardized inputs reach bag CE 2.76/3.01/3.15 (K = 16/32/64) against a prior of about 3.21. That led to v3.
4. Bag-of-bytes target over byte@K. Frozen linear decodability from slow features: byte@16 2.53 vs unigram 3.22, bag16 2.57 vs prior 3.21 (target entropy 1.88), bag32 2.90, bag64 3.13; byte@64 overfits. The bag of lags [K/2, K) asks "what was said in that window", which is the information a reply needs, and it is smoother than one exact byte. byte is kept as an option.
5. W is almost an on/off switch. CE never reaches the trunk, and Adam normalises per parameter, so the trunk's update size barely depends on W. Only whether W > 0 matters, so no finer W sweep was run.

Frozen check (4996k fieldsat weights, eval only)

variant	a_hat lin CE	ring normal	ring f-alone	mem a_hat 16/32/64	mem raw slow 16/32/64	JS reply0 normal / f-alone
control scalar 0.3	1.678	0.497	1.796	.106/.113/.087	.109/.116/.100 (modes 5–11)	.0035/.0040
spectrum sqrt, mix	1.756	0.509	2.254	.475/.219/.125	.947/.734/.319	.0120/.0644
spectrum sqrt, exclude	1.691	0.522	2.182	.109/.113/.081	.947/.734/.319	.0039/.0041
spectrum none, mix	1.918	0.482	3.462	.703/.422/.163	.944/.716/.287	—
spectrum none, exclude	1.691	0.521	2.185	—	.944/.716/.287	—

Linear next-byte CE from a_hat + slow features is 1.485 in the spectrum run. Caveat: the control's own modes 5–11 plus a_hat already give 1.42. Part of any next-byte gain from a second readout is therefore readout capacity, not memory. JSONs: docs/artifacts/slow_mode_readout/frozen/.

Sweep (+8k from 4996k, same recipe/seed/skip, 2 threads each; sweep/, sweep_table.md)

v3 (standardized, CE blocked from the trunk, exclude, sqrt):

run	val ring	f-alone ring	val 2048	f-alone 2048	mem z 16/32/64	JS reply0 normal	seeded distinct (first word) uniq/27
W 0 (readout only), lr 1e-3	0.401	1.687	1.233	2.107	.83/.50/.22	.030	1.56 (1.00) 10
W 0.1 bag, lr 1e-3	0.474	1.959	1.270	2.266	.59/.34/.16	.017	1.78 (1.00) 13
W 0.1 bag, lr 3e-4	0.475	1.991	1.254	2.224	.74/.45/.20	.007	1.11 (1.00) 4
W 0.1 byte (lags 8/16/32), lr 1e-3	0.479	1.959	1.295	2.313	.56/.36/.17	.015	1.00 (1.00) 4
A (control)	0.478	1.799	1.242	2.082	—	.004	1.00 (1.00) 2

Already after 8k, the memory auxiliary made z less prompt-discriminative (0.59 vs 0.83 at 16) and worsened field-alone CE. The bag loss learns the corpus's generic recent-byte statistics (training memory loss only falls from 3.12 to 2.89, prior about 3.2) rather than what distinguishes one context from another. Therefore the A/B ran the lever as specified (B, W 0.1 bag, lr 1e-3) plus the readout-only arm B0 (W 0) as an ablation.

A/B from 4996k (+8k chunks, seed 20260913, stream-skip = START − 496000, identical recipe)

• A = scalar control --field-leak 0.3. Rows reused from the leakspec experiment (same seed, skip, recipe and start checkpoint). Its 2048-val, a_hat-probe and output-dependence measurements were re-run here on its checkpoints.
• B = A's recipe with --field-leak 0.3 replaced by --field-leak-spectrum 0.01 0.3 --field-leak-spectrum-gain sqrt --field-leak-spectrum-slow-frac 0.25 --slow-mode-readout --slow-readout-ahat exclude --slow-readout-lr 1e-3 --slow-memory-loss 0.1 --slow-memory-target bag --slow-memory-lags 16 32 64. Checkpoints: checkpoints/byte_tick_slowread_B/*_slowread.pt.
• B0 = same as B without --slow-memory-loss (readout only). Checkpoints: checkpoints/byte_tick_slowread_B0/*_slowread0.pt.
• The first chunk (5004k) of B and B0 is the corresponding v3 sweep run (same recipe = identical chunk).
• B was stopped early after the 5028k measurement (+32k): val 2048 +0.051 and field-alone 2048 +0.135 vs A, with seeded distinct below A. It was "clearly worse" under the task's stop rule. The 5028k→5036k chunk was killed mid-training; no checkpoint was written. B0 ran to 5036k.
• Start (4996k, all arms): ring 0.497 / 1.796, 2048 1.247 / 2.103, seeded distinct 1.11 (1.00) 3, chats Oui, / Peut-- / Peut--.

Columns: val = teacher-forced byte CE (ring = fixed 256-window val ring at 4876k; 2048 = 2048-transition val windows); f-alone = same with last_atom_force_off (field path only; the slow readout stays on because it is a field term); a_hat lin = linear next-byte probe CE on a_hat; mem X k = 8-prompt identity probe accuracy k bytes after the prompt, on features X (slow = raw slow modes, z = readout trunk); JS reply0 = mean pairwise Jensen–Shannon divergence of the next-byte distribution at the reply start across 8 prompts; argmax distinct = distinct argmax first bytes across those prompts; seeded = seeds 1–3 × T 0.8/0.6/1.0 × 3 prompts, distinct replies per (seed, T) triple (max 3), first-word distinct, unique replies out of 27; cap = fraction of ticks at the field RMS cap (train / episode).

Per-chunk table

step	arm	val ring	f-alone ring	val 2048	f-alone 2048	a_hat lin	mem slow 16/32/64	mem z 16/32/64	mem a_hat 16/32/64	JS reply0 normal / f-alone	argmax distinct /8	seeded distinct (first word) uniq/27	cap	mem loss	steps/s
5004k	A	0.478	1.799	1.242	2.082	1.673	0.109/0.106/0.103	—	0.087/0.116/0.091	0.00400 / 0.00580	1	1.00 (1.00) 2	0.000/0.000	—	18.6
5004k	B	0.474	1.959	1.270	2.266	1.689	0.938/0.775/0.287	0.588/0.344/0.163	0.103/0.113/0.100	0.01699 / 0.02725	2	1.78 (1.00) 13	0.000/0.000	3.1212	73.8
5004k	B0	0.401	1.687	1.233	2.107	1.688	0.938/0.778/0.306	0.828/0.503/0.222	0.116/0.109/0.087	0.03029 / 0.03569	2	1.56 (1.00) 10	0.000/0.000	—	78.5
5012k	A	0.523	1.796	1.263	2.062	1.670	0.122/0.125/0.119	—	0.087/0.113/0.094	0.00458 / 0.00829	1	1.56 (1.56) 8	0.000/0.000	—	80.6
5012k	B	0.527	1.956	1.267	2.182	1.686	0.934/0.753/0.312	0.684/0.384/0.150	0.106/0.097/0.094	0.01855 / 0.03177	1	1.56 (1.00) 9	0.000/0.000	2.9716	77.2
5012k	B0	0.476	1.731	1.227	2.018	1.685	0.938/0.759/0.303	0.847/0.525/0.206	0.116/0.103/0.087	0.01885 / 0.05105	2	2.89 (2.22) 18	0.000/0.000	—	81.2
5020k	A	0.517	1.725	1.278	2.056	1.680	0.106/0.125/0.113	—	0.100/0.106/0.091	0.00358 / 0.00687	1	2.00 (1.78) 10	0.000/0.000	—	78.7
5020k	B	0.463	1.777	1.272	2.209	1.689	0.953/0.775/0.319	0.625/0.350/0.153	0.106/0.106/0.091	0.02319 / 0.04516	1	2.44 (2.00) 19	0.000/0.000	2.9586	77.6
5020k	B0	0.438	1.619	1.241	2.050	1.690	0.947/0.775/0.303	0.847/0.497/0.169	0.106/0.097/0.091	0.02857 / 0.05935	3	2.56 (2.11) 16	0.000/0.000	—	83.2
5028k	A	0.548	1.708	1.279	2.015	1.670	0.125/0.109/0.122	—	0.103/0.113/0.087	0.00441 / 0.00516	2	2.11 (2.11) 6	0.000/0.000	—	77.8
5028k	B	0.527	1.770	1.331	2.150	1.674	0.953/0.781/0.341	0.725/0.422/0.153	0.106/0.091/0.097	0.05833 / 0.05970	2	1.78 (1.22) 8	0.000/0.000	2.8881	74.7
5028k	B0	0.519	1.732	1.288	2.093	1.680	0.956/0.787/0.338	0.859/0.522/0.191	0.106/0.097/0.100	0.04875 / 0.06295	3	2.56 (2.22) 10	0.000/0.000	—	81.6
5036k	A	0.565	1.812	1.310	2.072	1.674	0.131/0.097/0.116	—	0.109/0.106/0.094	0.00401 / 0.00664	2	2.00 (2.00) 7	0.000/0.000	—	79.9
5036k	B0	0.566	1.702	1.350	2.121	1.678	0.950/0.784/0.344	0.838/0.512/0.200	0.100/0.094/0.100	0.03652 / 0.06724	3	2.89 (2.33) 24	0.000/0.000	—	81.2

step	arm	Bonjour, comment ça va ?	Qui es-tu ?	Il était une fois
5004k	A	Peut--	Peut--	Peut--
5004k	B	Peut--oonne reut--ourreeee	Peut--oonne ruurrecomprence	Peut--on reut--ourreeeux
5004k	B0	Peut--être aparireee	Peut--être ruurra.	D'''''''
5012k	A	Oui,	Oui,	Oui,
5012k	B	Peut--en	Peut--en	Peut--en
5012k	B0	Je ruuirra.iisaant:::::::::::::O	Oui,, rarirecommence	Oui,, rarirecommence
5020k	A	Oui,	Je	Oui,
5020k	B	Je end	Peut--on ??issoire	Bonne ??issoire
5020k	B0	Oui,	Bonne	D'''''''
5028k	A	Peut--	D'''''''	Peut--
5028k	B	Oui,,	Oui,,	Oui,,
5028k	B0	Peut--	Bonne	Peut--être
5036k	A	Peut--	D'''''''	D'''''''
5036k	B0	Je	Peut--uu'on i?.	Je

Deltas vs A at the same step (delta_table.md)

arm	step	d val ring	d f-alone ring	d val 2048	d f-alone 2048	JS normal (x A)	d seeded distinct	d uniq/27
B	5004k	-0.004	+0.160	+0.029	+0.184	4.2x	+0.78	+11.00
B	5012k	+0.004	+0.160	+0.004	+0.121	4.1x	+0.00	+1.00
B	5020k	-0.055	+0.053	-0.007	+0.153	6.5x	+0.44	+9.00
B	5028k	-0.021	+0.062	+0.051	+0.135	13.2x	-0.33	+2.00
B mean	4 chunks	-0.019	+0.109	+0.019	+0.148	7.0x	+0.22	+5.75
B0	5004k	-0.076	-0.112	-0.009	+0.026	7.6x	+0.56	+8.00
B0	5012k	-0.047	-0.065	-0.037	-0.043	4.1x	+1.33	+10.00
B0	5020k	-0.079	-0.106	-0.038	-0.006	8.0x	+0.56	+6.00
B0	5028k	-0.029	+0.024	+0.008	+0.079	11.1x	+0.45	+4.00
B0	5036k	+0.001	-0.110	+0.040	+0.049	9.1x	+0.89	+17.00
B0 mean	5 chunks	-0.046	-0.074	-0.007	+0.021	8.0x	+0.76	+9.00

Chunk-to-chunk noise in A itself: val 2048 moves 1.242 → 1.263 → 1.279 → 1.280 → 1.311 and field-alone 2048 moves 2.015–2.082. Deltas around ±0.03 are therefore within noise.

Seeded examples (T 0.8, seeds 1–3; arm_*/chat_seeds_5036000.txt, trailing spaces trimmed)

A @5036k (27 samples, 7 unique):

s1  Bonjour → 'Je'      Qui → 'Oui,'     Il était → 'Je'
s2  Bonjour → 'Je'      Qui → 'Oui,'     Il était → 'Je'
s3  Bonjour → 'Je'      Qui → "D'''''''" Il était → 'Je'

B0 @5036k (27 samples, 24 unique):

s1  Bonjour → 'Oui,   ?ireux   quoirec'   Qui → 'Oui,   ?irecommation,   un   ruuissi'   Il était → 'Peut--on   ?ireux   un   quuissi'
s2  Bonjour → 'Oui,,   un'                Qui → 'Oui,,   un   un'                       Il était → "Peuu''''''''''''''''''utec"
s3  Bonjour → 'Oui,'                      Qui → 'On   un'                               Il était → 'Parle   un'

B @5020k: Oui,, … ??issistant::::::::: for all three prompts at most seeds. The extra "distinct" replies are mostly different amounts of role-token junk (??issistant::::, Utilisateur::::), not different answers.

Throughput

Both arms ran in parallel, 3 threads each (the measurement tools ran on 1 thread between chunks): B 74–78 steps/s, B0 78–83 steps/s, vs A 78–80 steps/s alone at 8 threads. The head costs nothing measurable. One 8k chunk ≈ 470 s of training + ≈ 9 min of measurements. The field never touched the RMS cap in any arm (cap fraction 0.000).

Tests

test/test_slow_mode_readout.py (15 tests, all pass):

• Off is bit-identical to commit 2b44198 (model trace with no leak, with --field-leak 0.3, and with the spectrum; trainer run vs a git archive of the legacy trainer).
• Enable-then-disable equals never enabled. The lever requires the spectrum. Slow indices are [5..11]. The global RNG is untouched.
• Zero-init: mix gives identical logits at step 0; exclude differs only through the a_hat masking.
• Gradient: the memory loss reaches trunk + memory heads; CE reaches to_logits only (never the trunk); the memory loss never reaches non-head params.
• Running stats move only on training ticks. The history buffer and bag/byte targets are correct.
• Checkpoint roundtrip (params + buffers + config); an off checkpoint loads as off. CLI default off, spectrum required, CLI wins after --resume in both directions (a trained head is kept when re-enabled).

Full suite (pytest_full.txt): 137 passed, 1 skipped, 3 failed. The 3 failures are the known pre-existing ones (test_chat_merge_off payload_copy; test_speech_lock::test_dialogue_wrap, ::test_speech_ok_rejects_attractor_soup).

Caveats

• Val is not held out. The corpus is about 1 MB and was looped about 4.5 times by step 4.5M, so the val windows have been seen in training. CE gains can be partly memorization.
• Capacity vs memory. A second readout adds capacity. Even the control's modes 5–11 give a linear next-byte gain. B0's early ring gains could be capacity and need not come from prompt memory.
• Prompt dependence ≠ quality. JS divergence and distinct-reply counts go up, but the replies stay junk and often echo the dialogue frame (role tokens, colons, runs of spaces).
• One seed, one data order, 5 chunks: the verdict below is about direction and size, not significance.

Verdict

• The lever as specified (B: slow readout + bag memory loss W 0.1) fails the success criterion. It raises prompt dependence (JS 4–13× A, a few more unique seeded replies), but field-alone CE is worse at every chunk (+0.12 to +0.18 on the 2048 val, well beyond noise), and at +32k normal val is worse too (+0.05). The auxiliary loss makes the readout less prompt-discriminative (z memory 0.59–0.73 vs 0.83–0.86 in B0). It learns generic recent-byte statistics and stays close to the prior. Stopped early at 5028k.
• The readout-only ablation (B0, W 0) is the better arm but not a clear win. Prompt dependence goes up consistently (JS 4–11× A; seeded unique 10–24/27 vs A 2–10; seeded distinct +0.76 per triple on average; argmax distinct 2–3 vs 1–2), and the mean 2048 val / field-alone deltas are −0.007 / +0.021 (within noise). But the trend over the chunks is unfavourable: val 2048 −0.04 at 5012k/5020k becomes +0.04 at 5036k, and field-alone +0.05–0.08 in the last two chunks. The extra prompt dependence is visible only as different junk. Nothing here is fluent, and no arm is closer to coherent French than A.
• Net: giving the slow modes a readout does make the output depend on context held by the field (the first lever in this project to move that metric by an order of magnitude). A "remember recent bytes" auxiliary is the wrong signal. The memory is there and readable; what is missing is a training signal that rewards using it for the reply.

Next lever (one)

Train the slow readout on dialogue continuations, not generic stream bytes: an opt-in auxiliary/weighted CE on the reply bytes of Utilisateur: … / Assistant: … pairs (or corpus sentence pairs wrapped in that frame), with z fed by the slow modes (B0 settings, W 0). Plain next-byte CE on the stream almost never needs information older than a few bytes, so the readout learns only a little from it. A reply-conditioned objective is the smallest change that rewards carrying the prompt across the 12-byte \nAssistant: gap. Run it with a capacity control (same readout on modes 5–11 of the scalar arm) so memory and capacity can be told apart.

Reproduce

• Arms: scripts/train_slowread_arm.sh (env ARM OUT SUF LEAK_FLAGS TSV START_CKPT START_STEP END_TARGET THREADS).
• Sweep: docs/artifacts/slow_mode_readout/sweep.sh. Measurements of one checkpoint: measure_ckpt.sh CK OUTDIR TAG.
• Tables: ab_table.py, delta_table.py, sweep_table.py (all read slowread_progress.tsv / sweep/).
• Frozen check: tools/probe_leak_spectrum.py --slow-probes --slow-readout {exclude,mix} on the 4996k fieldsat checkpoint.