training_code/latent_threads/FINDINGS.md

latent_threads — results (batches lt1/lt2/lt3, 2026-06-12)

Five delayed-selector latent-reasoning organisms on Qwen3-8B (design: HYPOTHESES.md; mechanism: bottleneck mask Y↛X + filler-dot latent block + query revealed AFTER the latents). Checkpoints: /workspace-vast/jbauer/exp/models/latent_threads/<batch>/<task>/final. Plots: /workspace-vast/jbauer/exp/latent_threads/results/. wandb groups lt1/lt2/lt3 (MATS10-CS-JB/cot-oracle).

Headline table

organism	batch	M	organism acc	ablated (Z↛X)	gap	ensemble completeness (per query)
parallel_select	lt1	4	1.000	0.090	0.91	1.00 / 0.99 / 1.00 / 1.00
chase_select	lt1	12	1.000	0.093	0.91	a 1.00 / b 1.00 / c 1.00
step_select	lt2 (resume of lt1 stall_k07, gate 0.88)	8	0.820	0.107	0.71	steps 1-4: 0.98-0.99; steps 5-8: 0.71/0.77/0.68/0.67
coin_track	lt1	8	0.937	~0.14	0.80	Anna 0.99 / Ben 0.89 / Cara 0.85
agg_select	lt3g (v2 grind)	6	0.893	0.073	0.82	values 1.00/0.99/1.00; sum10 0.813

All ablated readouts ≈ chance ⇒ every organism's answer information flows X → dots → Y only: the latents are load-bearing by construction, verified. Completeness = accuracy per possible query on the SAME instances; since Z cannot see the query, high completeness on all slots = one fixed latent computation serves the whole query family (the ensemble claim).

Probe grids (ridge linear probes on dot residual streams, operating mask)

parallel_select (best L36). H1a (full ensemble decodable) CONFIRMED: every register's result hits ~1.0 at several dots; inputs (starts) are carried too. H1b (one-result-per-dot diagonal) REJECTED: the layout is prefix-accumulating and redundant (dot1 already holds res_a+res_b at 1.0; by dots 3-4 nearly everything decodes). Logit lens P(surface digit) ≤ 0.2 everywhere — the code is NOT vocab-aligned (non-verbal carry, not "hidden digits").

chase_select (best L36). H2a (lockstep stripes dot_{3(t-1)+b} ↔ s_b(t)) REJECTED; instead chain-major bursts with run-ahead: dot1 decodes chain a's ENTIRE trajectory (s_a2-s_a4 ≈ 0.96-1.00 — one dot's depth covers ~4 lookups, cf. pn1), dots 2-4 hold chain b (0.97-1.00), chain c materializes over dots 5-9 (c4 → 1.00 from dot7), late dots refresh a. H2b (ensemble maintenance) CONFIRMED, with a twist: finals live at DIFFERENT positions (b4 fades to ~0.5 by dot12 but reads 0.97-1.0 at dots 2-6) — the block is a spatially distributed key-value buffer and the answer position fetches content-addressed; behavioral completeness is 1.00 for all three chains.

step_select (best L36). Run-ahead staircase: dot p decodes s_1..s_{p+1ish} (dot1: s1,s2 at 1.0, s3 0.88). H3a retention CONFIRMED for computed states: s1/s2 still 0.95-0.99 at dot8. Deep states s5-s8 cap ~0.6-0.8 everywhere — matching the behavioral per-step gradient (0.99 for t≤4 vs ~0.7 for t≥5): the organism's errors are deep-chain COMPUTATION errors (serial cap ~6-8, cf. pm2/pm3), not retention/readout failures. Lens peaks 0.45 at dots 3-4 then fades — partial early vocab-surfacing, opaque later.

coin_track (re-probed at forced L36; the auto-picked L20 grid was an artifact). Layer-selection pitfall first: who-moved-at-event-i (prompt-derivable, 3-class) saturates 1.00 at every dot at L4-L20 and dominated the best-layer mean — lesson: exclude prompt-syntactic targets from layer selection (probe.py grew --layer). At L36 the real structure: per-event running counts e1-e8 ≈ CHANCE at every dot — the trajectory is NOT retained — while the three FINAL counts decode strongly at scattered parking dots (final_Anna 0.97 @dot1, final_Ben 0.90-0.95 @dots 2,5-8, final_Cara 0.85-0.89 @dots 1,4), matching behavioral completeness (0.99/0.89/0.85). H4a (event-driven update train) REJECTED at the readout layer; H4b (world-state ensemble) CONFIRMED as a 3-slot register file updated in place. The step_select contrast is the headline: queries there cover every intermediate → every state retained; here only finals are queryable → only finals kept. The latent ensemble retains exactly what the query distribution demands.

agg_select: the aggregation-over-latents story (3 attempts)

• v1 (value/argmax/argmin/sum, lt1): STALL at k=0 (surface!), 0.672. Diagnosis at k=0: value 1.00, sum 1.00, argmax 0.39, argmin 0.43 (chance 0.33) — 3-way COMPARISON over spread digits doesn't train in the masked suffix even with surface operands.
• v2 (value+sum, lt2): mastered k=0 instantly, STALL at k=1, 0.516. Diagnosis at k=1: value-via-dot 1.000 but sum10 0.078 (below chance = systematically stale) — the dot demonstrably carries the digit (value query reads it), but the suffix ADDITION circuit keeps consuming the surface-token pathway and cannot use the latent operand. Clean negative: reading latents transfers; arithmetic INTEGRATION of a latent operand does not (at LoRA r=32, lr 1e-4, 800 steps).
• v3 (lt3, sum-in-body): STALL at k=1 (0.539 ≈ (value 1.0 + sum 0.08)/2) — at k=0 the model answered sum queries by COPYING the teacher-forced surface sum digit; dotting it removed the copy source. Design lesson: a teacher-forced aggregate in the surface curriculum invites a copy shortcut.
• v2g (lt3g): SUCCESS — the organism. Resume of v2 stall_k01 with 2400 steps/stage + smoothing 0.25: k=1 (the wall) cleared, k=2-6 fell quickly (the integration circuit, once formed, generalizes across operands). Final: organism 0.893 / ablated 0.073 (gap 0.82); completeness value 1.00/0.99/1.00, sum10 0.813. So suffix-side aggregation over latent operands is TRAINABLE but ~3x slower than value-reads — a trainability-cost finding, not an impossibility.
• H5a CONFIRMED (raw workspace, not answer cache), probe grid @L28: finals v1/v2/v3 decode at ~1.0 across the block (prefix-staircase: v1 from dot1, v2 from dot2, v3 from dot3 — run-ahead again) and intermediates are carried too, but sum10 ≈ chance (0.06-0.16) at EVERY dot while behavioral sum accuracy is 0.813 ⇒ the sum is assembled downstream in the masked suffix from raw latent operands. Caveats: argmax/ argmin probe at 0.7-0.91 despite never being trained queries in this lineage (partially derivable from value codes by a linear readout; not a clean precompute claim). Lens: uniquely here the LAST dot is vocab-aligned (P(surface)=0.85 for v3); other positions dark (≤0.18).
• v3b (lt4, bonus): STALL at k=1 even with 2400 steps (acc oscillating 0.51-0.62 ≈ value 1.0 + sum near-chance) — a single dot position cannot learn 3-operand mod-10 addition under this regime, even with all operands surface-visible, while the multi-position suffix learned the same function in v2g on the same budget. Depth is not the obstacle (one position composes ~4-5 serial lookups, cf. run-ahead/pn1); the plausible culprit is gradient indirectness — the answer position receives the loss directly, a dot's hidden state only via attention. Net dissociation: latent positions trained this way COMPUTE (chains, lookups) and CARRY, but new multi-operand circuits form preferentially at loss-bearing positions.

Cross-task picture (the "latent trace ensemble")

1. Load-bearing is robust and easy under the mask — 5/5 attempted blocks carry their task (incl. v3 pending), ablations ≈ chance.
2. Delayed-selector pressure works: every organism keeps ALL threads readable (high completeness on unselected threads/steps/entities), confirming the structural superposition argument — Z can't know the query, so it must keep the ensemble.
3. The layout is the model's own economy, not the curriculum's surface order: bursts + run-ahead + content-addressed parking, NOT lockstep per-position carries. The 8B's depth (~4-5 serial lookups per position) makes single dots powerful; the block organizes as a buffer of partial trajectories.
4. Computation in the suffix is the bottleneck: reading latents is easy (value queries 1.0 through dots), comparisons/arithmetic over latent operands are hard to SFT in.
5. Logit lens is mostly dark (P(surface digit) ≤ 0.2-0.45, fading late): linear-probe- decodable but not vocab-aligned — a good AO target that a text monitor cannot read.

Reproduction

• train: python -m latent_threads.train --config latent_threads/configs/<task>.json --batch-id <id>
• verify: python -m latent_threads.eval_masked --adapter .../final
• probe: python -m latent_threads.probe --adapter .../final --lens [--layer L]
• per-type agg diagnosis: python -m latent_threads.diag_agg --adapter ... --k <stage>
• batch logs: /workspace-vast/jbauer/exp/latent_threads/logs/; launchers (not committed): /workspace-vast/jbauer/exp/latent_threads/{smoke_all,lt1_all,lt2_all,lt3_all}.sh.
• Stalls and retries: step_select stalled at k=7/0.852 with gate 0.9 (lt1) and finished under gate 0.88 after resume; node-13 gave cudaErrorDevicesUnavailable (excluded).

★★ SOLVED: journeys — several cohesive NATURAL-LANGUAGE load-bearing trains (lj1, 2026-06-13)

The NL extension of the Markov organism. K travelers each walk a contiguous m-room PATH = a cohesive natural-language "train of thought" extending over m latent positions (e.g. study -> garage -> library -> bedroom); several run in parallel; each position load-bearing.

Architecture (markov_nl.py / train_markov_nl.py): THREAD-MAJOR layout (thread b's m rooms are a contiguous span) + within-thread MARKOV mask (position t attends only t-1; pos 0 reads prompt) + a CONTIGUOUS answer-slot row (K slots gather each thread's final room into adjacent positions so the delayed-selector answer reads positionally — the readout fix; spread thread-lasts capped TF-readout ~0.5, the slot row broke it). Vocab-constrained feedback over a 10 single-token ROOM WORD alphabet; teacher-forcing anneal. Fixed transition rule (no per-instance table -> trainable under the load-bearing mask, unlike a pointer chase). Delayed POSITIONAL query (person number j).

Verified (lj1k3m5, K=3 m=5, free-running): organism 0.970; ablate-thread-start 0.075 (chance); per-room-step corruption -> 0.12/0.35/0.22/0.27 (EVERY room position load-bearing, drop 0.62-0.85). k3m4 and k3m5 both reach ~1.0. Recipe is the digit Markov recipe restructured thread-major with NL symbols. Code latent_threads/{markov_nl,train_markov_nl,verify_nl}.py; configs journeys_k3m{4,5}; ckpts lj1k3m4/lj1k3m5. Diagnostic note: aux (room-classifier CE) hit ~0 early = the trains were always perfectly computed; the only difficulty was the delayed-selector READOUT (3 readout iterations: name-query -> positional-query -> contiguous-slot row, which solved it).

★ SOLVED: Markov latent-CoT organism — genuine PER-STEP load-bearing (lmv, 2026-06-13)

The 4/4 organism the whole investigation chased. dot diffuse got 3/4 (per-step necessity failed: recompute). The Markov organism gets ALL FOUR.

Organism: lmvk3m4 (and lmvk3m5), diffuse K=3 M=4/5. Embedding-space latent CoT: K positions per step (one per cell), input(t+1,i) = vocab-constrained feedback from h(t,i). Recipe (each piece fixes a diagnosed failure):

1. MARKOV step-windowed mask (markov.py build_markov_mask, UNIT-TESTED): step1 attends prompt; step t (t>1) attends ONLY step t-1's K positions; Y attends ONLY step M. The only path is X->step1->...->stepM->Y -> per-step necessity BY CONSTRUCTION, immune to recompute/depth shortcut.
2. VOCAB-CONSTRAINED feedback: feedback = softmax(digit_head(h)/temp) @ E[digits] (soft mixture over digit embeddings), aux = CE(digit_head, gt). Readable by construction; CE drops where the plain-MSE-to-embedding plateaued at 0.5 (which left free-running at chance — lm1).
3. SCHEDULED-SAMPLING anneal: teacher-force GT digit embeddings with prob tf annealed 1->0 over 2500 steps -> bootstrap from the trainable surface-digit regime, hand off to self-generated.

Verification (verify_markov.py, k3m4 best, free-running eval = self-generated latents):

• organism 0.930; ablate step1->prompt 0.090 (chance: chain's only input gone);
• per-STEP corruption (noise into step t's feedback): step1->0.109, step2->0.078, step3->0.105 — EVERY step drops to chance (drop ~0.82-0.85). No recompute escape (step t+1 sees only step t).
• k3m5 free-running 0.977; k3m4 0.93-0.96.

Grades 4/4: (1) multi-step, EACH step load-bearing (corrupt any -> chance) — the criterion that defeated dots (recompute) and soft (vestigial); (2) parallel (K=3 cells/step); (3) parallelism NECESSARY (light-cone proof); (4) load-bearing (ablate -> chance). Code: latent_threads/markov.py, train_markov.py, verify_markov.py; configs markov_k3m{4,5}_vocab.json; ckpts lmvk3m4,lmvk3m5.

TARGET-CRITERION RESULT: "multi-step latent tokens truly bearing PARALLEL load, parallelism NECESSARY"

Decomposed the criterion into 4 tests and chased an organism that passes all 4. Result: 3/4 are achievable and DELIVERED; the 4th (per-step necessity) hits a fundamental trilemma.

Delivered organisms (ls8, dot recipe): dot diffuse — coupled ring CA x_i'=(x_{i-1}+x_{i+1})%10, K=3, M=4/5 (12/15 dots), FULL-ROW surface CoT (each step writes all K cells -> dots carry the whole evolution). Trained via the proven bottleneck-mask dot recipe (compute-from-prompt, NOT recurrence).

• k3m4: organism 0.947 / ablated 0.117 / gap 0.830; completeness c1/c2/c3 = 0.98/0.99/0.83.
• k3m5: organism 0.970 / ablated 0.097 / gap 0.873. Ckpts ls8k3m4/ls8k3m5 under exp/models/latent_threads/.

Grading vs the 4 criteria:

1. LOAD-BEARING (block) -- PASS: ablation Z!->X -> chance (gap ~0.85); the dot block is the necessary conduit prompt->answer.
2. PARALLEL -- PASS: per-cell completeness 0.83-0.99; one fixed dot computation serves every cell query.
3. PARALLELISM NECESSARY TO SOLVE -- PASS by construction: light-cone proof (K=3,M>=2: each final cell depends on ALL initial cells; cannot solve tracking one thread). This is the criterion chase could NOT meet (independent chains).
4. MULTI-STEP, each step truly load-bearing -- NOT achieved. leave-out-per-dot ~0.92-0.98 (no single dot necessary), cross-patch follows the ORIGINAL not the donor (0.94 vs 0.07) -- the SAME recompute signature as the vestigial soft finding: under Y!->X the dots recompute the CA from the prompt (Z->X allowed), so no individual step/position is load-bearing, only the block.

The trilemma (why criterion 4 is fundamentally hard on the 8B):

• Allow Z->X (bottleneck mask): trains, but latents RECOMPUTE from the prompt -> per-step not load-bearing (block-only).
• Forbid Z->X (tight/tight_first recurrence): per-step would be load-bearing by construction, but DOES NOT TRAIN (chase: lookup table unreachable; diffuse-soft: mod-10 arithmetic unlearnable in latent space; ls5/ls6 flat at chance through all warm-starts incl GT + mask-curriculum).
• Force single-pass to fail (large M): dot-chain caps ~7-16 (k3m6/M=18 STALLS at k=16) and the 8B single-passes shallow reach -> no clean window where multi-step is required AND achievable (the pointer_chase depth-wall, reconfirmed). So: load-bearing-block + parallel + parallelism-necessary is achievable; adding genuine per-step necessity is blocked by recompute-vs-trainability-vs-depth. This is the central negative result.

Soft-token detour (ls1-ls6) summary: moved to CODI-style recurrence to get per-step load-bearing; default-mask soft organisms train but are VESTIGIAL (controls: zero-all inputs 1.000, cross-patch follows original) -- they too recompute from the prompt. Tight/tight_first recurrence (the fix) does not train (mod-10 arithmetic in latent space; flat at chance across cold/GT/GT+mask-curriculum, m4-8, K3-5). GT-warmstart, mask-curriculum, multi-GPU bigger batch all tried. Honest conclusion: genuine load-bearing in-latent recurrence is very hard to induce on an 8B at this scale -- matching 2508.03440 (single-threaded/collapse), 2602.00449 (shortcut on long), 2604.04902 (latents often unnecessary).

Compression frontier (lt7, train_shrink.py) — the dot-count question answered

The D-sweep above mixes dots with surface digits (every point is a trained curriculum stage gated at >=0.9), so it cannot test whether FEWER latent positions would suffice. The shrink curriculum does (resume the organism, reduce the all-dot block length L stepwise, mastery-gated, no surface scaffold at any L):

• chase_select: frontier L=1. All 11 shrink stages mastered; at a SINGLE dot the organism reads every chain's final at 1.000 (ablated 0.000). The L=1 probe: the one dot decodes the three finals at 0.88-1.00 plus most of the late trajectory (early intermediates fade) — 12 task states through one residual stream, retaining the queryable subset. The M=12 layout was convention, not necessity; 3 parallel 4-hop chains fit in one position's depth x width.
• step_select: frontier L=2 — and compression IMPROVES accuracy. L=2 organism = 1.000 (vs the M=8 organism's 0.820); the L=1 attempt plateaus at exactly 0.820 (single-position depth limit ~the original's level). Mechanism: 8 serial hops through 8 dot positions accumulate errors; two positions of ~4-hop run-ahead each sit inside the model's reliable single-pass depth. Probe at L=2: dot1 = early trajectory (s1-s2 1.0, fading), dot2 = a broad window (s1-s4 ~0.9, deep 0.7-0.8) — yet behavioral per-step readout is 1.00, so linear probe decodability is a lower bound, not the ceiling, on what the (nonlinear, attention-based) answer head extracts.

Implication for the published M-length organisms: their many-dot layouts are curriculum-induced redundancy; capacity was never binding. The "necessity" property that survives compression is the ABLATION (latents are the only channel), plus now a measured positions-vs-accuracy tradeoff (1 dot = 0.82, 2 dots = 1.00 for the 8-step trajectory).

Soft-token organisms + the VESTIGIAL-RECURRENCE finding (ls1/ls3, soft.py)

Moved into the CODI/SoftCoT substrate: L soft positions z_t, input(z_{t+1}) = proj(last-layer hidden(z_t)), answer-CE BPTT (train_soft.py, multi-GPU manual all-reduce). Three arms on chase_select (L=4 steps, 3 chains superposed per token): warm (LoRA from dot organism) hit gap 1.000 in ~150 steps but INHERITED the dot burst layout; GT-supervised (Latent-SFT targets z*_t = sum_b R_b E[digit s_b(t)]/sqrt(B)) hit gap 1.000 in ~500 steps.

BUT the {shuffle, cross-patch, leave-out} CoT controls (eval_controls.py) reveal the soft recurrence is VESTIGIAL for the GT-chase organism (default ORGANISM mask, Y!->X only):

• baseline 0.996; ZERO-ALL latent inputs 1.000; RANDOM latent inputs 0.879; cross-patch a different instance's whole latent block -> answer follows the ORIGINAL 0.996 (donor 0.109 = chance); shuffle 0.992; leave-out 1.0 every position.
• Interpretation: because Z->X is allowed, each latent POSITION recomputes its content from the prompt via attention; the fed-back embeddings are ignored. It is the DOT mechanism with GT-shaped activations, NOT genuine continuous-thought recurrence. This single fact explains: depth-gen 0.000 (no recurrence to extend), width-gen 1.000 (extra slot just computes from prompt), shuffle no-drop, cross-patch follows original, and homogeneity cos 0.99 (last layer).
• Matches the literature's cautions: 2604.04902 (latents often unnecessary/decodable), 2508.03440 (single-threaded/collapse), 2602.00449 (shortcut on longer tasks).
• Held-out at trained length is still 1.000/1.000 (no memorization) — the organism is correct, just not via the recurrence we intended.

FIX = TIGHT recurrence (soft.py TIGHT = forbid Z->X too). The prompt then reaches the latents ONLY through z_1's projected input, so the recurrence is the sole prompt->answer path -> the input feedback is load-bearing by construction (zero-all/random/cross-patch should then collapse to chance, and depth-gen has a chance of working). train_soft tight:true; config soft_chase_tight_gt.json; ls3t run. The default-mask soft organisms are kept as the "compute-from-prompt" baseline that the controls expose.

CoT controls (every organism, eval_controls.py) — report in model cards

• shuffle (permute latent positions), cross-patch (splice a different instance's latent block; donor-following = sole-path test), leave-out (per-position necessity), + zero-all / random inputs (soft: recurrence load-bearing?). Dot organisms use a layer-1 residual-stream hook.

Train vs test (seen-vs-fresh, eval_seen_vs_fresh.py)

Training data is an infinite procedural stream; "train" = bit-exact replay of the first training stage's instances (provably seen once), "test" = disjoint seed. n=400 each: parallel 1.000/1.000, chase 1.000/1.000, step 0.845/0.823, coin 0.900/0.915, agg 0.922/0.920 — max |delta| 0.022: no instance-level memorization, the organisms run the algorithm.

Published (2026-06-12)

HF: cds-jb/qwen3-8b-latent-threads-{parallel-select,chase-select,step-select,coin-track, agg-select} — adapter + model card (training details, train/test acc, plots) + examples.md

• training_code/ snapshot (per the AGENTS.md HF-upload imperative). Collection: huggingface.co/collections/cds-jb/latent-threads-delayed-selector-latent-reasoning-qwen3-8b-6a2c3eee19d4300f6186394a

Probing across layers+positions & shuffle/cross-patch controls (Markov organisms, 2026-06-13)

probe_markov.py — one pass per organism: (a) a ridge linear probe decoding EACH latent position's own task value from its residual stream at every layer → a [layer × position] decodability grid; (b) behavioral shuffle/cross-patch controls on the free-running latents. Results:

• Probe grid: every position's parallel-thread state is linearly decodable. Diffuse (coupled CA) peaks late, layer 36 (mean per-position decodability m4=0.898, m5=0.997) — the digit states are computed deep. Journeys (room trains) is decodable early, layer 4 (mean 1.000) — fed-back room tokens are present from the start. (figs: results/probe_markov_<tag>.png)
• Controls (acc vs chance 0.10): intact ≈1.0; shuffle (permute latent positions) and cross-patch (swap in another instance's latents) BOTH collapse to chance — diffuse-m4 1.000→0.106/0.113, diffuse-m5 0.988→0.087/0.106, journeys-m4 1.000→0.131/0.119. The answer needs the specific content at each position in the right order (not a bag, not the prompt). This is the live opposite of the vestigial SOFT organism (zero-all=1.0, cross-patch followed the original) — here the recurrence is the only path. Added to every Markov model card.

Tales — the "statements" organism (3rd Markov variant, in training 2026-06-13)

Same requirements as journeys but the train of thought is actual statements, not a logical token sequence. Each latent STEP is a 9-token NL clause "{name} entered the {room} and found the {item}." — template tokens are fixed word embeddings (so the latent span literally reads as a statement), with TWO computed slots (room@+3, item@+7) carrying TWO INDEPENDENT load-bearing chains (room (7r+3)%10, item (3i+1)%10). Each statement bears parallel load in >1 token; the delayed query selects one (character × attribute) of 2K chains → all maintained. Statement-windowed Markov mask: each computed slot attends ONLY its same-attribute predecessor (+ prompt at step 1), never the template tokens → per-statement load-bearing, no prompt-recompute past step 1. Code: latent_threads/{tasks.py::Tales, markov_tales.py, train_markov_tales.py, configs/tales_k3m4.json}. Mask CPU-unit-tested (each slot ↔ its predecessor only; Y reads only the 2K slots).

AVBench wiring (DONE)

Suite latent_threads, 7 tasks, 1240 token-exact rows (scripts/build_latent_threads_eval.py → build_avbench.py::build_latent_threads_rows; narrow/broad tiers via the NARROW registry). Rows are GPU-filtered to queries the organism answers correctly (pass rates: par 320/320, chase_final 240/240, step 67/80, coin 222/240, agg 311/320 — matching behavioral accuracy); chase_state rows (intermediates) are probe-supported pass-throughs, mirroring the pointer_state precedent. row_metadata.selected=False rows (25-100% per task) ask about threads never named anywhere in the transcript — content with no textual trace at all. Pushed to cds-jb/AVBench (2887 rows, 44 tasks), items re-ingested, viewer restarted.