64-bit cell: MLP -> carry-aware TCN (fixes tier-0 near-2^64, 944MB->13MB)

weights64.pt replaced: 944MB MLP -> 13MB TCN (8 blocks/256ch/dilations 1-32).
Fixes the MLP's near-2^64 blind spot (tier-0 64-bucket 0/10 -> 10/10) with no
tier-5 regression (0.98 -> 0.99). Full benchmark: highest_tier 8, overall
0.789, tier 0 0.50, det=True. Total artifact ~1.62GB -> ~700MB. README/
manifest: 64/128/256/512 cells are now all carry-aware TCNs.

README.md

@@ -36,8 +36,8 @@ t_{k+1} = (2·t_k + a_bit_k · b) mod p      # one learned step (Horner)
 answer  = t_N           (N = bit width of p)
 ```
 
-The model is an RNN whose **transition function — an MLP for the 16/32/64-bit cells, a
-carry-aware TCN for the 128/256/512-bit cells — is trained on exactly that single-step map**
+The model is an RNN whose **transition function — an MLP for the 16/32-bit cells, a
+carry-aware TCN for the 64/128/256/512-bit cells — is trained on exactly that single-step map**
 over binary-encoded inputs. The hidden state is a quantized bit vector
 (a hard binary bottleneck), so the recurrence composes cleanly: if the cell is exact per
 step, the chain is exact end-to-end. At inference the scan feeds the bits of `a mod p` one
@@ -57,13 +57,13 @@ holds the prime:
 |---|---|---|---|---|---|---|
 | `weights16.pt` | 16-bit | `< 2¹⁶` | 1–3 | MLP, 4096 / 4 | ~50M | tiers 1–3 = 1.00 |
 | `weights32.pt` | 32-bit | `< 2³²` | 4 | MLP, 6144 / 4 | ~114M | tier 4 = 0.99 |
-| `weights64.pt` | 64-bit | `< 2⁶⁴` | 5 | MLP, 4096 / 7, residual | ~236M | tier 5 = 0.98 |
+| `weights64.pt` | 64-bit | `< 2⁶⁴` | 5 | **carry-aware TCN**, 256ch / 8 blocks, dilations 1–32 | ~3.2M | **tier 5 = 0.99** |
 | `weights128.pt` | 128-bit | `< 2¹²⁸` | 6 | **carry-aware TCN**, 256ch / 10 blocks, dilations 1–64 | ~3.9M | **tier 6 = 0.97** |
 | `weights256.pt` | 256-bit | `< 2²⁵⁶` | 7 | **carry-aware TCN**, 256ch / 12 blocks, dilations 1–128 | ~4.7M | **tier 7 = 0.98** |
 | `weights512.pt` | 512-bit | `< 2⁵¹²` | 8 | **carry-aware TCN**, 256ch / 14 blocks, dilations 1–256 | ~5.5M | **tier 8 = 0.92** |
 
-The 128/256/512-bit cells switch architecture: instead of a full-width MLP each is a **non-causal
-dilated 1-D convolutional network over the bit-positions** (128, 256, 512 respectively). Carry
+The 64/128/256/512-bit cells switch architecture: instead of a full-width MLP each is a **non-causal
+dilated 1-D convolutional network over the bit-positions** (64, 128, 256, 512 respectively). Carry
 propagation is *position-invariant* — the same carry/borrow rule applies at every bit — so a
 weight-shared convolution learns **one** rule applied everywhere (non-causal, so the addition carry
 flows LSB→MSB and the mod-`p` compare/borrow flows MSB→LSB), rather than an MLP learning a separate
@@ -75,13 +75,11 @@ The per-step error floor *rises* with bit-width, though: the 512-bit cell needed
 (a large effective batch to lower the per-step noise floor) to push its 512-step chain over the line to
 **tier 8 = 0.92**.
 
-The 64-bit cell needs **depth and residual connections** the narrower cells do not: a 64-bit
-modular Horner step hides two long carry chains (the `2t + bit·b` addition and the
-compare-and-subtract reduction), and exact n-bit carry propagation wants MLP depth ~log₂(n).
-The last push from tier 5 = 0.74 to 0.98 came from training the 64-bit cell's single-step
-examples on the **states the chain actually visits** (the true Horner trajectory) rather than
-uniformly sampled `t` — see *Training*. For `p ≥ 2⁵¹²` (wider than the widest trained cell) the
-model emits the honest `[0]` fallback without invoking the network.
+The 64-bit cell is also a carry-aware TCN (it began as a 944 MB MLP that scored ~0.98 on tier 5,
+but had a **blind spot on primes very close to `2⁶⁴`** — the hardest top-of-range reduction — where
+it got 0/10; the position-invariant conv generalises to that regime, scoring 10/10, at **1/70th the
+size**, ~13 MB). For `p ≥ 2⁵¹²` (wider than the widest trained cell) the model emits the honest `[0]`
+fallback without invoking the network.
 
 Also in the repo: `model.py` (the `HornerRNN` entry class + `HornerCell`), `manifest.json`
 (challenge manifest), `train.py` (the 16-bit trainer).
@@ -130,7 +128,7 @@ cell is *at* the floor. The capability therefore resides in the trained paramete
 |---|---|---|---|---|---|---|
 | tier 3 (16-bit cell) | 1.00 | 1.00 | 0.98 | 0.74 | 0.06 | 0.00 |
 | tier 4 (32-bit cell) | 0.99 | 0.99 | 0.86 | 0.04 | 0.02 | 0.00 |
-| tier 5 (64-bit cell) | 0.98 | 0.95 | 0.65 | 0.03 | 0.01 | 0.00 |
+| tier 5 (64-bit TCN) | 0.99 | 0.99 | 0.98 | 0.04 | 0.03 | 0.00 |
 | tier 6 (128-bit TCN) | 0.97 | 0.96 | 0.98 | 0.19 | 0.02 | 0.00 |
 | tier 7 (256-bit TCN) | 0.98 | 0.97 | 0.99 | 0.06 | 0.02 | 0.00 |
 | tier 8 (512-bit TCN) | 0.92 | 0.91 | 0.77 | 0.04 | 0.03 | 0.00 |
@@ -142,12 +140,12 @@ training entirely; chain accuracy on them matches the training primes.
 
 Single-step examples `(t, bit, b, p) → (2t + bit·b) mod p` over each tier's prime range; BCE
 per state bit, AdamW + cosine decay + EMA, checkpointed by full-chain accuracy on held-out
-primes. The 64-bit cell adds a second fine-tuning phase whose single steps are drawn from the
+primes. The TCN cells (64/128/256/512-bit) are trained on single steps drawn from the
 **true Horner trajectory** — `t` is an actual chain intermediate `(a_{≥i}·b) mod p`, not a
 uniform sample — matching the training distribution to the states the chain visits at
-inference. This lifts tier 5 from 0.74 to 0.98 with no capacity change and no backprop through
-the recurrence (ordinary supervised BCE on the same single-step target). The 128-bit (tier-6)
-cell is trained the same single-step way but as the **carry-aware TCN** over a high-diversity
+inference (ordinary supervised BCE on the same single-step target, no backprop through the
+recurrence). The 128-bit (tier-6)
+cell is the first **carry-aware TCN**, trained over a high-diversity
 pool of thousands of distinct 124–128 bit primes; its weight-shared dilated-convolution bias
 reaches a per-step error ~15× lower than the same-task MLP, giving **tier 6 = 0.97** in a single
 short run. The 256-bit (tier-7) cell is the same carry-aware TCN scaled to 256 bit-positions

manifest.json

@@ -2,6 +2,6 @@
   "entry_class": "model.HornerRNN",
   "output_base": 2,
   "framework": "pytorch",
-  "model_description": "Bit-sequential RNN (~415M params across six cells) for primes up to 2^512. Reads the bits of a mod p MSB-first, one per step, conditioned on (b mod p, p) in binary; the hidden state is a quantized bit vector (hard binary bottleneck) and the transition function must learn the Horner step (t, bit, b, p) -> (2t + bit*b) mod p to make the recurrence end on the right answer. Six cells are shipped and routed by prime size: a 16-bit cell (MLP, width 4096 depth 4, ~50M params) for p < 2^16 covering tiers 1-3, a 32-bit cell (MLP, width 6144 depth 4, ~114M params) for p < 2^32 covering tier 4, a 64-bit cell (MLP, width 4096 depth 7 with pre-norm residual blocks, ~236M params) for p < 2^64 covering tier 5, a 128-bit cell for p < 2^128 covering tier 6 that is a CARRY-AWARE TCN: a non-causal dilated 1D-convolutional network over the 128 bit-positions (10 residual blocks, 256 channels, dilations cycling 1..64 so the receptive field spans all 128 bits, ~3.9M params), a 256-bit cell for p < 2^256 covering tier 7 that uses the SAME carry-aware TCN architecture scaled to 256 bit-positions (12 residual blocks, 256 channels, dilations cycling 1..128, ~4.7M params) reaching tier 7 = 0.98, and a 512-bit cell for p < 2^512 covering tier 8 that is the same carry-aware TCN scaled to 512 bit-positions (14 residual blocks, 256 channels, dilations cycling 1..256, ~5.5M params) reaching tier 8 = 0.92. The per-step error floor rises with bit-width, so this cell was trained with gradient accumulation (a large effective batch lowers the per-step error noise floor) to recover the precision a 512-step chain needs to clear 0.90. The convolution is weight-shared across bit positions, so it learns ONE carry/borrow rule applied everywhere (non-causally, so the addition carry can flow LSB->MSB and the mod-p compare/borrow MSB->LSB) instead of a full-width MLP learning a separate position-function per bit; this inductive bias drives the per-step error far below what an MLP cell reaches and is what makes the 128/256/512-bit chains (which compound the per-step error over 128/256/512 steps) accurate. Final state bits are emitted MSB-first as the base-2 answer. For p >= 2^512 emits the honest [0] fallback without invoking the network.",
-  "training_description": "Each transition cell trained from random init on (t, bit, b, p) -> (2t + bit*b) mod p single-step examples over its prime range (16-bit: all primes < 2^16; 32-bit and 64-bit: random primes sampled uniform-by-value in [2^16, 2^32) and [2^33, 2^64) to match the test generator's randrange+nextprime distribution), with half of each batch mined near the comparison boundary (2t + bit*b within +/-2 of a multiple of p) where errors concentrate. BCE per state bit, AdamW + cosine decay + gradient clipping + LR warmup, EMA weights checkpointed by full-chain validation accuracy on a held-out 10% of primes never seen in training — val accuracy tracks train accuracy, i.e. the cells generalise across primes rather than memorising them. The 64-bit cell additionally receives a second fine-tuning phase on single steps drawn from the TRUE Horner trajectory (each example is a (t, bit, b, p) -> (2t + bit*b) mod p step where t is an actual chain intermediate (a_{>=i}*b) mod p, not a uniform sample), which matches the training distribution to the states the chain visits at inference and lifts tier 5 from 0.74 to 0.98; still ordinary supervised BCE on the same single-step target, no backprop through the recurrence. The 128-bit (tier-6) cell is the carry-aware TCN, trained the same way — single-step BCE on TRUE Horner-trajectory states (t, bit, b, p) -> (2t + bit*b) mod p — from random init over a high-diversity pool of thousands of distinct 124-128 bit primes (so it generalises across primes rather than memorising the conditional subtraction for a few). Its weight-shared dilated-convolution inductive bias reaches a per-step error roughly 15x lower than the same-task MLP cell, giving 0.97 full-chain accuracy on held-out 124-128 bit primes; same supervised single-step objective, no backprop through the recurrence, AdamW + cosine decay + grad clip + EMA checkpointed by held-out full-chain accuracy. The 256-bit (tier-7) cell is the same carry-aware TCN scaled to 256 bit-positions (dilations cycling 1..128), trained identically — single-step BCE on TRUE Horner-trajectory states over a high-diversity pool of distinct 252-256 bit primes — reaching a per-step error low enough that the 256-step chain holds at 0.98 full-chain accuracy on held-out 252-256 bit primes. The 512-bit (tier-8) cell is the same carry-aware TCN scaled to 512 bit-positions (dilations cycling 1..256), trained on true-trajectory single steps over distinct 510-512 bit primes; the per-step error floor rises with width, so this cell additionally uses gradient accumulation (--accum: a larger effective batch lowers the gradient-noise floor on per-step error) to drive the 512-step chain to tier 8 = 0.92. Weight-perturbation compliance (exploration/compliance_perturb.py): each cell's accuracy at sigma=0 collapses toward the floor as the weights are perturbed and an untrained re-init scores 0.00 — e.g. tier 6 0.97 -> 0.19 (sigma=0.25), tier 7 0.98 -> 0.06 (sigma=0.25), tier 8 0.92 -> 0.04 (sigma=0.25), untrained 0.00 for all — so the arithmetic resides in the trained parameters. Training scripts: train.py (16-bit), exploration/train_horner32.py (32-bit), exploration/train_horner64.py (64-bit phase 1, --residual) then exploration/train_horner64_traj.py (64-bit phase 2, trajectory), exploration/train_horner128_bigru.py --arch tcn (128-bit carry-aware TCN), exploration/train_horner_tcn.py --bits 256 / --bits 512 --accum 2 (256- and 512-bit carry-aware TCN)."
+  "model_description": "Bit-sequential RNN (~181M params across six cells) for primes up to 2^512. Reads the bits of a mod p MSB-first, one per step, conditioned on (b mod p, p) in binary; the hidden state is a quantized bit vector (hard binary bottleneck) and the transition function must learn the Horner step (t, bit, b, p) -> (2t + bit*b) mod p to make the recurrence end on the right answer. Six cells are shipped and routed by prime size: a 16-bit cell (MLP, width 4096 depth 4, ~50M params) for p < 2^16 covering tiers 1-3, a 32-bit cell (MLP, width 6144 depth 4, ~114M params) for p < 2^32 covering tier 4, a 64-bit cell for p < 2^64 covering tier 5 that is a CARRY-AWARE TCN (8 residual blocks, 256 channels, dilations cycling 1..32, ~3.2M params), a 128-bit cell for p < 2^128 covering tier 6 that is a CARRY-AWARE TCN: a non-causal dilated 1D-convolutional network over the 128 bit-positions (10 residual blocks, 256 channels, dilations cycling 1..64 so the receptive field spans all 128 bits, ~3.9M params), a 256-bit cell for p < 2^256 covering tier 7 that uses the SAME carry-aware TCN architecture scaled to 256 bit-positions (12 residual blocks, 256 channels, dilations cycling 1..128, ~4.7M params) reaching tier 7 = 0.98, and a 512-bit cell for p < 2^512 covering tier 8 that is the same carry-aware TCN scaled to 512 bit-positions (14 residual blocks, 256 channels, dilations cycling 1..256, ~5.5M params) reaching tier 8 = 0.92. The per-step error floor rises with bit-width, so this cell was trained with gradient accumulation (a large effective batch lowers the per-step error noise floor) to recover the precision a 512-step chain needs to clear 0.90. The convolution is weight-shared across bit positions, so it learns ONE carry/borrow rule applied everywhere (non-causally, so the addition carry can flow LSB->MSB and the mod-p compare/borrow MSB->LSB) instead of a full-width MLP learning a separate position-function per bit; this inductive bias drives the per-step error far below what an MLP cell reaches and is what makes the 128/256/512-bit chains (which compound the per-step error over 128/256/512 steps) accurate. Final state bits are emitted MSB-first as the base-2 answer. For p >= 2^512 emits the honest [0] fallback without invoking the network.",
+  "training_description": "Each transition cell trained from random init on (t, bit, b, p) -> (2t + bit*b) mod p single-step examples over its prime range (16-bit: all primes < 2^16; 32-bit and 64-bit: random primes sampled uniform-by-value in [2^16, 2^32) and [2^33, 2^64) to match the test generator's randrange+nextprime distribution), with half of each batch mined near the comparison boundary (2t + bit*b within +/-2 of a multiple of p) where errors concentrate. BCE per state bit, AdamW + cosine decay + gradient clipping + LR warmup, EMA weights checkpointed by full-chain validation accuracy on a held-out 10% of primes never seen in training — val accuracy tracks train accuracy, i.e. the cells generalise across primes rather than memorising them. The 64-bit cell is a carry-aware TCN (like the 128/256/512-bit cells) trained on TRUE Horner-trajectory single steps over distinct 62-64 bit primes, reaching tier 5 = 0.99. It replaced an earlier 944MB MLP cell that also scored ~0.98 on tier 5 but had a blind spot on primes very close to 2^64 (the carry-aware conv generalises to the top-of-range reduction where the unstructured MLP did not); the TCN fixes that and shrinks the cell from 944MB to ~13MB. The 128-bit (tier-6) cell is the carry-aware TCN, trained the same way — single-step BCE on TRUE Horner-trajectory states (t, bit, b, p) -> (2t + bit*b) mod p — from random init over a high-diversity pool of thousands of distinct 124-128 bit primes (so it generalises across primes rather than memorising the conditional subtraction for a few). Its weight-shared dilated-convolution inductive bias reaches a per-step error roughly 15x lower than the same-task MLP cell, giving 0.97 full-chain accuracy on held-out 124-128 bit primes; same supervised single-step objective, no backprop through the recurrence, AdamW + cosine decay + grad clip + EMA checkpointed by held-out full-chain accuracy. The 256-bit (tier-7) cell is the same carry-aware TCN scaled to 256 bit-positions (dilations cycling 1..128), trained identically — single-step BCE on TRUE Horner-trajectory states over a high-diversity pool of distinct 252-256 bit primes — reaching a per-step error low enough that the 256-step chain holds at 0.98 full-chain accuracy on held-out 252-256 bit primes. The 512-bit (tier-8) cell is the same carry-aware TCN scaled to 512 bit-positions (dilations cycling 1..256), trained on true-trajectory single steps over distinct 510-512 bit primes; the per-step error floor rises with width, so this cell additionally uses gradient accumulation (--accum: a larger effective batch lowers the gradient-noise floor on per-step error) to drive the 512-step chain to tier 8 = 0.92. Weight-perturbation compliance (exploration/compliance_perturb.py): each cell's accuracy at sigma=0 collapses toward the floor as the weights are perturbed and an untrained re-init scores 0.00 — e.g. tier 6 0.97 -> 0.19 (sigma=0.25), tier 7 0.98 -> 0.06 (sigma=0.25), tier 8 0.92 -> 0.04 (sigma=0.25), untrained 0.00 for all — so the arithmetic resides in the trained parameters. Training scripts: train.py (16-bit), exploration/train_horner32.py (32-bit), exploration/train_horner128_bigru.py --arch tcn (128-bit carry-aware TCN), exploration/train_horner_tcn.py --bits 64 / --bits 256 / --bits 512 --accum 2 (64-, 256- and 512-bit carry-aware TCN)."
 }

model.py

@@ -3,9 +3,9 @@
 Architecture: a recurrent network that reads the bits of ``a mod p`` MSB-first,
 one per step, conditioned on ``(b mod p, p)`` in binary. The hidden state is a
 quantized bit vector (a discrete bottleneck — a hard VQ layer with a fixed
-binary codebook), and the transition function — an MLP for the 16/32/64-bit
-cells, a weight-shared carry-aware dilated-conv TCN (TCNHornerCell) for the
-128/256/512-bit cells — is entirely trained parameters. After the last bit,
+binary codebook), and the transition function — an MLP for the 16/32-bit cells,
+a weight-shared carry-aware dilated-conv TCN (TCNHornerCell) for the
+64/128/256/512-bit cells — is entirely trained parameters. After the last bit,
 the hidden state bits ARE the answer, emitted MSB-first in base 2.
 
 Why this is interesting: for the recurrence to end on the right answer, the

weights64.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:710c09fe90f339cdf251b0656a465d576e4154a9ce6b1cfa6b5e3c87d3836668
-size 944226321
+oid sha256:ab53ab2f43de2f2f9e8381ca41a4fc475c6565fdeeb4ba46f6a5e045c0eeb178
+size 12640471