AbstractPhil commited on
Commit
ffe4d20
·
verified ·
1 Parent(s): b2ff15e

Add experiment: 2026-11-15_decay_arms

Browse files
2026-11-15_decay_arms/README.md ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Decay arms on a rebuilt recurrent CIFAR-10 bed: the decay read is invalidated as registered; the registered separation reads gating as the payer
2
+
3
+ Date: 2026-11-15. Outcome: a recurrent picture classifier from an August 2026 experiment of the program was rebuilt and reproduced its published numbers. Its recurrent cells were then swapped for a simple linear state cell in three forms: no decay, a fixed decay, and a learned decay that depends on the input. The learned decay reads about 0.11 higher in test accuracy than no decay on both seeds, about 12 times the noise bar. But the rule written down before the run also asks two different no-decay controls to agree, and on one seed they disagree by more than the bar, so the registered label of the decay read is **INVALIDATED (the two no-decay controls disagree beyond the bar)**. A second registered reading separates forgetting from gating: the fixed decay buys nothing over no decay, and the learned decay is far above the fixed one, so that reading is labeled **GATING IS THE PAYER**.
4
+
5
+ This check was carried out as a collaboration within the AbstractPhil research program.
6
+
7
+ ## 1. What was asked
8
+
9
+ The program's memory layer keeps an additive running state with no forgetting (a decay of exactly 1). Whether a learned decay helps had never been settled. An earlier experiment built two small beds for the question on CIFAR-10. The first put every recurrent arm below a plain linear probe on the pixels, so it could not judge anything. The second, a ReNet-style construct with GRU cells, cleared the probe by a wide margin but was never given a decay arm.
10
+
11
+ The question here: on that second bed, with its recurrent cells replaced by a state cell that either keeps everything, forgets on a fixed schedule, or forgets by an amount it reads from the input, does decay change test accuracy, and if so, is it the forgetting or the input-dependent gating that matters?
12
+
13
+ ## 2. The bed
14
+
15
+ | piece | value |
16
+ |---|---|
17
+ | data | CIFAR-10 from the official archive (checksums verified); 10,000 training pictures held out for validation by the original fixed permutation; the 10,000 test pictures |
18
+ | construct | 3 layers; each cuts its input into 2 x 2 patches and sweeps the grid four ways (down, up, right, left) with one recurrent cell per sweep, so 12 recurrent cells in all; then a 256-wide layer with ReLU, dropout 0.2, and 10 classes |
19
+ | training | 4,000 steps, batch 256, Adam with learning rate 0.002 and no weight decay, gradient clipping at 1.0, input masking 0.2, augmentation on; evaluation every 400 steps |
20
+ | the read | test accuracy at the step where validation accuracy is highest (the final-step test accuracy is kept as information only) |
21
+ | seeds | each run is seeded from a checksum of its name, exactly as the original did, so the probe and GRU runs replay the original seeds |
22
+ | hardware | one RTX 4090 |
23
+
24
+ ## 3. The three cell forms
25
+
26
+ Every decay arm uses the same cell in all 12 positions. For each channel, with input x_t at sweep position t:
27
+
28
+ z_t = W x_t + b
29
+ out_t = h_t / (n_t + 1e-6)
30
+
31
+ no decay: h_t = h_(t-1) + z_t n_t = n_(t-1) + 1
32
+ fixed decay: h_t = g_i * h_(t-1) + z_t n_t = g_i * n_(t-1) + 1
33
+ g_i = 1 - 2^-(5 + 7 i / (d - 1)) for channel i of d (0.96875 up to 0.999756; not trained)
34
+ learned decay: h_t = lam_t * h_(t-1) + z_t n_t = lam_t * n_(t-1) + 1
35
+ lam_t = exp( -softplus(A) * softplus(U x_t + c) ) (A, U, c trained; born just under 1)
36
+
37
+ With no decay the cell is a running mean along the sweep: a linear map. The fixed decay is still linear; it only weights recent positions more. The learned decay depends on the input, so it multiplies the state by an amount computed from the input: it adds a gate (a nonlinearity) as well as forgetting.
38
+
39
+ ## 4. The arms
40
+
41
+ | arm | what it is | parameters (total) | parameters (trained) |
42
+ |---|---|---|---|
43
+ | probe | a linear map from the pixels to the classes (the floor) | 30,730 | 30,730 |
44
+ | nodecay | no-decay cell in all 12 positions (the reference); the decay parameters exist but are frozen | 1,397,386 | 1,109,226 |
45
+ | lin | learned-decay cell | 1,397,386 | 1,397,386 |
46
+ | frozen | fixed-decay cell | 1,397,386 | 1,109,226 |
47
+ | nodecay_wide | no-decay cell, widened until its trained count matches the learned-decay arm (the second control) | 1,813,002 | 1,398,858 |
48
+ | renet | the original GRU construct, unchanged (the replication) | 1,364,362 | 1,364,362 |
49
+
50
+ The decay arms are matched in width to the GRU construct's 1,364,362 parameters (nearest step of the construct's 3:4:5 layer shape; +2.4%).
51
+
52
+ ## 5. The readings, written down before the run
53
+
54
+ | reading | rule |
55
+ |---|---|
56
+ | noise bar | the larger of two differences among the no-decay runs: nodecay seed 0 against the run named "repeat", and nodecay seed 0 against seed 1 |
57
+ | floor gate | no decay must beat the probe by more than the bar on both seeds, else no verdict |
58
+ | decay read | learned decay minus no decay: above the bar on both seeds = decay wins, below minus the bar on both = decay loses, else inconclusive |
59
+ | two-control check | the decay read is made against both no-decay controls on each seed; if on any seed the two deltas differ by more than the bar, the label is INVALIDATED (not refuted), and it replaces win, lose or inconclusive |
60
+ | separation (read only if the floor gate passes) | "forgetting alone pays" if fixed minus no decay is above the bar on both seeds; "gating is the payer" if learned minus fixed is above the bar on both seeds while fixed minus no decay is not; else "not separated" |
61
+ | replication (information) | the GRU construct must beat the probe by more than its own bar on both seeds; its numbers are set beside the original's |
62
+
63
+ **About the "repeat":** because every seed comes from the run's name, the run named "repeat" is not a same-seed rerun. It is a third seed. The bar is therefore the larger of two seed differences, not a measure of run-to-run noise at a fixed seed. The rule was kept as the original wrote it so the bar compares with the original's.
64
+
65
+ ## 6. Every run
66
+
67
+ | run | seed | test accuracy (validation-selected step) | at step | validation accuracy there | final-step test (information) | seconds |
68
+ |---|---|---|---|---|---|---|
69
+ | probe | 0 | 0.3531 | 2800 | 0.3453 | 0.3111 | 22.3 |
70
+ | probe | 1 | 0.3304 | 1600 | 0.3368 | 0.3314 | 19.4 |
71
+ | nodecay | 0 | 0.4405 | 3600 | 0.4347 | 0.4417 | 334.2 |
72
+ | nodecay | 1 | 0.4428 | 3600 | 0.4324 | 0.4376 | 315.7 |
73
+ | nodecay | "repeat" (a third seed) | 0.4317 | 3600 | 0.4299 | 0.4310 | 313.7 |
74
+ | lin | 0 | 0.5478 | 4000 | 0.5499 | 0.5478 | 668.3 |
75
+ | lin | 1 | 0.5541 | 3600 | 0.5566 | 0.5347 | 646.7 |
76
+ | frozen | 0 | 0.4385 | 4000 | 0.4353 | 0.4385 | 294.6 |
77
+ | frozen | 1 | 0.4348 | 3200 | 0.4297 | 0.4281 | 241.6 |
78
+ | nodecay_wide | 0 | 0.4241 | 4000 | 0.4263 | 0.4241 | 200.3 |
79
+ | nodecay_wide | 1 | 0.4391 | 3600 | 0.4387 | 0.4214 | 196.3 |
80
+ | renet | 0 | 0.7233 | 3200 | 0.7344 | 0.7068 | 385.5 |
81
+ | renet | 1 | 0.7197 | 4000 | 0.7321 | 0.7197 | 375.5 |
82
+ | renet | "repeat" (a third seed) | 0.7221 | 4000 | 0.7286 | 0.7221 | 376.9 |
83
+
84
+ ## 7. The replication
85
+
86
+ | run | the original | this run | difference |
87
+ |---|---|---|---|
88
+ | renet, seed 0 | 0.7275 | 0.7233 | -0.0042 |
89
+ | renet, "repeat" | 0.7217 | 0.7221 | +0.0004 |
90
+ | renet, seed 1 | 0.7162 | 0.7197 | +0.0035 |
91
+ | probe, seed 0 | 0.3589 | 0.3531 | -0.0058 |
92
+ | probe, seed 1 | 0.3365 | 0.3304 | -0.0061 |
93
+
94
+ Every number lands within 0.0061 of the original. The GRU construct beats the probe by 0.3702 and 0.3893 (its own bar here 0.0036; the original's was 0.0113): go on both seeds.
95
+
96
+ ## 8. The registered readings
97
+
98
+ Bar = 0.0088 (the "repeat" difference 0.0088; the seed difference 0.0023).
99
+
100
+ | reading | seed 0 | seed 1 | in bars (seed 0 / seed 1) | registered label |
101
+ |---|---|---|---|---|
102
+ | floor gate: no decay minus probe | +0.0874 | +0.1124 | 9.9 / 12.8 | pass |
103
+ | learned decay minus no decay | +0.1073 | +0.1113 | 12.2 / 12.6 | |
104
+ | learned decay minus no decay, widened | +0.1237 | +0.1150 | 14.1 / 13.1 | |
105
+ | difference between the two controls' deltas | 0.0164 | 0.0037 | 1.9 / 0.4 | seed 0: "controls DISAGREE beyond the bar -> INVALIDATED, not refuted"; seed 1: "ABOVE the bar on both controls -- decay pays here" |
106
+ | **the decay read** | | | | **INVALIDATED (the two no-decay controls disagree beyond the bar)** |
107
+ | fixed decay minus no decay | -0.0020 | -0.0080 | -0.2 / -0.9 | |
108
+ | learned decay minus fixed decay | +0.1093 | +0.1193 | 12.4 / 13.6 | **GATING IS THE PAYER** |
109
+ | also printed: learned decay minus the GRU construct | -0.1755 | -0.1656 | | |
110
+
111
+ On the final-step test accuracy (information only; its bar is 0.0107) the labels are the same: the two controls disagree beyond the bar on both seeds there, and the separation again reads gating is the payer.
112
+
113
+ ### Did the learned decay actually move?
114
+
115
+ | run | decay mean | share below 0.99 | share fully erased (below 1e-6) | share still exactly 1 | gradient on the decay parameters at step 1 |
116
+ |---|---|---|---|---|---|
117
+ | lin, seed 0 | 0.846 | 0.545 | 0.021 | 0.011 | 9.2e-06 |
118
+ | lin, seed 1 | 0.858 | 0.530 | 0.022 | 0.012 | 7.3e-06 |
119
+ | frozen | 0.9935 (its schedule) | 0.235 | 0 | 0 | 0 (not trained) |
120
+ | nodecay, nodecay_wide | 1 exactly everywhere | 0 | 0 | 1 | 0 (not trained) |
121
+
122
+ Read over all 12 cells on 256 test pictures. The learned decay was born just under 1, received a live gradient from the first step, and ended spread widely below 1. It did not stay stuck at 1.
123
+
124
+ ## 9. What this shows
125
+
126
+ The chain of cause and effect:
127
+
128
+ 1. With no decay the cell is a running mean, a linear map, so the no-decay arm's recurrent layers are linear in the pixels; only the 256-wide ReLU layer bends.
129
+ 2. A fixed decay keeps the cell linear and only weights recent positions more. Fixed decay against no decay: -0.0020 and -0.0080, inside the bar. Forgetting on a fixed schedule buys nothing here.
130
+ 3. The learned decay depends on the input, so it adds a multiplicative gate as well as forgetting. Learned against fixed: +0.1093 and +0.1193. Registered label: GATING IS THE PAYER.
131
+ 4. Against no decay the learned arm is far above the bar on both seeds, but the two no-decay controls disagree by 0.0164 on seed 0, beyond the bar of 0.0088. Registered label: INVALIDATED. This says the pair of controls is not consistent enough for the original rule to name a winner. It does not say that decay failed.
132
+
133
+ In plain terms, set apart from the registered labels: the learned decay's margin over no decay is about 12 bars on each seed, while the disagreement between the controls is about 1.9 bars on seed 0 and 0.4 on seed 1, so the margin is about 6.5 times the disagreement even at its smallest. The disagreement also runs against a capacity explanation: the widened control, with more trained parameters, reads lower than the narrow one on both seeds. One more observation: the bar's rule uses two of the three possible differences among the no-decay runs; the third (seed 1 against the "repeat") is 0.0111, larger than the bar, and the seed-0 disagreement of 0.0164 is larger still. The registered label stands as registered.
134
+
135
+ ## 10. What this does NOT show
136
+
137
+ - It is one bed (this GRU ReNet construct), one dataset (CIFAR-10), and 4,000 steps.
138
+ - The decay arms use a linear state cell. The learned-decay arm reads about 0.17 below the original GRU cell, and the no-decay and fixed-decay arms about 0.28 below. This is a comparison among weak cells, not a statement about a strong one.
139
+ - There are two seeds per arm (three for the no-decay reference). The two-control disagreement could shrink or grow with more seeds.
140
+ - The registered decay label is INVALIDATED. It is not a win, and it is not a refutation.
141
+ - "Gating is the payer" is measured on a linear cell. It does not carry over by itself to a cell that is already gated (such as a GRU), or to the program's memory layer, without a run of its own.
142
+ - The fixed decay is one schedule (a spread from 0.96875 to 0.999756 across channels), not a sweep of schedules.
143
+ - The decay census is one batch of 256 test pictures.
144
+
145
+ ## 11. Cost and how to reproduce
146
+
147
+ The 14 runs took 4,391 seconds (73.2 minutes) on one RTX 4090; the learned-decay runs took about twice as long as the no-decay runs (1,315 s against 650 s for a pair); the peak card memory of any run was 0.83 GB. In pseudocode:
148
+
149
+ data = CIFAR-10; hold out 10,000 training pictures by a fixed permutation (seed 1234) for validation
150
+ for each run name in [probe_s0, probe_s1, nodecay_s0, nodecay_s1, nodecay_s0_rep, lin_s0, lin_s1, frozen_s0, frozen_s1,
151
+ nodecay_wide_s0, nodecay_wide_s1, renet_s0, renet_s1, renet_s0_rep]:
152
+ seed = crc32(run name)
153
+ model = probe, or the 3-layer four-way-sweep construct with the arm's cell in all 12 positions, width-matched
154
+ train 4,000 steps, batch 256, Adam lr 0.002, clip 1.0; every 400 steps read validation and test accuracy
155
+ record test accuracy at the step of best validation accuracy
156
+ bar = max(|nodecay_s0 - nodecay_s0_rep|, |nodecay_s0 - nodecay_s1|)
157
+ apply the rules of section 5 per seed
158
+
159
+ `results.json` in this folder holds every run (its parameters, the accuracy history at every evaluation, the decay census at every cell, the step-1 gradients), the widths, the seconds per run, and both summaries (the validation-selected read and the final-step read). It is the lab's raw result file with only internal reference strings replaced.
2026-11-15_decay_arms/results.json ADDED
@@ -0,0 +1,1880 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "source": "an August 2026 experiment of the program: its GRU ReNet bed on CIFAR-10 rebuilt, plus linear-state-cell decay arms; see README.md",
3
+ "smoke": false,
4
+ "cfg": {
5
+ "arms": [
6
+ "probe",
7
+ "nodecay",
8
+ "lin",
9
+ "frozen",
10
+ "nodecay_wide",
11
+ "renet"
12
+ ],
13
+ "reference_arm": "nodecay",
14
+ "replication_arm": "renet",
15
+ "seeds": [
16
+ 0,
17
+ 1
18
+ ],
19
+ "repeat_reference": true,
20
+ "renet_dims": [
21
+ 48,
22
+ 64,
23
+ 80
24
+ ],
25
+ "patch": 2,
26
+ "d_fc": 256,
27
+ "dropout": 0.2,
28
+ "input_mask": 0.2,
29
+ "val_n": 10000,
30
+ "augment": true,
31
+ "steps": 4000,
32
+ "batch": 256,
33
+ "lr": 0.002,
34
+ "grad_clip": 1.0,
35
+ "eval_every": 400,
36
+ "eval_batch": 1000,
37
+ "decay_normalizer": "both",
38
+ "lambda_floor": null
39
+ },
40
+ "units_done": [
41
+ "probe_s0",
42
+ "probe_s1",
43
+ "nodecay_s0",
44
+ "nodecay_s1",
45
+ "nodecay_s0_rep",
46
+ "lin_s0",
47
+ "lin_s1",
48
+ "frozen_s0",
49
+ "frozen_s1",
50
+ "nodecay_wide_s0",
51
+ "nodecay_wide_s1",
52
+ "renet_s0",
53
+ "renet_s1",
54
+ "renet_s0_rep"
55
+ ],
56
+ "runs": {
57
+ "probe_s0": {
58
+ "run": "probe_s0",
59
+ "arm": "probe",
60
+ "seed": 0,
61
+ "suffix": "",
62
+ "torch_seed": 1627808380,
63
+ "params": 30730,
64
+ "params_active": 30730,
65
+ "unit": null,
66
+ "dims": null,
67
+ "val_selected_step": 2800,
68
+ "val_acc": 0.3453,
69
+ "test_acc": 0.3531,
70
+ "final_test": 0.3111,
71
+ "final_val": 0.3028,
72
+ "final_train_loss": 2.099102020263672,
73
+ "step1_grads": {
74
+ "decay": 0.0,
75
+ "rest": 6.0107776923071,
76
+ "ratio": 0.0,
77
+ "n_decay_tensors": 0
78
+ },
79
+ "lambda": null,
80
+ "wall_s": 18.7,
81
+ "history": [
82
+ {
83
+ "step": 400,
84
+ "loss": 2.279189348220825,
85
+ "val": 0.3149,
86
+ "test": 0.3223
87
+ },
88
+ {
89
+ "step": 800,
90
+ "loss": 2.1899023056030273,
91
+ "val": 0.3257,
92
+ "test": 0.3296
93
+ },
94
+ {
95
+ "step": 1200,
96
+ "loss": 2.190023422241211,
97
+ "val": 0.3346,
98
+ "test": 0.3417
99
+ },
100
+ {
101
+ "step": 1600,
102
+ "loss": 2.0192956924438477,
103
+ "val": 0.3245,
104
+ "test": 0.3282
105
+ },
106
+ {
107
+ "step": 2000,
108
+ "loss": 2.185375452041626,
109
+ "val": 0.3286,
110
+ "test": 0.3291
111
+ },
112
+ {
113
+ "step": 2400,
114
+ "loss": 2.2554547786712646,
115
+ "val": 0.324,
116
+ "test": 0.3251
117
+ },
118
+ {
119
+ "step": 2800,
120
+ "loss": 2.252396583557129,
121
+ "val": 0.3453,
122
+ "test": 0.3531
123
+ },
124
+ {
125
+ "step": 3200,
126
+ "loss": 2.132601737976074,
127
+ "val": 0.3346,
128
+ "test": 0.3273
129
+ },
130
+ {
131
+ "step": 3600,
132
+ "loss": 2.132026195526123,
133
+ "val": 0.3424,
134
+ "test": 0.3511
135
+ },
136
+ {
137
+ "step": 4000,
138
+ "loss": 2.099102020263672,
139
+ "val": 0.3028,
140
+ "test": 0.3111
141
+ }
142
+ ],
143
+ "device": "cuda",
144
+ "torch": "2.11.0+cu128",
145
+ "peak_card_bytes": 75721216,
146
+ "checkpoint": "probe_s0_steps4000_b256.pt"
147
+ },
148
+ "probe_s1": {
149
+ "run": "probe_s1",
150
+ "arm": "probe",
151
+ "seed": 1,
152
+ "suffix": "",
153
+ "torch_seed": 369185514,
154
+ "params": 30730,
155
+ "params_active": 30730,
156
+ "unit": null,
157
+ "dims": null,
158
+ "val_selected_step": 1600,
159
+ "val_acc": 0.3368,
160
+ "test_acc": 0.3304,
161
+ "final_test": 0.3314,
162
+ "final_val": 0.3214,
163
+ "final_train_loss": 2.2133331298828125,
164
+ "step1_grads": {
165
+ "decay": 0.0,
166
+ "rest": 5.291671433347546,
167
+ "ratio": 0.0,
168
+ "n_decay_tensors": 0
169
+ },
170
+ "lambda": null,
171
+ "wall_s": 19.4,
172
+ "history": [
173
+ {
174
+ "step": 400,
175
+ "loss": 2.1652724742889404,
176
+ "val": 0.3069,
177
+ "test": 0.3093
178
+ },
179
+ {
180
+ "step": 800,
181
+ "loss": 2.267091989517212,
182
+ "val": 0.3173,
183
+ "test": 0.3191
184
+ },
185
+ {
186
+ "step": 1200,
187
+ "loss": 2.233362913131714,
188
+ "val": 0.3274,
189
+ "test": 0.3362
190
+ },
191
+ {
192
+ "step": 1600,
193
+ "loss": 2.39182186126709,
194
+ "val": 0.3368,
195
+ "test": 0.3304
196
+ },
197
+ {
198
+ "step": 2000,
199
+ "loss": 2.095400333404541,
200
+ "val": 0.3183,
201
+ "test": 0.3359
202
+ },
203
+ {
204
+ "step": 2400,
205
+ "loss": 2.236255168914795,
206
+ "val": 0.3123,
207
+ "test": 0.3238
208
+ },
209
+ {
210
+ "step": 2800,
211
+ "loss": 2.286074161529541,
212
+ "val": 0.322,
213
+ "test": 0.3221
214
+ },
215
+ {
216
+ "step": 3200,
217
+ "loss": 2.1576802730560303,
218
+ "val": 0.305,
219
+ "test": 0.3017
220
+ },
221
+ {
222
+ "step": 3600,
223
+ "loss": 2.168860912322998,
224
+ "val": 0.3293,
225
+ "test": 0.3406
226
+ },
227
+ {
228
+ "step": 4000,
229
+ "loss": 2.2133331298828125,
230
+ "val": 0.3214,
231
+ "test": 0.3314
232
+ }
233
+ ],
234
+ "device": "cuda",
235
+ "torch": "2.11.0+cu128",
236
+ "peak_card_bytes": 75721216,
237
+ "checkpoint": "probe_s1_steps4000_b256.pt"
238
+ },
239
+ "nodecay_s0": {
240
+ "run": "nodecay_s0",
241
+ "arm": "nodecay",
242
+ "seed": 0,
243
+ "suffix": "",
244
+ "torch_seed": 1535074504,
245
+ "params": 1397386,
246
+ "params_active": 1109226,
247
+ "unit": 20,
248
+ "dims": [
249
+ 60,
250
+ 80,
251
+ 100
252
+ ],
253
+ "val_selected_step": 3600,
254
+ "val_acc": 0.4347,
255
+ "test_acc": 0.4405,
256
+ "final_test": 0.4417,
257
+ "final_val": 0.4346,
258
+ "final_train_loss": 1.80162513256073,
259
+ "step1_grads": {
260
+ "decay": 0.0,
261
+ "rest": 0.16399605515545906,
262
+ "ratio": 0.0,
263
+ "n_decay_tensors": 0
264
+ },
265
+ "lambda": {
266
+ "mean": 1.0,
267
+ "min": 1.0,
268
+ "max": 1.0,
269
+ "frac_below_0.99": 0.0,
270
+ "frac_erased_lt_1e-6": 0.0,
271
+ "frac_saturated_eq_1": 1.0,
272
+ "exactly_one_everywhere": true,
273
+ "per_cell_mean": {
274
+ "layers.0.v_f": 1.0,
275
+ "layers.0.v_r": 1.0,
276
+ "layers.0.h_f": 1.0,
277
+ "layers.0.h_r": 1.0,
278
+ "layers.1.v_f": 1.0,
279
+ "layers.1.v_r": 1.0,
280
+ "layers.1.h_f": 1.0,
281
+ "layers.1.h_r": 1.0,
282
+ "layers.2.v_f": 1.0,
283
+ "layers.2.v_r": 1.0,
284
+ "layers.2.h_f": 1.0,
285
+ "layers.2.h_r": 1.0
286
+ }
287
+ },
288
+ "wall_s": 334.1,
289
+ "history": [
290
+ {
291
+ "step": 400,
292
+ "loss": 1.9047083854675293,
293
+ "val": 0.3356,
294
+ "test": 0.3387
295
+ },
296
+ {
297
+ "step": 800,
298
+ "loss": 1.928490400314331,
299
+ "val": 0.3844,
300
+ "test": 0.3899
301
+ },
302
+ {
303
+ "step": 1200,
304
+ "loss": 1.7754831314086914,
305
+ "val": 0.4026,
306
+ "test": 0.4032
307
+ },
308
+ {
309
+ "step": 1600,
310
+ "loss": 1.8473857641220093,
311
+ "val": 0.4131,
312
+ "test": 0.4163
313
+ },
314
+ {
315
+ "step": 2000,
316
+ "loss": 1.767679214477539,
317
+ "val": 0.4005,
318
+ "test": 0.4076
319
+ },
320
+ {
321
+ "step": 2400,
322
+ "loss": 1.595026969909668,
323
+ "val": 0.417,
324
+ "test": 0.4195
325
+ },
326
+ {
327
+ "step": 2800,
328
+ "loss": 1.7287828922271729,
329
+ "val": 0.4278,
330
+ "test": 0.429
331
+ },
332
+ {
333
+ "step": 3200,
334
+ "loss": 1.6169073581695557,
335
+ "val": 0.4274,
336
+ "test": 0.4343
337
+ },
338
+ {
339
+ "step": 3600,
340
+ "loss": 1.6732802391052246,
341
+ "val": 0.4347,
342
+ "test": 0.4405
343
+ },
344
+ {
345
+ "step": 4000,
346
+ "loss": 1.80162513256073,
347
+ "val": 0.4346,
348
+ "test": 0.4417
349
+ }
350
+ ],
351
+ "device": "cuda",
352
+ "torch": "2.11.0+cu128",
353
+ "peak_card_bytes": 701897216,
354
+ "checkpoint": "nodecay_s0_steps4000_b256.pt"
355
+ },
356
+ "nodecay_s1": {
357
+ "run": "nodecay_s1",
358
+ "arm": "nodecay",
359
+ "seed": 1,
360
+ "suffix": "",
361
+ "torch_seed": 746082398,
362
+ "params": 1397386,
363
+ "params_active": 1109226,
364
+ "unit": 20,
365
+ "dims": [
366
+ 60,
367
+ 80,
368
+ 100
369
+ ],
370
+ "val_selected_step": 3600,
371
+ "val_acc": 0.4324,
372
+ "test_acc": 0.4428,
373
+ "final_test": 0.4376,
374
+ "final_val": 0.4295,
375
+ "final_train_loss": 1.8303117752075195,
376
+ "step1_grads": {
377
+ "decay": 0.0,
378
+ "rest": 0.1310279562926789,
379
+ "ratio": 0.0,
380
+ "n_decay_tensors": 0
381
+ },
382
+ "lambda": {
383
+ "mean": 1.0,
384
+ "min": 1.0,
385
+ "max": 1.0,
386
+ "frac_below_0.99": 0.0,
387
+ "frac_erased_lt_1e-6": 0.0,
388
+ "frac_saturated_eq_1": 1.0,
389
+ "exactly_one_everywhere": true,
390
+ "per_cell_mean": {
391
+ "layers.0.v_f": 1.0,
392
+ "layers.0.v_r": 1.0,
393
+ "layers.0.h_f": 1.0,
394
+ "layers.0.h_r": 1.0,
395
+ "layers.1.v_f": 1.0,
396
+ "layers.1.v_r": 1.0,
397
+ "layers.1.h_f": 1.0,
398
+ "layers.1.h_r": 1.0,
399
+ "layers.2.v_f": 1.0,
400
+ "layers.2.v_r": 1.0,
401
+ "layers.2.h_f": 1.0,
402
+ "layers.2.h_r": 1.0
403
+ }
404
+ },
405
+ "wall_s": 315.2,
406
+ "history": [
407
+ {
408
+ "step": 400,
409
+ "loss": 1.872191071510315,
410
+ "val": 0.3508,
411
+ "test": 0.3572
412
+ },
413
+ {
414
+ "step": 800,
415
+ "loss": 1.8747539520263672,
416
+ "val": 0.371,
417
+ "test": 0.3792
418
+ },
419
+ {
420
+ "step": 1200,
421
+ "loss": 1.9443190097808838,
422
+ "val": 0.3915,
423
+ "test": 0.3955
424
+ },
425
+ {
426
+ "step": 1600,
427
+ "loss": 1.7669308185577393,
428
+ "val": 0.4105,
429
+ "test": 0.4163
430
+ },
431
+ {
432
+ "step": 2000,
433
+ "loss": 1.7827770709991455,
434
+ "val": 0.4138,
435
+ "test": 0.4257
436
+ },
437
+ {
438
+ "step": 2400,
439
+ "loss": 1.7405424118041992,
440
+ "val": 0.4134,
441
+ "test": 0.4209
442
+ },
443
+ {
444
+ "step": 2800,
445
+ "loss": 1.7505507469177246,
446
+ "val": 0.4132,
447
+ "test": 0.4234
448
+ },
449
+ {
450
+ "step": 3200,
451
+ "loss": 1.7324721813201904,
452
+ "val": 0.4235,
453
+ "test": 0.4263
454
+ },
455
+ {
456
+ "step": 3600,
457
+ "loss": 1.831796646118164,
458
+ "val": 0.4324,
459
+ "test": 0.4428
460
+ },
461
+ {
462
+ "step": 4000,
463
+ "loss": 1.8303117752075195,
464
+ "val": 0.4295,
465
+ "test": 0.4376
466
+ }
467
+ ],
468
+ "device": "cuda",
469
+ "torch": "2.11.0+cu128",
470
+ "peak_card_bytes": 701897216,
471
+ "checkpoint": "nodecay_s1_steps4000_b256.pt"
472
+ },
473
+ "nodecay_s0_rep": {
474
+ "run": "nodecay_s0_rep",
475
+ "arm": "nodecay",
476
+ "seed": 0,
477
+ "suffix": "_rep",
478
+ "torch_seed": 48263812,
479
+ "params": 1397386,
480
+ "params_active": 1109226,
481
+ "unit": 20,
482
+ "dims": [
483
+ 60,
484
+ 80,
485
+ 100
486
+ ],
487
+ "val_selected_step": 3600,
488
+ "val_acc": 0.4299,
489
+ "test_acc": 0.4317,
490
+ "final_test": 0.431,
491
+ "final_val": 0.4268,
492
+ "final_train_loss": 1.7435688972473145,
493
+ "step1_grads": {
494
+ "decay": 0.0,
495
+ "rest": 0.13261467722565512,
496
+ "ratio": 0.0,
497
+ "n_decay_tensors": 0
498
+ },
499
+ "lambda": {
500
+ "mean": 1.0,
501
+ "min": 1.0,
502
+ "max": 1.0,
503
+ "frac_below_0.99": 0.0,
504
+ "frac_erased_lt_1e-6": 0.0,
505
+ "frac_saturated_eq_1": 1.0,
506
+ "exactly_one_everywhere": true,
507
+ "per_cell_mean": {
508
+ "layers.0.v_f": 1.0,
509
+ "layers.0.v_r": 1.0,
510
+ "layers.0.h_f": 1.0,
511
+ "layers.0.h_r": 1.0,
512
+ "layers.1.v_f": 1.0,
513
+ "layers.1.v_r": 1.0,
514
+ "layers.1.h_f": 1.0,
515
+ "layers.1.h_r": 1.0,
516
+ "layers.2.v_f": 1.0,
517
+ "layers.2.v_r": 1.0,
518
+ "layers.2.h_f": 1.0,
519
+ "layers.2.h_r": 1.0
520
+ }
521
+ },
522
+ "wall_s": 313.6,
523
+ "history": [
524
+ {
525
+ "step": 400,
526
+ "loss": 1.9135406017303467,
527
+ "val": 0.3693,
528
+ "test": 0.3763
529
+ },
530
+ {
531
+ "step": 800,
532
+ "loss": 1.921396017074585,
533
+ "val": 0.3684,
534
+ "test": 0.3816
535
+ },
536
+ {
537
+ "step": 1200,
538
+ "loss": 1.8668534755706787,
539
+ "val": 0.4005,
540
+ "test": 0.4032
541
+ },
542
+ {
543
+ "step": 1600,
544
+ "loss": 1.848726511001587,
545
+ "val": 0.4054,
546
+ "test": 0.4119
547
+ },
548
+ {
549
+ "step": 2000,
550
+ "loss": 1.8661603927612305,
551
+ "val": 0.3927,
552
+ "test": 0.407
553
+ },
554
+ {
555
+ "step": 2400,
556
+ "loss": 1.7571748495101929,
557
+ "val": 0.4141,
558
+ "test": 0.4131
559
+ },
560
+ {
561
+ "step": 2800,
562
+ "loss": 1.7830722332000732,
563
+ "val": 0.4214,
564
+ "test": 0.4241
565
+ },
566
+ {
567
+ "step": 3200,
568
+ "loss": 1.7614030838012695,
569
+ "val": 0.4263,
570
+ "test": 0.4339
571
+ },
572
+ {
573
+ "step": 3600,
574
+ "loss": 1.8259683847427368,
575
+ "val": 0.4299,
576
+ "test": 0.4317
577
+ },
578
+ {
579
+ "step": 4000,
580
+ "loss": 1.7435688972473145,
581
+ "val": 0.4268,
582
+ "test": 0.431
583
+ }
584
+ ],
585
+ "device": "cuda",
586
+ "torch": "2.11.0+cu128",
587
+ "peak_card_bytes": 701897216,
588
+ "checkpoint": "nodecay_s0_rep_steps4000_b256.pt"
589
+ },
590
+ "lin_s0": {
591
+ "run": "lin_s0",
592
+ "arm": "lin",
593
+ "seed": 0,
594
+ "suffix": "",
595
+ "torch_seed": 1141147802,
596
+ "params": 1397386,
597
+ "params_active": 1397386,
598
+ "unit": 20,
599
+ "dims": [
600
+ 60,
601
+ 80,
602
+ 100
603
+ ],
604
+ "val_selected_step": 4000,
605
+ "val_acc": 0.5499,
606
+ "test_acc": 0.5478,
607
+ "final_test": 0.5478,
608
+ "final_val": 0.5499,
609
+ "final_train_loss": 1.3436251878738403,
610
+ "step1_grads": {
611
+ "decay": 9.210208044345453e-06,
612
+ "rest": 0.1190783939660297,
613
+ "ratio": 7.734575297490922e-05,
614
+ "n_decay_tensors": 36
615
+ },
616
+ "lambda": {
617
+ "mean": 0.8463596701622009,
618
+ "min": 0.0,
619
+ "max": 1.0,
620
+ "frac_below_0.99": 0.5445767641067505,
621
+ "frac_erased_lt_1e-6": 0.02085978351533413,
622
+ "frac_saturated_eq_1": 0.011412955820560455,
623
+ "exactly_one_everywhere": false,
624
+ "per_cell_mean": {
625
+ "layers.0.v_f": 0.8456934690475464,
626
+ "layers.0.v_r": 0.8451754450798035,
627
+ "layers.0.h_f": 0.8312440514564514,
628
+ "layers.0.h_r": 0.8000173568725586,
629
+ "layers.1.v_f": 0.864902675151825,
630
+ "layers.1.v_r": 0.8573562502861023,
631
+ "layers.1.h_f": 0.9296594858169556,
632
+ "layers.1.h_r": 0.9079098105430603,
633
+ "layers.2.v_f": 0.8479119539260864,
634
+ "layers.2.v_r": 0.843068540096283,
635
+ "layers.2.h_f": 0.883732259273529,
636
+ "layers.2.h_r": 0.8604412078857422
637
+ }
638
+ },
639
+ "wall_s": 668.1,
640
+ "history": [
641
+ {
642
+ "step": 400,
643
+ "loss": 1.7576031684875488,
644
+ "val": 0.3729,
645
+ "test": 0.3755
646
+ },
647
+ {
648
+ "step": 800,
649
+ "loss": 1.6538734436035156,
650
+ "val": 0.4573,
651
+ "test": 0.4519
652
+ },
653
+ {
654
+ "step": 1200,
655
+ "loss": 1.5675301551818848,
656
+ "val": 0.4656,
657
+ "test": 0.4643
658
+ },
659
+ {
660
+ "step": 1600,
661
+ "loss": 1.5421319007873535,
662
+ "val": 0.5018,
663
+ "test": 0.5025
664
+ },
665
+ {
666
+ "step": 2000,
667
+ "loss": 1.5888330936431885,
668
+ "val": 0.4703,
669
+ "test": 0.4779
670
+ },
671
+ {
672
+ "step": 2400,
673
+ "loss": 1.4077179431915283,
674
+ "val": 0.4984,
675
+ "test": 0.4902
676
+ },
677
+ {
678
+ "step": 2800,
679
+ "loss": 1.5413758754730225,
680
+ "val": 0.5166,
681
+ "test": 0.5147
682
+ },
683
+ {
684
+ "step": 3200,
685
+ "loss": 1.4862645864486694,
686
+ "val": 0.5408,
687
+ "test": 0.5341
688
+ },
689
+ {
690
+ "step": 3600,
691
+ "loss": 1.3861064910888672,
692
+ "val": 0.5481,
693
+ "test": 0.548
694
+ },
695
+ {
696
+ "step": 4000,
697
+ "loss": 1.3436251878738403,
698
+ "val": 0.5499,
699
+ "test": 0.5478
700
+ }
701
+ ],
702
+ "device": "cuda",
703
+ "torch": "2.11.0+cu128",
704
+ "peak_card_bytes": 745470464,
705
+ "checkpoint": "lin_s0_steps4000_b256.pt"
706
+ },
707
+ "lin_s1": {
708
+ "run": "lin_s1",
709
+ "arm": "lin",
710
+ "seed": 1,
711
+ "suffix": "",
712
+ "torch_seed": 855881740,
713
+ "params": 1397386,
714
+ "params_active": 1397386,
715
+ "unit": 20,
716
+ "dims": [
717
+ 60,
718
+ 80,
719
+ 100
720
+ ],
721
+ "val_selected_step": 3600,
722
+ "val_acc": 0.5566,
723
+ "test_acc": 0.5541,
724
+ "final_test": 0.5347,
725
+ "final_val": 0.5343,
726
+ "final_train_loss": 1.3737794160842896,
727
+ "step1_grads": {
728
+ "decay": 7.27021011314642e-06,
729
+ "rest": 0.13034133110766435,
730
+ "ratio": 5.577824049641699e-05,
731
+ "n_decay_tensors": 36
732
+ },
733
+ "lambda": {
734
+ "mean": 0.8578829169273376,
735
+ "min": 0.0,
736
+ "max": 1.0,
737
+ "frac_below_0.99": 0.5296123623847961,
738
+ "frac_erased_lt_1e-6": 0.022028474137187004,
739
+ "frac_saturated_eq_1": 0.012149004265666008,
740
+ "exactly_one_everywhere": false,
741
+ "per_cell_mean": {
742
+ "layers.0.v_f": 0.8650097250938416,
743
+ "layers.0.v_r": 0.8182206153869629,
744
+ "layers.0.h_f": 0.8340145349502563,
745
+ "layers.0.h_r": 0.8495701551437378,
746
+ "layers.1.v_f": 0.8848231434822083,
747
+ "layers.1.v_r": 0.8776430487632751,
748
+ "layers.1.h_f": 0.9392439723014832,
749
+ "layers.1.h_r": 0.9162662625312805,
750
+ "layers.2.v_f": 0.8401320576667786,
751
+ "layers.2.v_r": 0.8596683144569397,
752
+ "layers.2.h_f": 0.8869063854217529,
753
+ "layers.2.h_r": 0.8694813251495361
754
+ }
755
+ },
756
+ "wall_s": 646.7,
757
+ "history": [
758
+ {
759
+ "step": 400,
760
+ "loss": 1.920438528060913,
761
+ "val": 0.3827,
762
+ "test": 0.3826
763
+ },
764
+ {
765
+ "step": 800,
766
+ "loss": 1.6001696586608887,
767
+ "val": 0.4508,
768
+ "test": 0.4488
769
+ },
770
+ {
771
+ "step": 1200,
772
+ "loss": 1.6149137020111084,
773
+ "val": 0.4402,
774
+ "test": 0.4374
775
+ },
776
+ {
777
+ "step": 1600,
778
+ "loss": 1.4953546524047852,
779
+ "val": 0.4766,
780
+ "test": 0.4675
781
+ },
782
+ {
783
+ "step": 2000,
784
+ "loss": 1.4973013401031494,
785
+ "val": 0.5022,
786
+ "test": 0.503
787
+ },
788
+ {
789
+ "step": 2400,
790
+ "loss": 1.4762473106384277,
791
+ "val": 0.5119,
792
+ "test": 0.5101
793
+ },
794
+ {
795
+ "step": 2800,
796
+ "loss": 1.367244005203247,
797
+ "val": 0.5247,
798
+ "test": 0.5184
799
+ },
800
+ {
801
+ "step": 3200,
802
+ "loss": 1.4474241733551025,
803
+ "val": 0.5323,
804
+ "test": 0.5317
805
+ },
806
+ {
807
+ "step": 3600,
808
+ "loss": 1.3756859302520752,
809
+ "val": 0.5566,
810
+ "test": 0.5541
811
+ },
812
+ {
813
+ "step": 4000,
814
+ "loss": 1.3737794160842896,
815
+ "val": 0.5343,
816
+ "test": 0.5347
817
+ }
818
+ ],
819
+ "device": "cuda",
820
+ "torch": "2.11.0+cu128",
821
+ "peak_card_bytes": 745470464,
822
+ "checkpoint": "lin_s1_steps4000_b256.pt"
823
+ },
824
+ "frozen_s0": {
825
+ "run": "frozen_s0",
826
+ "arm": "frozen",
827
+ "seed": 0,
828
+ "suffix": "",
829
+ "torch_seed": 1061315551,
830
+ "params": 1397386,
831
+ "params_active": 1109226,
832
+ "unit": 20,
833
+ "dims": [
834
+ 60,
835
+ 80,
836
+ 100
837
+ ],
838
+ "val_selected_step": 4000,
839
+ "val_acc": 0.4353,
840
+ "test_acc": 0.4385,
841
+ "final_test": 0.4385,
842
+ "final_val": 0.4353,
843
+ "final_train_loss": 1.7407853603363037,
844
+ "step1_grads": {
845
+ "decay": 0.0,
846
+ "rest": 0.15934257899095225,
847
+ "ratio": 0.0,
848
+ "n_decay_tensors": 0
849
+ },
850
+ "lambda": {
851
+ "mean": 0.9934642910957336,
852
+ "min": 0.96875,
853
+ "max": 0.999755859375,
854
+ "frac_below_0.99": 0.2347826212644577,
855
+ "frac_erased_lt_1e-6": 0.0,
856
+ "frac_saturated_eq_1": 0.0,
857
+ "exactly_one_everywhere": false,
858
+ "per_cell_mean": {
859
+ "layers.0.v_f": 0.9934502840042114,
860
+ "layers.0.v_r": 0.9934502840042114,
861
+ "layers.0.h_f": 0.9934502840042114,
862
+ "layers.0.h_r": 0.9934502840042114,
863
+ "layers.1.v_f": 0.9934908151626587,
864
+ "layers.1.v_r": 0.9934908151626587,
865
+ "layers.1.h_f": 0.9934908151626587,
866
+ "layers.1.h_r": 0.9934908151626587,
867
+ "layers.2.v_f": 0.9935148358345032,
868
+ "layers.2.v_r": 0.9935148358345032,
869
+ "layers.2.h_f": 0.9935148358345032,
870
+ "layers.2.h_r": 0.9935148358345032
871
+ }
872
+ },
873
+ "wall_s": 294.5,
874
+ "history": [
875
+ {
876
+ "step": 400,
877
+ "loss": 1.8613808155059814,
878
+ "val": 0.3579,
879
+ "test": 0.3697
880
+ },
881
+ {
882
+ "step": 800,
883
+ "loss": 1.8486320972442627,
884
+ "val": 0.385,
885
+ "test": 0.3898
886
+ },
887
+ {
888
+ "step": 1200,
889
+ "loss": 1.8923535346984863,
890
+ "val": 0.3895,
891
+ "test": 0.3912
892
+ },
893
+ {
894
+ "step": 1600,
895
+ "loss": 1.7093164920806885,
896
+ "val": 0.3965,
897
+ "test": 0.4073
898
+ },
899
+ {
900
+ "step": 2000,
901
+ "loss": 1.7840051651000977,
902
+ "val": 0.4122,
903
+ "test": 0.4099
904
+ },
905
+ {
906
+ "step": 2400,
907
+ "loss": 1.686863899230957,
908
+ "val": 0.4275,
909
+ "test": 0.4287
910
+ },
911
+ {
912
+ "step": 2800,
913
+ "loss": 1.8934996128082275,
914
+ "val": 0.4234,
915
+ "test": 0.4315
916
+ },
917
+ {
918
+ "step": 3200,
919
+ "loss": 1.8120520114898682,
920
+ "val": 0.428,
921
+ "test": 0.4302
922
+ },
923
+ {
924
+ "step": 3600,
925
+ "loss": 1.8429216146469116,
926
+ "val": 0.4221,
927
+ "test": 0.4195
928
+ },
929
+ {
930
+ "step": 4000,
931
+ "loss": 1.7407853603363037,
932
+ "val": 0.4353,
933
+ "test": 0.4385
934
+ }
935
+ ],
936
+ "device": "cuda",
937
+ "torch": "2.11.0+cu128",
938
+ "peak_card_bytes": 701194752,
939
+ "checkpoint": "frozen_s0_steps4000_b256.pt"
940
+ },
941
+ "frozen_s1": {
942
+ "run": "frozen_s1",
943
+ "arm": "frozen",
944
+ "seed": 1,
945
+ "suffix": "",
946
+ "torch_seed": 1212502857,
947
+ "params": 1397386,
948
+ "params_active": 1109226,
949
+ "unit": 20,
950
+ "dims": [
951
+ 60,
952
+ 80,
953
+ 100
954
+ ],
955
+ "val_selected_step": 3200,
956
+ "val_acc": 0.4297,
957
+ "test_acc": 0.4348,
958
+ "final_test": 0.4281,
959
+ "final_val": 0.4188,
960
+ "final_train_loss": 1.6491775512695312,
961
+ "step1_grads": {
962
+ "decay": 0.0,
963
+ "rest": 0.17970719703888988,
964
+ "ratio": 0.0,
965
+ "n_decay_tensors": 0
966
+ },
967
+ "lambda": {
968
+ "mean": 0.9934642910957336,
969
+ "min": 0.96875,
970
+ "max": 0.999755859375,
971
+ "frac_below_0.99": 0.2347826212644577,
972
+ "frac_erased_lt_1e-6": 0.0,
973
+ "frac_saturated_eq_1": 0.0,
974
+ "exactly_one_everywhere": false,
975
+ "per_cell_mean": {
976
+ "layers.0.v_f": 0.9934502840042114,
977
+ "layers.0.v_r": 0.9934502840042114,
978
+ "layers.0.h_f": 0.9934502840042114,
979
+ "layers.0.h_r": 0.9934502840042114,
980
+ "layers.1.v_f": 0.9934908151626587,
981
+ "layers.1.v_r": 0.9934908151626587,
982
+ "layers.1.h_f": 0.9934908151626587,
983
+ "layers.1.h_r": 0.9934908151626587,
984
+ "layers.2.v_f": 0.9935148358345032,
985
+ "layers.2.v_r": 0.9935148358345032,
986
+ "layers.2.h_f": 0.9935148358345032,
987
+ "layers.2.h_r": 0.9935148358345032
988
+ }
989
+ },
990
+ "wall_s": 241.5,
991
+ "history": [
992
+ {
993
+ "step": 400,
994
+ "loss": 1.9104502201080322,
995
+ "val": 0.3561,
996
+ "test": 0.3577
997
+ },
998
+ {
999
+ "step": 800,
1000
+ "loss": 1.9009696245193481,
1001
+ "val": 0.3841,
1002
+ "test": 0.3871
1003
+ },
1004
+ {
1005
+ "step": 1200,
1006
+ "loss": 1.9571985006332397,
1007
+ "val": 0.3943,
1008
+ "test": 0.3983
1009
+ },
1010
+ {
1011
+ "step": 1600,
1012
+ "loss": 1.8460667133331299,
1013
+ "val": 0.4064,
1014
+ "test": 0.4059
1015
+ },
1016
+ {
1017
+ "step": 2000,
1018
+ "loss": 1.9049961566925049,
1019
+ "val": 0.4211,
1020
+ "test": 0.4291
1021
+ },
1022
+ {
1023
+ "step": 2400,
1024
+ "loss": 1.817314863204956,
1025
+ "val": 0.4137,
1026
+ "test": 0.42
1027
+ },
1028
+ {
1029
+ "step": 2800,
1030
+ "loss": 1.7902936935424805,
1031
+ "val": 0.4124,
1032
+ "test": 0.413
1033
+ },
1034
+ {
1035
+ "step": 3200,
1036
+ "loss": 1.7846858501434326,
1037
+ "val": 0.4297,
1038
+ "test": 0.4348
1039
+ },
1040
+ {
1041
+ "step": 3600,
1042
+ "loss": 1.727637767791748,
1043
+ "val": 0.4169,
1044
+ "test": 0.4283
1045
+ },
1046
+ {
1047
+ "step": 4000,
1048
+ "loss": 1.6491775512695312,
1049
+ "val": 0.4188,
1050
+ "test": 0.4281
1051
+ }
1052
+ ],
1053
+ "device": "cuda",
1054
+ "torch": "2.11.0+cu128",
1055
+ "peak_card_bytes": 701194752,
1056
+ "checkpoint": "frozen_s1_steps4000_b256.pt"
1057
+ },
1058
+ "nodecay_wide_s0": {
1059
+ "run": "nodecay_wide_s0",
1060
+ "arm": "nodecay_wide",
1061
+ "seed": 0,
1062
+ "suffix": "",
1063
+ "torch_seed": 1479331954,
1064
+ "params": 1813002,
1065
+ "params_active": 1398858,
1066
+ "unit": 24,
1067
+ "dims": [
1068
+ 72,
1069
+ 96,
1070
+ 120
1071
+ ],
1072
+ "val_selected_step": 4000,
1073
+ "val_acc": 0.4263,
1074
+ "test_acc": 0.4241,
1075
+ "final_test": 0.4241,
1076
+ "final_val": 0.4263,
1077
+ "final_train_loss": 1.6947191953659058,
1078
+ "step1_grads": {
1079
+ "decay": 0.0,
1080
+ "rest": 0.16649678770303605,
1081
+ "ratio": 0.0,
1082
+ "n_decay_tensors": 0
1083
+ },
1084
+ "lambda": {
1085
+ "mean": 1.0,
1086
+ "min": 1.0,
1087
+ "max": 1.0,
1088
+ "frac_below_0.99": 0.0,
1089
+ "frac_erased_lt_1e-6": 0.0,
1090
+ "frac_saturated_eq_1": 1.0,
1091
+ "exactly_one_everywhere": true,
1092
+ "per_cell_mean": {
1093
+ "layers.0.v_f": 1.0,
1094
+ "layers.0.v_r": 1.0,
1095
+ "layers.0.h_f": 1.0,
1096
+ "layers.0.h_r": 1.0,
1097
+ "layers.1.v_f": 1.0,
1098
+ "layers.1.v_r": 1.0,
1099
+ "layers.1.h_f": 1.0,
1100
+ "layers.1.h_r": 1.0,
1101
+ "layers.2.v_f": 1.0,
1102
+ "layers.2.v_r": 1.0,
1103
+ "layers.2.h_f": 1.0,
1104
+ "layers.2.h_r": 1.0
1105
+ }
1106
+ },
1107
+ "wall_s": 198.2,
1108
+ "history": [
1109
+ {
1110
+ "step": 400,
1111
+ "loss": 1.9634010791778564,
1112
+ "val": 0.3353,
1113
+ "test": 0.3317
1114
+ },
1115
+ {
1116
+ "step": 800,
1117
+ "loss": 1.896615982055664,
1118
+ "val": 0.3582,
1119
+ "test": 0.3672
1120
+ },
1121
+ {
1122
+ "step": 1200,
1123
+ "loss": 1.959193229675293,
1124
+ "val": 0.3889,
1125
+ "test": 0.3939
1126
+ },
1127
+ {
1128
+ "step": 1600,
1129
+ "loss": 1.7899928092956543,
1130
+ "val": 0.394,
1131
+ "test": 0.3937
1132
+ },
1133
+ {
1134
+ "step": 2000,
1135
+ "loss": 1.841754674911499,
1136
+ "val": 0.3959,
1137
+ "test": 0.4021
1138
+ },
1139
+ {
1140
+ "step": 2400,
1141
+ "loss": 1.8209848403930664,
1142
+ "val": 0.4123,
1143
+ "test": 0.4161
1144
+ },
1145
+ {
1146
+ "step": 2800,
1147
+ "loss": 1.7542763948440552,
1148
+ "val": 0.423,
1149
+ "test": 0.4194
1150
+ },
1151
+ {
1152
+ "step": 3200,
1153
+ "loss": 1.6521923542022705,
1154
+ "val": 0.4237,
1155
+ "test": 0.4267
1156
+ },
1157
+ {
1158
+ "step": 3600,
1159
+ "loss": 1.7738583087921143,
1160
+ "val": 0.4235,
1161
+ "test": 0.4276
1162
+ },
1163
+ {
1164
+ "step": 4000,
1165
+ "loss": 1.6947191953659058,
1166
+ "val": 0.4263,
1167
+ "test": 0.4241
1168
+ }
1169
+ ],
1170
+ "device": "cuda",
1171
+ "torch": "2.11.0+cu128",
1172
+ "peak_card_bytes": 826973696,
1173
+ "checkpoint": "nodecay_wide_s0_steps4000_b256.pt"
1174
+ },
1175
+ "nodecay_wide_s1": {
1176
+ "run": "nodecay_wide_s1",
1177
+ "arm": "nodecay_wide",
1178
+ "seed": 1,
1179
+ "suffix": "",
1180
+ "torch_seed": 791404772,
1181
+ "params": 1813002,
1182
+ "params_active": 1398858,
1183
+ "unit": 24,
1184
+ "dims": [
1185
+ 72,
1186
+ 96,
1187
+ 120
1188
+ ],
1189
+ "val_selected_step": 3600,
1190
+ "val_acc": 0.4387,
1191
+ "test_acc": 0.4391,
1192
+ "final_test": 0.4214,
1193
+ "final_val": 0.4213,
1194
+ "final_train_loss": 1.780710220336914,
1195
+ "step1_grads": {
1196
+ "decay": 0.0,
1197
+ "rest": 0.13169671619861834,
1198
+ "ratio": 0.0,
1199
+ "n_decay_tensors": 0
1200
+ },
1201
+ "lambda": {
1202
+ "mean": 1.0,
1203
+ "min": 1.0,
1204
+ "max": 1.0,
1205
+ "frac_below_0.99": 0.0,
1206
+ "frac_erased_lt_1e-6": 0.0,
1207
+ "frac_saturated_eq_1": 1.0,
1208
+ "exactly_one_everywhere": true,
1209
+ "per_cell_mean": {
1210
+ "layers.0.v_f": 1.0,
1211
+ "layers.0.v_r": 1.0,
1212
+ "layers.0.h_f": 1.0,
1213
+ "layers.0.h_r": 1.0,
1214
+ "layers.1.v_f": 1.0,
1215
+ "layers.1.v_r": 1.0,
1216
+ "layers.1.h_f": 1.0,
1217
+ "layers.1.h_r": 1.0,
1218
+ "layers.2.v_f": 1.0,
1219
+ "layers.2.v_r": 1.0,
1220
+ "layers.2.h_f": 1.0,
1221
+ "layers.2.h_r": 1.0
1222
+ }
1223
+ },
1224
+ "wall_s": 196.2,
1225
+ "history": [
1226
+ {
1227
+ "step": 400,
1228
+ "loss": 1.9460245370864868,
1229
+ "val": 0.3332,
1230
+ "test": 0.3406
1231
+ },
1232
+ {
1233
+ "step": 800,
1234
+ "loss": 1.8643466234207153,
1235
+ "val": 0.3733,
1236
+ "test": 0.3747
1237
+ },
1238
+ {
1239
+ "step": 1200,
1240
+ "loss": 1.784814476966858,
1241
+ "val": 0.3841,
1242
+ "test": 0.3932
1243
+ },
1244
+ {
1245
+ "step": 1600,
1246
+ "loss": 1.8816750049591064,
1247
+ "val": 0.4114,
1248
+ "test": 0.4147
1249
+ },
1250
+ {
1251
+ "step": 2000,
1252
+ "loss": 1.8478457927703857,
1253
+ "val": 0.3963,
1254
+ "test": 0.4005
1255
+ },
1256
+ {
1257
+ "step": 2400,
1258
+ "loss": 1.8344430923461914,
1259
+ "val": 0.4203,
1260
+ "test": 0.4209
1261
+ },
1262
+ {
1263
+ "step": 2800,
1264
+ "loss": 1.838099718093872,
1265
+ "val": 0.3982,
1266
+ "test": 0.3908
1267
+ },
1268
+ {
1269
+ "step": 3200,
1270
+ "loss": 1.7225689888000488,
1271
+ "val": 0.422,
1272
+ "test": 0.4227
1273
+ },
1274
+ {
1275
+ "step": 3600,
1276
+ "loss": 1.7007989883422852,
1277
+ "val": 0.4387,
1278
+ "test": 0.4391
1279
+ },
1280
+ {
1281
+ "step": 4000,
1282
+ "loss": 1.780710220336914,
1283
+ "val": 0.4213,
1284
+ "test": 0.4214
1285
+ }
1286
+ ],
1287
+ "device": "cuda",
1288
+ "torch": "2.11.0+cu128",
1289
+ "peak_card_bytes": 826973696,
1290
+ "checkpoint": "nodecay_wide_s1_steps4000_b256.pt"
1291
+ },
1292
+ "renet_s0": {
1293
+ "run": "renet_s0",
1294
+ "arm": "renet",
1295
+ "seed": 0,
1296
+ "suffix": "",
1297
+ "torch_seed": 277084174,
1298
+ "params": 1364362,
1299
+ "params_active": 1364362,
1300
+ "unit": null,
1301
+ "dims": [
1302
+ 48,
1303
+ 64,
1304
+ 80
1305
+ ],
1306
+ "val_selected_step": 3200,
1307
+ "val_acc": 0.7344,
1308
+ "test_acc": 0.7233,
1309
+ "final_test": 0.7068,
1310
+ "final_val": 0.706,
1311
+ "final_train_loss": 0.6282953023910522,
1312
+ "step1_grads": {
1313
+ "decay": 0.0,
1314
+ "rest": 0.09216733004088452,
1315
+ "ratio": 0.0,
1316
+ "n_decay_tensors": 0
1317
+ },
1318
+ "lambda": null,
1319
+ "wall_s": 385.5,
1320
+ "history": [
1321
+ {
1322
+ "step": 400,
1323
+ "loss": 1.373478889465332,
1324
+ "val": 0.5382,
1325
+ "test": 0.5422
1326
+ },
1327
+ {
1328
+ "step": 800,
1329
+ "loss": 1.0481791496276855,
1330
+ "val": 0.6277,
1331
+ "test": 0.6229
1332
+ },
1333
+ {
1334
+ "step": 1200,
1335
+ "loss": 0.9306557774543762,
1336
+ "val": 0.6549,
1337
+ "test": 0.6512
1338
+ },
1339
+ {
1340
+ "step": 1600,
1341
+ "loss": 0.8042398691177368,
1342
+ "val": 0.6467,
1343
+ "test": 0.6502
1344
+ },
1345
+ {
1346
+ "step": 2000,
1347
+ "loss": 0.793249249458313,
1348
+ "val": 0.6731,
1349
+ "test": 0.6741
1350
+ },
1351
+ {
1352
+ "step": 2400,
1353
+ "loss": 0.8229799270629883,
1354
+ "val": 0.7041,
1355
+ "test": 0.7031
1356
+ },
1357
+ {
1358
+ "step": 2800,
1359
+ "loss": 0.72426438331604,
1360
+ "val": 0.701,
1361
+ "test": 0.6975
1362
+ },
1363
+ {
1364
+ "step": 3200,
1365
+ "loss": 0.6971091032028198,
1366
+ "val": 0.7344,
1367
+ "test": 0.7233
1368
+ },
1369
+ {
1370
+ "step": 3600,
1371
+ "loss": 0.6333722472190857,
1372
+ "val": 0.7318,
1373
+ "test": 0.7249
1374
+ },
1375
+ {
1376
+ "step": 4000,
1377
+ "loss": 0.6282953023910522,
1378
+ "val": 0.706,
1379
+ "test": 0.7068
1380
+ }
1381
+ ],
1382
+ "device": "cuda",
1383
+ "torch": "2.11.0+cu128",
1384
+ "peak_card_bytes": 738966016,
1385
+ "checkpoint": "renet_s0_steps4000_b256.pt"
1386
+ },
1387
+ "renet_s1": {
1388
+ "run": "renet_s1",
1389
+ "arm": "renet",
1390
+ "seed": 1,
1391
+ "suffix": "",
1392
+ "torch_seed": 1736755352,
1393
+ "params": 1364362,
1394
+ "params_active": 1364362,
1395
+ "unit": null,
1396
+ "dims": [
1397
+ 48,
1398
+ 64,
1399
+ 80
1400
+ ],
1401
+ "val_selected_step": 4000,
1402
+ "val_acc": 0.7321,
1403
+ "test_acc": 0.7197,
1404
+ "final_test": 0.7197,
1405
+ "final_val": 0.7321,
1406
+ "final_train_loss": 0.7341122627258301,
1407
+ "step1_grads": {
1408
+ "decay": 0.0,
1409
+ "rest": 0.09333898452133431,
1410
+ "ratio": 0.0,
1411
+ "n_decay_tensors": 0
1412
+ },
1413
+ "lambda": null,
1414
+ "wall_s": 375.5,
1415
+ "history": [
1416
+ {
1417
+ "step": 400,
1418
+ "loss": 1.4018796682357788,
1419
+ "val": 0.537,
1420
+ "test": 0.5398
1421
+ },
1422
+ {
1423
+ "step": 800,
1424
+ "loss": 1.1355798244476318,
1425
+ "val": 0.6298,
1426
+ "test": 0.6228
1427
+ },
1428
+ {
1429
+ "step": 1200,
1430
+ "loss": 0.9261863827705383,
1431
+ "val": 0.6512,
1432
+ "test": 0.6479
1433
+ },
1434
+ {
1435
+ "step": 1600,
1436
+ "loss": 0.8397046327590942,
1437
+ "val": 0.6798,
1438
+ "test": 0.6775
1439
+ },
1440
+ {
1441
+ "step": 2000,
1442
+ "loss": 0.8869620561599731,
1443
+ "val": 0.6891,
1444
+ "test": 0.681
1445
+ },
1446
+ {
1447
+ "step": 2400,
1448
+ "loss": 0.7484608292579651,
1449
+ "val": 0.7093,
1450
+ "test": 0.6978
1451
+ },
1452
+ {
1453
+ "step": 2800,
1454
+ "loss": 0.7494136691093445,
1455
+ "val": 0.7173,
1456
+ "test": 0.7164
1457
+ },
1458
+ {
1459
+ "step": 3200,
1460
+ "loss": 0.7631729245185852,
1461
+ "val": 0.7008,
1462
+ "test": 0.6978
1463
+ },
1464
+ {
1465
+ "step": 3600,
1466
+ "loss": 0.7102234363555908,
1467
+ "val": 0.6734,
1468
+ "test": 0.6718
1469
+ },
1470
+ {
1471
+ "step": 4000,
1472
+ "loss": 0.7341122627258301,
1473
+ "val": 0.7321,
1474
+ "test": 0.7197
1475
+ }
1476
+ ],
1477
+ "device": "cuda",
1478
+ "torch": "2.11.0+cu128",
1479
+ "peak_card_bytes": 738966016,
1480
+ "checkpoint": "renet_s1_steps4000_b256.pt"
1481
+ },
1482
+ "renet_s0_rep": {
1483
+ "run": "renet_s0_rep",
1484
+ "arm": "renet",
1485
+ "seed": 0,
1486
+ "suffix": "_rep",
1487
+ "torch_seed": 377069471,
1488
+ "params": 1364362,
1489
+ "params_active": 1364362,
1490
+ "unit": null,
1491
+ "dims": [
1492
+ 48,
1493
+ 64,
1494
+ 80
1495
+ ],
1496
+ "val_selected_step": 4000,
1497
+ "val_acc": 0.7286,
1498
+ "test_acc": 0.7221,
1499
+ "final_test": 0.7221,
1500
+ "final_val": 0.7286,
1501
+ "final_train_loss": 0.7075773477554321,
1502
+ "step1_grads": {
1503
+ "decay": 0.0,
1504
+ "rest": 0.1119405215359885,
1505
+ "ratio": 0.0,
1506
+ "n_decay_tensors": 0
1507
+ },
1508
+ "lambda": null,
1509
+ "wall_s": 376.8,
1510
+ "history": [
1511
+ {
1512
+ "step": 400,
1513
+ "loss": 1.3234918117523193,
1514
+ "val": 0.5477,
1515
+ "test": 0.5512
1516
+ },
1517
+ {
1518
+ "step": 800,
1519
+ "loss": 1.0493826866149902,
1520
+ "val": 0.6281,
1521
+ "test": 0.6297
1522
+ },
1523
+ {
1524
+ "step": 1200,
1525
+ "loss": 0.9839962720870972,
1526
+ "val": 0.6473,
1527
+ "test": 0.654
1528
+ },
1529
+ {
1530
+ "step": 1600,
1531
+ "loss": 0.8052473664283752,
1532
+ "val": 0.6803,
1533
+ "test": 0.6786
1534
+ },
1535
+ {
1536
+ "step": 2000,
1537
+ "loss": 0.8834858536720276,
1538
+ "val": 0.7113,
1539
+ "test": 0.708
1540
+ },
1541
+ {
1542
+ "step": 2400,
1543
+ "loss": 0.8952056765556335,
1544
+ "val": 0.6966,
1545
+ "test": 0.701
1546
+ },
1547
+ {
1548
+ "step": 2800,
1549
+ "loss": 0.8220301866531372,
1550
+ "val": 0.723,
1551
+ "test": 0.7169
1552
+ },
1553
+ {
1554
+ "step": 3200,
1555
+ "loss": 0.6942834854125977,
1556
+ "val": 0.7103,
1557
+ "test": 0.7122
1558
+ },
1559
+ {
1560
+ "step": 3600,
1561
+ "loss": 0.6144925951957703,
1562
+ "val": 0.7254,
1563
+ "test": 0.7186
1564
+ },
1565
+ {
1566
+ "step": 4000,
1567
+ "loss": 0.7075773477554321,
1568
+ "val": 0.7286,
1569
+ "test": 0.7221
1570
+ }
1571
+ ],
1572
+ "device": "cuda",
1573
+ "torch": "2.11.0+cu128",
1574
+ "peak_card_bytes": 738966016,
1575
+ "checkpoint": "renet_s0_rep_steps4000_b256.pt"
1576
+ }
1577
+ },
1578
+ "errors": [],
1579
+ "unit_seconds": {
1580
+ "probe_s0": 22.3,
1581
+ "probe_s1": 19.4,
1582
+ "nodecay_s0": 334.2,
1583
+ "nodecay_s1": 315.7,
1584
+ "nodecay_s0_rep": 313.7,
1585
+ "lin_s0": 668.3,
1586
+ "lin_s1": 646.7,
1587
+ "frozen_s0": 294.6,
1588
+ "frozen_s1": 241.6,
1589
+ "nodecay_wide_s0": 200.3,
1590
+ "nodecay_wide_s1": 196.3,
1591
+ "renet_s0": 385.5,
1592
+ "renet_s1": 375.5,
1593
+ "renet_s0_rep": 376.9
1594
+ },
1595
+ "budget": 1364362,
1596
+ "units": {
1597
+ "nodecay": 20,
1598
+ "lin": 20,
1599
+ "frozen": 20,
1600
+ "nodecay_wide": 24
1601
+ },
1602
+ "width_match": {
1603
+ "nodecay": {
1604
+ "unit": 20,
1605
+ "dims": [
1606
+ 60,
1607
+ 80,
1608
+ 100
1609
+ ],
1610
+ "total": 1397386,
1611
+ "active": 1109226
1612
+ },
1613
+ "lin": {
1614
+ "unit": 20,
1615
+ "dims": [
1616
+ 60,
1617
+ 80,
1618
+ 100
1619
+ ],
1620
+ "total": 1397386,
1621
+ "active": 1397386
1622
+ },
1623
+ "frozen": {
1624
+ "unit": 20,
1625
+ "dims": [
1626
+ 60,
1627
+ 80,
1628
+ 100
1629
+ ],
1630
+ "total": 1397386,
1631
+ "active": 1109226
1632
+ },
1633
+ "nodecay_wide": {
1634
+ "unit": 24,
1635
+ "dims": [
1636
+ 72,
1637
+ 96,
1638
+ 120
1639
+ ],
1640
+ "total": 1813002,
1641
+ "active": 1398858
1642
+ }
1643
+ },
1644
+ "summary": {
1645
+ "read": "test_acc",
1646
+ "bar_reference": {
1647
+ "ref": "nodecay",
1648
+ "repeat": 0.00880000000000003,
1649
+ "seed": 0.0023000000000000242,
1650
+ "bar": 0.00880000000000003,
1651
+ "complete": true
1652
+ },
1653
+ "floor_gate": {
1654
+ "per_seed": {
1655
+ "0": {
1656
+ "ref": 0.4405,
1657
+ "probe": 0.3531,
1658
+ "delta": 0.08739999999999998,
1659
+ "bars": 9.931818181818146,
1660
+ "clears": true
1661
+ },
1662
+ "1": {
1663
+ "ref": 0.4428,
1664
+ "probe": 0.3304,
1665
+ "delta": 0.1124,
1666
+ "bars": 12.772727272727229,
1667
+ "clears": true
1668
+ }
1669
+ },
1670
+ "pass": true
1671
+ },
1672
+ "decay": {
1673
+ "lin_minus_ref": {
1674
+ "0": 0.10729999999999995,
1675
+ "1": 0.11130000000000001
1676
+ },
1677
+ "bar": 0.00880000000000003,
1678
+ "two_control": {
1679
+ "0": {
1680
+ "deltas": {
1681
+ "nodecay": 0.10729999999999995,
1682
+ "nodecay_wide": 0.12369999999999998
1683
+ },
1684
+ "verdict": "controls DISAGREE beyond the bar -> INVALIDATED, not refuted",
1685
+ "two_controls": true
1686
+ },
1687
+ "1": {
1688
+ "deltas": {
1689
+ "nodecay": 0.11130000000000001,
1690
+ "nodecay_wide": 0.11500000000000005
1691
+ },
1692
+ "verdict": "ABOVE the bar on both controls -- decay pays here",
1693
+ "two_controls": true
1694
+ }
1695
+ },
1696
+ "verdict": "INVALIDATED (the two no-decay controls disagree beyond the bar)"
1697
+ },
1698
+ "replication": {
1699
+ "bar_renet": {
1700
+ "repeat": 0.0012000000000000899,
1701
+ "seed": 0.0036000000000000476,
1702
+ "bar": 0.0036000000000000476,
1703
+ "complete": true
1704
+ },
1705
+ "go_no_go": {
1706
+ "0": {
1707
+ "delta": 0.37020000000000003,
1708
+ "bars": 102.83333333333198,
1709
+ "go": true
1710
+ },
1711
+ "1": {
1712
+ "delta": 0.3893,
1713
+ "bars": 108.13888888888745,
1714
+ "go": true
1715
+ }
1716
+ },
1717
+ "vs_notebook": {
1718
+ "renet_s0": {
1719
+ "ours": 0.7233,
1720
+ "notebook": 0.7275,
1721
+ "diff": -0.0041999999999999815
1722
+ },
1723
+ "renet_s0_rep": {
1724
+ "ours": 0.7221,
1725
+ "notebook": 0.7217,
1726
+ "diff": 0.00039999999999995595
1727
+ },
1728
+ "renet_s1": {
1729
+ "ours": 0.7197,
1730
+ "notebook": 0.7162,
1731
+ "diff": 0.0035000000000000586
1732
+ },
1733
+ "probe_s0": {
1734
+ "ours": 0.3531,
1735
+ "notebook": 0.3589,
1736
+ "diff": -0.005799999999999972
1737
+ },
1738
+ "probe_s1": {
1739
+ "ours": 0.3304,
1740
+ "notebook": 0.3365,
1741
+ "diff": -0.006099999999999994
1742
+ }
1743
+ }
1744
+ },
1745
+ "separation": {
1746
+ "frozen_minus_ref": {
1747
+ "0": -0.0020000000000000018,
1748
+ "1": -0.008000000000000007
1749
+ },
1750
+ "lin_minus_frozen": {
1751
+ "0": 0.10929999999999995,
1752
+ "1": 0.11930000000000002
1753
+ },
1754
+ "bar": 0.00880000000000003,
1755
+ "verdict": "GATING IS THE PAYER"
1756
+ },
1757
+ "lin_minus_gru_renet": {
1758
+ "0": -0.1755000000000001,
1759
+ "1": -0.16559999999999997
1760
+ }
1761
+ },
1762
+ "summary_final_test": {
1763
+ "read": "final_test",
1764
+ "bar_reference": {
1765
+ "ref": "nodecay",
1766
+ "repeat": 0.010699999999999987,
1767
+ "seed": 0.0040999999999999925,
1768
+ "bar": 0.010699999999999987,
1769
+ "complete": true
1770
+ },
1771
+ "floor_gate": {
1772
+ "per_seed": {
1773
+ "0": {
1774
+ "ref": 0.4417,
1775
+ "probe": 0.3111,
1776
+ "delta": 0.1306,
1777
+ "bars": 12.205607476635528,
1778
+ "clears": true
1779
+ },
1780
+ "1": {
1781
+ "ref": 0.4376,
1782
+ "probe": 0.3314,
1783
+ "delta": 0.10620000000000002,
1784
+ "bars": 9.925233644859826,
1785
+ "clears": true
1786
+ }
1787
+ },
1788
+ "pass": true
1789
+ },
1790
+ "decay": {
1791
+ "lin_minus_ref": {
1792
+ "0": 0.10609999999999997,
1793
+ "1": 0.09709999999999996
1794
+ },
1795
+ "bar": 0.010699999999999987,
1796
+ "two_control": {
1797
+ "0": {
1798
+ "deltas": {
1799
+ "nodecay": 0.10609999999999997,
1800
+ "nodecay_wide": 0.12369999999999998
1801
+ },
1802
+ "verdict": "controls DISAGREE beyond the bar -> INVALIDATED, not refuted",
1803
+ "two_controls": true
1804
+ },
1805
+ "1": {
1806
+ "deltas": {
1807
+ "nodecay": 0.09709999999999996,
1808
+ "nodecay_wide": 0.11329999999999996
1809
+ },
1810
+ "verdict": "controls DISAGREE beyond the bar -> INVALIDATED, not refuted",
1811
+ "two_controls": true
1812
+ }
1813
+ },
1814
+ "verdict": "INVALIDATED (the two no-decay controls disagree beyond the bar)"
1815
+ },
1816
+ "replication": {
1817
+ "bar_renet": {
1818
+ "repeat": 0.01529999999999998,
1819
+ "seed": 0.012900000000000023,
1820
+ "bar": 0.01529999999999998,
1821
+ "complete": true
1822
+ },
1823
+ "go_no_go": {
1824
+ "0": {
1825
+ "delta": 0.3957,
1826
+ "bars": 25.862745098039248,
1827
+ "go": true
1828
+ },
1829
+ "1": {
1830
+ "delta": 0.38830000000000003,
1831
+ "bars": 25.379084967320296,
1832
+ "go": true
1833
+ }
1834
+ },
1835
+ "vs_notebook": {
1836
+ "renet_s0": {
1837
+ "ours": 0.7068,
1838
+ "notebook": 0.7275,
1839
+ "diff": -0.02070000000000005
1840
+ },
1841
+ "renet_s0_rep": {
1842
+ "ours": 0.7221,
1843
+ "notebook": 0.7217,
1844
+ "diff": 0.00039999999999995595
1845
+ },
1846
+ "renet_s1": {
1847
+ "ours": 0.7197,
1848
+ "notebook": 0.7162,
1849
+ "diff": 0.0035000000000000586
1850
+ },
1851
+ "probe_s0": {
1852
+ "ours": 0.3111,
1853
+ "notebook": 0.3589,
1854
+ "diff": -0.04780000000000001
1855
+ },
1856
+ "probe_s1": {
1857
+ "ours": 0.3314,
1858
+ "notebook": 0.3365,
1859
+ "diff": -0.005100000000000049
1860
+ }
1861
+ }
1862
+ },
1863
+ "separation": {
1864
+ "frozen_minus_ref": {
1865
+ "0": -0.0031999999999999806,
1866
+ "1": -0.009500000000000008
1867
+ },
1868
+ "lin_minus_frozen": {
1869
+ "0": 0.10929999999999995,
1870
+ "1": 0.10659999999999997
1871
+ },
1872
+ "bar": 0.010699999999999987,
1873
+ "verdict": "GATING IS THE PAYER"
1874
+ },
1875
+ "lin_minus_gru_renet": {
1876
+ "0": -0.15900000000000003,
1877
+ "1": -0.18500000000000005
1878
+ }
1879
+ }
1880
+ }
README.md CHANGED
@@ -13,5 +13,6 @@ experiment had to meet were written down before it ran. Code is not hosted here.
13
  | 2026-11-15 | Anchor drift by training length | The original 500-step test reproduces on all seven activations. Trained 20 times longer with 5 seeds, the average anchor drift does not settle at 0.2915 radians: all 35 runs read as passing through and end between 0.30 and 0.52 depending on the activation. A control with a built-in resting point reads as resting. One test bed. | `2026-11-15_anchor_drift` |
14
  | 2026-11-15 | Stored vision checkpoints on the CIFAR-100 test set | The rebuilt model code reads the published single-stream accuracies back (best 0.5816). The best of 13 stored two-stream checkpoints reads 0.5517, so neither 60% nor 66% is reproduced by the stored files. Removing the geometric features from the best single-stream model costs 0.0029. Stored checkpoints only. | `2026-11-15_vision_checkpoints` |
15
  | 2026-11-15 | Beeper twins on held-out text | Two small language models of August 2025, one plain and one trained with extra geometric loss terms on the same data, read on the same 8,903 held-out windows. The plain twin reproduces its own notebook's numbers. The twin with the geometric terms reads 0.0005 lower per token, about 2 standard errors: no difference is resolved by the rule set before the run. One training run per twin. | `2026-11-15_beeper_twins` |
 
16
 
17
  Each recap says what its result does not show. Work by AbstractPhil in collaboration with AI research assistants.
 
13
  | 2026-11-15 | Anchor drift by training length | The original 500-step test reproduces on all seven activations. Trained 20 times longer with 5 seeds, the average anchor drift does not settle at 0.2915 radians: all 35 runs read as passing through and end between 0.30 and 0.52 depending on the activation. A control with a built-in resting point reads as resting. One test bed. | `2026-11-15_anchor_drift` |
14
  | 2026-11-15 | Stored vision checkpoints on the CIFAR-100 test set | The rebuilt model code reads the published single-stream accuracies back (best 0.5816). The best of 13 stored two-stream checkpoints reads 0.5517, so neither 60% nor 66% is reproduced by the stored files. Removing the geometric features from the best single-stream model costs 0.0029. Stored checkpoints only. | `2026-11-15_vision_checkpoints` |
15
  | 2026-11-15 | Beeper twins on held-out text | Two small language models of August 2025, one plain and one trained with extra geometric loss terms on the same data, read on the same 8,903 held-out windows. The plain twin reproduces its own notebook's numbers. The twin with the geometric terms reads 0.0005 lower per token, about 2 standard errors: no difference is resolved by the rule set before the run. One training run per twin. | `2026-11-15_beeper_twins` |
16
+ | 2026-11-15 | Decay arms on a rebuilt recurrent CIFAR-10 bed | The original recurrent classifier reproduces within 0.006. With its cells replaced by a linear state cell, a fixed decay adds nothing over no decay, and a learned input-dependent decay reads about 0.11 higher on both seeds. The registered label of the decay read is invalidated, because two no-decay controls disagree beyond the noise bar on one seed; the registered separation reads gating as the payer. The learned-decay arm stays about 0.17 below the original cell. One bed, two seeds. | `2026-11-15_decay_arms` |
17
 
18
  Each recap says what its result does not show. Work by AbstractPhil in collaboration with AI research assistants.