GOVINDFROM commited on
Commit
997c9bc
Β·
verified Β·
1 Parent(s): ec6e864

Retract input-injection claim; correct batch-size pooling error

Browse files
Files changed (1) hide show
  1. README.md +34 -15
README.md CHANGED
@@ -35,7 +35,7 @@ digit relabelling. Metric: board accuracy, all 81 cells correct.
35
  | greedy MRV | – | – | 5.90% |
36
  | autoregressive prefix-LM | 2.4M | – | 0.30% |
37
  | direct recurrent classifier* | 212k | 10.5% | 31.00% |
38
- | recurrent, no input injection | 212k | – | 94.10% |
39
  | feed-forward diffusion, MDLM | 37.9M | 4.40% | 88.15% |
40
  | feed-forward diffusion, schedule-matched | 37.9M | 40.10% | 88.90% |
41
  | 32 distinct layers, no weight sharing | 6.34M | 70.20% | 94.50% |
@@ -50,20 +50,32 @@ evidence against their method**, and it is not used as a comparison.
50
 
51
  ## Ablations
52
 
53
- Built as a ladder, each step changing exactly one thing:
 
 
 
54
 
55
- | step | board | gain |
56
- |---|---|---|
57
- | feed-forward, 12 layers, 37.9M | 88.90% | – |
58
- | feed-forward, **32 layers**, 6.34M | 88.95% | **+0.05** |
59
- | + weight sharing (injection **off**), 212k | 94.10% | **+5.15** |
60
- | + input injection | **99.70%** | **+5.60** |
61
 
62
- **Depth alone is worth nothing** β€” 12β†’32 layers gains 0.05 points. Sharing one
63
- block across 32 applications gains 5.15 at 30Γ— fewer parameters; re-supplying the
64
- input each application gains a further 5.60. The weight-sharing comparison holds
65
- injection **off on both sides**, so the effects are separated rather than
66
- confounded.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
67
 
68
  Removing either from the full method: deep supervision **βˆ’1.95**, schedule
69
  matching **βˆ’3.15**.
@@ -91,8 +103,15 @@ supervision is dominated by copying the given clues. Low loss, no capability.
91
  clue range sits inside the Recurrent Transformer's 17–34 but excludes the
92
  hardest 17-clue instances, so 99.70% at 212k reads as *competitive with* their
93
  99.5% at 211k, not better.
94
- - **Input injection is not separated from weight sharing** in the 32-layer
95
- comparison; the isolating run is in progress.
 
 
 
 
 
 
 
96
  - The SATNet split is not used: constraint propagation alone solves 100% of it.
97
 
98
  ## Negative results, reported
 
35
  | greedy MRV | – | – | 5.90% |
36
  | autoregressive prefix-LM | 2.4M | – | 0.30% |
37
  | direct recurrent classifier* | 212k | 10.5% | 31.00% |
38
+ | recurrent, no input injection (batch 64) | 212k | – | 94.10% |
39
  | feed-forward diffusion, MDLM | 37.9M | 4.40% | 88.15% |
40
  | feed-forward diffusion, schedule-matched | 37.9M | 40.10% | 88.90% |
41
  | 32 distinct layers, no weight sharing | 6.34M | 70.20% | 94.50% |
 
50
 
51
  ## Ablations
52
 
53
+ **Compare only within a batch size.** An earlier version of this card pooled a
54
+ 32-layer run at batch 256 (94.50%) with one at batch 64 (83.40%) into a single
55
+ "88.95%", and drew two conclusions from the artefact. Both were wrong and are
56
+ corrected here.
57
 
58
+ **At batch 256:**
 
 
 
 
 
59
 
60
+ | step | params | board | gain |
61
+ |---|---|---|---|
62
+ | feed-forward, d=512, 12 layers | 37.9M | 88.90% (n=2) | – |
63
+ | feed-forward, d=128, 32 layers | 6.34M | 94.50% (n=1) | **+5.60** |
64
+ | + weight sharing, d=128, 1 layer Γ— 32 | **212k** | **99.70%** (n=2) | **+5.20** |
65
+
66
+ Parameters fall monotonically, accuracy rises monotonically: 37.9M β†’ 6.34M β†’
67
+ 212k against 88.90% β†’ 94.50% β†’ 99.70%.
68
+
69
+ **At batch 64, injection off on both sides:**
70
+
71
+ | step | params | board | gain |
72
+ |---|---|---|---|
73
+ | feed-forward, d=128, 32 layers | 6.34M | 83.40% (n=1) | – |
74
+ | + weight sharing | **212k** | **94.10%** (n=2) | **+10.70** |
75
+
76
+ **Corrections.** ~~"Depth alone is worth nothing (+0.05)"~~ β€” wrong and backwards;
77
+ at matched batch size the deeper, narrower, 6Γ— smaller model gains **+5.60**.
78
+ ~~"Input injection is worth +5.60"~~ β€” **retracted**, see below.
79
 
80
  Removing either from the full method: deep supervision **βˆ’1.95**, schedule
81
  matching **βˆ’3.15**.
 
103
  clue range sits inside the Recurrent Transformer's 17–34 but excludes the
104
  hardest 17-clue instances, so 99.70% at 212k reads as *competitive with* their
105
  99.5% at 211k, not better.
106
+ - **Input injection is retracted as a contribution.** In a matched test β€”
107
+ identical architecture, batch, and step count, with R-MDM's step embedding on
108
+ both sides β€” it is worth **+0.05 points** (96.85% β†’ 96.90%, best decode). The
109
+ previously claimed +5.60 compared an injection-ON arm at batch 256 against an
110
+ injection-OFF arm at batch 64; it measured batch size. The single injection-ON
111
+ seed lands inside the two-seed injection-OFF range at five of six decoding
112
+ budgets. All three arms stopped at step 80k of 110k on a wall-clock limit, and
113
+ the ON side is n=1; we could not add seeds, as the allocation had 3,030
114
+ billing-minutes left against ~3,600 for one run.
115
  - The SATNet split is not used: constraint propagation alone solves 100% of it.
116
 
117
  ## Negative results, reported