GOVINDFROM commited on
Commit
69d2186
·
verified ·
1 Parent(s): 55003bd

Results, figures and source (weights kept on scratch)

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +85 -16
  2. figures/beat_d12.json +56 -5
  3. figures/beat_d16.json +54 -3
  4. figures/fig3_depth_scaling.png +0 -0
  5. figures/fig4_text8_quality_vs_compute.png +0 -0
  6. figures/graph_ceiling.json +0 -0
  7. figures/summary.md +74 -20
  8. results/evallm.json +0 -0
  9. results/g14-d12-L6-s0.json +0 -0
  10. results/g14-d12-L6-s1.json +0 -0
  11. results/g14-d16-L2-s0.json +0 -0
  12. results/g14-d16-L2-s1.json +0 -0
  13. results/g14-d16-L3-s0.json +0 -0
  14. results/g14-d16-L3-s1.json +0 -0
  15. results/g14-d8-L12-s0.json +0 -0
  16. results/g14-d8-L12-s1.json +0 -0
  17. results/g15-finnish-distill-s0.json +0 -0
  18. results/g15-finnish-distill-s1.json +0 -0
  19. results/g15-finnish-matched-s0.json +0 -0
  20. results/g15-finnish-matched-s1.json +0 -0
  21. results/g15-georgian-distill-s0.json +0 -0
  22. results/g15-georgian-distill-s1.json +0 -0
  23. results/g15-georgian-matched-s0.json +0 -0
  24. results/g15-georgian-matched-s1.json +0 -0
  25. results/g15-german-distill-s1.json +0 -0
  26. results/g15-navajo-distill-s0.json +0 -0
  27. results/g15-navajo-distill-s1.json +0 -0
  28. results/g15-navajo-matched-s0.json +0 -0
  29. results/g15-navajo-matched-s1.json +0 -0
  30. results/g15-russian-distill-s0.json +0 -0
  31. results/g15-russian-distill-s1.json +0 -0
  32. results/g15-russian-matched-s0.json +0 -0
  33. results/g15-russian-matched-s1.json +0 -0
  34. results/g15-turkish-distill-s0.json +0 -0
  35. results/g15-turkish-distill-s1.json +0 -0
  36. results/g15-turkish-matched-s0.json +0 -0
  37. results/g15-turkish-matched-s1.json +0 -0
  38. results/g17-d16-mdlm2x-s0.json +0 -0
  39. results/g17-d16-mdlm2x-s1.json +0 -0
  40. results/g17-d20-mdlm2x-s0.json +0 -0
  41. results/g17-d20-mdlm2x-s1.json +0 -0
  42. results/g18-d20-anybudget-s0.json +0 -0
  43. results/g18-d20-anybudget-s1.json +0 -0
  44. results/g18-d20-anybudget-s2.json +0 -0
  45. results/g18-d20-anybudget_nc-s0.json +0 -0
  46. results/g18-d20-anybudget_nc-s1.json +0 -0
  47. results/g18-d20-anybudget_nc-s2.json +0 -0
  48. results/gr1-2regular-anybudget-s0.json +0 -0
  49. results/gr1-2regular-anybudget-s1.json +0 -0
  50. results/gr1-2regular-mdlm-s0.json +0 -0
README.md CHANGED
@@ -1,15 +1,9 @@
1
  ---
2
  license: apache-2.0
3
- tags: [masked-diffusion, discrete-diffusion, length-generalization, positional-encoding, from-scratch]
4
  ---
5
 
6
- # Length Generalization in Masked Diffusion Language Models
7
-
8
- From-scratch masked diffusion language models (MDLM-style absorbing state) and
9
- matched autoregressive baselines, trained on arithmetic and algorithmic tasks and
10
- evaluated far beyond the lengths seen in training.
11
-
12
- **No pretrained weights or tokenizers are used anywhere.**
13
 
14
  Code: https://github.com/GIND123/MSML612
15
 
@@ -40,19 +34,28 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
40
  | english | 95.3 | 95.2 | -0.1 |
41
  | finnish | 50.9 | 55.8 | +4.9 |
42
  | finnish | 48.6 | 56.6 | +8.0 |
 
 
43
  | finnish | 50.9 | 55.8 | +4.9 |
44
  | finnish | 52.9 | 57.9 | +4.9 |
 
 
45
  | finnish | 48.6 | 56.6 | +8.0 |
46
  | finnish | 54.3 | 61.7 | +7.4 |
47
  | georgian | 94.3 | 94.1 | -0.2 |
48
  | georgian | 95.4 | 95.3 | -0.1 |
 
 
49
  | georgian | 94.3 | 94.1 | -0.2 |
50
  | georgian | 93.8 | 94.0 | +0.1 |
 
 
51
  | georgian | 95.4 | 95.3 | -0.1 |
52
  | georgian | 93.9 | 93.9 | +0.0 |
53
  | german | 78.9 | 79.3 | +0.4 |
54
  | german | 75.6 | 79.5 | +3.9 |
55
  | german | 78.6 | 78.9 | +0.3 |
 
56
  | german | 78.9 | 79.3 | +0.4 |
57
  | german | 77.9 | 78.0 | +0.1 |
58
  | german | 79.4 | 81.0 | +1.7 |
@@ -62,22 +65,34 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
62
  | german | 77.2 | 78.3 | +1.0 |
63
  | navajo | 29.5 | 33.8 | +4.3 |
64
  | navajo | 31.4 | 40.2 | +8.9 |
 
 
65
  | navajo | 29.5 | 33.8 | +4.3 |
66
  | navajo | 24.3 | 29.0 | +4.8 |
 
 
67
  | navajo | 31.4 | 40.2 | +8.9 |
68
  | navajo | 24.6 | 31.7 | +7.1 |
69
  | russian | 72.7 | 72.0 | -0.7 |
70
  | russian | 69.1 | 71.7 | +2.6 |
 
 
71
  | russian | 72.7 | 72.0 | -0.7 |
72
  | russian | 71.9 | 72.0 | +0.1 |
 
 
73
  | russian | 69.1 | 71.7 | +2.6 |
74
  | russian | 74.1 | 75.9 | +1.8 |
75
  | spanish | 84.9 | 84.7 | -0.2 |
76
  | spanish | 83.9 | 84.1 | +0.1 |
77
  | turkish | 80.2 | 81.2 | +1.0 |
78
  | turkish | 77.5 | 82.2 | +4.7 |
 
 
79
  | turkish | 80.2 | 81.2 | +1.0 |
80
  | turkish | 84.5 | 85.1 | +0.7 |
 
 
81
  | turkish | 77.5 | 82.2 | +4.7 |
82
  | turkish | 79.9 | 82.2 | +2.3 |
83
  | turkish | 76.4 | 81.8 | +5.4 |
@@ -112,19 +127,73 @@ The account in THEORY.md predicts minimum depth growing as log2(n), since the ca
112
  | digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
113
  |---|---|---|---|---|---|---|---|---|
114
  | **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
115
- | **8** | 99 | 100 | 100 | 100 | 100 | – | 3.0 | 2 |
116
- | **12** | 48 | 100 | 100 | – | – | – | 3.6 | 3 |
 
117
 
118
  ## Table 7 — Which component carries the gain (inflection)
119
 
120
  | language | mdlm | matched | distill | full | (gap 1→32) |
121
  |---|---|---|---|---|---|
122
- | finnish | 51.4 | – | – | 51.9 | +7.7 / – / – / +4.9 |
123
- | georgian | 94.6 | – | – | 94.1 | -0.0 / – / – / -0.0 |
124
- | german | 76.0 | 77.9 | 78.6 | 78.4 | +3.2 / +1.4 / +0.3 / +0.2 |
125
- | navajo | 28.0 | – | – | 26.9 | +8.0 / – / – / +4.5 |
126
- | russian | 71.6 | – | – | 72.3 | +2.2 / – / – / -0.3 |
127
- | turkish | 78.7 | – | – | 82.3 | +3.5 / – / – / +0.8 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
128
 
129
  ## Table 9 — Did the baseline fail to parallelise, or fail to learn?
130
 
 
1
  ---
2
  license: apache-2.0
3
+ tags: [masked-diffusion, parallel-decoding, text8, qm9, from-scratch]
4
  ---
5
 
6
+ # Two Failure Modes of Parallel Decoding in Masked Diffusion
 
 
 
 
 
 
7
 
8
  Code: https://github.com/GIND123/MSML612
9
 
 
34
  | english | 95.3 | 95.2 | -0.1 |
35
  | finnish | 50.9 | 55.8 | +4.9 |
36
  | finnish | 48.6 | 56.6 | +8.0 |
37
+ | finnish | 48.3 | 53.5 | +5.3 |
38
+ | finnish | 50.6 | 55.6 | +4.9 |
39
  | finnish | 50.9 | 55.8 | +4.9 |
40
  | finnish | 52.9 | 57.9 | +4.9 |
41
+ | finnish | 50.3 | 57.0 | +6.7 |
42
+ | finnish | 53.6 | 60.9 | +7.3 |
43
  | finnish | 48.6 | 56.6 | +8.0 |
44
  | finnish | 54.3 | 61.7 | +7.4 |
45
  | georgian | 94.3 | 94.1 | -0.2 |
46
  | georgian | 95.4 | 95.3 | -0.1 |
47
+ | georgian | 94.8 | 94.5 | -0.3 |
48
+ | georgian | 93.9 | 93.8 | -0.1 |
49
  | georgian | 94.3 | 94.1 | -0.2 |
50
  | georgian | 93.8 | 94.0 | +0.1 |
51
+ | georgian | 94.4 | 94.9 | +0.5 |
52
+ | georgian | 94.1 | 93.8 | -0.3 |
53
  | georgian | 95.4 | 95.3 | -0.1 |
54
  | georgian | 93.9 | 93.9 | +0.0 |
55
  | german | 78.9 | 79.3 | +0.4 |
56
  | german | 75.6 | 79.5 | +3.9 |
57
  | german | 78.6 | 78.9 | +0.3 |
58
+ | german | 77.9 | 78.5 | +0.7 |
59
  | german | 78.9 | 79.3 | +0.4 |
60
  | german | 77.9 | 78.0 | +0.1 |
61
  | german | 79.4 | 81.0 | +1.7 |
 
65
  | german | 77.2 | 78.3 | +1.0 |
66
  | navajo | 29.5 | 33.8 | +4.3 |
67
  | navajo | 31.4 | 40.2 | +8.9 |
68
+ | navajo | 33.7 | 39.0 | +5.2 |
69
+ | navajo | 24.2 | 30.8 | +6.6 |
70
  | navajo | 29.5 | 33.8 | +4.3 |
71
  | navajo | 24.3 | 29.0 | +4.8 |
72
+ | navajo | 31.1 | 39.3 | +8.2 |
73
+ | navajo | 26.3 | 34.0 | +7.7 |
74
  | navajo | 31.4 | 40.2 | +8.9 |
75
  | navajo | 24.6 | 31.7 | +7.1 |
76
  | russian | 72.7 | 72.0 | -0.7 |
77
  | russian | 69.1 | 71.7 | +2.6 |
78
+ | russian | 73.5 | 73.0 | -0.5 |
79
+ | russian | 74.2 | 74.5 | +0.4 |
80
  | russian | 72.7 | 72.0 | -0.7 |
81
  | russian | 71.9 | 72.0 | +0.1 |
82
+ | russian | 73.1 | 73.3 | +0.3 |
83
+ | russian | 76.3 | 76.2 | -0.1 |
84
  | russian | 69.1 | 71.7 | +2.6 |
85
  | russian | 74.1 | 75.9 | +1.8 |
86
  | spanish | 84.9 | 84.7 | -0.2 |
87
  | spanish | 83.9 | 84.1 | +0.1 |
88
  | turkish | 80.2 | 81.2 | +1.0 |
89
  | turkish | 77.5 | 82.2 | +4.7 |
90
+ | turkish | 77.3 | 78.9 | +1.6 |
91
+ | turkish | 83.1 | 83.9 | +0.8 |
92
  | turkish | 80.2 | 81.2 | +1.0 |
93
  | turkish | 84.5 | 85.1 | +0.7 |
94
+ | turkish | 77.0 | 80.9 | +3.9 |
95
+ | turkish | 80.4 | 83.8 | +3.4 |
96
  | turkish | 77.5 | 82.2 | +4.7 |
97
  | turkish | 79.9 | 82.2 | +2.3 |
98
  | turkish | 76.4 | 81.8 | +5.4 |
 
127
  | digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
128
  |---|---|---|---|---|---|---|---|---|
129
  | **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
130
+ | **8** | 99 | 100 | 100 | 100 | 100 | 100 | 3.0 | 2 |
131
+ | **12** | 48 | 100 | 100 | 100 | – | – | 3.6 | 3 |
132
+ | **16** | 0 | 50 | – | – | – | – | 4.0 | – |
133
 
134
  ## Table 7 — Which component carries the gain (inflection)
135
 
136
  | language | mdlm | matched | distill | full | (gap 1→32) |
137
  |---|---|---|---|---|---|
138
+ | finnish | 51.4 | 52.0 | 49.4 | 51.9 | +7.7 / +7.0 / +5.1 / +4.9 |
139
+ | georgian | 94.6 | 94.3 | 94.3 | 94.1 | -0.0 / +0.1 / -0.2 / -0.0 |
140
+ | german | 76.0 | 77.9 | 78.2 | 78.4 | +3.2 / +1.4 / +0.5 / +0.2 |
141
+ | navajo | 28.0 | 28.7 | 29.0 | 26.9 | +8.0 / +8.0 / +5.9 / +4.5 |
142
+ | russian | 71.6 | 74.7 | 73.9 | 72.3 | +2.2 / +0.1 / -0.1 / -0.3 |
143
+ | turkish | 78.7 | 78.7 | 80.2 | 82.3 | +3.5 / +3.7 / +1.2 / +0.8 |
144
+
145
+ ## Table 8 — text8: quality against compute
146
+
147
+ Generative perplexity under a from-scratch autoregressive character evaluator, against the number of function evaluations actually used. Entropy and distinct-4-gram rate are reported beside it because generative perplexity alone is gamed by degenerate repetition; the real-text reference row is the target, not zero.
148
+
149
+ Real text8 reference — ppl 2.86, entropy 2.855, distinct-4 0.177
150
+
151
+ | training | decoding | NFE | gen ppl | entropy | distinct-4 | valid BPC |
152
+ |---|---|---|---|---|---|---|
153
+ | mdlm | conf_0.5 | 234.5 | 32.99 | 2.871 | 0.360 | 1.7225 |
154
+ | mdlm | conf_0.7 | 248.5 | 16.99 | 2.866 | 0.295 | 1.7225 |
155
+ | mdlm | conf_0.9 | 252.8 | 9.78 | 2.848 | 0.245 | 1.7225 |
156
+ | mdlm | conf_0.95 | 254.0 | 9.08 | 2.856 | 0.238 | 1.7225 |
157
+ | mdlm | conf_0.99 | 255.5 | 8.74 | 2.851 | 0.236 | 1.7225 |
158
+ | mdlm | entbudget_0.05 | 256.0 | 9.47 | 2.854 | 0.241 | 1.7225 |
159
+ | mdlm | entbudget_0.1 | 255.5 | 8.79 | 2.861 | 0.241 | 1.7225 |
160
+ | mdlm | entbudget_0.25 | 255.0 | 8.93 | 2.858 | 0.239 | 1.7225 |
161
+ | mdlm | entbudget_0.5 | 253.2 | 9.39 | 2.858 | 0.243 | 1.7225 |
162
+ | mdlm | entbudget_1.0 | 248.2 | 10.39 | 2.857 | 0.248 | 1.7225 |
163
+ | mdlm | entbudget_2.0 | 245.0 | 15.44 | 2.861 | 0.279 | 1.7225 |
164
+ | mdlm | entbudget_4.0 | 223.2 | 51.18 | 2.882 | 0.401 | 1.7225 |
165
+ | mdlm | entbudget_8.0 | 96.5 | 111.68 | 2.888 | 0.520 | 1.7225 |
166
+ | mdlm | fixedK_1 | 1.0 | 646.96 | 2.860 | 0.630 | 1.7225 |
167
+ | mdlm | fixedK_128 | 128.0 | 44.33 | 2.867 | 0.410 | 1.7225 |
168
+ | mdlm | fixedK_16 | 16.0 | 126.50 | 2.878 | 0.532 | 1.7225 |
169
+ | mdlm | fixedK_2 | 2.0 | 310.58 | 2.890 | 0.625 | 1.7225 |
170
+ | mdlm | fixedK_256 | 256.0 | 8.98 | 2.854 | 0.241 | 1.7225 |
171
+ | mdlm | fixedK_32 | 32.0 | 95.60 | 2.875 | 0.497 | 1.7225 |
172
+ | mdlm | fixedK_4 | 4.0 | 155.18 | 2.881 | 0.550 | 1.7225 |
173
+ | mdlm | fixedK_64 | 64.0 | 86.83 | 2.882 | 0.490 | 1.7225 |
174
+ | mdlm | fixedK_8 | 8.0 | 121.66 | 2.879 | 0.527 | 1.7225 |
175
+ | matched | conf_0.5 | 237.5 | 32.44 | 2.873 | 0.366 | 1.6946 |
176
+ | matched | conf_0.7 | 242.0 | 14.86 | 2.854 | 0.282 | 1.6946 |
177
+ | matched | conf_0.9 | 249.2 | 9.67 | 2.856 | 0.248 | 1.6946 |
178
+ | matched | conf_0.95 | 251.2 | 8.65 | 2.855 | 0.241 | 1.6946 |
179
+ | matched | conf_0.99 | 254.8 | 8.34 | 2.853 | 0.236 | 1.6946 |
180
+ | matched | entbudget_0.05 | 256.0 | 8.39 | 2.857 | 0.241 | 1.6946 |
181
+ | matched | entbudget_0.1 | 256.0 | 8.50 | 2.855 | 0.238 | 1.6946 |
182
+ | matched | entbudget_0.25 | 255.0 | 8.72 | 2.857 | 0.243 | 1.6946 |
183
+ | matched | entbudget_0.5 | 253.2 | 9.00 | 2.863 | 0.247 | 1.6946 |
184
+ | matched | entbudget_1.0 | 251.5 | 10.02 | 2.861 | 0.255 | 1.6946 |
185
+ | matched | entbudget_2.0 | 247.2 | 14.98 | 2.862 | 0.284 | 1.6946 |
186
+ | matched | entbudget_4.0 | 235.2 | 47.92 | 2.885 | 0.403 | 1.6946 |
187
+ | matched | entbudget_8.0 | 98.0 | 104.65 | 2.895 | 0.526 | 1.6946 |
188
+ | matched | fixedK_1 | 1.0 | 651.67 | 2.860 | 0.631 | 1.6946 |
189
+ | matched | fixedK_128 | 128.0 | 42.38 | 2.869 | 0.407 | 1.6946 |
190
+ | matched | fixedK_16 | 16.0 | 94.52 | 2.868 | 0.502 | 1.6946 |
191
+ | matched | fixedK_2 | 2.0 | 295.79 | 2.881 | 0.620 | 1.6946 |
192
+ | matched | fixedK_256 | 256.0 | 8.19 | 2.852 | 0.235 | 1.6946 |
193
+ | matched | fixedK_32 | 32.0 | 93.33 | 2.882 | 0.510 | 1.6946 |
194
+ | matched | fixedK_4 | 4.0 | 146.64 | 2.880 | 0.558 | 1.6946 |
195
+ | matched | fixedK_64 | 64.0 | 76.89 | 2.884 | 0.484 | 1.6946 |
196
+ | matched | fixedK_8 | 8.0 | 123.33 | 2.876 | 0.540 | 1.6946 |
197
 
198
  ## Table 9 — Did the baseline fail to parallelise, or fail to learn?
199
 
figures/beat_d12.json CHANGED
@@ -31,10 +31,10 @@
31
  ],
32
  "random": [
33
  0.766875,
34
- 0.764375,
35
  0.768125,
36
- 0.770625,
37
- 0.76625
 
38
  ]
39
  },
40
  "ours": {
@@ -46,10 +46,61 @@
46
  0.999375
47
  ]
48
  },
49
- "significance": {},
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
  "summary": {
51
  "best_inference_1pass": 0.766875,
52
- "best_inference_any_budget": 0.770625,
53
  "ours_1pass": 1.0
54
  }
55
  }
 
31
  ],
32
  "random": [
33
  0.766875,
 
34
  0.768125,
35
+ 0.77125,
36
+ 0.764375,
37
+ 0.77
38
  ]
39
  },
40
  "ours": {
 
46
  0.999375
47
  ]
48
  },
49
+ "significance": {
50
+ "confidence": {
51
+ "diff": 0.233125,
52
+ "ci95": [
53
+ 0.226875,
54
+ 0.23875
55
+ ],
56
+ "p_value": 0.0,
57
+ "n": 400,
58
+ "their_best_budget": 1
59
+ },
60
+ "entropy": {
61
+ "diff": 0.233125,
62
+ "ci95": [
63
+ 0.226875,
64
+ 0.23875
65
+ ],
66
+ "p_value": 0.0,
67
+ "n": 400,
68
+ "their_best_budget": 1
69
+ },
70
+ "entropy-gated": {
71
+ "diff": 0.233125,
72
+ "ci95": [
73
+ 0.226875,
74
+ 0.23875
75
+ ],
76
+ "p_value": 0.0,
77
+ "n": 400,
78
+ "their_best_budget": 1
79
+ },
80
+ "margin": {
81
+ "diff": 0.233125,
82
+ "ci95": [
83
+ 0.226875,
84
+ 0.23875
85
+ ],
86
+ "p_value": 0.0,
87
+ "n": 400,
88
+ "their_best_budget": 1
89
+ },
90
+ "random": {
91
+ "diff": 0.22875,
92
+ "ci95": [
93
+ 0.221875,
94
+ 0.235625
95
+ ],
96
+ "p_value": 0.0,
97
+ "n": 400,
98
+ "their_best_budget": 4
99
+ }
100
+ },
101
  "summary": {
102
  "best_inference_1pass": 0.766875,
103
+ "best_inference_any_budget": 0.77125,
104
  "ours_1pass": 1.0
105
  }
106
  }
figures/beat_d16.json CHANGED
@@ -33,8 +33,8 @@
33
  0.25,
34
  0.25,
35
  0.25,
36
- 0.249375,
37
- 0.25
38
  ]
39
  },
40
  "ours": {
@@ -46,7 +46,58 @@
46
  0.749375
47
  ]
48
  },
49
- "significance": {},
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
  "summary": {
51
  "best_inference_1pass": 0.25,
52
  "best_inference_any_budget": 0.25,
 
33
  0.25,
34
  0.25,
35
  0.25,
36
+ 0.25,
37
+ 0.249375
38
  ]
39
  },
40
  "ours": {
 
46
  0.749375
47
  ]
48
  },
49
+ "significance": {
50
+ "confidence": {
51
+ "diff": 0.5,
52
+ "ci95": [
53
+ 0.5,
54
+ 0.5
55
+ ],
56
+ "p_value": 0.0,
57
+ "n": 400,
58
+ "their_best_budget": 1
59
+ },
60
+ "entropy": {
61
+ "diff": 0.5,
62
+ "ci95": [
63
+ 0.5,
64
+ 0.5
65
+ ],
66
+ "p_value": 0.0,
67
+ "n": 400,
68
+ "their_best_budget": 1
69
+ },
70
+ "entropy-gated": {
71
+ "diff": 0.5,
72
+ "ci95": [
73
+ 0.5,
74
+ 0.5
75
+ ],
76
+ "p_value": 0.0,
77
+ "n": 400,
78
+ "their_best_budget": 1
79
+ },
80
+ "margin": {
81
+ "diff": 0.5,
82
+ "ci95": [
83
+ 0.5,
84
+ 0.5
85
+ ],
86
+ "p_value": 0.0,
87
+ "n": 400,
88
+ "their_best_budget": 1
89
+ },
90
+ "random": {
91
+ "diff": 0.5,
92
+ "ci95": [
93
+ 0.5,
94
+ 0.5
95
+ ],
96
+ "p_value": 0.0,
97
+ "n": 400,
98
+ "their_best_budget": 1
99
+ }
100
+ },
101
  "summary": {
102
  "best_inference_1pass": 0.25,
103
  "best_inference_any_budget": 0.25,
figures/fig3_depth_scaling.png CHANGED
figures/fig4_text8_quality_vs_compute.png ADDED
figures/graph_ceiling.json ADDED
File without changes
figures/summary.md CHANGED
@@ -23,19 +23,28 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
23
  | english | 95.3 | 95.2 | -0.1 |
24
  | finnish | 50.9 | 55.8 | +4.9 |
25
  | finnish | 48.6 | 56.6 | +8.0 |
 
 
26
  | finnish | 50.9 | 55.8 | +4.9 |
27
  | finnish | 52.9 | 57.9 | +4.9 |
 
 
28
  | finnish | 48.6 | 56.6 | +8.0 |
29
  | finnish | 54.3 | 61.7 | +7.4 |
30
  | georgian | 94.3 | 94.1 | -0.2 |
31
  | georgian | 95.4 | 95.3 | -0.1 |
 
 
32
  | georgian | 94.3 | 94.1 | -0.2 |
33
  | georgian | 93.8 | 94.0 | +0.1 |
 
 
34
  | georgian | 95.4 | 95.3 | -0.1 |
35
  | georgian | 93.9 | 93.9 | +0.0 |
36
  | german | 78.9 | 79.3 | +0.4 |
37
  | german | 75.6 | 79.5 | +3.9 |
38
  | german | 78.6 | 78.9 | +0.3 |
 
39
  | german | 78.9 | 79.3 | +0.4 |
40
  | german | 77.9 | 78.0 | +0.1 |
41
  | german | 79.4 | 81.0 | +1.7 |
@@ -45,22 +54,34 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
45
  | german | 77.2 | 78.3 | +1.0 |
46
  | navajo | 29.5 | 33.8 | +4.3 |
47
  | navajo | 31.4 | 40.2 | +8.9 |
 
 
48
  | navajo | 29.5 | 33.8 | +4.3 |
49
  | navajo | 24.3 | 29.0 | +4.8 |
 
 
50
  | navajo | 31.4 | 40.2 | +8.9 |
51
  | navajo | 24.6 | 31.7 | +7.1 |
52
  | russian | 72.7 | 72.0 | -0.7 |
53
  | russian | 69.1 | 71.7 | +2.6 |
 
 
54
  | russian | 72.7 | 72.0 | -0.7 |
55
  | russian | 71.9 | 72.0 | +0.1 |
 
 
56
  | russian | 69.1 | 71.7 | +2.6 |
57
  | russian | 74.1 | 75.9 | +1.8 |
58
  | spanish | 84.9 | 84.7 | -0.2 |
59
  | spanish | 83.9 | 84.1 | +0.1 |
60
  | turkish | 80.2 | 81.2 | +1.0 |
61
  | turkish | 77.5 | 82.2 | +4.7 |
 
 
62
  | turkish | 80.2 | 81.2 | +1.0 |
63
  | turkish | 84.5 | 85.1 | +0.7 |
 
 
64
  | turkish | 77.5 | 82.2 | +4.7 |
65
  | turkish | 79.9 | 82.2 | +2.3 |
66
  | turkish | 76.4 | 81.8 | +5.4 |
@@ -95,28 +116,61 @@ The account in THEORY.md predicts minimum depth growing as log2(n), since the ca
95
  | digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
96
  |---|---|---|---|---|---|---|---|---|
97
  | **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
98
- | **8** | 99 | 100 | 100 | 100 | 100 | – | 3.0 | 2 |
99
- | **12** | 48 | 100 | 100 | – | – | – | 3.6 | 3 |
 
100
 
101
  ## Table 7 — Which component carries the gain (inflection)
102
 
103
  | language | mdlm | matched | distill | full | (gap 1→32) |
104
  |---|---|---|---|---|---|
105
- | finnish | 51.4 | – | – | 51.9 | +7.7 / – / – / +4.9 |
106
- | georgian | 94.6 | – | – | 94.1 | -0.0 / – / – / -0.0 |
107
- | german | 76.0 | 77.9 | 78.6 | 78.4 | +3.2 / +1.4 / +0.3 / +0.2 |
108
- | navajo | 28.0 | – | – | 26.9 | +8.0 / – / – / +4.5 |
109
- | russian | 71.6 | – | – | 72.3 | +2.2 / – / – / -0.3 |
110
- | turkish | 78.7 | – | – | 82.3 | +3.5 / – / – / +0.8 |
111
-
112
- ## Table 9 — Did the baseline fail to parallelise, or fail to learn?
113
-
114
- A 20-digit answer occupies 21 slots, so the standard 16-pass grid is not a sequential decode. Re-decoded at one token per pass, the two explanations separate.
115
-
116
- | digits | mode | @1 pass | fully sequential | reading |
117
- |---|---|---|---|---|
118
- | 20 | full | 99.8 | 85.7 | learned it and parallelises |
119
- | 20 | mdlm | 0.0 | 0.0 | never learned the task |
120
- | 24 | full | 33.7 | 33.3 | learned it and parallelises |
121
- | 24 | mdlm | 0.0 | 0.0 | never learned the task |
122
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
  | english | 95.3 | 95.2 | -0.1 |
24
  | finnish | 50.9 | 55.8 | +4.9 |
25
  | finnish | 48.6 | 56.6 | +8.0 |
26
+ | finnish | 48.3 | 53.5 | +5.3 |
27
+ | finnish | 50.6 | 55.6 | +4.9 |
28
  | finnish | 50.9 | 55.8 | +4.9 |
29
  | finnish | 52.9 | 57.9 | +4.9 |
30
+ | finnish | 50.3 | 57.0 | +6.7 |
31
+ | finnish | 53.6 | 60.9 | +7.3 |
32
  | finnish | 48.6 | 56.6 | +8.0 |
33
  | finnish | 54.3 | 61.7 | +7.4 |
34
  | georgian | 94.3 | 94.1 | -0.2 |
35
  | georgian | 95.4 | 95.3 | -0.1 |
36
+ | georgian | 94.8 | 94.5 | -0.3 |
37
+ | georgian | 93.9 | 93.8 | -0.1 |
38
  | georgian | 94.3 | 94.1 | -0.2 |
39
  | georgian | 93.8 | 94.0 | +0.1 |
40
+ | georgian | 94.4 | 94.9 | +0.5 |
41
+ | georgian | 94.1 | 93.8 | -0.3 |
42
  | georgian | 95.4 | 95.3 | -0.1 |
43
  | georgian | 93.9 | 93.9 | +0.0 |
44
  | german | 78.9 | 79.3 | +0.4 |
45
  | german | 75.6 | 79.5 | +3.9 |
46
  | german | 78.6 | 78.9 | +0.3 |
47
+ | german | 77.9 | 78.5 | +0.7 |
48
  | german | 78.9 | 79.3 | +0.4 |
49
  | german | 77.9 | 78.0 | +0.1 |
50
  | german | 79.4 | 81.0 | +1.7 |
 
54
  | german | 77.2 | 78.3 | +1.0 |
55
  | navajo | 29.5 | 33.8 | +4.3 |
56
  | navajo | 31.4 | 40.2 | +8.9 |
57
+ | navajo | 33.7 | 39.0 | +5.2 |
58
+ | navajo | 24.2 | 30.8 | +6.6 |
59
  | navajo | 29.5 | 33.8 | +4.3 |
60
  | navajo | 24.3 | 29.0 | +4.8 |
61
+ | navajo | 31.1 | 39.3 | +8.2 |
62
+ | navajo | 26.3 | 34.0 | +7.7 |
63
  | navajo | 31.4 | 40.2 | +8.9 |
64
  | navajo | 24.6 | 31.7 | +7.1 |
65
  | russian | 72.7 | 72.0 | -0.7 |
66
  | russian | 69.1 | 71.7 | +2.6 |
67
+ | russian | 73.5 | 73.0 | -0.5 |
68
+ | russian | 74.2 | 74.5 | +0.4 |
69
  | russian | 72.7 | 72.0 | -0.7 |
70
  | russian | 71.9 | 72.0 | +0.1 |
71
+ | russian | 73.1 | 73.3 | +0.3 |
72
+ | russian | 76.3 | 76.2 | -0.1 |
73
  | russian | 69.1 | 71.7 | +2.6 |
74
  | russian | 74.1 | 75.9 | +1.8 |
75
  | spanish | 84.9 | 84.7 | -0.2 |
76
  | spanish | 83.9 | 84.1 | +0.1 |
77
  | turkish | 80.2 | 81.2 | +1.0 |
78
  | turkish | 77.5 | 82.2 | +4.7 |
79
+ | turkish | 77.3 | 78.9 | +1.6 |
80
+ | turkish | 83.1 | 83.9 | +0.8 |
81
  | turkish | 80.2 | 81.2 | +1.0 |
82
  | turkish | 84.5 | 85.1 | +0.7 |
83
+ | turkish | 77.0 | 80.9 | +3.9 |
84
+ | turkish | 80.4 | 83.8 | +3.4 |
85
  | turkish | 77.5 | 82.2 | +4.7 |
86
  | turkish | 79.9 | 82.2 | +2.3 |
87
  | turkish | 76.4 | 81.8 | +5.4 |
 
116
  | digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
117
  |---|---|---|---|---|---|---|---|---|
118
  | **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
119
+ | **8** | 99 | 100 | 100 | 100 | 100 | 100 | 3.0 | 2 |
120
+ | **12** | 48 | 100 | 100 | 100 | – | – | 3.6 | 3 |
121
+ | **16** | 0 | 50 | – | – | – | – | 4.0 | – |
122
 
123
  ## Table 7 — Which component carries the gain (inflection)
124
 
125
  | language | mdlm | matched | distill | full | (gap 1→32) |
126
  |---|---|---|---|---|---|
127
+ | finnish | 51.4 | 52.0 | 49.4 | 51.9 | +7.7 / +7.0 / +5.1 / +4.9 |
128
+ | georgian | 94.6 | 94.3 | 94.3 | 94.1 | -0.0 / +0.1 / -0.2 / -0.0 |
129
+ | german | 76.0 | 77.9 | 78.2 | 78.4 | +3.2 / +1.4 / +0.5 / +0.2 |
130
+ | navajo | 28.0 | 28.7 | 29.0 | 26.9 | +8.0 / +8.0 / +5.9 / +4.5 |
131
+ | russian | 71.6 | 74.7 | 73.9 | 72.3 | +2.2 / +0.1 / -0.1 / -0.3 |
132
+ | turkish | 78.7 | 78.7 | 80.2 | 82.3 | +3.5 / +3.7 / +1.2 / +0.8 |
133
+
134
+ ## Table 8 — text8: quality against compute
135
+
136
+ Generative perplexity under a from-scratch autoregressive character evaluator, against the number of function evaluations actually used. Entropy and distinct-4-gram rate are reported beside it because generative perplexity alone is gamed by degenerate repetition; the real-text reference row is the target, not zero.
137
+
138
+ Real text8 reference — ppl 2.86, entropy 2.855, distinct-4 0.177
139
+
140
+ | training | decoding | NFE | gen ppl | entropy | distinct-4 | valid BPC |
141
+ |---|---|---|---|---|---|---|
142
+ | mdlm | conf_0.5 | 234.5 | 32.99 | 2.871 | 0.360 | 1.7225 |
143
+ | mdlm | conf_0.7 | 248.5 | 16.99 | 2.866 | 0.295 | 1.7225 |
144
+ | mdlm | conf_0.9 | 252.8 | 9.78 | 2.848 | 0.245 | 1.7225 |
145
+ | mdlm | conf_0.95 | 254.0 | 9.08 | 2.856 | 0.238 | 1.7225 |
146
+ | mdlm | conf_0.99 | 255.5 | 8.74 | 2.851 | 0.236 | 1.7225 |
147
+ | mdlm | entbudget_0.05 | 256.0 | 9.47 | 2.854 | 0.241 | 1.7225 |
148
+ | mdlm | entbudget_0.1 | 255.5 | 8.79 | 2.861 | 0.241 | 1.7225 |
149
+ | mdlm | entbudget_0.25 | 255.0 | 8.93 | 2.858 | 0.239 | 1.7225 |
150
+ | mdlm | entbudget_0.5 | 253.2 | 9.39 | 2.858 | 0.243 | 1.7225 |
151
+ | mdlm | entbudget_1.0 | 248.2 | 10.39 | 2.857 | 0.248 | 1.7225 |
152
+ | mdlm | entbudget_2.0 | 245.0 | 15.44 | 2.861 | 0.279 | 1.7225 |
153
+ | mdlm | entbudget_4.0 | 223.2 | 51.18 | 2.882 | 0.401 | 1.7225 |
154
+ | mdlm | entbudget_8.0 | 96.5 | 111.68 | 2.888 | 0.520 | 1.7225 |
155
+ | mdlm | fixedK_1 | 1.0 | 646.96 | 2.860 | 0.630 | 1.7225 |
156
+ | mdlm | fixedK_128 | 128.0 | 44.33 | 2.867 | 0.410 | 1.7225 |
157
+ | mdlm | fixedK_16 | 16.0 | 126.50 | 2.878 | 0.532 | 1.7225 |
158
+ | mdlm | fixedK_2 | 2.0 | 310.58 | 2.890 | 0.625 | 1.7225 |
159
+ | mdlm | fixedK_256 | 256.0 | 8.98 | 2.854 | 0.241 | 1.7225 |
160
+ | mdlm | fixedK_32 | 32.0 | 95.60 | 2.875 | 0.497 | 1.7225 |
161
+ | mdlm | fixedK_4 | 4.0 | 155.18 | 2.881 | 0.550 | 1.7225 |
162
+ | mdlm | fixedK_64 | 64.0 | 86.83 | 2.882 | 0.490 | 1.7225 |
163
+ | mdlm | fixedK_8 | 8.0 | 121.66 | 2.879 | 0.527 | 1.7225 |
164
+ | matched | conf_0.5 | 237.5 | 32.44 | 2.873 | 0.366 | 1.6946 |
165
+ | matched | conf_0.7 | 242.0 | 14.86 | 2.854 | 0.282 | 1.6946 |
166
+ | matched | conf_0.9 | 249.2 | 9.67 | 2.856 | 0.248 | 1.6946 |
167
+ | matched | conf_0.95 | 251.2 | 8.65 | 2.855 | 0.241 | 1.6946 |
168
+ | matched | conf_0.99 | 254.8 | 8.34 | 2.853 | 0.236 | 1.6946 |
169
+ | matched | entbudget_0.05 | 256.0 | 8.39 | 2.857 | 0.241 | 1.6946 |
170
+ | matched | entbudget_0.1 | 256.0 | 8.50 | 2.855 | 0.238 | 1.6946 |
171
+ | matched | entbudget_0.25 | 255.0 | 8.72 | 2.857 | 0.243 | 1.6946 |
172
+ | matched | entbudget_0.5 | 253.2 | 9.00 | 2.863 | 0.247 | 1.6946 |
173
+ | matched | entbudget_1.0 | 251.5 | 10.02 | 2.861 | 0.255 | 1.6946 |
174
+ | matched | entbudget_2.0 | 247.2 | 14.98 | 2.862 | 0.284 | 1.6946 |
175
+ | matched | entbudget_4.0 | 235.2 | 47.92 | 2.885 | 0.403 | 1.6946 |
176
+ | matched
results/evallm.json ADDED
File without changes
results/g14-d12-L6-s0.json ADDED
File without changes
results/g14-d12-L6-s1.json ADDED
File without changes
results/g14-d16-L2-s0.json ADDED
File without changes
results/g14-d16-L2-s1.json ADDED
File without changes
results/g14-d16-L3-s0.json ADDED
File without changes
results/g14-d16-L3-s1.json ADDED
File without changes
results/g14-d8-L12-s0.json ADDED
File without changes
results/g14-d8-L12-s1.json ADDED
File without changes
results/g15-finnish-distill-s0.json ADDED
File without changes
results/g15-finnish-distill-s1.json ADDED
File without changes
results/g15-finnish-matched-s0.json ADDED
File without changes
results/g15-finnish-matched-s1.json ADDED
File without changes
results/g15-georgian-distill-s0.json ADDED
File without changes
results/g15-georgian-distill-s1.json ADDED
File without changes
results/g15-georgian-matched-s0.json ADDED
File without changes
results/g15-georgian-matched-s1.json ADDED
File without changes
results/g15-german-distill-s1.json ADDED
File without changes
results/g15-navajo-distill-s0.json ADDED
File without changes
results/g15-navajo-distill-s1.json ADDED
File without changes
results/g15-navajo-matched-s0.json ADDED
File without changes
results/g15-navajo-matched-s1.json ADDED
File without changes
results/g15-russian-distill-s0.json ADDED
File without changes
results/g15-russian-distill-s1.json ADDED
File without changes
results/g15-russian-matched-s0.json ADDED
File without changes
results/g15-russian-matched-s1.json ADDED
File without changes
results/g15-turkish-distill-s0.json ADDED
File without changes
results/g15-turkish-distill-s1.json ADDED
File without changes
results/g15-turkish-matched-s0.json ADDED
File without changes
results/g15-turkish-matched-s1.json ADDED
File without changes
results/g17-d16-mdlm2x-s0.json ADDED
File without changes
results/g17-d16-mdlm2x-s1.json ADDED
File without changes
results/g17-d20-mdlm2x-s0.json ADDED
File without changes
results/g17-d20-mdlm2x-s1.json ADDED
File without changes
results/g18-d20-anybudget-s0.json ADDED
File without changes
results/g18-d20-anybudget-s1.json ADDED
File without changes
results/g18-d20-anybudget-s2.json ADDED
File without changes
results/g18-d20-anybudget_nc-s0.json ADDED
File without changes
results/g18-d20-anybudget_nc-s1.json ADDED
File without changes
results/g18-d20-anybudget_nc-s2.json ADDED
File without changes
results/gr1-2regular-anybudget-s0.json ADDED
File without changes
results/gr1-2regular-anybudget-s1.json ADDED
File without changes
results/gr1-2regular-mdlm-s0.json ADDED
File without changes