Results, figures and source (weights kept on scratch)
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- README.md +85 -16
- figures/beat_d12.json +56 -5
- figures/beat_d16.json +54 -3
- figures/fig3_depth_scaling.png +0 -0
- figures/fig4_text8_quality_vs_compute.png +0 -0
- figures/graph_ceiling.json +0 -0
- figures/summary.md +74 -20
- results/evallm.json +0 -0
- results/g14-d12-L6-s0.json +0 -0
- results/g14-d12-L6-s1.json +0 -0
- results/g14-d16-L2-s0.json +0 -0
- results/g14-d16-L2-s1.json +0 -0
- results/g14-d16-L3-s0.json +0 -0
- results/g14-d16-L3-s1.json +0 -0
- results/g14-d8-L12-s0.json +0 -0
- results/g14-d8-L12-s1.json +0 -0
- results/g15-finnish-distill-s0.json +0 -0
- results/g15-finnish-distill-s1.json +0 -0
- results/g15-finnish-matched-s0.json +0 -0
- results/g15-finnish-matched-s1.json +0 -0
- results/g15-georgian-distill-s0.json +0 -0
- results/g15-georgian-distill-s1.json +0 -0
- results/g15-georgian-matched-s0.json +0 -0
- results/g15-georgian-matched-s1.json +0 -0
- results/g15-german-distill-s1.json +0 -0
- results/g15-navajo-distill-s0.json +0 -0
- results/g15-navajo-distill-s1.json +0 -0
- results/g15-navajo-matched-s0.json +0 -0
- results/g15-navajo-matched-s1.json +0 -0
- results/g15-russian-distill-s0.json +0 -0
- results/g15-russian-distill-s1.json +0 -0
- results/g15-russian-matched-s0.json +0 -0
- results/g15-russian-matched-s1.json +0 -0
- results/g15-turkish-distill-s0.json +0 -0
- results/g15-turkish-distill-s1.json +0 -0
- results/g15-turkish-matched-s0.json +0 -0
- results/g15-turkish-matched-s1.json +0 -0
- results/g17-d16-mdlm2x-s0.json +0 -0
- results/g17-d16-mdlm2x-s1.json +0 -0
- results/g17-d20-mdlm2x-s0.json +0 -0
- results/g17-d20-mdlm2x-s1.json +0 -0
- results/g18-d20-anybudget-s0.json +0 -0
- results/g18-d20-anybudget-s1.json +0 -0
- results/g18-d20-anybudget-s2.json +0 -0
- results/g18-d20-anybudget_nc-s0.json +0 -0
- results/g18-d20-anybudget_nc-s1.json +0 -0
- results/g18-d20-anybudget_nc-s2.json +0 -0
- results/gr1-2regular-anybudget-s0.json +0 -0
- results/gr1-2regular-anybudget-s1.json +0 -0
- results/gr1-2regular-mdlm-s0.json +0 -0
README.md
CHANGED
|
@@ -1,15 +1,9 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
-
tags: [masked-diffusion,
|
| 4 |
---
|
| 5 |
|
| 6 |
-
#
|
| 7 |
-
|
| 8 |
-
From-scratch masked diffusion language models (MDLM-style absorbing state) and
|
| 9 |
-
matched autoregressive baselines, trained on arithmetic and algorithmic tasks and
|
| 10 |
-
evaluated far beyond the lengths seen in training.
|
| 11 |
-
|
| 12 |
-
**No pretrained weights or tokenizers are used anywhere.**
|
| 13 |
|
| 14 |
Code: https://github.com/GIND123/MSML612
|
| 15 |
|
|
@@ -40,19 +34,28 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
|
|
| 40 |
| english | 95.3 | 95.2 | -0.1 |
|
| 41 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 42 |
| finnish | 48.6 | 56.6 | +8.0 |
|
|
|
|
|
|
|
| 43 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 44 |
| finnish | 52.9 | 57.9 | +4.9 |
|
|
|
|
|
|
|
| 45 |
| finnish | 48.6 | 56.6 | +8.0 |
|
| 46 |
| finnish | 54.3 | 61.7 | +7.4 |
|
| 47 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 48 |
| georgian | 95.4 | 95.3 | -0.1 |
|
|
|
|
|
|
|
| 49 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 50 |
| georgian | 93.8 | 94.0 | +0.1 |
|
|
|
|
|
|
|
| 51 |
| georgian | 95.4 | 95.3 | -0.1 |
|
| 52 |
| georgian | 93.9 | 93.9 | +0.0 |
|
| 53 |
| german | 78.9 | 79.3 | +0.4 |
|
| 54 |
| german | 75.6 | 79.5 | +3.9 |
|
| 55 |
| german | 78.6 | 78.9 | +0.3 |
|
|
|
|
| 56 |
| german | 78.9 | 79.3 | +0.4 |
|
| 57 |
| german | 77.9 | 78.0 | +0.1 |
|
| 58 |
| german | 79.4 | 81.0 | +1.7 |
|
|
@@ -62,22 +65,34 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
|
|
| 62 |
| german | 77.2 | 78.3 | +1.0 |
|
| 63 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 64 |
| navajo | 31.4 | 40.2 | +8.9 |
|
|
|
|
|
|
|
| 65 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 66 |
| navajo | 24.3 | 29.0 | +4.8 |
|
|
|
|
|
|
|
| 67 |
| navajo | 31.4 | 40.2 | +8.9 |
|
| 68 |
| navajo | 24.6 | 31.7 | +7.1 |
|
| 69 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 70 |
| russian | 69.1 | 71.7 | +2.6 |
|
|
|
|
|
|
|
| 71 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 72 |
| russian | 71.9 | 72.0 | +0.1 |
|
|
|
|
|
|
|
| 73 |
| russian | 69.1 | 71.7 | +2.6 |
|
| 74 |
| russian | 74.1 | 75.9 | +1.8 |
|
| 75 |
| spanish | 84.9 | 84.7 | -0.2 |
|
| 76 |
| spanish | 83.9 | 84.1 | +0.1 |
|
| 77 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 78 |
| turkish | 77.5 | 82.2 | +4.7 |
|
|
|
|
|
|
|
| 79 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 80 |
| turkish | 84.5 | 85.1 | +0.7 |
|
|
|
|
|
|
|
| 81 |
| turkish | 77.5 | 82.2 | +4.7 |
|
| 82 |
| turkish | 79.9 | 82.2 | +2.3 |
|
| 83 |
| turkish | 76.4 | 81.8 | +5.4 |
|
|
@@ -112,19 +127,73 @@ The account in THEORY.md predicts minimum depth growing as log2(n), since the ca
|
|
| 112 |
| digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
|
| 113 |
|---|---|---|---|---|---|---|---|---|
|
| 114 |
| **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
|
| 115 |
-
| **8** | 99 | 100 | 100 | 100 | 100 |
|
| 116 |
-
| **12** | 48 | 100 | 100 |
|
|
|
|
| 117 |
|
| 118 |
## Table 7 — Which component carries the gain (inflection)
|
| 119 |
|
| 120 |
| language | mdlm | matched | distill | full | (gap 1→32) |
|
| 121 |
|---|---|---|---|---|---|
|
| 122 |
-
| finnish | 51.4 |
|
| 123 |
-
| georgian | 94.6 |
|
| 124 |
-
| german | 76.0 | 77.9 | 78.
|
| 125 |
-
| navajo | 28.0 |
|
| 126 |
-
| russian | 71.6 |
|
| 127 |
-
| turkish | 78.7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
## Table 9 — Did the baseline fail to parallelise, or fail to learn?
|
| 130 |
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
tags: [masked-diffusion, parallel-decoding, text8, qm9, from-scratch]
|
| 4 |
---
|
| 5 |
|
| 6 |
+
# Two Failure Modes of Parallel Decoding in Masked Diffusion
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
|
| 8 |
Code: https://github.com/GIND123/MSML612
|
| 9 |
|
|
|
|
| 34 |
| english | 95.3 | 95.2 | -0.1 |
|
| 35 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 36 |
| finnish | 48.6 | 56.6 | +8.0 |
|
| 37 |
+
| finnish | 48.3 | 53.5 | +5.3 |
|
| 38 |
+
| finnish | 50.6 | 55.6 | +4.9 |
|
| 39 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 40 |
| finnish | 52.9 | 57.9 | +4.9 |
|
| 41 |
+
| finnish | 50.3 | 57.0 | +6.7 |
|
| 42 |
+
| finnish | 53.6 | 60.9 | +7.3 |
|
| 43 |
| finnish | 48.6 | 56.6 | +8.0 |
|
| 44 |
| finnish | 54.3 | 61.7 | +7.4 |
|
| 45 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 46 |
| georgian | 95.4 | 95.3 | -0.1 |
|
| 47 |
+
| georgian | 94.8 | 94.5 | -0.3 |
|
| 48 |
+
| georgian | 93.9 | 93.8 | -0.1 |
|
| 49 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 50 |
| georgian | 93.8 | 94.0 | +0.1 |
|
| 51 |
+
| georgian | 94.4 | 94.9 | +0.5 |
|
| 52 |
+
| georgian | 94.1 | 93.8 | -0.3 |
|
| 53 |
| georgian | 95.4 | 95.3 | -0.1 |
|
| 54 |
| georgian | 93.9 | 93.9 | +0.0 |
|
| 55 |
| german | 78.9 | 79.3 | +0.4 |
|
| 56 |
| german | 75.6 | 79.5 | +3.9 |
|
| 57 |
| german | 78.6 | 78.9 | +0.3 |
|
| 58 |
+
| german | 77.9 | 78.5 | +0.7 |
|
| 59 |
| german | 78.9 | 79.3 | +0.4 |
|
| 60 |
| german | 77.9 | 78.0 | +0.1 |
|
| 61 |
| german | 79.4 | 81.0 | +1.7 |
|
|
|
|
| 65 |
| german | 77.2 | 78.3 | +1.0 |
|
| 66 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 67 |
| navajo | 31.4 | 40.2 | +8.9 |
|
| 68 |
+
| navajo | 33.7 | 39.0 | +5.2 |
|
| 69 |
+
| navajo | 24.2 | 30.8 | +6.6 |
|
| 70 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 71 |
| navajo | 24.3 | 29.0 | +4.8 |
|
| 72 |
+
| navajo | 31.1 | 39.3 | +8.2 |
|
| 73 |
+
| navajo | 26.3 | 34.0 | +7.7 |
|
| 74 |
| navajo | 31.4 | 40.2 | +8.9 |
|
| 75 |
| navajo | 24.6 | 31.7 | +7.1 |
|
| 76 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 77 |
| russian | 69.1 | 71.7 | +2.6 |
|
| 78 |
+
| russian | 73.5 | 73.0 | -0.5 |
|
| 79 |
+
| russian | 74.2 | 74.5 | +0.4 |
|
| 80 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 81 |
| russian | 71.9 | 72.0 | +0.1 |
|
| 82 |
+
| russian | 73.1 | 73.3 | +0.3 |
|
| 83 |
+
| russian | 76.3 | 76.2 | -0.1 |
|
| 84 |
| russian | 69.1 | 71.7 | +2.6 |
|
| 85 |
| russian | 74.1 | 75.9 | +1.8 |
|
| 86 |
| spanish | 84.9 | 84.7 | -0.2 |
|
| 87 |
| spanish | 83.9 | 84.1 | +0.1 |
|
| 88 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 89 |
| turkish | 77.5 | 82.2 | +4.7 |
|
| 90 |
+
| turkish | 77.3 | 78.9 | +1.6 |
|
| 91 |
+
| turkish | 83.1 | 83.9 | +0.8 |
|
| 92 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 93 |
| turkish | 84.5 | 85.1 | +0.7 |
|
| 94 |
+
| turkish | 77.0 | 80.9 | +3.9 |
|
| 95 |
+
| turkish | 80.4 | 83.8 | +3.4 |
|
| 96 |
| turkish | 77.5 | 82.2 | +4.7 |
|
| 97 |
| turkish | 79.9 | 82.2 | +2.3 |
|
| 98 |
| turkish | 76.4 | 81.8 | +5.4 |
|
|
|
|
| 127 |
| digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
|
| 128 |
|---|---|---|---|---|---|---|---|---|
|
| 129 |
| **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
|
| 130 |
+
| **8** | 99 | 100 | 100 | 100 | 100 | 100 | 3.0 | 2 |
|
| 131 |
+
| **12** | 48 | 100 | 100 | 100 | – | – | 3.6 | 3 |
|
| 132 |
+
| **16** | 0 | 50 | – | – | – | – | 4.0 | – |
|
| 133 |
|
| 134 |
## Table 7 — Which component carries the gain (inflection)
|
| 135 |
|
| 136 |
| language | mdlm | matched | distill | full | (gap 1→32) |
|
| 137 |
|---|---|---|---|---|---|
|
| 138 |
+
| finnish | 51.4 | 52.0 | 49.4 | 51.9 | +7.7 / +7.0 / +5.1 / +4.9 |
|
| 139 |
+
| georgian | 94.6 | 94.3 | 94.3 | 94.1 | -0.0 / +0.1 / -0.2 / -0.0 |
|
| 140 |
+
| german | 76.0 | 77.9 | 78.2 | 78.4 | +3.2 / +1.4 / +0.5 / +0.2 |
|
| 141 |
+
| navajo | 28.0 | 28.7 | 29.0 | 26.9 | +8.0 / +8.0 / +5.9 / +4.5 |
|
| 142 |
+
| russian | 71.6 | 74.7 | 73.9 | 72.3 | +2.2 / +0.1 / -0.1 / -0.3 |
|
| 143 |
+
| turkish | 78.7 | 78.7 | 80.2 | 82.3 | +3.5 / +3.7 / +1.2 / +0.8 |
|
| 144 |
+
|
| 145 |
+
## Table 8 — text8: quality against compute
|
| 146 |
+
|
| 147 |
+
Generative perplexity under a from-scratch autoregressive character evaluator, against the number of function evaluations actually used. Entropy and distinct-4-gram rate are reported beside it because generative perplexity alone is gamed by degenerate repetition; the real-text reference row is the target, not zero.
|
| 148 |
+
|
| 149 |
+
Real text8 reference — ppl 2.86, entropy 2.855, distinct-4 0.177
|
| 150 |
+
|
| 151 |
+
| training | decoding | NFE | gen ppl | entropy | distinct-4 | valid BPC |
|
| 152 |
+
|---|---|---|---|---|---|---|
|
| 153 |
+
| mdlm | conf_0.5 | 234.5 | 32.99 | 2.871 | 0.360 | 1.7225 |
|
| 154 |
+
| mdlm | conf_0.7 | 248.5 | 16.99 | 2.866 | 0.295 | 1.7225 |
|
| 155 |
+
| mdlm | conf_0.9 | 252.8 | 9.78 | 2.848 | 0.245 | 1.7225 |
|
| 156 |
+
| mdlm | conf_0.95 | 254.0 | 9.08 | 2.856 | 0.238 | 1.7225 |
|
| 157 |
+
| mdlm | conf_0.99 | 255.5 | 8.74 | 2.851 | 0.236 | 1.7225 |
|
| 158 |
+
| mdlm | entbudget_0.05 | 256.0 | 9.47 | 2.854 | 0.241 | 1.7225 |
|
| 159 |
+
| mdlm | entbudget_0.1 | 255.5 | 8.79 | 2.861 | 0.241 | 1.7225 |
|
| 160 |
+
| mdlm | entbudget_0.25 | 255.0 | 8.93 | 2.858 | 0.239 | 1.7225 |
|
| 161 |
+
| mdlm | entbudget_0.5 | 253.2 | 9.39 | 2.858 | 0.243 | 1.7225 |
|
| 162 |
+
| mdlm | entbudget_1.0 | 248.2 | 10.39 | 2.857 | 0.248 | 1.7225 |
|
| 163 |
+
| mdlm | entbudget_2.0 | 245.0 | 15.44 | 2.861 | 0.279 | 1.7225 |
|
| 164 |
+
| mdlm | entbudget_4.0 | 223.2 | 51.18 | 2.882 | 0.401 | 1.7225 |
|
| 165 |
+
| mdlm | entbudget_8.0 | 96.5 | 111.68 | 2.888 | 0.520 | 1.7225 |
|
| 166 |
+
| mdlm | fixedK_1 | 1.0 | 646.96 | 2.860 | 0.630 | 1.7225 |
|
| 167 |
+
| mdlm | fixedK_128 | 128.0 | 44.33 | 2.867 | 0.410 | 1.7225 |
|
| 168 |
+
| mdlm | fixedK_16 | 16.0 | 126.50 | 2.878 | 0.532 | 1.7225 |
|
| 169 |
+
| mdlm | fixedK_2 | 2.0 | 310.58 | 2.890 | 0.625 | 1.7225 |
|
| 170 |
+
| mdlm | fixedK_256 | 256.0 | 8.98 | 2.854 | 0.241 | 1.7225 |
|
| 171 |
+
| mdlm | fixedK_32 | 32.0 | 95.60 | 2.875 | 0.497 | 1.7225 |
|
| 172 |
+
| mdlm | fixedK_4 | 4.0 | 155.18 | 2.881 | 0.550 | 1.7225 |
|
| 173 |
+
| mdlm | fixedK_64 | 64.0 | 86.83 | 2.882 | 0.490 | 1.7225 |
|
| 174 |
+
| mdlm | fixedK_8 | 8.0 | 121.66 | 2.879 | 0.527 | 1.7225 |
|
| 175 |
+
| matched | conf_0.5 | 237.5 | 32.44 | 2.873 | 0.366 | 1.6946 |
|
| 176 |
+
| matched | conf_0.7 | 242.0 | 14.86 | 2.854 | 0.282 | 1.6946 |
|
| 177 |
+
| matched | conf_0.9 | 249.2 | 9.67 | 2.856 | 0.248 | 1.6946 |
|
| 178 |
+
| matched | conf_0.95 | 251.2 | 8.65 | 2.855 | 0.241 | 1.6946 |
|
| 179 |
+
| matched | conf_0.99 | 254.8 | 8.34 | 2.853 | 0.236 | 1.6946 |
|
| 180 |
+
| matched | entbudget_0.05 | 256.0 | 8.39 | 2.857 | 0.241 | 1.6946 |
|
| 181 |
+
| matched | entbudget_0.1 | 256.0 | 8.50 | 2.855 | 0.238 | 1.6946 |
|
| 182 |
+
| matched | entbudget_0.25 | 255.0 | 8.72 | 2.857 | 0.243 | 1.6946 |
|
| 183 |
+
| matched | entbudget_0.5 | 253.2 | 9.00 | 2.863 | 0.247 | 1.6946 |
|
| 184 |
+
| matched | entbudget_1.0 | 251.5 | 10.02 | 2.861 | 0.255 | 1.6946 |
|
| 185 |
+
| matched | entbudget_2.0 | 247.2 | 14.98 | 2.862 | 0.284 | 1.6946 |
|
| 186 |
+
| matched | entbudget_4.0 | 235.2 | 47.92 | 2.885 | 0.403 | 1.6946 |
|
| 187 |
+
| matched | entbudget_8.0 | 98.0 | 104.65 | 2.895 | 0.526 | 1.6946 |
|
| 188 |
+
| matched | fixedK_1 | 1.0 | 651.67 | 2.860 | 0.631 | 1.6946 |
|
| 189 |
+
| matched | fixedK_128 | 128.0 | 42.38 | 2.869 | 0.407 | 1.6946 |
|
| 190 |
+
| matched | fixedK_16 | 16.0 | 94.52 | 2.868 | 0.502 | 1.6946 |
|
| 191 |
+
| matched | fixedK_2 | 2.0 | 295.79 | 2.881 | 0.620 | 1.6946 |
|
| 192 |
+
| matched | fixedK_256 | 256.0 | 8.19 | 2.852 | 0.235 | 1.6946 |
|
| 193 |
+
| matched | fixedK_32 | 32.0 | 93.33 | 2.882 | 0.510 | 1.6946 |
|
| 194 |
+
| matched | fixedK_4 | 4.0 | 146.64 | 2.880 | 0.558 | 1.6946 |
|
| 195 |
+
| matched | fixedK_64 | 64.0 | 76.89 | 2.884 | 0.484 | 1.6946 |
|
| 196 |
+
| matched | fixedK_8 | 8.0 | 123.33 | 2.876 | 0.540 | 1.6946 |
|
| 197 |
|
| 198 |
## Table 9 — Did the baseline fail to parallelise, or fail to learn?
|
| 199 |
|
figures/beat_d12.json
CHANGED
|
@@ -31,10 +31,10 @@
|
|
| 31 |
],
|
| 32 |
"random": [
|
| 33 |
0.766875,
|
| 34 |
-
0.764375,
|
| 35 |
0.768125,
|
| 36 |
-
0.
|
| 37 |
-
0.
|
|
|
|
| 38 |
]
|
| 39 |
},
|
| 40 |
"ours": {
|
|
@@ -46,10 +46,61 @@
|
|
| 46 |
0.999375
|
| 47 |
]
|
| 48 |
},
|
| 49 |
-
"significance": {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
"summary": {
|
| 51 |
"best_inference_1pass": 0.766875,
|
| 52 |
-
"best_inference_any_budget": 0.
|
| 53 |
"ours_1pass": 1.0
|
| 54 |
}
|
| 55 |
}
|
|
|
|
| 31 |
],
|
| 32 |
"random": [
|
| 33 |
0.766875,
|
|
|
|
| 34 |
0.768125,
|
| 35 |
+
0.77125,
|
| 36 |
+
0.764375,
|
| 37 |
+
0.77
|
| 38 |
]
|
| 39 |
},
|
| 40 |
"ours": {
|
|
|
|
| 46 |
0.999375
|
| 47 |
]
|
| 48 |
},
|
| 49 |
+
"significance": {
|
| 50 |
+
"confidence": {
|
| 51 |
+
"diff": 0.233125,
|
| 52 |
+
"ci95": [
|
| 53 |
+
0.226875,
|
| 54 |
+
0.23875
|
| 55 |
+
],
|
| 56 |
+
"p_value": 0.0,
|
| 57 |
+
"n": 400,
|
| 58 |
+
"their_best_budget": 1
|
| 59 |
+
},
|
| 60 |
+
"entropy": {
|
| 61 |
+
"diff": 0.233125,
|
| 62 |
+
"ci95": [
|
| 63 |
+
0.226875,
|
| 64 |
+
0.23875
|
| 65 |
+
],
|
| 66 |
+
"p_value": 0.0,
|
| 67 |
+
"n": 400,
|
| 68 |
+
"their_best_budget": 1
|
| 69 |
+
},
|
| 70 |
+
"entropy-gated": {
|
| 71 |
+
"diff": 0.233125,
|
| 72 |
+
"ci95": [
|
| 73 |
+
0.226875,
|
| 74 |
+
0.23875
|
| 75 |
+
],
|
| 76 |
+
"p_value": 0.0,
|
| 77 |
+
"n": 400,
|
| 78 |
+
"their_best_budget": 1
|
| 79 |
+
},
|
| 80 |
+
"margin": {
|
| 81 |
+
"diff": 0.233125,
|
| 82 |
+
"ci95": [
|
| 83 |
+
0.226875,
|
| 84 |
+
0.23875
|
| 85 |
+
],
|
| 86 |
+
"p_value": 0.0,
|
| 87 |
+
"n": 400,
|
| 88 |
+
"their_best_budget": 1
|
| 89 |
+
},
|
| 90 |
+
"random": {
|
| 91 |
+
"diff": 0.22875,
|
| 92 |
+
"ci95": [
|
| 93 |
+
0.221875,
|
| 94 |
+
0.235625
|
| 95 |
+
],
|
| 96 |
+
"p_value": 0.0,
|
| 97 |
+
"n": 400,
|
| 98 |
+
"their_best_budget": 4
|
| 99 |
+
}
|
| 100 |
+
},
|
| 101 |
"summary": {
|
| 102 |
"best_inference_1pass": 0.766875,
|
| 103 |
+
"best_inference_any_budget": 0.77125,
|
| 104 |
"ours_1pass": 1.0
|
| 105 |
}
|
| 106 |
}
|
figures/beat_d16.json
CHANGED
|
@@ -33,8 +33,8 @@
|
|
| 33 |
0.25,
|
| 34 |
0.25,
|
| 35 |
0.25,
|
| 36 |
-
0.
|
| 37 |
-
0.
|
| 38 |
]
|
| 39 |
},
|
| 40 |
"ours": {
|
|
@@ -46,7 +46,58 @@
|
|
| 46 |
0.749375
|
| 47 |
]
|
| 48 |
},
|
| 49 |
-
"significance": {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
"summary": {
|
| 51 |
"best_inference_1pass": 0.25,
|
| 52 |
"best_inference_any_budget": 0.25,
|
|
|
|
| 33 |
0.25,
|
| 34 |
0.25,
|
| 35 |
0.25,
|
| 36 |
+
0.25,
|
| 37 |
+
0.249375
|
| 38 |
]
|
| 39 |
},
|
| 40 |
"ours": {
|
|
|
|
| 46 |
0.749375
|
| 47 |
]
|
| 48 |
},
|
| 49 |
+
"significance": {
|
| 50 |
+
"confidence": {
|
| 51 |
+
"diff": 0.5,
|
| 52 |
+
"ci95": [
|
| 53 |
+
0.5,
|
| 54 |
+
0.5
|
| 55 |
+
],
|
| 56 |
+
"p_value": 0.0,
|
| 57 |
+
"n": 400,
|
| 58 |
+
"their_best_budget": 1
|
| 59 |
+
},
|
| 60 |
+
"entropy": {
|
| 61 |
+
"diff": 0.5,
|
| 62 |
+
"ci95": [
|
| 63 |
+
0.5,
|
| 64 |
+
0.5
|
| 65 |
+
],
|
| 66 |
+
"p_value": 0.0,
|
| 67 |
+
"n": 400,
|
| 68 |
+
"their_best_budget": 1
|
| 69 |
+
},
|
| 70 |
+
"entropy-gated": {
|
| 71 |
+
"diff": 0.5,
|
| 72 |
+
"ci95": [
|
| 73 |
+
0.5,
|
| 74 |
+
0.5
|
| 75 |
+
],
|
| 76 |
+
"p_value": 0.0,
|
| 77 |
+
"n": 400,
|
| 78 |
+
"their_best_budget": 1
|
| 79 |
+
},
|
| 80 |
+
"margin": {
|
| 81 |
+
"diff": 0.5,
|
| 82 |
+
"ci95": [
|
| 83 |
+
0.5,
|
| 84 |
+
0.5
|
| 85 |
+
],
|
| 86 |
+
"p_value": 0.0,
|
| 87 |
+
"n": 400,
|
| 88 |
+
"their_best_budget": 1
|
| 89 |
+
},
|
| 90 |
+
"random": {
|
| 91 |
+
"diff": 0.5,
|
| 92 |
+
"ci95": [
|
| 93 |
+
0.5,
|
| 94 |
+
0.5
|
| 95 |
+
],
|
| 96 |
+
"p_value": 0.0,
|
| 97 |
+
"n": 400,
|
| 98 |
+
"their_best_budget": 1
|
| 99 |
+
}
|
| 100 |
+
},
|
| 101 |
"summary": {
|
| 102 |
"best_inference_1pass": 0.25,
|
| 103 |
"best_inference_any_budget": 0.25,
|
figures/fig3_depth_scaling.png
CHANGED
|
|
figures/fig4_text8_quality_vs_compute.png
ADDED
|
figures/graph_ceiling.json
ADDED
|
File without changes
|
figures/summary.md
CHANGED
|
@@ -23,19 +23,28 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
|
|
| 23 |
| english | 95.3 | 95.2 | -0.1 |
|
| 24 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 25 |
| finnish | 48.6 | 56.6 | +8.0 |
|
|
|
|
|
|
|
| 26 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 27 |
| finnish | 52.9 | 57.9 | +4.9 |
|
|
|
|
|
|
|
| 28 |
| finnish | 48.6 | 56.6 | +8.0 |
|
| 29 |
| finnish | 54.3 | 61.7 | +7.4 |
|
| 30 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 31 |
| georgian | 95.4 | 95.3 | -0.1 |
|
|
|
|
|
|
|
| 32 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 33 |
| georgian | 93.8 | 94.0 | +0.1 |
|
|
|
|
|
|
|
| 34 |
| georgian | 95.4 | 95.3 | -0.1 |
|
| 35 |
| georgian | 93.9 | 93.9 | +0.0 |
|
| 36 |
| german | 78.9 | 79.3 | +0.4 |
|
| 37 |
| german | 75.6 | 79.5 | +3.9 |
|
| 38 |
| german | 78.6 | 78.9 | +0.3 |
|
|
|
|
| 39 |
| german | 78.9 | 79.3 | +0.4 |
|
| 40 |
| german | 77.9 | 78.0 | +0.1 |
|
| 41 |
| german | 79.4 | 81.0 | +1.7 |
|
|
@@ -45,22 +54,34 @@ Answers are unique but only ~10 characters with shallow, local dependencies. One
|
|
| 45 |
| german | 77.2 | 78.3 | +1.0 |
|
| 46 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 47 |
| navajo | 31.4 | 40.2 | +8.9 |
|
|
|
|
|
|
|
| 48 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 49 |
| navajo | 24.3 | 29.0 | +4.8 |
|
|
|
|
|
|
|
| 50 |
| navajo | 31.4 | 40.2 | +8.9 |
|
| 51 |
| navajo | 24.6 | 31.7 | +7.1 |
|
| 52 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 53 |
| russian | 69.1 | 71.7 | +2.6 |
|
|
|
|
|
|
|
| 54 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 55 |
| russian | 71.9 | 72.0 | +0.1 |
|
|
|
|
|
|
|
| 56 |
| russian | 69.1 | 71.7 | +2.6 |
|
| 57 |
| russian | 74.1 | 75.9 | +1.8 |
|
| 58 |
| spanish | 84.9 | 84.7 | -0.2 |
|
| 59 |
| spanish | 83.9 | 84.1 | +0.1 |
|
| 60 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 61 |
| turkish | 77.5 | 82.2 | +4.7 |
|
|
|
|
|
|
|
| 62 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 63 |
| turkish | 84.5 | 85.1 | +0.7 |
|
|
|
|
|
|
|
| 64 |
| turkish | 77.5 | 82.2 | +4.7 |
|
| 65 |
| turkish | 79.9 | 82.2 | +2.3 |
|
| 66 |
| turkish | 76.4 | 81.8 | +5.4 |
|
|
@@ -95,28 +116,61 @@ The account in THEORY.md predicts minimum depth growing as log2(n), since the ca
|
|
| 95 |
| digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
|
| 96 |
|---|---|---|---|---|---|---|---|---|
|
| 97 |
| **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
|
| 98 |
-
| **8** | 99 | 100 | 100 | 100 | 100 |
|
| 99 |
-
| **12** | 48 | 100 | 100 |
|
|
|
|
| 100 |
|
| 101 |
## Table 7 — Which component carries the gain (inflection)
|
| 102 |
|
| 103 |
| language | mdlm | matched | distill | full | (gap 1→32) |
|
| 104 |
|---|---|---|---|---|---|
|
| 105 |
-
| finnish | 51.4 |
|
| 106 |
-
| georgian | 94.6 |
|
| 107 |
-
| german | 76.0 | 77.9 | 78.
|
| 108 |
-
| navajo | 28.0 |
|
| 109 |
-
| russian | 71.6 |
|
| 110 |
-
| turkish | 78.7 |
|
| 111 |
-
|
| 112 |
-
## Table
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
|
| 119 |
-
|
|
| 120 |
-
|
|
| 121 |
-
|
|
| 122 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
| english | 95.3 | 95.2 | -0.1 |
|
| 24 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 25 |
| finnish | 48.6 | 56.6 | +8.0 |
|
| 26 |
+
| finnish | 48.3 | 53.5 | +5.3 |
|
| 27 |
+
| finnish | 50.6 | 55.6 | +4.9 |
|
| 28 |
| finnish | 50.9 | 55.8 | +4.9 |
|
| 29 |
| finnish | 52.9 | 57.9 | +4.9 |
|
| 30 |
+
| finnish | 50.3 | 57.0 | +6.7 |
|
| 31 |
+
| finnish | 53.6 | 60.9 | +7.3 |
|
| 32 |
| finnish | 48.6 | 56.6 | +8.0 |
|
| 33 |
| finnish | 54.3 | 61.7 | +7.4 |
|
| 34 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 35 |
| georgian | 95.4 | 95.3 | -0.1 |
|
| 36 |
+
| georgian | 94.8 | 94.5 | -0.3 |
|
| 37 |
+
| georgian | 93.9 | 93.8 | -0.1 |
|
| 38 |
| georgian | 94.3 | 94.1 | -0.2 |
|
| 39 |
| georgian | 93.8 | 94.0 | +0.1 |
|
| 40 |
+
| georgian | 94.4 | 94.9 | +0.5 |
|
| 41 |
+
| georgian | 94.1 | 93.8 | -0.3 |
|
| 42 |
| georgian | 95.4 | 95.3 | -0.1 |
|
| 43 |
| georgian | 93.9 | 93.9 | +0.0 |
|
| 44 |
| german | 78.9 | 79.3 | +0.4 |
|
| 45 |
| german | 75.6 | 79.5 | +3.9 |
|
| 46 |
| german | 78.6 | 78.9 | +0.3 |
|
| 47 |
+
| german | 77.9 | 78.5 | +0.7 |
|
| 48 |
| german | 78.9 | 79.3 | +0.4 |
|
| 49 |
| german | 77.9 | 78.0 | +0.1 |
|
| 50 |
| german | 79.4 | 81.0 | +1.7 |
|
|
|
|
| 54 |
| german | 77.2 | 78.3 | +1.0 |
|
| 55 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 56 |
| navajo | 31.4 | 40.2 | +8.9 |
|
| 57 |
+
| navajo | 33.7 | 39.0 | +5.2 |
|
| 58 |
+
| navajo | 24.2 | 30.8 | +6.6 |
|
| 59 |
| navajo | 29.5 | 33.8 | +4.3 |
|
| 60 |
| navajo | 24.3 | 29.0 | +4.8 |
|
| 61 |
+
| navajo | 31.1 | 39.3 | +8.2 |
|
| 62 |
+
| navajo | 26.3 | 34.0 | +7.7 |
|
| 63 |
| navajo | 31.4 | 40.2 | +8.9 |
|
| 64 |
| navajo | 24.6 | 31.7 | +7.1 |
|
| 65 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 66 |
| russian | 69.1 | 71.7 | +2.6 |
|
| 67 |
+
| russian | 73.5 | 73.0 | -0.5 |
|
| 68 |
+
| russian | 74.2 | 74.5 | +0.4 |
|
| 69 |
| russian | 72.7 | 72.0 | -0.7 |
|
| 70 |
| russian | 71.9 | 72.0 | +0.1 |
|
| 71 |
+
| russian | 73.1 | 73.3 | +0.3 |
|
| 72 |
+
| russian | 76.3 | 76.2 | -0.1 |
|
| 73 |
| russian | 69.1 | 71.7 | +2.6 |
|
| 74 |
| russian | 74.1 | 75.9 | +1.8 |
|
| 75 |
| spanish | 84.9 | 84.7 | -0.2 |
|
| 76 |
| spanish | 83.9 | 84.1 | +0.1 |
|
| 77 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 78 |
| turkish | 77.5 | 82.2 | +4.7 |
|
| 79 |
+
| turkish | 77.3 | 78.9 | +1.6 |
|
| 80 |
+
| turkish | 83.1 | 83.9 | +0.8 |
|
| 81 |
| turkish | 80.2 | 81.2 | +1.0 |
|
| 82 |
| turkish | 84.5 | 85.1 | +0.7 |
|
| 83 |
+
| turkish | 77.0 | 80.9 | +3.9 |
|
| 84 |
+
| turkish | 80.4 | 83.8 | +3.4 |
|
| 85 |
| turkish | 77.5 | 82.2 | +4.7 |
|
| 86 |
| turkish | 79.9 | 82.2 | +2.3 |
|
| 87 |
| turkish | 76.4 | 81.8 | +5.4 |
|
|
|
|
| 116 |
| digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
|
| 117 |
|---|---|---|---|---|---|---|---|---|
|
| 118 |
| **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
|
| 119 |
+
| **8** | 99 | 100 | 100 | 100 | 100 | 100 | 3.0 | 2 |
|
| 120 |
+
| **12** | 48 | 100 | 100 | 100 | – | – | 3.6 | 3 |
|
| 121 |
+
| **16** | 0 | 50 | – | – | – | – | 4.0 | – |
|
| 122 |
|
| 123 |
## Table 7 — Which component carries the gain (inflection)
|
| 124 |
|
| 125 |
| language | mdlm | matched | distill | full | (gap 1→32) |
|
| 126 |
|---|---|---|---|---|---|
|
| 127 |
+
| finnish | 51.4 | 52.0 | 49.4 | 51.9 | +7.7 / +7.0 / +5.1 / +4.9 |
|
| 128 |
+
| georgian | 94.6 | 94.3 | 94.3 | 94.1 | -0.0 / +0.1 / -0.2 / -0.0 |
|
| 129 |
+
| german | 76.0 | 77.9 | 78.2 | 78.4 | +3.2 / +1.4 / +0.5 / +0.2 |
|
| 130 |
+
| navajo | 28.0 | 28.7 | 29.0 | 26.9 | +8.0 / +8.0 / +5.9 / +4.5 |
|
| 131 |
+
| russian | 71.6 | 74.7 | 73.9 | 72.3 | +2.2 / +0.1 / -0.1 / -0.3 |
|
| 132 |
+
| turkish | 78.7 | 78.7 | 80.2 | 82.3 | +3.5 / +3.7 / +1.2 / +0.8 |
|
| 133 |
+
|
| 134 |
+
## Table 8 — text8: quality against compute
|
| 135 |
+
|
| 136 |
+
Generative perplexity under a from-scratch autoregressive character evaluator, against the number of function evaluations actually used. Entropy and distinct-4-gram rate are reported beside it because generative perplexity alone is gamed by degenerate repetition; the real-text reference row is the target, not zero.
|
| 137 |
+
|
| 138 |
+
Real text8 reference — ppl 2.86, entropy 2.855, distinct-4 0.177
|
| 139 |
+
|
| 140 |
+
| training | decoding | NFE | gen ppl | entropy | distinct-4 | valid BPC |
|
| 141 |
+
|---|---|---|---|---|---|---|
|
| 142 |
+
| mdlm | conf_0.5 | 234.5 | 32.99 | 2.871 | 0.360 | 1.7225 |
|
| 143 |
+
| mdlm | conf_0.7 | 248.5 | 16.99 | 2.866 | 0.295 | 1.7225 |
|
| 144 |
+
| mdlm | conf_0.9 | 252.8 | 9.78 | 2.848 | 0.245 | 1.7225 |
|
| 145 |
+
| mdlm | conf_0.95 | 254.0 | 9.08 | 2.856 | 0.238 | 1.7225 |
|
| 146 |
+
| mdlm | conf_0.99 | 255.5 | 8.74 | 2.851 | 0.236 | 1.7225 |
|
| 147 |
+
| mdlm | entbudget_0.05 | 256.0 | 9.47 | 2.854 | 0.241 | 1.7225 |
|
| 148 |
+
| mdlm | entbudget_0.1 | 255.5 | 8.79 | 2.861 | 0.241 | 1.7225 |
|
| 149 |
+
| mdlm | entbudget_0.25 | 255.0 | 8.93 | 2.858 | 0.239 | 1.7225 |
|
| 150 |
+
| mdlm | entbudget_0.5 | 253.2 | 9.39 | 2.858 | 0.243 | 1.7225 |
|
| 151 |
+
| mdlm | entbudget_1.0 | 248.2 | 10.39 | 2.857 | 0.248 | 1.7225 |
|
| 152 |
+
| mdlm | entbudget_2.0 | 245.0 | 15.44 | 2.861 | 0.279 | 1.7225 |
|
| 153 |
+
| mdlm | entbudget_4.0 | 223.2 | 51.18 | 2.882 | 0.401 | 1.7225 |
|
| 154 |
+
| mdlm | entbudget_8.0 | 96.5 | 111.68 | 2.888 | 0.520 | 1.7225 |
|
| 155 |
+
| mdlm | fixedK_1 | 1.0 | 646.96 | 2.860 | 0.630 | 1.7225 |
|
| 156 |
+
| mdlm | fixedK_128 | 128.0 | 44.33 | 2.867 | 0.410 | 1.7225 |
|
| 157 |
+
| mdlm | fixedK_16 | 16.0 | 126.50 | 2.878 | 0.532 | 1.7225 |
|
| 158 |
+
| mdlm | fixedK_2 | 2.0 | 310.58 | 2.890 | 0.625 | 1.7225 |
|
| 159 |
+
| mdlm | fixedK_256 | 256.0 | 8.98 | 2.854 | 0.241 | 1.7225 |
|
| 160 |
+
| mdlm | fixedK_32 | 32.0 | 95.60 | 2.875 | 0.497 | 1.7225 |
|
| 161 |
+
| mdlm | fixedK_4 | 4.0 | 155.18 | 2.881 | 0.550 | 1.7225 |
|
| 162 |
+
| mdlm | fixedK_64 | 64.0 | 86.83 | 2.882 | 0.490 | 1.7225 |
|
| 163 |
+
| mdlm | fixedK_8 | 8.0 | 121.66 | 2.879 | 0.527 | 1.7225 |
|
| 164 |
+
| matched | conf_0.5 | 237.5 | 32.44 | 2.873 | 0.366 | 1.6946 |
|
| 165 |
+
| matched | conf_0.7 | 242.0 | 14.86 | 2.854 | 0.282 | 1.6946 |
|
| 166 |
+
| matched | conf_0.9 | 249.2 | 9.67 | 2.856 | 0.248 | 1.6946 |
|
| 167 |
+
| matched | conf_0.95 | 251.2 | 8.65 | 2.855 | 0.241 | 1.6946 |
|
| 168 |
+
| matched | conf_0.99 | 254.8 | 8.34 | 2.853 | 0.236 | 1.6946 |
|
| 169 |
+
| matched | entbudget_0.05 | 256.0 | 8.39 | 2.857 | 0.241 | 1.6946 |
|
| 170 |
+
| matched | entbudget_0.1 | 256.0 | 8.50 | 2.855 | 0.238 | 1.6946 |
|
| 171 |
+
| matched | entbudget_0.25 | 255.0 | 8.72 | 2.857 | 0.243 | 1.6946 |
|
| 172 |
+
| matched | entbudget_0.5 | 253.2 | 9.00 | 2.863 | 0.247 | 1.6946 |
|
| 173 |
+
| matched | entbudget_1.0 | 251.5 | 10.02 | 2.861 | 0.255 | 1.6946 |
|
| 174 |
+
| matched | entbudget_2.0 | 247.2 | 14.98 | 2.862 | 0.284 | 1.6946 |
|
| 175 |
+
| matched | entbudget_4.0 | 235.2 | 47.92 | 2.885 | 0.403 | 1.6946 |
|
| 176 |
+
| matched
|
results/evallm.json
ADDED
|
File without changes
|
results/g14-d12-L6-s0.json
ADDED
|
File without changes
|
results/g14-d12-L6-s1.json
ADDED
|
File without changes
|
results/g14-d16-L2-s0.json
ADDED
|
File without changes
|
results/g14-d16-L2-s1.json
ADDED
|
File without changes
|
results/g14-d16-L3-s0.json
ADDED
|
File without changes
|
results/g14-d16-L3-s1.json
ADDED
|
File without changes
|
results/g14-d8-L12-s0.json
ADDED
|
File without changes
|
results/g14-d8-L12-s1.json
ADDED
|
File without changes
|
results/g15-finnish-distill-s0.json
ADDED
|
File without changes
|
results/g15-finnish-distill-s1.json
ADDED
|
File without changes
|
results/g15-finnish-matched-s0.json
ADDED
|
File without changes
|
results/g15-finnish-matched-s1.json
ADDED
|
File without changes
|
results/g15-georgian-distill-s0.json
ADDED
|
File without changes
|
results/g15-georgian-distill-s1.json
ADDED
|
File without changes
|
results/g15-georgian-matched-s0.json
ADDED
|
File without changes
|
results/g15-georgian-matched-s1.json
ADDED
|
File without changes
|
results/g15-german-distill-s1.json
ADDED
|
File without changes
|
results/g15-navajo-distill-s0.json
ADDED
|
File without changes
|
results/g15-navajo-distill-s1.json
ADDED
|
File without changes
|
results/g15-navajo-matched-s0.json
ADDED
|
File without changes
|
results/g15-navajo-matched-s1.json
ADDED
|
File without changes
|
results/g15-russian-distill-s0.json
ADDED
|
File without changes
|
results/g15-russian-distill-s1.json
ADDED
|
File without changes
|
results/g15-russian-matched-s0.json
ADDED
|
File without changes
|
results/g15-russian-matched-s1.json
ADDED
|
File without changes
|
results/g15-turkish-distill-s0.json
ADDED
|
File without changes
|
results/g15-turkish-distill-s1.json
ADDED
|
File without changes
|
results/g15-turkish-matched-s0.json
ADDED
|
File without changes
|
results/g15-turkish-matched-s1.json
ADDED
|
File without changes
|
results/g17-d16-mdlm2x-s0.json
ADDED
|
File without changes
|
results/g17-d16-mdlm2x-s1.json
ADDED
|
File without changes
|
results/g17-d20-mdlm2x-s0.json
ADDED
|
File without changes
|
results/g17-d20-mdlm2x-s1.json
ADDED
|
File without changes
|
results/g18-d20-anybudget-s0.json
ADDED
|
File without changes
|
results/g18-d20-anybudget-s1.json
ADDED
|
File without changes
|
results/g18-d20-anybudget-s2.json
ADDED
|
File without changes
|
results/g18-d20-anybudget_nc-s0.json
ADDED
|
File without changes
|
results/g18-d20-anybudget_nc-s1.json
ADDED
|
File without changes
|
results/g18-d20-anybudget_nc-s2.json
ADDED
|
File without changes
|
results/gr1-2regular-anybudget-s0.json
ADDED
|
File without changes
|
results/gr1-2regular-anybudget-s1.json
ADDED
|
File without changes
|
results/gr1-2regular-mdlm-s0.json
ADDED
|
File without changes
|