Full results: addition, text8, graphs, QM9, SATNet Sudoku

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 figures/fig2_frontier.png filter=lfs diff=lfs merge=lfs -text
+figures/fig4_text8_quality_vs_compute.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,209 +1,145 @@
 ---
 license: apache-2.0
-tags: [masked-diffusion, parallel-decoding, text8, qm9, from-scratch]
+tags:
+  - masked-diffusion
+  - discrete-diffusion
+  - parallel-decoding
+  - constraint-satisfaction
+  - text8
+  - qm9
+  - sudoku
+  - from-scratch
 ---
 
 # Two Failure Modes of Parallel Decoding in Masked Diffusion
 
+Everything here is trained from scratch: no pretrained weights, no pretrained
+tokenizer, and the autoregressive model used to score generated text is itself
+trained from scratch on the same corpus.
+
 Code: https://github.com/GIND123/MSML612
 
-## Results
-
-# Results
-
-## Table 1 — Addition: exact match at ONE forward pass
-
-The whole answer committed in a single pass. Depth scaled as log2(digits); learning rate 1e-4 with 4000 warmup steps, which matters - see Table 4.
-
-| digits | MDLM baseline | full method | seeds >90% (base → ours) |
-|---|---|---|---|
-| **8** | 100.0 [100,100] | 100.0 [100,100] | 4/4 → 4/4 |
-| **12** | 76.7 [30,100] | 100.0 [100,100] | 3/4 → 4/4 |
-| **16** | 25.0 [0,75] | 75.0 [25,100] | 1/4 → 3/4 |
-| **24** | 0.0 [0,0] | 0.0 [0,0] | 0/4 → 0/4 |
-| **32** | 0.0 [0,0] | 0.0 [0,0] | 0/4 → 0/4 |
-
-## Table 2 — Morphological inflection: where the problem does NOT arise
-
-Answers are unique but only ~10 characters with shallow, local dependencies. One pass already suffices for the plain baseline, so there is no gap for any method to close. This is the boundary the account predicts, and it is reported as such.
-
-| language | 1 pass | 32 passes | gap |
-|---|---|---|---|
-| english | 95.5 | 95.1 | -0.4 |
-| english | 95.0 | 95.2 | +0.2 |
-| english | 95.3 | 95.2 | -0.1 |
-| finnish | 50.9 | 55.8 | +4.9 |
-| finnish | 48.6 | 56.6 | +8.0 |
-| finnish | 48.3 | 53.5 | +5.3 |
-| finnish | 50.6 | 55.6 | +4.9 |
-| finnish | 50.9 | 55.8 | +4.9 |
-| finnish | 52.9 | 57.9 | +4.9 |
-| finnish | 50.3 | 57.0 | +6.7 |
-| finnish | 53.6 | 60.9 | +7.3 |
-| finnish | 48.6 | 56.6 | +8.0 |
-| finnish | 54.3 | 61.7 | +7.4 |
-| georgian | 94.3 | 94.1 | -0.2 |
-| georgian | 95.4 | 95.3 | -0.1 |
-| georgian | 94.8 | 94.5 | -0.3 |
-| georgian | 93.9 | 93.8 | -0.1 |
-| georgian | 94.3 | 94.1 | -0.2 |
-| georgian | 93.8 | 94.0 | +0.1 |
-| georgian | 94.4 | 94.9 | +0.5 |
-| georgian | 94.1 | 93.8 | -0.3 |
-| georgian | 95.4 | 95.3 | -0.1 |
-| georgian | 93.9 | 93.9 | +0.0 |
-| german | 78.9 | 79.3 | +0.4 |
-| german | 75.6 | 79.5 | +3.9 |
-| german | 78.6 | 78.9 | +0.3 |
-| german | 77.9 | 78.5 | +0.7 |
-| german | 78.9 | 79.3 | +0.4 |
-| german | 77.9 | 78.0 | +0.1 |
-| german | 79.4 | 81.0 | +1.7 |
-| german | 76.5 | 77.6 | +1.0 |
-| german | 75.6 | 79.5 | +3.9 |
-| german | 76.4 | 78.9 | +2.5 |
-| german | 77.2 | 78.3 | +1.0 |
-| navajo | 29.5 | 33.8 | +4.3 |
-| navajo | 31.4 | 40.2 | +8.9 |
-| navajo | 33.7 | 39.0 | +5.2 |
-| navajo | 24.2 | 30.8 | +6.6 |
-| navajo | 29.5 | 33.8 | +4.3 |
-| navajo | 24.3 | 29.0 | +4.8 |
-| navajo | 31.1 | 39.3 | +8.2 |
-| navajo | 26.3 | 34.0 | +7.7 |
-| navajo | 31.4 | 40.2 | +8.9 |
-| navajo | 24.6 | 31.7 | +7.1 |
-| russian | 72.7 | 72.0 | -0.7 |
-| russian | 69.1 | 71.7 | +2.6 |
-| russian | 73.5 | 73.0 | -0.5 |
-| russian | 74.2 | 74.5 | +0.4 |
-| russian | 72.7 | 72.0 | -0.7 |
-| russian | 71.9 | 72.0 | +0.1 |
-| russian | 73.1 | 73.3 | +0.3 |
-| russian | 76.3 | 76.2 | -0.1 |
-| russian | 69.1 | 71.7 | +2.6 |
-| russian | 74.1 | 75.9 | +1.8 |
-| spanish | 84.9 | 84.7 | -0.2 |
-| spanish | 83.9 | 84.1 | +0.1 |
-| turkish | 80.2 | 81.2 | +1.0 |
-| turkish | 77.5 | 82.2 | +4.7 |
-| turkish | 77.3 | 78.9 | +1.6 |
-| turkish | 83.1 | 83.9 | +0.8 |
-| turkish | 80.2 | 81.2 | +1.0 |
-| turkish | 84.5 | 85.1 | +0.7 |
-| turkish | 77.0 | 80.9 | +3.9 |
-| turkish | 80.4 | 83.8 | +3.4 |
-| turkish | 77.5 | 82.2 | +4.7 |
-| turkish | 79.9 | 82.2 | +2.3 |
-| turkish | 76.4 | 81.8 | +5.4 |
-
-## Table 3 — Non-unique answers: the limit no training removes
-
-On probes where a group must agree but the value is free, the likelihood-optimal marginal is uniform over the valid values, so two independent draws agree with probability 1/V. Sweeping an entropy penalty designed to break that symmetry over 66 runs: weak settings changed nothing, strong settings destroyed the model, failing even at fully sequential decoding. Succeeding would require abandoning the data distribution.
-
-## Table 4 — Why earlier runs appeared to fail (10-digit addition)
-
-Every apparent failure at 10+ digits was a learning-rate artifact, not a limit of the method or of depth.
-
-| layers | lr | warmup | mean @1 pass | seeds >90% |
-|---|---|---|---|---|
-| 6 | 5e-05 | 500 | 100.0 | 3/3 |
-| 6 | 5e-05 | 4000 | 100.0 | 3/3 |
-| 6 | 0.0001 | 500 | 100.0 | 3/3 |
-| 6 | 0.0001 | 4000 | 100.0 | 3/3 |
-| 6 | 0.0003 | 500 | 70.6 | 2/3 |
-| 6 | 0.0003 | 4000 | 100.0 | 3/3 |
-| 10 | 5e-05 | 500 | 100.0 | 3/3 |
-| 10 | 5e-05 | 4000 | 100.0 | 3/3 |
-| 10 | 0.0001 | 500 | 100.0 | 3/3 |
-| 10 | 0.0001 | 4000 | 100.0 | 3/3 |
-| 10 | 0.0003 | 500 | 47.3 | 1/3 |
-| 10 | 0.0003 | 4000 | 100.0 | 3/3 |
-
-## Table 6 — Minimum depth for one-pass addition
-
-The account in THEORY.md predicts minimum depth growing as log2(n), since the carry is a prefix scan. A ripple-carry account predicts linear growth; an impossibility account predicts no depth suffices.
-
-| digits | 2L | 3L | 4L | 6L | 8L | 12L | log2(n) | min depth >90% |
-|---|---|---|---|---|---|---|---|---|
-| **4** | 100 | 100 | 100 | 100 | 100 | 100 | 2.0 | 2 |
-| **8** | 99 | 100 | 100 | 100 | 100 | 100 | 3.0 | 2 |
-| **12** | 48 | 100 | 100 | 100 | – | – | 3.6 | 3 |
-| **16** | 0 | 50 | – | – | – | – | 4.0 | – |
-
-## Table 7 — Which component carries the gain (inflection)
-
-| language | mdlm | matched | distill | full | (gap 1→32) |
-|---|---|---|---|---|---|
-| finnish | 51.4 | 52.0 | 49.4 | 51.9 | +7.7 / +7.0 / +5.1 / +4.9 |
-| georgian | 94.6 | 94.3 | 94.3 | 94.1 | -0.0 / +0.1 / -0.2 / -0.0 |
-| german | 76.0 | 77.9 | 78.2 | 78.4 | +3.2 / +1.4 / +0.5 / +0.2 |
-| navajo | 28.0 | 28.7 | 29.0 | 26.9 | +8.0 / +8.0 / +5.9 / +4.5 |
-| russian | 71.6 | 74.7 | 73.9 | 72.3 | +2.2 / +0.1 / -0.1 / -0.3 |
-| turkish | 78.7 | 78.7 | 80.2 | 82.3 | +3.5 / +3.7 / +1.2 / +0.8 |
-
-## Table 8 — text8: quality against compute
-
-Generative perplexity under a from-scratch autoregressive character evaluator, against the number of function evaluations actually used. Entropy and distinct-4-gram rate are reported beside it because generative perplexity alone is gamed by degenerate repetition; the real-text reference row is the target, not zero.
-
-Real text8 reference — ppl 2.86, entropy 2.855, distinct-4 0.177
-
-| training | decoding | NFE | gen ppl | entropy | distinct-4 | valid BPC |
-|---|---|---|---|---|---|---|
-| mdlm | conf_0.5 | 234.5 | 32.99 | 2.871 | 0.360 | 1.7225 |
-| mdlm | conf_0.7 | 248.5 | 16.99 | 2.866 | 0.295 | 1.7225 |
-| mdlm | conf_0.9 | 252.8 | 9.78 | 2.848 | 0.245 | 1.7225 |
-| mdlm | conf_0.95 | 254.0 | 9.08 | 2.856 | 0.238 | 1.7225 |
-| mdlm | conf_0.99 | 255.5 | 8.74 | 2.851 | 0.236 | 1.7225 |
-| mdlm | entbudget_0.05 | 256.0 | 9.47 | 2.854 | 0.241 | 1.7225 |
-| mdlm | entbudget_0.1 | 255.5 | 8.79 | 2.861 | 0.241 | 1.7225 |
-| mdlm | entbudget_0.25 | 255.0 | 8.93 | 2.858 | 0.239 | 1.7225 |
-| mdlm | entbudget_0.5 | 253.2 | 9.39 | 2.858 | 0.243 | 1.7225 |
-| mdlm | entbudget_1.0 | 248.2 | 10.39 | 2.857 | 0.248 | 1.7225 |
-| mdlm | entbudget_2.0 | 245.0 | 15.44 | 2.861 | 0.279 | 1.7225 |
-| mdlm | entbudget_4.0 | 223.2 | 51.18 | 2.882 | 0.401 | 1.7225 |
-| mdlm | entbudget_8.0 | 96.5 | 111.68 | 2.888 | 0.520 | 1.7225 |
-| mdlm | fixedK_1 | 1.0 | 646.96 | 2.860 | 0.630 | 1.7225 |
-| mdlm | fixedK_128 | 128.0 | 44.33 | 2.867 | 0.410 | 1.7225 |
-| mdlm | fixedK_16 | 16.0 | 126.50 | 2.878 | 0.532 | 1.7225 |
-| mdlm | fixedK_2 | 2.0 | 310.58 | 2.890 | 0.625 | 1.7225 |
-| mdlm | fixedK_256 | 256.0 | 8.98 | 2.854 | 0.241 | 1.7225 |
-| mdlm | fixedK_32 | 32.0 | 95.60 | 2.875 | 0.497 | 1.7225 |
-| mdlm | fixedK_4 | 4.0 | 155.18 | 2.881 | 0.550 | 1.7225 |
-| mdlm | fixedK_64 | 64.0 | 86.83 | 2.882 | 0.490 | 1.7225 |
-| mdlm | fixedK_8 | 8.0 | 121.66 | 2.879 | 0.527 | 1.7225 |
-| matched | conf_0.5 | 237.5 | 32.44 | 2.873 | 0.366 | 1.6946 |
-| matched | conf_0.7 | 242.0 | 14.86 | 2.854 | 0.282 | 1.6946 |
-| matched | conf_0.9 | 249.2 | 9.67 | 2.856 | 0.248 | 1.6946 |
-| matched | conf_0.95 | 251.2 | 8.65 | 2.855 | 0.241 | 1.6946 |
-| matched | conf_0.99 | 254.8 | 8.34 | 2.853 | 0.236 | 1.6946 |
-| matched | entbudget_0.05 | 256.0 | 8.39 | 2.857 | 0.241 | 1.6946 |
-| matched | entbudget_0.1 | 256.0 | 8.50 | 2.855 | 0.238 | 1.6946 |
-| matched | entbudget_0.25 | 255.0 | 8.72 | 2.857 | 0.243 | 1.6946 |
-| matched | entbudget_0.5 | 253.2 | 9.00 | 2.863 | 0.247 | 1.6946 |
-| matched | entbudget_1.0 | 251.5 | 10.02 | 2.861 | 0.255 | 1.6946 |
-| matched | entbudget_2.0 | 247.2 | 14.98 | 2.862 | 0.284 | 1.6946 |
-| matched | entbudget_4.0 | 235.2 | 47.92 | 2.885 | 0.403 | 1.6946 |
-| matched | entbudget_8.0 | 98.0 | 104.65 | 2.895 | 0.526 | 1.6946 |
-| matched | fixedK_1 | 1.0 | 651.67 | 2.860 | 0.631 | 1.6946 |
-| matched | fixedK_128 | 128.0 | 42.38 | 2.869 | 0.407 | 1.6946 |
-| matched | fixedK_16 | 16.0 | 94.52 | 2.868 | 0.502 | 1.6946 |
-| matched | fixedK_2 | 2.0 | 295.79 | 2.881 | 0.620 | 1.6946 |
-| matched | fixedK_256 | 256.0 | 8.19 | 2.852 | 0.235 | 1.6946 |
-| matched | fixedK_32 | 32.0 | 93.33 | 2.882 | 0.510 | 1.6946 |
-| matched | fixedK_4 | 4.0 | 146.64 | 2.880 | 0.558 | 1.6946 |
-| matched | fixedK_64 | 64.0 | 76.89 | 2.884 | 0.484 | 1.6946 |
-| matched | fixedK_8 | 8.0 | 123.33 | 2.876 | 0.540 | 1.6946 |
-
-## Table 9 — Did the baseline fail to parallelise, or fail to learn?
-
-A 20-digit answer occupies 21 slots, so the standard 16-pass grid is not a sequential decode. Re-decoded at one token per pass, the two explanations separate.
-
-| digits | mode | @1 pass | fully sequential | reading |
-|---|---|---|---|---|
-| 20 | full | 99.8 | 85.7 | learned it and parallelises |
-| 20 | mdlm | 0.0 | 0.0 | never learned the task |
-| 24 | full | 33.7 | 33.3 | learned it and parallelises |
-| 24 | mdlm | 0.0 | 0.0 | never learned the task |
+---
+
+## The claim
+
+Parallel decoding in masked diffusion loses accuracy for two reasons that need
+**opposite** fixes, and the literature treats them as one.
+
+| | failure mode | fixable by |
+|---|---|---|
+| **(i)** | the fully-masked state is never trained — a *K*-pass decode starts at `t = 1`, and continuous `t ~ U(0,1)` assigns that state probability **zero** | **training only** |
+| **(ii)** | committed positions carry real mutual information — parallel decoding samples the product of marginals when the truth is the joint | **inference only** |
 
+Prior work applies inference-time fixes to both, which is why those methods
+flatline on arithmetic, where (ii) is identically zero.
 
+## The bound
+
+For a data distribution `P` uniform on a family `F`, let `V*` be the one-pass
+validity achievable from the **true** marginals. Then
+
+```
+V*  ≥  2^(−TC),     TC = D_KL( P ‖ ∏ᵢ pᵢ ),   equality iff ∏ᵢ pᵢ is constant on F
+```
+
+**Every bit of total correlation among jointly committed variables at most halves
+the one-pass success probability.**
+
+Verified by exhaustive enumeration over graph families, with the enumerator itself
+checked against Cayley's formula (spanning trees at n = 5, 6, 7 → 125, 1296,
+16807):
+
+| family | TC (bits) | V* | 2^(−TC) | ratio |
+|---|---|---|---|---|
+| bipartite (n=7) | 0.026 | 98.2816% | 98.1817% | 1.001 |
+| tree (n=6) | 3.435 | **9.2488%** | **9.2488%** | **1.00000** |
+| matching (n=6) | 6.922 | **0.8246%** | **0.8246%** | **1.00000** |
+| 2-regular (n=7) | 10.423 | **0.0728%** | **0.0728%** | **1.00000** |
+
+The inequality holds in every case; equality holds to five decimals on exactly
+the fixed-edge-count families, and fails precisely where edge counts vary — which
+is what the Jensen step requires.
+
+## Finding 1 — train the schedule you intend to decode at
+
+| task | baseline | ours |
+|---|---|---|
+| 20-digit addition, 1 pass | **0.00%** (also 0.00% at **2× compute**, both seeds) | **99.9%** (3/3 seeds) |
+| text8, NFE 16 | 116.88 gen-ppl | **92.74** (−20.7%) |
+| text8, validation BPC | 1.7292 | **1.7010** |
+
+The text8 gain appears in generative quality **and** likelihood simultaneously,
+so it is not bought by trading one for the other.
+
+A 48-run ablation isolates the mechanism: the `t = 1` term alone removes **49%**
+of the parallelism penalty, against 17% for reshaping the rest of the schedule.
+
+## Finding 2 — one-pass decoding needs absolute position information
+
+At `t = 1` every input token is `[MASK]`, so the sequence is constant. With a
+relative encoding the attention score depends only on `(i − j)`, and identical
+value vectors give identical outputs at every position. **Position-dependent
+marginals are inexpressible at any training budget.**
+
+| condition | learned (APE) | parameter-free (sinusoidal) |
+|---|---|---|
+| marginals **vary** | **+88.037 pts** | **+88.074 pts** |
+| marginals **constant** (control) | **−0.024 pts** | **+0.000 pts** |
+
+A ~3,700× dissociation. The parameter-free variant matching the learned one rules
+out capacity; the control rules out "absolute is simply better". Predicted 5/5
+before measurement, and the treated arm lands at **100.3% of `V*`** — at the
+bound, not merely higher.
+
+## Benchmark: SATNet Sudoku
+
+9x9 Sudoku, SATNet's conventional 9,000/1,000 split, **board accuracy** = all 81
+cells correct. Published number: **98.3%** (Wang et al., ICML 2019).
+
+Un-augmented arm, which is the only like-for-like comparison:
+
+| denoising passes | board accuracy |
+|---|---|
+| **1** | **87.85%** |
+| **2** | **98.65%** |
+| 8 | 99.70% |
+| **32** | **100.00%** |
+
+Two passes exceed the published number; 32 passes solve **all 1,000** test
+puzzles exactly. One pass — the entire grid committed at once — solves 87.85%.
+
+Leakage audited first: 0/1000 test puzzles in training, 0/1000 test solutions in
+training, and 0/1000 test solutions matching a training solution even up to digit
+relabelling. All 10,000 solutions are distinct.
+
+**Scoped honestly:** this comes from the *plain MDLM baseline* with fixed-`K`
+decoding. Neither finding above contributes — the standard formulation already
+saturates this benchmark. It is a result for masked diffusion on SATNet Sudoku,
+not evidence for our method.
+
+## Negative results, reported
+
+- **Any-budget conditioning fails.** Sampling the decoding budget during training
+  is worse than the baseline at every budget on graphs (−4.2 at K=8, −31.7 at
+  K=15) and destroys 20-digit addition outright (0.0% vs 99.9%), because it
+  dilutes `t = 1` training 4.5×.
+- **The entropy budget is mixed** — competitive on graphs, loses to fixed-`K` on
+  text8.
+- **Half the QM9 gap was our own representation.** Giving the baseline the atom
+  count that DiGress samples first lifts it from 77.5% to 89.1%; the gap narrows
+  from 22.2 to 10.7 points.
+- **16-digit addition is unsettled** — a compute-matched baseline reaches 100% on
+  one seed of two.
+- **Our Sudoku prediction was refuted by our own framework.** We registered that
+  entropy-budgeted decoding would beat fixed-`K` on Sudoku by the largest margin
+  in the project. It loses on every arm by 0.35–1.40 points. The prediction
+  contradicted the bound: Sudoku has a unique answer, so `TC = 0` and there is
+  nothing for a mode-(ii) fix to repair.
+
+## Audit
+
+`src/audit.py`: 46 assertions across 11 sections, 0 failures. It caught two live
+bugs that would each have produced confidently wrong numbers — a sampler emitting
+the absorbing state so decoding silently stalled, and an aromatic-bond encoding
+that made round-trip validity on *real* molecules 92.65% instead of 100%.

figures/graph_ceiling.json

@@ -0,0 +1,23 @@
+{
+  "2regular/mdlm": {
+    "V": 0.002889476997120002,
+    "p": 0.003173828125,
+    "lo": 0.0021668792659426802,
+    "hi": 0.004646527181541615,
+    "verdict": "at the limit"
+  },
+  "matching/anybudget": {
+    "V": 0.008246337208320003,
+    "p": 0.008544921875,
+    "lo": 0.006769272649384511,
+    "hi": 0.010781286160829684,
+    "verdict": "at the limit"
+  },
+  "matching/mdlm": {
+    "V": 0.008246337208320003,
+    "p": 0.0076904296875,
+    "lo": 0.006015871984354349,
+    "hi": 0.009826503495425543,
+    "verdict": "at the limit"
+  }
+}

figures/summary.md

@@ -173,4 +173,62 @@ Real text8 reference — ppl 2.86, entropy 2.855, distinct-4 0.177
 | matched | entbudget_1.0 | 251.5 | 10.02 | 2.861 | 0.255 | 1.6946 |
 | matched | entbudget_2.0 | 247.2 | 14.98 | 2.862 | 0.284 | 1.6946 |
 | matched | entbudget_4.0 | 235.2 | 47.92 | 2.885 | 0.403 | 1.6946 |
-| matched
+| matched | entbudget_8.0 | 98.0 | 104.65 | 2.895 | 0.526 | 1.6946 |
+| matched | fixedK_1 | 1.0 | 651.67 | 2.860 | 0.631 | 1.6946 |
+| matched | fixedK_128 | 128.0 | 42.38 | 2.869 | 0.407 | 1.6946 |
+| matched | fixedK_16 | 16.0 | 94.52 | 2.868 | 0.502 | 1.6946 |
+| matched | fixedK_2 | 2.0 | 295.79 | 2.881 | 0.620 | 1.6946 |
+| matched | fixedK_256 | 256.0 | 8.19 | 2.852 | 0.235 | 1.6946 |
+| matched | fixedK_32 | 32.0 | 93.33 | 2.882 | 0.510 | 1.6946 |
+| matched | fixedK_4 | 4.0 | 146.64 | 2.880 | 0.558 | 1.6946 |
+| matched | fixedK_64 | 64.0 | 76.89 | 2.884 | 0.484 | 1.6946 |
+| matched | fixedK_8 | 8.0 | 123.33 | 2.876 | 0.540 | 1.6946 |
+
+## Table 9b — Any-budget conditioning vs a fixed budget
+
+20-digit addition, one configuration. A fixed-K method wins at the budget it was trained for and loses away from it; the question is whether conditioning on the budget removes that trade.
+
+| training | @1 pass | @sequential | spread |
+|---|---|---|---|
+| MDLM baseline | 0.0 | 0.0 | 0.0 |
+| fixed-K (K=1) | 99.9 | 85.8 | 14.2 |
+| any-budget (no conditioning) | 0.2 | 0.3 | 0.1 |
+| any-budget (conditioned) | 0.0 | 0.0 | 0.0 |
+
+## Table 9 — Did the baseline fail to parallelise, or fail to learn?
+
+A 20-digit answer occupies 21 slots, so the standard 16-pass grid is not a sequential decode. Re-decoded at one token per pass, the two explanations separate.
+
+| digits | mode | @1 pass | fully sequential | reading |
+|---|---|---|---|---|
+| 20 | full | 99.8 | 85.7 | learned it and parallelises |
+| 20 | mdlm | 0.0 | 0.0 | never learned the task |
+| 24 | full | 33.7 | 33.3 | learned it and parallelises |
+| 24 | mdlm | 0.0 | 0.0 | never learned the task |
+
+## Table 10 — Structured graphs: measured against an exact ceiling
+
+V* is the one-pass validity achievable from the TRUE marginals, so it bounds any model that reproduces the data distribution. Where the interval contains V* the residual failure is information-theoretic and untrainable; where it falls short, the shortfall is trainable.
+
+| family | TC (bits) | V* | achieved @1 pass | reading |
+|---|---|---|---|---|
+| 2regular | 8.43 | 0.29% | 0.32% | at the ceiling — untrainable |
+| matching | 6.92 | 0.82% | 0.77% | at the ceiling — untrainable |
+| tree | 3.43 | 9.25% | 9.20% | at the ceiling — untrainable |
+| bipartite | 0.11 | 93.63% | 5.88% | below ceiling — see Table 11 |
+
+## Table 11 — One-pass decoding needs ABSOLUTE position information
+
+At t = 1 every input token is [MASK], so the sequence is constant and a relative encoding gives every position the same output. Position-dependent marginals are then inexpressible at any training budget. Matching is the control: its marginals are identical at every slot, so absolute information should buy nothing there.
+
+| family | marginals | encoding | @1 pass | % of V* |
+|---|---|---|---|---|
+| bipartite | **vary** | rope | 5.87% | 6.3% |
+| bipartite | **vary** | ape | 93.91% | 100.3% |
+| bipartite | **vary** | sin | 93.95% | 100.3% |
+| matching | constant | rope | 0.78% | 94.7% |
+| matching | constant | ape | 0.76% | 91.8% |
+
+- absolute − relative on **bipartite**: **+88.037** points
+- absolute − relative on **matching**: **-0.024** points
+

results/evallm.json

@@ -0,0 +1,17 @@
+{
+  "args": {
+    "root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/text8",
+    "seq_len": 256,
+    "d": 512,
+    "layers": 8,
+    "heads": 8,
+    "bs": 128,
+    "steps": 50000,
+    "lr": 0.0003,
+    "warmup": 2000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/evallm"
+  },
+  "test_bpc": 1.5393671429136824,
+  "vocab": 28
+}

results/g14-d12-L6-s0.json

@@ -0,0 +1,39 @@
+{
+  "args": {
+    "mode": "full",
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 12,
+    "seq_len": 40,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 6,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d12-L6-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 1.0,
+    "2": 1.0,
+    "3": 1.0,
+    "4": 1.0,
+    "6": 1.0,
+    "8": 1.0,
+    "12": 1.0,
+    "16": 1.0
+  },
+  "minutes": 17.56450060606003,
+  "compute": {
+    "params": 10649088,
+    "train_tokens": 307200000,
+    "fwd_per_step": 2
+  }
+}

results/g14-d12-L6-s1.json

@@ -0,0 +1,39 @@
+{
+  "args": {
+    "mode": "full",
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 12,
+    "seq_len": 40,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 6,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d12-L6-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 1.0,
+    "2": 1.0,
+    "3": 1.0,
+    "4": 1.0,
+    "6": 1.0,
+    "8": 1.0,
+    "12": 1.0,
+    "16": 1.0
+  },
+  "minutes": 17.586296463012694,
+  "compute": {
+    "params": 10649088,
+    "train_tokens": 307200000,
+    "fwd_per_step": 2
+  }
+}

results/g14-d16-L2-s0.json

@@ -0,0 +1,39 @@
+{
+  "args": {
+    "mode": "full",
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 16,
+    "seq_len": 52,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 2,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d16-L2-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0
+  },
+  "minutes": 7.497824923197428,
+  "compute": {
+    "params": 3557376,
+    "train_tokens": 399360000,
+    "fwd_per_step": 2
+  }
+}

results/g14-d16-L2-s1.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "full",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 16,
+    "seq_len": 52,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 2,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d16-L2-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "17": 0.0
+  },
+  "minutes": 7.427204525470733,
+  "compute": {
+    "params": 3557376,
+    "train_tokens": 399360000,
+    "fwd_per_step": 2
+  }
+}

results/g14-d16-L3-s0.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "full",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 16,
+    "seq_len": 52,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 3,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d16-L3-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.9975,
+    "2": 0.6275,
+    "3": 0.4825,
+    "4": 0.44499999999999995,
+    "6": 0.3825,
+    "8": 0.35,
+    "12": 0.3425,
+    "16": 0.315,
+    "17": 0.315
+  },
+  "minutes": 9.988970235983532,
+  "compute": {
+    "params": 5330304,
+    "train_tokens": 399360000,
+    "fwd_per_step": 2
+  }
+}

results/g14-d16-L3-s1.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "full",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 16,
+    "seq_len": 52,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 3,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d16-L3-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "17": 0.0
+  },
+  "minutes": 10.037210019429525,
+  "compute": {
+    "params": 5330304,
+    "train_tokens": 399360000,
+    "fwd_per_step": 2
+  }
+}

results/g14-d8-L12-s0.json

@@ -0,0 +1,39 @@
+{
+  "args": {
+    "mode": "full",
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 8,
+    "seq_len": 28,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 12,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d8-L12-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 1.0,
+    "2": 1.0,
+    "3": 1.0,
+    "4": 1.0,
+    "6": 1.0,
+    "8": 1.0,
+    "12": 1.0,
+    "16": 1.0
+  },
+  "minutes": 31.55148322979609,
+  "compute": {
+    "params": 21286656,
+    "train_tokens": 215040000,
+    "fwd_per_step": 2
+  }
+}

results/g14-d8-L12-s1.json

@@ -0,0 +1,39 @@
+{
+  "args": {
+    "mode": "full",
+    "K_max": 16,
+    "K": 4,
+    "lam": 1.0,
+    "digits": 8,
+    "seq_len": 28,
+    "n_train": 800000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 384,
+    "layers": 12,
+    "heads": 6,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 60000,
+    "lr": 0.0001,
+    "warmup": 4000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g14-d8-L12-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 1.0,
+    "2": 1.0,
+    "3": 1.0,
+    "4": 1.0,
+    "6": 1.0,
+    "8": 1.0,
+    "12": 1.0,
+    "16": 1.0
+  },
+  "minutes": 32.08436821699142,
+  "compute": {
+    "params": 21286656,
+    "train_tokens": 215040000,
+    "fwd_per_step": 2
+  }
+}

results/g15-finnish-distill-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "finnish",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-finnish-distill-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.48252918956043955,
+    "2": 0.4772493131868132,
+    "4": 0.5012985061813187,
+    "8": 0.5235770089285714,
+    "16": 0.5343191964285714,
+    "32": 0.5352957589285714,
+    "ar": 0.05587869162087912
+  },
+  "minutes": 4.343921728928884,
+  "compute": {
+    "params": 4769792,
+    "answer_len_mean": 15.12968635559082
+  }
+}

results/g15-finnish-distill-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "finnish",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-finnish-distill-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.5063637534340659,
+    "2": 0.5069754464285714,
+    "4": 0.5237594436813187,
+    "8": 0.5517148866758241,
+    "16": 0.5556211366758241,
+    "32": 0.5556211366758241,
+    "ar": 0.05587869162087912
+  },
+  "minutes": 4.371706000963846,
+  "compute": {
+    "params": 4769792,
+    "answer_len_mean": 15.12968635559082
+  }
+}

results/g15-finnish-matched-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "finnish",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-finnish-matched-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.5032194368131868,
+    "2": 0.5125879979395604,
+    "4": 0.5278803228021978,
+    "8": 0.5622746394230769,
+    "16": 0.5710637019230769,
+    "32": 0.5700871394230769,
+    "ar": 0.05685525412087912
+  },
+  "minutes": 2.444117879867554,
+  "compute": {
+    "params": 4769792,
+    "answer_len_mean": 15.12968635559082
+  }
+}

results/g15-finnish-matched-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "finnish",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-finnish-matched-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.535875257554945,
+    "2": 0.551747081043956,
+    "4": 0.5798849587912087,
+    "8": 0.5892535199175823,
+    "16": 0.6087847699175823,
+    "32": 0.6087847699175823,
+    "ar": 0.05783181662087912
+  },
+  "minutes": 2.467513418197632,
+  "compute": {
+    "params": 4769792,
+    "answer_len_mean": 15.12968635559082
+  }
+}

results/g15-georgian-distill-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "georgian",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-georgian-distill-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.9483173076923077,
+    "2": 0.9461388221153846,
+    "4": 0.9432091346153846,
+    "8": 0.9453876201923077,
+    "16": 0.9453876201923077,
+    "32": 0.9453876201923077,
+    "ar": 0.1258263221153846
+  },
+  "minutes": 4.3569332202275595,
+  "compute": {
+    "params": 4766208,
+    "answer_len_mean": 9.932000160217285
+  }
+}

results/g15-georgian-distill-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "georgian",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-georgian-distill-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.9385516826923077,
+    "2": 0.9385516826923077,
+    "4": 0.9395282451923077,
+    "8": 0.9375751201923077,
+    "16": 0.9375751201923077,
+    "32": 0.9375751201923077,
+    "ar": 0.12289663461538461
+  },
+  "minutes": 4.348151512940725,
+  "compute": {
+    "params": 4766208,
+    "answer_len_mean": 9.932000160217285
+  }
+}

results/g15-georgian-matched-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "georgian",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-georgian-matched-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.9441856971153846,
+    "2": 0.9471153846153846,
+    "4": 0.9510216346153846,
+    "8": 0.9500450721153846,
+    "16": 0.9490685096153846,
+    "32": 0.9490685096153846,
+    "ar": 0.1258263221153846
+  },
+  "minutes": 2.449139161904653,
+  "compute": {
+    "params": 4766208,
+    "answer_len_mean": 9.932000160217285
+  }
+}

results/g15-georgian-matched-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "georgian",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-georgian-matched-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.9412560096153846,
+    "2": 0.9402794471153846,
+    "4": 0.9390775240384616,
+    "8": 0.9378756009615384,
+    "16": 0.9378756009615384,
+    "32": 0.9378756009615384,
+    "ar": 0.12364783653846154
+  },
+  "minutes": 2.45454421043396,
+  "compute": {
+    "params": 4766208,
+    "answer_len_mean": 9.932000160217285
+  }
+}

results/g15-german-distill-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "german",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-german-distill-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.77875,
+    "2": 0.7770703125,
+    "4": 0.7834765625,
+    "8": 0.7815234375,
+    "16": 0.7854296875,
+    "32": 0.7854296875,
+    "ar": 0.0795703125
+  },
+  "minutes": 4.314774318536123,
+  "compute": {
+    "params": 4776448,
+    "answer_len_mean": 10.743975639343262
+  }
+}

results/g15-navajo-distill-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "navajo",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-navajo-distill-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.33743990384615385,
+    "2": 0.3449519230769231,
+    "4": 0.36763822115384615,
+    "8": 0.3858924278846154,
+    "16": 0.3897986778846154,
+    "32": 0.3897986778846154,
+    "ar": 0.07474459134615385
+  },
+  "minutes": 4.2984689474105835,
+  "compute": {
+    "params": 4765696,
+    "answer_len_mean": 10.005999565124512
+  }
+}

results/g15-navajo-distill-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "navajo",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-navajo-distill-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.24166165865384615,
+    "2": 0.2635967548076923,
+    "4": 0.2813251201923077,
+    "8": 0.3040114182692308,
+    "16": 0.3079176682692308,
+    "32": 0.3079176682692308,
+    "ar": 0.06377704326923077
+  },
+  "minutes": 4.306906080245971,
+  "compute": {
+    "params": 4765696,
+    "answer_len_mean": 10.005999565124512
+  }
+}

results/g15-navajo-matched-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "navajo",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-navajo-matched-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.310546875,
+    "2": 0.3400691105769231,
+    "4": 0.36275540865384615,
+    "8": 0.3937049278846154,
+    "16": 0.3927283653846154,
+    "32": 0.3927283653846154,
+    "ar": 0.07767427884615385
+  },
+  "minutes": 2.491582691669464,
+  "compute": {
+    "params": 4765696,
+    "answer_len_mean": 10.005999565124512
+  }
+}

results/g15-navajo-matched-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "navajo",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-navajo-matched-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.2626201923076923,
+    "2": 0.2931189903846154,
+    "4": 0.31212439903846156,
+    "8": 0.3390925480769231,
+    "16": 0.3400691105769231,
+    "32": 0.3400691105769231,
+    "ar": 0.0645282451923077
+  },
+  "minutes": 2.493058661619822,
+  "compute": {
+    "params": 4765696,
+    "answer_len_mean": 10.005999565124512
+  }
+}

results/g15-russian-distill-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "russian",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-russian-distill-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.7351542771464646,
+    "2": 0.7225773358585859,
+    "4": 0.7267696496212122,
+    "8": 0.7284367108585859,
+    "16": 0.7303898358585859,
+    "32": 0.7303898358585859,
+    "ar": 0.035896070075757576
+  },
+  "minutes": 4.300656799475352,
+  "compute": {
+    "params": 4770304,
+    "answer_len_mean": 11.276381492614746
+  }
+}

results/g15-russian-distill-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "russian",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-russian-distill-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.7418718434343434,
+    "2": 0.7421579071969697,
+    "4": 0.7457780934343434,
+    "8": 0.7460641571969697,
+    "16": 0.745373658459596,
+    "32": 0.745373658459596,
+    "ar": 0.03813525883838384
+  },
+  "minutes": 4.345847002665201,
+  "compute": {
+    "params": 4770304,
+    "answer_len_mean": 11.276381492614746
+  }
+}

results/g15-russian-matched-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "russian",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-russian-matched-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.7305575284090909,
+    "2": 0.7367029671717171,
+    "4": 0.7263652146464646,
+    "8": 0.7283183396464646,
+    "16": 0.7332011521464646,
+    "32": 0.7332011521464646,
+    "ar": 0.032966382575757576
+  },
+  "minutes": 2.497041336695353,
+  "compute": {
+    "params": 4770304,
+    "answer_len_mean": 11.276381492614746
+  }
+}

results/g15-russian-matched-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "russian",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-russian-matched-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.7626657196969697,
+    "2": 0.759045533459596,
+    "4": 0.7642144097222222,
+    "8": 0.760022095959596,
+    "16": 0.761975220959596,
+    "32": 0.761975220959596,
+    "ar": 0.03911182133838384
+  },
+  "minutes": 2.4522072474161782,
+  "compute": {
+    "params": 4770304,
+    "answer_len_mean": 11.276381492614746
+  }
+}

results/g15-turkish-distill-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "turkish",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-turkish-distill-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.7729124663978495,
+    "2": 0.7768187163978495,
+    "4": 0.7752331149193549,
+    "8": 0.783171622983871,
+    "16": 0.789030997983871,
+    "32": 0.789030997983871,
+    "ar": 0.09997689852150538
+  },
+  "minutes": 4.311886747678121,
+  "compute": {
+    "params": 4772864,
+    "answer_len_mean": 13.684530258178711
+  }
+}

results/g15-turkish-distill-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "turkish",
+    "size": "medium",
+    "mode": "distill",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-turkish-distill-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.8306661626344086,
+    "2": 0.8190629200268817,
+    "4": 0.8260248655913979,
+    "8": 0.8345724126344086,
+    "16": 0.838846186155914,
+    "32": 0.838846186155914,
+    "ar": 0.099609375
+  },
+  "minutes": 4.282213552792867,
+  "compute": {
+    "params": 4772864,
+    "answer_len_mean": 13.684530258178711
+  }
+}

results/g15-turkish-matched-s0.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "turkish",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 0,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-turkish-matched-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.7699932795698925,
+    "2": 0.7765877016129032,
+    "4": 0.7892830141129032,
+    "8": 0.8050340221774194,
+    "16": 0.8093077956989247,
+    "32": 0.8093077956989247,
+    "ar": 0.09997689852150538
+  },
+  "minutes": 2.4571212331453958,
+  "compute": {
+    "params": 4772864,
+    "answer_len_mean": 13.684530258178711
+  }
+}

results/g15-turkish-matched-s1.json

@@ -0,0 +1,36 @@
+{
+  "args": {
+    "lang": "turkish",
+    "size": "medium",
+    "mode": "matched",
+    "seq_len": 48,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 64,
+    "steps": 15000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "K": 4,
+    "lam": 1.0,
+    "seed": 1,
+    "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig",
+    "eval_bs": 128,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g15-turkish-matched-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.8041729670698925,
+    "2": 0.8061260920698925,
+    "4": 0.8206590221774194,
+    "8": 0.8376281081989247,
+    "16": 0.8379956317204301,
+    "32": 0.8379956317204301,
+    "ar": 0.10388314852150538
+  },
+  "minutes": 2.4274060289065043,
+  "compute": {
+    "params": 4772864,
+    "answer_len_mean": 13.684530258178711
+  }
+}

results/g17-d16-mdlm2x-s0.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "mdlm",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 16,
+    "seq_len": 52,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 300000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g17-d16-mdlm2x-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 1.0,
+    "2": 1.0,
+    "3": 1.0,
+    "4": 1.0,
+    "6": 1.0,
+    "8": 1.0,
+    "12": 1.0,
+    "16": 1.0,
+    "17": 1.0
+  },
+  "minutes": 103.74741363922755,
+  "compute": {
+    "params": 37819392,
+    "train_tokens": 1996800000,
+    "fwd_per_step": 1
+  }
+}

results/g17-d16-mdlm2x-s1.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "mdlm",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 16,
+    "seq_len": 52,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 300000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g17-d16-mdlm2x-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "17": 0.0
+  },
+  "minutes": 104.48015933036804,
+  "compute": {
+    "params": 37819392,
+    "train_tokens": 1996800000,
+    "fwd_per_step": 1
+  }
+}

results/g17-d20-mdlm2x-s0.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "mdlm",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 300000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g17-d20-mdlm2x-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "21": 0.0
+  },
+  "minutes": 116.37444167931875,
+  "compute": {
+    "params": 37819392,
+    "train_tokens": 2457600000,
+    "fwd_per_step": 1
+  }
+}

results/g17-d20-mdlm2x-s1.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "mdlm",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 300000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g17-d20-mdlm2x-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "21": 0.0
+  },
+  "minutes": 117.45276789665222,
+  "compute": {
+    "params": 37819392,
+    "train_tokens": 2457600000,
+    "fwd_per_step": 1
+  }
+}

results/g18-d20-anybudget-s0.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "anybudget",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 150000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g18-d20-anybudget-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "21": 0.0
+  },
+  "minutes": 58.222007815043135,
+  "compute": {
+    "params": 37824512,
+    "train_tokens": 1228800000,
+    "fwd_per_step": 1
+  }
+}

results/g18-d20-anybudget-s1.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "anybudget",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 150000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g18-d20-anybudget-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "21": 0.0
+  },
+  "minutes": 58.380110347270964,
+  "compute": {
+    "params": 37824512,
+    "train_tokens": 1228800000,
+    "fwd_per_step": 1
+  }
+}

results/g18-d20-anybudget-s2.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "anybudget",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 150000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 2,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g18-d20-anybudget-s2"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "21": 0.0
+  },
+  "minutes": 58.75475962559382,
+  "compute": {
+    "params": 37824512,
+    "train_tokens": 1228800000,
+    "fwd_per_step": 1
+  }
+}

results/g18-d20-anybudget_nc-s0.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "anybudget_nc",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 150000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 0,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g18-d20-anybudget_nc-s0"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0075,
+    "2": 0.01,
+    "3": 0.01,
+    "4": 0.01,
+    "6": 0.005,
+    "8": 0.0125,
+    "12": 0.0075,
+    "16": 0.01,
+    "21": 0.01
+  },
+  "minutes": 58.74203004837036,
+  "compute": {
+    "params": 37819392,
+    "train_tokens": 1228800000,
+    "fwd_per_step": 1
+  }
+}

results/g18-d20-anybudget_nc-s1.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "anybudget_nc",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 150000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 1,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g18-d20-anybudget_nc-s1"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "21": 0.0
+  },
+  "minutes": 58.15160437424978,
+  "compute": {
+    "params": 37819392,
+    "train_tokens": 1228800000,
+    "fwd_per_step": 1
+  }
+}

results/g18-d20-anybudget_nc-s2.json

@@ -0,0 +1,41 @@
+{
+  "args": {
+    "mode": "anybudget_nc",
+    "budget_bins": 10,
+    "K_max": 16,
+    "K": 8,
+    "lam": 1.0,
+    "digits": 20,
+    "seq_len": 64,
+    "n_train": 1500000,
+    "n_eval": 400,
+    "eval_bs": 100,
+    "d": 512,
+    "layers": 12,
+    "heads": 8,
+    "pe": "rope",
+    "bs": 128,
+    "steps": 150000,
+    "lr": 0.0001,
+    "warmup": 6000,
+    "seed": 2,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g18-d20-anybudget_nc-s2"
+  },
+  "accuracy_by_passes": {
+    "1": 0.0,
+    "2": 0.0,
+    "3": 0.0,
+    "4": 0.0,
+    "6": 0.0,
+    "8": 0.0,
+    "12": 0.0,
+    "16": 0.0,
+    "21": 0.0
+  },
+  "minutes": 58.52561570803324,
+  "compute": {
+    "params": 37819392,
+    "train_tokens": 1228800000,
+    "fwd_per_step": 1
+  }
+}

results/gr1-2regular-anybudget-s0.json

@@ -0,0 +1,78 @@
+{
+  "args": {
+    "family": "2regular",
+    "n": 6,
+    "mode": "anybudget",
+    "budget_bins": 8,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "bs": 256,
+    "steps": 30000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "seed": 0,
+    "n_gen": 4096,
+    "gen_bs": 512,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/gr1-2regular-anybudget-s0"
+  },
+  "exact": {
+    "family": "2regular",
+    "n": 6,
+    "n_edges": 15,
+    "size": 70,
+    "sum_marginal_entropy_bits": 14.56425891682003,
+    "joint_entropy_bits": 6.129283016944966,
+    "total_correlation_bits": 8.434975899875063,
+    "one_pass_ceiling": 0.002889476997120002,
+    "bound_holds": true
+  },
+  "decode": {
+    "fixedK_1": {
+      "validity": 0.00390625,
+      "nfe": 1.0,
+      "unique": 0.9375
+    },
+    "fixedK_2": {
+      "validity": 0.250732421875,
+      "nfe": 2.0,
+      "unique": 0.06134371957156767
+    },
+    "fixedK_4": {
+      "validity": 0.607666015625,
+      "nfe": 4.0,
+      "unique": 0.02812374447569305
+    },
+    "fixedK_8": {
+      "validity": 0.91162109375,
+      "nfe": 8.0,
+      "unique": 0.018746652383502947
+    },
+    "fixedK_15": {
+      "validity": 0.845703125,
+      "nfe": 15.0,
+      "unique": 0.015877598152424944
+    },
+    "entbudget_0.05": {
+      "validity": 0.982177734375,
+      "nfe": 13.0,
+      "unique": 0.017399950285856326
+    },
+    "entbudget_0.25": {
+      "validity": 0.982421875,
+      "nfe": 13.0,
+      "unique": 0.01739562624254473
+    },
+    "entbudget_1.0": {
+      "validity": 0.96435546875,
+      "nfe": 11.0,
+      "unique": 0.017721518987341773
+    },
+    "entbudget_4.0": {
+      "validity": 0.13330078125,
+      "nfe": 3.0,
+      "unique": 0.10989010989010989
+    }
+  },
+  "reached_ceiling": true
+}

results/gr1-2regular-anybudget-s1.json

@@ -0,0 +1,78 @@
+{
+  "args": {
+    "family": "2regular",
+    "n": 6,
+    "mode": "anybudget",
+    "budget_bins": 8,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "bs": 256,
+    "steps": 30000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "seed": 1,
+    "n_gen": 4096,
+    "gen_bs": 512,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/gr1-2regular-anybudget-s1"
+  },
+  "exact": {
+    "family": "2regular",
+    "n": 6,
+    "n_edges": 15,
+    "size": 70,
+    "sum_marginal_entropy_bits": 14.56425891682003,
+    "joint_entropy_bits": 6.129283016944966,
+    "total_correlation_bits": 8.434975899875063,
+    "one_pass_ceiling": 0.002889476997120002,
+    "bound_holds": true
+  },
+  "decode": {
+    "fixedK_1": {
+      "validity": 0.002197265625,
+      "nfe": 1.0,
+      "unique": 0.8888888888888888
+    },
+    "fixedK_2": {
+      "validity": 0.25732421875,
+      "nfe": 2.0,
+      "unique": 0.061669829222011384
+    },
+    "fixedK_4": {
+      "validity": 0.671142578125,
+      "nfe": 4.0,
+      "unique": 0.025463805020007276
+    },
+    "fixedK_8": {
+      "validity": 0.824462890625,
+      "nfe": 8.0,
+      "unique": 0.020728457210541902
+    },
+    "fixedK_15": {
+      "validity": 0.2802734375,
+      "nfe": 15.0,
+      "unique": 0.020905923344947737
+    },
+    "entbudget_0.05": {
+      "validity": 0.991455078125,
+      "nfe": 13.0,
+      "unique": 0.017237133710908643
+    },
+    "entbudget_0.25": {
+      "validity": 0.991943359375,
+      "nfe": 13.0,
+      "unique": 0.017228648781688408
+    },
+    "entbudget_1.0": {
+      "validity": 0.975341796875,
+      "nfe": 10.0,
+      "unique": 0.017521902377972465
+    },
+    "entbudget_4.0": {
+      "validity": 0.3955078125,
+      "nfe": 3.0,
+      "unique": 0.04197530864197531
+    }
+  },
+  "reached_ceiling": false
+}

results/gr1-2regular-mdlm-s0.json

@@ -0,0 +1,78 @@
+{
+  "args": {
+    "family": "2regular",
+    "n": 6,
+    "mode": "mdlm",
+    "budget_bins": 8,
+    "d": 256,
+    "layers": 6,
+    "heads": 8,
+    "bs": 256,
+    "steps": 30000,
+    "lr": 0.0001,
+    "warmup": 1000,
+    "seed": 0,
+    "n_gen": 4096,
+    "gen_bs": 512,
+    "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/gr1-2regular-mdlm-s0"
+  },
+  "exact": {
+    "family": "2regular",
+    "n": 6,
+    "n_edges": 15,
+    "size": 70,
+    "sum_marginal_entropy_bits": 14.56425891682003,
+    "joint_entropy_bits": 6.129283016944966,
+    "total_correlation_bits": 8.434975899875063,
+    "one_pass_ceiling": 0.002889476997120002,
+    "bound_holds": true
+  },
+  "decode": {
+    "fixedK_1": {
+      "validity": 0.00244140625,
+      "nfe": 1.0,
+      "unique": 0.8
+    },
+    "fixedK_2": {
+      "validity": 0.240478515625,
+      "nfe": 2.0,
+      "unique": 0.06294416243654823
+    },
+    "fixedK_4": {
+      "validity": 0.6591796875,
+      "nfe": 4.0,
+      "unique": 0.025555555555555557
+    },
+    "fixedK_8": {
+      "validity": 0.951171875,
+      "nfe": 8.0,
+      "unique": 0.017967145790554414
+    },
+    "fixedK_15": {
+      "validity": 0.999755859375,
+      "nfe": 15.0,
+      "unique": 0.017094017094017096
+    },
+    "entbudget_0.05": {
+      "validity": 0.99951171875,
+      "nfe": 13.0,
+      "unique": 0.01709819247679531
+    },
+    "entbudget_0.25": {
+      "validity": 0.999755859375,
+      "nfe": 13.0,
+      "unique": 0.017094017094017096
+    },
+    "entbudget_1.0": {
+      "validity": 0.989501953125,
+      "nfe": 10.0,
+      "unique": 0.017271157167530225
+    },
+    "entbudget_4.0": {
+      "validity": 0.28466796875,
+      "nfe": 3.0,
+      "unique": 0.05660377358490566
+    }
+  },
+  "reached_ceiling": false
+}