diff --git a/README.md b/README.md index 6a5a6f00d2d085da2a586f0a28eac8e99d68a8f1..44340535306bda94a3d15344ce9ab307b2b3fb35 100644 --- a/README.md +++ b/README.md @@ -3,120 +3,85 @@ license: apache-2.0 tags: - masked-diffusion - discrete-diffusion - - parallel-decoding - constraint-satisfaction - - text8 - - qm9 - sudoku + - recurrent-transformer - from-scratch --- -# Two Failure Modes of Parallel Decoding in Masked Diffusion +# Recurrent Masked Diffusion for Constraint Satisfaction -Everything here is trained from scratch: no pretrained weights, no pretrained -tokenizer, and the autoregressive model used to score generated text is itself -trained from scratch on the same corpus. +**Constraint propagation is an iterative algorithm; masked diffusion models are +trained as fixed-depth denoisers. Making the denoiser recurrent — one +weight-shared block applied many times, supervised at every application — +reaches 99.70% on hard Sudoku with 212k parameters, where a 37.9M-parameter +feed-forward denoiser reaches 88.90%.** -Code: https://github.com/GIND123/MSML612 - ---- - -## The claim - -Parallel decoding in masked diffusion loses accuracy for two reasons that need -**opposite** fixes, and the literature treats them as one. - -| | failure mode | fixable by | -|---|---|---| -| **(i)** | the fully-masked state is never trained — a *K*-pass decode starts at `t = 1`, and continuous `t ~ U(0,1)` assigns that state probability **zero** | **training only** | -| **(ii)** | committed positions carry real mutual information — parallel decoding samples the product of marginals when the truth is the joint | **inference only** | - -Prior work applies inference-time fixes to both, which is why those methods -flatline on arithmetic, where (ii) is identically zero. +Trained from scratch. No pretrained weights, no pretrained tokenizer. -## The bound - -For a data distribution `P` uniform on a family `F`, let `V*` be the one-pass -validity achievable from the **true** marginals. Then - -``` -V* ≥ 2^(−TC), TC = D_KL( P ‖ ∏ᵢ pᵢ ), equality iff ∏ᵢ pᵢ is constant on F -``` - -**Every bit of total correlation among jointly committed variables at most halves -the one-pass success probability.** +Code: https://github.com/GIND123/MSML612 -Verified by exhaustive enumeration over graph families, with the enumerator itself -checked against Cayley's formula (spanning trees at n = 5, 6, 7 → 125, 1296, -16807): +## Benchmark -| family | TC (bits) | V* | 2^(−TC) | ratio | -|---|---|---|---|---| -| bipartite (n=7) | 0.026 | 98.2816% | 98.1817% | 1.001 | -| tree (n=6) | 3.435 | **9.2488%** | **9.2488%** | **1.00000** | -| matching (n=6) | 6.922 | **0.8246%** | **0.8246%** | **1.00000** | -| 2-regular (n=7) | 10.423 | **0.0728%** | **0.0728%** | **1.00000** | +1,000 held-out minimal Sudoku puzzles, 20–30 clues (mean 24.4). Uniqueness +verified after **every** cell removal; train/test solutions disjoint even up to +digit relabelling. Metric: board accuracy, all 81 cells correct. -The inequality holds in every case; equality holds to five decimals on exactly -the fixed-edge-count families, and fails precisely where edge counts vary — which -is what the Jensen step requires. +## Results -## Finding 1 — train the schedule you intend to decode at +| method | params | 1 pass | best | +|---|---|---|---| +| constraint propagation (no search) | – | – | 3.00% | +| greedy MRV | – | – | 5.90% | +| autoregressive prefix-LM | 2.4M | – | 0.30% | +| direct recurrent classifier* | 212k | 10.5% | 31.00% | +| feed-forward diffusion, MDLM | 37.9M | 4.40% | 88.15% | +| feed-forward diffusion, schedule-matched | 37.9M | 40.10% | 88.90% | +| 32 distinct layers, no weight sharing | 6.34M | 70.20% | 94.50% | +| recurrent diffusion, MDLM | 212k | 87.35% | 96.55% | +| recurrent, no deep supervision | 212k | 93.35% | 97.75% | +| **recurrent diffusion, full method** | **212k** | **97.35%** | **99.70%** | +| full backtracking search | – | – | 100% (423 nodes) | -| task | baseline | ours | -|---|---|---| -| 20-digit addition, 1 pass | **0.00%** (also 0.00% at **2× compute**, both seeds) | **99.9%** (3/3 seeds) | -| text8, NFE 16 | 116.88 gen-ppl | **92.74** (−20.7%) | -| text8, validation BPC | 1.7292 | **1.7010** | +\* Our reimplementation of the published recurrent classifier reaches 31% +against their reported 99.5%. **That is a failed reproduction on our side, not +evidence against their method**, and it is not used as a comparison. -The text8 gain appears in generative quality **and** likelihood simultaneously, -so it is not bought by trading one for the other. +## Ablations -A 48-run ablation isolates the mechanism: the `t = 1` term alone removes **49%** -of the parallelism penalty, against 17% for reshaping the rest of the schedule. +| ingredient | gain | +|---|---| +| recurrence vs feed-forward | +10.80 | +| weight sharing vs 32 distinct layers (matched depth) | +3.75 | +| schedule matching | +3.15 | +| deep supervision | +1.95 | -## Finding 2 — one-pass decoding needs absolute position information +## Recurrence extrapolates past its training depth -At `t = 1` every input token is `[MASK]`, so the sequence is constant. With a -relative encoding the attention score depends only on `(i − j)`, and identical -value vectors give identical outputs at every position. **Position-dependent -marginals are inexpressible at any training budget.** +Trained at R = 32, evaluated elsewhere without retraining: -| condition | learned (APE) | parameter-free (sinusoidal) | -|---|---|---| -| marginals **vary** | **+88.037 pts** | **+88.074 pts** | -| marginals **constant** (control) | **−0.024 pts** | **+0.000 pts** | +| R | 1 | 4 | 8 | 16 | **32** | **64** | 128 | +|---|---|---|---|---|---|---|---| +| board | 0.10% | 48.40% | 82.00% | 92.40% | **98.30%** | **99.40%** | 99.10% / 39.70% | -A ~3,700× dissociation. The parameter-free variant matching the learned one rules -out capacity; the control rules out "absolute is simply better". Predicted 5/5 -before measurement, and the treated arm lands at **100.3% of `V*`** — at the -bound, not merely higher. +2× extrapolation is reliable; 4× collapses on one seed. -## Benchmark: SATNet Sudoku +## The instructive failure -9x9 Sudoku, SATNet's conventional 9,000/1,000 split, **board accuracy** = all 81 -cells correct. Published number: **98.3%** (Wang et al., ICML 2019). +Both non-diffusion baselines reach **lower training loss** than our method and +solve almost nothing — AR at 0.0486 loss gives 0.30% boards, direct all-cell +supervision at 0.0722 gives 55%. AR's errors compound unrecoverably; all-cell +supervision is dominated by copying the given clues. Low loss, no capability. -Un-augmented arm, which is the only like-for-like comparison: +## Honest limits -| denoising passes | board accuracy | -|---|---| -| **1** | **87.85%** | -| **2** | **98.65%** | -| 8 | 99.70% | -| **32** | **100.00%** | - -Two passes exceed the published number; 32 passes solve **all 1,000** test -puzzles exactly. One pass — the entire grid committed at once — solves 87.85%. - -Leakage audited first: 0/1000 test puzzles in training, 0/1000 test solutions in -training, and 0/1000 test solutions matching a training solution even up to digit -relabelling. All 10,000 solutions are distinct. - -**Scoped honestly:** this comes from the *plain MDLM baseline* with fixed-`K` -decoding. Neither finding above contributes — the standard formulation already -saturates this benchmark. It is a result for masked diffusion on SATNet Sudoku, -not evidence for our method. +- Published comparison is across **different puzzle distributions**: our 20–30 + clue range sits inside the Recurrent Transformer's 17–34 but excludes the + hardest 17-clue instances, so 99.70% at 212k reads as *competitive with* their + 99.5% at 211k, not better. +- **Input injection is not separated from weight sharing** in the 32-layer + comparison; the isolating run is in progress. +- The SATNet split is not used: constraint propagation alone solves 100% of it. ## Negative results, reported diff --git a/figures/compare_recurrent.json b/figures/compare_recurrent.json new file mode 100644 index 0000000000000000000000000000000000000000..3bac109e1d0084955cba123f28305f62c1cb5e17 --- /dev/null +++ b/figures/compare_recurrent.json @@ -0,0 +1,119 @@ +{ + "n": 1000, + "clues": { + "min": 21, + "max": 28, + "mean": 24.318 + }, + "classical": { + "propagation": 0.03, + "greedy_mrv": 0.059, + "backtracking": 1.0, + "backtrack_nodes_mean": 423.126, + "backtrack_nodes_median": 211.5 + }, + "diffusion": { + "recurrent-full": { + "1": { + "board": 0.974, + "per_seed": [ + 0.974, + 0.974 + ] + }, + "4": { + "board": 0.985, + "per_seed": [ + 0.982, + 0.988 + ] + }, + "8": { + "board": 0.9895, + "per_seed": [ + 0.987, + 0.992 + ] + }, + "16": { + "board": 0.992, + "per_seed": [ + 0.99, + 0.994 + ] + }, + "32": { + "board": 0.995, + "per_seed": [ + 0.993, + 0.997 + ] + }, + "61": { + "board": 0.9955, + "per_seed": [ + 0.994, + 0.997 + ], + "by_clue": { + "22": 0.9767441860465116, + "23": 1.0, + "24": 0.9942028985507246, + "25": 0.9964539007092199, + "26": 0.984251968503937 + } + } + }, + "recurrent-mdlm": { + "1": { + "board": 0.8705, + "per_seed": [ + 0.911, + 0.83 + ] + }, + "4": { + "board": 0.9155, + "per_seed": [ + 0.938, + 0.893 + ] + }, + "8": { + "board": 0.9315, + "per_seed": [ + 0.94, + 0.923 + ] + }, + "16": { + "board": 0.947, + "per_seed": [ + 0.951, + 0.943 + ] + }, + "32": { + "board": 0.949, + "per_seed": [ + 0.954, + 0.944 + ] + }, + "61": { + "board": 0.9515, + "per_seed": [ + 0.957, + 0.946 + ], + "by_clue": { + "22": 0.9534883720930233, + "23": 0.9723756906077348, + "24": 0.9652173913043478, + "25": 0.9468085106382979, + "26": 0.952755905511811 + } + } + } + } +} \ No newline at end of file diff --git a/figures/mol_fair.json b/figures/mol_fair.json new file mode 100644 index 0000000000000000000000000000000000000000..2c3f0d1c508550f150228915f7d9bffea91111cd --- /dev/null +++ b/figures/mol_fair.json @@ -0,0 +1,106 @@ +{ + "matched-ape": { + "1": [ + 0.3076171875, + 0.30078125 + ], + "4": [ + 0.5634765625, + 0.5458984375 + ], + "8": [ + 0.70068359375, + 0.7294921875 + ], + "16": [ + 0.8759765625, + 0.86572265625 + ], + "32": [ + 0.97119140625, + 0.970703125 + ], + "45": [ + 0.9951171875, + 0.998046875 + ] + }, + "matched-rope": { + "1": [ + 0.173828125, + 0.17236328125 + ], + "4": [ + 0.40771484375, + 0.4462890625 + ], + "8": [ + 0.6943359375, + 0.5380859375 + ], + "16": [ + 0.74072265625, + 0.57958984375 + ], + "32": [ + 0.7822265625, + 0.62939453125 + ], + "45": [ + 0.7880859375, + 0.705078125 + ] + }, + "mdlm-ape": { + "1": [ + 0.3115234375, + 0.30810546875 + ], + "4": [ + 0.5458984375, + 0.5400390625 + ], + "8": [ + 0.7041015625, + 0.72314453125 + ], + "16": [ + 0.85693359375, + 0.86181640625 + ], + "32": [ + 0.9638671875, + 0.9609375 + ], + "45": [ + 0.99853515625, + 0.998046875 + ] + }, + "mdlm-rope": { + "1": [ + 0.18359375, + 0.166015625 + ], + "4": [ + 0.4169921875, + 0.41845703125 + ], + "8": [ + 0.7265625, + 0.68505859375 + ], + "16": [ + 0.7861328125, + 0.75732421875 + ], + "32": [ + 0.92578125, + 0.75 + ], + "45": [ + 0.97607421875, + 0.806640625 + ] + } +} \ No newline at end of file diff --git a/figures/sudoku_compare.json b/figures/sudoku_compare.json new file mode 100644 index 0000000000000000000000000000000000000000..d39bf0f24fd3dfa04f09c5b89aea390480859c22 --- /dev/null +++ b/figures/sudoku_compare.json @@ -0,0 +1,119 @@ +{ + "n": 1000, + "clues": { + "min": 21, + "max": 28, + "mean": 24.318 + }, + "classical": { + "propagation": 0.03, + "greedy_mrv": 0.059, + "backtracking": 1.0, + "backtrack_nodes_mean": 423.126, + "backtrack_nodes_median": 211.5 + }, + "diffusion": { + "full": { + "1": { + "board": 0.40349999999999997, + "per_seed": [ + 0.421, + 0.386 + ] + }, + "4": { + "board": 0.7925, + "per_seed": [ + 0.786, + 0.799 + ] + }, + "8": { + "board": 0.8445, + "per_seed": [ + 0.84, + 0.849 + ] + }, + "16": { + "board": 0.855, + "per_seed": [ + 0.842, + 0.868 + ] + }, + "32": { + "board": 0.868, + "per_seed": [ + 0.861, + 0.875 + ] + }, + "61": { + "board": 0.894, + "per_seed": [ + 0.891, + 0.897 + ], + "by_clue": { + "22": 0.8372093023255814, + "23": 0.9171270718232044, + "24": 0.8898550724637682, + "25": 0.8936170212765957, + "26": 0.889763779527559 + } + } + }, + "mdlm": { + "1": { + "board": 0.046, + "per_seed": [ + 0.057, + 0.035 + ] + }, + "4": { + "board": 0.709, + "per_seed": [ + 0.727, + 0.691 + ] + }, + "8": { + "board": 0.783, + "per_seed": [ + 0.798, + 0.768 + ] + }, + "16": { + "board": 0.843, + "per_seed": [ + 0.861, + 0.825 + ] + }, + "32": { + "board": 0.8645, + "per_seed": [ + 0.864, + 0.865 + ] + }, + "61": { + "board": 0.874, + "per_seed": [ + 0.884, + 0.864 + ], + "by_clue": { + "22": 0.8372093023255814, + "23": 0.9116022099447514, + "24": 0.8956521739130435, + "25": 0.900709219858156, + "26": 0.8031496062992126 + } + } + } + } +} \ No newline at end of file diff --git a/figures/sudoku_hard.json b/figures/sudoku_hard.json new file mode 100644 index 0000000000000000000000000000000000000000..a973692d3ccbe096d945634d5e09692d27071406 --- /dev/null +++ b/figures/sudoku_hard.json @@ -0,0 +1,86 @@ +{ + "full-aug|30": { + "fixedK_1": 0.0, + "fixedK_8": 0.5249999999999999, + "fixedK_45": 0.7208333333333333, + "entbudget_0.01": 0.11666666666666667, + "entbudget_0.2": 0.008333333333333333 + }, + "full-aug|26": { + "fixedK_1": 0.0, + "fixedK_8": 0.18333333333333332, + "fixedK_45": 0.3375, + "entbudget_0.01": 0.024999999999999998, + "entbudget_0.2": 0.004166666666666667 + }, + "full-aug|22": { + "fixedK_1": 0.0, + "fixedK_8": 0.12083333333333333, + "fixedK_45": 0.26666666666666666, + "entbudget_0.01": 0.004166666666666667, + "entbudget_0.2": 0.0 + }, + "full-plain|30": { + "fixedK_1": 0.0, + "fixedK_8": 0.5375, + "fixedK_45": 0.6458333333333333, + "entbudget_0.01": 0.04583333333333334, + "entbudget_0.2": 0.0 + }, + "full-plain|26": { + "fixedK_1": 0.0, + "fixedK_8": 0.23333333333333334, + "fixedK_45": 0.37916666666666665, + "entbudget_0.01": 0.008333333333333333, + "entbudget_0.2": 0.004166666666666667 + }, + "full-plain|22": { + "fixedK_1": 0.0, + "fixedK_8": 0.075, + "fixedK_45": 0.19583333333333333, + "entbudget_0.01": 0.0125, + "entbudget_0.2": 0.0 + }, + "mdlm-aug|30": { + "fixedK_1": 0.0, + "fixedK_8": 0.4916666666666667, + "fixedK_45": 0.6791666666666667, + "entbudget_0.01": 0.475, + "entbudget_0.2": 0.16249999999999998 + }, + "mdlm-aug|26": { + "fixedK_1": 0.0, + "fixedK_8": 0.2125, + "fixedK_45": 0.33333333333333337, + "entbudget_0.01": 0.15833333333333333, + "entbudget_0.2": 0.008333333333333333 + }, + "mdlm-aug|22": { + "fixedK_1": 0.0, + "fixedK_8": 0.10416666666666666, + "fixedK_45": 0.16666666666666666, + "entbudget_0.01": 0.12083333333333333, + "entbudget_0.2": 0.0125 + }, + "mdlm-plain|30": { + "fixedK_1": 0.0, + "fixedK_8": 0.39583333333333337, + "fixedK_45": 0.5875, + "entbudget_0.01": 0.20833333333333331, + "entbudget_0.2": 0.020833333333333332 + }, + "mdlm-plain|26": { + "fixedK_1": 0.0, + "fixedK_8": 0.14583333333333331, + "fixedK_45": 0.2875, + "entbudget_0.01": 0.06666666666666667, + "entbudget_0.2": 0.0 + }, + "mdlm-plain|22": { + "fixedK_1": 0.0, + "fixedK_8": 0.03333333333333333, + "fixedK_45": 0.13333333333333333, + "entbudget_0.01": 0.016666666666666666, + "entbudget_0.2": 0.0 + } +} \ No newline at end of file diff --git a/figures/sudoku_remask.json b/figures/sudoku_remask.json new file mode 100644 index 0000000000000000000000000000000000000000..ce5c5e25ef08a69cef955c6a1811806d409cabeb --- /dev/null +++ b/figures/sudoku_remask.json @@ -0,0 +1,190 @@ +{ + "sudh-full-s0": { + "fixedK_16": { + "board": 0.85, + "cell": 0.9471260717688155, + "nfe": 16.0 + }, + "fixedK_61": { + "board": 0.877, + "cell": 0.9471966409089305, + "nfe": 59.5 + }, + "entbudget": { + "board": 0.882, + "cell": 0.9523305458522988, + "nfe": 34.5 + }, + "verifier_b32_r4": { + "board": 0.886, + "cell": 0.9486433082812886, + "nfe": 154.75 + }, + "verifier_b32_r8": { + "board": 0.886, + "cell": 0.9505486750643943, + "nfe": 183.75 + }, + "verifier_b61_r4": { + "board": 0.884, + "cell": 0.9470555026287004, + "nfe": 217.25 + }, + "verifier_b61_r8": { + "board": 0.884, + "cell": 0.9467379414981828, + "nfe": 253.75 + }, + "verifier_narrow": { + "board": 0.878, + "cell": 0.9480611128753396, + "nfe": 89.5 + }, + "random_remask_control": { + "board": 0.888, + "cell": 0.9505486750643943, + "nfe": 174.25 + } + }, + "sudh-full-s1": { + "fixedK_16": { + "board": 0.865, + "cell": 0.9489608694118062, + "nfe": 16.0 + }, + "fixedK_61": { + "board": 0.898, + "cell": 0.9554179457323313, + "nfe": 59.5 + }, + "entbudget": { + "board": 0.887, + "cell": 0.9549063194664973, + "nfe": 30.0 + }, + "verifier_b32_r4": { + "board": 0.895, + "cell": 0.9561942062735965, + "nfe": 151.25 + }, + "verifier_b32_r8": { + "board": 0.896, + "cell": 0.9564941251190854, + "nfe": 179.75 + }, + "verifier_b61_r4": { + "board": 0.9, + "cell": 0.9574644507956671, + "nfe": 212.25 + }, + "verifier_b61_r8": { + "board": 0.895, + "cell": 0.9557707914329064, + "nfe": 241.75 + }, + "verifier_narrow": { + "board": 0.897, + "cell": 0.9576938005010409, + "nfe": 83.75 + }, + "random_remask_control": { + "board": 0.899, + "cell": 0.9584700610423063, + "nfe": 174.25 + } + }, + "sudh-mdlm-s0": { + "fixedK_16": { + "board": 0.86, + "cell": 0.9477435517448219, + "nfe": 16.0 + }, + "fixedK_61": { + "board": 0.881, + "cell": 0.9500017642285029, + "nfe": 59.5 + }, + "entbudget": { + "board": 0.892, + "cell": 0.955506157157475, + "nfe": 26.75 + }, + "verifier_b32_r4": { + "board": 0.887, + "cell": 0.9510603013302283, + "nfe": 153.25 + }, + "verifier_b32_r8": { + "board": 0.899, + "cell": 0.9541477012102607, + "nfe": 184.0 + }, + "verifier_b61_r4": { + "board": 0.893, + "cell": 0.9520482692918387, + "nfe": 214.0 + }, + "verifier_b61_r8": { + "board": 0.886, + "cell": 0.9489432271267775, + "nfe": 240.0 + }, + "verifier_narrow": { + "board": 0.886, + "cell": 0.9512720087505734, + "nfe": 88.0 + }, + "random_remask_control": { + "board": 0.889, + "cell": 0.9519777001517237, + "nfe": 174.75 + } + }, + "sudh-mdlm-s1": { + "fixedK_16": { + "board": 0.807, + "cell": 0.9294132175999436, + "nfe": 16.0 + }, + "fixedK_61": { + "board": 0.873, + "cell": 0.9472848523340743, + "nfe": 59.5 + }, + "entbudget": { + "board": 0.874, + "cell": 0.9481669665855121, + "nfe": 23.5 + }, + "verifier_b32_r4": { + "board": 0.871, + "cell": 0.9441445255989556, + "nfe": 155.25 + }, + "verifier_b32_r8": { + "board": 0.873, + "cell": 0.9436505416181504, + "nfe": 189.5 + }, + "verifier_b61_r4": { + "board": 0.876, + "cell": 0.9451148512755372, + "nfe": 223.25 + }, + "verifier_b61_r8": { + "board": 0.88, + "cell": 0.9460146078120039, + "nfe": 243.75 + }, + "verifier_narrow": { + "board": 0.871, + "cell": 0.9458029003916587, + "nfe": 89.75 + }, + "random_remask_control": { + "board": 0.866, + "cell": 0.9417275325500158, + "nfe": 175.5 + } + } +} \ No newline at end of file diff --git a/figures/sudoku_rsweep.json b/figures/sudoku_rsweep.json new file mode 100644 index 0000000000000000000000000000000000000000..b51ff30fc6c5b168641db1a7812191239f395ae1 --- /dev/null +++ b/figures/sudoku_rsweep.json @@ -0,0 +1,42 @@ +{ + "rec1-full-s0": { + "1": 0.001, + "2": 0.054, + "4": 0.484, + "8": 0.82, + "16": 0.924, + "32": 0.983, + "64": 0.994, + "128": 0.991 + }, + "rec1-full-s1": { + "1": 0.006, + "2": 0.061, + "4": 0.482, + "8": 0.815, + "16": 0.924, + "32": 0.982, + "64": 0.992, + "128": 0.397 + }, + "rec1-mdlm-s0": { + "1": 0.002, + "2": 0.053, + "4": 0.407, + "8": 0.799, + "16": 0.896, + "32": 0.933, + "64": 0.95, + "128": 0.949 + }, + "rec1-mdlm-s1": { + "1": 0.0, + "2": 0.024, + "4": 0.366, + "8": 0.769, + "16": 0.889, + "32": 0.916, + "64": 0.938, + "128": 0.935 + } +} \ No newline at end of file diff --git a/figures/sudoku_validate.json b/figures/sudoku_validate.json new file mode 100644 index 0000000000000000000000000000000000000000..7007eddb8c5231096f6d67481f12868f49867a55 --- /dev/null +++ b/figures/sudoku_validate.json @@ -0,0 +1,68 @@ +{ + "classical_propagation": 1.0, + "full/aug": { + "mean": 1.0, + "per_seed": [ + 1.0, + 1.0 + ], + "ci": [ + 0.9961731014136095, + 1.0 + ], + "p_vs_satnet": 3.576990874029317e-08 + }, + "full/plain": { + "mean": 1.0, + "per_seed": [ + 1.0, + 1.0 + ], + "ci": [ + 0.9961731014136095, + 1.0 + ], + "p_vs_satnet": 3.576990874029317e-08 + }, + "mdlm/aug": { + "mean": 1.0, + "per_seed": [ + 1.0, + 1.0 + ], + "ci": [ + 0.9961731014136095, + 1.0 + ], + "p_vs_satnet": 3.576990874029317e-08 + }, + "mdlm/plain": { + "mean": 1.0, + "per_seed": [ + 1.0, + 1.0 + ], + "ci": [ + 0.9961731014136095, + 1.0 + ], + "p_vs_satnet": 3.576990874029317e-08 + }, + "harder": { + "30": { + "n": 120, + "clues": 30.0, + "classical_propagation": 0.275 + }, + "26": { + "n": 120, + "clues": 26.033333333333335, + "classical_propagation": 0.041666666666666664 + }, + "22": { + "n": 120, + "clues": 23.583333333333332, + "classical_propagation": 0.03333333333333333 + } + } +} \ No newline at end of file diff --git a/results/abl-deep32L-s0.json b/results/abl-deep32L-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..f6f634ec6dfb207cec2302f5696d558f4aa48e8c --- /dev/null +++ b/results/abl-deep32L-s0.json @@ -0,0 +1,141 @@ +{ + "args": { + "mode": "full", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": false, + "R": 32, + "no_deep_sup": false, + "R_infer": 0, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 32, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/abl-deep32L-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.702, + "cell": 0.9354115945097209, + "valid_sudoku": 0.702, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.832, + "cell": 0.9493666419674677, + "valid_sudoku": 0.832, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.865, + "cell": 0.9574115239405808, + "valid_sudoku": 0.865, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.906, + "cell": 0.9639391694012208, + "valid_sudoku": 0.906, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.911, + "cell": 0.9625807134540065, + "valid_sudoku": 0.911, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.918, + "cell": 0.9660386013196429, + "valid_sudoku": 0.918, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.928, + "cell": 0.9694082777601355, + "valid_sudoku": 0.928, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.945, + "cell": 0.9751596626795103, + "valid_sudoku": 0.945, + "clues_kept": 1.0, + "nfe": 45.25 + }, + "entbudget_0.05": { + "board": 0.944, + "cell": 0.9752478741046541, + "valid_sudoku": 0.944, + "clues_kept": 1.0, + "nfe": 33.25 + }, + "entbudget_0.2": { + "board": 0.933, + "cell": 0.9735189301718359, + "valid_sudoku": 0.933, + "clues_kept": 1.0, + "nfe": 24.75 + }, + "entbudget_0.5": { + "board": 0.922, + "cell": 0.9706961645672347, + "valid_sudoku": 0.922, + "clues_kept": 1.0, + "nfe": 16.25 + }, + "entbudget_1.0": { + "board": 0.873, + "cell": 0.9671147807063971, + "valid_sudoku": 0.873, + "clues_kept": 1.0, + "nfe": 10.25 + }, + "entbudget_2.0": { + "board": 0.823, + "cell": 0.9554532303023887, + "valid_sudoku": 0.823, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "conf_0.9": { + "board": 0.837, + "cell": 0.9656328287639815, + "valid_sudoku": 0.837, + "clues_kept": 1.0, + "nfe": 14.75 + }, + "conf_0.99": { + "board": 0.931, + "cell": 0.9755477929501429, + "valid_sudoku": 0.931, + "clues_kept": 1.0, + "nfe": 27.5 + }, + "conf_0.999": { + "board": 0.942, + "cell": 0.9744186867083025, + "valid_sudoku": 0.942, + "clues_kept": 1.0, + "nfe": 41.0 + } + }, + "best_board": 0.945 +} \ No newline at end of file diff --git a/results/abl-nodeepsup-s0.json b/results/abl-nodeepsup-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..46ccc0515313683475521b1b2e18ca16a1b97da5 --- /dev/null +++ b/results/abl-nodeepsup-s0.json @@ -0,0 +1,141 @@ +{ + "args": { + "mode": "full", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "no_deep_sup": true, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/abl-nodeepsup-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.932, + "cell": 0.9677146183973748, + "valid_sudoku": 0.932, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.935, + "cell": 0.9694964891852793, + "valid_sudoku": 0.935, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.94, + "cell": 0.9735012878868071, + "valid_sudoku": 0.94, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.952, + "cell": 0.9796408030768146, + "valid_sudoku": 0.952, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.957, + "cell": 0.9813344624395752, + "valid_sudoku": 0.957, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.964, + "cell": 0.9835926749232561, + "valid_sudoku": 0.964, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.97, + "cell": 0.985268692000988, + "valid_sudoku": 0.97, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.963, + "cell": 0.9857097491267068, + "valid_sudoku": 0.963, + "clues_kept": 1.0, + "nfe": 35.75 + }, + "entbudget_0.05": { + "board": 0.97, + "cell": 0.9853569034261318, + "valid_sudoku": 0.97, + "clues_kept": 1.0, + "nfe": 24.75 + }, + "entbudget_0.2": { + "board": 0.966, + "cell": 0.9850040577255567, + "valid_sudoku": 0.966, + "clues_kept": 1.0, + "nfe": 26.0 + }, + "entbudget_0.5": { + "board": 0.961, + "cell": 0.9856391799865919, + "valid_sudoku": 0.961, + "clues_kept": 1.0, + "nfe": 22.75 + }, + "entbudget_1.0": { + "board": 0.956, + "cell": 0.9811051127342013, + "valid_sudoku": 0.956, + "clues_kept": 1.0, + "nfe": 14.5 + }, + "entbudget_2.0": { + "board": 0.958, + "cell": 0.982798772096962, + "valid_sudoku": 0.958, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "conf_0.9": { + "board": 0.959, + "cell": 0.9802229984827635, + "valid_sudoku": 0.959, + "clues_kept": 1.0, + "nfe": 14.0 + }, + "conf_0.99": { + "board": 0.973, + "cell": 0.9887618644366818, + "valid_sudoku": 0.973, + "clues_kept": 1.0, + "nfe": 24.25 + }, + "conf_0.999": { + "board": 0.968, + "cell": 0.9864860096679722, + "valid_sudoku": 0.968, + "clues_kept": 1.0, + "nfe": 32.25 + } + }, + "best_board": 0.973 +} \ No newline at end of file diff --git a/results/abl-nodeepsup-s1.json b/results/abl-nodeepsup-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..cf7ffdc7572b7d01b88198764717a7cc0f64ef95 --- /dev/null +++ b/results/abl-nodeepsup-s1.json @@ -0,0 +1,141 @@ +{ + "args": { + "mode": "full", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "no_deep_sup": true, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 1, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/abl-nodeepsup-s1" + }, + "decode": { + "fixedK_1": { + "board": 0.935, + "cell": 0.9744892558484175, + "valid_sudoku": 0.935, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.94, + "cell": 0.9745951095585901, + "valid_sudoku": 0.94, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.956, + "cell": 0.9811756818743164, + "valid_sudoku": 0.956, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.968, + "cell": 0.9852157651459017, + "valid_sudoku": 0.968, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.972, + "cell": 0.9868917822236336, + "valid_sudoku": 0.972, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.975, + "cell": 0.9883031650259342, + "valid_sudoku": 0.975, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.976, + "cell": 0.9873328393493526, + "valid_sudoku": 0.976, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.982, + "cell": 0.9916904837514555, + "valid_sudoku": 0.982, + "clues_kept": 1.0, + "nfe": 31.25 + }, + "entbudget_0.05": { + "board": 0.98, + "cell": 0.9899968243886949, + "valid_sudoku": 0.98, + "clues_kept": 1.0, + "nfe": 22.5 + }, + "entbudget_0.2": { + "board": 0.979, + "cell": 0.9905966620796726, + "valid_sudoku": 0.979, + "clues_kept": 1.0, + "nfe": 19.5 + }, + "entbudget_0.5": { + "board": 0.975, + "cell": 0.9885148724462792, + "valid_sudoku": 0.975, + "clues_kept": 1.0, + "nfe": 12.5 + }, + "entbudget_1.0": { + "board": 0.961, + "cell": 0.9830634063723933, + "valid_sudoku": 0.961, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "entbudget_2.0": { + "board": 0.949, + "cell": 0.9780353551391976, + "valid_sudoku": 0.949, + "clues_kept": 1.0, + "nfe": 7.0 + }, + "conf_0.9": { + "board": 0.959, + "cell": 0.9837338132034861, + "valid_sudoku": 0.959, + "clues_kept": 1.0, + "nfe": 8.5 + }, + "conf_0.99": { + "board": 0.974, + "cell": 0.989044140997142, + "valid_sudoku": 0.974, + "clues_kept": 1.0, + "nfe": 20.5 + }, + "conf_0.999": { + "board": 0.979, + "cell": 0.9893969866977171, + "valid_sudoku": 0.979, + "clues_kept": 1.0, + "nfe": 30.0 + } + }, + "best_board": 0.982 +} \ No newline at end of file diff --git a/results/base-ar-s0.json b/results/base-ar-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..c16af852d855776eae7f3d366d31ad319604bc09 --- /dev/null +++ b/results/base-ar-s0.json @@ -0,0 +1,36 @@ +{ + "args": { + "mode": "mdlm", + "objective": "ar", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": false, + "R": 32, + "no_deep_sup": false, + "R_infer": 0, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 12, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/base-ar-s0" + }, + "decode": { + "ar": { + "board": 0.0, + "cell": 0.16823683003422604, + "valid_sudoku": 0.0, + "clues_kept": 1.0, + "nfe": 81.0 + } + } +} \ No newline at end of file diff --git a/results/base-directrec-s0.json b/results/base-directrec-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..06d71a9de66f40484b0570d5b81e2789b2cb0f68 --- /dev/null +++ b/results/base-directrec-s0.json @@ -0,0 +1,142 @@ +{ + "args": { + "mode": "mdlm", + "objective": "direct", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "no_deep_sup": false, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/base-directrec-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.358, + "cell": 0.8523870011643908, + "valid_sudoku": 0.358, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.077, + "cell": 0.7139127059736777, + "valid_sudoku": 0.077, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.0, + "cell": 0.5832715853357326, + "valid_sudoku": 0.0, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.0, + "cell": 0.574468085106383, + "valid_sudoku": 0.0, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.0, + "cell": 0.5897286616562577, + "valid_sudoku": 0.0, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.0, + "cell": 0.5926043541159451, + "valid_sudoku": 0.0, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.0, + "cell": 0.5926396386860026, + "valid_sudoku": 0.0, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.129, + "cell": 0.760117850463992, + "valid_sudoku": 0.129, + "clues_kept": 1.0, + "nfe": 57.0 + }, + "entbudget_0.05": { + "board": 0.318, + "cell": 0.8145795843477647, + "valid_sudoku": 0.318, + "clues_kept": 1.0, + "nfe": 51.25 + }, + "entbudget_0.2": { + "board": 0.381, + "cell": 0.8257471507709678, + "valid_sudoku": 0.381, + "clues_kept": 1.0, + "nfe": 41.75 + }, + "entbudget_0.5": { + "board": 0.39, + "cell": 0.8347623584206626, + "valid_sudoku": 0.39, + "clues_kept": 1.0, + "nfe": 32.5 + }, + "entbudget_1.0": { + "board": 0.381, + "cell": 0.8299812991778696, + "valid_sudoku": 0.381, + "clues_kept": 1.0, + "nfe": 26.5 + }, + "entbudget_2.0": { + "board": 0.38, + "cell": 0.8337391058889947, + "valid_sudoku": 0.38, + "clues_kept": 1.0, + "nfe": 20.0 + }, + "conf_0.9": { + "board": 0.401, + "cell": 0.8617550545146607, + "valid_sudoku": 0.401, + "clues_kept": 1.0, + "nfe": 25.75 + }, + "conf_0.99": { + "board": 0.409, + "cell": 0.8534278959810875, + "valid_sudoku": 0.409, + "clues_kept": 1.0, + "nfe": 44.25 + }, + "conf_0.999": { + "board": 0.329, + "cell": 0.8302282911682721, + "valid_sudoku": 0.329, + "clues_kept": 1.0, + "nfe": 54.0 + } + }, + "best_board": 0.409 +} \ No newline at end of file diff --git a/results/base-directrec-s1.json b/results/base-directrec-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..b0dbc2fd14a8b33435a813e3013b8048a4e687ab --- /dev/null +++ b/results/base-directrec-s1.json @@ -0,0 +1,142 @@ +{ + "args": { + "mode": "mdlm", + "objective": "direct", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "no_deep_sup": false, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 1, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/base-directrec-s1" + }, + "decode": { + "fixedK_1": { + "board": 0.015, + "cell": 0.7376415793373557, + "valid_sudoku": 0.015, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.009, + "cell": 0.7314314950072334, + "valid_sudoku": 0.009, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.003, + "cell": 0.7721146042835468, + "valid_sudoku": 0.003, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.005, + "cell": 0.7732437105253872, + "valid_sudoku": 0.005, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.0, + "cell": 0.7650753325570728, + "valid_sudoku": 0.0, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.002, + "cell": 0.7653223245474754, + "valid_sudoku": 0.002, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.003, + "cell": 0.7670689107653224, + "valid_sudoku": 0.003, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.008, + "cell": 0.7843759923785328, + "valid_sudoku": 0.008, + "clues_kept": 1.0, + "nfe": 53.75 + }, + "entbudget_0.05": { + "board": 0.037, + "cell": 0.7805652588123214, + "valid_sudoku": 0.037, + "clues_kept": 1.0, + "nfe": 46.0 + }, + "entbudget_0.2": { + "board": 0.06, + "cell": 0.7796831445608835, + "valid_sudoku": 0.06, + "clues_kept": 1.0, + "nfe": 39.0 + }, + "entbudget_0.5": { + "board": 0.063, + "cell": 0.7810592427931266, + "valid_sudoku": 0.063, + "clues_kept": 1.0, + "nfe": 29.5 + }, + "entbudget_1.0": { + "board": 0.064, + "cell": 0.7823647718852546, + "valid_sudoku": 0.064, + "clues_kept": 1.0, + "nfe": 22.75 + }, + "entbudget_2.0": { + "board": 0.057, + "cell": 0.7639462263152323, + "valid_sudoku": 0.057, + "clues_kept": 1.0, + "nfe": 15.0 + }, + "conf_0.9": { + "board": 0.061, + "cell": 0.7680039518718464, + "valid_sudoku": 0.061, + "clues_kept": 1.0, + "nfe": 24.5 + }, + "conf_0.99": { + "board": 0.066, + "cell": 0.7839702198228714, + "valid_sudoku": 0.066, + "clues_kept": 1.0, + "nfe": 40.0 + }, + "conf_0.999": { + "board": 0.057, + "cell": 0.7903920115733389, + "valid_sudoku": 0.057, + "clues_kept": 1.0, + "nfe": 49.25 + } + }, + "best_board": 0.066 +} \ No newline at end of file diff --git a/results/fix-arprefix-s0.json b/results/fix-arprefix-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..0f5e20b88ec8e5cf4daf918b1a6ab945f424c03f --- /dev/null +++ b/results/fix-arprefix-s0.json @@ -0,0 +1,37 @@ +{ + "args": { + "mode": "mdlm", + "all_cells": false, + "objective": "ar", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": false, + "R": 32, + "no_deep_sup": false, + "R_infer": 0, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 12, + "heads": 4, + "bs": 256, + "steps": 110000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/fix-arprefix-s0" + }, + "decode": { + "ar": { + "board": 0.003, + "cell": 0.430683462122014, + "valid_sudoku": 0.043, + "clues_kept": 0.208, + "nfe": 81.0 + } + } +} \ No newline at end of file diff --git a/results/fix-directall-s0.json b/results/fix-directall-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..51a948a662336c93a58e9d93a276a844e1c87069 --- /dev/null +++ b/results/fix-directall-s0.json @@ -0,0 +1,143 @@ +{ + "args": { + "mode": "mdlm", + "all_cells": true, + "objective": "direct", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "no_deep_sup": false, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 110000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/fix-directall-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.105, + "cell": 0.6825270809075191, + "valid_sudoku": 0.105, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.154, + "cell": 0.7321371864083837, + "valid_sudoku": 0.154, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.147, + "cell": 0.7680568787269327, + "valid_sudoku": 0.147, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.259, + "cell": 0.8128506404149466, + "valid_sudoku": 0.259, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.318, + "cell": 0.8437775660703575, + "valid_sudoku": 0.318, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.398, + "cell": 0.8605553791327053, + "valid_sudoku": 0.398, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.452, + "cell": 0.881620267457041, + "valid_sudoku": 0.452, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.501, + "cell": 0.893017183585618, + "valid_sudoku": 0.501, + "clues_kept": 1.0, + "nfe": 58.5 + }, + "entbudget_0.05": { + "board": 0.525, + "cell": 0.9052609293955753, + "valid_sudoku": 0.525, + "clues_kept": 1.0, + "nfe": 57.75 + }, + "entbudget_0.2": { + "board": 0.549, + "cell": 0.9146819096009315, + "valid_sudoku": 0.549, + "clues_kept": 1.0, + "nfe": 56.5 + }, + "entbudget_0.5": { + "board": 0.493, + "cell": 0.9018383260999965, + "valid_sudoku": 0.493, + "clues_kept": 1.0, + "nfe": 55.0 + }, + "entbudget_1.0": { + "board": 0.432, + "cell": 0.905578490526093, + "valid_sudoku": 0.432, + "clues_kept": 1.0, + "nfe": 49.0 + }, + "entbudget_2.0": { + "board": 0.322, + "cell": 0.8803676652199993, + "valid_sudoku": 0.322, + "clues_kept": 1.0, + "nfe": 32.75 + }, + "conf_0.9": { + "board": 0.33, + "cell": 0.9106241840443174, + "valid_sudoku": 0.33, + "clues_kept": 1.0, + "nfe": 54.75 + }, + "conf_0.99": { + "board": 0.551, + "cell": 0.9083306869905791, + "valid_sudoku": 0.551, + "clues_kept": 1.0, + "nfe": 58.0 + }, + "conf_0.999": { + "board": 0.532, + "cell": 0.906954588758336, + "valid_sudoku": 0.532, + "clues_kept": 1.0, + "nfe": 58.75 + } + }, + "best_board": 0.551 +} \ No newline at end of file diff --git a/results/fix-directall-s1.json b/results/fix-directall-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..4d1f3c311965bec5ecae254e9421074582aaad34 --- /dev/null +++ b/results/fix-directall-s1.json @@ -0,0 +1,143 @@ +{ + "args": { + "mode": "mdlm", + "all_cells": true, + "objective": "direct", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "no_deep_sup": false, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 110000, + "lr": 0.0003, + "warmup": 3000, + "seed": 1, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/fix-directall-s1" + }, + "decode": { + "fixedK_1": { + "board": 0.094, + "cell": 0.5431353868953107, + "valid_sudoku": 0.094, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.084, + "cell": 0.5583606788751279, + "valid_sudoku": 0.084, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.03, + "cell": 0.5073568328569916, + "valid_sudoku": 0.03, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.007, + "cell": 0.5128965103560214, + "valid_sudoku": 0.007, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.01, + "cell": 0.5899050845065453, + "valid_sudoku": 0.01, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.007, + "cell": 0.6276948590381426, + "valid_sudoku": 0.007, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.014, + "cell": 0.6588158498288699, + "valid_sudoku": 0.014, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.088, + "cell": 0.715923926466956, + "valid_sudoku": 0.088, + "clues_kept": 1.0, + "nfe": 58.25 + }, + "entbudget_0.05": { + "board": 0.138, + "cell": 0.7457393881655552, + "valid_sudoku": 0.138, + "clues_kept": 1.0, + "nfe": 57.75 + }, + "entbudget_0.2": { + "board": 0.187, + "cell": 0.7710913517518789, + "valid_sudoku": 0.187, + "clues_kept": 1.0, + "nfe": 58.0 + }, + "entbudget_0.5": { + "board": 0.212, + "cell": 0.7857344483257471, + "valid_sudoku": 0.212, + "clues_kept": 1.0, + "nfe": 57.0 + }, + "entbudget_1.0": { + "board": 0.214, + "cell": 0.788504287075262, + "valid_sudoku": 0.214, + "clues_kept": 1.0, + "nfe": 57.0 + }, + "entbudget_2.0": { + "board": 0.189, + "cell": 0.8019829928372323, + "valid_sudoku": 0.189, + "clues_kept": 1.0, + "nfe": 57.5 + }, + "conf_0.9": { + "board": 0.194, + "cell": 0.7980840478458769, + "valid_sudoku": 0.194, + "clues_kept": 1.0, + "nfe": 57.75 + }, + "conf_0.99": { + "board": 0.213, + "cell": 0.7785893228891005, + "valid_sudoku": 0.213, + "clues_kept": 1.0, + "nfe": 58.0 + }, + "conf_0.999": { + "board": 0.149, + "cell": 0.7357009279841925, + "valid_sudoku": 0.149, + "clues_kept": 1.0, + "nfe": 58.25 + } + }, + "best_board": 0.214 +} \ No newline at end of file diff --git a/results/rec1-full-s0.json b/results/rec1-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..88fbd37e7b9d5baad53b8588be950ff135df5c95 --- /dev/null +++ b/results/rec1-full-s0.json @@ -0,0 +1,140 @@ +{ + "args": { + "mode": "full", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/rec1-full-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.974, + "cell": 0.9903849546593275, + "valid_sudoku": 0.974, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.975, + "cell": 0.9906319466497301, + "valid_sudoku": 0.975, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.981, + "cell": 0.9929254437034685, + "valid_sudoku": 0.981, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.985, + "cell": 0.9943721110758266, + "valid_sudoku": 0.985, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.992, + "cell": 0.9960481281535585, + "valid_sudoku": 0.992, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.993, + "cell": 0.9976182915211178, + "valid_sudoku": 0.993, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.995, + "cell": 0.9980946332168943, + "valid_sudoku": 0.995, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.996, + "cell": 0.9975653646660315, + "valid_sudoku": 0.996, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "entbudget_0.05": { + "board": 0.993, + "cell": 0.9970360961151689, + "valid_sudoku": 0.993, + "clues_kept": 1.0, + "nfe": 20.5 + }, + "entbudget_0.2": { + "board": 0.992, + "cell": 0.9954835750326382, + "valid_sudoku": 0.992, + "clues_kept": 1.0, + "nfe": 13.75 + }, + "entbudget_0.5": { + "board": 0.992, + "cell": 0.9982886983522106, + "valid_sudoku": 0.992, + "clues_kept": 1.0, + "nfe": 12.0 + }, + "entbudget_1.0": { + "board": 0.986, + "cell": 0.9937722733848489, + "valid_sudoku": 0.986, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "entbudget_2.0": { + "board": 0.98, + "cell": 0.9918316220316855, + "valid_sudoku": 0.98, + "clues_kept": 1.0, + "nfe": 4.75 + }, + "conf_0.9": { + "board": 0.982, + "cell": 0.9933135739741011, + "valid_sudoku": 0.982, + "clues_kept": 1.0, + "nfe": 12.25 + }, + "conf_0.99": { + "board": 0.992, + "cell": 0.9959246321583571, + "valid_sudoku": 0.992, + "clues_kept": 1.0, + "nfe": 13.75 + }, + "conf_0.999": { + "board": 0.993, + "cell": 0.9963304047140186, + "valid_sudoku": 0.993, + "clues_kept": 1.0, + "nfe": 21.75 + } + }, + "best_board": 0.996 +} \ No newline at end of file diff --git a/results/rec1-full-s1.json b/results/rec1-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..cee801319fea11df231147b7bae9541e31bc7ab1 --- /dev/null +++ b/results/rec1-full-s1.json @@ -0,0 +1,140 @@ +{ + "args": { + "mode": "full", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 1, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/rec1-full-s1" + }, + "decode": { + "fixedK_1": { + "board": 0.973, + "cell": 0.9908612963551039, + "valid_sudoku": 0.973, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.973, + "cell": 0.9926078825729509, + "valid_sudoku": 0.973, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.986, + "cell": 0.9944073956458841, + "valid_sudoku": 0.986, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.992, + "cell": 0.9958011361631558, + "valid_sudoku": 0.992, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.992, + "cell": 0.9961186972936735, + "valid_sudoku": 0.992, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.998, + "cell": 0.9995060160191948, + "valid_sudoku": 0.998, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.997, + "cell": 0.9989591051833033, + "valid_sudoku": 0.997, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.996, + "cell": 0.9984298366324407, + "valid_sudoku": 0.996, + "clues_kept": 1.0, + "nfe": 14.75 + }, + "entbudget_0.05": { + "board": 0.994, + "cell": 0.9970184538301401, + "valid_sudoku": 0.994, + "clues_kept": 1.0, + "nfe": 16.75 + }, + "entbudget_0.2": { + "board": 0.993, + "cell": 0.9975830069510603, + "valid_sudoku": 0.993, + "clues_kept": 1.0, + "nfe": 10.75 + }, + "entbudget_0.5": { + "board": 0.993, + "cell": 0.9971948766804276, + "valid_sudoku": 0.993, + "clues_kept": 1.0, + "nfe": 7.0 + }, + "entbudget_1.0": { + "board": 0.991, + "cell": 0.9968243886948237, + "valid_sudoku": 0.991, + "clues_kept": 1.0, + "nfe": 6.0 + }, + "entbudget_2.0": { + "board": 0.981, + "cell": 0.9932606471190149, + "valid_sudoku": 0.981, + "clues_kept": 1.0, + "nfe": 4.25 + }, + "conf_0.9": { + "board": 0.988, + "cell": 0.9962774778589323, + "valid_sudoku": 0.988, + "clues_kept": 1.0, + "nfe": 6.5 + }, + "conf_0.99": { + "board": 0.994, + "cell": 0.9969831692600826, + "valid_sudoku": 0.994, + "clues_kept": 1.0, + "nfe": 11.75 + }, + "conf_0.999": { + "board": 0.995, + "cell": 0.9978476412264916, + "valid_sudoku": 0.995, + "clues_kept": 1.0, + "nfe": 18.0 + } + }, + "best_board": 0.998 +} \ No newline at end of file diff --git a/results/rec1-mdlm-s0.json b/results/rec1-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..70b21ee6faa879b0c817c43cbc050c9e8da72a8c --- /dev/null +++ b/results/rec1-mdlm-s0.json @@ -0,0 +1,140 @@ +{ + "args": { + "mode": "mdlm", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 0, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/rec1-mdlm-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.912, + "cell": 0.9668148618609083, + "valid_sudoku": 0.912, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.925, + "cell": 0.971842913094104, + "valid_sudoku": 0.925, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.94, + "cell": 0.973871775872411, + "valid_sudoku": 0.94, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.944, + "cell": 0.978370558554744, + "valid_sudoku": 0.944, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.949, + "cell": 0.9784058431248015, + "valid_sudoku": 0.949, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.955, + "cell": 0.9792526728061819, + "valid_sudoku": 0.955, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.959, + "cell": 0.9808934053138563, + "valid_sudoku": 0.959, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.971, + "cell": 0.9863448713877422, + "valid_sudoku": 0.971, + "clues_kept": 1.0, + "nfe": 36.5 + }, + "entbudget_0.05": { + "board": 0.958, + "cell": 0.9812109664443739, + "valid_sudoku": 0.958, + "clues_kept": 1.0, + "nfe": 31.0 + }, + "entbudget_0.2": { + "board": 0.962, + "cell": 0.9824988532514731, + "valid_sudoku": 0.962, + "clues_kept": 1.0, + "nfe": 23.75 + }, + "entbudget_0.5": { + "board": 0.962, + "cell": 0.982869341237077, + "valid_sudoku": 0.962, + "clues_kept": 1.0, + "nfe": 19.25 + }, + "entbudget_1.0": { + "board": 0.948, + "cell": 0.978370558554744, + "valid_sudoku": 0.948, + "clues_kept": 1.0, + "nfe": 13.25 + }, + "entbudget_2.0": { + "board": 0.942, + "cell": 0.9765357609117533, + "valid_sudoku": 0.942, + "clues_kept": 1.0, + "nfe": 7.5 + }, + "conf_0.9": { + "board": 0.946, + "cell": 0.9811403973042588, + "valid_sudoku": 0.946, + "clues_kept": 1.0, + "nfe": 19.5 + }, + "conf_0.99": { + "board": 0.96, + "cell": 0.9817755195652941, + "valid_sudoku": 0.96, + "clues_kept": 1.0, + "nfe": 28.25 + }, + "conf_0.999": { + "board": 0.962, + "cell": 0.9814756007198052, + "valid_sudoku": 0.962, + "clues_kept": 1.0, + "nfe": 37.5 + } + }, + "best_board": 0.971 +} \ No newline at end of file diff --git a/results/rec1-mdlm-s1.json b/results/rec1-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..1be1bc530ce7469ba6dc1edde4b0fab133429eb2 --- /dev/null +++ b/results/rec1-mdlm-s1.json @@ -0,0 +1,140 @@ +{ + "args": { + "mode": "mdlm", + "K": 8, + "lam": 1.0, + "pe": "ape", + "recurrent": true, + "R": 32, + "R_infer": 64, + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 128, + "layers": 1, + "heads": 4, + "bs": 256, + "steps": 120000, + "lr": 0.0003, + "warmup": 3000, + "seed": 1, + "eval_bs": 250, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/rec1-mdlm-s1" + }, + "decode": { + "fixedK_1": { + "board": 0.835, + "cell": 0.9497900568081578, + "valid_sudoku": 0.835, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.855, + "cell": 0.9570586782400057, + "valid_sudoku": 0.855, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.898, + "cell": 0.962333721463604, + "valid_sudoku": 0.898, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.923, + "cell": 0.9695141314703081, + "valid_sudoku": 0.923, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.94, + "cell": 0.9763593380614657, + "valid_sudoku": 0.94, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.944, + "cell": 0.9770826717476447, + "valid_sudoku": 0.944, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.947, + "cell": 0.9754595815249991, + "valid_sudoku": 0.947, + "clues_kept": 1.0, + "nfe": 59.5 + }, + "entbudget_0.01": { + "board": 0.947, + "cell": 0.9762005574962069, + "valid_sudoku": 0.947, + "clues_kept": 1.0, + "nfe": 52.25 + }, + "entbudget_0.05": { + "board": 0.96, + "cell": 0.9831869023675947, + "valid_sudoku": 0.96, + "clues_kept": 1.0, + "nfe": 40.5 + }, + "entbudget_0.2": { + "board": 0.946, + "cell": 0.9753360855297978, + "valid_sudoku": 0.946, + "clues_kept": 1.0, + "nfe": 32.25 + }, + "entbudget_0.5": { + "board": 0.944, + "cell": 0.9752655163896828, + "valid_sudoku": 0.944, + "clues_kept": 1.0, + "nfe": 23.0 + }, + "entbudget_1.0": { + "board": 0.925, + "cell": 0.9730425884760594, + "valid_sudoku": 0.925, + "clues_kept": 1.0, + "nfe": 12.0 + }, + "entbudget_2.0": { + "board": 0.889, + "cell": 0.9628453477294379, + "valid_sudoku": 0.889, + "clues_kept": 1.0, + "nfe": 8.5 + }, + "conf_0.9": { + "board": 0.884, + "cell": 0.9695141314703081, + "valid_sudoku": 0.884, + "clues_kept": 1.0, + "nfe": 15.25 + }, + "conf_0.99": { + "board": 0.946, + "cell": 0.976482834056667, + "valid_sudoku": 0.946, + "clues_kept": 1.0, + "nfe": 35.0 + }, + "conf_0.999": { + "board": 0.956, + "cell": 0.9794820225115557, + "valid_sudoku": 0.956, + "clues_kept": 1.0, + "nfe": 48.25 + } + }, + "best_board": 0.96 +} \ No newline at end of file diff --git a/results/sudh-full-s0.json b/results/sudh-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..cfa1241ed67bf19de564753bac6fce310aa3329d --- /dev/null +++ b/results/sudh-full-s0.json @@ -0,0 +1,137 @@ +{ + "args": { + "mode": "full", + "K": 8, + "lam": 1.0, + "pe": "ape", + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 512, + "layers": 12, + "heads": 8, + "bs": 256, + "steps": 150000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "eval_bs": 200, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/sudh-full-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.419, + "cell": 0.8971101937122896, + "valid_sudoku": 0.419, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.703, + "cell": 0.9253025651882432, + "valid_sudoku": 0.703, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.792, + "cell": 0.9413394022793832, + "valid_sudoku": 0.792, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.847, + "cell": 0.9466850146430966, + "valid_sudoku": 0.847, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.853, + "cell": 0.9441268833139268, + "valid_sudoku": 0.853, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.87, + "cell": 0.9467908683532691, + "valid_sudoku": 0.87, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.883, + "cell": 0.9494901379626689, + "valid_sudoku": 0.883, + "clues_kept": 1.0, + "nfe": 59.4 + }, + "entbudget_0.01": { + "board": 0.876, + "cell": 0.9501605447937617, + "valid_sudoku": 0.876, + "clues_kept": 1.0, + "nfe": 30.6 + }, + "entbudget_0.05": { + "board": 0.866, + "cell": 0.9512190818954871, + "valid_sudoku": 0.866, + "clues_kept": 1.0, + "nfe": 23.6 + }, + "entbudget_0.2": { + "board": 0.795, + "cell": 0.9393105395010761, + "valid_sudoku": 0.795, + "clues_kept": 1.0, + "nfe": 17.4 + }, + "entbudget_0.5": { + "board": 0.731, + "cell": 0.9327652517554074, + "valid_sudoku": 0.731, + "clues_kept": 1.0, + "nfe": 10.4 + }, + "entbudget_1.0": { + "board": 0.636, + "cell": 0.9199745951095586, + "valid_sudoku": 0.636, + "clues_kept": 1.0, + "nfe": 7.8 + }, + "entbudget_2.0": { + "board": 0.531, + "cell": 0.9122119896969055, + "valid_sudoku": 0.531, + "clues_kept": 1.0, + "nfe": 5.4 + }, + "conf_0.9": { + "board": 0.582, + "cell": 0.9171341872199287, + "valid_sudoku": 0.582, + "clues_kept": 1.0, + "nfe": 7.8 + }, + "conf_0.99": { + "board": 0.81, + "cell": 0.9423802970960798, + "valid_sudoku": 0.81, + "clues_kept": 1.0, + "nfe": 22.6 + }, + "conf_0.999": { + "board": 0.866, + "cell": 0.9504604636392505, + "valid_sudoku": 0.866, + "clues_kept": 1.0, + "nfe": 30.0 + } + }, + "best_board": 0.883 +} \ No newline at end of file diff --git a/results/sudh-full-s1.json b/results/sudh-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..818c5f1b4811a43ba6eca994e7b4a9f646d750d6 --- /dev/null +++ b/results/sudh-full-s1.json @@ -0,0 +1,137 @@ +{ + "args": { + "mode": "full", + "K": 8, + "lam": 1.0, + "pe": "ape", + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 512, + "layers": 12, + "heads": 8, + "bs": 256, + "steps": 150000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "eval_bs": 200, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/sudh-full-s1" + }, + "decode": { + "fixedK_1": { + "board": 0.383, + "cell": 0.8876362866518471, + "valid_sudoku": 0.383, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.682, + "cell": 0.9200804488197312, + "valid_sudoku": 0.682, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.801, + "cell": 0.9390635475106736, + "valid_sudoku": 0.801, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.851, + "cell": 0.9509720899050845, + "valid_sudoku": 0.851, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.87, + "cell": 0.9515719275960622, + "valid_sudoku": 0.87, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.886, + "cell": 0.9570233936699482, + "valid_sudoku": 0.886, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.884, + "cell": 0.9524716841325288, + "valid_sudoku": 0.884, + "clues_kept": 1.0, + "nfe": 59.4 + }, + "entbudget_0.01": { + "board": 0.895, + "cell": 0.9573409548004658, + "valid_sudoku": 0.895, + "clues_kept": 1.0, + "nfe": 30.4 + }, + "entbudget_0.05": { + "board": 0.851, + "cell": 0.9468437952083554, + "valid_sudoku": 0.851, + "clues_kept": 1.0, + "nfe": 20.2 + }, + "entbudget_0.2": { + "board": 0.788, + "cell": 0.9315479340884232, + "valid_sudoku": 0.788, + "clues_kept": 1.0, + "nfe": 13.6 + }, + "entbudget_0.5": { + "board": 0.697, + "cell": 0.919215976853322, + "valid_sudoku": 0.697, + "clues_kept": 1.0, + "nfe": 9.8 + }, + "entbudget_1.0": { + "board": 0.604, + "cell": 0.9126001199675382, + "valid_sudoku": 0.604, + "clues_kept": 1.0, + "nfe": 6.4 + }, + "entbudget_2.0": { + "board": 0.482, + "cell": 0.9062136127871282, + "valid_sudoku": 0.482, + "clues_kept": 1.0, + "nfe": 5.0 + }, + "conf_0.9": { + "board": 0.538, + "cell": 0.9054549945308916, + "valid_sudoku": 0.538, + "clues_kept": 1.0, + "nfe": 7.8 + }, + "conf_0.99": { + "board": 0.777, + "cell": 0.9316890723686532, + "valid_sudoku": 0.777, + "clues_kept": 1.0, + "nfe": 17.0 + }, + "conf_0.999": { + "board": 0.884, + "cell": 0.9553473765922162, + "valid_sudoku": 0.884, + "clues_kept": 1.0, + "nfe": 26.2 + } + }, + "best_board": 0.895 +} \ No newline at end of file diff --git a/results/sudh-mdlm-s0.json b/results/sudh-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..ed82eb043f58136f6ee00d3642b5270784cabfa8 --- /dev/null +++ b/results/sudh-mdlm-s0.json @@ -0,0 +1,137 @@ +{ + "args": { + "mode": "mdlm", + "K": 8, + "lam": 1.0, + "pe": "ape", + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 512, + "layers": 12, + "heads": 8, + "bs": 256, + "steps": 150000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "eval_bs": 200, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/sudh-mdlm-s0" + }, + "decode": { + "fixedK_1": { + "board": 0.05, + "cell": 0.7941851028545217, + "valid_sudoku": 0.05, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.511, + "cell": 0.8766980699340179, + "valid_sudoku": 0.511, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.71, + "cell": 0.9216682544723193, + "valid_sudoku": 0.71, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.807, + "cell": 0.9365407007515614, + "valid_sudoku": 0.807, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.856, + "cell": 0.9451501358455947, + "valid_sudoku": 0.856, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.872, + "cell": 0.947020218058643, + "valid_sudoku": 0.872, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.879, + "cell": 0.948537454571116, + "valid_sudoku": 0.879, + "clues_kept": 1.0, + "nfe": 59.4 + }, + "entbudget_0.01": { + "board": 0.89, + "cell": 0.9501252602237041, + "valid_sudoku": 0.89, + "clues_kept": 1.0, + "nfe": 24.6 + }, + "entbudget_0.05": { + "board": 0.872, + "cell": 0.9483786740058573, + "valid_sudoku": 0.872, + "clues_kept": 1.0, + "nfe": 22.2 + }, + "entbudget_0.2": { + "board": 0.856, + "cell": 0.9515542853110335, + "valid_sudoku": 0.856, + "clues_kept": 1.0, + "nfe": 16.2 + }, + "entbudget_0.5": { + "board": 0.766, + "cell": 0.9330298860308387, + "valid_sudoku": 0.766, + "clues_kept": 1.0, + "nfe": 11.2 + }, + "entbudget_1.0": { + "board": 0.647, + "cell": 0.9091951589569881, + "valid_sudoku": 0.647, + "clues_kept": 1.0, + "nfe": 7.8 + }, + "entbudget_2.0": { + "board": 0.44, + "cell": 0.8767333545040754, + "valid_sudoku": 0.44, + "clues_kept": 1.0, + "nfe": 5.0 + }, + "conf_0.9": { + "board": 0.413, + "cell": 0.8837726262305494, + "valid_sudoku": 0.413, + "clues_kept": 1.0, + "nfe": 9.6 + }, + "conf_0.99": { + "board": 0.847, + "cell": 0.9466144455029816, + "valid_sudoku": 0.847, + "clues_kept": 1.0, + "nfe": 20.0 + }, + "conf_0.999": { + "board": 0.89, + "cell": 0.9560354257083378, + "valid_sudoku": 0.89, + "clues_kept": 1.0, + "nfe": 24.6 + } + }, + "best_board": 0.89 +} \ No newline at end of file diff --git a/results/sudh-mdlm-s1.json b/results/sudh-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..def9a2d5974e2d396dff134cab25449a4bd41f11 --- /dev/null +++ b/results/sudh-mdlm-s1.json @@ -0,0 +1,137 @@ +{ + "args": { + "mode": "mdlm", + "K": 8, + "lam": 1.0, + "pe": "ape", + "root": "data/sudoku", + "hard": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sudoku_hard", + "augment": true, + "d": 512, + "layers": 12, + "heads": 8, + "bs": 256, + "steps": 150000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "eval_bs": 200, + "ckpt_every": 10000, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/sudh-mdlm-s1" + }, + "decode": { + "fixedK_1": { + "board": 0.038, + "cell": 0.7791185914399633, + "valid_sudoku": 0.038, + "clues_kept": 1.0, + "nfe": 1.0 + }, + "fixedK_2": { + "board": 0.455, + "cell": 0.8588970043400022, + "valid_sudoku": 0.455, + "clues_kept": 1.0, + "nfe": 2.0 + }, + "fixedK_4": { + "board": 0.696, + "cell": 0.9148406901661903, + "valid_sudoku": 0.696, + "clues_kept": 1.0, + "nfe": 4.0 + }, + "fixedK_8": { + "board": 0.779, + "cell": 0.9273314279665502, + "valid_sudoku": 0.779, + "clues_kept": 1.0, + "nfe": 8.0 + }, + "fixedK_16": { + "board": 0.822, + "cell": 0.9361878550509862, + "valid_sudoku": 0.822, + "clues_kept": 1.0, + "nfe": 16.0 + }, + "fixedK_32": { + "board": 0.858, + "cell": 0.9440210296037542, + "valid_sudoku": 0.858, + "clues_kept": 1.0, + "nfe": 32.0 + }, + "fixedK_61": { + "board": 0.873, + "cell": 0.9453794855509685, + "valid_sudoku": 0.873, + "clues_kept": 1.0, + "nfe": 59.4 + }, + "entbudget_0.01": { + "board": 0.869, + "cell": 0.9446737941498183, + "valid_sudoku": 0.869, + "clues_kept": 1.0, + "nfe": 26.6 + }, + "entbudget_0.05": { + "board": 0.865, + "cell": 0.9473554214741894, + "valid_sudoku": 0.865, + "clues_kept": 1.0, + "nfe": 21.2 + }, + "entbudget_0.2": { + "board": 0.817, + "cell": 0.9398750926219964, + "valid_sudoku": 0.817, + "clues_kept": 1.0, + "nfe": 17.8 + }, + "entbudget_0.5": { + "board": 0.752, + "cell": 0.9303129741364101, + "valid_sudoku": 0.752, + "clues_kept": 1.0, + "nfe": 9.6 + }, + "entbudget_1.0": { + "board": 0.602, + "cell": 0.8949578349387812, + "valid_sudoku": 0.602, + "clues_kept": 1.0, + "nfe": 7.2 + }, + "entbudget_2.0": { + "board": 0.425, + "cell": 0.8687413993860484, + "valid_sudoku": 0.425, + "clues_kept": 1.0, + "nfe": 5.0 + }, + "conf_0.9": { + "board": 0.402, + "cell": 0.8780918104512897, + "valid_sudoku": 0.402, + "clues_kept": 1.0, + "nfe": 11.0 + }, + "conf_0.99": { + "board": 0.815, + "cell": 0.9398045234818814, + "valid_sudoku": 0.815, + "clues_kept": 1.0, + "nfe": 21.6 + }, + "conf_0.999": { + "board": 0.861, + "cell": 0.9431389153523164, + "valid_sudoku": 0.861, + "clues_kept": 1.0, + "nfe": 26.4 + } + }, + "best_board": 0.873 +} \ No newline at end of file diff --git a/slurm/abl.sh b/slurm/abl.sh new file mode 100644 index 0000000000000000000000000000000000000000..9d8a8d6cf35a2901603de4d1db9b24eadb8b4311 --- /dev/null +++ b/slurm/abl.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Ablations that test the mechanism rather than decorate the result. +# +# 0,1 recurrent WITHOUT deep supervision -> is per-step supervision the +# ingredient, or is recurrence alone enough? +# 2,3 32 DISTINCT layers, d=128, no weight sharing -> is it the depth, or the +# weight sharing? Same depth, ~32x the parameters. +# 4,5 R=8 instead of 32 -> how much recurrence does the task actually need? +# +# All use the schedule-matched objective, which is the arm that reached 99.70%. +#SBATCH -J abl +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 5:00:00 +#SBATCH -o logs/abl-%A_%a.out +#SBATCH --array=0-5%6 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +I=$SLURM_ARRAY_TASK_ID +S=$((I % 2)) +COMMON="--mode full --seed $S --pe ape --hard $SCR/pcmd/data/sudoku_hard --augment \ + --heads 4 --bs 256 --steps 120000 --lr 3e-4 --warmup 3000 --K 8 --lam 1.0 \ + --eval_bs 250 --ckpt_every 10000" +case $((I / 2)) in + 0) NAME="abl-nodeepsup-s$S" + ARGS="--recurrent --R 32 --R_infer 64 --no_deep_sup --d 128 --layers 1" ;; + 1) NAME="abl-deep32L-s$S" + ARGS="--d 128 --layers 32" ;; + 2) NAME="abl-R8-s$S" + ARGS="--recurrent --R 8 --R_infer 16 --d 128 --layers 1" ;; +esac +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py $ARGS $COMMON --out $SCR/pcmd/runs/$NAME diff --git a/slurm/b_gpu-a100_1g.5gb --gres=gpu:a100_1g.5gb:1.sh b/slurm/b_gpu-a100_1g.5gb --gres=gpu:a100_1g.5gb:1.sh new file mode 100644 index 0000000000000000000000000000000000000000..d3d2b2c3f9f902d4f70e7a21f97c7e033edb18dd --- /dev/null +++ b/slurm/b_gpu-a100_1g.5gb --gres=gpu:a100_1g.5gb:1.sh @@ -0,0 +1,14 @@ +#!/bin/bash +#SBATCH -J bench-gpu-a100_1g.5gb --gres=gpu:a100_1g.5gb:1 +#SBATCH -p gpu-a100_1g.5gb --gres=gpu:a100_1g.5gb:1 +#SBATCH +#SBATCH -c 4 +#SBATCH --mem=16g +#SBATCH -t 0:20:00 +#SBATCH -o logs/bench-gpu-a100_1g.5gb --gres=gpu:a100_1g.5gb:1-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u bench.py diff --git a/slurm/b_gpu-h100 --gpus=h100:1.sh b/slurm/b_gpu-h100 --gpus=h100:1.sh new file mode 100644 index 0000000000000000000000000000000000000000..b1dee84643776135ac3df4abf602d2b2607be6f3 --- /dev/null +++ b/slurm/b_gpu-h100 --gpus=h100:1.sh @@ -0,0 +1,14 @@ +#!/bin/bash +#SBATCH -J bench-gpu-h100 --gpus=h100:1 +#SBATCH -p gpu-h100 --gpus=h100:1 +#SBATCH +#SBATCH -c 4 +#SBATCH --mem=16g +#SBATCH -t 0:20:00 +#SBATCH -o logs/bench-gpu-h100 --gpus=h100:1-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u bench.py diff --git a/slurm/base.sh b/slurm/base.sh new file mode 100644 index 0000000000000000000000000000000000000000..f423402d50f850702c2a3bdb8c44812c68856d6b --- /dev/null +++ b/slurm/base.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# The baselines that can overturn the paper's framing. +# +# 0,1 DIRECT + RECURRENT: predict every blank from the puzzle, no masking +# schedule, recurrent with deep supervision. This is the Recurrent +# Transformer setup reimplemented in our codebase - the control that asks +# whether the DIFFUSION framing earns its place at all. If it matches +# 99.70%, diffusion contributes nothing and the paper's framing collapses. +# 2,3 DIRECT + FEED-FORWARD: same, without recurrence. Isolates recurrence +# from the objective. +# 4,5 AUTOREGRESSIVE: left-to-right cell filling at matched size. Asks whether +# any iterative model would do, making the generative framing irrelevant. +#SBATCH -J base +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 5:00:00 +#SBATCH -o logs/base-%A_%a.out +#SBATCH --array=0-5%6 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +I=$SLURM_ARRAY_TASK_ID +S=$((I % 2)) +COMMON="--seed $S --pe ape --hard $SCR/pcmd/data/sudoku_hard --augment \ + --heads 4 --bs 256 --steps 120000 --lr 3e-4 --warmup 3000 \ + --eval_bs 250 --ckpt_every 10000" +case $((I / 2)) in + 0) NAME="base-directrec-s$S" + ARGS="--objective direct --recurrent --R 32 --R_infer 64 --d 128 --layers 1" ;; + 1) NAME="base-directff-s$S" + ARGS="--objective direct --d 128 --layers 12" ;; + 2) NAME="base-ar-s$S" + ARGS="--objective ar --d 128 --layers 12" ;; +esac +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py $ARGS $COMMON --out $SCR/pcmd/runs/$NAME diff --git a/slurm/bh.sh b/slurm/bh.sh new file mode 100644 index 0000000000000000000000000000000000000000..49fb04af95ebcbd2315a98593e331f10d05beb7a --- /dev/null +++ b/slurm/bh.sh @@ -0,0 +1,14 @@ +#!/bin/bash +#SBATCH -J benchH +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 4 +#SBATCH --mem=16g +#SBATCH -t 0:20:00 +#SBATCH -o logs/benchH-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u bench.py diff --git a/slurm/bm.sh b/slurm/bm.sh new file mode 100644 index 0000000000000000000000000000000000000000..c542add35f8408031e6bd8af68de40ef5fbf6177 --- /dev/null +++ b/slurm/bm.sh @@ -0,0 +1,14 @@ +#!/bin/bash +#SBATCH -J benchM +#SBATCH -p gpu-a100_1g.5gb +#SBATCH --gres=gpu:a100_1g.5gb:1 +#SBATCH -c 4 +#SBATCH --mem=16g +#SBATCH -t 0:20:00 +#SBATCH -o logs/benchM-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u bench.py diff --git a/slurm/cmpall.sh b/slurm/cmpall.sh new file mode 100644 index 0000000000000000000000000000000000000000..b46b0aad727a7b219d28c4cfc3d07addc2e548f4 --- /dev/null +++ b/slurm/cmpall.sh @@ -0,0 +1,15 @@ +#!/bin/bash +#SBATCH -J cmpall +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 16 +#SBATCH --mem=48g +#SBATCH -t 2:00:00 +#SBATCH -o logs/cmpall-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u sudoku_compare.py --hard $SCR/pcmd/data/sudoku_hard \ + --runs $SCR/pcmd/runs --pattern 'rec1-*' --out $SCR/pcmd/figures/compare_recurrent.json diff --git a/slurm/fix.sh b/slurm/fix.sh new file mode 100644 index 0000000000000000000000000000000000000000..d46767c3b8c25ba140770a947a428dfa68807f14 --- /dev/null +++ b/slurm/fix.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Properly specified baselines, replacing two broken ones. +# +# 0 DIRECT + all-cell supervision: the published recurrent classifier trains +# on all 81 cells, not just the blanks. Our blanks-only version plateaued at +# loss 0.34 against diffusion's 0.15, which is the likeliest cause. +# 1 same, second seed - the first attempt had huge seed variance (40.9 / 6.6) +# 2 AR PREFIX-LM: [puzzle][solution], causal, loss on the solution half. The +# earlier in-place causal version was mis-specified: Sudoku clues are +# scattered, so a causal model could not attend to clues on its right. +#SBATCH -J fix +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 2:00:00 +#SBATCH -o logs/fix-%A_%a.out +#SBATCH --array=0-2%3 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +I=$SLURM_ARRAY_TASK_ID +COMMON="--pe ape --hard $SCR/pcmd/data/sudoku_hard --augment --heads 4 --bs 256 \ + --steps 110000 --lr 3e-4 --warmup 3000 --eval_bs 250 --ckpt_every 10000" +case $I in + 0) NAME="fix-directall-s0"; ARGS="--objective direct --all_cells --recurrent --R 32 --R_infer 64 --d 128 --layers 1 --seed 0" ;; + 1) NAME="fix-directall-s1"; ARGS="--objective direct --all_cells --recurrent --R 32 --R_infer 64 --d 128 --layers 1 --seed 1" ;; + 2) NAME="fix-arprefix-s0"; ARGS="--objective ar --d 128 --layers 12 --seed 0" ;; +esac +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py $ARGS $COMMON --out $SCR/pcmd/runs/$NAME diff --git a/slurm/genhard.sh b/slurm/genhard.sh new file mode 100644 index 0000000000000000000000000000000000000000..a33ee4b3313a69b24bd021d5d709bf1070958c1f --- /dev/null +++ b/slurm/genhard.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# CPU-only: generate the hard Sudoku benchmark. No GPU needed, so this costs +# almost nothing against the class GPU allocation. +#SBATCH -J genhard +#SBATCH -p standard +#SBATCH -c 64 +#SBATCH --mem=64g +#SBATCH -t 3:00:00 +#SBATCH -o logs/genhard-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u gen_hard_sudoku.py --n_train 100000 --n_test 1000 \ + --workers 64 --out $SCR/pcmd/data/sudoku_hard diff --git a/slurm/inj.sh b/slurm/inj.sh new file mode 100644 index 0000000000000000000000000000000000000000..49eb620e4ff6dc73608d56f4f70808d7336b3078 --- /dev/null +++ b/slurm/inj.sh @@ -0,0 +1,40 @@ +#!/bin/bash +# Closes the one acknowledged gap in the ablation. +# +# The headline comparison - recurrent 99.70% against 32 distinct layers 94.50% - +# conflates TWO differences: weight sharing, and re-supplying the input at every +# application. --no_inject holds architecture, depth, parameter count and +# objective fixed and changes only the injection, which isolates them. +# +# no-injection stays near 98% -> weight sharing is the mechanism +# no-injection falls toward 94% -> injection is, and our explanation is wrong +# +# Runs on a 5GB MIG slice: at 212k parameters and 81-token sequences the +# optimiser state is a few MB, and that partition bills at 9/min against the +# H100's 144 - 16x more compute from the same remaining allocation. +#SBATCH -J inj +#SBATCH -p gpu-a100_1g.5gb +#SBATCH --gres=gpu:a100_1g.5gb:1 +#SBATCH -c 4 +#SBATCH --mem=16g +#SBATCH -t 6:00:00 +#SBATCH -o logs/inj-%A_%a.out +#SBATCH --array=0-4%5 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +I=$SLURM_ARRAY_TASK_ID +COMMON="--mode full --pe ape --hard $SCR/pcmd/data/sudoku_hard --augment \ + --heads 4 --bs 256 --steps 110000 --lr 3e-4 --warmup 3000 --K 8 --lam 1.0 \ + --eval_bs 200 --ckpt_every 8000" +case $I in + 0) NAME="inj-noinject-s0"; ARGS="--recurrent --R 32 --R_infer 64 --no_inject --d 128 --layers 1 --seed 0" ;; + 1) NAME="inj-noinject-s1"; ARGS="--recurrent --R 32 --R_infer 64 --no_inject --d 128 --layers 1 --seed 1" ;; + 2) NAME="inj-deep32L-s1"; ARGS="--d 128 --layers 32 --seed 1" ;; + 3) NAME="inj-rec-s2"; ARGS="--recurrent --R 32 --R_infer 64 --d 128 --layers 1 --seed 2" ;; + 4) NAME="inj-rec-s3"; ARGS="--recurrent --R 32 --R_infer 64 --d 128 --layers 1 --seed 3" ;; +esac +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py $ARGS $COMMON --out $SCR/pcmd/runs/$NAME diff --git a/slurm/inj2.sh b/slurm/inj2.sh new file mode 100644 index 0000000000000000000000000000000000000000..ec4257c015a025b66609b18cd9dbaaded1be99ad --- /dev/null +++ b/slurm/inj2.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# Closes the one acknowledged gap in the ablation. +# +# The headline comparison - recurrent 99.70% against 32 distinct layers 94.50% - +# conflates TWO differences: weight sharing, and re-supplying the input at every +# application. --no_inject holds architecture, depth, parameter count and +# objective fixed and changes only the injection, which isolates them. +# +# no-injection stays near 98% -> weight sharing is the mechanism +# no-injection falls toward 94% -> injection is, and our explanation is wrong +# +# Runs on a 5GB MIG slice: at 212k parameters and 81-token sequences the +# optimiser state is a few MB, and that partition bills at 9/min against the +# H100's 144 - 16x more compute from the same remaining allocation. +#SBATCH -J inj +#SBATCH -p gpu-a100_1g.5gb +#SBATCH --gres=gpu:a100_1g.5gb:1 +#SBATCH -c 4 +#SBATCH --mem=16g +#SBATCH -t 6:00:00 +#SBATCH -o logs/inj-%A_%a.out +#SBATCH --array=0-2%3 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True +cd $SCR/pcmd/src +I=$SLURM_ARRAY_TASK_ID +COMMON="--mode full --pe ape --hard $SCR/pcmd/data/sudoku_hard --augment \ + --heads 4 --bs 64 --steps 110000 --lr 3e-4 --warmup 3000 --K 8 --lam 1.0 \ + --eval_bs 200 --ckpt_every 8000" +case $I in + 0) NAME="inj-noinject-s0"; ARGS="--recurrent --R 32 --R_infer 64 --no_inject --d 128 --layers 1 --seed 0" ;; + 1) NAME="inj-noinject-s1"; ARGS="--recurrent --R 32 --R_infer 64 --no_inject --d 128 --layers 1 --seed 1" ;; + 2) NAME="inj-deep32L-s1"; ARGS="--d 128 --layers 32 --seed 1" ;; + 3) NAME="inj-rec-s2"; ARGS="--recurrent --R 32 --R_infer 64 --d 128 --layers 1 --seed 2" ;; + 4) NAME="inj-rec-s3"; ARGS="--recurrent --R 32 --R_infer 64 --d 128 --layers 1 --seed 3" ;; +esac +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py $ARGS $COMMON --out $SCR/pcmd/runs/$NAME diff --git a/slurm/mol1.sh b/slurm/mol1.sh new file mode 100644 index 0000000000000000000000000000000000000000..6c123342c6faec9661483b4ae7c49e5621efe688 --- /dev/null +++ b/slurm/mol1.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# QM9 molecular generation - validity retained as denoising steps fall. +# +# This is the axis the discrete graph-diffusion literature competes on. DiGress +# and its successors use hundreds of denoising steps; the question is how much +# validity survives at few steps, and that depends on the training schedule and +# the decoding rule rather than on model scale - which is why it is winnable at +# this budget when text8 bits-per-character is not. +# +# Both of the project's findings are under test here at once, and QM9 is the +# strongest possible case for the second. Its marginals are extremely +# position-dependent: node slot 8 is NEVER occupied, and edge slots range from +# never-bonded to always-bonded, giving a mean total-variation distance of 0.435 +# between a slot and the average slot. Finding 2 therefore predicts that a +# relative-only encoding badly caps few-step generation here, and that absolute +# position information produces a large gain. Registered before the runs. +#SBATCH -J mol1 +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 3:00:00 +#SBATCH -o logs/mol1-%A_%a.out +#SBATCH --array=0-7%8 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +MODES=(mdlm matched); PES=(rope ape); SEEDS=(0 1) +I=$SLURM_ARRAY_TASK_ID +M=${MODES[$((I / 4))]}; P=${PES[$(((I % 4) / 2))]}; S=${SEEDS[$((I % 2))]} +NAME="mol1-$M-$P-s$S" +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_mol.py --mode $M --pe $P --seed $S --K 8 \ + --root $SCR/pcmd/data/qm9 --d 384 --layers 8 --heads 8 --bs 256 \ + --steps 60000 --lr 1e-4 --warmup 2000 --n_gen 4096 --gen_bs 512 \ + --out $SCR/pcmd/runs/$NAME diff --git a/slurm/molfair.sh b/slurm/molfair.sh new file mode 100644 index 0000000000000000000000000000000000000000..b9bd30d86e73ef7f84fd43a06251ecd8104c0258 --- /dev/null +++ b/slurm/molfair.sh @@ -0,0 +1,15 @@ +#!/bin/bash +#SBATCH -J molfair +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 2:00:00 +#SBATCH -o logs/molfair-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u mol_fair.py --runs $SCR/pcmd/runs --root $SCR/pcmd/data/qm9 \ + --n_gen 2048 --gen_bs 512 --out $SCR/pcmd/figures/mol_fair.json diff --git a/slurm/rec1.sh b/slurm/rec1.sh new file mode 100644 index 0000000000000000000000000000000000000000..88fcdb777336857c1a4a82f1de09eda6e2456b50 --- /dev/null +++ b/slurm/rec1.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# Recurrent masked diffusion with deep supervision, on hard Sudoku. +# +# The gap this targets is measured, not guessed. Our 37.86M-parameter 12-layer +# denoiser reaches 89.4%; Yang et al. (2023) reach 99.5% on a comparable split +# with 211k parameters - 180x smaller - by applying ONE block 32 times with loss +# at every recurrence. Constraint propagation is iterative, and a fixed-depth +# network cannot express thirty rounds of it at any width. +# +# Config follows theirs (d=128, 1 layer, 4 heads, R=32) so the comparison is +# about the diffusion framing rather than about scale. Inference runs R=64, +# more than training, which deep supervision is what makes possible. +#SBATCH -J rec1 +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 5:00:00 +#SBATCH -o logs/rec1-%A_%a.out +#SBATCH --array=0-3%4 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +MODES=(mdlm full); SEEDS=(0 1) +I=$SLURM_ARRAY_TASK_ID +M=${MODES[$((I / 2))]}; S=${SEEDS[$((I % 2))]} +NAME="rec1-$M-s$S" +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py --recurrent --R 32 --R_infer 64 \ + --mode $M --seed $S --pe ape --hard $SCR/pcmd/data/sudoku_hard --augment \ + --d 128 --layers 1 --heads 4 --bs 256 --steps 120000 --lr 3e-4 --warmup 3000 \ + --K 8 --lam 1.0 --eval_bs 250 --ckpt_every 10000 \ + --out $SCR/pcmd/runs/$NAME diff --git a/slurm/rsw.sh b/slurm/rsw.sh new file mode 100644 index 0000000000000000000000000000000000000000..a6bfd64a5bc4b0c2b1ca85e8e4112899c4bf5fea --- /dev/null +++ b/slurm/rsw.sh @@ -0,0 +1,15 @@ +#!/bin/bash +#SBATCH -J rsweep +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 2:00:00 +#SBATCH -o logs/rsweep-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u sudoku_rsweep.py --hard $SCR/pcmd/data/sudoku_hard \ + --runs $SCR/pcmd/runs --pattern 'rec1-*' --out $SCR/pcmd/figures/sudoku_rsweep.json diff --git a/slurm/sud1.sh b/slurm/sud1.sh new file mode 100644 index 0000000000000000000000000000000000000000..09eb89a6e6f78311d005156c775165b017ce8aaa --- /dev/null +++ b/slurm/sud1.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Sudoku on the SATNet benchmark (Wang et al., ICML 2019), 9000/1000 split, +# scored by BOARD-level accuracy - all 81 cells correct - which is the published +# metric. SATNet reports 98.3%. +# +# The registered prediction: Sudoku has a unique answer, so TC = 0 and there is +# no information-theoretic obstacle at all; the whole difficulty is mode (i) plus +# the ORDER of commitment. An entropy-budgeted decoder commits the forced cells +# first, recomputes and repeats, which IS constraint propagation. A fixed-K +# decoder commits a fixed share per pass regardless of certainty. The gap between +# them should be the largest in this project. +# +# Augmentation is by digit relabelling, a genuine Sudoku symmetry that leaves the +# constraint structure untouched. Both arms are run because SATNet did not +# augment, so the un-augmented number is the like-for-like one. +#SBATCH -J sud1 +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 4:00:00 +#SBATCH -o logs/sud1-%A_%a.out +#SBATCH --array=0-7%4 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +MODES=(mdlm full); AUG=(plain aug); SEEDS=(0 1) +I=$SLURM_ARRAY_TASK_ID +M=${MODES[$((I / 4))]}; A=${AUG[$(((I % 4) / 2))]}; S=${SEEDS[$((I % 2))]} +FLAG=""; [ "$A" = "aug" ] && FLAG="--augment" +NAME="sud1-$M-$A-s$S" +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py --mode $M $FLAG --seed $S --pe ape \ + --root $SCR/pcmd/data/sudoku --d 512 --layers 10 --heads 8 --bs 128 \ + --steps 60000 --lr 1e-4 --warmup 2000 --K 8 --lam 1.0 --eval_bs 200 \ + --out $SCR/pcmd/runs/$NAME diff --git a/slurm/sudcompare.sh b/slurm/sudcompare.sh new file mode 100644 index 0000000000000000000000000000000000000000..f52dbe32041b256d95a1d77b4b3475c2b424b328 --- /dev/null +++ b/slurm/sudcompare.sh @@ -0,0 +1,15 @@ +#!/bin/bash +#SBATCH -J sudcmp +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 16 +#SBATCH --mem=48g +#SBATCH -t 2:00:00 +#SBATCH -o logs/sudcmp-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u sudoku_compare.py --hard $SCR/pcmd/data/sudoku_hard \ + --runs $SCR/pcmd/runs --out $SCR/pcmd/figures/sudoku_compare.json diff --git a/slurm/sudhard.sh b/slurm/sudhard.sh new file mode 100644 index 0000000000000000000000000000000000000000..475917e48abd7d5da034dd363f20a72164bf90a5 --- /dev/null +++ b/slurm/sudhard.sh @@ -0,0 +1,15 @@ +#!/bin/bash +#SBATCH -J sudhard +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 1:30:00 +#SBATCH -o logs/sudhard-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u sudoku_hard.py --runs $SCR/pcmd/runs --figs $SCR/pcmd/figures \ + --out $SCR/pcmd/figures/sudoku_hard.json diff --git a/slurm/sudhard_train.sh b/slurm/sudhard_train.sh new file mode 100644 index 0000000000000000000000000000000000000000..49f6cff867226d286d23f8cd2f777417a8fad4b8 --- /dev/null +++ b/slurm/sudhard_train.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# Hard Sudoku: minimal puzzles at ~24 clues, where search-free constraint +# propagation solves about 4%. Unlike the SATNet split - which a classical +# solver saturates at 100% - this discriminates. +# +# Every puzzle was verified to have exactly one solution at generation time, and +# train/test solutions are disjoint even up to digit relabelling (an earlier +# generator leaked 767/1000 and the assertion caught it). +# +# Baselines to beat, measured on this exact test set: +# constraint propagation, no search : ~4% +# backtracking search : 100%, but it is search +# The question is what a masked diffusion model reaches in a fixed, small number +# of forward passes. +#SBATCH -J sudhard +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 4:00:00 +#SBATCH -o logs/sudhard-%A_%a.out +#SBATCH --array=0-3%4 +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +MODES=(mdlm full); SEEDS=(0 1) +I=$SLURM_ARRAY_TASK_ID +M=${MODES[$((I / 2))]}; S=${SEEDS[$((I % 2))]} +NAME="sudh-$M-s$S" +echo "=== $NAME ===" +$SCR/hf_env/bin/python -u train_sudoku.py --mode $M --seed $S --pe ape \ + --hard $SCR/pcmd/data/sudoku_hard --augment \ + --d 512 --layers 12 --heads 8 --bs 256 --steps 150000 --lr 1e-4 --warmup 4000 \ + --K 8 --lam 1.0 --eval_bs 200 --ckpt_every 10000 \ + --out $SCR/pcmd/runs/$NAME diff --git a/slurm/sudremask.sh b/slurm/sudremask.sh new file mode 100644 index 0000000000000000000000000000000000000000..cd59a18d23b483236bddf8ea26a6cfff6714fb61 --- /dev/null +++ b/slurm/sudremask.sh @@ -0,0 +1,15 @@ +#!/bin/bash +#SBATCH -J sudrm +#SBATCH -p gpu-h100 +#SBATCH --gpus=h100:1 +#SBATCH -c 8 +#SBATCH --mem=48g +#SBATCH -t 2:00:00 +#SBATCH -o logs/sudrm-%j.out +source /etc/profile +SCR=/scratch/zt1/project/msml612/user/$USER +module load python/3.10.10 +export PYTHONPATH= +cd $SCR/pcmd/src +$SCR/hf_env/bin/python -u sudoku_remask.py --hard $SCR/pcmd/data/sudoku_hard \ + --runs $SCR/pcmd/runs --pattern 'sudh-*' --out $SCR/pcmd/figures/sudoku_remask.json diff --git a/src/bench.py b/src/bench.py new file mode 100644 index 0000000000000000000000000000000000000000..5540ac2398abe059e2b299b247e0a114ae1af5b0 --- /dev/null +++ b/src/bench.py @@ -0,0 +1,38 @@ +"""Throughput of the ACTUAL Sudoku model on whatever GPU this job got. + +The class allocation bills an H100 at 144/min and a 5GB MIG slice at 9/min - 16x +apart. That only matters if the MIG slice is less than 16x slower on OUR +workload, which is 81-token sequences and a ~32M-parameter model: a size where an +H100 is largely idle. Measured rather than assumed. +""" +import os, sys, time, torch +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from model import Transformer +import diffusion as dfn +from sudoku import CELLS, SudokuTokenizer + +dev = "cuda" +tok = SudokuTokenizer() +m = Transformer(len(tok), 512, 10, 8, "ape", causal=False, max_len=CELLS + 8).to(dev) +opt = torch.optim.AdamW(m.parameters(), lr=1e-4) +x = torch.randint(0, 9, (128, CELLS), device=dev) +a = torch.rand(128, CELLS, device=dev) < 0.55 + +for _ in range(10): # warm up + with torch.autocast("cuda", dtype=torch.bfloat16): + loss = dfn.pmd_loss(m, x, a, tok, "full", 8, 1.0) + opt.zero_grad(set_to_none=True); loss.backward(); opt.step() +torch.cuda.synchronize() + +t0 = time.time() +N = 100 +for _ in range(N): + with torch.autocast("cuda", dtype=torch.bfloat16): + loss = dfn.pmd_loss(m, x, a, tok, "full", 8, 1.0) + opt.zero_grad(set_to_none=True); loss.backward(); opt.step() +torch.cuda.synchronize() +el = time.time() - t0 +name = torch.cuda.get_device_name(0) +print(f"GPU={name}") +print(f" {N/el:.1f} steps/sec ({el/N*1000:.1f} ms/step)") +print(f" 60k steps would take {60000*el/N/60:.1f} min") diff --git a/src/diffusion.py b/src/diffusion.py index 6e88bd4323657a00146f6ac0878a0c8fa1973f1f..dff7c156f3717b498f1860631866464b51a7b8c1 100644 --- a/src/diffusion.py +++ b/src/diffusion.py @@ -808,3 +808,117 @@ def cond_decode_budget(model, x0, amask, tok, steps, device, condition=True): if rem.any() and pred is not None: x[rem] = pred[rem] return x + + +# --------------------------------------------- verifier-guided remasking --- +# Masked diffusion decoding is ABSORBING: once a position is unmasked it can +# never be revised. On a constraint problem that is fatal - three wrong cells out +# of fifty-seven destroy a whole board, which is exactly the observed failure +# mode (94.9% of cells correct but only 87.9% of boards). +# +# When the task has a checkable constraint, the repair signal is exact rather +# than heuristic: we know precisely which positions conflict. Remask only those, +# re-decode them against the surviving context, and repeat. This is the diffusion +# analogue of the propagate-detect-backtrack loop every classical solver runs, +# and unlike a confidence heuristic it never remasks a position that is provably +# fine. +# +# The per-position remask counter is what keeps it terminating: a cell that has +# already been reopened `max_revisits` times is frozen, so two mutually +# inconsistent cells cannot flip each other forever. + +@torch.no_grad() +def verifier_remask_decode(model, x, tok, device, violation_fn, base_steps=16, + rounds=8, max_revisits=3, fillable=None, + expand_peers=None, allowed=None, temperature=1.0): + """Decode, then repeatedly remask whatever a verifier says is wrong. + + violation_fn(x) -> bool mask of positions involved in a violation, where x + holds the current token ids. `expand_peers`, if given, maps a violation mask + to a wider mask (for Sudoku: the conflicting cells plus their units), which + gives the model room to re-solve the local subproblem rather than re-guess a + single cell in an unchanged context. + + Returns (x, nfe) with nfe counting every forward pass, so the cost of the + repair rounds is charged honestly against the fixed-K budgets it is compared + with. + """ + x = x.clone().to(device) + can = (x == tok.mask) if fillable is None else fillable.to(device) + B, L = x.shape + + x, nfe = fixed_k_decode(model, x, tok, device, base_steps, fillable=can, + allowed=allowed, temperature=temperature) + revisits = torch.zeros((B, L), dtype=torch.long, device=device) + + for _ in range(rounds): + bad = violation_fn(x).to(device) & can + if expand_peers is not None: + bad = expand_peers(bad) & can + # a position already reopened too often is frozen, or two conflicting + # cells would keep overwriting each other indefinitely + bad = bad & (revisits < max_revisits) + if not bad.any(): + break + revisits = revisits + bad.long() + x = torch.where(bad, torch.full_like(x, tok.mask), x) + # re-decode only the reopened positions, against everything still held + steps = max(1, min(base_steps, int(bad.sum(1).max()))) + x, n2 = fixed_k_decode(model, x, tok, device, steps, fillable=bad, + allowed=allowed, temperature=temperature) + nfe += n2 + return x, nfe + + +def recurrent_cond_loss(model, x0, amask, tok, mode="mdlm", K=8, lam=1.0, + recurrences=None, deep_sup=True): + """Conditional masked-diffusion loss with DEEP SUPERVISION over recurrences. + + Loss is taken at every recurrence rather than only the last. That is the + ingredient Yang et al. (2023) identify as critical for constraint + satisfaction: it forces every application of the shared block to be a valid + one-step refinement, which in turn is what lets inference run more + recurrences than training. + + Later recurrences are weighted more heavily - a linear ramp - so the final + output still receives the strongest signal while the early ones are kept + honest. + """ + B, L = x0.shape + dev = x0.device + if mode == "matched" or mode == "full": + t = torch.randint(1, K + 1, (B,), device=dev).float() / K + else: + t = torch.rand(B, device=dev).clamp_min(1.0 / L) + + m = _cond_mask(x0, amask, tok, t) + xt = torch.where(m, torch.full_like(x0, tok.mask), x0) + outs = model(xt, recurrences=recurrences, return_all=True) + + R = len(outs) + if not deep_sup: + outs = outs[-1:] # ablation: supervise only the last + total, wsum = 0.0, 0.0 + for r, logits in enumerate(outs): + w = (r + 1) / len(outs) if deep_sup else 1.0 + ce = F.cross_entropy(logits.reshape(-1, logits.size(-1)), + x0.reshape(-1), reduction="none").view(B, L) + total = total + w * ((ce * m).sum(1) / m.sum(1).clamp_min(1)).mean() + wsum += w + loss = total / wsum + + if mode == "full": + # the t=1 term, also deep-supervised + xf = torch.where(amask, torch.full_like(x0, tok.mask), x0) + outs_f = model(xf, recurrences=recurrences, return_all=True) + if not deep_sup: + outs_f = outs_f[-1:] + tot_f, ws = 0.0, 0.0 + for r, logits in enumerate(outs_f): + w = (r + 1) / len(outs_f) if deep_sup else 1.0 + ce = F.cross_entropy(logits.reshape(-1, logits.size(-1)), + x0.reshape(-1), reduction="none").view(B, L) + tot_f = tot_f + w * ((ce * amask).sum(1) / amask.sum(1).clamp_min(1)).mean() + ws += w + loss = loss + lam * tot_f / ws + return loss diff --git a/src/gen_hard_sudoku.py b/src/gen_hard_sudoku.py new file mode 100644 index 0000000000000000000000000000000000000000..44f6f5194828e87d7d7adb9c2417d8a68dbee2e0 --- /dev/null +++ b/src/gen_hard_sudoku.py @@ -0,0 +1,200 @@ +"""Generate a HARD Sudoku benchmark, with every puzzle verified unique. + +The SATNet split is saturated - search-free constraint propagation solves +1000/1000 of it - so nothing measured there discriminates between methods. This +builds instances that do. + +Each puzzle is MINIMAL: cells are removed from a completed grid in random order +and a removal is kept only if the solution is still unique, verified by an +exhaustive counter that stops at two. The result carries roughly 19-30 clues +(mean 24), where constraint propagation alone solves about 4%. No puzzle is accepted without +that uniqueness check, so board accuracy remains a well-defined quantity. + +Solutions come from a randomised backtracking fill of an EMPTY grid, which +reaches the full space of roughly 6.7e21 valid grids. + +An earlier version applied the Sudoku symmetry group to a single seed grid +instead. That was much faster and completely wrong: every grid it produced lay +in one equivalence class, whose size modulo digit relabelling is only about 3.4 +million, so 101,000 draws collided constantly and 767 of 1000 test solutions +already appeared in training. The assertion below caught it. It is kept, and it +is the reason the slower fill is used. + +Train and test are built from DISJOINT solution grids and the script asserts +that no test solution appears in training, in any relabelling. +""" +import argparse, os, sys, time +from multiprocessing import Pool + +import numpy as np + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from sudoku import CELLS, GRID + +PEERS = None + + +def _peers(): + out = [] + for i in range(CELLS): + r, c = divmod(i, GRID) + br, bc = 3 * (r // 3), 3 * (c // 3) + s = set() + for k in range(GRID): + s.add(r * GRID + k) + s.add(k * GRID + c) + for dr in range(3): + for dc in range(3): + s.add((br + dr) * GRID + bc + dc) + s.discard(i) + out.append(sorted(s)) + return out + + +PEERS = _peers() + + + +def candidates(g): + cand = [0x1FF if v == 0 else (1 << (v - 1)) for v in g] + for i, v in enumerate(g): + if v: + bit = 1 << (v - 1) + for p in PEERS[i]: + if cand[p] & bit: + cand[p] &= ~bit + if cand[p] == 0: + return None + return cand + + +def count_solutions(grid, limit=2): + g = [int(v) for v in grid] + + def rec(): + cand = candidates(g) + if cand is None: + return 0 + best, bi = 10, -1 + for i in range(CELLS): + if g[i] == 0: + n = bin(cand[i]).count("1") + if n < best: + best, bi = n, i + if n == 1: + break + if bi < 0: + return 1 + tot = 0 + for d in range(9): + if cand[bi] & (1 << d): + g[bi] = d + 1 + tot += rec() + g[bi] = 0 + if tot >= limit: + return tot + return tot + + return rec() + + +def random_solution(rng): + """A uniformly-ish random completed grid, by randomised backtracking fill. + + The previous version applied the Sudoku symmetry group to a single seed grid. + Every grid it produced therefore lay in ONE equivalence class, whose size + modulo digit relabelling is only about 3.4 million - so drawing 101,000 + samples collided constantly, and 767 of 1000 test solutions turned out to + already appear in training. Filling an empty grid instead reaches the full + space of roughly 6.7e21 valid grids, where collisions are impossible in + practice. + """ + g = [0] * CELLS + while True: + cand = candidates(g) + if cand is None: + return None + best, bi = 10, -1 + for i in range(CELLS): + if g[i] == 0: + n = bin(cand[i]).count("1") + if n < best: + best, bi = n, i + if bi < 0: + return np.array(g, dtype=np.int64) + digits = [d + 1 for d in range(9) if cand[bi] & (1 << d)] + rng.shuffle(digits) + placed = False + for d in digits: + g[bi] = d + if candidates(g) is not None: + placed = True + break + g[bi] = 0 + if not placed: + # dead end: restart rather than backtrack; restarts are rare and a + # fresh fill is cheaper than unwinding a deep search + g = [0] * CELLS + + +def make_one(seed): + rng = np.random.default_rng(seed) + sol = None + while sol is None: + sol = random_solution(rng) + g = [int(v) for v in sol] + for c in rng.permutation(CELLS): + keep = g[c] + g[c] = 0 + if count_solutions(g, 2) != 1: + g[c] = keep + return np.array(g, dtype=np.int64), sol + + +def canon(g): + m, out = {}, [] + for v in g: + if v not in m: + m[v] = len(m) + 1 + out.append(m[v]) + return bytes(out) + + +if __name__ == "__main__": + p = argparse.ArgumentParser() + p.add_argument("--n_train", type=int, default=100000) + p.add_argument("--n_test", type=int, default=1000) + p.add_argument("--workers", type=int, default=64) + p.add_argument("--out", default="data/sudoku_hard") + a = p.parse_args() + os.makedirs(a.out, exist_ok=True) + + total = a.n_train + a.n_test + t0 = time.time() + with Pool(a.workers) as pool: + got = pool.map(make_one, range(total), chunksize=32) + P = np.stack([g for g, _ in got]) + S = np.stack([s for _, s in got]) + print(f"generated {len(P)} minimal puzzles in {time.time()-t0:.0f}s", flush=True) + print(f"clues: min={(P>0).sum(1).min()} max={(P>0).sum(1).max()} " + f"mean={(P>0).sum(1).mean():.1f}", flush=True) + + Ptr, Str = P[:a.n_train], S[:a.n_train] + Pte, Ste = P[a.n_train:], S[a.n_train:] + + # a test solution must not appear in training, even up to digit relabelling + tr = {canon(r) for r in Str} + leak = sum(canon(r) in tr for r in Ste) + print(f"test solutions also in training (up to relabelling): {leak}", flush=True) + assert leak == 0, "leakage between train and test" + + # every puzzle must still admit exactly one solution + bad = sum(count_solutions(Pte[i], 2) != 1 for i in range(0, len(Pte), 25)) + print(f"test puzzles without a unique solution (sampled): {bad}", flush=True) + assert bad == 0 + + np.save(os.path.join(a.out, "train_puz.npy"), Ptr) + np.save(os.path.join(a.out, "train_sol.npy"), Str) + np.save(os.path.join(a.out, "test_puz.npy"), Pte) + np.save(os.path.join(a.out, "test_sol.npy"), Ste) + print(f"saved to {a.out}: train {Ptr.shape} test {Pte.shape}", flush=True) diff --git a/src/model.py b/src/model.py index 14b2c33830ce09632f18491a23c486714eb47bb1..46cf36a416c4710060fa38238d42c879a593cfb2 100644 --- a/src/model.py +++ b/src/model.py @@ -162,3 +162,74 @@ class Transformer(nn.Module): def n_params(self): return sum(p.numel() for p in self.parameters() if p.requires_grad) + + +class RecurrentDenoiser(nn.Module): + """A weight-shared block applied R times, supervised at every step. + + Motivated by a measurement, not a hunch. On hard Sudoku a 37.86M-parameter + 12-layer denoiser reaches 89.4%, while the Recurrent Transformer of Yang et + al. (2023) reaches 99.5% on a comparable split with 211k parameters - 180x + smaller. The difference is not capacity, it is that constraint propagation is + an ITERATIVE algorithm: solving a hard Sudoku takes tens of rounds of + "eliminate, propagate, repeat". A fixed 12-layer feedforward network cannot + express thirty rounds of propagation at any width; one block applied thirty + times can. + + This is the same argument THEORY.md makes for addition - that a K-pass decode + has effective depth L*K and "borrows depth from the decoding loop" - applied + to the architecture instead of to the decoder. + + Two details carry the method: + + INPUT INJECTION. The token embedding is re-added at every recurrence. + Without it the input signal decays through thirty applications of the same + block and the model forgets the clues it is solving for. + + DEEP SUPERVISION. Training loss is taken at EVERY recurrence, not only the + last. That forces each application to make progress on its own rather than + letting the stack learn one entangled thirty-step function, and it is what + lets inference run MORE recurrences than training - the per-step objective + is the same at every step, so the map is iterable beyond where it was fit. + """ + + def __init__(self, vocab, d=128, n_layers=1, n_heads=4, pe="ape", + max_len=128, recurrences=32, hidden_mult=4): + super().__init__() + self.pe_kind, self.causal = pe, False + self.recurrences = recurrences + self.emb = nn.Embedding(vocab, d) + if pe == "ape": + self.pos = nn.Embedding(max_len, d) + elif pe == "sin": + self.register_buffer("pos_sin", sinusoidal(max_len, d), persistent=False) + self.blocks = nn.ModuleList([Block(d, n_heads, pe, False, max_len) + for _ in range(n_layers)]) + self.norm = nn.LayerNorm(d) + self.head = nn.Linear(d, vocab, bias=False) + self.apply(Transformer._init) + + def _inject(self, idx): + x = self.emb(idx) + if self.pe_kind == "ape": + x = x + self.pos(torch.arange(idx.shape[1], device=idx.device))[None] + elif self.pe_kind == "sin": + x = x + self.pos_sin[torch.arange(idx.shape[1], device=idx.device)][None] + return x + + def forward(self, idx, pad_mask=None, pos_ids=None, budget=None, + recurrences=None, return_all=False): + """Returns final logits, or the list of logits from every recurrence.""" + R = recurrences or self.recurrences + inp = self._inject(idx) + h = inp + outs = [] + for _ in range(R): + h = h + inp # input injection: keep the clues alive + for b in self.blocks: + h = b(h, pad_mask, pos_ids) + outs.append(self.head(self.norm(h))) + return outs if return_all else outs[-1] + + def n_params(self): + return sum(p.numel() for p in self.parameters() if p.requires_grad) diff --git a/src/sudoku.py b/src/sudoku.py index 18cdfccde8a3f1715b28bf94c2f3bb5d707bc3cc..559ec41678ef7ea9b659490b7821753777f8acca 100644 --- a/src/sudoku.py +++ b/src/sudoku.py @@ -142,3 +142,53 @@ if __name__ == "__main__": tr = {bytes(r) for r in Xtr[::3]} ov = sum(bytes(r) in tr for r in Xte) print(f"test puzzles also seen in the training sample: {ov} (must be 0)") + + +def _unit_list(): + us = [] + for r in range(GRID): + us.append([r * GRID + c for c in range(GRID)]) + for c in range(GRID): + us.append([r * GRID + c for r in range(GRID)]) + for br in range(0, GRID, 3): + for bc in range(0, GRID, 3): + us.append([(br + dr) * GRID + bc + dc for dr in range(3) for dc in range(3)]) + return us + + +UNIT_LIST = _unit_list() + + +def violation_mask(x): + """Positions involved in a row/column/box duplicate. Exact, not heuristic. + + x is (B, 81) of token ids 0-8. A cell is flagged when its value appears more + than once in any unit it belongs to. A solved grid flags nothing, so this + doubles as the stopping condition for verifier-guided remasking. + """ + import torch + B = x.shape[0] + bad = torch.zeros_like(x, dtype=torch.bool) + for unit in UNIT_LIST: + idx = torch.tensor(unit, device=x.device) + vals = x[:, idx] # (B, 9) + same = vals[:, :, None] == vals[:, None, :] # (B, 9, 9) + dup = same.sum(-1) > 1 # appears more than once + bad[:, idx] |= dup + return bad + + +def expand_to_units(bad): + """Widen a violation mask to every cell sharing a unit with a flagged cell. + + Re-deciding a single conflicting cell in an otherwise unchanged context tends + to reproduce the same value. Reopening its row, column and box gives the + model a subproblem it can actually re-solve. + """ + import torch + out = torch.zeros_like(bad) + for unit in UNIT_LIST: + idx = torch.tensor(unit, device=bad.device) + hit = bad[:, idx].any(1, keepdim=True) # (B, 1) + out[:, idx] |= hit + return out diff --git a/src/sudoku_compare.py b/src/sudoku_compare.py new file mode 100644 index 0000000000000000000000000000000000000000..c5e6bee99e0d43cf77394f20a4b2a59716de17d1 --- /dev/null +++ b/src/sudoku_compare.py @@ -0,0 +1,279 @@ +"""The full comparison: our diffusion model against classical solvers and the +published neural results, on a benchmark whose difficulty is stated rather than +assumed. + +Published Sudoku numbers are reported on different puzzle distributions, and +quoting them side by side without saying so is exactly the error that cost this +project its SATNet claim. So: + + * classical baselines are MEASURED on our test set, not cited; + * our model is reported STRATIFIED BY CLUE COUNT, so a reader can compare at + whatever difficulty a published number used; + * published numbers appear only in a clearly separated block, with their + distribution named. + +Three classical baselines, each measuring something different: + + propagation naked + hidden singles to fixpoint, NO search. The "free" solver. + backtracking full search. Solves everything, so the informative quantity is + its COST - how many search nodes it needs - which is the honest + yardstick for "how hard is this puzzle really". + greedy-MRV one pass of most-constrained-variable assignment without + backtracking: what a single confident sweep achieves. + +For the diffusion model the quantity of interest is board accuracy at a small +fixed number of forward passes, which is the comparison that makes the +generative framing worth anything. +""" +import argparse, glob, json, os, sys +from collections import defaultdict + +import numpy as np +import torch + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from model import RecurrentDenoiser, Transformer +import diffusion as dfn +from sudoku import CELLS, GRID, SudokuTokenizer +from gen_hard_sudoku import candidates, count_solutions, PEERS + +# published results, with the distribution each was measured on +PUBLISHED = [ + ("SATNet (Wang et al. ICML 2019)", "SATNet easy split, ~36 clues", 0.983), + ("SATNet, evaluated on RRN hard", "RRN hard, 17 clues", 0.061), + ("RRN (Palm et al. NeurIPS 2018)", "RRN hard, 17 clues", 0.967), + ("Recurrent Transformer (Yang et al. 2023)", "RRN hard, 17 clues", 0.995), + ("Recurrent Transformer", "SATNet easy split", 1.000), +] +PARAMS = {"Recurrent Transformer (Yang et al. 2023)": "211k", + "RRN (Palm et al. NeurIPS 2018)": "201k", + "SATNet (Wang et al. ICML 2019)": "618k"} + + +def units(): + us = [] + for r in range(GRID): + us.append([r * GRID + c for c in range(GRID)]) + for c in range(GRID): + us.append([r * GRID + c for r in range(GRID)]) + for br in range(0, GRID, 3): + for bc in range(0, GRID, 3): + us.append([(br + dr) * GRID + bc + dc + for dr in range(3) for dc in range(3)]) + return us + + +UNITS = units() + + +def propagate(grid): + g = [int(v) for v in grid] + while True: + cand = candidates(g) + if cand is None: + return None + changed = False + for i in range(CELLS): + if g[i] == 0 and bin(cand[i]).count("1") == 1: + g[i] = cand[i].bit_length() + changed = True + if not changed: + for unit in UNITS: + for d in range(9): + bit = 1 << d + spots = [i for i in unit if g[i] == 0 and cand[i] & bit] + if len(spots) == 1: + g[spots[0]] = d + 1 + changed = True + if not changed: + return g + + +def greedy_mrv(grid): + """Most-constrained-variable, one sweep, no backtracking.""" + g = [int(v) for v in grid] + for _ in range(CELLS): + cand = candidates(g) + if cand is None: + return None + best, bi = 10, -1 + for i in range(CELLS): + if g[i] == 0: + n = bin(cand[i]).count("1") + if n < best: + best, bi = n, i + if bi < 0: + return g + g[bi] = cand[bi].bit_length() if best == 1 else \ + [d + 1 for d in range(9) if cand[bi] & (1 << d)][0] + return g + + +def backtrack_cost(grid): + """Solve by full search; return (solution, number of search nodes).""" + g = [int(v) for v in grid] + nodes = [0] + + def rec(): + nodes[0] += 1 + cand = candidates(g) + if cand is None: + return False + best, bi = 10, -1 + for i in range(CELLS): + if g[i] == 0: + n = bin(cand[i]).count("1") + if n < best: + best, bi = n, i + if n == 1: + break + if bi < 0: + return True + for d in range(9): + if cand[bi] & (1 << d): + g[bi] = d + 1 + if rec(): + return True + g[bi] = 0 + return False + + ok = rec() + return (np.array(g) if ok else None), nodes[0] + + +p = argparse.ArgumentParser() +p.add_argument("--hard", default="data/sudoku_hard") +p.add_argument("--runs", default="runs") +p.add_argument("--pattern", default="sudh-*") +p.add_argument("--eval_bs", type=int, default=200) +p.add_argument("--out", default="figures/sudoku_compare.json") +a = p.parse_args() + +P = np.load(os.path.join(a.hard, "test_puz.npy")) +S = np.load(os.path.join(a.hard, "test_sol.npy")) +clues = (P > 0).sum(1) +print(f"benchmark: {len(P)} puzzles, clues {clues.min()}-{clues.max()} " + f"(mean {clues.mean():.1f})\n", flush=True) + +res = {"n": int(len(P)), "clues": {"min": int(clues.min()), "max": int(clues.max()), + "mean": float(clues.mean())}} + +print("=" * 88) +print("CLASSICAL BASELINES, measured on THIS test set") +print("=" * 88) +prop_ok = np.array([(lambda r: r is not None and all(v > 0 for v in r) + and np.array_equal(r, S[i]))(propagate(P[i])) + for i in range(len(P))]) +mrv_ok = np.array([(lambda r: r is not None and all(v > 0 for v in r) + and np.array_equal(r, S[i]))(greedy_mrv(P[i])) + for i in range(len(P))]) +bt = [backtrack_cost(P[i]) for i in range(len(P))] +bt_ok = np.array([g is not None and np.array_equal(g, S[i]) for i, (g, _) in enumerate(bt)]) +nodes = np.array([n for _, n in bt]) +res["classical"] = {"propagation": float(prop_ok.mean()), + "greedy_mrv": float(mrv_ok.mean()), + "backtracking": float(bt_ok.mean()), + "backtrack_nodes_mean": float(nodes.mean()), + "backtrack_nodes_median": float(np.median(nodes))} +print(f" propagation only (no search) : {prop_ok.mean()*100:6.2f}%") +print(f" greedy MRV, one sweep : {mrv_ok.mean()*100:6.2f}%") +print(f" full backtracking search : {bt_ok.mean()*100:6.2f}% " + f"(mean {nodes.mean():.0f} search nodes, median {np.median(nodes):.0f})") +print(f"\n >>> propagation leaves {(1-prop_ok.mean())*100:.1f}% unsolved, so this") +print(f" >>> benchmark discriminates. Search solves all of it, at a cost.") + +print() +print("=" * 88) +print("OUR FROM-SCRATCH MASKED DIFFUSION MODEL") +print("=" * 88) +dev = "cuda" if torch.cuda.is_available() else "cpu" +tok = SudokuTokenizer() +runs = defaultdict(list) +for d in sorted(glob.glob(os.path.join(a.runs, a.pattern))): + if os.path.exists(os.path.join(d, "result.json")): + r = json.load(open(os.path.join(d, "result.json"))) + tag = ("recurrent-" if r["args"].get("recurrent") else "feedforward-") \ + + r["args"]["mode"] + runs[tag].append((d, r)) + +if not runs: + print(" (no trained models yet)") +else: + Pt = torch.from_numpy(P) + St = torch.from_numpy(S - 1) + blank = Pt == 0 + + def load(d): + cfg = json.load(open(os.path.join(d, "result.json")))["args"] + if cfg.get("recurrent"): + m = RecurrentDenoiser(len(tok), cfg["d"], cfg["layers"], cfg["heads"], + cfg["pe"], max_len=CELLS + 8, + recurrences=cfg.get("R_infer") or cfg["R"]).to(dev) + else: + m = Transformer(len(tok), cfg["d"], cfg["layers"], cfg["heads"], + cfg["pe"], causal=False, max_len=CELLS + 8).to(dev) + m.load_state_dict(torch.load(os.path.join(d, "model.pt"), map_location=dev)) + m.eval() + return m + + @torch.no_grad() + def run(m, steps): + out = [] + for i in range(0, len(Pt), a.eval_bs): + gold = St[i:i + a.eval_bs].to(dev) + bl = blank[i:i + a.eval_bs].to(dev) + x = torch.where(bl, torch.full_like(gold, tok.mask), gold) + y, _ = dfn.fixed_k_decode(m, x, tok, dev, steps, fillable=bl) + out.append(y.cpu().numpy() + 1) + return np.concatenate(out) + + res["diffusion"] = {} + for mode, lst in sorted(runs.items()): + npar = json.load(open(os.path.join(lst[0][0], "result.json")))["args"] + print(f"\n {mode} ({len(lst)} seeds, d={npar['d']} layers={npar['layers']}" + + (f" R={npar.get('R')}→{npar.get('R_infer') or npar.get('R')}" + if npar.get('recurrent') else "") + ")") + print(f" {'passes':>8}{'board':>10}{'vs propagation':>17}") + per_pass = {} + for K in (1, 4, 8, 16, 32, 61): + accs, preds = [], None + for d, _ in lst: + m = load(d) + pr = run(m, K) + accs.append(float((pr == S).all(axis=1).mean())) + preds = pr if preds is None else preds + del m + torch.cuda.empty_cache() + mean = float(np.mean(accs)) + per_pass[K] = {"board": mean, "per_seed": accs} + if K == 61: + per_pass[K]["by_clue"] = { + int(c): float((preds[clues == c] == S[clues == c]).all(axis=1).mean()) + for c in sorted(set(clues.tolist())) + if (clues == c).sum() >= 20} + print(f" {K:>8}{mean*100:>9.2f}%{mean/max(prop_ok.mean(),1e-9):>15.1f}x") + res["diffusion"][mode] = per_pass + + best_mode = max(res["diffusion"], key=lambda m: res["diffusion"][m][61]["board"]) + bc = res["diffusion"][best_mode][61].get("by_clue", {}) + if bc: + print(f"\n stratified by clue count (mode={best_mode}, 61 passes):") + print(f" {'clues':>7}{'n':>6}{'board':>10}") + for c in sorted(bc): + print(f" {c:>7}{int((clues==c).sum()):>6}{bc[c]*100:>9.2f}%") + +print() +print("=" * 88) +print("PUBLISHED NEURAL RESULTS — different puzzle distributions, NOT directly comparable") +print("=" * 88) +for name, dist, acc in PUBLISHED: + pm = PARAMS.get(name, "") + print(f" {acc*100:6.1f}% {name:<42} [{dist}]" + (f" {pm} params" if pm else "")) +print(f"\n Our benchmark is {clues.min()}-{clues.max()} clues (mean {clues.mean():.1f}).") +print(" The 17-clue RRN split is strictly harder than ours; the SATNet split,") +print(" at ~36 clues, is strictly easier and is fully solved by propagation alone.") +print(" The stratified table above is what permits a matched-difficulty comparison.") + +os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True) +json.dump(res, open(a.out, "w"), indent=2) +print(f"\nwrote {a.out}") diff --git a/src/sudoku_hard.py b/src/sudoku_hard.py new file mode 100644 index 0000000000000000000000000000000000000000..7cd5f49ee3e7f481169448f24334117f37c2edff --- /dev/null +++ b/src/sudoku_hard.py @@ -0,0 +1,136 @@ +"""The Sudoku evaluation that actually discriminates. + +The SATNet split is saturated: plain constraint propagation, with no search at +all, solves 1000/1000 of its test puzzles. Any method that scores 100% there has +demonstrated nothing, ours included, and the published 98.3% is below what a +forty-line classical solver achieves. + +Harder instances are built by stripping clues from the SAME solutions while +verifying after every removal that the solution stays unique, so accuracy remains +well defined. Constraint propagation alone solves 27.5% of the 30-clue set, 4.2% +of the 26-clue set and 3.3% of the 22-clue set - these discriminate. + +Two things are being measured at once, and both are worth having: + + * CAPABILITY. Can the diffusion model solve puzzles a search-free classical + solver cannot? That is a real question with a real answer. + * GENERALISATION. The models were trained on 31-42 clue puzzles, so 30, 26 and + 22 clues are out of distribution. This asks whether they learned Sudoku or + learned a clue density. + +The decoding comparison is repeated here because this is where it should finally +matter: with fewer clues the model's marginals are no longer near-perfect, so +commit order has something to do. +""" +import argparse, glob, json, os, sys +from collections import defaultdict + +import numpy as np +import torch + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from model import Transformer +import diffusion as dfn +from sudoku import CELLS, SudokuTokenizer, is_valid_solution + +p = argparse.ArgumentParser() +p.add_argument("--runs", default="runs") +p.add_argument("--figs", default="figures") +p.add_argument("--eval_bs", type=int, default=120) +p.add_argument("--out", default="figures/sudoku_hard.json") +a = p.parse_args() + +dev = "cuda" if torch.cuda.is_available() else "cpu" +tok = SudokuTokenizer() + +sets = {} +for t in (30, 26, 22): + pf = os.path.join(a.figs, f"hard_{t}_puz.npy") + sf = os.path.join(a.figs, f"hard_{t}_sol.npy") + if os.path.exists(pf): + sets[t] = (np.load(pf), np.load(sf)) +if not sets: + sys.exit("no hard puzzle sets found - run sudoku_validate.py first") +for t, (P, S) in sets.items(): + print(f" {t}-clue set: {len(P)} puzzles, mean {(P>0).sum(1).mean():.1f} clues", + flush=True) + + +def load(run_dir): + cfg = json.load(open(os.path.join(run_dir, "result.json")))["args"] + m = Transformer(len(tok), cfg["d"], cfg["layers"], cfg["heads"], cfg["pe"], + causal=False, max_len=CELLS + 8).to(dev) + m.load_state_dict(torch.load(os.path.join(run_dir, "model.pt"), map_location=dev)) + m.eval() + return m, cfg + + +@torch.no_grad() +def solve(model, P, S, fn): + """Board accuracy on a hard set: every one of the 81 cells correct.""" + ok = [] + for i in range(0, len(P), a.eval_bs): + pz = torch.from_numpy(P[i:i + a.eval_bs]).to(dev) + gold = torch.from_numpy(S[i:i + a.eval_bs] - 1).to(dev) + blank = pz == 0 + x = torch.where(blank, torch.full_like(gold, tok.mask), gold) + y, _ = fn(x, blank) + pred = (y.cpu().numpy() + 1) + ok.extend((pred == S[i:i + a.eval_bs]).all(axis=1).tolist()) + return float(np.mean(ok)) + + +res = defaultdict(dict) +runs = sorted(glob.glob(os.path.join(a.runs, "sud1-*"))) +print(f"\nevaluating {len(runs)} trained models on the hard sets", flush=True) + +for d in runs: + if not os.path.exists(os.path.join(d, "model.pt")): + continue + m, cfg = load(d) + arm = f"{cfg['mode']}-{'aug' if cfg.get('augment') else 'plain'}" + for t, (P, S) in sets.items(): + for tag, fn in [ + ("fixedK_1", lambda x, b: dfn.fixed_k_decode(m, x, tok, dev, 1, fillable=b)), + ("fixedK_8", lambda x, b: dfn.fixed_k_decode(m, x, tok, dev, 8, fillable=b)), + ("fixedK_45", lambda x, b: dfn.fixed_k_decode(m, x, tok, dev, 45, fillable=b)), + ("entbudget_0.01", lambda x, b: dfn.entropy_budget_decode( + m, x, tok, dev, 0.01, max_steps=CELLS + 2, fillable=b)), + ("entbudget_0.2", lambda x, b: dfn.entropy_budget_decode( + m, x, tok, dev, 0.2, max_steps=CELLS + 2, fillable=b))]: + res[(arm, t)].setdefault(tag, []).append(solve(m, P, S, fn)) + del m + torch.cuda.empty_cache() + print(f" done {os.path.basename(d)}", flush=True) + +CP = {30: 0.275, 26: 0.042, 22: 0.033} # measured in sudoku_validate.py + +print() +print("=" * 86) +print("HARD SUDOKU — board accuracy, against a search-free classical solver") +print("=" * 86) +print("Models were trained on 31-42 clues, so every column here is out of distribution.\n") +arms = sorted({k[0] for k in res}) +for t in sorted(sets, reverse=True): + print(f"--- {t}-clue puzzles (constraint propagation alone: {CP[t]*100:.1f}%)") + print(f" {'arm':<16}" + "".join(f"{g:>16}" for g in + ("fixedK_1", "fixedK_8", "fixedK_45", + "entbud_0.01", "entbud_0.2"))) + for arm in arms: + r = res.get((arm, t)) + if not r: + continue + cells = [] + for g in ("fixedK_1", "fixedK_8", "fixedK_45", "entbudget_0.01", "entbudget_0.2"): + v = r.get(g) + cells.append(f"{np.mean(v)*100:>15.1f}%" if v else f"{'—':>16}") + print(f" {arm:<16}" + "".join(cells)) + print() + +json.dump({f"{k[0]}|{k[1]}": {g: float(np.mean(v)) for g, v in d.items()} + for k, d in res.items()}, + open(a.out, "w"), indent=2) +print(f"wrote {a.out}") +print("\nReading: a number above the classical rate is a capability the search-free") +print("solver does not have. A number below it means the model is worse than the") +print("first thing anyone would write, and should be reported that way.") diff --git a/src/sudoku_remask.py b/src/sudoku_remask.py new file mode 100644 index 0000000000000000000000000000000000000000..ffab2978bd081bc43fe313195001c8f4758c32e6 --- /dev/null +++ b/src/sudoku_remask.py @@ -0,0 +1,180 @@ +"""Does exact verifier feedback fix the absorbing-commit failure? + +Diagnosis from the trained models: 94.9% of cells are correct but only 87.9% of +boards. Failed boards therefore carry roughly three wrong cells out of +fifty-seven - they are near misses, not collapses. Masked diffusion decoding is +absorbing, so those three cells can never be revised and each one destroys a +whole board. + +Sudoku constraints are exactly checkable, so the repair signal is not a +heuristic: we know precisely which cells conflict. Remask exactly those (widened +to their row, column and box, since re-deciding one cell in an unchanged context +reproduces the same value), re-decode, repeat. + +Every forward pass is counted, including the repair rounds, so the comparison +against fixed-K budgets is honest rather than flattering: a method that needs +sixty passes to repair is reported at sixty. + +The control that matters is RANDOM remasking of the same number of cells. If +random repair helps as much as verifier-guided repair, then the gain is just +extra compute and the verifier is doing nothing. +""" +import argparse, glob, json, os, sys + +import numpy as np +import torch + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from model import Transformer +import diffusion as dfn +from sudoku import (CELLS, SudokuTokenizer, expand_to_units, violation_mask) + +p = argparse.ArgumentParser() +p.add_argument("--hard", default="data/sudoku_hard") +p.add_argument("--runs", default="runs") +p.add_argument("--pattern", default="sudh-*") +p.add_argument("--eval_bs", type=int, default=250) +p.add_argument("--out", default="figures/sudoku_remask.json") +a = p.parse_args() + +dev = "cuda" if torch.cuda.is_available() else "cpu" +tok = SudokuTokenizer() +P = np.load(os.path.join(a.hard, "test_puz.npy")) +S = np.load(os.path.join(a.hard, "test_sol.npy")) +clues = (P > 0).sum(1) +print(f"benchmark: {len(P)} puzzles, clues {clues.min()}-{clues.max()} " + f"(mean {clues.mean():.1f})", flush=True) + +Pt, St = torch.from_numpy(P), torch.from_numpy(S - 1) +blank = Pt == 0 + + +def load(d): + cfg = json.load(open(os.path.join(d, "result.json")))["args"] + m = Transformer(len(tok), cfg["d"], cfg["layers"], cfg["heads"], cfg["pe"], + causal=False, max_len=CELLS + 8).to(dev) + m.load_state_dict(torch.load(os.path.join(d, "model.pt"), map_location=dev)) + m.eval() + return m, cfg + + +def random_remask_like(bad_counts): + """Control: remask the same NUMBER of cells, chosen at random.""" + def fn(x): + B, L = x.shape + out = torch.zeros_like(x, dtype=torch.bool) + for b in range(B): + k = int(bad_counts[b]) + if k: + idx = torch.randperm(L, device=x.device)[:k] + out[b, idx] = True + return out + return fn + + +@torch.no_grad() +def evaluate(m, fn): + preds, nfes = [], [] + for i in range(0, len(Pt), a.eval_bs): + gold = St[i:i + a.eval_bs].to(dev) + bl = blank[i:i + a.eval_bs].to(dev) + x = torch.where(bl, torch.full_like(gold, tok.mask), gold) + y, nfe = fn(x, bl) + preds.append(y.cpu().numpy() + 1) + nfes.append(nfe) + pred = np.concatenate(preds) + board = float((pred == S).all(axis=1).mean()) + cell = float((pred[P == 0] == S[P == 0]).mean()) + return board, cell, float(np.mean(nfes)), pred + + +res = {} +dirs = sorted(glob.glob(os.path.join(a.runs, a.pattern))) +dirs = [d for d in dirs if os.path.exists(os.path.join(d, "model.pt"))] +print(f"evaluating {len(dirs)} models\n", flush=True) + +for d in dirs: + m, cfg = load(d) + name = os.path.basename(d) + print(f"--- {name}", flush=True) + print(f" {'decoder':<34}{'NFE':>8}{'BOARD':>10}{'cell':>9}", flush=True) + row = {} + + for K in (16, 61): + b, c, n, _ = evaluate(m, lambda x, bl, K=K: dfn.fixed_k_decode( + m, x, tok, dev, K, fillable=bl)) + row[f"fixedK_{K}"] = {"board": b, "cell": c, "nfe": n} + print(f" {'fixed-K K=' + str(K):<34}{n:>8.1f}{b*100:>9.2f}%{c*100:>8.2f}%", + flush=True) + + b, c, n, _ = evaluate(m, lambda x, bl: dfn.entropy_budget_decode( + m, x, tok, dev, 0.01, max_steps=CELLS + 2, fillable=bl)) + row["entbudget"] = {"board": b, "cell": c, "nfe": n} + print(f" {'entropy-budget B=0.01':<34}{n:>8.1f}{b*100:>9.2f}%{c*100:>8.2f}%", + flush=True) + + # Repair must start from the STRONGEST available decode, not a weak one. + # An earlier version used base_steps=16, which decodes to ~83% before repair, + # so the repaired result never beat plain 61-pass decoding at ~88% - the + # method was handicapped by its starting point, not by the idea. + for base in (32, 61): + for rounds in (4, 8): + b, c, n, _ = evaluate( + m, lambda x, bl, r=rounds, bs=base: dfn.verifier_remask_decode( + m, x, tok, dev, violation_mask, base_steps=bs, rounds=r, + max_revisits=3, fillable=bl, expand_peers=expand_to_units)) + row[f"verifier_b{base}_r{rounds}"] = {"board": b, "cell": c, "nfe": n} + print(f" {'VERIFIER base=' + str(base) + ' rounds=' + str(rounds):<34}" + f"{n:>8.1f}{b*100:>9.2f}%{c*100:>8.2f}%", flush=True) + + # narrow repair: remask ONLY the conflicting cells, not their whole units + b, c, n, _ = evaluate(m, lambda x, bl: dfn.verifier_remask_decode( + m, x, tok, dev, violation_mask, base_steps=61, rounds=8, + max_revisits=3, fillable=bl, expand_peers=None)) + row["verifier_narrow"] = {"board": b, "cell": c, "nfe": n} + print(f" {'VERIFIER base=61 narrow (no units)':<34}{n:>8.1f}{b*100:>9.2f}%" + f"{c*100:>8.2f}%", flush=True) + + # control: same budget, but remask at random instead of where it is wrong + b, c, n, _ = evaluate(m, lambda x, bl: dfn.verifier_remask_decode( + m, x, tok, dev, + lambda z: (torch.rand_like(z, dtype=torch.float) < 0.12), + base_steps=61, rounds=8, max_revisits=3, fillable=bl, + expand_peers=None)) + row["random_remask_control"] = {"board": b, "cell": c, "nfe": n} + print(f" {'CONTROL: random remask, 8 rounds':<34}{n:>8.1f}{b*100:>9.2f}%" + f"{c*100:>8.2f}%", flush=True) + + res[name] = row + del m + torch.cuda.empty_cache() + print(flush=True) + +if res: + print("=" * 76) + print("SUMMARY (mean over models)") + print("=" * 76) + keys = list(next(iter(res.values())).keys()) + print(f"{'decoder':<34}{'NFE':>8}{'BOARD':>10}") + base = None + for k in keys: + bs = [r[k]["board"] for r in res.values()] + ns = [r[k]["nfe"] for r in res.values()] + if k == "fixedK_61": + base = float(np.mean(bs)) + print(f"{k:<34}{np.mean(ns):>8.1f}{np.mean(bs)*100:>9.2f}%") + if base is not None: + best_v = max((np.mean([r[k]["board"] for r in res.values()]) + for k in keys if k.startswith("verifier_")), default=0) + ctrl = np.mean([r["random_remask_control"]["board"] for r in res.values()]) + print(f"\n best verifier-guided : {best_v*100:.2f}%") + print(f" fixed-K at 61 passes : {base*100:.2f}% " + f"(verifier gains {(best_v-base)*100:+.2f} pts)") + print(f" random-remask control: {ctrl*100:.2f}% " + f"({(ctrl-base)*100:+.2f} pts)") + print("\n If the control gains as much, the verifier is doing nothing and") + print(" the improvement is only the extra compute.") + +os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True) +json.dump(res, open(a.out, "w"), indent=2) +print(f"\nwrote {a.out}") diff --git a/src/sudoku_rsweep.py b/src/sudoku_rsweep.py new file mode 100644 index 0000000000000000000000000000000000000000..e1c00a8c420d830061b2f8f4d4d47f550e32be8c --- /dev/null +++ b/src/sudoku_rsweep.py @@ -0,0 +1,75 @@ +"""Does deep supervision make the recurrence EXTRAPOLATE past its training depth? + +The claim behind supervising every recurrence is that it forces each application +of the shared block to be a valid one-step refinement. If that holds, the map is +iterable beyond where it was fit, and running more recurrences at inference than +at training should keep helping - or at least not collapse. + +This is free: no retraining, only re-evaluating trained checkpoints at different +recurrence counts. A model whose accuracy peaks exactly at its training depth and +degrades past it has learned one entangled R-step function rather than a +repeatable refinement step. +""" +import argparse, glob, json, os, sys +from collections import defaultdict + +import numpy as np +import torch + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from model import RecurrentDenoiser +import diffusion as dfn +from sudoku import CELLS, SudokuTokenizer + +p = argparse.ArgumentParser() +p.add_argument("--hard", default="data/sudoku_hard") +p.add_argument("--runs", default="runs") +p.add_argument("--pattern", default="rec1-*") +p.add_argument("--passes", type=int, default=16) +p.add_argument("--eval_bs", type=int, default=250) +p.add_argument("--out", default="figures/sudoku_rsweep.json") +a = p.parse_args() + +dev = "cuda" if torch.cuda.is_available() else "cpu" +tok = SudokuTokenizer() +P = np.load(os.path.join(a.hard, "test_puz.npy")) +S = np.load(os.path.join(a.hard, "test_sol.npy")) +Pt, St = torch.from_numpy(P), torch.from_numpy(S - 1) +blank = Pt == 0 + +res = defaultdict(dict) +for d in sorted(glob.glob(os.path.join(a.runs, a.pattern))): + rf = os.path.join(d, "result.json") + if not (os.path.exists(rf) and os.path.exists(os.path.join(d, "model.pt"))): + continue + cfg = json.load(open(rf))["args"] + if not cfg.get("recurrent"): + continue + m = RecurrentDenoiser(len(tok), cfg["d"], cfg["layers"], cfg["heads"], cfg["pe"], + max_len=CELLS + 8, recurrences=cfg["R"]).to(dev) + m.load_state_dict(torch.load(os.path.join(d, "model.pt"), map_location=dev)) + m.eval() + name = os.path.basename(d) + trained_R = cfg["R"] + print(f"\n--- {name} (trained with R={trained_R})", flush=True) + print(f" {'R at inference':>16}{'board':>10}", flush=True) + for R in (1, 2, 4, 8, 16, 32, 64, 128): + m.recurrences = R + preds = [] + with torch.no_grad(): + for i in range(0, len(Pt), a.eval_bs): + gold = St[i:i + a.eval_bs].to(dev) + bl = blank[i:i + a.eval_bs].to(dev) + x = torch.where(bl, torch.full_like(gold, tok.mask), gold) + y, _ = dfn.fixed_k_decode(m, x, tok, dev, a.passes, fillable=bl) + preds.append(y.cpu().numpy() + 1) + acc = float((np.concatenate(preds) == S).all(axis=1).mean()) + res[name][R] = acc + flag = " <- trained here" if R == trained_R else "" + print(f" {R:>16}{acc*100:>9.2f}%{flag}", flush=True) + del m + torch.cuda.empty_cache() + +os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True) +json.dump(res, open(a.out, "w"), indent=2) +print(f"\nwrote {a.out}") diff --git a/src/sudoku_validate.py b/src/sudoku_validate.py new file mode 100644 index 0000000000000000000000000000000000000000..d8bc5cb280acd44b7e6a5e1676335405db093911 --- /dev/null +++ b/src/sudoku_validate.py @@ -0,0 +1,265 @@ +"""Is the Sudoku result meaningful, or is the benchmark simply saturated? + +A neural solver scoring 100% on a test split proves nothing on its own. The +questions that decide whether the number means anything: + +1. HOW HARD ARE THESE PUZZLES? If plain constraint propagation - no search, no + backtracking, the first thing anyone writes - already solves every test + puzzle, then the benchmark is trivial and 100% is the expected result for any + competent method rather than evidence of anything. This is the test that + matters most, and it is run first. + +2. IS THE MARGIN STATISTICALLY REAL? 100% against a published 98.3% on 1000 + puzzles: exact binomial, not eyeballing. + +3. DOES IT HOLD PER SEED? A mean of 100% could hide a split. + +4. DOES IT GENERALISE TO HARDER PUZZLES? SATNet's puzzles carry 31-42 clues. + Stripping clues while VERIFYING the solution stays unique gives strictly + harder instances from the same distribution, and shows whether the model + learned Sudoku or learned this clue-density. + +Every puzzle generated for (4) is checked to have exactly one solution by an +exhaustive counter, so "accuracy" remains well defined. +""" +import argparse, glob, json, os, sys +from collections import defaultdict + +import numpy as np + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from sudoku import GRID, CELLS, load_satnet, split + +PEERS = None + + +def _build_peers(): + """For each cell, the 20 cells sharing its row, column or box.""" + peers = [] + for i in range(CELLS): + r, c = divmod(i, GRID) + br, bc = 3 * (r // 3), 3 * (c // 3) + s = set() + for k in range(GRID): + s.add(r * GRID + k) + s.add(k * GRID + c) + for dr in range(3): + for dc in range(3): + s.add((br + dr) * GRID + bc + dc) + s.discard(i) + peers.append(sorted(s)) + return peers + + +def candidates(grid): + """Per-cell candidate sets as 9-bit masks; None if a contradiction appears. + + Values are forced to Python ints: numpy integer scalars lack .bit_length(), + and mixing the two silently produces numpy masks that then fail. + """ + g = [int(v) for v in grid] + cand = [0x1FF if v == 0 else (1 << (v - 1)) for v in g] + for i, v in enumerate(g): + if v: + bit = 1 << (v - 1) + for p in PEERS[i]: + if cand[p] & bit: + cand[p] &= ~bit + if cand[p] == 0: + return None + return cand + + +def propagate(grid): + """Constraint propagation only: naked singles + hidden singles to fixpoint. + + This is the baseline every Sudoku solver starts from and it uses no search. + If it solves the whole test split, the split is trivial. + """ + g = [int(v) for v in grid] + while True: + cand = candidates(g) + if cand is None: + return None + changed = False + for i in range(CELLS): + if g[i] == 0 and bin(cand[i]).count("1") == 1: + g[i] = cand[i].bit_length() + changed = True + if not changed: + # hidden singles: a digit with only one home in some unit + for unit in UNITS: + for d in range(9): + bit = 1 << d + spots = [i for i in unit if g[i] == 0 and cand[i] & bit] + if len(spots) == 1: + g[spots[0]] = d + 1 + changed = True + if not changed: + return g + + +def _units(): + us = [] + for r in range(GRID): + us.append([r * GRID + c for c in range(GRID)]) + for c in range(GRID): + us.append([r * GRID + c for r in range(GRID)]) + for br in range(0, GRID, 3): + for bc in range(0, GRID, 3): + us.append([(br + dr) * GRID + bc + dc for dr in range(3) for dc in range(3)]) + return us + + +def count_solutions(grid, limit=2): + """Exhaustive count of solutions, stopping at `limit`. Used for uniqueness.""" + g = [int(v) for v in grid] + + def rec(): + cand = candidates(g) + if cand is None: + return 0 + best, bi = 10, -1 + for i in range(CELLS): + if g[i] == 0: + n = bin(cand[i]).count("1") + if n < best: + best, bi = n, i + if n == 1: + break + if bi < 0: + return 1 + total = 0 + for d in range(9): + if cand[bi] & (1 << d): + g[bi] = d + 1 + total += rec() + g[bi] = 0 + if total >= limit: + return total + return total + + return rec() + + +def wilson(k, n, z=1.96): + if n == 0: + return 0.0, 0.0 + p = k / n + d = 1 + z * z / n + c = (p + z * z / (2 * n)) / d + h = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d + return max(0.0, c - h), min(1.0, c + h) + + +def binom_p(k, n, p0): + """One-sided exact binomial: P(X >= k) under H0: rate = p0.""" + from math import comb + return float(sum(comb(n, i) * p0 ** i * (1 - p0) ** (n - i) for i in range(k, n + 1))) + + +PEERS = _build_peers() +UNITS = _units() + +p = argparse.ArgumentParser() +p.add_argument("--root", default="data/sudoku") +p.add_argument("--runs", default="runs") +p.add_argument("--n_hard", type=int, default=200) +p.add_argument("--out", default="figures/sudoku_validate.json") +a = p.parse_args() + +X, Y, tok = load_satnet(a.root) +(Xtr, Ytr), (Xte, Yte) = split(X, Y) +out = {} + +print("=" * 78) +print("1. HOW HARD IS THIS BENCHMARK? (the test that decides if 100% means anything)") +print("=" * 78) +cp_solved = 0 +for i in range(len(Xte)): + g = propagate(Xte[i]) + if g is not None and all(v > 0 for v in g) and np.array_equal(g, Yte[i]): + cp_solved += 1 +cp_rate = cp_solved / len(Xte) +out["classical_propagation"] = cp_rate +print(f" constraint propagation alone, NO search: {cp_solved}/{len(Xte)} = {cp_rate*100:.2f}%") +print(f" clues per test puzzle: min={(Xte>0).sum(1).min()} " + f"max={(Xte>0).sum(1).max()} mean={(Xte>0).sum(1).mean():.1f}") +if cp_rate > 0.99: + print(" >>> The split is SATURATED: a search-free classical solver already") + print(" >>> solves it. 100% is the expected result for any competent method,") + print(" >>> and the neural number should NOT be presented as a capability claim.") +else: + print(f" >>> Not saturated: propagation alone leaves {(1-cp_rate)*100:.1f}% unsolved,") + print(" >>> so the benchmark does discriminate between methods.") + +print() +print("=" * 78) +print("2. IS THE MARGIN STATISTICALLY REAL?") +print("=" * 78) +runs = defaultdict(list) +for f in sorted(glob.glob(os.path.join(a.runs, "sud1-*/result.json"))): + r = json.load(open(f)) + arg = r["args"] + runs[(arg["mode"], "aug" if arg.get("augment") else "plain")].append(r) + +for key in sorted(runs): + rs = runs[key] + best = max((max(r["decode"].values(), key=lambda v: v["board"]) for r in rs), + key=lambda v: v["board"]) + per_seed = [max(r["decode"].values(), key=lambda v: v["board"])["board"] for r in rs] + k = int(round(np.mean(per_seed) * len(Xte))) + lo, hi = wilson(k, len(Xte)) + pv = binom_p(k, len(Xte), 0.983) + print(f" {key[0]}/{key[1]:<6} best {np.mean(per_seed)*100:6.2f}% " + f"95% CI [{lo*100:.2f}, {hi*100:.2f}] per-seed {[round(x*100,2) for x in per_seed]}" + f" p(vs 98.3%)={pv:.2g}") + out[f"{key[0]}/{key[1]}"] = {"mean": float(np.mean(per_seed)), + "per_seed": per_seed, "ci": [lo, hi], "p_vs_satnet": pv} + +print() +print("=" * 78) +print(f"3. DOES IT GENERALISE TO HARDER PUZZLES? (stripping clues, uniqueness verified)") +print("=" * 78) +rng = np.random.default_rng(0) +hard = {} +for target in (30, 26, 22): + made, tries = [], 0 + for idx in range(len(Xte)): + if len(made) >= a.n_hard or tries > a.n_hard * 12: + break + tries += 1 + sol = Yte[idx] + cells = list(rng.permutation(CELLS)) + g = list(sol) + removed = 0 + for c in cells: + if CELLS - removed <= target: + break + keep = g[c] + g[c] = 0 + if count_solutions(g, 2) == 1: + removed += 1 + else: + g[c] = keep + if sum(1 for v in g if v) <= target + 2: + made.append((np.array(g), sol)) + if made: + P = np.stack([m[0] for m in made]) + S = np.stack([m[1] for m in made]) + cp = sum(1 for i in range(len(P)) + if (lambda r: r is not None and all(v > 0 for v in r) + and np.array_equal(r, S[i]))(propagate(P[i]))) + hard[target] = {"n": len(P), "clues": float((P > 0).sum(1).mean()), + "classical_propagation": cp / len(P)} + print(f" target ~{target} clues: built {len(P)} unique puzzles " + f"(mean {(P>0).sum(1).mean():.1f} clues) " + f"constraint propagation solves {cp/len(P)*100:.1f}%") + np.save(os.path.join(os.path.dirname(a.out) or ".", f"hard_{target}_puz.npy"), P) + np.save(os.path.join(os.path.dirname(a.out) or ".", f"hard_{target}_sol.npy"), S) +out["harder"] = hard + +os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True) +json.dump(out, open(a.out, "w"), indent=2, default=float) +print(f"\nwrote {a.out}") +print("harder puzzle sets saved next to it for the model to be evaluated on") diff --git a/src/train_sudoku.py b/src/train_sudoku.py index 041a910e55696bfbc510b4bcac009c108c2363c1..75b83bfe95f74012fe883a302768711a637bc89b 100644 --- a/src/train_sudoku.py +++ b/src/train_sudoku.py @@ -21,7 +21,7 @@ import argparse, json, math, os, time import numpy as np import torch -from model import Transformer +from model import RecurrentDenoiser, Transformer import diffusion as dfn from sudoku import (CELLS, SudokuTokenizer, board_accuracy, cell_accuracy, consistent_with_clues, is_valid_solution, load_satnet, split) @@ -30,10 +30,35 @@ from sudoku import (CELLS, SudokuTokenizer, board_accuracy, cell_accuracy, def get_args(): p = argparse.ArgumentParser() p.add_argument("--mode", choices=["mdlm", "matched", "full"], default="mdlm") + p.add_argument("--all_cells", action="store_true", + help="direct objective: supervise all 81 cells, not only the " + "blanks. Denser signal - the model must also reproduce " + "the given clues - which is how the published recurrent " + "classifier is trained and the most likely reason our " + "first attempt plateaued at loss 0.34 against 0.15") + p.add_argument("--objective", choices=["diffusion", "direct", "ar"], + default="diffusion", + help="diffusion = mask at a random ratio and denoise; " + "direct = predict every blank from the puzzle with no " + "masking schedule, which is the Recurrent Transformer " + "setup and the baseline that asks whether the diffusion " + "framing earns its place; " + "ar = autoregressive, blanks filled left to right") p.add_argument("--K", type=int, default=8) p.add_argument("--lam", type=float, default=1.0) p.add_argument("--pe", default="ape", choices=["ape", "rope", "sin"]) + p.add_argument("--recurrent", action="store_true", + help="weight-shared block applied R times with deep supervision") + p.add_argument("--R", type=int, default=32, help="training recurrences") + p.add_argument("--no_deep_sup", action="store_true", + help="ablation: supervise only the final recurrence") + p.add_argument("--R_infer", type=int, default=0, + help="inference recurrences (0 = same as training)") p.add_argument("--root", default="data/sudoku") + p.add_argument("--hard", default="", + help="directory of the generated HARD benchmark (npy files); " + "overrides --root. Minimal puzzles at ~24 clues, where " + "constraint propagation alone solves about 4%") p.add_argument("--augment", action="store_true", help="relabel digits by a random permutation (a Sudoku symmetry)") p.add_argument("--d", type=int, default=512) @@ -45,6 +70,8 @@ def get_args(): p.add_argument("--warmup", type=int, default=2000) p.add_argument("--seed", type=int, default=0) p.add_argument("--eval_bs", type=int, default=200) + p.add_argument("--ckpt_every", type=int, default=4000, + help="steps between checkpoints; 0 disables") p.add_argument("--out", default="runs/sud") return p.parse_args() @@ -54,10 +81,22 @@ torch.manual_seed(a.seed); np.random.seed(a.seed) dev = "cuda" if torch.cuda.is_available() else "cpu" os.makedirs(a.out, exist_ok=True) -X, Y, tok = load_satnet(a.root) -(Xtr, Ytr), (Xte, Yte) = split(X, Y) -print(f"sudoku: train {Xtr.shape} test {Xte.shape} vocab {len(tok)} " - f"clues/puzzle mean {(X>0).sum(1).mean():.1f}", flush=True) +if a.hard: + # the generated benchmark: minimal puzzles, uniqueness verified per removal, + # train/test solutions disjoint even up to digit relabelling + Xtr = np.load(os.path.join(a.hard, "train_puz.npy")) + Ytr = np.load(os.path.join(a.hard, "train_sol.npy")) + Xte = np.load(os.path.join(a.hard, "test_puz.npy")) + Yte = np.load(os.path.join(a.hard, "test_sol.npy")) + tok = SudokuTokenizer() + print(f"HARD sudoku: train {Xtr.shape} test {Xte.shape} " + f"clues mean {(Xtr>0).sum(1).mean():.1f} " + f"(min {(Xtr>0).sum(1).min()} max {(Xtr>0).sum(1).max()})", flush=True) +else: + X, Y, tok = load_satnet(a.root) + (Xtr, Ytr), (Xte, Yte) = split(X, Y) + print(f"sudoku: train {Xtr.shape} test {Xte.shape} vocab {len(tok)} " + f"clues/puzzle mean {(X>0).sum(1).mean():.1f}", flush=True) # tokens are the SOLUTION digits shifted to 0-8; blanks are what gets masked sol_tr = torch.from_numpy(Ytr - 1) @@ -65,18 +104,54 @@ blank_tr = torch.from_numpy(Xtr == 0) sol_te = torch.from_numpy(Yte - 1) blank_te = torch.from_numpy(Xte == 0) -model = Transformer(len(tok), a.d, a.layers, a.heads, a.pe, causal=False, - max_len=CELLS + 8).to(dev) -print(f"params {model.n_params()/1e6:.2f}M mode={a.mode} pe={a.pe}", flush=True) +if a.recurrent: + model = RecurrentDenoiser(len(tok), a.d, a.layers, a.heads, a.pe, + max_len=CELLS + 8, recurrences=a.R).to(dev) + print(f"params {model.n_params()/1e3:.0f}k RECURRENT R={a.R} " + f"layers={a.layers} d={a.d} mode={a.mode} pe={a.pe}", flush=True) +else: + model = Transformer(len(tok), a.d, a.layers, a.heads, a.pe, + causal=(a.objective == "ar"), + max_len=(2 * CELLS + 8 if a.objective == "ar" + else CELLS + 8)).to(dev) + print(f"params {model.n_params()/1e6:.2f}M mode={a.mode} pe={a.pe}", flush=True) opt = torch.optim.AdamW(model.parameters(), lr=a.lr, weight_decay=0.01) sched = torch.optim.lr_scheduler.LambdaLR( opt, lambda s: min(1.0, (s + 1) / a.warmup) * 0.5 * (1 + math.cos(math.pi * min(1.0, s / a.steps)))) +# ---- checkpoint / resume --------------------------------------------------- +# The class allocation bills an H100 at 144/min against 9/min for a 5GB MIG +# slice, so long runs belong on the cheap partition - which has shorter time +# limits. Resuming from a checkpoint is what makes a long run survive being +# split across several short jobs, and it costs no extra GPU-time: the +# allocation charges GPU-seconds consumed, not jobs submitted. +CKPT = os.path.join(a.out, "ckpt.pt") +start_step = 0 +if os.path.exists(CKPT): + st = torch.load(CKPT, map_location=dev) + model.load_state_dict(st["model"]) + opt.load_state_dict(st["opt"]) + sched.load_state_dict(st["sched"]) + start_step = st["step"] + torch.set_rng_state(st["torch_rng"].cpu()) + np.random.set_state(st["np_rng"]) + print(f"resumed from {CKPT} at step {start_step}", flush=True) + + +def save_ckpt(step): + tmp = CKPT + ".tmp" + torch.save({"model": model.state_dict(), "opt": opt.state_dict(), + "sched": sched.state_dict(), "step": step, + "torch_rng": torch.get_rng_state(), + "np_rng": np.random.get_state()}, tmp) + os.replace(tmp, CKPT) # atomic: a job killed mid-write leaves the old one + + rng = np.random.default_rng(a.seed) t0 = time.time() -for step in range(a.steps): +for step in range(start_step, a.steps): idx = torch.randint(0, len(sol_tr), (a.bs,)) xb, ab = sol_tr[idx].clone(), blank_tr[idx] if a.augment: @@ -87,14 +162,66 @@ for step in range(a.steps): xb, ab = xb.to(dev), ab.to(dev) opt.zero_grad(set_to_none=True) with torch.autocast("cuda", dtype=torch.bfloat16, enabled=(dev == "cuda")): - loss = dfn.pmd_loss(model, xb, ab, tok, a.mode, a.K, a.lam) + if a.objective == "direct": + # No masking schedule at all: the model always sees the real puzzle + # and predicts every blank. This is what a recurrent classifier does, + # and it is the control for whether diffusion adds anything. + xin = torch.where(ab, torch.full_like(xb, tok.mask), xb) + sup = torch.ones_like(ab) if a.all_cells else ab + if a.recurrent: + outs = model(xin, recurrences=None, return_all=True) + if a.no_deep_sup: + outs = outs[-1:] + R = len(outs) + loss, wsum = 0.0, 0.0 + for r, lg in enumerate(outs): + w = (r + 1) / R + ce = torch.nn.functional.cross_entropy( + lg.reshape(-1, lg.size(-1)), xb.reshape(-1), + reduction="none").view(xb.shape) + loss = loss + w * ((ce * sup).sum(1) / sup.sum(1).clamp_min(1)).mean() + wsum += w + loss = loss / wsum + else: + lg = model(xin) + ce = torch.nn.functional.cross_entropy( + lg.reshape(-1, lg.size(-1)), xb.reshape(-1), + reduction="none").view(xb.shape) + loss = ((ce * sup).sum(1) / sup.sum(1).clamp_min(1)).mean() + elif a.objective == "ar": + # Prefix-LM: [puzzle 81 tokens][solution 81 tokens], causal, loss on + # the solution half only. Sudoku clues are scattered, so an in-place + # causal model cannot see clues to its right; giving it the whole + # puzzle as a prefix is the setup that actually makes AR competitive. + puz = torch.where(ab, torch.full_like(xb, tok.mask), xb) + seq = torch.cat([puz, xb], dim=1) # (B, 162) + lg = model(seq) + tgt = seq[:, 1:] + lgt = lg[:, :-1] + sup = torch.zeros_like(tgt, dtype=torch.bool) + sup[:, CELLS - 1:] = True # solution half only + ce = torch.nn.functional.cross_entropy( + lgt.reshape(-1, lg.size(-1)), tgt.reshape(-1), + reduction="none").view(tgt.shape) + loss = ((ce * sup).sum(1) / sup.sum(1)).mean() + elif a.recurrent: + loss = dfn.recurrent_cond_loss(model, xb, ab, tok, a.mode, a.K, a.lam, + deep_sup=not a.no_deep_sup) + else: + loss = dfn.pmd_loss(model, xb, ab, tok, a.mode, a.K, a.lam) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step(); sched.step() if step % 5000 == 0: print(f"step {step:6d} loss {loss.item():.4f} ({time.time()-t0:.0f}s)", flush=True) + if a.ckpt_every and step and step % a.ckpt_every == 0: + save_ckpt(step) +save_ckpt(a.steps) model.eval() +if a.recurrent and a.R_infer: + model.recurrences = a.R_infer + print(f"inference recurrences set to {a.R_infer} (trained with {a.R})", flush=True) res = {"args": vars(a), "decode": {}} @@ -124,11 +251,28 @@ def report(tag, key, r): f"{r['valid_sudoku']*100:>9.2f}%", flush=True) +if a.objective == "ar": + @torch.no_grad() + def ar_decode(x, bl): + """Generate the solution half token by token, conditioned on the puzzle.""" + puz = x # already puzzle-with-masks + seq = torch.cat([puz, torch.full_like(puz, tok.mask)], dim=1) + for j in range(CELLS): + pos = CELLS + j + lg = model(seq[:, :pos])[:, -1] + seq[:, pos] = lg.argmax(-1) + return seq[:, CELLS:], CELLS + report("autoregressive (L->R)", "ar", evaluate(ar_decode)) + torch.save(model.state_dict(), os.path.join(a.out, "model.pt")) + json.dump(res, open(os.path.join(a.out, "result.json"), "w"), indent=2) + print(f"saved to {a.out}", flush=True) + raise SystemExit + print("\n=== board accuracy against decoding budget ===", flush=True) print(f" {'rule':<26}{'NFE':>7}{'BOARD':>10}{'cell':>9}{'valid':>9}", flush=True) print("fixed-K (commits a fixed share per pass, ignoring certainty):", flush=True) -for K in (1, 2, 4, 8, 16, 32, 45): +for K in (1, 2, 4, 8, 16, 32, 61): report(f"fixed-K K={K}", f"fixedK_{K}", evaluate(lambda x, bl, K=K: dfn.fixed_k_decode( model, x, tok, dev, K, fillable=bl)))