Document the from-scratch constraint in checkable form
Browse files
REPORT.md
CHANGED
|
@@ -8,7 +8,9 @@ Sudoku benchmark is **saturated** and replaces it, measures **classical solvers*
|
|
| 8 |
that prior work omits, and reports a **negative result**: the one architectural
|
| 9 |
component we added does not help once R-MDM's own conditioning is present.**
|
| 10 |
|
| 11 |
-
Trained from scratch
|
|
|
|
|
|
|
| 12 |
|
| 13 |
> ### Relation to prior work — read this first
|
| 14 |
>
|
|
@@ -382,6 +384,28 @@ not used because constraint propagation alone solves 100% of it.
|
|
| 382 |
- Sudoku only. Whether recurrence transfers to other constraint families is
|
| 383 |
untested here.
|
| 384 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 385 |
## Datasets
|
| 386 |
|
| 387 |
**Primary — generated, not downloaded.** The standard split is saturated, so it
|
|
@@ -451,6 +475,27 @@ python src/sudoku_compare.py --hard data/sudoku_hard --runs runs --pattern 'rec*
|
|
| 451 |
python src/plot_sudoku.py
|
| 452 |
```
|
| 453 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 454 |
---
|
| 455 |
|
| 456 |
## The earlier study: two failure modes of parallel decoding
|
|
|
|
| 8 |
that prior work omits, and reports a **negative result**: the one architectural
|
| 9 |
component we added does not help once R-MDM's own conditioning is present.**
|
| 10 |
|
| 11 |
+
Trained from scratch: every weight randomly initialised, every tokenizer
|
| 12 |
+
hand-written, the primary benchmark generated rather than downloaded.
|
| 13 |
+
[Verify it in under a minute](#built-from-scratch--how-to-verify-it-in-under-a-minute).
|
| 14 |
|
| 15 |
> ### Relation to prior work — read this first
|
| 16 |
>
|
|
|
|
| 384 |
- Sudoku only. Whether recurrence transfers to other constraint families is
|
| 385 |
untested here.
|
| 386 |
|
| 387 |
+
## Built from scratch — how to verify it in under a minute
|
| 388 |
+
|
| 389 |
+
Every model in this repository is randomly initialised and trained from zero.
|
| 390 |
+
No pretrained weights, no pretrained tokenizer, no pretrained embeddings, no
|
| 391 |
+
distillation from a pretrained teacher, at any stage. This is a hard constraint
|
| 392 |
+
on the project, so it is stated here in checkable form rather than as an
|
| 393 |
+
assurance.
|
| 394 |
+
|
| 395 |
+
| what | where | how to check |
|
| 396 |
+
|---|---|---|
|
| 397 |
+
| **Weights** are random, always | `src/model.py` — `self.apply(self._init)` | every `nn.Linear`/`nn.Embedding` gets `N(0, 0.02²)`; there is no other initialisation path |
|
| 398 |
+
| **No pretrained loader exists** | whole tree | `grep -rn "from_pretrained\|AutoModel\|AutoTokenizer\|torch.hub\|timm" src/` returns **nothing** |
|
| 399 |
+
| **Tokenizers are hand-written**, 7 of them | `src/{sudoku,text8,data,graphs,qm9,sigmorphon}.py` | `grep -n "class .*Tokenizer" src/*.py` — each is a literal symbol list built in-repo |
|
| 400 |
+
| **The Sudoku vocabulary is 10 symbols** | `src/sudoku.py:32` | digits `1`–`9` plus `[MASK]`; no vendored vocab file |
|
| 401 |
+
| **Downloads are data, never weights** | `src/{data,text8,qm9,sigmorphon,sudoku}.py` | the only URLs are the text8 corpus, the QM9 CSV, SIGMORPHON TSVs and the SATNet zip — raw datasets |
|
| 402 |
+
| **The primary benchmark is generated** | `src/gen_hard_sudoku.py` | nothing is downloaded at all; puzzles come from a randomised backtracking fill |
|
| 403 |
+
| **Even the evaluator is from scratch** | `src/train_eval_lm.py` | generative perplexity in the earlier study is scored by a character LM we trained on text8 ourselves, not GPT-2 |
|
| 404 |
+
|
| 405 |
+
The largest model here is 37.9M parameters and the headline model is **212k** —
|
| 406 |
+
small enough that from-scratch training is the only sensible choice, and small
|
| 407 |
+
enough to retrain end to end on one MIG slice of an A100.
|
| 408 |
+
|
| 409 |
## Datasets
|
| 410 |
|
| 411 |
**Primary — generated, not downloaded.** The standard split is saturated, so it
|
|
|
|
| 475 |
python src/plot_sudoku.py
|
| 476 |
```
|
| 477 |
|
| 478 |
+
**The matched R-MDM comparison of §2b.** Three arms differing in one component,
|
| 479 |
+
with their step embedding on both sides. `--step_embed` is R-MDM's per-loop
|
| 480 |
+
conditioning `f(h, â„“, L)`; `--no_inject` turns off the component we tested:
|
| 481 |
+
|
| 482 |
+
```bash
|
| 483 |
+
for cfg in "--step_embed --no_inject --seed 0:rmdm-faithful-s0" \
|
| 484 |
+
"--step_embed --no_inject --seed 1:rmdm-faithful-s1" \
|
| 485 |
+
"--step_embed --seed 0:rmdm-both-s0"; do
|
| 486 |
+
python src/train_sudoku.py ${cfg%%:*} --recurrent --R 32 --R_infer 64 \
|
| 487 |
+
--mode full --hard data/sudoku_hard --augment --d 128 --layers 1 --heads 4 \
|
| 488 |
+
--bs 64 --steps 110000 --lr 3e-4 --ckpt_every 8000 --out runs/${cfg##*:}
|
| 489 |
+
done
|
| 490 |
+
python src/eval_ckpt.py --hard data/sudoku_hard --runs runs # reads ckpt.pt
|
| 491 |
+
python src/plot_rmdm.py # -> sud_fig9_rmdm.png
|
| 492 |
+
```
|
| 493 |
+
|
| 494 |
+
`eval_ckpt.py` evaluates from `ckpt.pt` rather than `model.pt` because all three
|
| 495 |
+
of our runs hit the wall clock at step 80,000 and never reached the trainer's
|
| 496 |
+
evaluation block. **Keep the batch size identical across arms** — the retracted
|
| 497 |
+
+5.60 came from comparing arms trained at batch 256 and batch 64 (§4).
|
| 498 |
+
|
| 499 |
---
|
| 500 |
|
| 501 |
## The earlier study: two failure modes of parallel decoding
|