GOVINDFROM commited on
Commit
ec70788
·
verified ·
1 Parent(s): 831c9dd

Document the from-scratch constraint in checkable form

Browse files
Files changed (1) hide show
  1. REPORT.md +46 -1
REPORT.md CHANGED
@@ -8,7 +8,9 @@ Sudoku benchmark is **saturated** and replaces it, measures **classical solvers*
8
  that prior work omits, and reports a **negative result**: the one architectural
9
  component we added does not help once R-MDM's own conditioning is present.**
10
 
11
- Trained from scratch. No pretrained weights, no pretrained tokenizer.
 
 
12
 
13
  > ### Relation to prior work — read this first
14
  >
@@ -382,6 +384,28 @@ not used because constraint propagation alone solves 100% of it.
382
  - Sudoku only. Whether recurrence transfers to other constraint families is
383
  untested here.
384
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
385
  ## Datasets
386
 
387
  **Primary — generated, not downloaded.** The standard split is saturated, so it
@@ -451,6 +475,27 @@ python src/sudoku_compare.py --hard data/sudoku_hard --runs runs --pattern 'rec*
451
  python src/plot_sudoku.py
452
  ```
453
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
454
  ---
455
 
456
  ## The earlier study: two failure modes of parallel decoding
 
8
  that prior work omits, and reports a **negative result**: the one architectural
9
  component we added does not help once R-MDM's own conditioning is present.**
10
 
11
+ Trained from scratch: every weight randomly initialised, every tokenizer
12
+ hand-written, the primary benchmark generated rather than downloaded.
13
+ [Verify it in under a minute](#built-from-scratch--how-to-verify-it-in-under-a-minute).
14
 
15
  > ### Relation to prior work — read this first
16
  >
 
384
  - Sudoku only. Whether recurrence transfers to other constraint families is
385
  untested here.
386
 
387
+ ## Built from scratch — how to verify it in under a minute
388
+
389
+ Every model in this repository is randomly initialised and trained from zero.
390
+ No pretrained weights, no pretrained tokenizer, no pretrained embeddings, no
391
+ distillation from a pretrained teacher, at any stage. This is a hard constraint
392
+ on the project, so it is stated here in checkable form rather than as an
393
+ assurance.
394
+
395
+ | what | where | how to check |
396
+ |---|---|---|
397
+ | **Weights** are random, always | `src/model.py` — `self.apply(self._init)` | every `nn.Linear`/`nn.Embedding` gets `N(0, 0.02²)`; there is no other initialisation path |
398
+ | **No pretrained loader exists** | whole tree | `grep -rn "from_pretrained\|AutoModel\|AutoTokenizer\|torch.hub\|timm" src/` returns **nothing** |
399
+ | **Tokenizers are hand-written**, 7 of them | `src/{sudoku,text8,data,graphs,qm9,sigmorphon}.py` | `grep -n "class .*Tokenizer" src/*.py` — each is a literal symbol list built in-repo |
400
+ | **The Sudoku vocabulary is 10 symbols** | `src/sudoku.py:32` | digits `1`–`9` plus `[MASK]`; no vendored vocab file |
401
+ | **Downloads are data, never weights** | `src/{data,text8,qm9,sigmorphon,sudoku}.py` | the only URLs are the text8 corpus, the QM9 CSV, SIGMORPHON TSVs and the SATNet zip — raw datasets |
402
+ | **The primary benchmark is generated** | `src/gen_hard_sudoku.py` | nothing is downloaded at all; puzzles come from a randomised backtracking fill |
403
+ | **Even the evaluator is from scratch** | `src/train_eval_lm.py` | generative perplexity in the earlier study is scored by a character LM we trained on text8 ourselves, not GPT-2 |
404
+
405
+ The largest model here is 37.9M parameters and the headline model is **212k** —
406
+ small enough that from-scratch training is the only sensible choice, and small
407
+ enough to retrain end to end on one MIG slice of an A100.
408
+
409
  ## Datasets
410
 
411
  **Primary — generated, not downloaded.** The standard split is saturated, so it
 
475
  python src/plot_sudoku.py
476
  ```
477
 
478
+ **The matched R-MDM comparison of §2b.** Three arms differing in one component,
479
+ with their step embedding on both sides. `--step_embed` is R-MDM's per-loop
480
+ conditioning `f(h, â„“, L)`; `--no_inject` turns off the component we tested:
481
+
482
+ ```bash
483
+ for cfg in "--step_embed --no_inject --seed 0:rmdm-faithful-s0" \
484
+ "--step_embed --no_inject --seed 1:rmdm-faithful-s1" \
485
+ "--step_embed --seed 0:rmdm-both-s0"; do
486
+ python src/train_sudoku.py ${cfg%%:*} --recurrent --R 32 --R_infer 64 \
487
+ --mode full --hard data/sudoku_hard --augment --d 128 --layers 1 --heads 4 \
488
+ --bs 64 --steps 110000 --lr 3e-4 --ckpt_every 8000 --out runs/${cfg##*:}
489
+ done
490
+ python src/eval_ckpt.py --hard data/sudoku_hard --runs runs # reads ckpt.pt
491
+ python src/plot_rmdm.py # -> sud_fig9_rmdm.png
492
+ ```
493
+
494
+ `eval_ckpt.py` evaluates from `ckpt.pt` rather than `model.pt` because all three
495
+ of our runs hit the wall clock at step 80,000 and never reached the trainer's
496
+ evaluation block. **Keep the batch size identical across arms** — the retracted
497
+ +5.60 came from comparing arms trained at batch 256 and batch 64 (§4).
498
+
499
  ---
500
 
501
  ## The earlier study: two failure modes of parallel decoding