Automated push from Zaratan

README.md

@@ -5,48 +5,13 @@ tags: [masked-diffusion, discrete-diffusion, length-generalization, positional-e
 
 # Length Generalization in Masked Diffusion Language Models
 
-From-scratch masked diffusion language models (MDLM-style absorbing-state
-diffusion) and matched autoregressive baselines, trained on multi-digit addition
-and evaluated on operand lengths never seen in training.
+From-scratch masked diffusion language models (MDLM-style absorbing state) and
+matched autoregressive baselines, trained on arithmetic and algorithmic tasks and
+evaluated far beyond the lengths seen in training.
 
-**No pretrained weights or tokenizers are used anywhere.** Every model is trained
-from random initialization with a symbol-level vocabulary built from the data.
+**No pretrained weights or tokenizers are used anywhere.**
 
-## What is here
-
-- `figures/` — plots and the summary table
-- `results/` — one JSON per run (config + accuracy on every test length)
-- `weights/` — checkpoints (~10.7M parameters each)
-- `src/` — full training, evaluation and plotting code
-
-## Method
-
-**Significance-aligned position ids.** Instead of numbering tokens by sequence
-index, every digit is numbered by its place value, so digits that must be
-combined share an id:
-
-```
- 4   7   +   8   5   =   1   3   2
- 2   1   0   2   1   0   3   2   1
-```
-
-The rule "combine equal ids, carry into id+1" does not depend on operand length,
-which is what allows extrapolation. A random per-example offset is added to all
-ids so the model keys on relative place value and encounters large ids during
-training.
-
-## Findings
-
-1. **Positional encodings do not transfer between the two architectures.**
-   Removing positional information entirely, and sinusoidal encoding, both give
-   100% in-distribution accuracy for autoregressive models and **0% for masked
-   diffusion** — the diffusion model cannot learn the task at all (3 seeds each).
-   A diffusion model attends bidirectionally and so has no implicit order to
-   fall back on.
-
-2. **Length-generalization accuracy has very high seed variance** on this task
-   (individual seeds span 3%–68% for one configuration), so single-seed
-   comparisons in this setting are unreliable.
+Code: https://github.com/GIND123/MSML612
 
 ## Results
 
@@ -114,12 +79,3 @@ Every apparent failure at 10+ digits was a learning-rate artifact, not a limit o
 | 10 | 0.0003 | 4000 | 100.0 | 3/3 |
 
 
-
-## Setup
-
-6 layers, 384 hidden dimension, 6 heads, ~10.7M parameters. AdamW, learning rate
-1e-4, cosine schedule, bfloat16, effective batch 256. Trained on operands of 1-5
-digits; evaluated at 5, 6, 7, 8, 10 and 12 digits. Exact-match accuracy — the
-entire answer must be correct.
-
-Trained on the University of Maryland Zaratan cluster (A100 MIG slices).

slurm/auditjob.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+#SBATCH -J audit
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=48g
+#SBATCH -t 02:00:00
+#SBATCH -o logs/audit-%j.out
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+cd $SCR/pcmd/src
+echo "########## AUDIT ##########"
+RUNS=$SCR/pcmd/runs SIGROOT=$SCR/pcmd/data/sig $SCR/hf_env/bin/python audit.py
+echo "########## HEAD-TO-HEAD vs INFERENCE-TIME METHODS ##########"
+for D in 12 16; do
+  $SCR/hf_env/bin/python beat_baselines.py --runs $SCR/pcmd/runs --digits $D \
+    --out $SCR/pcmd/figures/beat_d$D.json
+done

slurm/g10.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+# Grid 10 - is depth failing for OPTIMISATION reasons? This must be settled
+# before the depth-scaling test means anything.
+#
+# At 8 digits a 6-layer/384 model reached 99.9%, while 10-layer/512 reached
+# 66.7% with more training. Deeper transformers usually need a lower learning
+# rate and longer warmup; if that is the cause, the depth sweep at a single LR
+# would fake a "depth does not help" result.
+#SBATCH -J g10
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=64g
+#SBATCH -t 10:00:00
+#SBATCH -o logs/g10-%A_%a.out
+#SBATCH --array=0-35%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+PY=$SCR/hf_env/bin/python
+cd $SCR/pcmd/src
+
+I=$SLURM_ARRAY_TASK_ID
+LAYERS=(6 10); LR=(3e-4 1e-4 5e-5); WARM=(500 4000); SEEDS=(0 1 2)
+L=${LAYERS[$((I / 18))]}
+R=${LR[$(((I % 18) / 6))]}
+W=${WARM[$(((I % 6) / 3))]}
+S=${SEEDS[$((I % 3))]}
+
+NAME="g10-L$L-lr$R-w$W-s$S"
+echo "=== $NAME ==="
+$PY train_add.py --mode full --digits 10 --seq_len 34 --seed $S \
+  --n_train 500000 --n_eval 400 --eval_bs 100 \
+  --d 384 --layers $L --heads 6 --bs 128 --steps 40000 --lr $R --warmup $W \
+  --K 4 --lam 1.0 --out $SCR/pcmd/runs/$NAME

slurm/g11.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+# Grid 11 - THE HEADLINE EXPERIMENT, with hyperparameters that are not broken.
+#
+# Grid 10 showed every "the method fails at 10+ digits" result was a learning
+# rate artifact: at lr 3e-4 with 500 warmup a 10-layer model reached 47%, while
+# at lr 1e-4 the same setting reached 100% on 3/3 seeds. Depth was never the
+# problem. This reruns the full difficulty sweep at lr 1e-4 / warmup 4000, with
+# depth scaled as log2(digits) per THEORY.md.
+#SBATCH -J g11
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=64g
+#SBATCH -t 12:00:00
+#SBATCH -o logs/g11-%A_%a.out
+#SBATCH --array=0-39%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+PY=$SCR/hf_env/bin/python
+cd $SCR/pcmd/src
+
+I=$SLURM_ARRAY_TASK_ID
+DIG=(8 12 16 24 32); MODES=(mdlm full); SEEDS=(0 1 2 3)
+D=${DIG[$((I / 8))]}
+M=${MODES[$(((I % 8) / 4))]}
+S=${SEEDS[$((I % 4))]}
+SL=$(( 3 * D + 4 ))
+# depth ~ 2*log2(n), generous but still logarithmic
+case $D in
+  8)  L=8  ;;
+  12) L=8  ;;
+  16) L=10 ;;
+  24) L=10 ;;
+  32) L=12 ;;
+esac
+
+NAME="g11-d$D-$M-s$S"
+echo "=== $NAME (layers=$L) ==="
+$PY train_add.py --mode $M --digits $D --seq_len $SL --seed $S \
+  --n_train 800000 --n_eval 400 --eval_bs 100 \
+  --d 384 --layers $L --heads 6 --bs 128 --steps 60000 --lr 1e-4 --warmup 4000 \
+  --K 4 --lam 1.0 --out $SCR/pcmd/runs/$NAME

slurm/g12.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+# Grid 12 - inflection across typologically diverse languages.
+# Reported as the BOUNDARY: answers are unique but shallow, so one pass already
+# works and no method is needed. Breadth here is across languages, not seeds.
+#SBATCH -J g12
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=32g
+#SBATCH -t 06:00:00
+#SBATCH -o logs/g12-%A_%a.out
+#SBATCH --array=0-23%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+cd $SCR/pcmd/src
+LANGS=(english german spanish turkish finnish russian navajo georgian)
+MODES=(mdlm full); SEEDS=(0)
+I=$SLURM_ARRAY_TASK_ID
+L=${LANGS[$((I / 3))]}
+M=${MODES[$(((I % 3) / 2))]}
+NAME="g12-$L-$M"
+echo "=== $NAME ==="
+$SCR/hf_env/bin/python train_sig.py --lang $L --mode $M --size medium --seed 0 \
+  --steps 15000 --d 256 --layers 6 --lr 1e-4 --warmup 1000 \
+  --data_root $SCR/pcmd/data/sig --out $SCR/pcmd/runs/$NAME

slurm/g13.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+# Grid 13 - push the frontier past the 24-digit wall.
+# At 24+ digits both methods score 0 even at sequential decoding, so the task
+# is simply not learned. Per THEORY.md depth should only need to grow with
+# log2(n), so this raises depth modestly and training substantially.
+#SBATCH -J g13
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=64g
+#SBATCH -t 16:00:00
+#SBATCH -o logs/g13-%A_%a.out
+#SBATCH --array=0-11%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+cd $SCR/pcmd/src
+DIG=(20 24); MODES=(mdlm full); SEEDS=(0 1 2)
+I=$SLURM_ARRAY_TASK_ID
+D=${DIG[$((I / 6))]}; M=${MODES[$(((I % 6) / 3))]}; S=${SEEDS[$((I % 3))]}
+SL=$(( 3 * D + 4 ))
+NAME="g13-d$D-$M-s$S"
+echo "=== $NAME ==="
+$SCR/hf_env/bin/python train_add.py --mode $M --digits $D --seq_len $SL --seed $S \
+  --n_train 1500000 --n_eval 400 --eval_bs 100 \
+  --d 512 --layers 12 --heads 8 --bs 128 --steps 150000 --lr 1e-4 --warmup 6000 \
+  --K 4 --lam 1.0 --out $SCR/pcmd/runs/$NAME

slurm/g6.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+# Grid 6 - push past saturation. 4 and 6 digits are 100% for every method, so
+# the effect only shows once the carry chain exceeds what one forward pass can
+# compute. This traces the frontier and checks the gap widens with depth.
+#SBATCH -J g6
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=64g
+#SBATCH -t 08:00:00
+#SBATCH -o logs/g6-%A_%a.out
+#SBATCH --array=0-47%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+PY=$SCR/hf_env/bin/python
+cd $SCR/pcmd/src
+
+I=$SLURM_ARRAY_TASK_ID
+MODES=(mdlm matched distill full); DIG=(10 12 14 16); SEEDS=(0 1 2)
+M=${MODES[$((I / 12))]}
+D=${DIG[$(((I % 12) / 3))]}
+S=${SEEDS[$((I % 3))]}
+SL=$(( 3 * D + 4 ))
+
+NAME="g6-d$D-$M-s$S"
+echo "=== $NAME ==="
+$PY train_add.py --mode $M --digits $D --seq_len $SL --seed $S \
+  --n_train 300000 --n_eval 500 --eval_bs 100 \
+  --d 384 --layers 6 --heads 6 --bs 128 --steps 30000 --lr 3e-4 \
+  --K 4 --lam 1.0 --out $SCR/pcmd/runs/$NAME

slurm/g7.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# Grid 7 - LEARNABILITY GATE for long addition, before any parallelism claim.
+#
+# At 14-16 digits every method scored 0 even at FULLY SEQUENTIAL decoding, so
+# the models had not learned addition at all. Any statement about parallel
+# decoding there would be vacuous. This sweeps capacity and training length to
+# find where the task becomes learnable; only then does measuring parallelism
+# mean anything.
+#SBATCH -J g7
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=64g
+#SBATCH -t 10:00:00
+#SBATCH -o logs/g7-%A_%a.out
+#SBATCH --array=0-23%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+PY=$SCR/hf_env/bin/python
+cd $SCR/pcmd/src
+
+I=$SLURM_ARRAY_TASK_ID
+# capacity x training length, on the two settings that failed outright
+CFG=(
+  "12 8 512 60000" "12 12 512 60000" "12 8 512 120000"
+  "16 8 512 60000" "16 12 512 60000" "16 12 768 120000"
+)
+MODES=(mdlm full)
+C=${CFG[$((I / 4))]}
+M=${MODES[$(((I % 4) / 2))]}
+S=$((I % 2))
+read -r D LY DM ST <<< "$C"
+SL=$(( 3 * D + 4 ))
+
+NAME="g7-d$D-L$LY-dm$DM-st$ST-$M-s$S"
+echo "=== $NAME ==="
+$PY train_add.py --mode $M --digits $D --seq_len $SL --seed $S \
+  --n_train 500000 --n_eval 400 --eval_bs 100 \
+  --d $DM --layers $LY --heads 8 --bs 128 --steps $ST --lr 3e-4 \
+  --K 4 --lam 1.0 --out $SCR/pcmd/runs/$NAME

slurm/g8.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# Grid 8 - does the parallelism curriculum hold ACROSS difficulties?
+#
+# The previous method was excellent at 8 digits (3/3 seeds ~100%) but unstable
+# at 10-12 (1/3 seeds) because it demanded a full carry chain in one pass from
+# random initialisation. `prog` anneals the number of parallel commits 16->1 and
+# ramps the one-pass term in, so each stage is a small increment. Capacity is
+# also raised, since 14-16 digits previously failed even at sequential decoding
+# and that is a learnability failure, not a parallelism one.
+#SBATCH -J g8
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=64g
+#SBATCH -t 10:00:00
+#SBATCH -o logs/g8-%A_%a.out
+#SBATCH --array=0-29%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+PY=$SCR/hf_env/bin/python
+cd $SCR/pcmd/src
+
+I=$SLURM_ARRAY_TASK_ID
+DIG=(8 10 12 14 16); MODES=(mdlm prog); SEEDS=(0 1 2)
+D=${DIG[$((I / 6))]}
+M=${MODES[$(((I % 6) / 3))]}
+S=${SEEDS[$((I % 3))]}
+SL=$(( 3 * D + 4 ))
+
+NAME="g8-d$D-$M-s$S"
+echo "=== $NAME ==="
+$PY train_add.py --mode $M --digits $D --seq_len $SL --seed $S \
+  --n_train 500000 --n_eval 400 --eval_bs 100 \
+  --d 512 --layers 10 --heads 8 --bs 128 --steps 60000 --lr 3e-4 \
+  --K_max 16 --lam 1.0 --out $SCR/pcmd/runs/$NAME

slurm/g9_depth.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Grid 9 - THE FALSIFIABLE TEST of the theory in THEORY.md.
+#
+# Addition's carry is a prefix scan, computable in O(log n) depth by carry
+# lookahead, and constant-depth transformers sit inside uniform TC0 which
+# contains addition. So one-pass decoding should become possible once depth
+# reaches ~log2(n), and the minimum depth should grow LOGARITHMICALLY with the
+# number of digits.
+#
+#   log-depth account  : min depth ~ log2(n)   -> 2, 3, 4, 5
+#   ripple-carry account: min depth ~ n        -> 4, 8, 16, 32
+#   "transformers cannot do this": no depth suffices
+#
+# These are cleanly distinguishable. Sweep depth x digits and read off where
+# one-pass accuracy crosses 90%.
+#SBATCH -J g9
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=64g
+#SBATCH -t 12:00:00
+#SBATCH -o logs/g9-%A_%a.out
+#SBATCH --array=0-47%12
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+PY=$SCR/hf_env/bin/python
+cd $SCR/pcmd/src
+
+I=$SLURM_ARRAY_TASK_ID
+DIG=(4 8 16 32); LAYERS=(2 3 4 6 8 12); SEEDS=(0 1)
+D=${DIG[$((I / 12))]}
+L=${LAYERS[$(((I % 12) / 2))]}
+S=${SEEDS[$((I % 2))]}
+SL=$(( 3 * D + 4 ))
+
+NAME="g9-d$D-L$L-s$S"
+echo "=== $NAME ==="
+$PY train_add.py --mode prog --digits $D --seq_len $SL --seed $S \
+  --n_train 500000 --n_eval 400 --eval_bs 100 \
+  --d 384 --layers $L --heads 6 --bs 128 --steps 60000 --lr 3e-4 \
+  --K_max 16 --lam 1.0 --out $SCR/pcmd/runs/$NAME

slurm/gate.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# LEARNABILITY GATE for SIGMORPHON, before any parallel-decoding claim.
+# Published transformer baselines reach roughly 90%+ exact match on medium data
+# for these languages. If our harness cannot get near that with sequential
+# decoding, nothing measured at one pass would mean anything.
+#SBATCH -J siggate
+#SBATCH -p gpu-h100
+#SBATCH --gpus=h100:1
+#SBATCH -c 8
+#SBATCH --mem=32g
+#SBATCH -t 02:00:00
+#SBATCH -o logs/siggate-%j.out
+source /etc/profile
+SCR=/scratch/zt1/project/msml612/user/$USER
+module load python/3.10.10
+export PYTHONPATH=
+cd $SCR/pcmd/src
+for L in english german turkish; do
+  echo "--- $L (mdlm baseline) ---"
+  $SCR/hf_env/bin/python train_sig.py --lang $L --mode mdlm --size medium \
+    --steps 12000 --d 256 --layers 6 --lr 1e-4 --warmup 1000 \
+    --data_root $SCR/pcmd/data/sig --out $SCR/pcmd/runs/gate-$L 2>&1 | tail -2
+done