PROGRESS.md

PROGRESS — BLT-LLM

Last updated: 2026-06-08

Current phase

Phase 3 — End-to-end training: DONE & verified. Full 20k-step run on the 2.2GB TinyStories TRAIN split finished: held-out BPB 8.0 → 0.71 (best 0.7077 @ step 19500), plateaued; causality gate passes; samples are coherent TinyStories prose. All three Phase 3 done-criteria met. Next up: Phase 4 (scale config) — rented GPU only, not local.

Done & verified

• Repo under git, pushed to private GitHub. .gitignore excludes reference/, data/, logs/, caches.

• blt/ package scaffolded: config.py, data.py, entropy_model.py, train_entropy.py, patcher.py, tests/.

• Entropy byte-LM (blt/entropy_model.py): 1,705,216 params. 2 layers × dim 256, 4 heads, sliding-window (local block-causal) attention window 128, RoPE θ=500000, RMSNorm, SwiGLU, tied embeddings.

• Data (blt/data.py): TinyStoriesV2-GPT4-valid.txt via huggingface_hub → ~22.3M train bytes / 225K val bytes.

• Training (blt/train_entropy.py): 8000 steps; AdamW β(0.9,0.95) wd 0.1 clip 1.0; cosine LR 4e-4, 300 warmup. Final val BPB = 0.9278 (step-0 baseline 8.06 = log₂256). ~30 min on RTX 3050 6GB. → checkpoints/entropy_model.pt.

• Patcher (blt/patcher.py): entropy in nats (-Σ p·log p); causal global-threshold rule (byte i starts a patch iff H(x_i | x_<i) > θ); threshold tuning by bisection. Tuned θ = 1.0917 (official repo default ≈ 1.335). → checkpoints/patcher_threshold.json.

• Phase 0 isolation test (blt/tests/test_phase0_patcher.py): ALL PASS

  • achieved avg patch on held-out 100 kB = 4.36 (target 4–6) ✓
  • reproducible ✓, causal / no future leakage ✓, patch lengths sum to n ✓, first byte = boundary ✓
  • in-dist "Once upon a time…" → big patches (low entropy); OOD GoT sentence over-segments (expected).
  • artifacts/phase0_entropy.png (paper Fig. 4 style).

• BLT model (blt/model.py): 55,392,768 params (~55M). Assembles encoder → global → decoder. hE=hD=256, hG=768 (k=3), encoder 1×256/4h, global 6×768/12h, decoder 4×256/4h, cross 4h. Tied byte embed/output.

• Shared layers (blt/layers.py): RMSNorm, RoPE θ=500000, SwiGLU, SelfAttention (windowed or full causal), TransformerBlock.

• Hash n-grams (blt/ngram_hash.py): RollPolyHash (reference primes), precomputed IDs, per-n tables (4096×256), e_i normalized by (n_sizes+1).

• Encoder (blt/encoder.py): windowed-causal byte layer; max-pool seeds patch queries; encoder cross-attn (patch↔own bytes) AFTER the layer. Returns byte_hidden (carry-over, not detached) + patch_reps.

• Global (blt/global_transformer.py): block-causal over patches + key-padding for padded patches.

• Decoder (blt/decoder.py): cross-attn BEFORE layers; byte i attends previous patch o_{j-1} (causal shift — first-patch bytes get no global ctx); windowed-causal byte layers; output → 256.

• Dummy patcher (blt/patcher.py:strided_patch_lengths): fixed stride, but emits the same [B,M] format as the entropy patcher (model never assumes equal size).

• Tests PASS: test_shapes.py (hand-checked 2-patch masks + T→M→T transitions + padded patches all finite); test_overfit.py (single batch, loss 5.63→0.047 in 41 steps).

Phase 2 (real entropy patcher) — DONE & verified

• Added batched patching to blt/patcher.py: batch_entropies, batch_boundaries, boundaries_to_patch_lengths ([B,Mmax] right-padded), entropy_patch_lengths (drop-in for strided_patch_lengths).
• test_phase2.py PASS: real patcher gives variable per-row patches (counts e.g. [5,19,23,23], lengths 1..30), exercises the padded-patch path with real data (finite), and the single-batch overfit still collapses (5.63→0.0496).
• HARD GATE — test_causality.py PASS (no future leakage), required before Phase 2 done: full model + real patcher, 4 batches × 7 t-values × 2 tests:
  • gradient leakage: ∂logit_t/∂embed_s == 0 (exact) for all s>t and all other batch rows;
  • perturbation: scramble bytes>t (recompute hashes) → logits≤t bit-identical (diff exactly 0). Added inputs_embeds path to model.py/encoder.py to put grad on input byte embeds.
• Phase 1 strided overfit + shapes still PASS (no regression).

Phase 3 (train end-to-end) — DONE & verified

• Full run result (python -m blt.train --data train --steps 20000 --batch-size 8 --seq-len 512 --grad-accum 2, ~2.4 h on RTX 3050 6GB, ~2.33 it/s): held-out BPB 8.07 → 0.71 (best 0.7077 @ step 19500), curve flat over the last ~4k steps → plateaued. Untrained baseline = 8.0 (log₂256); brief's rough target ≤ 1.5 → beaten.
  • BPB curve: 1.39(500) 1.16(1k) 1.04(2k) 0.96(3k) 0.87(5k) 0.80(9k) 0.73(13k) 0.71(17k+).
  • Best weights+optimizer → checkpoints/blt_model.pt (635M, gitignored; see below).
• Final sample (greedy-ish, temp 0.7, from the best checkpoint): "Once upon a time, in a small house, there was a boy named Tim. One day, Tim went to the store with his mom. They needed to buy a toy… Tim said to his mom, "Mom, can I give t…" → coherent, named characters, dialogue, paragraph breaks. Recognizable text ✓.
• Causality leakage gate: PASS (exact-zero; architecture-level, re-run after the Phase 3 code and unchanged since). Three Phase-3 done-criteria all met.

Phase 3 (train end-to-end) — code (built earlier in the phase)

• blt/config.py:BLTTrainConfig: AdamW β(0.9,0.95) wd 0.1 clip 1.0; cosine LR 4e-4, 300 warmup; defaults sized for 6GB (batch 8 × grad_accum 2, seq_len 512). Eval = BPB.
• blt/train.py: full loop. Each batch → FROZEN entropy patcher decides per-row patch boundaries (causal, no grad) → precomputed n-gram hash IDs → BLT forward → next-byte CE. Held-out BPB eval, best+last checkpointing, --resume, in-loop sampling, --smoke, --data train|valid. Smoke (40 steps, valid): BPB 8.07 → 4.6; ckpt+resume+sample OK. Target batch (8×2, seq 512) fits 6GB with headroom.
• blt/generate.py: autoregressive byte generation. Recomputes patch boundaries over the running prefix each step (correct-first; causal patcher → no future leak). Temperature / top-k. Loads BLT ckpt + frozen entropy model + θ. Verified end-to-end on the smoke ckpt.
• test_causality.py HARD GATE re-run after the Phase 3 code — still ALL PASS (worst future-grad / cross-batch / perturb |diff| all 0.000e+00). Safe to train.
• Real run launched (background, detached): python -m blt.train --data train --steps 20000 --batch-size 8 --seq-len 512 --grad-accum 2 → logs/train.log, ckpts in checkpoints/. Downloads the 2.2GB TinyStories TRAIN split first (cached), then trains (~2.2B bytes ≫ 55M params, so it won't memorize). Resumable with --resume.

In progress

• Nothing. Phases 0–3 complete and publicly released on both GitHub (public) and Hugging Face. Best model weights backed up via git-LFS (checkpoints/blt_model_weights.pt, 212M, model-only, val_bpb 0.7078 @ step 19500). The 635M full checkpoint (with optimizer) stays local + gitignored. git-lfs binary in ~/.local/bin (no system install).
• Public artifacts (anyone can clone + pip install -r requirements.txt + python -m blt.generate):
  • GitHub (public): https://github.com/shaikh-saud705/blt-llm — code + weights (LFS) + README.
  • HF model (public): https://huggingface.co/sssssaud/blt-llm-tinystories-55m — code mirror
    • weights + entropy patcher + θ + model card.
  • README.md (root) + requirements.txt added; blt/generate.py now defaults --ckpt to checkpoints/blt_model_weights.pt so it runs straight from a clone.

Next action

• Phase 4 (scale-up) — rented GPU only, NEVER local (BLT_LLM.md sec 4 / gotcha #10): swap TinyConfig for the ~1.5B config (paper 1B row widened), add bf16/mixed precision, gradient checkpointing, FlexAttention for the patch-masked cross-attn, robust ckpt save/resume. Done when it trains stably on the rented GPU and checkpoints resume. The architecture itself is now proven end-to-end (this was the whole point of Phases 0–3).
• Optional polish before scaling: trim model toward 50M (ngram_table_size or one global layer) if desired. (Tiny checkpoint is already backed up via git-LFS — see In progress.)

Blockers

• None.

Reproduce

python -m blt.train_entropy --steps 8000     # Phase 0: train entropy model -> checkpoints/entropy_model.pt
python -m blt.tests.test_phase0_patcher      # Phase 0 isolation test -> artifacts/ + threshold json
python -m blt.tests.test_shapes              # Phase 1 shapes + masks
python -m blt.tests.test_overfit             # Phase 1 single-batch overfit
python -m blt.tests.test_phase2              # Phase 2 real patcher + overfit
python -m blt.tests.test_causality           # HARD GATE: no future leakage (re-run after any mask change)
python -m blt.train --smoke                  # Phase 3 quick wiring check (valid split, 40 steps)
python -m blt.train --data train --steps 20000 --batch-size 8 --seq-len 512 --grad-accum 2   # Phase 3 full run
python -m blt.generate --prompt "Once upon a time"   # sample from the trained model