# PROGRESS — BLT-LLM

Last updated: 2026-06-08

## Current phase
**Phase 3 — End-to-end training: DONE & verified.** Full 20k-step run on the 2.2GB
TinyStories TRAIN split finished: held-out **BPB 8.0 → 0.71** (best 0.7077 @ step 19500),
plateaued; causality gate passes; samples are coherent TinyStories prose. All three
Phase 3 done-criteria met. Next up: Phase 4 (scale config) — **rented GPU only, not local**.

## Done & verified
- Repo under git, pushed to private GitHub. `.gitignore` excludes `reference/`, `data/`, `logs/`, caches.
- `blt/` package scaffolded: `config.py`, `data.py`, `entropy_model.py`, `train_entropy.py`, `patcher.py`, `tests/`.
- **Entropy byte-LM** (`blt/entropy_model.py`): **1,705,216 params**. 2 layers × dim 256, 4 heads,
  sliding-window (local block-causal) attention window 128, RoPE θ=500000, RMSNorm, SwiGLU, tied embeddings.
- **Data** (`blt/data.py`): `TinyStoriesV2-GPT4-valid.txt` via `huggingface_hub` → ~22.3M train bytes / 225K val bytes.
- **Training** (`blt/train_entropy.py`): 8000 steps; AdamW β(0.9,0.95) wd 0.1 clip 1.0; cosine LR 4e-4, 300 warmup.
  **Final val BPB = 0.9278** (step-0 baseline 8.06 = log₂256). ~30 min on RTX 3050 6GB. → `checkpoints/entropy_model.pt`.
- **Patcher** (`blt/patcher.py`): entropy in nats (`-Σ p·log p`); causal global-threshold rule
  (byte *i* starts a patch iff `H(x_i | x_<i) > θ`); threshold tuning by bisection.
  Tuned **θ = 1.0917** (official repo default ≈ 1.335). → `checkpoints/patcher_threshold.json`.
- **Phase 0 isolation test** (`blt/tests/test_phase0_patcher.py`): **ALL PASS**
  - achieved avg patch on held-out 100 kB = **4.36** (target 4–6) ✓
  - reproducible ✓, causal / no future leakage ✓, patch lengths sum to n ✓, first byte = boundary ✓
  - in-dist "Once upon a time…" → big patches (low entropy); OOD GoT sentence over-segments (expected).
  - `artifacts/phase0_entropy.png` (paper Fig. 4 style).

- **BLT model** (`blt/model.py`): **55,392,768 params** (~55M). Assembles encoder → global → decoder.
  hE=hD=256, hG=768 (k=3), encoder 1×256/4h, global 6×768/12h, decoder 4×256/4h, cross 4h. Tied byte embed/output.
- **Shared layers** (`blt/layers.py`): RMSNorm, RoPE θ=500000, SwiGLU, SelfAttention (windowed or full causal), TransformerBlock.
- **Hash n-grams** (`blt/ngram_hash.py`): RollPolyHash (reference primes), precomputed IDs, per-n tables (4096×256), e_i normalized by (n_sizes+1).
- **Encoder** (`blt/encoder.py`): windowed-causal byte layer; **max-pool** seeds patch queries; encoder cross-attn (patch↔own bytes) AFTER the layer. Returns byte_hidden (carry-over, not detached) + patch_reps.
- **Global** (`blt/global_transformer.py`): block-causal over patches + key-padding for padded patches.
- **Decoder** (`blt/decoder.py`): cross-attn BEFORE layers; **byte i attends previous patch o_{j-1}** (causal shift — first-patch bytes get no global ctx); windowed-causal byte layers; output → 256.
- **Dummy patcher** (`blt/patcher.py:strided_patch_lengths`): fixed stride, but emits the same [B,M] format as the entropy patcher (model never assumes equal size).
- **Tests PASS**: `test_shapes.py` (hand-checked 2-patch masks + T→M→T transitions + padded patches all finite); `test_overfit.py` (single batch, loss 5.63→0.047 in 41 steps).

### Phase 2 (real entropy patcher) — DONE & verified
- Added batched patching to `blt/patcher.py`: `batch_entropies`, `batch_boundaries`,
  `boundaries_to_patch_lengths` ([B,Mmax] right-padded), `entropy_patch_lengths` (drop-in for `strided_patch_lengths`).
- `test_phase2.py` PASS: real patcher gives variable per-row patches (counts e.g. [5,19,23,23], lengths 1..30),
  exercises the padded-patch path with real data (finite), and the single-batch overfit still collapses (5.63→0.0496).
- **HARD GATE — `test_causality.py` PASS (no future leakage), required before Phase 2 done:**
  full model + real patcher, 4 batches × 7 t-values × 2 tests:
  - gradient leakage: ∂logit_t/∂embed_s == 0 (exact) for all s>t and all other batch rows;
  - perturbation: scramble bytes>t (recompute hashes) → logits≤t bit-identical (diff exactly 0).
  Added `inputs_embeds` path to `model.py`/`encoder.py` to put grad on input byte embeds.
- Phase 1 strided overfit + shapes still PASS (no regression).

### Phase 3 (train end-to-end) — DONE & verified
- **Full run result** (`python -m blt.train --data train --steps 20000 --batch-size 8
  --seq-len 512 --grad-accum 2`, ~2.4 h on RTX 3050 6GB, ~2.33 it/s):
  held-out BPB **8.07 → 0.71** (best **0.7077** @ step 19500), curve flat over the last
  ~4k steps → plateaued. Untrained baseline = 8.0 (log₂256); brief's rough target ≤ 1.5 → beaten.
  - BPB curve: 1.39(500) 1.16(1k) 1.04(2k) 0.96(3k) 0.87(5k) 0.80(9k) 0.73(13k) 0.71(17k+).
  - Best weights+optimizer → `checkpoints/blt_model.pt` (635M, gitignored; see below).
- **Final sample** (greedy-ish, temp 0.7, from the best checkpoint):
  *"Once upon a time, in a small house, there was a boy named Tim. One day, Tim went to the
  store with his mom. They needed to buy a toy… Tim said to his mom, \"Mom, can I give t…"*
  → coherent, named characters, dialogue, paragraph breaks. Recognizable text ✓.
- **Causality leakage gate**: PASS (exact-zero; architecture-level, re-run after the Phase 3
  code and unchanged since). Three Phase-3 done-criteria all met.

### Phase 3 (train end-to-end) — code (built earlier in the phase)
- **`blt/config.py:BLTTrainConfig`**: AdamW β(0.9,0.95) wd 0.1 clip 1.0; cosine LR 4e-4,
  300 warmup; defaults sized for 6GB (batch 8 × grad_accum 2, seq_len 512). Eval = BPB.
- **`blt/train.py`**: full loop. Each batch → FROZEN entropy patcher decides per-row patch
  boundaries (causal, no grad) → precomputed n-gram hash IDs → BLT forward → next-byte CE.
  Held-out BPB eval, best+last checkpointing, `--resume`, in-loop sampling, `--smoke`,
  `--data train|valid`. Smoke (40 steps, valid): BPB 8.07 → 4.6; ckpt+resume+sample OK.
  Target batch (8×2, seq 512) fits 6GB with headroom.
- **`blt/generate.py`**: autoregressive byte generation. Recomputes patch boundaries over
  the running prefix each step (correct-first; causal patcher → no future leak). Temperature
  / top-k. Loads BLT ckpt + frozen entropy model + θ. Verified end-to-end on the smoke ckpt.
- **`test_causality.py` HARD GATE re-run after the Phase 3 code** — still ALL PASS
  (worst future-grad / cross-batch / perturb |diff| all 0.000e+00). Safe to train.
- **Real run launched** (background, detached): `python -m blt.train --data train --steps 20000
  --batch-size 8 --seq-len 512 --grad-accum 2` → `logs/train.log`, ckpts in `checkpoints/`.
  Downloads the 2.2GB TinyStories TRAIN split first (cached), then trains (~2.2B bytes ≫ 55M
  params, so it won't memorize). Resumable with `--resume`.

## In progress
- Nothing. Phases 0–3 complete and **publicly released** on both GitHub (public) and
  Hugging Face. Best model weights backed up via **git-LFS** (`checkpoints/blt_model_weights.pt`,
  212M, model-only, val_bpb 0.7078 @ step 19500). The 635M full checkpoint (with optimizer)
  stays local + gitignored. git-lfs binary in `~/.local/bin` (no system install).
- **Public artifacts** (anyone can clone + `pip install -r requirements.txt` + `python -m
  blt.generate`):
  - GitHub (public): https://github.com/shaikh-saud705/blt-llm — code + weights (LFS) + README.
  - HF model (public): https://huggingface.co/sssssaud/blt-llm-tinystories-55m — code mirror
    + weights + entropy patcher + θ + model card.
  - `README.md` (root) + `requirements.txt` added; `blt/generate.py` now defaults `--ckpt`
    to `checkpoints/blt_model_weights.pt` so it runs straight from a clone.

## Next action
- **Phase 4 (scale-up) — rented GPU only, NEVER local** (BLT_LLM.md sec 4 / gotcha #10):
  swap TinyConfig for the ~1.5B config (paper 1B row widened), add bf16/mixed precision,
  gradient checkpointing, FlexAttention for the patch-masked cross-attn, robust ckpt
  save/resume. Done when it trains stably on the rented GPU and checkpoints resume.
  The architecture itself is now proven end-to-end (this was the whole point of Phases 0–3).
- Optional polish before scaling: trim model toward 50M (ngram_table_size or one global
  layer) if desired. (Tiny checkpoint is already backed up via git-LFS — see In progress.)

## Blockers
- None.

## Reproduce
```
python -m blt.train_entropy --steps 8000     # Phase 0: train entropy model -> checkpoints/entropy_model.pt
python -m blt.tests.test_phase0_patcher      # Phase 0 isolation test -> artifacts/ + threshold json
python -m blt.tests.test_shapes              # Phase 1 shapes + masks
python -m blt.tests.test_overfit             # Phase 1 single-batch overfit
python -m blt.tests.test_phase2              # Phase 2 real patcher + overfit
python -m blt.tests.test_causality           # HARD GATE: no future leakage (re-run after any mask change)
python -m blt.train --smoke                  # Phase 3 quick wiring check (valid split, 40 steps)
python -m blt.train --data train --steps 20000 --batch-size 8 --seq-len 512 --grad-accum 2   # Phase 3 full run
python -m blt.generate --prompt "Once upon a time"   # sample from the trained model
```