bcdennis/concept-attn-pilot-baseline

Concept-Attention Pilot — Dense Baseline (CA_MODE=none)

Part of the concept-attention-at-generation pilot (design doc: NCP-ArchPreview arXiv 2609.10715 twist — attend to concepts during generation instead of predicting them). Tag: arxiv:2609.10715, arxiv:2602.08984

Config

From-scratch pre-LN decoder: 33,888,384 params, d_model 384, 8 layers, 6 heads, seq 1024, tied GPT-2 BPE embeddings (50257). Data: HuggingFaceFW/fineweb-edu sample-10BT, deterministic stream (seed 1234), 400M tokens = 1525 optimizer steps @ 262,144 tokens/step. AdamW lr 3e-4 (warmup 100, cosine), wd 0.1 (ndim>=2), bf16 autocast, grad clip 1.0.

Result

Final val_ce 5.1985 (val = 2,685,222 tokens from first 2500 held-out docs). Training 2.55h on 1x L4.

Eval history (optimizer step -> tokens -> val_ce): 100 -> 26.2M -> 7.0803; 1000 -> 262.1M -> 5.2988; 1100 -> 288.4M -> 5.2599; 1200 -> 314.6M -> 5.2339; 1300 -> 340.8M -> 5.2130; 1400 -> 367.0M -> 5.2017; 1500 -> 393.2M -> 5.1987; final 5.1985. (Intermediate evals 200-900 not recovered from the log tail; full history was in metrics.json, lost with the job's ephemeral storage.)

Verdict context

Head-to-head at matched tokens: Cond-0 (concept attention, no NCP loss) finished at 5.1940 vs this baseline's 5.1985 — Cond-0 leads at every matched eval point. Weight upload failed on an API kwarg bug (fixed in train.py v19); weights are not recoverable, metrics are as recorded here.