Byrne-15M-Looped

Small toy / smoke-test for Byrne-100M-Ultra-MC. I ran looping here first, at ~15M, to see if the stack still trained before I spent the 114M budget. Not a product. Not a matched scale-up of Ultra-MC.

~15.2M active / ~18.9M total looped SpikeWhale (MoE, loop_count=3, effective depth 30). Trained from scratch on UltraX, then SFT on UltraChat, then DPO. Inference code, checkpoints, and a safetensors export for leaderboard owners are in this repo.

It's a small model. Fluent-ish English, weak knowledge, weak code. Numbers in Evaluation. Base continues text. SFT/DPO take ChatML; they are a probe, not a chat product.

The scored pretrained dir is safetensors/base/ (WikiText-2 byte_ppl 2.943). Pretrain tokens: 246M at the 40k matched A/B, 419M at step 68.25k. SFT adds 82M packed tokens (10k × batch 8 × 1024). Almost all of the budget is the base.

Checkpoints

file	stage	use
safetensors/base/	pretrained (checkpoints/best)	scored. Lowest byte_ppl. Continues text.
checkpoints/base_40k.pt	pretrain step 40k	loop vs unlooped A/B. FINDINGS.md
checkpoints/base_68k.pt	pretrain step 68.25k	later base, val_ppl 59.8
checkpoints/sft_ultrachat10k.pt	SFT 10k UltraChat	--chat
checkpoints/dpo_seg1.pt	DPO on the SFT	--chat (default). Scored.

Same loader for all of them. Context is 1024 everywhere. Safetensors mirrors are under safetensors/ — that is what to point a harness at.

Architecture

Looped decoder (model_v2.py, SpikeWhale v2). MoE is on. Vocab 16,512 via a byte-level Length-MAX tokenizer (spike_tokenizer.py, tokenizer.json) from Dong & Su, Length-MAX Tokenizer for Language Models (arXiv:2511.20849).

Active / total params	15.2M / 18.9M
Hidden size	384
Layers	10 × loop_count 3 (effective 30)
Attention	8 heads, 1 KV (MQA), MLA
Head dim	64 (RoPE 32 + NoPE 32)
FFN	MoE, 4 routed + 1 shared, top-2, width 160
Tie embeddings	yes
Context	1024

Also on: fractal RoPE (γ=1), Elo, Engram (table 1024), HRM refine ×3 with deep supervision, abstain head, XSA, hyper-connections. No Memory Cache. That branch lives on Ultra-MC.

Config: byrne_15m.yaml + the config baked into each .pt.

The +1,152 params vs the unlooped 15M baseline are per-pass embeddings. Everything else is weight-tied across the three passes.

Evaluation

Full, uncapped. Same harness as Ultra-MC. Raw JSON: benchmarks/Byrne-15M-Looped.json (base) and benchmarks/Byrne-15M-DPOseg1.json. Tables: BENCHMARKS.md.

Metric	base	DPO seg1	chance
WikiText-2 byte_ppl ↓	2.943	3.014	—
BLiMP acc ↑	0.734	0.711	0.50
arc_easy acc	0.335	0.325	0.25
arc_easy acc_norm	0.328	0.326	0.25
arc_challenge acc	0.182	0.179	0.25
arc_challenge acc_norm	0.224	0.232	0.25
hellaswag acc	0.267	0.269	0.25
hellaswag acc_norm	0.272	0.271	0.25
winogrande acc	0.509	0.512	0.50
piqa acc	0.533	0.554	0.50
piqa acc_norm	0.521	0.539	0.50
openbookqa acc	0.126	0.138	0.25
openbookqa acc_norm	0.248	0.262	0.25
boolq acc	0.378	0.397	0.50
ArithMark-3.0 acc	0.345	0.322	0.25
ArithMark-3.0 acc_norm	0.347	0.327	0.25

n: arc_easy 2376, arc_challenge 1172, hellaswag 10042, winogrande 1267, piqa 1838, openbookqa 500, boolq 3270, arithmark 1000, BLiMP 1800. Not a 200-cap.

Base wins LM metrics and ArithMark. DPO moves PIQA / BoolQ a little and costs PPL. Most MC is chance. 15M.

Loop vs no-loop (step 40k, 246M tokens)

Param-matched A/B against loop_count=1. Looped spends 3× compute per token. Writeup: FINDINGS.md. Checkpoint: base_40k.pt.

Val loss 4.195 vs 4.231. Sampled repetition 0.011 vs 0.049. The unlooped baseline's % % % collapse (rep 0.92) never showed up on the looped arm.

68k, capped 200

checkpoints/base_68k.pt. Same harness, --limit 200. Does not replace the full table above. JSON: benchmarks/Byrne-15M-Looped-base68k-cap200.json.

Metric	68k cap-200	chance
WikiText-2 byte_ppl ↓	2.937	—
BLiMP acc ↑	0.728	0.50
arc_easy acc	0.360	0.25
hellaswag acc_norm	0.330	0.25
piqa acc	0.515	0.50
boolq acc	0.365	0.50
ArithMark-3.0 acc_norm	0.335	0.25

n=200 on MC / ArithMark. Full table: BENCHMARKS.md.

Usage

# base (text continuation) — scored pretrained dir
python generate.py --ckpt safetensors/base -p "The capital of France is" --temp 0.7 --top-k 40 --rp 1.3

# DPO chat (default checkpoint)
python generate.py --chat -p "Explain why the sky is blue in one sentence." --temp 0.7 --top-k 40 --rp 1.3

# SFT chat
python generate.py --ckpt checkpoints/sft_ultrachat10k.pt --chat -p "Give me a tip for staying focused."

Defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy is a test tool. DECODING-DEFAULTS.md.

python verify.py          # greedy cached decode == full recompute
python safetensors/load.py base

Files

config.py  model_v2.py  spike_tokenizer.py  special_tokens.py  fractal.py
chat_format.py  generate.py  tokenizer.json
byrne_15m.yaml  package.json  requirements.txt
verify.py  DECODING-DEFAULTS.md  PROVENANCE.md  FINDINGS.md
checkpoints/{base_40k,base_68k,sft_ultrachat10k,dpo_seg1}.pt
safetensors/{base,base_40k,base_68k,sft_ultrachat10k,dpo_seg1}/
benchmarks/

Leaderboard: safetensors/base/ (config.json + model.safetensors + tokenizer.json). You still need this repo's model_v2.py. Call tie_weights() after load. safetensors/README.md.

Training recipe

Pretrain: UltraX-Preview (UltraX-Ultra-FineWeb), batch 6, seq 1024, lr 6e-4, cosine 500k planned. Released bases: 40k (246M tokens) and 68.25k (419M). SFT: UltraChat train_sft, 10k steps, batch 8, block 1024, from checkpoints/best. DPO: UltraFeedback, 1k steps, β=0.1, seg1 is the scored chat stage.

Limitations

• 15M. Factual recall is weak. MC tasks near chance.
• Code and multi-step reasoning are weak.
• English. No safety tuning past DPO.
• Base continues text. Chat template will not make it answer questions.
• Looping costs 3× FLOPs per token vs the dense 15M baseline.

What this is

A toy. Smoke-test for Byrne-100M-Ultra-MC. I wanted to know the looped stack would train at all before I put 2.0B tokens into the 114M run.

While I was here I also checked whether loop_count=3 beats loop_count=1 at equal parameters. It does, on val loss and on degeneration, at 3× the compute per token. Then I SFT'd and DPO'd it the same way I do the bigger Byrne runs. That is extra. The point of this repo is the smoke test.

SFT/DPO are a probe. If you want instruct, start from safetensors/base and train your own.

Citation

@misc{byrne15mlooped,
  title        = {Byrne-15M-Looped: A ~15M-parameter looped SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-15M-Looped}},
  note         = {Quazim0t0/Byrne-15M-Looped}
}