ddgoede/pythia-70m-ce-matched

Pythia-70M, cross-entropy, recipe-matched control

A Pythia-70M model trained from scratch with cross-entropy on the deduplicated Pile, using the exact training code, data pipeline, seed and schedule as ddgoede/pythia-70m-distilled-from-410m, with the distillation term switched off (alpha = 0).

Its purpose is to serve as a controlled baseline: comparing it against the distilled student isolates the effect of the loss function, since every other part of the recipe is shared. The public EleutherAI/pythia-70m-deduped checkpoint is trained by a different recipe (LR decayed to a 0.1x floor rather than to zero, fp16 rather than bf16, weight decay 0.1 rather than 0.01, gradient clipping, and a different global data order), so a comparison against it confounds the loss with the recipe.

Checkpoints

One branch (revision) per checkpoint step, following the Pythia schedule: step1, step2, step4, ..., step512, then step1000 through step143000 in steps of 1000 (153 revisions).

from transformers import GPTNeoXForCausalLM, AutoTokenizer

model = GPTNeoXForCausalLM.from_pretrained(
    "ddgoede/pythia-70m-ce-matched", revision="step143000"
)
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-70m-deduped")

Training details

Objective	cross-entropy (alpha = 0)
Data	EleutherAI/the_pile_deduplicated, streamed, shuffle buffer 10,000, seed 42
Steps	143,000
Batch	1024 sequences x 2048 tokens (2M tokens/step, ~293B tokens total)
Optimizer	AdamW, lr 1e-3, betas (0.9, 0.95), weight decay 0.01
Schedule	linear warmup 1430 steps, cosine decay to zero
Precision	bf16 mixed precision
Hardware	8x A5000

Training code: training/llm/train_distill.py in the wiggle-room-hypothesis-project repository.