ppt-c4-pre350

SmolLM-135M trained on C4 after 350 steps of pre-pretraining on shuffle-Dyck-k64 followed by main C4 pretraining (the PPT+PT treatment). Part of the EPFL CS-552 (Spring 2026) Power Rangers project: Mechanisms of Pre-Pretraining Transfer in Language Models.

• Architecture: LlamaForCausalLM (30 layers × 9 query heads × 576 d_model, GQA with 3 KV heads, head_dim 64)
• Vocab: 49 152 (SmolLM BPE)
• Training data: allenai/c4 (en)
• Phase 1: 350 steps on k=64 shuffle-Dyck. Optimizer reset between phases. Phase 2: ~9 650 steps on C4.

Reproducibility

Both checkpoints in this pair were trained on the EPFL Alps cluster by team collaborators (M. Rofin and G. Yüce). The original training artifacts live in the tml-ppt/ppt-project wandb project; this HF copy is for downstream mechanistic-interpretability analysis.

Use

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("VityaVitalich/ppt-c4-pre350")
tokenizer = AutoTokenizer.from_pretrained("VityaVitalich/ppt-c4-pre350")