--- language: - en license: apache-2.0 library_name: transformers tags: - pretraining - educational - pedagogical - sutra - smollm2 - llama pipeline_tag: text-generation model-index: - name: SmolLM2-70M results: - task: type: text-generation name: Text Generation dataset: type: ai2_arc name: ARC-Easy config: ARC-Easy metrics: - type: acc_norm value: 33.00 name: Normalized Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: ai2_arc name: ARC-Challenge config: ARC-Challenge metrics: - type: acc_norm value: 22.35 name: Normalized Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: boolq name: BoolQ metrics: - type: acc value: 39.66 name: Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: hellaswag name: HellaSwag metrics: - type: acc_norm value: 26.14 name: Normalized Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: piqa name: PIQA metrics: - type: acc_norm value: 54.84 name: Normalized Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: sciq name: SciQ metrics: - type: acc_norm value: 45.20 name: Normalized Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: winogrande name: WinoGrande metrics: - type: acc value: 50.04 name: Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: truthful_qa name: TruthfulQA MC2 metrics: - type: acc value: 48.02 name: Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: gsm8k name: GSM8K metrics: - type: exact_match value: 0.53 name: Exact Match (5-shot) - task: type: text-generation name: Text Generation dataset: type: cais/mmlu name: MMLU metrics: - type: acc value: 22.96 name: Accuracy (0-shot) - task: type: text-generation name: Text Generation dataset: type: openbookqa name: OpenBookQA metrics: - type: acc_norm value: 27.60 name: Normalized Accuracy (0-shot) base_model: HuggingFaceTB/SmolLM2-70M datasets: - codelion/sutra-10B --- # SmolLM2-70M A SmolLM2-70M model pretrained on the [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) pedagogical dataset for 3 epochs (~30.6B tokens total). This model demonstrates that a 69M parameter model can be trained to near-capacity performance using dense, curated educational data. ## Model Details | Property | Value | |----------|-------| | Architecture | LlamaForCausalLM | | Parameters | 69.2M | | Hidden Size | 384 | | Layers | 32 | | Attention Heads | 6 (2 KV heads) | | Context Length | 8,192 | | Vocabulary | 49,152 | | Precision | bfloat16 | | Base Model | [SmolLM2-70M](https://huggingface.co/HuggingFaceTB/SmolLM2-70M) | | Training Dataset | [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) (10.2B tokens) | ## Training The model was trained for 3 epochs on the Sutra-10B dataset using a single NVIDIA L40S GPU (46GB). This checkpoint is the best perplexity checkpoint from epoch 3. | Epoch | Tokens | Training Time | Learning Rate | Best Perplexity | |-------|--------|---------------|---------------|-----------------| | 1 | 10.2B | 25.82h | 3e-4 → 3e-5 | 39.50 | | 2 | 10.2B | 25.78h | 1e-4 → 1e-5 | 37.81 | | 3 | 10.2B | 26.16h | 3e-5 → 3e-6 | 37.72 | | **Total** | **30.6B** | **77.76h** | — | **37.72** | Training configuration: - Optimizer: AdamW (fused), weight decay 0.1 - Schedule: Cosine with warmup - Batch size: 4 per device, gradient accumulation 8 (effective ~262K tokens/step) - Sequence length: 8,192 - Flash Attention 2, TF32 matmul, torch.compile - Throughput: ~110K tokens/sec ## Benchmark Results All benchmarks evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-eval) v0.4.11. All tasks are 0-shot except GSM8K (5-shot). ### This Model vs Training Progression | Benchmark | **E3-best** | E3-final | E2-best | E2-final | E1-final | |-----------|:-----------:|:--------:|:-------:|:--------:|:--------:| | ARC-Easy | **33.00** | 33.16 | 32.83 | 33.12 | 33.46 | | ARC-Challenge | **22.35** | 21.67 | 22.61 | 22.44 | 22.44 | | BoolQ | **39.66** | 39.66 | 39.79 | 39.54 | 39.79 | | HellaSwag | **26.14** | 26.03 | 26.08 | 25.91 | 26.03 | | PIQA | **54.84** | 55.01 | 54.24 | 54.13 | 54.62 | | SciQ | **45.20** | 46.30 | 44.10 | 45.50 | 43.60 | | WinoGrande | **50.04** | 49.33 | 50.51 | 48.70 | 48.78 | | TruthfulQA | **48.02** | 47.93 | 48.30 | 48.14 | 48.30 | | GSM8K | **0.53** | 0.61 | 0.68 | 0.83 | 0.15 | | MMLU | **22.96** | 22.87 | 23.00 | 22.98 | 22.99 | | OpenBookQA | **27.60** | 27.60 | — | — | — | | **Average (10)** | **34.27** | 34.26 | 34.21 | 34.13 | 34.02 | ### Comparison with 1B Token Baselines (SmolLM2-70M) These are results from training the same SmolLM2-70M model on various 1B-token datasets from the [Pre-training Dataset Samples](https://huggingface.co/collections/codelion/pre-training-dataset-samples-686bd760abf1a43b0ce32829) collection for 1 epoch, showing that Sutra-10B at 3 epochs achieves the highest performance for this model size. | Dataset (1B tokens) | HellaSwag | PIQA | WinoGrande | ARC-C | MMLU | TruthfulQA | GSM8K | Avg | |---------------------|-----------|------|------------|-------|------|------------|-------|-----| | **Sutra-10B (3 epochs)** | **26.14** | **54.84** | **50.04** | **22.35** | 22.96 | **48.02** | 0.53 | **34.27** | | [Sutra-1B](https://huggingface.co/datasets/codelion/sutra-1B) | 25.43 | 53.86 | 49.41 | 23.04 | 22.91 | 49.09 | 1.14 | 32.13 | | [FineWiki-1B](https://huggingface.co/datasets/HuggingFaceFW/finewiki) | 25.56 | 51.69 | 48.86 | 24.15 | **23.34** | 51.16 | 0.91 | 32.24 | | [FinePDFs-1B](https://huggingface.co/datasets/HuggingFaceFW/FinePDFs) | 25.58 | 52.56 | 50.51 | 22.44 | 22.95 | 51.41 | 1.21 | 32.38 | | [DCLM-Baseline-1B](https://huggingface.co/datasets/codelion/dclm-baseline-1B) | 25.85 | 55.17 | 50.20 | 21.08 | 22.97 | 49.21 | 0.68 | 32.16 | | [FineWeb-Edu-1B](https://huggingface.co/datasets/codelion/fineweb-edu-1B) | 25.72 | 55.11 | 50.36 | 21.25 | 22.96 | 48.11 | 1.21 | 32.10 | | [Essential-Web-1B](https://huggingface.co/datasets/sumukshashidhar-archive/essential-web-v1.0-sample-1B) | 26.02 | 55.44 | 48.30 | 20.99 | 22.95 | 49.59 | 1.29 | 32.08 | | [Synth-1B](https://huggingface.co/datasets/codelion/synth-1B) | 26.63 | 50.98 | 48.78 | 21.93 | 23.24 | 47.10 | 1.29 | 31.42 | ## Key Findings 1. **Capacity ceiling**: The 70M parameter model reaches its capacity ceiling at approximately 10B tokens. Additional epochs (up to 30.6B total tokens) yield only marginal improvements in benchmark scores (+0.25 average from epoch 1 to epoch 3), despite continued perplexity improvement (39.50 → 37.72). 2. **Perplexity vs benchmarks**: Perplexity continues to decrease across epochs, but downstream benchmark performance plateaus, suggesting the model's representational capacity is the bottleneck rather than data exposure. 3. **Data quality matters**: Even at 1B tokens, Sutra outperforms or matches larger web-crawled datasets (DCLM, FineWeb-Edu, Essential-Web) on average, demonstrating the value of curated pedagogical content. ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("codelion/SmolLM2-70M", trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained("codelion/SmolLM2-70M") input_text = "The theory of relativity states that" inputs = tokenizer(input_text, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=100) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` ## Limitations - This is a 69M parameter base model (not instruction-tuned) — it generates completions, not conversational responses - Performance is at the capacity ceiling for this model size; larger models would benefit more from the Sutra-10B dataset - The model was trained primarily on English educational content ## Related Resources - **Dataset**: [codelion/sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) — 10B token pedagogical pretraining dataset - **Sutra Framework**: Generates structured educational content optimized for LLM pretraining ## Citation ```bibtex @article{sharma2026sutra, title={Scaling Pedagogical Pretraining: From Optimal Mixing to 10 Billion Tokens}, author={Sharma, Asankhaya}, year={2026}, url={https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens} } ``` ## License Apache 2.0