| | --- |
| | language: |
| | - en |
| | license: apache-2.0 |
| | library_name: transformers |
| | tags: |
| | - pretraining |
| | - educational |
| | - pedagogical |
| | - sutra |
| | - smollm2 |
| | - llama |
| | pipeline_tag: text-generation |
| | model-index: |
| | - name: SmolLM2-70M |
| | results: |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: ai2_arc |
| | name: ARC-Easy |
| | config: ARC-Easy |
| | metrics: |
| | - type: acc_norm |
| | value: 33.00 |
| | name: Normalized Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: ai2_arc |
| | name: ARC-Challenge |
| | config: ARC-Challenge |
| | metrics: |
| | - type: acc_norm |
| | value: 22.35 |
| | name: Normalized Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: boolq |
| | name: BoolQ |
| | metrics: |
| | - type: acc |
| | value: 39.66 |
| | name: Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: hellaswag |
| | name: HellaSwag |
| | metrics: |
| | - type: acc_norm |
| | value: 26.14 |
| | name: Normalized Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: piqa |
| | name: PIQA |
| | metrics: |
| | - type: acc_norm |
| | value: 54.84 |
| | name: Normalized Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: sciq |
| | name: SciQ |
| | metrics: |
| | - type: acc_norm |
| | value: 45.20 |
| | name: Normalized Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: winogrande |
| | name: WinoGrande |
| | metrics: |
| | - type: acc |
| | value: 50.04 |
| | name: Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: truthful_qa |
| | name: TruthfulQA MC2 |
| | metrics: |
| | - type: acc |
| | value: 48.02 |
| | name: Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: gsm8k |
| | name: GSM8K |
| | metrics: |
| | - type: exact_match |
| | value: 0.53 |
| | name: Exact Match (5-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: cais/mmlu |
| | name: MMLU |
| | metrics: |
| | - type: acc |
| | value: 22.96 |
| | name: Accuracy (0-shot) |
| | - task: |
| | type: text-generation |
| | name: Text Generation |
| | dataset: |
| | type: openbookqa |
| | name: OpenBookQA |
| | metrics: |
| | - type: acc_norm |
| | value: 27.60 |
| | name: Normalized Accuracy (0-shot) |
| | base_model: HuggingFaceTB/SmolLM2-70M |
| | datasets: |
| | - codelion/sutra-10B |
| | --- |
| | |
| | # SmolLM2-70M |
| |
|
| | A SmolLM2-70M model pretrained on the [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) pedagogical dataset for 3 epochs (~30.6B tokens total). This model demonstrates that a 69M parameter model can be trained to near-capacity performance using dense, curated educational data. |
| |
|
| | ## Model Details |
| |
|
| | | Property | Value | |
| | |----------|-------| |
| | | Architecture | LlamaForCausalLM | |
| | | Parameters | 69.2M | |
| | | Hidden Size | 384 | |
| | | Layers | 32 | |
| | | Attention Heads | 6 (2 KV heads) | |
| | | Context Length | 8,192 | |
| | | Vocabulary | 49,152 | |
| | | Precision | bfloat16 | |
| | | Base Model | [SmolLM2-70M](https://huggingface.co/HuggingFaceTB/SmolLM2-70M) | |
| | | Training Dataset | [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) (10.2B tokens) | |
| |
|
| | ## Training |
| |
|
| | The model was trained for 3 epochs on the Sutra-10B dataset using a single NVIDIA L40S GPU (46GB). This checkpoint is the best perplexity checkpoint from epoch 3. |
| |
|
| | | Epoch | Tokens | Training Time | Learning Rate | Best Perplexity | |
| | |-------|--------|---------------|---------------|-----------------| |
| | | 1 | 10.2B | 25.82h | 3e-4 → 3e-5 | 39.50 | |
| | | 2 | 10.2B | 25.78h | 1e-4 → 1e-5 | 37.81 | |
| | | 3 | 10.2B | 26.16h | 3e-5 → 3e-6 | 37.72 | |
| | | **Total** | **30.6B** | **77.76h** | — | **37.72** | |
| |
|
| | Training configuration: |
| | - Optimizer: AdamW (fused), weight decay 0.1 |
| | - Schedule: Cosine with warmup |
| | - Batch size: 4 per device, gradient accumulation 8 (effective ~262K tokens/step) |
| | - Sequence length: 8,192 |
| | - Flash Attention 2, TF32 matmul, torch.compile |
| | - Throughput: ~110K tokens/sec |
| |
|
| | ## Benchmark Results |
| |
|
| | All benchmarks evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-eval) v0.4.11. All tasks are 0-shot except GSM8K (5-shot). |
| |
|
| | ### This Model vs Training Progression |
| |
|
| | | Benchmark | **E3-best** | E3-final | E2-best | E2-final | E1-final | |
| | |-----------|:-----------:|:--------:|:-------:|:--------:|:--------:| |
| | | ARC-Easy | **33.00** | 33.16 | 32.83 | 33.12 | 33.46 | |
| | | ARC-Challenge | **22.35** | 21.67 | 22.61 | 22.44 | 22.44 | |
| | | BoolQ | **39.66** | 39.66 | 39.79 | 39.54 | 39.79 | |
| | | HellaSwag | **26.14** | 26.03 | 26.08 | 25.91 | 26.03 | |
| | | PIQA | **54.84** | 55.01 | 54.24 | 54.13 | 54.62 | |
| | | SciQ | **45.20** | 46.30 | 44.10 | 45.50 | 43.60 | |
| | | WinoGrande | **50.04** | 49.33 | 50.51 | 48.70 | 48.78 | |
| | | TruthfulQA | **48.02** | 47.93 | 48.30 | 48.14 | 48.30 | |
| | | GSM8K | **0.53** | 0.61 | 0.68 | 0.83 | 0.15 | |
| | | MMLU | **22.96** | 22.87 | 23.00 | 22.98 | 22.99 | |
| | | OpenBookQA | **27.60** | 27.60 | — | — | — | |
| | | **Average (10)** | **34.27** | 34.26 | 34.21 | 34.13 | 34.02 | |
| |
|
| | ### Comparison with 1B Token Baselines (SmolLM2-70M) |
| |
|
| | These are results from training the same SmolLM2-70M model on various 1B-token datasets from the [Pre-training Dataset Samples](https://huggingface.co/collections/codelion/pre-training-dataset-samples-686bd760abf1a43b0ce32829) collection for 1 epoch, showing that Sutra-10B at 3 epochs achieves the highest performance for this model size. |
| |
|
| | | Dataset (1B tokens) | HellaSwag | PIQA | WinoGrande | ARC-C | MMLU | TruthfulQA | GSM8K | Avg | |
| | |---------------------|-----------|------|------------|-------|------|------------|-------|-----| |
| | | **Sutra-10B (3 epochs)** | **26.14** | **54.84** | **50.04** | **22.35** | 22.96 | **48.02** | 0.53 | **34.27** | |
| | | [Sutra-1B](https://huggingface.co/datasets/codelion/sutra-1B) | 25.43 | 53.86 | 49.41 | 23.04 | 22.91 | 49.09 | 1.14 | 32.13 | |
| | | [FineWiki-1B](https://huggingface.co/datasets/HuggingFaceFW/finewiki) | 25.56 | 51.69 | 48.86 | 24.15 | **23.34** | 51.16 | 0.91 | 32.24 | |
| | | [FinePDFs-1B](https://huggingface.co/datasets/HuggingFaceFW/FinePDFs) | 25.58 | 52.56 | 50.51 | 22.44 | 22.95 | 51.41 | 1.21 | 32.38 | |
| | | [DCLM-Baseline-1B](https://huggingface.co/datasets/codelion/dclm-baseline-1B) | 25.85 | 55.17 | 50.20 | 21.08 | 22.97 | 49.21 | 0.68 | 32.16 | |
| | | [FineWeb-Edu-1B](https://huggingface.co/datasets/codelion/fineweb-edu-1B) | 25.72 | 55.11 | 50.36 | 21.25 | 22.96 | 48.11 | 1.21 | 32.10 | |
| | | [Essential-Web-1B](https://huggingface.co/datasets/sumukshashidhar-archive/essential-web-v1.0-sample-1B) | 26.02 | 55.44 | 48.30 | 20.99 | 22.95 | 49.59 | 1.29 | 32.08 | |
| | | [Synth-1B](https://huggingface.co/datasets/codelion/synth-1B) | 26.63 | 50.98 | 48.78 | 21.93 | 23.24 | 47.10 | 1.29 | 31.42 | |
| |
|
| | ## Key Findings |
| |
|
| | 1. **Capacity ceiling**: The 70M parameter model reaches its capacity ceiling at approximately 10B tokens. Additional epochs (up to 30.6B total tokens) yield only marginal improvements in benchmark scores (+0.25 average from epoch 1 to epoch 3), despite continued perplexity improvement (39.50 → 37.72). |
| |
|
| | 2. **Perplexity vs benchmarks**: Perplexity continues to decrease across epochs, but downstream benchmark performance plateaus, suggesting the model's representational capacity is the bottleneck rather than data exposure. |
| |
|
| | 3. **Data quality matters**: Even at 1B tokens, Sutra outperforms or matches larger web-crawled datasets (DCLM, FineWeb-Edu, Essential-Web) on average, demonstrating the value of curated pedagogical content. |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | from transformers import AutoModelForCausalLM, AutoTokenizer |
| | |
| | model = AutoModelForCausalLM.from_pretrained("codelion/SmolLM2-70M", trust_remote_code=True) |
| | tokenizer = AutoTokenizer.from_pretrained("codelion/SmolLM2-70M") |
| | |
| | input_text = "The theory of relativity states that" |
| | inputs = tokenizer(input_text, return_tensors="pt") |
| | outputs = model.generate(**inputs, max_new_tokens=100) |
| | print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| | ``` |
| |
|
| | ## Limitations |
| |
|
| | - This is a 69M parameter base model (not instruction-tuned) — it generates completions, not conversational responses |
| | - Performance is at the capacity ceiling for this model size; larger models would benefit more from the Sutra-10B dataset |
| | - The model was trained primarily on English educational content |
| |
|
| | ## Related Resources |
| |
|
| | - **Dataset**: [codelion/sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) — 10B token pedagogical pretraining dataset |
| | - **Sutra Framework**: Generates structured educational content optimized for LLM pretraining |
| |
|
| | ## Citation |
| |
|
| | ```bibtex |
| | @article{sharma2026sutra, |
| | title={Scaling Pedagogical Pretraining: From Optimal Mixing to 10 Billion Tokens}, |
| | author={Sharma, Asankhaya}, |
| | year={2026}, |
| | url={https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens} |
| | } |
| | ``` |
| |
|
| | ## License |
| |
|
| | Apache 2.0 |
| |
|
