Upload folder using huggingface_hub

README.md

@@ -0,0 +1,243 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- pretraining
+- educational
+- pedagogical
+- sutra
+- smollm2
+- llama
+pipeline_tag: text-generation
+model-index:
+- name: SmolLM2-70M
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: ai2_arc
+      name: ARC-Easy
+      config: ARC-Easy
+    metrics:
+    - type: acc_norm
+      value: 33.00
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: ai2_arc
+      name: ARC-Challenge
+      config: ARC-Challenge
+    metrics:
+    - type: acc_norm
+      value: 22.35
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: boolq
+      name: BoolQ
+    metrics:
+    - type: acc
+      value: 39.66
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: hellaswag
+      name: HellaSwag
+    metrics:
+    - type: acc_norm
+      value: 26.14
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: piqa
+      name: PIQA
+    metrics:
+    - type: acc_norm
+      value: 54.84
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: sciq
+      name: SciQ
+    metrics:
+    - type: acc_norm
+      value: 45.20
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: winogrande
+      name: WinoGrande
+    metrics:
+    - type: acc
+      value: 50.04
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: truthful_qa
+      name: TruthfulQA MC2
+    metrics:
+    - type: acc
+      value: 48.02
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: gsm8k
+      name: GSM8K
+    metrics:
+    - type: exact_match
+      value: 0.53
+      name: Exact Match (5-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: cais/mmlu
+      name: MMLU
+    metrics:
+    - type: acc
+      value: 22.96
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: openbookqa
+      name: OpenBookQA
+    metrics:
+    - type: acc_norm
+      value: 27.60
+      name: Normalized Accuracy (0-shot)
+base_model: HuggingFaceTB/SmolLM2-70M
+datasets:
+- codelion/sutra-10B
+---
+
+# SmolLM2-70M
+
+A SmolLM2-70M model pretrained on the [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) pedagogical dataset for 3 epochs (~30.6B tokens total). This model demonstrates that a 69M parameter model can be trained to near-capacity performance using dense, curated educational data.
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| Architecture | LlamaForCausalLM |
+| Parameters | 69.2M |
+| Hidden Size | 384 |
+| Layers | 32 |
+| Attention Heads | 6 (2 KV heads) |
+| Context Length | 8,192 |
+| Vocabulary | 49,152 |
+| Precision | bfloat16 |
+| Base Model | [SmolLM2-70M](https://huggingface.co/HuggingFaceTB/SmolLM2-70M) |
+| Training Dataset | [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) (10.2B tokens) |
+
+## Training
+
+The model was trained for 3 epochs on the Sutra-10B dataset using a single NVIDIA L40S GPU (46GB). This checkpoint is the best perplexity checkpoint from epoch 3.
+
+| Epoch | Tokens | Training Time | Learning Rate | Best Perplexity |
+|-------|--------|---------------|---------------|-----------------|
+| 1 | 10.2B | 25.82h | 3e-4 → 3e-5 | 39.50 |
+| 2 | 10.2B | 25.78h | 1e-4 → 1e-5 | 37.81 |
+| 3 | 10.2B | 26.16h | 3e-5 → 3e-6 | 37.72 |
+| **Total** | **30.6B** | **77.76h** | — | **37.72** |
+
+Training configuration:
+- Optimizer: AdamW (fused), weight decay 0.1
+- Schedule: Cosine with warmup
+- Batch size: 4 per device, gradient accumulation 8 (effective ~262K tokens/step)
+- Sequence length: 8,192
+- Flash Attention 2, TF32 matmul, torch.compile
+- Throughput: ~110K tokens/sec
+
+## Benchmark Results
+
+All benchmarks evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-eval) v0.4.11. All tasks are 0-shot except GSM8K (5-shot).
+
+### This Model vs Training Progression
+
+| Benchmark | **E3-best** | E3-final | E2-best | E2-final | E1-final |
+|-----------|:-----------:|:--------:|:-------:|:--------:|:--------:|
+| ARC-Easy | **33.00** | 33.16 | 32.83 | 33.12 | 33.46 |
+| ARC-Challenge | **22.35** | 21.67 | 22.61 | 22.44 | 22.44 |
+| BoolQ | **39.66** | 39.66 | 39.79 | 39.54 | 39.79 |
+| HellaSwag | **26.14** | 26.03 | 26.08 | 25.91 | 26.03 |
+| PIQA | **54.84** | 55.01 | 54.24 | 54.13 | 54.62 |
+| SciQ | **45.20** | 46.30 | 44.10 | 45.50 | 43.60 |
+| WinoGrande | **50.04** | 49.33 | 50.51 | 48.70 | 48.78 |
+| TruthfulQA | **48.02** | 47.93 | 48.30 | 48.14 | 48.30 |
+| GSM8K | **0.53** | 0.61 | 0.68 | 0.83 | 0.15 |
+| MMLU | **22.96** | 22.87 | 23.00 | 22.98 | 22.99 |
+| OpenBookQA | **27.60** | 27.60 | — | — | — |
+| **Average (10)** | **34.27** | 34.26 | 34.21 | 34.13 | 34.02 |
+
+### Comparison with 1B Token Baselines (SmolLM2-70M)
+
+These are results from training the same SmolLM2-70M model on various 1B-token datasets for 1 epoch, showing that Sutra-10B at 3 epochs achieves the highest performance for this model size.
+
+| Dataset (1B tokens) | HellaSwag | PIQA | WinoGrande | ARC-C | MMLU | TruthfulQA | GSM8K | Avg |
+|---------------------|-----------|------|------------|-------|------|------------|-------|-----|
+| **Sutra-10B (3 epochs)** | **26.14** | **54.84** | **50.04** | **22.35** | 22.96 | **48.02** | 0.53 | **34.27** |
+| Sutra-1B | 25.43 | 53.86 | 49.41 | 23.04 | 22.91 | 49.09 | 1.14 | 32.13 |
+| FineWiki-1B | 25.56 | 51.69 | 48.86 | 24.15 | **23.34** | 51.16 | 0.91 | 32.24 |
+| FinePDFs-1B | 25.58 | 52.56 | 50.51 | 22.44 | 22.95 | 51.41 | 1.21 | 32.38 |
+| DCLM-Baseline-1B | 25.85 | 55.17 | 50.20 | 21.08 | 22.97 | 49.21 | 0.68 | 32.16 |
+| FineWeb-Edu-1B | 25.72 | 55.11 | 50.36 | 21.25 | 22.96 | 48.11 | 1.21 | 32.10 |
+| Essential-Web-1B | 26.02 | 55.44 | 48.30 | 20.99 | 22.95 | 49.59 | 1.29 | 32.08 |
+| Synth-1B | 26.63 | 50.98 | 48.78 | 21.93 | 23.24 | 47.10 | 1.29 | 31.42 |
+
+## Key Findings
+
+1. **Capacity ceiling**: The 70M parameter model reaches its capacity ceiling at approximately 10B tokens. Additional epochs (up to 30.6B total tokens) yield only marginal improvements in benchmark scores (+0.25 average from epoch 1 to epoch 3), despite continued perplexity improvement (39.50 → 37.72).
+
+2. **Perplexity vs benchmarks**: Perplexity continues to decrease across epochs, but downstream benchmark performance plateaus, suggesting the model's representational capacity is the bottleneck rather than data exposure.
+
+3. **Data quality matters**: Even at 1B tokens, Sutra outperforms or matches larger web-crawled datasets (DCLM, FineWeb-Edu, Essential-Web) on average, demonstrating the value of curated pedagogical content.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("codelion/SmolLM2-70M", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("codelion/SmolLM2-70M")
+
+input_text = "The theory of relativity states that"
+inputs = tokenizer(input_text, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=100)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Limitations
+
+- This is a 69M parameter base model (not instruction-tuned) — it generates completions, not conversational responses
+- Performance is at the capacity ceiling for this model size; larger models would benefit more from the Sutra-10B dataset
+- The model was trained primarily on English educational content
+
+## Related Resources
+
+- **Dataset**: [codelion/sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) — 10B token pedagogical pretraining dataset
+- **Sutra Framework**: Generates structured educational content optimized for LLM pretraining
+
+## License
+
+Apache 2.0

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 384,
+  "initializer_range": 0.041666666666666664,
+  "intermediate_size": 1024,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 6,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 100000,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 49152
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.57.6"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e777a572b1103d8b91543c1e2bdb632d1aeec9bb3879fac90a27b4ce45c92a17
+size 138494280

special_tokens_map.json

@@ -0,0 +1,42 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,168 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 8192,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}