| --- |
| license: mit |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| - HuggingFaceTB/smollm-corpus |
| pipeline_tag: text-generation |
| library_name: transformers |
| --- |
| # TinyMixtral 1B — Post-Trained |
|
|
| A 1.18B-parameter Mixture-of-Experts language model (351M active), post-trained on 1B tokens of educational and web text. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | Architecture | Decoder-only Transformer with Sparse MoE | |
| | Total Parameters | 1,182,172,160 | |
| | Active Parameters | ~351M | |
| | Hidden Size | 1024 | |
| | Layers | 16 | |
| | Experts | 8 (top-2 routing) | |
| | Attention Heads | 16 query / 4 key-value (GQA) | |
| | Head Dimension | 64 | |
| | Intermediate Size | 2,816 (per expert) | |
| | Vocabulary | 32,000 | |
| | Context Length | 2,048 | |
| | Position Encoding | RoPE (theta=1e6) | |
| | Activation | SiLU | |
| | Norm | RMSNorm | |
| | Tied Embeddings | Yes | |
|
|
| ## Training |
|
|
| **Pre-training (4B tokens):** |
| - Data: FineWeb-Edu + Cosmopedia (89:11 blend) |
| - Schedule: WSD (warmup 2,000 → stable → decay) |
| - Peak LR: 7e-4 |
| - Batch: 16 × 1,024 = 16,384 tokens/step |
| - Steps: 244,141 |
| - Duration: ~102.5 hours |
|
|
| **Post-training (1B tokens):** |
| - Data: FineWeb-Edu + Cosmopedia continuation (second 1B slice) |
| - Schedule: WSD (warmup 2,000 → stable → decay) |
| - Peak LR: 2e-5 |
| - Batch: 16 × 1,024 |
| - Steps: 60,975 |
| - Duration: ~25.9 hours |
|
|
| ## Benchmark Results |
|
|
| ### Harness (0-shot) |
|
|
| | Benchmark | Score | |
| |---|---| |
| | HellaSwag (acc_norm) | 0.313 | |
| | PIQA (acc) | 0.609 | |
| | Winogrande (acc) | 0.505 | |
| | ARC-Easy (acc_norm) | 0.410 | |
| | ARC-Challenge (acc_norm) | 0.272 | |
| | OpenBookQA (acc_norm) | 0.290 | |
| | BoolQ (acc) | 0.528 | |
| | LAMBADA (acc) | 0.195 | |
|
|
| ### IFEval (instruction-following) |
|
|
| | Model | inst_strict | |
| |---|---| |
| | **1B post-train** | **0.2338** | |
| | v1.1 | 0.2182 | |
| |
| ### SAMSum (Dialogue Summarization) |
| |
| | Model | ROUGE-1 | ROUGE-2 | ROUGE-L | |
| |---|---|---|---| |
| | **1B (0-shot)** | 9.83 | 0.50 | 7.85 | |
| | **1B (fine-tuned, 15ep)** | **28.82** | **8.55** | **24.08** | |
| | T5-small (60M) | 35.7 | 13.4 | 31.4 | |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "publish_posttrain/", |
| trust_remote_code=True, |
| torch_dtype="bfloat16", |
| device_map="auto", |
| ) |
| tokenizer = AutoTokenizer.from_pretrained("publish_posttrain/", legacy=False) |
| |
| prompt = "The capital of France is" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| output = model.generate(**inputs, max_new_tokens=20, do_sample=False) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
| |
| ## Limitations |
| |
| - **Data budget:** Trained on only 4B tokens (pre-training) + 1B tokens (post-training). Comparable models use 100-1000× more data. |
| - **Reasoning:** Limited multi-step reasoning and mathematical capability. |
| - **Hallucination:** May generate plausible but incorrect facts. |
| - **Context:** Effective context is shorter than the 2,048-token window. |
| |
| ## License |
| |
| MIT License. See [LICENSE](LICENSE) for details. |
