File size: 2,952 Bytes
1bc6993 afb3d19 4764731 afb3d19 4764731 afb3d19 4764731 afb3d19 1bc6993 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | ---
license: mit
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smollm-corpus
pipeline_tag: text-generation
library_name: transformers
---
# TinyMixtral 1B — Post-Trained
A 1.18B-parameter Mixture-of-Experts language model (351M active), post-trained on 1B tokens of educational and web text.
## Model Details
| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer with Sparse MoE |
| Total Parameters | 1,182,172,160 |
| Active Parameters | ~351M |
| Hidden Size | 1024 |
| Layers | 16 |
| Experts | 8 (top-2 routing) |
| Attention Heads | 16 query / 4 key-value (GQA) |
| Head Dimension | 64 |
| Intermediate Size | 2,816 (per expert) |
| Vocabulary | 32,000 |
| Context Length | 2,048 |
| Position Encoding | RoPE (theta=1e6) |
| Activation | SiLU |
| Norm | RMSNorm |
| Tied Embeddings | Yes |
## Training
**Pre-training (4B tokens):**
- Data: FineWeb-Edu + Cosmopedia (89:11 blend)
- Schedule: WSD (warmup 2,000 → stable → decay)
- Peak LR: 7e-4
- Batch: 16 × 1,024 = 16,384 tokens/step
- Steps: 244,141
- Duration: ~102.5 hours
**Post-training (1B tokens):**
- Data: FineWeb-Edu + Cosmopedia continuation (second 1B slice)
- Schedule: WSD (warmup 2,000 → stable → decay)
- Peak LR: 2e-5
- Batch: 16 × 1,024
- Steps: 60,975
- Duration: ~25.9 hours
## Benchmark Results
### Harness (0-shot)
| Benchmark | Score |
|---|---|
| HellaSwag (acc_norm) | 0.313 |
| PIQA (acc) | 0.609 |
| Winogrande (acc) | 0.505 |
| ARC-Easy (acc_norm) | 0.410 |
| ARC-Challenge (acc_norm) | 0.272 |
| OpenBookQA (acc_norm) | 0.290 |
| BoolQ (acc) | 0.528 |
| LAMBADA (acc) | 0.195 |
### IFEval (instruction-following)
| Model | inst_strict |
|---|---|
| **1B post-train** | **0.2338** |
| v1.1 | 0.2182 |
### SAMSum (Dialogue Summarization)
| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| **1B (0-shot)** | 9.83 | 0.50 | 7.85 |
| **1B (fine-tuned, 15ep)** | **28.82** | **8.55** | **24.08** |
| T5-small (60M) | 35.7 | 13.4 | 31.4 |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"publish_posttrain/",
trust_remote_code=True,
torch_dtype="bfloat16",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("publish_posttrain/", legacy=False)
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
## Limitations
- **Data budget:** Trained on only 4B tokens (pre-training) + 1B tokens (post-training). Comparable models use 100-1000× more data.
- **Reasoning:** Limited multi-step reasoning and mathematical capability.
- **Hallucination:** May generate plausible but incorrect facts.
- **Context:** Effective context is shorter than the 2,048-token window.
## License
MIT License. See [LICENSE](LICENSE) for details. |