---
license: mit
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smollm-corpus
pipeline_tag: text-generation
library_name: transformers
---
# TinyMixtral 1B — Post-Trained

A 1.18B-parameter Mixture-of-Experts language model (351M active), post-trained on 1B tokens of educational and web text.

## Model Details

| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer with Sparse MoE |
| Total Parameters | 1,182,172,160 |
| Active Parameters | ~351M |
| Hidden Size | 1024 |
| Layers | 16 |
| Experts | 8 (top-2 routing) |
| Attention Heads | 16 query / 4 key-value (GQA) |
| Head Dimension | 64 |
| Intermediate Size | 2,816 (per expert) |
| Vocabulary | 32,000 |
| Context Length | 2,048 |
| Position Encoding | RoPE (theta=1e6) |
| Activation | SiLU |
| Norm | RMSNorm |
| Tied Embeddings | Yes |

## Training

**Pre-training (4B tokens):**
- Data: FineWeb-Edu + Cosmopedia (89:11 blend)
- Schedule: WSD (warmup 2,000 → stable → decay)
- Peak LR: 7e-4
- Batch: 16 × 1,024 = 16,384 tokens/step
- Steps: 244,141
- Duration: ~102.5 hours

**Post-training (1B tokens):**
- Data: FineWeb-Edu + Cosmopedia continuation (second 1B slice)
- Schedule: WSD (warmup 2,000 → stable → decay)
- Peak LR: 2e-5
- Batch: 16 × 1,024
- Steps: 60,975
- Duration: ~25.9 hours

## Benchmark Results

### Harness (0-shot)

| Benchmark | Score |
|---|---|
| HellaSwag (acc_norm) | 0.313 |
| PIQA (acc) | 0.609 |
| Winogrande (acc) | 0.505 |
| ARC-Easy (acc_norm) | 0.410 |
| ARC-Challenge (acc_norm) | 0.272 |
| OpenBookQA (acc_norm) | 0.290 |
| BoolQ (acc) | 0.528 |
| LAMBADA (acc) | 0.195 |

### IFEval (instruction-following)

| Model | inst_strict |
|---|---|
| **1B post-train** | **0.2338** |
| v1.1 | 0.2182 |

### SAMSum (Dialogue Summarization)

| Model | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| **1B (0-shot)** | 9.83 | 0.50 | 7.85 |
| **1B (fine-tuned, 15ep)** | **28.82** | **8.55** | **24.08** |
| T5-small (60M) | 35.7 | 13.4 | 31.4 |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "publish_posttrain/",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("publish_posttrain/", legacy=False)

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Limitations

- **Data budget:** Trained on only 4B tokens (pre-training) + 1B tokens (post-training). Comparable models use 100-1000× more data.
- **Reasoning:** Limited multi-step reasoning and mathematical capability.
- **Hallucination:** May generate plausible but incorrect facts.
- **Context:** Effective context is shorter than the 2,048-token window.

## License

MIT License. See [LICENSE](LICENSE) for details.