pruned_olmo3_5120_24_32
WARNING: This model is PRUNED ONLY, NOT retrained or distilled!
Performance will be degraded compared to the original model. This is a structural pruning checkpoint intended as a starting point for knowledge distillation or fine-tuning.
Description
Structurally pruned version of allenai/OLMo-3-7B-Instruct.
Pruning Configuration
| Parameter | Original | Pruned |
|---|---|---|
| Intermediate size (MLP) | 11008 | 5120 |
| Attention heads | 32 | 24 |
| Layers | 32 | 32 |
| Hidden size | 4096 | 4096 (unchanged) |
Important Notes
- This model has NOT been retrained after pruning
- Performance will be significantly degraded compared to the original
- Intended use: Starting checkpoint for distillation/fine-tuning
- For the distillation training data, see hbfreed/dolci-distill-packed
- Downloads last month
- 13