| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - llama |
| - pretrained |
| - from-scratch |
| - gpuburnout |
| pipeline_tag: text-generation |
| --- |
| |
| # GPUburnout-3B-75K |
|
|
| A 3.12 billion parameter Llama-style decoder-only transformer, pretrained from scratch as the Season 4 model in the GPUburnout blog series. Final pretraining checkpoint at 75,000 steps. |
|
|
| This is the base model, before any instruction tuning. For the chat-tuned versions, see [GPUburnout/GPUburnout-3B-75K-Chat](https://huggingface.co/GPUburnout/GPUburnout-3B-75K-Chat) (the shipped SFT champion). |
|
|
| ## Model Details |
|
|
| - **Architecture:** Llama-style decoder-only transformer |
| - **Parameters:** 3.12B |
| - **Hidden size:** 2,560 |
| - **Intermediate size:** 10,240 |
| - **Layers:** 32 |
| - **Attention heads:** 40 query, 10 key/value (GQA) |
| - **Head dim:** 64 |
| - **Vocab size:** 32,005 |
| - **Max position:** 2,048 |
| - **Tie word embeddings:** True |
| - **Precision:** float16 |
|
|
| ## Training |
|
|
| - **Steps:** 75,000 |
| - **Final val loss:** 2.2475 |
| - **Training cost:** ~$425 (RunPod A100/H200) |
| - **Pretraining data mix:** FineWeb-Edu, FineMath, Stack-Edu-Python, PubMed abstracts |
| - **Optimizer:** 8-bit AdamW |
| - **Hardware:** Mixed A100 SXM 80GB and H200 NVL across training run |
|
|
| ## Benchmarks (0-shot, float16) |
|
|
| | Benchmark | Score | |
| |---|---| |
| | TruthfulQA MC2 | 47.61% | |
| | HellaSwag (acc_norm) | 28.30% | |
| | ARC-Easy (acc_norm) | 43.06% | |
| | ARC-Challenge (acc_norm) | 21.84% | |
| | MMLU (5-shot acc) | 23.02% | |
| |
| These are typical small-model-on-limited-tokens numbers. The model has not seen enough data to develop deep academic knowledge (MMLU is near random). It absorbed enough factual content for ARC-Easy to land ~2x random and for TruthfulQA to score above random on calibrated truthfulness, which is the strongest signal at this scale. |
| |
| ## Related Models |
| |
| - [GPUburnout/GPUburnout-3B-75K-Chat](https://huggingface.co/GPUburnout/GPUburnout-3B-75K-Chat) - SFT champion (step 1500, lr=5e-5, r=16) |
| - [GPUburnout/GPUburnout-3B-75K-Chat-step3000](https://huggingface.co/GPUburnout/GPUburnout-3B-75K-Chat-step3000) - Alternative SFT checkpoint (lowest val loss, not shipped) |
| - [GPUburnout/GPUburnout-3B-checkpoints](https://huggingface.co/GPUburnout/GPUburnout-3B-checkpoints) - Intermediate pretraining checkpoints |
| |
| ## Blog |
| |
| This model is part of the GPUburnout LLM-from-scratch blog series. Season 4 documents the 3B build, including the platform pivot from Thunder to RunPod, MooseFS storage debugging, and the 75K-step pretraining run. |
| |
| https://gpuburnout.com |
| |
| ## License |
| |
| Apache 2.0. Free to use, modify, redistribute. Attribution appreciated. |
| |
