| language: en | |
| license: apache-2.0 | |
| tags: | |
| - language-model | |
| - from-scratch | |
| - dense-transformer | |
| # HobbyLM-30M | |
| A 31.9M parameter dense transformer trained from scratch on 1B tokens of FineWeb. | |
| Built on top of [HobbyLM](https://github.com/harishsg993010/HobbyLM) by rootxhacker. | |
| ## Training | |
| - **Parameters:** 31.9M (fully dense) | |
| - **Dataset:** FineWeb (1B tokens) | |
| - **Steps:** 3800 | |
| - **Final val loss:** 3.9077 | |
| - **Architecture:** 8 layers, d_model=384, 6 heads, GQA, RoPE, RMSNorm, Muon optimizer | |
| ## This is a base model | |
| No instruction tuning. Generates fluent English but drifts off topic — expected at this scale. | |