---
language: en
license: apache-2.0
tags:
- language-model
- from-scratch
- dense-transformer
---

# HobbyLM-30M

A 31.9M parameter dense transformer trained from scratch on 1B tokens of FineWeb.

Built on top of [HobbyLM](https://github.com/harishsg993010/HobbyLM) by rootxhacker.

## Training
- **Parameters:** 31.9M (fully dense)
- **Dataset:** FineWeb (1B tokens)
- **Steps:** 3800
- **Final val loss:** 3.9077
- **Architecture:** 8 layers, d_model=384, 6 heads, GQA, RoPE, RMSNorm, Muon optimizer

## This is a base model
No instruction tuning. Generates fluent English but drifts off topic — expected at this scale.