mini-llm-base
A 124M-parameter language model pretrained from scratch on 2.5B tokens of educational web text. It continues text; it does not answer questions.
Built from scratch in plain PyTorch as a learning project. The code, the training pipeline, the evals and a day-by-day walkthrough are at github.com/a-ramakrishnan/mini_llm.
| Model | Stage | What it is |
|---|---|---|
| mini-llm-base (this one) | 2 | text completion: pretrained on 2.5B tokens of Mini FineWeb-Edu + Cosmopedia |
| mini-llm-chat-sft | 3 | the base, instruction-tuned on smol-smoltalk |
| mini-llm-chat | 4 | the instruction-tuned model after DPO against repetition: the final chat model |
Architecture
| Parameters | 123,552,768 |
| Layers × width | 12 × 768, 12 heads |
| Context | 512 tokens |
| Positions | RoPE (rotary), no position parameters |
| Feed-forward | SwiGLU, hidden width 2,048 |
| Attention | QK-norm: queries and keys RMS-normalized per head |
| Biases | none |
| Normalization | LayerNorm, pre-norm |
| Embeddings | input and output tied |
| Tokenizer | GPT-2 BPE (50,257 tokens), via tiktoken |
| Language | English |
GPT-2 small's shape with four changes: RoPE, SwiGLU, QK-norm and no biases.
How to use it
This is a custom architecture, loaded with the code in the mini_llm repo. It does not load with
transformers.AutoModel yet; a transformers class is planned.
git clone https://github.com/a-ramakrishnan/mini_llm.git
cd mini_llm
pip install -r requirements.txt
python3 scripts/download_weights.py mini-llm-base
python3 scripts/chat.py --complete "The water cycle is the process by which" --model mini-llm-base
from mini_llm import checkpoint
from mini_llm.inference import PRESETS, generate_stream
from mini_llm.tokenizer import EOT, get_tokenizer
model, info, device = checkpoint.load("mini-llm-base")
enc = get_tokenizer()
prompt = [EOT] + enc.encode_ordinary("The water cycle is the process by which")
s = dict(PRESETS["balanced"])
tokens = [ev["id"] for ev in generate_stream(model, prompt, device, 100, s.pop("temperature"), stop_token=EOT, sampling=s)]
print(enc.decode(tokens))
Results
Measured with the mini_llm eval suite (Day 7) on an Apple M5.
| Held-out loss (lower is better) | |
|---|---|
| Mini FineWeb-Edu | 3.296 |
| TinyStories (not trained on) | 2.751 |
| WikiText-2 (not trained on) | 5.047 |
| Writing | |
|---|---|
| Text completion, greedy: repeated 4-token runs | 54.8% |
| Text completion, sampled: repeated 4-token runs | 1.0% |
| Instructions followed, checked by code | 6 of 30 |
| Easy factual questions | 11 of 40 |
Zero-shot benchmarks (accuracy; length-normalized for HellaSwag), measured in the source project:
| HellaSwag | LAMBADA | ARC-Easy | PIQA | Winogrande | Average | |
|---|---|---|---|---|---|---|
| OpenAI GPT-2 small (124M) | 31.1 | 32.6 | 43.8 | 62.9 | 51.6 | 44.4 |
| mini-llm-base | 32.3 | 20.3 | 46.8 | 62.4 | 49.9 | 42.3 |
| mini-llm-chat-sft | 30.1 | 13.5 | 41.8 | 59.1 | 47.9 | 38.5 |
| mini-llm-chat | 30.1 | 12.9 | 42.0 | 59.0 | 47.9 | 38.4 |
Limits
- It is small. 124M parameters, 2.5B tokens of training. It writes fluent English and is often wrong.
- Fluency is not knowledge. It will tell you the capital of Egypt is New York City, in a well-formed sentence.
- It does not follow constraints such as "answer in one word" or "yes or no".
- It mostly cannot do arithmetic.
- The context is 512 tokens, and it is English only.
- Not for anything that matters. It is a model to learn from. It has had no safety tuning beyond what its training data carries.
How it was trained
Stage 1
- data: Mini FineWeb-Edu (small) + five Cosmopedia subsets (stories, wikihow, openstax, stanford, khanacademy), mixed by size
- steps: 30,500
- tokens: 499,712,000
- batch size: 32
- context: 512
- learning rate: 0.0012
- warmup steps: 1,000
- hardware: Colab L4, bfloat16
- time: 4 h 20 min
Stage 2
- data: Mini FineWeb-Edu (full) + the same Cosmopedia subsets; stage 1's validation documents removed from the training data
- continued from step: 30,500
- steps: 152,600
- tokens: 2,500,198,400
- hardware: Colab L4 to step 79,000, then A100; four sessions
- best val loss: 2.9633
- best val note: on the stage 2 data mix
Every stage is a script in the mini_llm repo, with a day-by-day walkthrough.
Licences and data
The weights are released under the MIT licence. The training data has its own terms:
| Dataset | Licence | Used for |
|---|---|---|
| Mini FineWeb-Edu, a subset of FineWeb-Edu | ODC-By 1.0 | pretraining |
| Cosmopedia | Apache 2.0 | pretraining |
| smol-smoltalk | Apache 2.0 | instruction tuning, DPO pairs |
Files
| File | What it is |
|---|---|
model.safetensors |
the weights; the output layer shares the token embedding matrix, stored once |
model_config.json |
the model's configuration under "model", and a training summary |