mini-llm-chat-sft
mini-llm-base after instruction tuning: it answers questions. An intermediate checkpoint; mini-llm-chat is the one to use.
Built from scratch in plain PyTorch as a learning project. The code, the training pipeline, the evals and a day-by-day walkthrough are at github.com/a-ramakrishnan/mini_llm.
| Model | Stage | What it is |
|---|---|---|
| mini-llm-base | 2 | text completion: pretrained on 2.5B tokens of Mini FineWeb-Edu + Cosmopedia |
| mini-llm-chat-sft (this one) | 3 | the base, instruction-tuned on smol-smoltalk |
| mini-llm-chat | 4 | the instruction-tuned model after DPO against repetition: the final chat model |
Architecture
| Parameters | 123,552,768 |
| Layers × width | 12 × 768, 12 heads |
| Context | 512 tokens |
| Positions | RoPE (rotary), no position parameters |
| Feed-forward | SwiGLU, hidden width 2,048 |
| Attention | QK-norm: queries and keys RMS-normalized per head |
| Biases | none |
| Normalization | LayerNorm, pre-norm |
| Embeddings | input and output tied |
| Tokenizer | GPT-2 BPE (50,257 tokens), via tiktoken |
| Language | English |
GPT-2 small's shape with four changes: RoPE, SwiGLU, QK-norm and no biases.
How to use it
This is a custom architecture, loaded with the code in the mini_llm repo. It does not load with
transformers.AutoModel yet; a transformers class is planned.
git clone https://github.com/a-ramakrishnan/mini_llm.git
cd mini_llm
pip install -r requirements.txt
python3 scripts/download_weights.py mini-llm-chat-sft
python3 scripts/chat.py "What is the water cycle?" --model mini-llm-chat-sft
python3 scripts/chat.py --model mini-llm-chat-sft # a conversation
from mini_llm import checkpoint
from mini_llm.chat import format_prompt
from mini_llm.inference import generate_stream
from mini_llm.tokenizer import EOT, get_tokenizer
model, info, device = checkpoint.load("mini-llm-chat-sft")
enc = get_tokenizer()
prompt = format_prompt([{"role": "user", "content": "What is the water cycle?"}], enc)
tokens = [ev["id"] for ev in generate_stream(model, prompt, device, 200, temperature=0.0, stop_token=EOT)]
print(enc.decode(tokens))
The chat format
Plain text, no special chat tokens (smoltalk-plain-v1). Every assistant turn ends with <|endoftext|>:
<|endoftext|>System: Be brief.
User: What is the water cycle?
Assistant: Water evaporates, forms clouds, and falls as rain.<|endoftext|>User: What drives it?
Assistant:
The system line is optional. mini_llm.chat.format_prompt builds this from a list of messages.
Results
Measured with the mini_llm eval suite (Day 7) on an Apple M5.
| Held-out loss (lower is better) | |
|---|---|
| Mini FineWeb-Edu | 4.264 |
| TinyStories (not trained on) | 3.182 |
| WikiText-2 (not trained on) | 6.542 |
| Writing | |
|---|---|
| Text completion, greedy: repeated 4-token runs | 32.9% |
| Text completion, sampled: repeated 4-token runs | 0.0% |
| 100 unseen questions, greedy: repeated 4-token runs | 16.1% |
| 100 unseen questions: answers that end by themselves | 73% |
| Instructions followed, checked by code | 12 of 30 |
| Easy factual questions | 19 of 40 |
Zero-shot benchmarks (accuracy; length-normalized for HellaSwag), measured in the source project:
| HellaSwag | LAMBADA | ARC-Easy | PIQA | Winogrande | Average | |
|---|---|---|---|---|---|---|
| OpenAI GPT-2 small (124M) | 31.1 | 32.6 | 43.8 | 62.9 | 51.6 | 44.4 |
| mini-llm-base | 32.3 | 20.3 | 46.8 | 62.4 | 49.9 | 42.3 |
| mini-llm-chat-sft | 30.1 | 13.5 | 41.8 | 59.1 | 47.9 | 38.5 |
| mini-llm-chat | 30.1 | 12.9 | 42.0 | 59.0 | 47.9 | 38.4 |
Limits
- It is small. 124M parameters, 2.5B tokens of training. It writes fluent English and is often wrong.
- Fluency is not knowledge. It will tell you the capital of Egypt is New York City, in a well-formed sentence.
- It does not follow constraints such as "answer in one word" or "yes or no".
- It mostly cannot do arithmetic.
- The context is 512 tokens, and it is English only.
- Not for anything that matters. It is a model to learn from. It has had no safety tuning beyond what its training data carries.
How it was trained
Stage 1
- data: Mini FineWeb-Edu (small) + five Cosmopedia subsets (stories, wikihow, openstax, stanford, khanacademy), mixed by size
- steps: 30,500
- tokens: 499,712,000
- batch size: 32
- context: 512
- learning rate: 0.0012
- warmup steps: 1,000
- hardware: Colab L4, bfloat16
- time: 4 h 20 min
Stage 2
- data: Mini FineWeb-Edu (full) + the same Cosmopedia subsets; stage 1's validation documents removed from the training data
- continued from step: 30,500
- steps: 152,600
- tokens: 2,500,198,400
- hardware: Colab L4 to step 79,000, then A100; four sessions
- best val loss: 2.9633
- best val note: on the stage 2 data mix
Instruction tuning (this checkpoint: mini-llm-chat-sft)
- data: HuggingFaceTB/smol-smoltalk, conversations packed whole into 361,106 windows of 513 tokens; loss on the assistant's tokens only
- updates: 11,284
- batch size: 32
- epochs: 1.0
- learning rate: 0.0002
- warmup: 200
- hardware: Colab A100, bfloat16
- time: 19 min
- held out answer loss: 2.316 -> 1.390
Every stage is a script in the mini_llm repo, with a day-by-day walkthrough.
Licences and data
The weights are released under the MIT licence. The training data has its own terms:
| Dataset | Licence | Used for |
|---|---|---|
| Mini FineWeb-Edu, a subset of FineWeb-Edu | ODC-By 1.0 | pretraining |
| Cosmopedia | Apache 2.0 | pretraining |
| smol-smoltalk | Apache 2.0 | instruction tuning, DPO pairs |
Files
| File | What it is |
|---|---|
model.safetensors |
the weights; the output layer shares the token embedding matrix, stored once |
model_config.json |
the model's configuration under "model", and a training summary |