mini-llm-chat-sft

mini-llm-base after instruction tuning: it answers questions. An intermediate checkpoint; mini-llm-chat is the one to use.

Built from scratch in plain PyTorch as a learning project. The code, the training pipeline, the evals and a day-by-day walkthrough are at github.com/a-ramakrishnan/mini_llm.

Model Stage What it is
mini-llm-base 2 text completion: pretrained on 2.5B tokens of Mini FineWeb-Edu + Cosmopedia
mini-llm-chat-sft (this one) 3 the base, instruction-tuned on smol-smoltalk
mini-llm-chat 4 the instruction-tuned model after DPO against repetition: the final chat model

Architecture

Parameters 123,552,768
Layers × width 12 × 768, 12 heads
Context 512 tokens
Positions RoPE (rotary), no position parameters
Feed-forward SwiGLU, hidden width 2,048
Attention QK-norm: queries and keys RMS-normalized per head
Biases none
Normalization LayerNorm, pre-norm
Embeddings input and output tied
Tokenizer GPT-2 BPE (50,257 tokens), via tiktoken
Language English

GPT-2 small's shape with four changes: RoPE, SwiGLU, QK-norm and no biases.

How to use it

This is a custom architecture, loaded with the code in the mini_llm repo. It does not load with transformers.AutoModel yet; a transformers class is planned.

git clone https://github.com/a-ramakrishnan/mini_llm.git
cd mini_llm
pip install -r requirements.txt
python3 scripts/download_weights.py mini-llm-chat-sft
python3 scripts/chat.py "What is the water cycle?" --model mini-llm-chat-sft
python3 scripts/chat.py --model mini-llm-chat-sft        # a conversation
from mini_llm import checkpoint
from mini_llm.chat import format_prompt
from mini_llm.inference import generate_stream
from mini_llm.tokenizer import EOT, get_tokenizer

model, info, device = checkpoint.load("mini-llm-chat-sft")
enc = get_tokenizer()
prompt = format_prompt([{"role": "user", "content": "What is the water cycle?"}], enc)
tokens = [ev["id"] for ev in generate_stream(model, prompt, device, 200, temperature=0.0, stop_token=EOT)]
print(enc.decode(tokens))

The chat format

Plain text, no special chat tokens (smoltalk-plain-v1). Every assistant turn ends with <|endoftext|>:

<|endoftext|>System: Be brief.

User: What is the water cycle?

Assistant: Water evaporates, forms clouds, and falls as rain.<|endoftext|>User: What drives it?

Assistant:

The system line is optional. mini_llm.chat.format_prompt builds this from a list of messages.

Results

Measured with the mini_llm eval suite (Day 7) on an Apple M5.

Held-out loss (lower is better)
Mini FineWeb-Edu 4.264
TinyStories (not trained on) 3.182
WikiText-2 (not trained on) 6.542
Writing
Text completion, greedy: repeated 4-token runs 32.9%
Text completion, sampled: repeated 4-token runs 0.0%
100 unseen questions, greedy: repeated 4-token runs 16.1%
100 unseen questions: answers that end by themselves 73%
Instructions followed, checked by code 12 of 30
Easy factual questions 19 of 40

Zero-shot benchmarks (accuracy; length-normalized for HellaSwag), measured in the source project:

HellaSwag LAMBADA ARC-Easy PIQA Winogrande Average
OpenAI GPT-2 small (124M) 31.1 32.6 43.8 62.9 51.6 44.4
mini-llm-base 32.3 20.3 46.8 62.4 49.9 42.3
mini-llm-chat-sft 30.1 13.5 41.8 59.1 47.9 38.5
mini-llm-chat 30.1 12.9 42.0 59.0 47.9 38.4

Limits

  • It is small. 124M parameters, 2.5B tokens of training. It writes fluent English and is often wrong.
  • Fluency is not knowledge. It will tell you the capital of Egypt is New York City, in a well-formed sentence.
  • It does not follow constraints such as "answer in one word" or "yes or no".
  • It mostly cannot do arithmetic.
  • The context is 512 tokens, and it is English only.
  • Not for anything that matters. It is a model to learn from. It has had no safety tuning beyond what its training data carries.

How it was trained

Stage 1

  • data: Mini FineWeb-Edu (small) + five Cosmopedia subsets (stories, wikihow, openstax, stanford, khanacademy), mixed by size
  • steps: 30,500
  • tokens: 499,712,000
  • batch size: 32
  • context: 512
  • learning rate: 0.0012
  • warmup steps: 1,000
  • hardware: Colab L4, bfloat16
  • time: 4 h 20 min

Stage 2

  • data: Mini FineWeb-Edu (full) + the same Cosmopedia subsets; stage 1's validation documents removed from the training data
  • continued from step: 30,500
  • steps: 152,600
  • tokens: 2,500,198,400
  • hardware: Colab L4 to step 79,000, then A100; four sessions
  • best val loss: 2.9633
  • best val note: on the stage 2 data mix

Instruction tuning (this checkpoint: mini-llm-chat-sft)

  • data: HuggingFaceTB/smol-smoltalk, conversations packed whole into 361,106 windows of 513 tokens; loss on the assistant's tokens only
  • updates: 11,284
  • batch size: 32
  • epochs: 1.0
  • learning rate: 0.0002
  • warmup: 200
  • hardware: Colab A100, bfloat16
  • time: 19 min
  • held out answer loss: 2.316 -> 1.390

Every stage is a script in the mini_llm repo, with a day-by-day walkthrough.

Licences and data

The weights are released under the MIT licence. The training data has its own terms:

Dataset Licence Used for
Mini FineWeb-Edu, a subset of FineWeb-Edu ODC-By 1.0 pretraining
Cosmopedia Apache 2.0 pretraining
smol-smoltalk Apache 2.0 instruction tuning, DPO pairs

Files

File What it is
model.safetensors the weights; the output layer shares the token embedding matrix, stored once
model_config.json the model's configuration under "model", and a training summary
Downloads last month

-

Downloads are not tracked for this model. How to track
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for aramkris/mini-llm-chat-sft

Finetuned
(1)
this model
Finetunes
1 model

Datasets used to train aramkris/mini-llm-chat-sft