Haiku H2
Haiku H2 is the canonical successor to Haiku Mini — a 217M-parameter small language model from Rootcomputer. It features a modernized architecture, a complete SFT + DPO post-training pipeline, and a published benchmark profile that wins its weight class.
H2 is available for public testing in open beta via chathaiku.com. It is an experimental research model, designed to make small-model behavior studyable end-to-end, validate training pipelines, and publish reproducible benchmarks against comparable open small models.
Model Specifications
H2 is a decoder-only transformer architecture optimized to keep parameter counts low without sacrificing capacity.
| Specification | Value |
|---|---|
| Parameters | 217M |
| Transformer Layers | 16 |
| Embedding Dimension | 1,024 |
| Attention Mechanism | Grouped-Query Attention (GQA) with 4 KV heads |
| Positional Base | 50k RoPE (Rotary Positional Embeddings) |
| Activation Function | SwiGLU |
| Embeddings | Tied Input/Output Embeddings |
| Precision & Scaling | bf16 (Flash Attention 2 supported) |
Training Pipeline
H2 is trained across three distinct stages. Each stage is validated independently against a stable upstream checkpoint to allow cheap, rapid iteration on commodity hardware.
- Pretrain: Base language modeling executed on the Rootcomputer multi-source corpus, comprising web text, books, news, scientific papers, and biomedical sources.
- SFT (Supervised Fine-Tuning): Conversational tuning that teaches the model the shape of instructions, responses, refusals, and proper output formatting.
- DPO (Direct Preference Optimization): Preference alignment on accumulated chosen/rejected pairs to sharpen identity consistency, formatting adherence, false-premise handling, and response quality. Note: DPO is always retrained from the SFT base, never stacked.
Evaluation: Featherweight v1 Benchmark
H2 is evaluated on Featherweight v1, a 100-question multiple-choice benchmark designed for the sub-1B parameter class. The benchmark spans 10 categories (10 questions each) with balanced multiple-choice keys (25% each for A/B/C/D) to eliminate position bias.
Testing uses deterministic greedy decoding (temperature=0, top_k=1, max_new_tokens=20) across two formats:
- Plain Prompt: Matches deployed evaluators, ending with "Answer with just the letter of the correct choice."
- Completion Prompt: An MMLU-style format ending with a bare "Answer:" (fairer to non-instruction-tuned base models).
Key Takeaways
- 38–39% Accuracy: Solid performance across both prompt formats (random chance is 25%).
- 1st in Weight Class: Beats every tested model $\le$ 217M parameters by 7–8 points.
- Highly Robust: Only a 1 percentage point swing between plain and completion prompts, making it the most prompt-robust model tested.
Comparison Leaderboard
| Model | Params | Plain | Completion | $\Delta$ |
|---|---|---|---|---|
| distilgpt2 | 82M | 27% | 24% | −3 |
| gpt2 | 124M | 23% | 26% | +3 |
| opt-125m | 125M | 28% | 23% | −5 |
| smollm-135m | 135M | 26% | 19% | −7 |
| smollm-135m-instruct | 135M | 30% | 32% | +2 |
| pythia-160m | 162M | 25% | 28% | +3 |
| Haiku H2 | 217M | 38% | 39% | +1 |
| gpt2-medium | 355M | 24% | 23% | −1 |
| smollm-360m | 362M | 20% | 29% | +9 |
| pythia-410m | 405M | 25% | 32% | +7 |
| qwen2.5-0.5b | 494M | 40% | 70% | +30 |
| tinyllama-1.1b-chat | 1.1B | 45% | 60% | +15 |
$\Delta$ = completion − plain. With n = 100, the 95% confidence interval on individual scores is roughly $\pm$9 percentage points.
Capability Profile
On the completion prompt, H2's category-specific breakdown shows an uneven but stable profile that remains strictly above the 25% random chance floor everywhere:
- Strongest Categories: Geography (60%) and Arithmetic (50%).
- Weakest Categories: Categorization (20%) and Logic (30%).
Size and Recency Gap
When plotted against larger systems like Qwen 2.5 (494M) and TinyLlama 1.1B Chat, H2's performance curve sits nested uniformly inside theirs. Larger, more recent 2024-vintage systems outperform H2 on knowledge-heavy fields (e.g., Qwen reaches 100% on categorization). H2's primary claim is that it represents the strongest option within its explicit weight class and against older architectures up through ~410M parameters.
Key Model Claims
- Wins its weight class: Outperforms all tested models $\le$ 217M parameters on both prompt settings.
- Beats larger base models: Comfortably clears larger pre-pretrained models like
gpt2-medium(355M) andpythia-410m(405M) by 7–18 points, demonstrating that SFT+DPO alignment optimizes small weights far better than raw scale alone. - Unmatched prompt robustness: Shifts by only 1% between testing formats, while competitors swing by up to 15% (TinyLlama) or 30% (Qwen 2.5).
Intended Use & Limitations
Best For
- Research baseline: A fully-documented, reproducible small architecture to compare against, fork, or iterate on.
- Small-model behavior studies: Investigating how SFT and DPO alignment mechanisms behave at the 200M parameter scale.
- Pipeline validation: Testing architecture changes and data mixtures quickly and cheaply before scaling to larger models.
Out of Scope & Limitations
- Multi-step reasoning: Logical reasoning is near chance (20–30%). Avoid chains of inference or syllogisms.
- Unreliable arithmetic: Arithmetic beyond single-digit operations is unstable and prone to hallucinations (e.g., outputs like
7 × 8 = 57). - Corpus bounds: Factual knowledge is strictly limited to the training corpus. Factual outputs should always be independently verified.
- Safety critical zones: Not evaluated or certified for medical, legal, financial, or safety-critical deployment.
Reproducibility
The validation framework is entirely open. Anyone using a standard CUDA GPU environment with pip install transformers torch can verify these benchmarks.
featherweight.jsonl: Contains the 100 benchmark questions balanced evenly across choice keys.run_featherweight.py: Standalone evaluation script supporting local Hugging Face pipelines, HTTP requests, and OpenAI-compatible backends using deterministic greedy decoding.
Model card generated based on Featherweight v1 evaluation metadata, 2026-06-13.