| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - text-generation |
| - small-language-model |
| - slm |
| pipeline_tag: text-generation |
| inference: true |
| widget: |
| - text: "What is the capital of France?" |
| example_title: "Geography" |
| - text: "7 * 8 =" |
| example_title: "Arithmetic" |
| --- |
| |
| # Haiku H2 |
|
|
| Haiku H2 is the canonical successor to Haiku Mini — a 217M-parameter small language model from **Rootcomputer**. It features a modernized architecture, a complete SFT + DPO post-training pipeline, and a published benchmark profile that wins its weight class. |
|
|
| H2 is available for public testing in open beta via [chathaiku.com](https://chathaiku.com/). It is an experimental research model, designed to make small-model behavior studyable end-to-end, validate training pipelines, and publish reproducible benchmarks against comparable open small models. |
|
|
| --- |
|
|
| ## Model Specifications |
|
|
| H2 is a decoder-only transformer architecture optimized to keep parameter counts low without sacrificing capacity. |
|
|
| | Specification | Value | |
| | :--- | :--- | |
| | **Parameters** | 217M | |
| | **Transformer Layers** | 16 | |
| | **Embedding Dimension** | 1,024 | |
| | **Attention Mechanism** | Grouped-Query Attention (GQA) with 4 KV heads | |
| | **Positional Base** | 50k RoPE (Rotary Positional Embeddings) | |
| | **Activation Function** | SwiGLU | |
| | **Embeddings** | Tied Input/Output Embeddings | |
| | **Precision & Scaling** | bf16 (Flash Attention 2 supported) | |
|
|
| --- |
|
|
| ## Training Pipeline |
|
|
| H2 is trained across three distinct stages. Each stage is validated independently against a stable upstream checkpoint to allow cheap, rapid iteration on commodity hardware. |
|
|
| 1. **Pretrain:** Base language modeling executed on the Rootcomputer multi-source corpus, comprising web text, books, news, scientific papers, and biomedical sources. |
| 2. **SFT (Supervised Fine-Tuning):** Conversational tuning that teaches the model the shape of instructions, responses, refusals, and proper output formatting. |
| 3. **DPO (Direct Preference Optimization):** Preference alignment on accumulated chosen/rejected pairs to sharpen identity consistency, formatting adherence, false-premise handling, and response quality. *Note: DPO is always retrained from the SFT base, never stacked.* |
|
|
| --- |
|
|
| ## Evaluation: Featherweight v1 Benchmark |
|
|
| H2 is evaluated on **Featherweight v1**, a 100-question multiple-choice benchmark designed for the sub-1B parameter class. The benchmark spans 10 categories (10 questions each) with balanced multiple-choice keys (25% each for A/B/C/D) to eliminate position bias. |
|
|
| Testing uses deterministic greedy decoding (`temperature=0`, `top_k=1`, `max_new_tokens=20`) across two formats: |
| * **Plain Prompt:** Matches deployed evaluators, ending with *"Answer with just the letter of the correct choice."* |
| * **Completion Prompt:** An MMLU-style format ending with a bare *"Answer:"* (fairer to non-instruction-tuned base models). |
|
|
| ### Key Takeaways |
| * **38–39% Accuracy:** Solid performance across both prompt formats (random chance is 25%). |
| * **1st in Weight Class:** Beats every tested model $\le$ 217M parameters by 7–8 points. |
| * **Highly Robust:** Only a 1 percentage point swing between plain and completion prompts, making it the most prompt-robust model tested. |
|
|
| ### Comparison Leaderboard |
|
|
| | Model | Params | Plain | Completion | $\Delta$ | |
| | :--- | :---: | :---: | :---: | :---: | |
| | distilgpt2 | 82M | 27% | 24% | −3 | |
| | gpt2 | 124M | 23% | 26% | +3 | |
| | opt-125m | 125M | 28% | 23% | −5 | |
| | smollm-135m | 135M | 26% | 19% | −7 | |
| | smollm-135m-instruct | 135M | 30% | 32% | +2 | |
| | pythia-160m | 162M | 25% | 28% | +3 | |
| | **Haiku H2** | **217M** | **38%** | **39%** | **+1** | |
| | gpt2-medium | 355M | 24% | 23% | −1 | |
| | smollm-360m | 362M | 20% | 29% | +9 | |
| | pythia-410m | 405M | 25% | 32% | +7 | |
| | qwen2.5-0.5b | 494M | 40% | 70% | +30 | |
| | tinyllama-1.1b-chat | 1.1B | 45% | 60% | +15 | |
|
|
| *$\Delta$ = completion − plain. With n = 100, the 95% confidence interval on individual scores is roughly $\pm$9 percentage points.* |
|
|
| --- |
|
|
| ## Capability Profile |
|
|
| On the completion prompt, H2's category-specific breakdown shows an uneven but stable profile that remains strictly above the 25% random chance floor everywhere: |
|
|
| * **Strongest Categories:** Geography (60%) and Arithmetic (50%). |
| * **Weakest Categories:** Categorization (20%) and Logic (30%). |
|
|
| ### Size and Recency Gap |
| When plotted against larger systems like **Qwen 2.5 (494M)** and **TinyLlama 1.1B Chat**, H2's performance curve sits nested uniformly inside theirs. Larger, more recent 2024-vintage systems outperform H2 on knowledge-heavy fields (e.g., Qwen reaches 100% on categorization). H2's primary claim is that it represents the strongest option within its explicit weight class and against older architectures up through ~410M parameters. |
|
|
| --- |
|
|
| ## Key Model Claims |
|
|
| * **Wins its weight class:** Outperforms all tested models $\le$ 217M parameters on both prompt settings. |
| * **Beats larger base models:** Comfortably clears larger pre-pretrained models like `gpt2-medium` (355M) and `pythia-410m` (405M) by 7–18 points, demonstrating that SFT+DPO alignment optimizes small weights far better than raw scale alone. |
| * **Unmatched prompt robustness:** Shifts by only 1% between testing formats, while competitors swing by up to 15% (TinyLlama) or 30% (Qwen 2.5). |
|
|
| --- |
|
|
| ## Intended Use & Limitations |
|
|
| ### Best For |
| * **Research baseline:** A fully-documented, reproducible small architecture to compare against, fork, or iterate on. |
| * **Small-model behavior studies:** Investigating how SFT and DPO alignment mechanisms behave at the 200M parameter scale. |
| * **Pipeline validation:** Testing architecture changes and data mixtures quickly and cheaply before scaling to larger models. |
|
|
| ### Out of Scope & Limitations |
| * **Multi-step reasoning:** Logical reasoning is near chance (20–30%). Avoid chains of inference or syllogisms. |
| * **Unreliable arithmetic:** Arithmetic beyond single-digit operations is unstable and prone to hallucinations (e.g., outputs like `7 × 8 = 57`). |
| * **Corpus bounds:** Factual knowledge is strictly limited to the training corpus. Factual outputs should always be independently verified. |
| * **Safety critical zones:** Not evaluated or certified for medical, legal, financial, or safety-critical deployment. |
|
|
| --- |
|
|
| ## Reproducibility |
|
|
| The validation framework is entirely open. Anyone using a standard CUDA GPU environment with `pip install transformers torch` can verify these benchmarks. |
|
|
| * `featherweight.jsonl`: Contains the 100 benchmark questions balanced evenly across choice keys. |
| * `run_featherweight.py`: Standalone evaluation script supporting local Hugging Face pipelines, HTTP requests, and OpenAI-compatible backends using deterministic greedy decoding. |
|
|
| --- |
| *Model card generated based on Featherweight v1 evaluation metadata, 2026-06-13.* |
