--- language: en license: apache-2.0 tags: - text-generation - small-language-model - slm pipeline_tag: text-generation inference: true widget: - text: "What is the capital of France?" example_title: "Geography" - text: "7 * 8 =" example_title: "Arithmetic" --- # Haiku H2 Haiku H2 is the canonical successor to Haiku Mini — a 217M-parameter small language model from **Rootcomputer**. It features a modernized architecture, a complete SFT + DPO post-training pipeline, and a published benchmark profile that wins its weight class. H2 is available for public testing in open beta via [chathaiku.com](https://chathaiku.com/). It is an experimental research model, designed to make small-model behavior studyable end-to-end, validate training pipelines, and publish reproducible benchmarks against comparable open small models. --- ## Model Specifications H2 is a decoder-only transformer architecture optimized to keep parameter counts low without sacrificing capacity. | Specification | Value | | :--- | :--- | | **Parameters** | 217M | | **Transformer Layers** | 16 | | **Embedding Dimension** | 1,024 | | **Attention Mechanism** | Grouped-Query Attention (GQA) with 4 KV heads | | **Positional Base** | 50k RoPE (Rotary Positional Embeddings) | | **Activation Function** | SwiGLU | | **Embeddings** | Tied Input/Output Embeddings | | **Precision & Scaling** | bf16 (Flash Attention 2 supported) | --- ## Training Pipeline H2 is trained across three distinct stages. Each stage is validated independently against a stable upstream checkpoint to allow cheap, rapid iteration on commodity hardware. 1. **Pretrain:** Base language modeling executed on the Rootcomputer multi-source corpus, comprising web text, books, news, scientific papers, and biomedical sources. 2. **SFT (Supervised Fine-Tuning):** Conversational tuning that teaches the model the shape of instructions, responses, refusals, and proper output formatting. 3. **DPO (Direct Preference Optimization):** Preference alignment on accumulated chosen/rejected pairs to sharpen identity consistency, formatting adherence, false-premise handling, and response quality. *Note: DPO is always retrained from the SFT base, never stacked.* --- ## Evaluation: Featherweight v1 Benchmark H2 is evaluated on **Featherweight v1**, a 100-question multiple-choice benchmark designed for the sub-1B parameter class. The benchmark spans 10 categories (10 questions each) with balanced multiple-choice keys (25% each for A/B/C/D) to eliminate position bias. Testing uses deterministic greedy decoding (`temperature=0`, `top_k=1`, `max_new_tokens=20`) across two formats: * **Plain Prompt:** Matches deployed evaluators, ending with *"Answer with just the letter of the correct choice."* * **Completion Prompt:** An MMLU-style format ending with a bare *"Answer:"* (fairer to non-instruction-tuned base models). ### Key Takeaways * **38–39% Accuracy:** Solid performance across both prompt formats (random chance is 25%). * **1st in Weight Class:** Beats every tested model $\le$ 217M parameters by 7–8 points. * **Highly Robust:** Only a 1 percentage point swing between plain and completion prompts, making it the most prompt-robust model tested. ### Comparison Leaderboard | Model | Params | Plain | Completion | $\Delta$ | | :--- | :---: | :---: | :---: | :---: | | distilgpt2 | 82M | 27% | 24% | −3 | | gpt2 | 124M | 23% | 26% | +3 | | opt-125m | 125M | 28% | 23% | −5 | | smollm-135m | 135M | 26% | 19% | −7 | | smollm-135m-instruct | 135M | 30% | 32% | +2 | | pythia-160m | 162M | 25% | 28% | +3 | | **Haiku H2** | **217M** | **38%** | **39%** | **+1** | | gpt2-medium | 355M | 24% | 23% | −1 | | smollm-360m | 362M | 20% | 29% | +9 | | pythia-410m | 405M | 25% | 32% | +7 | | qwen2.5-0.5b | 494M | 40% | 70% | +30 | | tinyllama-1.1b-chat | 1.1B | 45% | 60% | +15 | *$\Delta$ = completion − plain. With n = 100, the 95% confidence interval on individual scores is roughly $\pm$9 percentage points.* --- ## Capability Profile On the completion prompt, H2's category-specific breakdown shows an uneven but stable profile that remains strictly above the 25% random chance floor everywhere: * **Strongest Categories:** Geography (60%) and Arithmetic (50%). * **Weakest Categories:** Categorization (20%) and Logic (30%). ### Size and Recency Gap When plotted against larger systems like **Qwen 2.5 (494M)** and **TinyLlama 1.1B Chat**, H2's performance curve sits nested uniformly inside theirs. Larger, more recent 2024-vintage systems outperform H2 on knowledge-heavy fields (e.g., Qwen reaches 100% on categorization). H2's primary claim is that it represents the strongest option within its explicit weight class and against older architectures up through ~410M parameters. --- ## Key Model Claims * **Wins its weight class:** Outperforms all tested models $\le$ 217M parameters on both prompt settings. * **Beats larger base models:** Comfortably clears larger pre-pretrained models like `gpt2-medium` (355M) and `pythia-410m` (405M) by 7–18 points, demonstrating that SFT+DPO alignment optimizes small weights far better than raw scale alone. * **Unmatched prompt robustness:** Shifts by only 1% between testing formats, while competitors swing by up to 15% (TinyLlama) or 30% (Qwen 2.5). --- ## Intended Use & Limitations ### Best For * **Research baseline:** A fully-documented, reproducible small architecture to compare against, fork, or iterate on. * **Small-model behavior studies:** Investigating how SFT and DPO alignment mechanisms behave at the 200M parameter scale. * **Pipeline validation:** Testing architecture changes and data mixtures quickly and cheaply before scaling to larger models. ### Out of Scope & Limitations * **Multi-step reasoning:** Logical reasoning is near chance (20–30%). Avoid chains of inference or syllogisms. * **Unreliable arithmetic:** Arithmetic beyond single-digit operations is unstable and prone to hallucinations (e.g., outputs like `7 × 8 = 57`). * **Corpus bounds:** Factual knowledge is strictly limited to the training corpus. Factual outputs should always be independently verified. * **Safety critical zones:** Not evaluated or certified for medical, legal, financial, or safety-critical deployment. --- ## Reproducibility The validation framework is entirely open. Anyone using a standard CUDA GPU environment with `pip install transformers torch` can verify these benchmarks. * `featherweight.jsonl`: Contains the 100 benchmark questions balanced evenly across choice keys. * `run_featherweight.py`: Standalone evaluation script supporting local Hugging Face pipelines, HTTP requests, and OpenAI-compatible backends using deterministic greedy decoding. --- *Model card generated based on Featherweight v1 evaluation metadata, 2026-06-13.*