README.md

---
language: en
license: apache-2.0
tags:
- text-generation
- small-language-model
- slm
pipeline_tag: text-generation
inference: true
widget:
- text: "What is the capital of France?"
  example_title: "Geography"
- text: "7 * 8 ="
  example_title: "Arithmetic"
---

# Haiku H2

Haiku H2 is the canonical successor to Haiku Mini — a 217M-parameter small language model from **Rootcomputer**. It features a modernized architecture, a complete SFT + DPO post-training pipeline, and a published benchmark profile that wins its weight class.

H2 is available for public testing in open beta via [chathaiku.com](https://chathaiku.com/). It is an experimental research model, designed to make small-model behavior studyable end-to-end, validate training pipelines, and publish reproducible benchmarks against comparable open small models.

---

## Model Specifications

H2 is a decoder-only transformer architecture optimized to keep parameter counts low without sacrificing capacity.

| Specification | Value |
| :--- | :--- |
| **Parameters** | 217M |
| **Transformer Layers** | 16 |
| **Embedding Dimension** | 1,024 |
| **Attention Mechanism** | Grouped-Query Attention (GQA) with 4 KV heads |
| **Positional Base** | 50k RoPE (Rotary Positional Embeddings) |
| **Activation Function** | SwiGLU |
| **Embeddings** | Tied Input/Output Embeddings |
| **Precision & Scaling** | bf16 (Flash Attention 2 supported) |

---

## Training Pipeline

H2 is trained across three distinct stages. Each stage is validated independently against a stable upstream checkpoint to allow cheap, rapid iteration on commodity hardware.

1. **Pretrain:** Base language modeling executed on the Rootcomputer multi-source corpus, comprising web text, books, news, scientific papers, and biomedical sources.
2. **SFT (Supervised Fine-Tuning):** Conversational tuning that teaches the model the shape of instructions, responses, refusals, and proper output formatting.
3. **DPO (Direct Preference Optimization):** Preference alignment on accumulated chosen/rejected pairs to sharpen identity consistency, formatting adherence, false-premise handling, and response quality. *Note: DPO is always retrained from the SFT base, never stacked.*

---

## Evaluation: Featherweight v1 Benchmark

H2 is evaluated on **Featherweight v1**, a 100-question multiple-choice benchmark designed for the sub-1B parameter class. The benchmark spans 10 categories (10 questions each) with balanced multiple-choice keys (25% each for A/B/C/D) to eliminate position bias. 

Testing uses deterministic greedy decoding (`temperature=0`, `top_k=1`, `max_new_tokens=20`) across two formats:
* **Plain Prompt:** Matches deployed evaluators, ending with *"Answer with just the letter of the correct choice."*
* **Completion Prompt:** An MMLU-style format ending with a bare *"Answer:"* (fairer to non-instruction-tuned base models).

### Key Takeaways
* **38–39% Accuracy:** Solid performance across both prompt formats (random chance is 25%).
* **1st in Weight Class:** Beats every tested model $\le$ 217M parameters by 7–8 points.
* **Highly Robust:** Only a 1 percentage point swing between plain and completion prompts, making it the most prompt-robust model tested.

### Comparison Leaderboard

| Model | Params | Plain | Completion | $\Delta$ |
| :--- | :---: | :---: | :---: | :---: |
| distilgpt2 | 82M | 27% | 24% | −3 |
| gpt2 | 124M | 23% | 26% | +3 |
| opt-125m | 125M | 28% | 23% | −5 |
| smollm-135m | 135M | 26% | 19% | −7 |
| smollm-135m-instruct | 135M | 30% | 32% | +2 |
| pythia-160m | 162M | 25% | 28% | +3 |
| **Haiku H2** | **217M** | **38%** | **39%** | **+1** |
| gpt2-medium | 355M | 24% | 23% | −1 |
| smollm-360m | 362M | 20% | 29% | +9 |
| pythia-410m | 405M | 25% | 32% | +7 |
| qwen2.5-0.5b | 494M | 40% | 70% | +30 |
| tinyllama-1.1b-chat | 1.1B | 45% | 60% | +15 |

*$\Delta$ = completion − plain. With n = 100, the 95% confidence interval on individual scores is roughly $\pm$9 percentage points.*

---

## Capability Profile

On the completion prompt, H2's category-specific breakdown shows an uneven but stable profile that remains strictly above the 25% random chance floor everywhere:

* **Strongest Categories:** Geography (60%) and Arithmetic (50%).
* **Weakest Categories:** Categorization (20%) and Logic (30%).

### Size and Recency Gap
When plotted against larger systems like **Qwen 2.5 (494M)** and **TinyLlama 1.1B Chat**, H2's performance curve sits nested uniformly inside theirs. Larger, more recent 2024-vintage systems outperform H2 on knowledge-heavy fields (e.g., Qwen reaches 100% on categorization). H2's primary claim is that it represents the strongest option within its explicit weight class and against older architectures up through ~410M parameters.

---

## Key Model Claims

* **Wins its weight class:** Outperforms all tested models $\le$ 217M parameters on both prompt settings.
* **Beats larger base models:** Comfortably clears larger pre-pretrained models like `gpt2-medium` (355M) and `pythia-410m` (405M) by 7–18 points, demonstrating that SFT+DPO alignment optimizes small weights far better than raw scale alone.
* **Unmatched prompt robustness:** Shifts by only 1% between testing formats, while competitors swing by up to 15% (TinyLlama) or 30% (Qwen 2.5).

---

## Intended Use & Limitations

### Best For
* **Research baseline:** A fully-documented, reproducible small architecture to compare against, fork, or iterate on.
* **Small-model behavior studies:** Investigating how SFT and DPO alignment mechanisms behave at the 200M parameter scale.
* **Pipeline validation:** Testing architecture changes and data mixtures quickly and cheaply before scaling to larger models.

### Out of Scope & Limitations
* **Multi-step reasoning:** Logical reasoning is near chance (20–30%). Avoid chains of inference or syllogisms.
* **Unreliable arithmetic:** Arithmetic beyond single-digit operations is unstable and prone to hallucinations (e.g., outputs like `7 × 8 = 57`).
* **Corpus bounds:** Factual knowledge is strictly limited to the training corpus. Factual outputs should always be independently verified.
* **Safety critical zones:** Not evaluated or certified for medical, legal, financial, or safety-critical deployment. 

---

## Reproducibility

The validation framework is entirely open. Anyone using a standard CUDA GPU environment with `pip install transformers torch` can verify these benchmarks. 

* `featherweight.jsonl`: Contains the 100 benchmark questions balanced evenly across choice keys.
* `run_featherweight.py`: Standalone evaluation script supporting local Hugging Face pipelines, HTTP requests, and OpenAI-compatible backends using deterministic greedy decoding.

---
*Model card generated based on Featherweight v1 evaluation metadata, 2026-06-13.*