---
license: mit
library_name: transformers
tags:
  - text-generation
  - chat-model
  - qwen2
  - english
  - chinese
  - pytorch
  - real-time
  - sft
---

# OpenSI-Haiku

**An open conversational model (SFT from the OpenSI 0.3B base).**

`Qwen2` architecture · **113.9M params** · 2048 context · **Interactive (low-latency)** served profile · MIT License

> ⚠️ Name note: the project name uses "Haiku" to denote its **smallest/fastest real-time** variant. Actual parameter count: **113,916,672** (113.9M), not "0.3B". Treat as a ~0.11B chat model.

## Quick facts

| | |
|---|---|
| Parameters | **113,916,672** (113.9M) |
| Layers / hidden | 12 / 768 |
| Heads (GQA) | 12 query, 4 KV |
| FFN (SwiGLU) | 2048 |
| Vocab | 50,000 (ByteLevel BPE) |
| Context | 2048 |
| Embeddings | tied (`lm_head.weight` = `embed_tokens.weight`) |
| Base ckpt (seed) | `runs/opensi-0.3B-seed/opensi-0.3B.ckpt.pt` (step 9536, train EMA loss 3.0499) |
| SFT method | **Full-parameter SFT** (no LoRA), masked CE on assistant tokens only |
| Training steps | **120** (3 epochs) |
| Best val loss | **2.7959** (ppl ~16.4) |
| Tokens (SFT) | ~10.4M train tokens packed into 5,091 windows, val 196k (94 windows) |
| Hardware | NVIDIA GB10, BF16 training, eval/serve in FP32 |
| Dtype exported | float32 (numerically verified) |

## SFT data

Mixture (kept after dedup and length filtering, max 2048):

- [`databricks/databricks-dolly-15k`](https://huggingface.co/datasets/databricks/databricks-dolly-15k) (CC BY-SA 3.0, English)
- [`shibing624/alpaca-zh`](https://huggingface.co/datasets/shibing624/alpaca-zh) (CC BY 4.0, Chinese, Alpaca-GPT4)

SFT mirrors the seed architecture exactly; tokenizer unchanged (vocab 50000). No new special tokens were introduced.

## Chat format (canonical)

Format is plain-text role markers with EOS terminator. This matches the single-source-of-truth implementation:

- `### System:\n{system}\n`
- `### User:\n{user}\n`
- `### Assistant:\n{content}</s>` (assistant turns end with existing `</s>` id=2)
- If no system is provided, a default system is auto-inserted during training/serving (see below).

**Default system:**  
`You are OpenSI-Haiku, a helpful and concise AI assistant.`

The HF `chat_template` is bit-exact equivalent to `src/chat_format.py` (verified by `chat_format.self_check` against the tokenizer). The assistant generation prompt ends with `### Assistant:\n` and the model is trained to emit `</s>` to terminate.

## Inference

Two generation endpoints (OpenAI-compatible):

- `POST /v1/chat/completions` — multi-turn with system/user/assistant. Auto-inserts default system if missing. Supports `temperature`, `top_p`, `top_k`, `repetition_penalty`, `stop`, `max_tokens`, `min_tokens`, `seed`, `request_id`, `logprobs` (shape/format consistent with completions path). Returns OpenAI-style `chat.completion`. `stream` and `n>1` are not supported by this static-batch engine.
- `POST /v1/completions` — continuation (also works on chat models).

Serving modes: low-latency interactive profile (`--batch-window-ms 0`) is recommended for real-time chat. Deterministic greedy behavior is reproducible under fixed seeds in practice (FP32).

## Verification

- HF export numerical alignment: max abs error **4.18e-5**, cosine similarity **1.0000000000**, argmax agreement **100.00%** (vs custom GPT).
- Template consistency: `chat_format.self_check` passes for representative cases.
- Endpoint checks: 14/14 pass via `scripts/serve/verify_chat.py` (including correct 400s for invalid requests).
- Base backward compatibility: `/health` and `/v1/*` semantics preserved for the existing base service (restarted with updated, backward-compatible server).

## Limitations (important)

This is a **113.9M** model trained on a small, mostly instruction-style SFT set (~55k examples mixed EN/ZH). Expect:  
- Fluent, format-correct chat responses  
- Good at short, structured tasks and English commonsense patterns  
- **Limited factual recall / domain knowledge** (will confidently hallucinate on niche facts)  
- Weaker multi-hop reasoning and arithmetic than larger models  
- May echo or ramble on very long prompts near context; prefer concise instructions

## License

MIT (as inherited from the OpenSI seed/base line). SFT data sources retain their original licenses (CC BY-SA 3.0 / CC BY 4.0).

## Acknowledgements

Built on the OpenSI codebase; adapted to chat via masked SFT with EOS termination and a Jinja-equivalent canonical format. The HF export produces a `Qwen2ForCausalLM`-compatible directory (`tokenizer_config.json` includes the chat template).