Instructions to use OpenSI-SI/OpenSI-Haiku with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenSI-SI/OpenSI-Haiku with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenSI-SI/OpenSI-Haiku")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OpenSI-SI/OpenSI-Haiku")
model = AutoModelForCausalLM.from_pretrained("OpenSI-SI/OpenSI-Haiku", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpenSI-SI/OpenSI-Haiku with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenSI-SI/OpenSI-Haiku"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenSI-SI/OpenSI-Haiku",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OpenSI-SI/OpenSI-Haiku

SGLang

How to use OpenSI-SI/OpenSI-Haiku with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenSI-SI/OpenSI-Haiku" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenSI-SI/OpenSI-Haiku",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenSI-SI/OpenSI-Haiku" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenSI-SI/OpenSI-Haiku",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use OpenSI-SI/OpenSI-Haiku with Docker Model Runner:
```
docker model run hf.co/OpenSI-SI/OpenSI-Haiku
```

OpenSI-Haiku

File size: 4,597 Bytes

69d0e9d

---
license: mit
library_name: transformers
tags:
  - text-generation
  - chat-model
  - qwen2
  - english
  - chinese
  - pytorch
  - real-time
  - sft
---

# OpenSI-Haiku

**An open conversational model (SFT from the OpenSI 0.3B base).**

`Qwen2` architecture · **113.9M params** · 2048 context · **Interactive (low-latency)** served profile · MIT License

> ⚠️ Name note: the project name uses "Haiku" to denote its **smallest/fastest real-time** variant. Actual parameter count: **113,916,672** (113.9M), not "0.3B". Treat as a ~0.11B chat model.

## Quick facts

| | |
|---|---|
| Parameters | **113,916,672** (113.9M) |
| Layers / hidden | 12 / 768 |
| Heads (GQA) | 12 query, 4 KV |
| FFN (SwiGLU) | 2048 |
| Vocab | 50,000 (ByteLevel BPE) |
| Context | 2048 |
| Embeddings | tied (`lm_head.weight` = `embed_tokens.weight`) |
| Base ckpt (seed) | `runs/opensi-0.3B-seed/opensi-0.3B.ckpt.pt` (step 9536, train EMA loss 3.0499) |
| SFT method | **Full-parameter SFT** (no LoRA), masked CE on assistant tokens only |
| Training steps | **120** (3 epochs) |
| Best val loss | **2.7959** (ppl ~16.4) |
| Tokens (SFT) | ~10.4M train tokens packed into 5,091 windows, val 196k (94 windows) |
| Hardware | NVIDIA GB10, BF16 training, eval/serve in FP32 |
| Dtype exported | float32 (numerically verified) |

## SFT data

Mixture (kept after dedup and length filtering, max 2048):

- [`databricks/databricks-dolly-15k`](https://huggingface.co/datasets/databricks/databricks-dolly-15k) (CC BY-SA 3.0, English)
- [`shibing624/alpaca-zh`](https://huggingface.co/datasets/shibing624/alpaca-zh) (CC BY 4.0, Chinese, Alpaca-GPT4)

SFT mirrors the seed architecture exactly; tokenizer unchanged (vocab 50000). No new special tokens were introduced.

## Chat format (canonical)

Format is plain-text role markers with EOS terminator. This matches the single-source-of-truth implementation:

- `### System:\n{system}\n`
- `### User:\n{user}\n`
- `### Assistant:\n{content}</s>` (assistant turns end with existing `</s>` id=2)
- If no system is provided, a default system is auto-inserted during training/serving (see below).

**Default system:**  
`You are OpenSI-Haiku, a helpful and concise AI assistant.`

The HF `chat_template` is bit-exact equivalent to `src/chat_format.py` (verified by `chat_format.self_check` against the tokenizer). The assistant generation prompt ends with `### Assistant:\n` and the model is trained to emit `</s>` to terminate.

## Inference

Two generation endpoints (OpenAI-compatible):

- `POST /v1/chat/completions` — multi-turn with system/user/assistant. Auto-inserts default system if missing. Supports `temperature`, `top_p`, `top_k`, `repetition_penalty`, `stop`, `max_tokens`, `min_tokens`, `seed`, `request_id`, `logprobs` (shape/format consistent with completions path). Returns OpenAI-style `chat.completion`. `stream` and `n>1` are not supported by this static-batch engine.
- `POST /v1/completions` — continuation (also works on chat models).

Serving modes: low-latency interactive profile (`--batch-window-ms 0`) is recommended for real-time chat. Deterministic greedy behavior is reproducible under fixed seeds in practice (FP32).

## Verification

- HF export numerical alignment: max abs error **4.18e-5**, cosine similarity **1.0000000000**, argmax agreement **100.00%** (vs custom GPT).
- Template consistency: `chat_format.self_check` passes for representative cases.
- Endpoint checks: 14/14 pass via `scripts/serve/verify_chat.py` (including correct 400s for invalid requests).
- Base backward compatibility: `/health` and `/v1/*` semantics preserved for the existing base service (restarted with updated, backward-compatible server).

## Limitations (important)

This is a **113.9M** model trained on a small, mostly instruction-style SFT set (~55k examples mixed EN/ZH). Expect:  
- Fluent, format-correct chat responses  
- Good at short, structured tasks and English commonsense patterns  
- **Limited factual recall / domain knowledge** (will confidently hallucinate on niche facts)  
- Weaker multi-hop reasoning and arithmetic than larger models  
- May echo or ramble on very long prompts near context; prefer concise instructions

## License

MIT (as inherited from the OpenSI seed/base line). SFT data sources retain their original licenses (CC BY-SA 3.0 / CC BY 4.0).

## Acknowledgements

Built on the OpenSI codebase; adapted to chat via masked SFT with EOS termination and a Jinja-equivalent canonical format. The HF export produces a `Qwen2ForCausalLM`-compatible directory (`tokenizer_config.json` includes the chat template).