run-saina/saina-helm-2-0.8b

Saina Helm 2 0.8B

A small decision model. Give it some state and any number of typed questions (yes/no, single choice, multiple choice or rating) and it returns a probability for every answer. No text is generated.

Options are supplied at request time, so one model handles intent routing, triage, tool selection, review checklists and similar decisions without retraining.

It scores 39.26 on the public part of Decision Index 0.3, the highest of any model under 1B on the board, and has the highest tool-use (API-Bank, BFCL, When2Call) and RAGTruth hallucination scores of any model under 1B. See Evaluation.

New in Helm 2: every question in a request is answered from one pass over the shared state, instead of one pass per question, and the readout gains small routed expert heads. Previous model: saina-helm-0.8b.

Intended use

Decisions where a wrong answer is recoverable or reviewed before it takes effect: intent routing, triage, tool shortlisting, review checklists. Where a wrong answer has a cost, run with mode='decision' so low-confidence questions escalate instead of guessing. Validate on your own labelled data, for your own label space, before use.

Out of scope

• Generating text of any kind; the model only scores the options you supply.
• Final, unreviewed decisions with material consequences for a person (credit, employment, medical, legal), even at high confidence.
• Classifying people: personal attributes, demographics, or anything used to profile individuals.
• Safety filtering, harm detection or content moderation; not trained or evaluated for it.
• Non-English input. Training data is mostly English with limited non-English material, every reported evaluation is English, and other languages are untested.
• Decisions taken directly on untrusted text. The model scores whatever state it is given, so injected content can steer its answers; bound the authority you grant the output.

Usage

pip install "saina[local]>=0.3.2"

from saina.transformers import HelmModel

helm = HelmModel.from_pretrained('run-saina/saina-helm-2-0.8b', revision='<commit>', device='cuda:0')

answers = helm.ask(
    model='saina-helm-2-0.8b',
    state='Order #4417: parcel arrived crushed, customer wants their money back. Third complaint this month.',
    questions={
        'refund': {'type': 'yes_no', 'question': 'Is a refund requested?'},
        'team': {'type': 'single_choice', 'question': 'Which team should handle this?',
                 'options': {'billing': 'Payments and refunds', 'shipping': 'Delivery and carriers', 'tech': None}},
        'issues': {'type': 'multi_choice', 'question': 'Which issues apply?',
                   'options': {'damaged': 'Item damaged', 'late': 'Late delivery', 'repeat': 'Repeat complainer'}},
        'urgency': {'type': 'rating', 'question': 'How urgent is this?',
                    'levels': ['Low', 'Medium', 'High', 'Critical']},
    },
)

Pin revision to a commit. The saina package builds the exact prompt the model was trained on and applies the trained head; scores depend on that format, so use it rather than a custom prompt or generate(). No remote code is executed. Helm 2 needs saina 0.3 or newer; older versions refuse this checkpoint.

The same package runs an authenticated HTTP server (pip install "saina[local,server]", see run-saina/saina).

Question types

Type	You provide	You get
yes_no	question, optional descriptions: {yes, no}	yes, no, confidence
single_choice	question, options: {key: description or None} (2–255)	probabilities, selection, confidence
multi_choice	question, options (2–255)	independent memberships per option
rating	question, ordered levels (2–10)	probabilities per level, expected_level, confidence

Decision mode. Pass mode='decision' with a threshold (default 0.8) and, for yes_no / single_choice, an optional min_margin; both can be overridden per question. Answers then include selected / selection / selections / level, set to None with a reason (below_threshold, below_margin, tie) when the model is not confident enough, so you can abstain and escalate instead of guessing.

Context length

The checkpoint was trained with a 262,144-token limit (the base model's), which saina uses by default. A request is one forward pass over the whole input, so GPU memory grows with input length: about 3.2 GiB at 15K tokens and 7.9 GiB at 58K on an RTX 3090 (measured), roughly 0.11 GiB per 1K tokens, so the full 262K needs on the order of 30 GiB. Pass max_length= to serve a limit that fits your hardware. Longer inputs are rejected, never truncated. Installing flash-linear-attention and causal-conv1d enables faster kernels.

Evaluation

This model scores 39.26 on the public part of Decision Index 0.3, the highest among models under 1B parameters on the board, ahead of vLLM-SR d3-lite (36.93) and Crivo-150M (36.39). OneJev1.5-0.8B scores 42.71 but serves about 1.1B parameters, so it is not counted here. Larger models on the board score higher. The board's official Full score, which adds the maintainers' private tests, is 20.82 for the earlier revision 06f2fec2 (public part 30.47), fifth among models under 1B; it is not yet available for 72349026. Board scores are from the leaderboard data of 2026-10-16.

Run against the Decision Index 0.3 benchmark suite with the reproducible harness in run-saina/saina-decision-index: all 43 benchmarks, 140,178 requests, every one answered, on revision 72349026.

By area (chance-corrected, 0 = random, 100 = perfect):

Area	Score
Tools & Automation	61.3
Retrieval & Classification	50.5
Language Understanding	44.0
Knowledge & Reasoning	16.6
Arts & Human Taste	22.5

Per benchmark, each scored on its complete set, against six sub-1B models from the leaderboard's saved public results of 2026-10-10 (jiwo 0.8B, Dinah-0, Sifr 0.8B, Bekko 400M, OneJev 0.8B, Eos 0.8B). Rank is among these seven.

Benchmark	Metric	Result	Rank	Best of the others
CLINC150 + OOS	macro-F1	92.6	1st	Dinah-0 88.8
BFCL	case exact	92.5	1st	Sifr 0.8B 79.6
BANKING77	macro-F1	91.0	1st	Dinah-0 89.1
ContractNLI	macro-F1	85.4	1st	jiwo 0.8B 79.0
When2Call	accuracy	83.8	1st	Dinah-0 78.4
API-Bank tool selection	accuracy	81.7	1st	OneJev 0.8B 66.5
RouterBench	selected quality	78.5	1st	Eos 0.8B 76.5
RAGTruth	hallucination F1	77.5	1st	jiwo 0.8B 69.2
ARC-Challenge	accuracy	71.7	1st	jiwo 0.8B 68.9
VAST	macro-F1	69.6	1st	Dinah-0 62.6
cfcolor	accuracy	61.3	1st	OneJev 0.8B 58.6
Humicroedit	accuracy	59.9	1st	jiwo 0.8B 55.5
New Yorker	accuracy	53.4	1st	OneJev 0.8B 44.3
Habermas	accuracy	51.8	1st	jiwo 0.8B 42.1
ACOS	per-review F1	37.3	1st	Bekko 400M 9.7
GSM8K	accuracy	34.7	1st	Bekko 400M 29.5
iSarcasmEval	sarcasm F1, English (task A)	29.1	1st	jiwo 0.8B 27.7
Home appliances	case exact	6.8	1st	OneJev 0.8B 3.4
HoVer	accuracy	82.0	2nd	Dinah-0 82.2
CLadder	accuracy	76.5	2nd	Dinah-0 83.8
BPoMP	accuracy	60.8	2nd	Bekko 400M 72.1
MMLU	accuracy	49.2	2nd	jiwo 0.8B 50.0
ANLI	macro-F1	48.2	2nd	Sifr 0.8B 51.6
Amazon ESCI	macro-F1	47.9	2nd	jiwo 0.8B 48.1
ChessBench	accuracy	21.8	2nd	Dinah-0 24.3
POP909-CL	accuracy	3.9	2nd	Eos 0.8B 7.7
ARC-Easy	accuracy	85.4	3rd	Sifr 0.8B 85.9
FinEntity	macro-F1	74.8	3rd	Sifr 0.8B 85.7
HellaSwag	accuracy	65.5	3rd	jiwo 0.8B 74.9
NLI4CT	macro-F1	63.4	3rd	Eos 0.8B 63.6
WinoGrande	accuracy	62.4	3rd	Bekko 400M 69.2
MuSR	accuracy	54.9	3rd	Bekko 400M 56.9
MMLU-Pro	accuracy	24.7	3rd	jiwo 0.8B 28.6
SATA-Bench	case exact	14.2	3rd	Sifr 0.8B 33.3
BBH	accuracy	42.9	4th	jiwo 0.8B 48.5
HLE	accuracy	13.6	4th	Dinah-0, Sifr 0.8B 15.8
PhishNChips	accuracy	50.9	5th	Sifr 0.8B 99.9
ToolRet	nDCG@10	48.4	6th	Bekko 400M 63.4
SGD/SGD-X	macro-F1	43.9	6th	Eos 0.8B 60.9
BRIGHT	nDCG@10	21.2	6th	jiwo 0.8B 37.8
CRUXEval	accuracy	30.9	7th	Dinah-0 43.0
GPQA Diamond	accuracy	24.5	7th	Dinah-0, Bekko 400M 28.6
SimpleBench	accuracy (10 questions)	0.0	—	no public results for the others

Helm 2 is strongest on tool use and structured decisions (BFCL, API-Bank, When2Call, ContractNLI, RAGTruth); its BFCL (92.5), API-Bank (81.7), When2Call (83.8) and RAGTruth (77.5) scores are the highest of any sub-1B model on the board (per-benchmark results of 2026-10-16). It is weak on knowledge and reasoning (GPQA Diamond, CRUXEval, BBH) and on retrieval ranking (BRIGHT, ToolRet), where it places fourth to seventh of seven. On PhishNChips it scores close to chance (50.9), and on Home appliances it gets only 6.8% of cases fully right, far behind Crivo-150M (79.5).

This run sends every benchmark through the same request rendering API callers get from /v1/systemone; the results above are what users get.

At 81.7% API-Bank accuracy roughly one tool selection in five is wrong: treat tool choice as a ranked shortlist to review, not a final answer.

Training

Official test records were excluded from training by an exact-match audit; near-duplicates were not checked.

Limitations

Scope:

• Fairness across dialects, demographics and languages has not been evaluated.
• Domains and label spaces far from the training mix are untested; the results above do not predict them.
• Results are sensitive to how questions and options are phrased. Fix a wording, keep it stable, and revalidate when it changes.

Mechanics:

• Probabilities are not calibrated; set thresholds on your own data.
• Helm 2 runs with the saina Python runtime, and with llama.cpp (GGUF) and MLX through the scorer each ships. Its head is not part of the GGUF, so stock Ollama, llama-server and llama-cli can't serve it.

Feedback

Issue reports and results from your own tasks are welcome on run-saina/saina.

License

The weights in this repository are released under Apache-2.0. Training data sources keep their own terms, and some are share-alike or have unclear terms; no representation is made that the weights are cleared for commercial use. Review your use accordingly.