GyrTechnology/saif-4b

Saif 4B

A fast, calibrated decision model. Send it a situation and up to 64 typed questions in one request; it returns a probability for every answer in tens of milliseconds. It does no text generation and no chat: it only makes decisions.

Saif 4B speaks the System One / Jev typed-decision format, so an existing Jev or Clef integration can switch to it by changing the endpoint and model name.

• Decision Index 0.3 (public suite): 50.79, ahead of the published d1-3B (48.99) at a similar size.
• Fast: 79 ms median, 193 ms p95 per request on a single GPU.
• Calibrated: on 7,624 held-out decisions, its confidence matches its accuracy to within 1.3% (expected calibration error): when it says 85%, it is right about 85% of the time.
• Decision-only by design: no text-generation head is shipped, so it can't be turned into a chatbot by accident.
• Self-contained: the merged weights and loader are included. No adapter, Unsloth install or separate base download is needed.

What it does

Every request has a state (any JSON: a message, a tool catalogue, a document, a conversation) and a set of questions in three types:

Type	Asks	Returns
noul	a yes/no question	the probability of yes
choice	pick one option from a set you define	the chosen option, its confidence, and a probability for every option
score	rate on an ordered scale	the expected level, its confidence, and a probability for every level

Typical uses include:

• Routing and intent: which tool, team or flow should handle this?
• Tool use: does this request need a tool? Which one? Which argument value?
• Guardrails: is this a prompt injection? Is it out of scope?
• Classification and triage: sentiment, topic and stance.
• Claim checking: is this claim supported by the text?
• Ranking: how relevant is this document?

Results

Decision Index 0.3, public suite (37 benchmarks), run with the official Decision Index kit, one pass per question. This is our own run, not a leaderboard submission.

Area	Saif 4B	d1-3B
Overall index	50.79	48.99
Tools & Automation	71.2	74.5
Retrieval & Classification	62.0	52.8
Language Understanding	54.4	56.5
Arts & Human Taste	35.4	38.9
Knowledge & Reasoning	30.1	24.3

Strongest benchmarks: BFCL function calling 95.1, CLINC150 intent with out-of-scope detection 94.5, BANKING77 89.6, HellaSwag 89.0, FinEntity 89.0, When2Call 86.2, ContractNLI 73.6. Scores are chance-corrected skill (0 is random guessing, 100 is perfect).

Calibration

Every answer comes with a full probability distribution, and the probabilities are calibrated. Temperatures are fitted after training on a separate 9,115-decision calibration set, per question type and option count, and applied at inference.

Measured on a held-out development set of 7,624 decisions, which the calibration never saw:

Stated confidence	Decisions	Actually right
90–100%	4,639	97.7% (stated 98.0%)
80–90%	697	86.5% (stated 85.3%)
70–80%	474	78.9% (stated 75.1%)
60–70%	514	65.2% (stated 65.0%)
50–60%	542	57.0% (stated 55.0%)
40–50%	347	51.6% (stated 45.8%)

Overall: mean confidence 83.9%, accuracy 84.8%, expected calibration error 1.3%. The model is slightly cautious in most areas (Tools: 94.9% stated, 96.9% right) and slightly confident on retrieval (82.3% stated, 79.4% right). The held-out set comes from the same sources as the training data; expect calibration to loosen somewhat on very different tasks.

Quick start

Docker (recommended)

The Dockerfile is in the serving/ folder of this repo. Build the image, then run it; it downloads the model from Hugging Face on first start.

git clone https://huggingface.co/GyrTechnology/saif-4b
docker build -t saif-4b saif-4b/serving

# GPU (needs the NVIDIA Container Toolkit)
docker run --rm --gpus all -p 8000:8000 -e SAIF_DEVICE=cuda \
  -e SAIF_HF_REPO=GyrTechnology/saif-4b saif-4b

# CPU
docker run --rm -p 8000:8000 -e SAIF_DEVICE=cpu \
  -e SAIF_HF_REPO=GyrTechnology/saif-4b saif-4b

To use a local copy instead of downloading, mount it and set SAIF_MODEL_PATH: -e SAIF_MODEL_PATH=/model -v "$PWD/saif-4b:/model:ro".

curl http://localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "saif-4b",
  "state": {"message": "My card was charged twice for one order."},
  "questions": {
    "needs_human": {"type": "noul", "instructions": "Should this go to a human agent?"},
    "intent": {"type": "choice", "instructions": "What does the customer want?",
               "criteria": {"refund": "A refund", "tracking": "Order tracking", "other": "Something else"}},
    "urgency": {"type": "score", "instructions": "How urgent is it?",
                "criteria": ["low", "medium", "high"]}
  }
}'

Python

from joint_schema_model import load   # run from the downloaded model folder

model = load(".")
answers = model.decide(
    {"message": "My card was charged twice for one order."},
    {
        "intent": {
            "type": "choice",
            "instructions": "What does the customer want?",
            "criteria": {"refund": "A refund", "tracking": "Order tracking", "other": "Something else"},
        },
    },
)
print(answers)

The serving kit (server, CLI, Dockerfile and pinned requirements) is in serving/; see serving/SERVING.md.

Inference engines

Saif 4B is a single-pass decision model with a custom head, not a text generator. It runs with the included server or loader (plain PyTorch and Transformers), and can be wrapped in Triton's Python backend. Text-generation engines (Ollama, llama.cpp/GGUF, vLLM, SGLang, LM Studio, TGI) don't apply to it.

How it works

A Qwen3.5-4B backbone reads the state and the questions once. A joint decision head then scores every option of every question from the same pass, which is why many questions cost about as much as one. Probabilities are calibrated with temperatures fitted on held-out data for each question type and option count.

Training: LoRA (rank 64) plus the decision head, one epoch over about 82,000 records and 142,000 decisions from more than 80 public datasets: tool use, intent, retrieval, NLI, stance, safety, reasoning and taste. The adapter is merged into the backbone in this release.

Good to know

• Context: up to 131,072 tokens per request, state and questions together, without truncation. This matches the evaluated inference configuration; training used inputs up to about 16k tokens (16,320), so longer inputs extrapolate beyond the training length. The HTTP server can enforce a smaller limit with SAIF_MAX_TOKENS (default 131072).
• Single pass: Saif 4B decides in one fast pass with no step-by-step reasoning. It is excellent at routing, tool selection and classification. Tasks that require working something out in steps (maths word problems, code tracing, chess) are better served by a reasoning model.
• Text only: no image input.
• Hardware: runs on GPU or CPU. The latency figures above are from a single NVIDIA RTX PRO 6000; CPU requests take a few seconds.
• Verified release: the merged weights were checked against the trained adapter on 320 held-out records. They give the same top answer on every question that isn't a near-tie, with a mean probability difference of 0.0005. Details are in EQUIVALENCE.json and DECISION_ONLY_TESTS.json.

License

The weights are released under CC-BY-NC-4.0 (non-commercial), because some of the training data is licensed for non-commercial or research use only. These sources include ANLI, AgentDecision, Chordonomicon and RACE, plus research-only sources such as VAST, NLI4CT, ACOS and MS MARCO. Every source and its licence is listed below.

• Base model: Qwen/Qwen3.5-4B, Apache-2.0 (LICENSE-QWEN).
• Decision architecture and encoder adapted from Cloudflare Clef, Apache-2.0 (LICENSE-CLEF).
• The optimised head kernel code is from Unsloth, AGPL-3.0 (LICENSE-AGPL).
• See NOTICE.

Training data

Every training source is listed below with its recorded licence. Licences are recorded as published upstream; check each source's own terms before relying on them. Sources with zero records were used only for transfer evaluation.

Source entry	Dataset ID	Split	Recorded license	Training records
supergpqa	m-a-p/SuperGPQA	train	odc-by	2543
winogrande	allenai/winogrande	train	HF card states More Information Needed under Licensing Information; no licence inferred	2308
aqua	deepmind/aqua_rat	train	apache-2.0	1191
gsm8k	openai/gsm8k	train	mit	0
medmcqa	openlifescienceai/medmcqa	train	apache-2.0	2334
arc	allenai/ai2_arc	train	cc-by-sa-4.0	499
openbookqa	allenai/openbookqa	train	unknown	383
ruletaker	tasksource/ruletaker	train	Upstream licence not declared in conversion receipt	592
quality	tasksource/QuALITY	train	cc-by-4.0	1933
anli	facebook/anli	train	cc-by-nc-4.0	2076
hellaswag	Rowan/hellaswag	train	MIT; pinned HF README Licensing Information	1894
isarcasm	isarcasm	train	MIT	1016
ragtruth	ragtruth	train	MIT	186
mnli	nyu-mll/multi_nli	train	cc-by-3.0, cc-by-sa-3.0, mit, other	2665
vast	vast	train	No licence file in pinned upstream checkout; research-only review required	2034
nli4ct	nli4ct	train	No licence file in pinned upstream checkout; research-only review required	880
acos	acos	train	No licence file in pinned upstream checkout; research-only review required	85
boolq	google/boolq	train	cc-by-sa-3.0	400
race	ehovy/race	train	non-commercial-research-only	291
cosmosqa	allenai/cosmos_qa	train	cc-by-4.0	200
socialiqa	allenai/social_i_qa	train	cc-by-4.0	200
paws_language	google-research-datasets/paws	train	other	262
banking77	mteb/banking77	train	MIT (mteb mirror card); CC-BY-4.0 (PolyAI original, existing source citation)	3425
clinc150	clinc/clinc_oos	train	cc-by-3.0	4262
msmarco	microsoft/ms_marco	train	Upstream card lacks licence declaration; research use only	2750
esci	amazon-science/esci-data	train	apache-2.0	1071
hover	hover	train	MIT	1420
fever	copenlu/fever_gold_evidence	train	cc-by-sa-3.0; gpl-3.0 (HF card)	455
massive_en	AmazonScience/massive	train	cc-by-4.0	498
massive_ar	AmazonScience/massive	train	cc-by-4.0	297
agentdecision	JevAgentHarness/AgentDecision-train	train	cc-by-nc-4.0	5870
when2call	nvidia/When2Call	train	cc-by-4.0	443
toolace	Team-ACE/ToolACE	train	apache-2.0	2682
humicroedit	Humicroedit	train	Research dataset; see upstream SemEval task terms	1498
newyorker	jmhessel/newyorker_caption_contest	train	cc-by-4.0	851
helpsteer2_helpfulness	nvidia/HelpSteer2	train	cc-by-4.0	3464
helpsteer2_verbosity	nvidia/HelpSteer2	train	cc-by-4.0	494
go_emotions	google-research-datasets/go_emotions	train	apache-2.0	1200
measuring_hate_speech	ucberkeley-dlab/measuring-hate-speech	train	cc-by-4.0	600
wildguard	allenai/wildguardmix	train	odc-by	1077
civil_comments	google/civil_comments	train	cc0-1.0	104
prompt_injections	deepset/prompt-injections	train	apache-2.0	270
paws_safety	google-research-datasets/paws	train	other	397
ar_guardrail	oddadmix/arabic-guardrail	train	apache-2.0	300
jevify:stsb	Praveenrajus/jev-bench	train	cc-by-sa-4.0 (STS Benchmark)	0
public:typed-decisions	LocalLLaMA/typed-decisions	train	apache-2.0	930
inventory_banking77	mteb/banking77	train	mit	59
inventory_clinc150	clinc/clinc_oos	train	cc-by-3.0	53
inventory_go_emotions	google-research-datasets/go_emotions	train	apache-2.0	176
inventory_acos	acos	train	No licence file in pinned upstream checkout; research-only review required	331
contractnli	stanfordnlp/contract-nli	train	CC-BY-4.0; pinned archive LICENSE and TERMS	76
seal_tools	casey-martin/Seal-Tools	train	mit	205
toolret_train	mangopy/ToolRet-Training-20w	train	apache-2.0	207
multiwoz_slots	pfb30/multi_woz_v22	train	apache-2.0	199
sgd_slots	google-research-datasets/dstc8-schema-guided-dialogue	train	cc-by-sa-4.0	845
cuad_clauses	theatticusproject/cuad	train	cc-by-4.0	64
sentfin	tasksource/sentfin	corpus_hash_train	mit	2225
poetry_pairs	yoonholee/poetry-greats-public-domain	corpus_hash_train	cc0-1.0	1446
home_states	acon96/Home-Assistant-Requests	train	mit	258
stance_semeval	SetFit/tweet_eval_stance	train	undeclared	980
chordonomicon	ailsntua/Chordonomicon	corpus_hash_train	cc-by-nc-4.0	2433
glaive_tools	glaiveai/glaive-function-calling-v2	train	apache-2.0	1313
xlam_irrelevance	MadeAgents/xlam-irrelevance-7.5k	train	cc-by-4.0	250
toolace_multicall	Team-ACE/ToolACE	train	apache-2.0	146
dialogue_multiwoz	pfb30/multi_woz_v22	train	apache-2.0	36
agent_eval:when2call	nvidia/When2Call	train	cc-by-4.0	0
agent_eval:glaive_tools	glaiveai/glaive-function-calling-v2	train	apache-2.0	0
agent_eval:xlam_irrelevance	MadeAgents/xlam-irrelevance-7.5k	train	cc-by-4.0	0
agent_eval:toolace_multicall	Team-ACE/ToolACE	train	apache-2.0	0
agent_eval:agentdecision	JevAgentHarness/AgentDecision-train	train	cc-by-nc-4.0	0
toolace_arguments	Team-ACE/ToolACE	train	apache-2.0	1464
glaive_arguments	glaiveai/glaive-function-calling-v2	train	apache-2.0	1829
when2call_actions	nvidia/When2Call	train	cc-by-4.0	2131
medqa	GBaker/MedQA-USMLE-4-options	train	cc-by-4.0	1992
sst5	SetFit/sst5	train	unspecified on upstream card; research-only review	381
agent_eval:toolace_arguments	Team-ACE/ToolACE	train	apache-2.0	0
agent_eval:glaive_arguments	glaiveai/glaive-function-calling-v2	train	apache-2.0	0
agent_eval:when2call_actions	nvidia/When2Call	train	cc-by-4.0	0
procedural	tasksource/procedural-typed-decisions	train	apache-2.0	826
plumb	crh225/plumb-decisions	train	apache-2.0	1715
habermas_train	google-deepmind/habermas_machine	TRAINING_DATA_V1-V5	CC-BY-4.0 data; Apache-2.0 software	1810
transfer:sciq	allenai/sciq	validation	cc-by-nc-3.0	0
transfer:qasc	allenai/qasc	validation	cc-by-4.0	0
transfer:commonsenseqa	tau/commonsense_qa	validation	mit	0
transfer:emotion	dair-ai/emotion	validation	other; see pinned card	0
transfer:unfair_tos	coastalcph/lex_glue	validation	cc-by-4.0	0
transfer:scotus	coastalcph/lex_glue	validation	cc-by-4.0	0
transfer:graph_navigation	tasksource/procedural-typed-decisions	validation	apache-2.0	0
transfer:table_lookup	tasksource/procedural-typed-decisions	validation	apache-2.0	0
transfer:evidence_sufficiency	tasksource/procedural-typed-decisions	validation	apache-2.0	0
transfer:partial_observation_calibration	tasksource/procedural-typed-decisions	validation	apache-2.0	0
dialogue_sgd	google-research-datasets/dstc8-schema-guided-dialogue	train	cc-by-sa-4.0	90