decider-2b: typed decisions with calibrated probabilities in one forward pass

A language model that does not generate text. It reads a state and one or more typed questions, each with an explicit option list, and returns a probability distribution over the options for every question from one forward pass. There is no decoding, no parsing and no output outside the options you defined. It is called from software, not chatted with. It is an open reproduction of the "System One" model class (TypeSafe AI's Jev).

Base model: Qwen/Qwen3.5-2B-Base (1.9B parameters). The supervised stages (v1 to v8) fine-tune it with cross-entropy, a proper scoring rule, on a mixture of about 95 public decision datasets, agent trajectories, web element choice, game states and teacher-written custom questions, in two prompt layouts and with isolated Score levels. This repository holds v10: the v8 weights continued for 384 steps of calibration-aware reinforcement learning whose only rewards are outcomes (live browser task checkers and the exact probability laws of games), with a hard KL limit to the v8 weights on replayed training rows. Code, data registry, training scripts and the recipe are at https://github.com/Mapika/decider; decider/ in this repository is the inference subset of that package.

What changed from v8, measured on the same rows: live browser click tasks 83% to 93% sampled success (held-out tasks 73% to 92%), stated beliefs about action outcomes 0.47 to 0.22 nats above the exact law, Mind2Web +1.5 points, general accuracy and Bespoke's public suite unchanged, OpenJev −0.8 points. Details under Evaluation.

Usage

from decider.infer import Decider          # decider/ is included in this repo
d = Decider("Mapika/decider-2b")
d.decide("My card was charged twice for the same purchase.",
         [{"question": "Which department should handle this?", "options": ["billing", "technical support", "sales"]},
          {"question": "Does this need a refund action?", "options": ["no", "yes"]}])
# [{'choice': 'billing', 'confidence': 0.99, 'probs': {...}}, {'choice': 'yes', 'confidence': 0.99, 'probs': {...}}]

decide_batch scores many states, each with many questions, in one call. abstain_below=t returns None for decisions with confidence under t. A question can have 2 to 255 options (more than 10 options use one label token per option, see decider/prompt.py).

The same request shape as TypeSafe's Jev (POST /v1/systemone), in process or over HTTP:

d.system_one({"ticket": {"messages": [{"from": "customer", "text": "I was charged twice for order A-104. Please refund the duplicate."}]},
              "refund_policy": "Duplicate charges are eligible for a refund."},
             {"department": {"type": "choice", "instructions": "Which team should handle this?",
                             "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
                                          "billing": {"what": "Charges, invoices", "not_for": "delivery"}, "other": None}},
              "refund_requested": {"type": "noul", "instructions": "Does `ticket.messages[0].text` request a refund?"},
              "frustration": {"type": "score", "instructions": "How frustrated is the customer?", "criteria": ["calm", "frustrated", "very frustrated"]}})
# {"model": "decider-v10", "answers": {"department": {"type": "choice", "choice": "billing", "confidence": ..., "certainty": ..., "probabilities": {...}},
#  "refund_requested": {"type": "noul", "noul": ...}, "frustration": {"type": "score", "score": ..., "legend": {...}, ...}}, "usage": {...}}

The state may be a string, object or array (up to 32k tokens with the questions). instructions and every option description may be a string or any JSON value. Question ids are never shown to the model. Each question is scored in its own row, so an answer does not depend on which other questions are asked (independent=False packs them into one row, about half the latency for short states). Each Score level is likewise judged in its own row, without its number or its neighbours, and the per-level fits are normalised ("isolated": false restores listwise scoring). The answer also reports level_fit and their sum fit_mass, which is near 1 when exactly one level fits.

For a fixed set of questions, s = d.schema(questions) computes the question prefix once and s(state) / s.batch(states) then run only the state (1.2 to 2.4x faster per request, up to 19x per batch). It uses a questions-first prompt layout that costs accuracy: about 1.5 points on fixed label sets, 5 on per-example options, more on 50 or more options and on states of several thousand tokens. decider.serve exposes the same thing as POST /v1/systemone; the official typesafe-sdk works against it unchanged with TYPESAFE_BASE_URL pointing at the server.

Requirements: torch, transformers>=5, and flash-linear-attention (Triton kernels for the Qwen3.5 linear-attention layers; the model runs without it but several times slower). Python 3.11 or newer lets those kernels use torch.compile.

Without the helper package, the same computation in plain transformers:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained(REPO); m = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.bfloat16).cuda().eval()
prompt = ("Context:\nMy card was charged twice for the same purchase.\n\n"
          "Question: Which department should handle this?\nOptions:\n(A) billing\n(B) technical support\n(C) sales\nAnswer: (")
ids = tok(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
    logits = m(**ids).logits[0, -1]
letters = [tok.encode(L, add_special_tokens=False)[0] for L in "ABC"]
probs = torch.softmax(logits[letters].float() / 1.30, -1)      # -> P(billing), P(technical support), P(sales); 1.30 is the stored temperature

For several questions in one pass, append further Question k: ... Answer k: ( blocks and read the logits at each ( position (see decider/prompt.py).

How it works

The prompt is Context: ... followed by, for each question, the question text, the lettered options (A) ... (B) ... and an answer slot Answer k: (. The hidden state at each slot is projected with the option-letter rows of the LM head and softmaxed over the valid letters, divided by the temperature in decider_config.json. Letters are never generated, so all slots are read from one pass. Large label sets were sub-sampled to at most 10 options per training example (gold always kept, order shuffled), so the model conditions on the supplied candidates rather than on a fixed head.

Field types

• noul: probability of "yes".
• choice with criteria {name: description | JSON | null}: the argmax option, its probability (confidence, the calibrated number), certainty (1 minus the normalised entropy) and the full distribution.
• score with criteria [level descriptions]: the expected level, the probability of the most likely level, the distribution, and the per-level fits.

Training

Supervised stages (v1 to v8). One epoch on a mixture of public decision datasets (intent detection, ticket routing, topic classification, sentiment, emotion, moderation, NLI, paraphrase, fact verification, passage relevance, reading comprehension, multiple-choice QA, ordinal rating scales, pairwise response preference, tool selection), then continuation epochs that added next-action choice from agent trajectories (AgentGym), web element choice (Mind2Web), teacher-written situations and game states, the input shapes of the Jev API (described options, up to 255 options, JSON states with path references, long inputs), teacher- written custom questions with a generic option next to a catch-all, a second cacheable prompt layout, and isolated Score levels. In 10% of questions with three or more options an abstain option is added; in a quarter of those the option list is replaced by labels from an unrelated task so that the abstain option is correct. The full list of components with sizes is in decider/data/mixture.py of the GitHub repository; scripts/train.sh full reproduces the supervised stages in one run.

Reinforcement learning stage (v8 to v10). 384 optimizer steps at a peak learning rate of 1e-6 (cosine, 16 warm-up steps), selected among the checkpoints of a 576-step run. Each of the 48 iterations plays 4 live MiniWoB++ click tasks, 4 minesweeper boards and 4 game boards (a 5x5 grid with a slippery move, draws from bags of known composition), 4 repeats each, through the same one-pass readout that serves requests. Three loss terms use those rollouts: a PPO clipped surrogate (clip 0.2) on the terminal outcome with a leave-one-replicate-out baseline; a proper log score of the model's stated belief about the immediate outcome of its action against the exact law (games, minesweeper) or the realised outcome (browser); and a rendering-consistency term that pulls the model's answer in the other prompt layout and the reversed option order toward its served answer. A fourth term keeps the model where it was: on 8 replayed supervised rows per step, KL(v8 ‖ student) on the served distribution must stay under 0.01 nats on average and 0.05 on any row, otherwise the step drops the reward terms and follows only the KL gradient. Six browser tasks were held out from reward and used for validation only. No gold labels were used. The recipe and every measurement are in docs/RL.md of the GitHub repository.

Evaluation

94 public tasks, original protocol. Large label sets sub-sampled to 10 options; one temperature fitted on in-task data and stored in decider_config.json. "In-task" means the test splits of the training datasets; "held-out" means datasets never seen in training (TREC, BBC news, PAWS, SciQ, Social IQa, StrategyQA, PubMedQA, TruthfulQA, tweet irony, financial sentiment, ADE, MASSIVE scenario, student question categories, Dolly categories, CR reviews, Financial PhraseBank, CommitmentBank, QuALITY, XStoryCloze, RewardBench, Arena preferences, Hermes tool selection, and an abstention probe). ECE is the expected calibration error with 15 bins.

model	in-task (69 tasks) acc / NLL / ECE	held-out (24 tasks) acc / NLL / ECE
Qwen3.5-2B-Base, zero-shot	0.620 / 0.908 / 0.121	0.642 / 0.853 / 0.105
decider-2b v8, T=1.30	0.811 / 0.460 / 0.037	0.741 / 0.655 / 0.088
decider-2b v9, T=1.36	0.812 / 0.464 / 0.041	0.741 / 0.655 / 0.087
decider-2b v8, rebuilt set (67 / 28 tasks, see note), T=1.30	0.806 / 0.473 / 0.038	0.757 / 0.622 / 0.083
decider-2b v10 (this repository), rebuilt set, T=1.30	0.805 / 0.474 / 0.037	0.755 / 0.622 / 0.084
v8, questions-first layout (schema cache), T=1.18	0.790 / 0.500 / 0.038	0.707 / 0.757 / 0.104

The two "rebuilt set" rows were measured after the data pipeline was rebuilt on another machine: two datasets no longer download (TREC-fine, the game states) and the current mixture adds held-out probes, so that set has 67 in-task and 28 held-out tasks. Its numbers are comparable to each other, not to the rows above. v10 matches v8 on it.

Per-task accuracy / ECE on the held-out datasets of the rebuilt set, v8 against v10:

task	v8 acc / ECE	v10 acc / ECE
abstain_probe	0.633 / 0.112	0.606 / 0.134
ade	0.811 / 0.044	0.817 / 0.038
arena_pref	0.487 / 0.173	0.483 / 0.189
bbc_news	0.924 / 0.014	0.927 / 0.013
cb	0.911 / 0.090	0.857 / 0.093
cr_reviews	0.900 / 0.027	0.903 / 0.031
dbpedia_l2	0.948 / 0.017	0.950 / 0.018
dbpedia_l3	0.989 / 0.007	0.987 / 0.005
dolly_category	0.291 / 0.209	0.299 / 0.203
fin_phrasebank	0.684 / 0.043	0.694 / 0.042
fin_sentiment	0.794 / 0.069	0.793 / 0.058
hermes_tools	0.718 / 0.209	0.723 / 0.208
hwu64	0.964 / 0.031	0.961 / 0.030
massive_scenario	0.766 / 0.040	0.756 / 0.041
offtopic_probe	0.841 / 0.033	0.841 / 0.027
paws	0.707 / 0.169	0.724 / 0.145
pubmedqa	0.752 / 0.083	0.756 / 0.085
quality	0.495 / 0.236	0.494 / 0.233
quality_full	0.505 / 0.205	0.508 / 0.198
reward_bench	0.825 / 0.042	0.819 / 0.045
sciq	0.982 / 0.022	0.982 / 0.024
social_iqa	0.698 / 0.072	0.708 / 0.077
strategyqa	0.559 / 0.123	0.552 / 0.138
student_questions	0.927 / 0.036	0.925 / 0.045
trec	0.792 / 0.057	0.784 / 0.066
truthfulqa	0.529 / 0.102	0.537 / 0.090
tweet_irony	0.801 / 0.048	0.795 / 0.052
xstory_cloze	0.962 / 0.017	0.962 / 0.017

v10 against v8 on the same rows. Every row below is scored by both models on identical inputs and seeds. Intervals are 95% bootstrap or paired intervals.

	v8	v10	difference
live MiniWoB++ click tasks, 22 tasks x 8 seeds, sampled play	83.0%	93.2%	+10.2 (+5.1 to +15.9)
the 6 tasks never used for reward	72.9%	91.7%	+18.8 (+6.2 to +31.2)
same tasks, greedy play	90.3%	90.9%	+0.6
Mind2Web element and action choice, 1,770 rows	81.1%	82.7%	+1.5 (+0.7 to +2.4)
bag-draw games, win rate, 64 boards x 4	35.2%	41.4%	+6.2 (+0.8 to +11.7)
slippery-grid games, win rate, 64 boards x 4	14.1%	18.8%	+4.7 (−2.0 to +11.3)
stated belief, nats above the exact law (lower is better)	0.473	0.219
click-outcome prediction, log score (higher is better)	−0.349	−0.034
TypeSafe workflow decisions, 102 rows, accuracy / NLL	78.4% / 0.594	80.4% / 0.585	+2.0 (−2.0 to +5.9)
847 in-task validation rows, accuracy / NLL	83.6% / 0.443	83.2% / 0.444	−0.4 (−1.3 to +0.6)
Bespoke's public suite, 13 subsets, macro accuracy	0.706	0.704
JevBench public items, easy / standard / hard accuracy	1.000 / 0.861 / 0.459	1.000 / 0.847 / 0.459
OpenJev, 5,252 rows, accuracy / NLL	64.1% / 0.906	63.3% / 0.916	−0.8 (−1.3 to −0.3)

The browser gain is in the served distribution rather than in the argmax: sampled play improves by ten points, greedy play by under one. Tic-tac-toe and minesweeper play did not change; a 2B model without search loses most of those games either way. The one measured regression is OpenJev, under one point.

Bespoke's public suite (13 human-labelled subsets, 3,880 records in Jev's wire format, answered through system_one as shipped). decider-2b v10 macro 0.704 / micro 0.711; v9 0.701 / 0.711; Nimble-9B 0.748 / 0.759; Jev 1.13.0 0.760 / 0.773 (the last two copied from Bespoke's report). Per-subset numbers, the JevBench public-item comparison (decider-2b v10 is at 1.000 / 0.847 / 0.459 on the easy / standard / hard public items, against Jev 1.13.0 at 1.000 / 0.986 / 0.730) and recordings of both versions on the same browser pages and game boards are in the GitHub README.

Speed

One NVIDIA GH200, bf16, unchanged from v8 (same architecture, readout and temperature). decider.infer.Decider uses shape-bucketed CUDA graphs; the batching server is decider/serve.py. Support-ticket states of about 230 tokens with 3 to 5 typed questions each:

setting	p50 latency	throughput
single request, eager PyTorch	49 ms
single request, CUDA graphs + torch.compile (helper default)	4.0 ms
batch of 32, in-process, bf16	70 ms	about 1,370 decisions/s
batch of 32, in-process, FP8 linears	58 ms	about 1,670 decisions/s
HTTP server (FP8), 1 client	6.8 ms	134 req/s
HTTP server (FP8), 64 clients	126 ms	431 req/s, 2,152 decisions/s

With the schema cache (Decider.schema), 10 described questions on short chat messages run at 11,180 decisions/s in a batch, and one question with 151 options at 19x the full-forward rate. FP8 (e4m3 weights, per-token activation scales) changes accuracy and calibration by less than the evaluation noise.

Limitations

• A 2B model without reasoning. Knowledge-heavy multiple choice (MMLU, MedQA, ARC) improves little over the base model, and a judgment that needs several steps should be split into several questions.
• English only. Calibration is measured on public datasets and teacher-labelled probes, not on your traffic. Check it on your own labels before using confidence for routing.
• v10 continues the v8 weights. The v9 data for terse bucket names (support, help, account next to other) is not in it: on held-out terse-bucket messages v8 chose the generic bucket correctly 59% of the time where v9 reached 86%. Name or describe the generic option as a bucket (general_support, or a description).
• Rules written into the question ("fill if empty, otherwise skip") are not followed at this size. State the decision as a plain question with described options.
• Picking one record out of a long JSON array by position is the least accurate input shape (0.51 with 64 records against 0.70 with one). Address records by key, or let the helper write the index into the array (0.62).
• Full label sets cost accuracy against 10 sampled options: CLINC 151-way 0.88 against 0.98; DBpedia level 2 with 70 labels is the least calibrated case (ECE 0.14).
• Questions packed into one row (independent=False) see the earlier question texts, and reversing their order changes up to 12% of answers. The default path scores each question alone.
• The v10 browser results are on 22 click-only MiniWoB++ tasks: small synthetic pages with the elements listed as text. Typing, scrolling and real websites were not tested. OpenJev accuracy is 0.8 points lower than v8.
• Abstention: a catch-all option ("none of the above", "other", "unsure") is chosen when nothing on offer fits, not when the exact fine-grained label is merely absent. Wordings far from the training data remain the main risk.
• One in-task dataset, tweet_hate (SemEval-2019 HatEval), stays near chance on its test split, whose collection and label definition differ from the training split. The number is reported as measured.

Reproduction

Code, data registry, training and evaluation scripts, the RL recipe and the per-version history: https://github.com/Mapika/decider. Each release is staged with scripts/stage_release.py and uploaded with scripts/upload_hf.py; the previous weights are kept under the tag v8 in this repository.