ktcapraz/t5gemma2-decision-270m-v2

T5Gemma2 Decision 270M v2

A small decision model on an encoder-decoder base. It reads a text and answers typed questions with probabilities: choice (one option of a set), noul (probability of yes) and score (a position on ordered levels). It also highlights the parts of the text it used.

It scores 51.8 on S1MB, rank 19 of 100 published models when it was scored (2026-10-08). On tests whose task it trained on, it beats bekko-system-one-v0-400m (61.7 vs 59.1). On tests it never trained on, it is weak (27.1 vs 39.8).

To our knowledge, most open decision models use an encoder-only or decoder-only base. This one is a fine-tune of google/t5gemma-2-270m-270m.

The request format follows TypeSafe's System One API. This model is not made by, or affiliated with, TypeSafe.

Results

S1MB is 100 x the mean skill over 137 English tests. Scores use the official per-test metrics and are from results/s1mb_tree.json.

Model	All 137	Trained tasks (98)	Never-trained tasks (39)	choice (57)	noul (59)	score (21)
v2, fast readout	51.8	61.7	27.1	55.9	50.6	44.2
v2, final readout	51.7	-	26.7	55.8	50.2	44.4
bekko-system-one-v0-400m	53.6	59.1	39.8	61.3	51.2	39.2
Jev 1.13	63.0	64.3	59.6	67.2	64.6	46.9

"Trained" means that the task has a training split in hotchpotch/bekko-system-one-dataset-v0, which we used. We never trained on S1MB test rows; its test split is the test split of that dataset, and training rows were filtered against it.

Other sets: accuracy 0.737 on 10,788 dev questions (final readout), and 72.1 on our own 147-question real-use set.

Quick start

The model needs the code in decisions/. It does not load with plain AutoModel.

pip install -r requirements.txt
python run_example.py

Output from a real run (CPU, 4 threads, about 0.9 s):

billing {"type": "noul", "noul": 0.984755}
tone {"type": "choice", "choice": "angry", "confidence": 0.138584, "probabilities": {"angry": 0.569292, "calm": 0.430708}}
urgency {"type": "score", "score": 1.2781, "confidence": 0.237147, "probabilities": {"0": 0.115214, "1": 0.491431, "2": 0.393355}}

In your own code:

import os
os.environ["T5D_READOUT"] = "final"   # or "fast", or "both"

from decisions.engine import Engine, Timer
from decisions.format import parse_request

engine = Engine(".", threads=6, max_state_tokens=8192, layout="isolated", local_attn=256)
state, questions = parse_request({
    "state": "I was charged twice for my subscription this month. Please refund the extra charge.",
    "questions": {
        "billing": {"type": "noul", "instructions": "Is this message about billing?"},
        "tone": {"type": "choice", "instructions": "What is the tone of this message?",
                 "criteria": {"angry": "An upset or hostile message", "calm": "A neutral or polite message"}},
        "urgency": {"type": "score", "instructions": "How urgent is this message?",
                    "criteria": ["Can wait", "Needs attention this week", "Needs attention today"]},
    },
})
answers, usage, meta = engine.decide(state, questions, Timer())
print(answers["billing"]["noul"], answers["billing"]["evidence"], answers["billing"]["quote"])

Each answer also has evidence (spans with probabilities, as character offsets), quote and nei (a "not in the text" signal; we found it unreliable on clear texts).

Use layout="isolated" when questions must stay independent (see Limitations).

Speed on a server CPU: about 0.9 s for 1 question on a 900-token text. With 3 questions on 900 tokens, shared takes about 1.3 s and isolated about 3.5 s.

How it works

• Base: T5Gemma 2 270M-270M. 18 encoder layers and 18 decoder layers, hidden size 640. The word table and the tied LM head stay frozen.
• Encoder row: the state, then each question, then its options. Options share start positions and are masked from each other, so option order cannot change an answer.
• Fast readout at Answer:: LM-head score of the option words ("voice") + a bilinear score with the pooled option vector + a per-option bias.
• Final readout: one decoder branch per option, ending in True?. Its score is Yes minus No from the frozen LM head, plus a helper capped at +-3.
• Highlight: a token score over the state for each question. The top tokens go into the decoder as a quote.
• Calibration: a temperature per question type, per readout, fitted on dev sets.

The training run also had a verifier row and a "think longer" loop. Both had zero effect, so they are not in this release.

What we learned by examining the model

We examined the trained model directly after training. These results matter if you build on it:

• The encoder makes the answer. With a constant decoder state, S1MB is 51.4 against 51.8 for the full path.
• The decoder does not read the text. With the text hidden from it, 0% of answers flip. With only the text visible, its verdict barely beats no input on new tasks (0.33 vs 0.28).
• Text helps less on new tasks. It lifts accuracy from 0.34 to 0.43 on new tasks, and from 0.44 to 0.74 on practiced rows.
• Layer 14 carries most of the answer: 46% of the logit size on new tasks and 34% on practiced rows.

Limitations

• Weak on tasks it never trained on (27.1, against 59.6 for Jev 1.13). The gap to Jev is almost all here.
• Questions can change each other in shared layout. State tokens read every question in the row. In one test, a second question moved a first answer from refund 0.909 to replacement 0.795. isolated fixes this for normal questions and takes about 2.7 times as long. Chained questions (depends_on) always share a row, so a child can still change its parent.
• Sensitive to word form. Capitalizing 30% of the words, with no change in meaning, flips 17 to 27% of answers. Real-use accuracy falls from 0.727 to 0.644 in that test. Natural changes such as a trailing new line flip 1 to 10%.
• Yes/no cutoff on new tests. New tests rank well (AUC 0.787), but the 0.5 cutoff is often wrong: skill is 30.5, and 40.1 with a cutoff tuned per test. Do not treat 0.5 as a universal threshold on a new task.
• Highlights do not drive decisions. Treat them as a hint, not as proof of the reason for an answer.
• Tested on English only.
• No safety evaluation. Do not use it alone for decisions about people, or in medical, legal or financial settings.

Training

The path from the base was: s2, s3, s4, s5 (decisions-first data), p1 (bekko data), p1c (more span labels), s6b (tree decoder, step 10000), v2.

The final run took 4 h 40 min on one A100-80GB, stopped at step 14,274, at learning rate 1e-5.

Group	Share of the v2 mix	Source
weak	38.6%	28 subsets of hotchpotch/bekko-system-one-dataset-v0, 30K questions each
bekko	24.1%	Other subsets of the same training split
evidence	19.3%	Span-labelled questions (for example ERASER, HotpotQA)
others	rest	avbiswas/bev-decision, LocalLLaMA/typed-decisions, n4ze3m/typed-decisions-synth, Praveenrajus/jev-bench, tasksource, and judgment sets such as HelpSteer, UltraFeedback, Feedback-Collection, ToxicChat, PKU-SafeRLHF, Bitext, CaseHOLD, UnfairToS, CoS-E, ProSocial, Enron spam

Training data licences differ between sources, and some are non-commercial (for example PKU-SafeRLHF and ToxicChat). Some source datasets contain labels made by other models. Check these before commercial use.

Files

File	Content
model.safetensors	Text encoder and decoder weights (fp32, no image encoder)
head.safetensors	Decision head
decision_config.json	Input format, tree settings, calibration, training metrics
config.json, generation_config.json, tokenizer.json, tokenizer_config.json	Base model files
decisions/	Inference code
run_example.py, requirements.txt	Example and pinned versions
results/s1mb_tree.json	S1MB per-test skill for each readout
NOTICE	Gemma notice and list of changes

License

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. This model is a modified version of google/t5gemma-2-270m-270m: the weights were changed by fine-tuning, and a decision head was added. The Gemma Prohibited Use Policy applies to this model and to anything built from it. The code in decisions/ is provided under the same terms as the model.