fabriciocarraro/CatDecide-1

CatDecide v0.11

CatDecide is a decision model for Catalan. It reads a state (a text, or a JSON value) and answers typed questions about it: a choice among options, yes or no, or a score on ordered levels. It reads each answer from the next-token probabilities of the options' codes, so no text is generated, and the probabilities are calibrated. Catalan comes first; the model also reads Spanish and English.

It has 9B parameters in bf16 and is fine-tuned from a base model that is not public. v0.11 was frozen on 3 Nov 2026.

Access is granted on request. Please do not redistribute the weights.

How to use

You give the model a state and typed questions about it. Each question becomes one prompt: the state fenced between two marks, the question, and the options, each behind a one-letter code. The answer is read from the probabilities of those codes at the next token. A question costs one forward pass per option order read (per group of options, for long lists), and nothing is sampled.

In Python, with Transformers

example.py, in this repository, is a self-contained copy of the served readout in about 200 lines: the prompt, the option orders, the calibration and the answer rules.

Access is gated: request it on this page, then log in with the approved account.

pip install torch "transformers>=5" huggingface_hub   # transformers 5: the config writes its YaRN rope in that format
hf auth login
hf download fabriciocarraro/catdecide-v0.11 example.py --local-dir .
python example.py                                      # downloads the weights (18 GB) and answers the request below

from example import CatDecide

model = CatDecide()   # the weights from the Hub, on "cuda"; or CatDecide("/a/local/folder")
answers = model.decide(
    "Bon dia. Fa tres setmanes que el fanal de davant del número 12 del carrer Major no s'encén, i a la nit el tram "
    "queda completament a les fosques. Ho vaig comunicar fa quinze dies i encara no hi ha vingut ningú.",
    {
        "area": {"type": "choice", "instructions": "A quin servei municipal s'ha d'enviar el missatge?",
                 "criteria": {"enllumenat": "Enllumenat públic", "neteja": "Neteja viària",
                              "obres": "Obres i via pública", "padro": "Padró d'habitants"}},
        "queixa": {"type": "noul", "instructions": "El missatge és una queixa?"},
        "urgencia": {"type": "score", "instructions": "Quina urgència té el missatge?",
                     "criteria": ["Baixa", "Normal", "Alta", "Urgent"]},
    },
)

What it returns (rounded):

{"area":     {"choice": "enllumenat", "confidence": 0.984, "decided": false,
              "probabilities": {"enllumenat": 0.984, "neteja": 0.001, "obres": 0.014, "padro": 0.001}},
 "queixa":   {"choice": "true", "confidence": 0.951, "decided": false,
              "probabilities": {"true": 0.951, "false": 0.049}},
 "urgencia": {"choice": "2", "confidence": 0.340, "within_one": 0.822, "decided": false,
              "probabilities": {"0": 0.178, "1": 0.286, "2": 0.340, "3": 0.196}}}

None of these answers reaches its 1 % threshold (below); area and queixa clear the 5 % ones.

The three question types:

type	criteria	choice in the answer
choice	option id → description; 2 or more options (above 26, read in groups of 20 and a final pass)	the most probable option id
noul (yes/no)	none: the prompt shows "Sí" and "No"	"true" or "false"
score	the levels' descriptions, in order; 2 to 10	the level index ("2" is "Alta" above): the lower median of the distribution

Reading the answer:

• probabilities are calibrated, with a temperature per question type (in calibration.json, fitted on a development set), and averaged over the option orders read: a choice in four orders, a yes/no question in both.
• confidence is the probability of the answer given.
• decided says whether the answer reaches the development set's 1 % error rate. A choice needs a confidence of at least 0.987, a yes/no answer 0.954. A score is decided when within_one, the probability that the true level lies within one level of the answer, reaches 0.978.
• An undecided answer is still the model's best answer: send it to a person, or use the 5 % thresholds in calibration.json (0.891 for a choice, 0.887 for yes/no).

Good practice:

• Ask every question about a state in one call. Each question is read alone: none sees another's options or answer.
• Describe the options in words ("Enllumenat públic", not "E1"). Instructions and options can be in Catalan, Spanish or English; the prompt's fixed parts are Catalan.
• When the state may not settle a question, offer that as an option ("El text no ho diu"): the model takes it when the state does not decide.
• The state is fenced and read as data, not as instructions. The model is hard to steer from inside the state, but not impossible to steer.
• Keep thinking off (enable_thinking=False in the chat template; the example sets it): with thinking on, the template opens a thought where the answer should be.
• Do not call generate(). The answer is the calibrated distribution averaged over the orders, not a sampled token.

Speed: served with vLLM on one H100, with the state's prefill shared among its questions, one question on a fresh 3k-token state takes 144 ms over HTTP, and twenty questions on a 4k-token state 336 ms (38 and 137 ms when the state is cached). On a fresh 30,000-token state, one question takes 1.6 s and twenty take 2.2 s.

Evaluation

The model was read as served: vLLM in bf16, choice questions in four orders, its calibration, and score questions answered by the median level. The items are drawn from the test splits of public benchmarks, none of which was used in training.

Benchmark	Task	Options	Items	Accuracy
TE-ca	textual entailment	3	999	79.8 %
XNLI-ca	natural language inference	3	999	87.3 %
XNLI-va	the same, in Valencian	3	999	85.5 %
Parafraseja	paraphrase or not	yes/no	1,000	89.4 %
InToxiCat	toxic or not	yes/no	1,200	79.4 %
CaBBQ	social bias in question answering	3	1,200	89.4 %
NLUCat	intent	88	1,440	83.4 %
TeCla	news topic	53	1,500	61.8 %
GuiaCat	review rating	5 levels	1,000	59.9 % (98.7 % within one level)
STS-ca	sentence similarity	6 levels	500	54.4 % (91.2 % within one level)

Knowledge questions, answered from what the model knows rather than from the state:

Benchmark	Items	Accuracy
ARC-Easy (Catalan)	2,375	91.1 %
ARC-Challenge (Catalan)	1,171	79.0 %
OpenBookQA (Catalan)	500	75.6 %
VeritasQA (Catalan / Spanish / English)	284 / 284 / 286	61.3 / 59.9 / 58.0 %

Tool use and safety, on English benchmarks:

Benchmark	Question	Items	Result
When2Call	an assistant's next step: call a listed tool, ask the user, say no listed tool can do it, or answer directly	3,423	82.3 % right
WildGuardTest	is the request harmful? is the reply a refusal? is the reply harmful?	880	F1 91.9, 95.9 and 88.6
XSTest	is the request harmful?	450	flags 95.0 % of the unsafe requests and 15.2 % of the safe ones
JailbreakBench	is the request harmful?	500	flags 95.0 % of the plain harmful requests, 70.3 % of the jailbreak prompts and 35.0 % of the benign requests

When2Call's right answer is a tool call (88.4 % right), a question to the user (87.9 %) or that no listed tool can do it (72.6 %). WildGuardTest is read on the 880 of its 1,725 items whose three annotators agree on all three labels.

Limitations

• Knowledge. Questions that need facts the state does not give, and common misconceptions above all (VeritasQA, 58–61 %), are its weak side.
• Computations. It reads dates, amounts and formats well, but a computation of a kind it has not seen (a new fee rule, a deadline counted a new way) is unreliable. Compute it in code and ask the model about the text.
• Scores are coarse. The exact level is right 54–60 % of the time, and within one level 91–99 % of the time.
• Consistency across languages. The same text in Catalan and in Spanish can get different answers.
• Tools. Seeing that none of the listed tools can do what is asked is its weakest tool decision.
• Safety screening. It flags some safe requests that only look unsafe, and it lets about three jailbreak prompts in ten through. Use it as one layer of a guardrail, not the only one.
• Length. It is served with prompts of up to 32,768 tokens; split longer states.

Intended use

Routing, classifying and checking texts in Catalan administration and services: citizens' messages, documents and records. It can also choose an assistant's next step with tools, and screen requests and replies. The calibrated probabilities can gate automation: act on decided answers and send the rest to a person.

It is not meant for generating text, for open questions, or for decisions about people without human oversight.