license: apache-2.0
language: en
base_model: answerdotai/ModernBERT-base
library_name: decide
pipeline_tag: zero-shot-classification
tags:
- modernbert
- decision-model
- zero-shot-classification
- calibrated
decide-0.2.0
A small, fast, non-autoregressive decision model. Given some state, an instruction and a set of candidate options, it scores every option in a single forward pass and returns calibrated probabilities. It answers three kinds of question with the same weights:
- choice: a distribution over K mutually exclusive options
- noul: the probability that a condition holds (a yes/no question)
- score: an expected level on an ordered scale
It is fine-tuned from ModernBERT-base (149M parameters).
The packed input is <instructions> <state> [SEP] [MASK] option0 [MASK] option1 ...; a small MLP head scores the
encoder state at each [MASK]. Each option attends only to the shared prefix and to itself and its position ids restart
after the prefix, so an option's score depends on (premise, that option) alone: permuting the options permutes the
scores and changes nothing else (measured max logit change under permutation: 8.9e-06).
The model, tokenizer, training loop, evaluation and this card were all produced with the pure-Go toolchain in jmwri/decide; no Python or PyTorch was used to train it.
Usage
import "github.com/jmwri/decide"
decide.SetDefault(decide.NewLocal("/path/to/this/model")) // or: decide pull --repo jmwri/decide
ans, _ := decide.Decide(ctx, "Replication lag exceeded 45 seconds.", decide.Options{
{Key: "infrastructure", Description: "Database, hardware, network or server failures"},
{Key: "billing", Description: "Invoices, payments, refunds"},
}, "Classify the root cause domain.")
Evaluation
Accuracy is top-option accuracy. ECE is the expected calibration error (15 bins) of the top-option confidence,
after the per-kind temperature scaling shipped in decide.json.
Held-out validation splits of the training tasks
| task | n | accuracy | NLL | ECE |
|---|---|---|---|---|
| all | 8127 | 80.2% | 0.505 | 0.007 |
| kind: choice | 4745 | 78.7% | 0.569 | 0.011 |
| kind: noul | 2703 | 89.5% | 0.259 | 0.009 |
| kind: score | 679 | 53.5% | 1.041 | 0.034 |
| amazon_polarity | 300 | 94.0% | 0.178 | 0.035 |
| arc | 200 | 49.0% | 1.272 | 0.124 |
| arc_easy | 200 | 66.0% | 0.857 | 0.079 |
| banking77 | 300 | 96.7% | 0.094 | 0.032 |
| bitext_support | 300 | 100.0% | 0.011 | 0.011 |
| boolq | 300 | 83.3% | 0.412 | 0.043 |
| civil_comments | 156 | 76.9% (MAE 0.48 levels) | 0.543 | 0.086 |
| clinc150 | 300 | 98.3% | 0.058 | 0.019 |
| cola | 200 | 79.0% | 0.419 | 0.086 |
| commonsense_qa | 300 | 65.0% | 0.880 | 0.042 |
| dbpedia14 | 300 | 70.0% | 1.027 | 0.163 |
| go_emotions | 256 | 84.4% | 0.463 | 0.079 |
| hellaswag | 300 | 68.7% | 0.818 | 0.065 |
| massive_intent | 300 | 97.0% | 0.113 | 0.037 |
| massive_scenario | 300 | 96.3% | 0.141 | 0.040 |
| mnli | 300 | 87.3% | 0.359 | 0.061 |
| mrpc | 200 | 90.5% | 0.234 | 0.048 |
| newsgroups | 296 | 85.5% | 0.440 | 0.058 |
| openbookqa | 250 | 59.6% | 0.957 | 0.056 |
| qasc | 250 | 49.6% | 1.370 | 0.053 |
| qnli | 200 | 90.5% | 0.228 | 0.059 |
| qqp | 200 | 87.0% | 0.331 | 0.057 |
| rte | 200 | 82.0% | 0.391 | 0.038 |
| sms_spam | 300 | 100.0% | 0.009 | 0.009 |
| snli | 295 | 87.5% | 0.346 | 0.073 |
| sst2 | 300 | 95.0% | 0.177 | 0.048 |
| sst5 | 300 | 54.7% (MAE 0.55 levels) | 1.034 | 0.069 |
| stsb | 300 | 51.3% (MAE 0.60 levels) | 1.093 | 0.049 |
| toxic_conversations | 124 | 91.9% | 0.258 | 0.054 |
| wanli | 300 | 74.7% | 0.583 | 0.042 |
| winogrande | 300 | 66.0% | 0.715 | 0.110 |
Tasks in this second table were never trained on (zero-shot to the model):
Held-out tasks
| task | n | accuracy | NLL | ECE |
|---|---|---|---|---|
| all | 4500 | 47.6% | 1.268 | 0.133 |
| kind: choice | 2500 | 42.8% | 1.415 | 0.128 |
| kind: noul | 1000 | 58.9% | 1.011 | 0.266 |
| kind: score | 1000 | 48.2% | 1.159 | 0.072 |
| amazon_stars | 1000 | 48.2% (MAE 0.71 levels) | 1.159 | 0.072 |
| emotion | 1000 | 55.1% | 1.244 | 0.084 |
| enron_spam | 1000 | 58.9% | 1.011 | 0.266 |
| mmlu | 1500 | 34.5% | 1.529 | 0.166 |
Training data
Fine-tuned on 301661 examples built from the public datasets below (options are re-sampled and verbalised in several styles; classification tasks are also posed as yes/no verification questions). Licenses are as declared on the Hugging Face hub; review them before redistributing or commercialising the weights, since several datasets carry share-alike or unspecified terms.
| dataset | train examples | license (as declared) |
|---|---|---|
| amazon_polarity | 8000 | apache-2.0 |
| amazon_stars (held out, evaluation only) | 0 | unspecified (mteb/amazon_reviews_multi) |
| arc | 1119 | cc-by-sa-4.0 |
| arc_easy | 2251 | cc-by-sa-4.0 |
| banking77 | 9993 | cc-by-4.0 |
| bitext_support | 8990 | cdla-sharing-1.0 |
| boolq | 9000 | cc-by-sa-3.0 |
| civil_comments | 9391 | cc0-1.0 |
| clinc150 | 15000 | cc-by-3.0 |
| cola | 6000 | other (GLUE) |
| commonsense_qa | 9000 | mit |
| dbpedia14 | 8000 | cc-by-sa-3.0 |
| emotion (held out, evaluation only) | 0 | other (dair-ai) |
| enron_spam (held out, evaluation only) | 0 | unspecified (SetFit/enron_spam) |
| go_emotions | 7520 | apache-2.0 |
| hellaswag | 22000 | mit |
| massive_intent | 11499 | apache-2.0 |
| massive_scenario | 11500 | apache-2.0 |
| mmlu (held out, evaluation only) | 0 | mit |
| mnli | 30000 | cc-by-3.0/cc-by-sa-3.0/mit/other |
| mrpc | 3668 | other (GLUE) |
| newsgroups | 10707 | unspecified (SetFit/20_newsgroups) |
| openbookqa | 4957 | unknown |
| qasc | 8000 | cc-by-4.0 |
| qnli | 8000 | other (GLUE) |
| qqp | 8000 | other (GLUE) |
| rte | 2490 | other (GLUE) |
| sms_spam | 4433 | unknown (UCI) |
| snli | 15976 | cc-by-sa-4.0 |
| sst2 | 8000 | other (GLUE) |
| sst5 | 8500 | unspecified (SetFit/sst5) |
| stsb | 5748 | other (GLUE) |
| toxic_conversations | 6138 | cc-by-4.0 |
| wanli | 16000 | cc-by-4.0 |
| winogrande | 22000 | cc-by-4.0 |
Limitations
- English only. Inputs are limited to 8192 tokens; option sets are best kept under ~10 (use a two-stage taxonomy beyond that).
- Trained on public classification, inference and multiple-choice datasets, so it is strongest on those styles of question. Explicit, descriptive option texts work better than bare labels. Domain-specific jargon or rubrics with no descriptive criteria fall back to general language priors.
- Probabilities are calibrated on held-out data from the training tasks; on very different distributions they will be over-confident. Use a confidence gate and route uncertain cases to review.
- Not for decisions that affect people's rights, safety or livelihood without human oversight.
License
Weights: Apache-2.0 (ModernBERT-base is Apache-2.0), subject to the training-data terms above.