---
license: apache-2.0
language: en
base_model: answerdotai/ModernBERT-base
library_name: decide
pipeline_tag: zero-shot-classification
tags:
  - modernbert
  - decision-model
  - zero-shot-classification
  - calibrated
---

# decide-0.2.0

A small, fast, **non-autoregressive decision model**. Given some state, an instruction and a set of candidate
options, it scores every option in a single forward pass and returns calibrated probabilities. It answers
three kinds of question with the same weights:

- **choice**: a distribution over K mutually exclusive options
- **noul**: the probability that a condition holds (a yes/no question)
- **score**: an expected level on an ordered scale

It is fine-tuned from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) (149M parameters).
The packed input is `<instructions> <state> [SEP] [MASK] option0 [MASK] option1 ...`; a small MLP head scores the
encoder state at each `[MASK]`. Each option attends only to the shared prefix and to itself and its position ids restart
after the prefix, so an option's score depends on *(premise, that option)* alone: **permuting the options permutes the
scores and changes nothing else** (measured max logit change under permutation: 8.9e-06).

The model, tokenizer, training loop, evaluation and this card were all produced with the pure-Go toolchain in
[jmwri/decide](https://github.com/jmwri/decide); no Python or PyTorch was used to train it.

## Usage

```go
import "github.com/jmwri/decide"

decide.SetDefault(decide.NewLocal("/path/to/this/model")) // or: decide pull --repo jmwri/decide
ans, _ := decide.Decide(ctx, "Replication lag exceeded 45 seconds.", decide.Options{
    {Key: "infrastructure", Description: "Database, hardware, network or server failures"},
    {Key: "billing", Description: "Invoices, payments, refunds"},
}, "Classify the root cause domain.")
```

## Evaluation

Accuracy is top-option accuracy. ECE is the expected calibration error (15 bins) of the top-option confidence,
after the per-kind temperature scaling shipped in `decide.json`.

### Held-out validation splits of the training tasks

| task | n | accuracy | NLL | ECE |
| :-- | --: | --: | --: | --: |
| **all** | 8127 | **80.2%** | 0.505 | 0.007 |
| kind: choice | 4745 | 78.7% | 0.569 | 0.011 |
| kind: noul | 2703 | 89.5% | 0.259 | 0.009 |
| kind: score | 679 | 53.5% | 1.041 | 0.034 |
| amazon_polarity | 300 | 94.0% | 0.178 | 0.035 |
| arc | 200 | 49.0% | 1.272 | 0.124 |
| arc_easy | 200 | 66.0% | 0.857 | 0.079 |
| banking77 | 300 | 96.7% | 0.094 | 0.032 |
| bitext_support | 300 | 100.0% | 0.011 | 0.011 |
| boolq | 300 | 83.3% | 0.412 | 0.043 |
| civil_comments | 156 | 76.9% (MAE 0.48 levels) | 0.543 | 0.086 |
| clinc150 | 300 | 98.3% | 0.058 | 0.019 |
| cola | 200 | 79.0% | 0.419 | 0.086 |
| commonsense_qa | 300 | 65.0% | 0.880 | 0.042 |
| dbpedia14 | 300 | 70.0% | 1.027 | 0.163 |
| go_emotions | 256 | 84.4% | 0.463 | 0.079 |
| hellaswag | 300 | 68.7% | 0.818 | 0.065 |
| massive_intent | 300 | 97.0% | 0.113 | 0.037 |
| massive_scenario | 300 | 96.3% | 0.141 | 0.040 |
| mnli | 300 | 87.3% | 0.359 | 0.061 |
| mrpc | 200 | 90.5% | 0.234 | 0.048 |
| newsgroups | 296 | 85.5% | 0.440 | 0.058 |
| openbookqa | 250 | 59.6% | 0.957 | 0.056 |
| qasc | 250 | 49.6% | 1.370 | 0.053 |
| qnli | 200 | 90.5% | 0.228 | 0.059 |
| qqp | 200 | 87.0% | 0.331 | 0.057 |
| rte | 200 | 82.0% | 0.391 | 0.038 |
| sms_spam | 300 | 100.0% | 0.009 | 0.009 |
| snli | 295 | 87.5% | 0.346 | 0.073 |
| sst2 | 300 | 95.0% | 0.177 | 0.048 |
| sst5 | 300 | 54.7% (MAE 0.55 levels) | 1.034 | 0.069 |
| stsb | 300 | 51.3% (MAE 0.60 levels) | 1.093 | 0.049 |
| toxic_conversations | 124 | 91.9% | 0.258 | 0.054 |
| wanli | 300 | 74.7% | 0.583 | 0.042 |
| winogrande | 300 | 66.0% | 0.715 | 0.110 |

Tasks in this second table were **never trained on** (zero-shot to the model):

### Held-out tasks

| task | n | accuracy | NLL | ECE |
| :-- | --: | --: | --: | --: |
| **all** | 4500 | **47.6%** | 1.268 | 0.133 |
| kind: choice | 2500 | 42.8% | 1.415 | 0.128 |
| kind: noul | 1000 | 58.9% | 1.011 | 0.266 |
| kind: score | 1000 | 48.2% | 1.159 | 0.072 |
| amazon_stars | 1000 | 48.2% (MAE 0.71 levels) | 1.159 | 0.072 |
| emotion | 1000 | 55.1% | 1.244 | 0.084 |
| enron_spam | 1000 | 58.9% | 1.011 | 0.266 |
| mmlu | 1500 | 34.5% | 1.529 | 0.166 |

## Training data

Fine-tuned on 301661 examples built from the public datasets below (options are re-sampled and verbalised in several
styles; classification tasks are also posed as yes/no verification questions). Licenses are as declared on the
Hugging Face hub; **review them before redistributing or commercialising the weights**, since several datasets
carry share-alike or unspecified terms.

| dataset | train examples | license (as declared) |
| :-- | --: | :-- |
| amazon_polarity | 8000 | apache-2.0 |
| amazon_stars (held out, evaluation only) | 0 | unspecified (mteb/amazon_reviews_multi) |
| arc | 1119 | cc-by-sa-4.0 |
| arc_easy | 2251 | cc-by-sa-4.0 |
| banking77 | 9993 | cc-by-4.0 |
| bitext_support | 8990 | cdla-sharing-1.0 |
| boolq | 9000 | cc-by-sa-3.0 |
| civil_comments | 9391 | cc0-1.0 |
| clinc150 | 15000 | cc-by-3.0 |
| cola | 6000 | other (GLUE) |
| commonsense_qa | 9000 | mit |
| dbpedia14 | 8000 | cc-by-sa-3.0 |
| emotion (held out, evaluation only) | 0 | other (dair-ai) |
| enron_spam (held out, evaluation only) | 0 | unspecified (SetFit/enron_spam) |
| go_emotions | 7520 | apache-2.0 |
| hellaswag | 22000 | mit |
| massive_intent | 11499 | apache-2.0 |
| massive_scenario | 11500 | apache-2.0 |
| mmlu (held out, evaluation only) | 0 | mit |
| mnli | 30000 | cc-by-3.0/cc-by-sa-3.0/mit/other |
| mrpc | 3668 | other (GLUE) |
| newsgroups | 10707 | unspecified (SetFit/20_newsgroups) |
| openbookqa | 4957 | unknown |
| qasc | 8000 | cc-by-4.0 |
| qnli | 8000 | other (GLUE) |
| qqp | 8000 | other (GLUE) |
| rte | 2490 | other (GLUE) |
| sms_spam | 4433 | unknown (UCI) |
| snli | 15976 | cc-by-sa-4.0 |
| sst2 | 8000 | other (GLUE) |
| sst5 | 8500 | unspecified (SetFit/sst5) |
| stsb | 5748 | other (GLUE) |
| toxic_conversations | 6138 | cc-by-4.0 |
| wanli | 16000 | cc-by-4.0 |
| winogrande | 22000 | cc-by-4.0 |

## Limitations

- English only. Inputs are limited to 8192 tokens; option sets are best kept under ~10 (use a two-stage taxonomy beyond that).
- Trained on public classification, inference and multiple-choice datasets, so it is strongest on those styles of
  question. Explicit, descriptive option texts work better than bare labels. Domain-specific jargon or rubrics with
  no descriptive criteria fall back to general language priors.
- Probabilities are calibrated on held-out data from the training tasks; on very different distributions they will
  be over-confident. Use a confidence gate and route uncertain cases to review.
- Not for decisions that affect people's rights, safety or livelihood without human oversight.

## License

Weights: Apache-2.0 (ModernBERT-base is Apache-2.0), subject to the training-data terms above.