| --- |
| license: apache-2.0 |
| language: en |
| base_model: answerdotai/ModernBERT-base |
| library_name: decide |
| pipeline_tag: zero-shot-classification |
| tags: |
| - modernbert |
| - decision-model |
| - zero-shot-classification |
| - calibrated |
| --- |
| |
| # decide-0.2.0 |
|
|
| A small, fast, **non-autoregressive decision model**. Given some state, an instruction and a set of candidate |
| options, it scores every option in a single forward pass and returns calibrated probabilities. It answers |
| three kinds of question with the same weights: |
|
|
| - **choice**: a distribution over K mutually exclusive options |
| - **noul**: the probability that a condition holds (a yes/no question) |
| - **score**: an expected level on an ordered scale |
|
|
| It is fine-tuned from [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) (149M parameters). |
| The packed input is `<instructions> <state> [SEP] [MASK] option0 [MASK] option1 ...`; a small MLP head scores the |
| encoder state at each `[MASK]`. Each option attends only to the shared prefix and to itself and its position ids restart |
| after the prefix, so an option's score depends on *(premise, that option)* alone: **permuting the options permutes the |
| scores and changes nothing else** (measured max logit change under permutation: 8.9e-06). |
|
|
| The model, tokenizer, training loop, evaluation and this card were all produced with the pure-Go toolchain in |
| [jmwri/decide](https://github.com/jmwri/decide); no Python or PyTorch was used to train it. |
|
|
| ## Usage |
|
|
| ```go |
| import "github.com/jmwri/decide" |
| |
| decide.SetDefault(decide.NewLocal("/path/to/this/model")) // or: decide pull --repo jmwri/decide |
| ans, _ := decide.Decide(ctx, "Replication lag exceeded 45 seconds.", decide.Options{ |
| {Key: "infrastructure", Description: "Database, hardware, network or server failures"}, |
| {Key: "billing", Description: "Invoices, payments, refunds"}, |
| }, "Classify the root cause domain.") |
| ``` |
|
|
| ## Evaluation |
|
|
| Accuracy is top-option accuracy. ECE is the expected calibration error (15 bins) of the top-option confidence, |
| after the per-kind temperature scaling shipped in `decide.json`. |
|
|
| ### Held-out validation splits of the training tasks |
|
|
| | task | n | accuracy | NLL | ECE | |
| | :-- | --: | --: | --: | --: | |
| | **all** | 8127 | **80.2%** | 0.505 | 0.007 | |
| | kind: choice | 4745 | 78.7% | 0.569 | 0.011 | |
| | kind: noul | 2703 | 89.5% | 0.259 | 0.009 | |
| | kind: score | 679 | 53.5% | 1.041 | 0.034 | |
| | amazon_polarity | 300 | 94.0% | 0.178 | 0.035 | |
| | arc | 200 | 49.0% | 1.272 | 0.124 | |
| | arc_easy | 200 | 66.0% | 0.857 | 0.079 | |
| | banking77 | 300 | 96.7% | 0.094 | 0.032 | |
| | bitext_support | 300 | 100.0% | 0.011 | 0.011 | |
| | boolq | 300 | 83.3% | 0.412 | 0.043 | |
| | civil_comments | 156 | 76.9% (MAE 0.48 levels) | 0.543 | 0.086 | |
| | clinc150 | 300 | 98.3% | 0.058 | 0.019 | |
| | cola | 200 | 79.0% | 0.419 | 0.086 | |
| | commonsense_qa | 300 | 65.0% | 0.880 | 0.042 | |
| | dbpedia14 | 300 | 70.0% | 1.027 | 0.163 | |
| | go_emotions | 256 | 84.4% | 0.463 | 0.079 | |
| | hellaswag | 300 | 68.7% | 0.818 | 0.065 | |
| | massive_intent | 300 | 97.0% | 0.113 | 0.037 | |
| | massive_scenario | 300 | 96.3% | 0.141 | 0.040 | |
| | mnli | 300 | 87.3% | 0.359 | 0.061 | |
| | mrpc | 200 | 90.5% | 0.234 | 0.048 | |
| | newsgroups | 296 | 85.5% | 0.440 | 0.058 | |
| | openbookqa | 250 | 59.6% | 0.957 | 0.056 | |
| | qasc | 250 | 49.6% | 1.370 | 0.053 | |
| | qnli | 200 | 90.5% | 0.228 | 0.059 | |
| | qqp | 200 | 87.0% | 0.331 | 0.057 | |
| | rte | 200 | 82.0% | 0.391 | 0.038 | |
| | sms_spam | 300 | 100.0% | 0.009 | 0.009 | |
| | snli | 295 | 87.5% | 0.346 | 0.073 | |
| | sst2 | 300 | 95.0% | 0.177 | 0.048 | |
| | sst5 | 300 | 54.7% (MAE 0.55 levels) | 1.034 | 0.069 | |
| | stsb | 300 | 51.3% (MAE 0.60 levels) | 1.093 | 0.049 | |
| | toxic_conversations | 124 | 91.9% | 0.258 | 0.054 | |
| | wanli | 300 | 74.7% | 0.583 | 0.042 | |
| | winogrande | 300 | 66.0% | 0.715 | 0.110 | |
|
|
| Tasks in this second table were **never trained on** (zero-shot to the model): |
|
|
| ### Held-out tasks |
|
|
| | task | n | accuracy | NLL | ECE | |
| | :-- | --: | --: | --: | --: | |
| | **all** | 4500 | **47.6%** | 1.268 | 0.133 | |
| | kind: choice | 2500 | 42.8% | 1.415 | 0.128 | |
| | kind: noul | 1000 | 58.9% | 1.011 | 0.266 | |
| | kind: score | 1000 | 48.2% | 1.159 | 0.072 | |
| | amazon_stars | 1000 | 48.2% (MAE 0.71 levels) | 1.159 | 0.072 | |
| | emotion | 1000 | 55.1% | 1.244 | 0.084 | |
| | enron_spam | 1000 | 58.9% | 1.011 | 0.266 | |
| | mmlu | 1500 | 34.5% | 1.529 | 0.166 | |
|
|
| ## Training data |
|
|
| Fine-tuned on 301661 examples built from the public datasets below (options are re-sampled and verbalised in several |
| styles; classification tasks are also posed as yes/no verification questions). Licenses are as declared on the |
| Hugging Face hub; **review them before redistributing or commercialising the weights**, since several datasets |
| carry share-alike or unspecified terms. |
|
|
| | dataset | train examples | license (as declared) | |
| | :-- | --: | :-- | |
| | amazon_polarity | 8000 | apache-2.0 | |
| | amazon_stars (held out, evaluation only) | 0 | unspecified (mteb/amazon_reviews_multi) | |
| | arc | 1119 | cc-by-sa-4.0 | |
| | arc_easy | 2251 | cc-by-sa-4.0 | |
| | banking77 | 9993 | cc-by-4.0 | |
| | bitext_support | 8990 | cdla-sharing-1.0 | |
| | boolq | 9000 | cc-by-sa-3.0 | |
| | civil_comments | 9391 | cc0-1.0 | |
| | clinc150 | 15000 | cc-by-3.0 | |
| | cola | 6000 | other (GLUE) | |
| | commonsense_qa | 9000 | mit | |
| | dbpedia14 | 8000 | cc-by-sa-3.0 | |
| | emotion (held out, evaluation only) | 0 | other (dair-ai) | |
| | enron_spam (held out, evaluation only) | 0 | unspecified (SetFit/enron_spam) | |
| | go_emotions | 7520 | apache-2.0 | |
| | hellaswag | 22000 | mit | |
| | massive_intent | 11499 | apache-2.0 | |
| | massive_scenario | 11500 | apache-2.0 | |
| | mmlu (held out, evaluation only) | 0 | mit | |
| | mnli | 30000 | cc-by-3.0/cc-by-sa-3.0/mit/other | |
| | mrpc | 3668 | other (GLUE) | |
| | newsgroups | 10707 | unspecified (SetFit/20_newsgroups) | |
| | openbookqa | 4957 | unknown | |
| | qasc | 8000 | cc-by-4.0 | |
| | qnli | 8000 | other (GLUE) | |
| | qqp | 8000 | other (GLUE) | |
| | rte | 2490 | other (GLUE) | |
| | sms_spam | 4433 | unknown (UCI) | |
| | snli | 15976 | cc-by-sa-4.0 | |
| | sst2 | 8000 | other (GLUE) | |
| | sst5 | 8500 | unspecified (SetFit/sst5) | |
| | stsb | 5748 | other (GLUE) | |
| | toxic_conversations | 6138 | cc-by-4.0 | |
| | wanli | 16000 | cc-by-4.0 | |
| | winogrande | 22000 | cc-by-4.0 | |
|
|
| ## Limitations |
|
|
| - English only. Inputs are limited to 8192 tokens; option sets are best kept under ~10 (use a two-stage taxonomy beyond that). |
| - Trained on public classification, inference and multiple-choice datasets, so it is strongest on those styles of |
| question. Explicit, descriptive option texts work better than bare labels. Domain-specific jargon or rubrics with |
| no descriptive criteria fall back to general language priors. |
| - Probabilities are calibrated on held-out data from the training tasks; on very different distributions they will |
| be over-confident. Use a confidence gate and route uncertain cases to review. |
| - Not for decisions that affect people's rights, safety or livelihood without human oversight. |
|
|
| ## License |
|
|
| Weights: Apache-2.0 (ModernBERT-base is Apache-2.0), subject to the training-data terms above. |
|
|
