Upload local-decision-model

README.md

@@ -0,0 +1,83 @@
+---
+license: cc-by-nc-4.0
+language: en
+library_name: pytorch
+base_model: answerdotai/ModernBERT-base
+pipeline_tag: zero-shot-classification
+tags: [decision-model, zero-shot-classification, structured-output, guardrails, extraction, modernbert]
+---
+
+# local-decision-model
+
+A small, fast decision model: **unstructured state in, typed probabilistic decisions out.**
+One forward pass of a 150M parameter encoder answers a whole schema of typed questions about a piece of text,
+typically in 10 to 25 ms on a consumer GPU. It never generates text, so every output is guaranteed to match its declared type.
+
+Code, training recipe, experiments and full results: https://github.com/Pdbz199/local-decision-model
+
+![demo](https://github.com/Pdbz199/local-decision-model/raw/main/assets/hero.gif)
+
+## Usage
+
+These weights need the small `decisionmodel` package from the GitHub repo (they are not a standard `transformers` head).
+
+```bash
+pip install git+https://github.com/Pdbz199/local-decision-model
+```
+
+```python
+from decisionmodel import DecisionModel, Bool, Enum, MultiLabel, Int, Extract
+
+dm = DecisionModel.load("Pdbz199/local-decision-model")
+
+d = dm.decide("My order #48213 arrived with a cracked screen. Refund me today or I dispute the charge.", {
+    "urgent":   Bool("Does this need a response within the hour?"),
+    "team":     Enum("Which team should handle this ticket?", ["billing and refunds", "technical support", "sales"]),
+    "anger":    Int("How angry is the customer, from 1 (calm) to 5 (furious)?", 1, 5),
+    "order_id": Extract("What is the order number?"),
+})
+print(d.team.value, d.team.confidence, d.urgent.p(), d.order_id.value)
+```
+
+Types: `Bool`, `Enum` (2 to 255 options), `MultiLabel`, `Int`, `Float`, `Extract` (a verbatim substring of the input, or `None`).
+
+## How it works
+
+ModernBERT-base with two small heads. The schema is written into the input as
+`[CLS] [MODE] question [OPT] option 1 [OPT] option 2 ... [SEP] state [SEP]`. The hidden state at each `[OPT]` token is scored,
+so all options are judged in the same pass; a span head handles extraction. Trained with log loss on about one million
+examples recast from roughly 50 public datasets, then temperature-calibrated on tasks that were never trained on.
+
+## Results on tasks never seen in training
+
+| Held-out task | Type | Result |
+| --- | --- | --- |
+| IMDB sentiment | Bool | 94.4% accuracy, calibration error 0.016 |
+| SQuAD v2 dev | Extract | 82.8% exact match, F1 85.8 |
+| AG News topics | Enum (4) | 74.0% |
+| Twitter financial news sentiment | Enum (3) | 74.1% |
+| Banking77 intents | Enum (77) | 63.0% top 1, 80.2% top 3; 83.6% on the 48% of calls with confidence of at least 0.9 |
+| deepset prompt injections | Bool | 69.6% accuracy, AUROC 0.87 |
+| SMS spam | Bool | 64.9% with a vague question, 89.4% with a precise one (AUROC 0.94 to 0.96) |
+
+Latency (RTX 5070 Ti, bfloat16, median): single yes/no 10 ms, 8 typed fields 16 ms, 77 options 11 ms, 255 options 21 ms, 8,000 token input 351 ms.
+
+## Limitations
+
+* A small model: good at judgments a person could make at a glance, weak at implied or conditional meaning.
+* Sensitive to question wording. Test a schema on a few real examples before trusting it.
+* `MultiLabel` under-predicts when several options are true at once. Ask important tags as their own `Bool`.
+* Confidence is well calibrated on familiar kinds of task and can be overconfident on unfamiliar ones.
+* English only. Guardrail decisions can be wrong: do not use it as a sole security boundary.
+
+## Licensing
+
+The code is MIT licensed. The base model is Apache 2.0. These weights were trained on public datasets with mixed
+licenses, some of which restrict commercial use (for example lmsys/toxic-chat, CC BY-NC 4.0, and the Yelp reviews
+dataset). The weights are therefore released under cc-by-nc-4.0. If you need different terms, retrain with
+`scripts/build_data.py` after removing the sources that do not fit your use; every source is listed in that file.
+
+## Relation to Jev
+
+This project was inspired by TypeSafe AI's blog post introducing "System One Models" and their model Jev.
+It is an independent, generic implementation built only from that public post, and is not affiliated with TypeSafe AI.

encoder/config.json

@@ -0,0 +1,77 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "vocab_size": 50368
+}

encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6fe3bedd74500a32f74257a264d65b9787e0fc84fa93cd48e6f8730699d0d19
+size 596070136

encoder/tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}

heads.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb92553c26b875e6675c06eca4ee6e467374cb20c2d8cd49687f8e7398e17075
+size 4747396

meta.json

@@ -0,0 +1,168 @@
+{
+  "backbone": "answerdotai/ModernBERT-base",
+  "temperature": {
+    "one": 1.2999999999999998,
+    "any": 1.109569472067845,
+    "span": 1.0717734625362931
+  },
+  "calibration_task_accuracy": {
+    "calib/benayas/snips": 0.87,
+    "calib/crowdflower/airline-sentiment": 0.735,
+    "calib/emo/emo2019": 0.445,
+    "calib/health_fact": 0.115,
+    "calib/insincere-questions": 0.93,
+    "calib/lex_glue/ledgar": 0.3875,
+    "calib/logical-fallacy": 0.2,
+    "calib/rotten_tomatoes": 0.905,
+    "calib/scicite": 0.76,
+    "calib/silicone/dyda_e": 0.7725,
+    "calib/subjectivity": 0.6989,
+    "calib/tuetschek/atis": 0.7725
+  },
+  "temperature_k_slope": 0.0,
+  "val_accuracy": {
+    "20ng": 0.8208,
+    "anli": 0.8773,
+    "bitext": 0.984,
+    "boolq": 0.796,
+    "civil": 0.9351,
+    "clinc": 0.9509,
+    "counterfactual": 0.988,
+    "csqa": 0.74,
+    "customer_reviews": 0.944,
+    "davidson": 0.936,
+    "dbpedia": 0.996,
+    "emotion": 0.9361,
+    "enron_spam": 0.992,
+    "fever_nli": 0.896,
+    "fewnerd": 0.896,
+    "gandalf": 0.987,
+    "go_emotions": 0.9568,
+    "jailbreak": 0.9903,
+    "massive_intent": 0.872,
+    "massive_scenario": 0.94,
+    "mnli": 0.868,
+    "multi_claims": 0.9086,
+    "multi_entities": 0.8863,
+    "multi_mix_bitext": 0.987,
+    "multi_mix_clinc": 0.9679,
+    "multi_mix_emotion": 0.9167,
+    "multi_mix_yahoo": 0.8529,
+    "paws": 0.976,
+    "prompt_injection_mix": 0.992,
+    "race": 0.832,
+    "safeguard": 1.0,
+    "scales_toxicity": 0.58,
+    "scales_yelp": 0.736,
+    "snli": 0.88,
+    "spml": 0.996,
+    "squad_v2": 0.748,
+    "sst2": 0.908,
+    "stsb": 0.544,
+    "synthetic_fields": 1.0,
+    "toxic_chat": 0.96,
+    "trec": 0.944,
+    "ts/CONDAQA": 0.0,
+    "ts/CREAK": 1.0,
+    "ts/EffectiveFeedbackStudentWriting": 0.5,
+    "ts/I2D2": 1.0,
+    "ts/PARARULE-Plus": 1.0,
+    "ts/Touche23-ValueEval": 1.0,
+    "ts/VUAC": 1.0,
+    "ts/ade_corpus_v2/Ade_corpus_v2_classification": 1.0,
+    "ts/args_me": 1.0,
+    "ts/avicenna": 0.6667,
+    "ts/blog_authorship_corpus/job": 1.0,
+    "ts/boolq-natural-perturbations": 1.0,
+    "ts/circa": 1.0,
+    "ts/citation_intent": 1.0,
+    "ts/clcd-english": 1.0,
+    "ts/commonsense_qa_2.0": 1.0,
+    "ts/crowdflower/corporate-messaging": 1.0,
+    "ts/crowdflower/economic-news": 1.0,
+    "ts/crowdflower/political-media-audience": 0.6667,
+    "ts/crowdflower/political-media-bias": 1.0,
+    "ts/crowdflower/political-media-message": 0.5,
+    "ts/crowdflower/text_emotion": 0.6667,
+    "ts/crowdflower/tweet_global_warming": 1.0,
+    "ts/cycic_classification": 0.5,
+    "ts/defeasible-nli/atomic": 1.0,
+    "ts/defeasible-nli/snli": 1.0,
+    "ts/dynahate": 1.0,
+    "ts/dynasent/dynabench.dynasent.r1.all/r1": 1.0,
+    "ts/dynasent/dynabench.dynasent.r2.all/r2": 0.5,
+    "ts/ethics/deontology": 1.0,
+    "ts/ethics/justice": 0.3333,
+    "ts/ethics/virtue": 1.0,
+    "ts/ethos/binary": 0.5,
+    "ts/fever-evidence-related/mwong--fever-related": 1.0,
+    "ts/glue/cola": 0.5,
+    "ts/glue/qqp": 1.0,
+    "ts/hate_speech18": 1.0,
+    "ts/hlgd": 1.0,
+    "ts/hope_edi/english": 0.8333,
+    "ts/humicroedit/subtask-2": 0.3333,
+    "ts/implicit-hate-stg1": 1.0,
+    "ts/language-identification": 1.0,
+    "ts/lex_glue/scotus": 0.4,
+    "ts/mbib-base/cognitive-bias": 0.6667,
+    "ts/mbib-base/fake-news": 0.0,
+    "ts/mbib-base/hate-speech": 0.5,
+    "ts/mbib-base/linguistic-bias": 1.0,
+    "ts/mbib-base/political-bias": 0.6667,
+    "ts/mbib-base/racial-bias": 1.0,
+    "ts/mbib-base/text-level-bias": 0.75,
+    "ts/mc_taco": 1.0,
+    "ts/medical_questions_pairs": 1.0,
+    "ts/natural-language-satisfiability": 0.0,
+    "ts/numer_sense": 0.5,
+    "ts/open_question_type": 1.0,
+    "ts/parade": 0.25,
+    "ts/phrase_similarity": 0.6667,
+    "ts/pragmeval/emergent": 0.0,
+    "ts/pragmeval/gum": 1.0,
+    "ts/pragmeval/mrda": 0.3333,
+    "ts/pragmeval/pdtb": 0.3333,
+    "ts/pragmeval/sarcasm": 1.0,
+    "ts/pragmeval/stac": 0.5,
+    "ts/pragmeval/verifiability": 1.0,
+    "ts/probability_words_nli/reasoning_1hop": 0.75,
+    "ts/probability_words_nli/reasoning_2hop": 1.0,
+    "ts/probability_words_nli/usnli": 0.4,
+    "ts/quarel": 0.6,
+    "ts/robust_nli_li_ts": 1.0,
+    "ts/rumoureval_2019/RumourEval2019": 1.0,
+    "ts/sciie": 1.0,
+    "ts/scruples": 0.6,
+    "ts/sem_eval_2010_task_8": 1.0,
+    "ts/silicone/dyda_da": 1.0,
+    "ts/silicone/maptask": 1.0,
+    "ts/silicone/meld_e": 0.6667,
+    "ts/silicone/meld_s": 0.6667,
+    "ts/silicone/oasis": 0.6,
+    "ts/silicone/sem": 0.75,
+    "ts/spartqa-yn": 0.6667,
+    "ts/starcon": 0.5,
+    "ts/strategy-qa": 1.0,
+    "ts/super_glue/multirc": 0.4,
+    "ts/super_glue/wic": 0.5,
+    "ts/tweet_eval/emotion": 1.0,
+    "ts/tweet_eval/stance_feminist": 1.0,
+    "ts/tweet_eval/stance_hillary": 1.0,
+    "ts/tweets_hate_speech_detection": 0.6667,
+    "ts/twentyquestions": 1.0,
+    "ts/vitaminc/tals--vitaminc": 1.0,
+    "ts/wiki_qa": 1.0,
+    "ts/winowhy": 1.0,
+    "tweet_hate": 0.868,
+    "tweet_irony": 0.62,
+    "tweet_offensive": 0.852,
+    "tweet_sentiment": 0.732,
+    "tweet_sentiment2": 0.792,
+    "web_spam": 0.996,
+    "yahoo": 0.8035,
+    "yelp_sent": 0.8053,
+    "yelp_stars": 0.6204
+  },
+  "source": "https://github.com/Pdbz199/local-decision-model"
+}