RealFalconsAI commited on
Commit
1d4fa51
·
verified ·
1 Parent(s): 99347a4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +409 -224
README.md CHANGED
@@ -1,243 +1,428 @@
1
  ---
2
- license: apache-2.0
3
- language: [en]
4
- library_name: transformers
5
- tags: [decision-model, system-one, calibrated-decisions, multiple-choice, zero-shot-classification, falcondec, lightdec]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  ---
7
- # LightDec_V2_Long v1.0.0 — FalconDec architecture
8
 
9
- Single-pass, typed (`choice` / `noul` / `score`), calibrated closed-set decision model. Successor to
10
- [`Falconsai/proof_v2`](https://huggingface.co/Falconsai/proof_v2).
11
 
12
- * Built with the FalconDec notebook V3.7 · backbone `jhu-clsp/ettin-encoder-150m` · mode `scratch` · preset `long` · lineage: jhu-clsp/ettin-encoder-150m
13
- * Parameters: 159.7M · weights: fp16 305 MB, int8 153 MB
14
- * Layout: `[CLS] question [SEP] [MASK] opt1 … [MASK] optk [SEP] state [SEP]`; set-transformer option head;
15
- temperature per (question type × option-count bucket).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
 
17
  ## Usage
18
 
 
 
19
  ```python
20
  import importlib.util
21
- from huggingface_hub import snapshot_download
22
- path = snapshot_download("<repo>") # or a local LightDec_V2_Long-v1.0.0 directory
23
- spec = importlib.util.spec_from_file_location("falcondec_modeling", f"{path}/falcondec_modeling.py")
24
- fdm = importlib.util.module_from_spec(spec); spec.loader.exec_module(fdm)
25
- model, tok = fdm.load_falcondec(path) # add "/compact-int8" for the int8 artefact
26
- out = fdm.decide(model, tok, state={"body": "I was charged twice, refund me or I cancel."}, questions={
27
- "dept": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "payments", "tech": "bugs"}},
28
- "churn": {"type": "noul", "instructions": "Does the user threaten to leave?"},
29
- "urgency": {"type": "score", "instructions": "How urgent?", "criteria": ["low", "medium", "high"]}})
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
  ```
31
 
32
- ## Test results
33
-
34
- Overall: micro acc **0.784**, task-macro acc **0.789**, ECE **0.029**,
35
- NLL 0.563, Brier 0.295, AURC 0.068.
36
-
37
- | task | domain | heldout | n | chance | acc | proof_v2 (card) | ece |
38
- |---|---|---|---|---|---|---|---|
39
- | counsel/critique_quality | agentic | False | 201 | 0.333 | 0.627 | | 0.271 |
40
- | agenttrek/finish_now | agentic | False | 151 | 0.500 | 0.788 | | 0.075 |
41
- | counsel/step_has_error | agentic | False | 201 | 0.500 | 0.831 | | 0.148 |
42
- | agenttrek/next_action_type | agentic | False | 487 | 0.191 | 0.864 | | 0.092 |
43
- | hotpotqa/retrieve | agentic | False | 497 | 0.167 | 0.893 | | 0.043 |
44
- | hotpotqa/comparison_yes_no | agentic | False | 26 | 0.500 | 0.923 | | 0.082 |
45
- | sst5/score | classification | True | 500 | 0.200 | 0.406 | | 0.069 |
46
- | emotion/6way | classification | True | 500 | 0.167 | 0.472 | | 0.211 |
47
- | yelp/score | classification | False | 500 | 0.200 | 0.668 | | 0.091 |
48
- | ag_news/topic | classification | False | 500 | 0.250 | 0.900 | | 0.052 |
49
- | devign/vulnerability | code | False | 500 | 0.500 | 0.632 | 0.542 | 0.070 |
50
- | humaneval/completion | code | True | 119 | 0.394 | 0.849 | 0.575 | 0.140 |
51
- | mbpp/bugspot | code | False | 256 | 0.406 | 0.855 | 0.475 | 0.037 |
52
- | codexglue/func_name | code | False | 467 | 0.255 | 0.955 | 0.901 | 0.039 |
53
- | bigclonebench/clone | code | False | 500 | 0.500 | 0.962 | 0.383 | 0.031 |
54
- | codexglue/doc_to_code | code | False | 504 | 0.279 | 0.976 | 0.961 | 0.013 |
55
- | mbpp/solution | code | False | 500 | 0.250 | 0.978 | 0.992 | 0.020 |
56
- | codexglue/code_to_doc | code | False | 504 | 0.262 | 0.990 | 0.969 | 0.010 |
57
- | codexglue/lang_id | code | False | 504 | 0.235 | 1.000 | 0.997 | 0.001 |
58
- | prompt_injections/detect | guardrails | True | 116 | 0.500 | 0.595 | | 0.333 |
59
- | agentharm/refuse | guardrails | True | 416 | 0.500 | 0.700 | | 0.089 |
60
- | civil_comments/toxic | guardrails | False | 500 | 0.500 | 0.926 | | 0.051 |
61
- | jailbreak/detect | guardrails | False | 262 | 0.500 | 0.973 | | 0.031 |
62
- | banking77/intent_77 | intents | True | 300 | 0.013 | 0.580 | | 0.241 |
63
- | banking77/intent | intents | True | 500 | 0.321 | 0.922 | 0.883 | 0.037 |
64
- | massive_en/intent | intents | False | 500 | 0.130 | 0.930 | | 0.028 |
65
- | clinc150/intent | intents | False | 500 | 0.124 | 0.970 | 0.850 | 0.015 |
66
- | long/contract_clause | long_context | False | 150 | 0.200 | 1.000 | | 0.000 |
67
- | long/email_thread | long_context | False | 150 | 0.250 | 1.000 | | 0.000 |
68
- | long/service_log | long_context | False | 150 | 0.200 | 1.000 | | 0.001 |
69
- | policy/table_count_transfer | policy | False | 500 | 0.121 | 0.324 | | 0.541 |
70
- | policy/invoice_total_transfer | policy | False | 500 | 0.500 | 0.474 | | 0.082 |
71
- | policy/sla_urgency_transfer | policy | False | 500 | 0.250 | 0.606 | | 0.131 |
72
- | policy/invoice_overdue_transfer | policy | False | 500 | 0.500 | 0.824 | | 0.020 |
73
- | policy/count_threshold_transfer | policy | False | 500 | 0.105 | 0.880 | | 0.048 |
74
- | policy/table_compare_transfer | policy | False | 500 | 0.500 | 0.928 | | 0.049 |
75
- | policy/table_extreme_transfer | policy | False | 500 | 0.122 | 0.932 | | 0.053 |
76
- | policy/free_shipping_transfer | policy | False | 500 | 0.500 | 0.938 | | 0.041 |
77
- | policy/refund_approval_transfer | policy | False | 500 | 0.333 | 0.962 | | 0.019 |
78
- | policy/access_control_transfer | policy | False | 500 | 0.333 | 1.000 | | 0.002 |
79
- | policy/return_window_transfer | policy | False | 500 | 0.333 | 1.000 | | 0.026 |
80
- | aqua_rat/math | reasoning | False | 247 | 0.200 | 0.340 | | 0.080 |
81
- | mmlu/mcq | reasoning | True | 500 | 0.250 | 0.390 | | 0.155 |
82
- | anli/nli | reasoning | False | 498 | 0.333 | 0.486 | | 0.146 |
83
- | arc_challenge/mcq | reasoning | True | 500 | 0.250 | 0.490 | 0.308 | 0.168 |
84
- | openbookqa/mcq | reasoning | False | 500 | 0.250 | 0.572 | 0.292 | 0.218 |
85
- | hellaswag/continuation | reasoning | False | 500 | 0.250 | 0.576 | | 0.075 |
86
- | arc_easy/mcq | reasoning | True | 500 | 0.250 | 0.612 | 0.425 | 0.122 |
87
- | commonsense_qa/mcq | reasoning | False | 493 | 0.200 | 0.643 | 0.442 | 0.143 |
88
- | winogrande/blank | reasoning | False | 500 | 0.500 | 0.662 | | 0.116 |
89
- | gsm8k/math | reasoning | False | 500 | 0.250 | 0.700 | 0.275 | 0.085 |
90
- | boolq/yes_no | reasoning | False | 500 | 0.500 | 0.822 | 0.717 | 0.080 |
91
- | mnli/claim | reasoning | False | 500 | 0.333 | 0.860 | 0.492 | 0.081 |
92
- | snli/nli | reasoning | False | 988 | 0.333 | 0.895 | | 0.108 |
93
- | sciq/mcq | reasoning | False | 498 | 0.250 | 0.954 | 0.692 | 0.021 |
94
- | scitail/support | reasoning | False | 500 | 0.500 | 0.962 | | 0.030 |
95
- | qasc/mcq | reasoning | False | 500 | 0.125 | 0.986 | | 0.008 |
96
- | snli/must_be_true | reasoning | False | 500 | 0.333 | 0.986 | 0.908 | 0.043 |
97
- | snli/contradicts | reasoning | False | 500 | 0.333 | 0.990 | 0.892 | 0.019 |
98
- | triage/support_email | support | False | 505 | 0.323 | 0.945 | | 0.046 |
99
- | bitext/category | support | False | 500 | 0.161 | 1.000 | | 0.001 |
100
- | bitext/route | support | False | 500 | 0.190 | 1.000 | 0.958 | 0.000 |
101
- | tev1_test/sst5 | tev1_benchmark | True | 150 | 0.200 | 0.393 | | 0.108 |
102
- | tev1_test/routing | tev1_benchmark | True | 600 | 0.200 | 0.458 | | 0.038 |
103
- | tev1_test/policy | tev1_benchmark | True | 1200 | 0.333 | 0.529 | | 0.059 |
104
- | tev1_test/banking77 | tev1_benchmark | True | 200 | 0.178 | 0.720 | | 0.137 |
105
- | tev1_test/mnli | tev1_benchmark | True | 300 | 0.333 | 0.723 | | 0.075 |
106
- | tev1_test/boolq | tev1_benchmark | True | 200 | 0.500 | 0.835 | | 0.050 |
107
- | tev1_test/ag_news | tev1_benchmark | True | 150 | 0.250 | 0.920 | | 0.098 |
108
- | typed_decisions/agent_trace_observability | workflows | False | 500 | 0.300 | 0.734 | | 0.222 |
109
- | typed_decisions/customer_service | workflows | False | 500 | 0.280 | 0.764 | | 0.210 |
110
- | typed_decisions/security_incidents | workflows | False | 500 | 0.340 | 0.768 | | 0.235 |
111
- | typed_decisions/invoice_processing | workflows | False | 500 | 0.350 | 0.826 | | 0.195 |
112
-
113
-
114
- ### Head-to-head with proof_v2 on identical test decisions
115
-
116
- | task | n | LightDec | proof_v2 | Δ |
117
- |---|---|---|---|---|
118
- | ag_news/topic | 120.000 | 0.850 | 0.450 | 0.400 |
119
- | agentharm/refuse | 120.000 | 0.667 | 0.450 | 0.217 |
120
- | agenttrek/finish_now | 120.000 | 0.783 | 0.258 | 0.525 |
121
- | agenttrek/next_action_type | 120.000 | 0.875 | 0.075 | 0.800 |
122
- | anli/nli | 120.000 | 0.592 | 0.325 | 0.267 |
123
- | aqua_rat/math | 120.000 | 0.350 | 0.242 | 0.108 |
124
- | arc_challenge/mcq | 120.000 | 0.533 | 0.308 | 0.225 |
125
- | arc_easy/mcq | 120.000 | 0.650 | 0.458 | 0.192 |
126
- | banking77/intent | 120.000 | 0.925 | 0.867 | 0.058 |
127
- | bigclonebench/clone | 120.000 | 0.967 | 0.242 | 0.725 |
128
- | bitext/category | 120.000 | 1.000 | 0.858 | 0.142 |
129
- | bitext/route | 120.000 | 1.000 | 0.925 | 0.075 |
130
- | boolq/yes_no | 120.000 | 0.825 | 0.667 | 0.158 |
131
- | civil_comments/toxic | 120.000 | 0.933 | 0.358 | 0.575 |
132
- | clinc150/intent | 120.000 | 0.975 | 0.642 | 0.333 |
133
- | codexglue/code_to_doc | 120.000 | 1.000 | 0.933 | 0.067 |
134
- | codexglue/doc_to_code | 120.000 | 0.983 | 0.975 | 0.008 |
135
- | codexglue/func_name | 120.000 | 0.992 | 0.917 | 0.075 |
136
- | codexglue/lang_id | 120.000 | 1.000 | 1.000 | 0.000 |
137
- | commonsense_qa/mcq | 120.000 | 0.683 | 0.408 | 0.275 |
138
- | counsel/critique_quality | 120.000 | 0.608 | 0.258 | 0.350 |
139
- | counsel/step_has_error | 120.000 | 0.825 | 0.750 | 0.075 |
140
- | devign/vulnerability | 120.000 | 0.667 | 0.567 | 0.100 |
141
- | emotion/6way | 120.000 | 0.492 | 0.367 | 0.125 |
142
- | gsm8k/math | 120.000 | 0.733 | 0.125 | 0.608 |
143
- | hellaswag/continuation | 120.000 | 0.608 | 0.292 | 0.317 |
144
- | hotpotqa/comparison_yes_no | 26.000 | 0.923 | 0.462 | 0.462 |
145
- | hotpotqa/retrieve | 120.000 | 0.875 | 0.292 | 0.583 |
146
- | humaneval/completion | 119.000 | 0.849 | 0.513 | 0.336 |
147
- | jailbreak/detect | 120.000 | 0.983 | 0.567 | 0.417 |
148
- | long/contract_clause | 120.000 | 1.000 | 0.233 | 0.767 |
149
- | long/email_thread | 120.000 | 1.000 | 0.050 | 0.950 |
150
- | long/service_log | 120.000 | 1.000 | 0.242 | 0.758 |
151
- | massive_en/intent | 120.000 | 0.950 | 0.742 | 0.208 |
152
- | mbpp/bugspot | 120.000 | 0.858 | 0.575 | 0.283 |
153
- | mbpp/solution | 120.000 | 1.000 | 0.850 | 0.150 |
154
- | mmlu/mcq | 120.000 | 0.408 | 0.308 | 0.100 |
155
- | mnli/claim | 120.000 | 0.875 | 0.317 | 0.558 |
156
- | openbookqa/mcq | 120.000 | 0.650 | 0.283 | 0.367 |
157
- | policy/access_control_transfer | 120.000 | 1.000 | 0.233 | 0.767 |
158
- | policy/count_threshold_transfer | 120.000 | 0.867 | 0.108 | 0.758 |
159
- | policy/free_shipping_transfer | 120.000 | 0.925 | 0.708 | 0.217 |
160
- | policy/invoice_overdue_transfer | 120.000 | 0.808 | 0.658 | 0.150 |
161
- | policy/invoice_total_transfer | 120.000 | 0.450 | 0.525 | -0.075 |
162
- | policy/refund_approval_transfer | 120.000 | 0.958 | 0.325 | 0.633 |
163
- | policy/return_window_transfer | 120.000 | 1.000 | 0.342 | 0.658 |
164
- | policy/sla_urgency_transfer | 120.000 | 0.517 | 0.442 | 0.075 |
165
- | policy/table_compare_transfer | 120.000 | 0.917 | 0.492 | 0.425 |
166
- | policy/table_count_transfer | 120.000 | 0.317 | 0.158 | 0.158 |
167
- | policy/table_extreme_transfer | 120.000 | 0.925 | 0.142 | 0.783 |
168
- | prompt_injections/detect | 116.000 | 0.595 | 0.543 | 0.052 |
169
- | qasc/mcq | 120.000 | 0.992 | 0.858 | 0.133 |
170
- | sciq/mcq | 120.000 | 0.967 | 0.867 | 0.100 |
171
- | scitail/support | 120.000 | 0.958 | 0.425 | 0.533 |
172
- | snli/contradicts | 120.000 | 1.000 | 0.358 | 0.642 |
173
- | snli/must_be_true | 120.000 | 0.992 | 0.925 | 0.067 |
174
- | snli/nli | 120.000 | 0.908 | 0.342 | 0.567 |
175
- | sst5/score | 120.000 | 0.383 | 0.283 | 0.100 |
176
- | tev1_test/ag_news | 120.000 | 0.925 | 0.425 | 0.500 |
177
- | tev1_test/banking77 | 120.000 | 0.717 | 0.550 | 0.167 |
178
- | tev1_test/boolq | 120.000 | 0.842 | 0.558 | 0.283 |
179
- | tev1_test/mnli | 120.000 | 0.750 | 0.392 | 0.358 |
180
- | tev1_test/policy | 120.000 | 0.500 | 0.408 | 0.092 |
181
- | tev1_test/routing | 120.000 | 0.442 | 0.292 | 0.150 |
182
- | tev1_test/sst5 | 120.000 | 0.392 | 0.275 | 0.117 |
183
- | triage/support_email | 120.000 | 0.958 | 0.392 | 0.567 |
184
- | typed_decisions/agent_trace_observability | 120.000 | 0.808 | 0.333 | 0.475 |
185
- | typed_decisions/customer_service | 120.000 | 0.742 | 0.283 | 0.458 |
186
- | typed_decisions/invoice_processing | 120.000 | 0.817 | 0.475 | 0.342 |
187
- | typed_decisions/security_incidents | 120.000 | 0.758 | 0.500 | 0.258 |
188
- | winogrande/blank | 120.000 | 0.692 | 0.558 | 0.133 |
189
- | yelp/score | 120.000 | 0.625 | 0.325 | 0.300 |
190
-
191
- ### Tev1 benchmark (tev1 v2.1 test records, held out)
192
-
193
- | task | n | accuracy | source also in LightDec training |
194
- |---|---|---|---|
195
- | tev1_test/ag_news | 150 | 0.920 | yes |
196
- | tev1_test/banking77 | 200 | 0.720 | no |
197
- | tev1_test/boolq | 200 | 0.835 | yes |
198
- | tev1_test/mnli | 300 | 0.723 | yes |
199
- | tev1_test/policy | 1200 | 0.529 | no |
200
- | tev1_test/routing | 600 | 0.458 | no |
201
- | tev1_test/sst5 | 150 | 0.393 | no |
202
-
203
- All categories 0.584 on 2800 decisions; excluding sources LightDec also trains on 0.518. Tev1-4B's published results (880/1,000 main decisions,
204
- 300/300 policy transfer) are reused development benchmarks and may not be the same records.
205
-
206
- ### Support-email triage (`triage/support_email`, 101 held-out emails)
207
-
208
- Per question: topic 0.990 · refund 1.000 · breakage 0.990 · anger 0.921 · judgment 0.822. Routing-pile accuracy (person / engineering / feature log / reply): 0.941.
209
-
210
- ### Long-state tasks at max_len 2048
211
-
212
- long/contract_clause 1.000 (chance 0.20) · long/email_thread 1.000 (chance 0.25) · long/service_log 1.000 (chance 0.20)
213
-
214
- ### Latency (measured on NVIDIA RTX PRO 6000 Blackwell Server Edition)
215
-
216
- | device | weights | questions per call | p50 ms | p95 ms |
217
- |---|---|---|---|---|
218
- | cuda | fp16 | 1 | 10.1 | 10.4 |
219
- | cuda | fp16 | 5 | 11.3 | 11.4 |
220
- | cuda | fp16 | 10 | 12.6 | 12.7 |
221
- | cpu fp32 | int8 file | 1 | 48.4 | 48.8 |
222
-
223
- Batched throughput: 2589 decisions/s.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
224
 
225
  ## Training
226
 
227
- Strictly proper objective (log + 0.5·spherical + 1.0·RPS for ordinal), soft teacher targets where available,
228
- task-balanced sampling (α = 0.5), NOTA augmentation (8%), per-epoch option reshuffling,
229
- AdamW + LLRD 0.9, EMA 0.999, 10 epoch(s), RLCD stage off/reverted.
230
- Training time 291.7 min on NVIDIA RTX PRO 6000 Blackwell Server Edition.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
231
 
232
  ## Limitations
233
 
234
- * Closed world: the model ranks the options you give it. Add "None of the above" when that is a valid answer (it was trained with it).
235
- * Multi-step arithmetic (GSM8K / AQuA) is a System-2 task; route low-confidence answers to an LLM (`defer` flag, threshold 0.7).
236
- * English only; code judgement (vulnerabilities, clones) is still weak; check calibration on your own traffic.
 
 
 
 
 
 
 
237
 
238
- ## References
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
239
 
240
- Warner et al. 2024 (ModernBERT) · Weller et al. 2025 (Ettin) · Gneiting & Raftery 2007 (strictly proper scoring rules) ·
241
- Guo et al. 2017 (temperature scaling) · Geifman & El-Yaniv 2017 (selective classification) · Shao et al. 2024 (GRPO) ·
242
- Williams 1992 (REINFORCE) · Zaheer et al. 2017 / Lee et al. 2019 (Deep Sets / Set Transformer) · Hinton et al. 2015 (distillation) ·
243
- Laya (convaiinnovations) · TypeSafe Jev · Together Tev1.
 
1
  ---
2
+ license: apache-2.0 # TODO: confirm before publishing (the Ettin backbone is MIT-licensed)
3
+ language:
4
+ - en
5
+ library_name: pytorch
6
+ pipeline_tag: text-classification
7
+ base_model: jhu-clsp/ettin-encoder-150m
8
+ tags:
9
+ - falcondec
10
+ - decision-model
11
+ - multiple-choice
12
+ - zero-shot-classification
13
+ - calibration
14
+ - routing
15
+ - guardrails
16
+ - agents
17
+ - long-context
18
+ - safetensors
19
+ - falconsai
20
  ---
 
21
 
22
+ # LightDec_V2 (Long)
 
23
 
24
+ **LightDec_V2** is a small, fast, calibrated *decision model* from [Falcons.ai](https://huggingface.co/Falconsai). Given a piece of state (an email, a ticket, an agent trace, a contract, a log, a JSON record) and one or more typed questions, it picks an answer from a closed set of options in a single forward pass and returns calibrated probabilities, so your application knows when to trust the answer and when to defer to a human or a larger model.
25
+
26
+ It is the long-context successor to [Falconsai/LightDec](https://huggingface.co/Falconsai/LightDec). Every call now uses a **2,048-token window** (LightDec v1 used 512 tokens by default and only switched to 2,048 for questions with more than 24 options), so long email threads, contracts and service logs fit without truncation.
27
+
28
+ | | |
29
+ |---|---|
30
+ | **Architecture** | FalconDec (encoder + permutation-equivariant option head) |
31
+ | **Backbone** | [`jhu-clsp/ettin-encoder-150m`](https://huggingface.co/jhu-clsp/ettin-encoder-150m) |
32
+ | **Parameters** | ~0.16B (fp16 weights ≈ 319 MB; int8 export available) |
33
+ | **Context** | 2,048 tokens per call |
34
+ | **Question types** | `choice`, `noul` (yes/no), `score` (ordinal levels) |
35
+ | **Options per question** | 2 to 96 in one pass; more via an automatic tournament |
36
+ | **Test accuracy** | 78.4% micro / 78.9% macro over 31,990 test items, 73 tasks |
37
+ | **Calibration** | ECE 0.029 (per-type, per-option-count temperature scaling) |
38
+ | **Latency** | ~10 ms per call on GPU (fp16), ~48 ms on CPU (int8) |
39
+ | **Version** | 1.0.0 (trained from the pretrained backbone, not fine-tuned from LightDec v1) |
40
+
41
+ ## What it does
42
+
43
+ LightDec_V2 answers the kind of small, structured questions that sit inside real systems: *Which queue does this ticket go to? Does this email ask for a refund? How angry is the customer, 0–2? Is this agent step wrong? Is this prompt a jailbreak? Does this invoice violate the policy?* It is not a generative model and never produces free text; it only chooses among the options you give it, which makes its output easy to validate, log and act on.
44
+
45
+ Three question types are supported, and several questions about the same state can be asked in one call:
46
+
47
+ - **`choice`**: pick one of N labelled options (options can carry descriptions, e.g. `{"billing": "Charges, invoices, refunds"}`).
48
+ - **`noul`**: a yes/no judgment about a statement (returns `p_true`).
49
+ - **`score`**: an ordinal scale such as `["Calm", "Annoyed", "Furious"]` (also returns an `expected_level`).
50
+
51
+ Every answer includes a `confidence` and a `defer` flag. With the default `defer_threshold` of 0.7, the model answered **68.0%** of test questions and was **90.6%** accurate on those; the remainder are flagged for review.
52
+
53
+ ## How it works
54
+
55
+ The question, the options and the state are packed into one sequence:
56
+
57
+ ```
58
+ [CLS] question [SEP] [MASK] option_1 [MASK] option_2 ... [MASK] option_k [SEP] state [SEP]
59
+ ```
60
+
61
+ The encoder reads the whole sequence once. The hidden vector at each `[MASK]` marker represents one option; it is combined with the `[CLS]` context and a question-type embedding, then passed through a 2-layer, 8-head set transformer in which options attend to each other **without positional encoding**, so the result does not depend on option order. An MLP produces one logit per option and a softmax turns them into a distribution. Finally a temperature, learned per question type and per option-count bucket (2, 3–5, 6–12, 13+ options) and stored in `falcondec_config.json`, calibrates the probabilities.
62
+
63
+ Questions with more than 96 options are resolved with a tournament: options are scored in chunks, the best of each chunk advance, and a final pass ranks the survivors.
64
 
65
  ## Usage
66
 
67
+ The model ships with its own loader, `falcondec_modeling.py`, which requires `torch`, `transformers`, `safetensors` and `huggingface_hub`.
68
+
69
  ```python
70
  import importlib.util
71
+ from huggingface_hub import hf_hub_download
72
+
73
+ path = hf_hub_download("Falconsai/LightDec_V2", "falcondec_modeling.py")
74
+ spec = importlib.util.spec_from_file_location("falcondec_modeling", path)
75
+ fd = importlib.util.module_from_spec(spec)
76
+ spec.loader.exec_module(fd)
77
+
78
+ model, tok = fd.load_falcondec("Falconsai/LightDec_V2") # GPU if available, else CPU
79
+
80
+ state = {
81
+ "email": {
82
+ "subject": "Charged twice this month",
83
+ "body": "I was billed twice for my Pro plan in March. Please refund the duplicate charge. This is the second time!",
84
+ },
85
+ "customer": {"plan": "Pro", "customer_since": "2021"},
86
+ }
87
+
88
+ questions = {
89
+ "topic": {
90
+ "type": "choice",
91
+ "instructions": "What is this support email primarily about?",
92
+ "criteria": {
93
+ "billing": "Charges, invoices, payments, refunds or subscription costs",
94
+ "technical": "Something in the product is broken, failing or erroring",
95
+ "feature_request": "Asking for a new feature or an improvement",
96
+ "account": "Login, password, profile, seats or account settings",
97
+ "none": "None of these",
98
+ },
99
+ },
100
+ "refund": {"type": "noul", "instructions": "The email asks for money to be returned or credited"},
101
+ "anger": {"type": "score", "instructions": "How angry does the customer sound?",
102
+ "criteria": ["Calm", "Annoyed", "Furious"]},
103
+ }
104
+
105
+ out = fd.decide(model, tok, state, questions)
106
+ for key, r in out["answers"].items():
107
+ print(key, r["choice"], round(r["confidence"], 3), "DEFER" if r["defer"] else "")
108
  ```
109
 
110
+ Each result contains `choice`, `choice_text`, `confidence`, the full `probs` distribution and `defer`; `noul` answers add `p_true` and `score` answers add `expected_level`. Pass `defer_threshold=` to `decide` to trade coverage for accuracy. For lower-level control, `fd.score_items(model, tok, items)` scores raw `{"state", "question", "options", "type"}` items in batches.
111
+
112
+ **CPU / small footprint.** If the repository includes the int8 export (as LightDec v1 does in `compact-int8/`), load that folder the same way; the loader dequantizes the per-channel int8 weights automatically.
113
+
114
+ ## Evaluation
115
+
116
+ All numbers come from `falcondec_report.json`. The test split has 31,990 items across 73 tasks in 11 domains. Tasks marked *held-out* are flagged as such in the training report.
117
+
118
+ ### Overall
119
+
120
+ | Metric | Value |
121
+ |---|---:|
122
+ | Micro accuracy | 78.4% |
123
+ | Macro accuracy (mean over tasks) | 78.9% |
124
+ | Negative log-likelihood | 0.563 |
125
+ | Brier score | 0.295 |
126
+ | Expected calibration error | 0.029 |
127
+ | AURC (area under risk–coverage) | 0.068 |
128
+ | Mean absolute error on `score` questions (levels) | 0.51 |
129
+ | Coverage at defer threshold 0.7 | 68.0% |
130
+ | Accuracy on covered questions | 90.6% |
131
+
132
+ ### By domain
133
+
134
+ | Domain | Tasks | Test items | Accuracy |
135
+ |---|---:|---:|---:|
136
+ | long_context | 3 | 450 | 100.0% |
137
+ | support | 3 | 1,505 | 98.2% |
138
+ | code | 9 | 3,854 | 91.1% |
139
+ | intents | 4 | 1,800 | 85.0% |
140
+ | agentic | 6 | 1,563 | 82.1% |
141
+ | policy | 11 | 5,500 | 80.6% |
142
+ | guardrails | 4 | 1,294 | 79.8% |
143
+ | workflows | 4 | 2,000 | 77.3% |
144
+ | reasoning | 18 | 9,224 | 71.8% |
145
+ | tev1_benchmark | 7 | 2,800 | 65.4% |
146
+ | classification | 4 | 2,000 | 61.2% |
147
+
148
+ ### Head-to-head against the proof_v2 baseline
149
+
150
+ On 72 shared tasks (up to 120 items each), LightDec_V2 averaged **79.6%** against **46.7%** for the `proof_v2` baseline, winning on 70 tasks, tying on 1 and losing on 1 (`policy/invoice_total_transfer`).
151
+
152
+ <details>
153
+ <summary>Per-task head-to-head</summary>
154
+
155
+ | Task | n | LightDec_V2 | proof_v2 | Δ |
156
+ |---|---:|---:|---:|---:|
157
+ | `long/email_thread` | 120 | 100.0% | 5.0% | +95.0 |
158
+ | `agenttrek/next_action_type` | 120 | 87.5% | 7.5% | +80.0 |
159
+ | `policy/table_extreme_transfer` | 120 | 92.5% | 14.2% | +78.3 |
160
+ | `long/contract_clause` | 120 | 100.0% | 23.3% | +76.7 |
161
+ | `policy/access_control_transfer` | 120 | 100.0% | 23.3% | +76.7 |
162
+ | `long/service_log` | 120 | 100.0% | 24.2% | +75.8 |
163
+ | `policy/count_threshold_transfer` | 120 | 86.7% | 10.8% | +75.8 |
164
+ | `bigclonebench/clone` | 120 | 96.7% | 24.2% | +72.5 |
165
+ | `policy/return_window_transfer` | 120 | 100.0% | 34.2% | +65.8 |
166
+ | `snli/contradicts` | 120 | 100.0% | 35.8% | +64.2 |
167
+ | `policy/refund_approval_transfer` | 120 | 95.8% | 32.5% | +63.3 |
168
+ | `gsm8k/math` | 120 | 73.3% | 12.5% | +60.8 |
169
+ | `hotpotqa/retrieve` | 120 | 87.5% | 29.2% | +58.3 |
170
+ | `civil_comments/toxic` | 120 | 93.3% | 35.8% | +57.5 |
171
+ | `snli/nli` | 120 | 90.8% | 34.2% | +56.7 |
172
+ | `triage/support_email` | 120 | 95.8% | 39.2% | +56.7 |
173
+ | `mnli/claim` | 120 | 87.5% | 31.7% | +55.8 |
174
+ | `scitail/support` | 120 | 95.8% | 42.5% | +53.3 |
175
+ | `agenttrek/finish_now` | 120 | 78.3% | 25.8% | +52.5 |
176
+ | `tev1_test/ag_news` | 120 | 92.5% | 42.5% | +50.0 |
177
+ | `typed_decisions/agent_trace_observability` | 120 | 80.8% | 33.3% | +47.5 |
178
+ | `hotpotqa/comparison_yes_no` | 26 | 92.3% | 46.2% | +46.2 |
179
+ | `typed_decisions/customer_service` | 120 | 74.2% | 28.3% | +45.8 |
180
+ | `policy/table_compare_transfer` | 120 | 91.7% | 49.2% | +42.5 |
181
+ | `jailbreak/detect` | 120 | 98.3% | 56.7% | +41.7 |
182
+ | `ag_news/topic` | 120 | 85.0% | 45.0% | +40.0 |
183
+ | `openbookqa/mcq` | 120 | 65.0% | 28.3% | +36.7 |
184
+ | `tev1_test/mnli` | 120 | 75.0% | 39.2% | +35.8 |
185
+ | `counsel/critique_quality` | 120 | 60.8% | 25.8% | +35.0 |
186
+ | `typed_decisions/invoice_processing` | 120 | 81.7% | 47.5% | +34.2 |
187
+ | `humaneval/completion` | 119 | 84.9% | 51.3% | +33.6 |
188
+ | `clinc150/intent` | 120 | 97.5% | 64.2% | +33.3 |
189
+ | `hellaswag/continuation` | 120 | 60.8% | 29.2% | +31.7 |
190
+ | `yelp/score` | 120 | 62.5% | 32.5% | +30.0 |
191
+ | `mbpp/bugspot` | 120 | 85.8% | 57.5% | +28.3 |
192
+ | `tev1_test/boolq` | 120 | 84.2% | 55.8% | +28.3 |
193
+ | `commonsense_qa/mcq` | 120 | 68.3% | 40.8% | +27.5 |
194
+ | `anli/nli` | 120 | 59.2% | 32.5% | +26.7 |
195
+ | `typed_decisions/security_incidents` | 120 | 75.8% | 50.0% | +25.8 |
196
+ | `arc_challenge/mcq` | 120 | 53.3% | 30.8% | +22.5 |
197
+ | `policy/free_shipping_transfer` | 120 | 92.5% | 70.8% | +21.7 |
198
+ | `agentharm/refuse` | 120 | 66.7% | 45.0% | +21.7 |
199
+ | `massive_en/intent` | 120 | 95.0% | 74.2% | +20.8 |
200
+ | `arc_easy/mcq` | 120 | 65.0% | 45.8% | +19.2 |
201
+ | `tev1_test/banking77` | 120 | 71.7% | 55.0% | +16.7 |
202
+ | `boolq/yes_no` | 120 | 82.5% | 66.7% | +15.8 |
203
+ | `policy/table_count_transfer` | 120 | 31.7% | 15.8% | +15.8 |
204
+ | `mbpp/solution` | 120 | 100.0% | 85.0% | +15.0 |
205
+ | `policy/invoice_overdue_transfer` | 120 | 80.8% | 65.8% | +15.0 |
206
+ | `tev1_test/routing` | 120 | 44.2% | 29.2% | +15.0 |
207
+ | `bitext/category` | 120 | 100.0% | 85.8% | +14.2 |
208
+ | `qasc/mcq` | 120 | 99.2% | 85.8% | +13.3 |
209
+ | `winogrande/blank` | 120 | 69.2% | 55.8% | +13.3 |
210
+ | `emotion/6way` | 120 | 49.2% | 36.7% | +12.5 |
211
+ | `tev1_test/sst5` | 120 | 39.2% | 27.5% | +11.7 |
212
+ | `aqua_rat/math` | 120 | 35.0% | 24.2% | +10.8 |
213
+ | `sst5/score` | 120 | 38.3% | 28.3% | +10.0 |
214
+ | `devign/vulnerability` | 120 | 66.7% | 56.7% | +10.0 |
215
+ | `mmlu/mcq` | 120 | 40.8% | 30.8% | +10.0 |
216
+ | `sciq/mcq` | 120 | 96.7% | 86.7% | +10.0 |
217
+ | `tev1_test/policy` | 120 | 50.0% | 40.8% | +9.2 |
218
+ | `codexglue/func_name` | 120 | 99.2% | 91.7% | +7.5 |
219
+ | `policy/sla_urgency_transfer` | 120 | 51.7% | 44.2% | +7.5 |
220
+ | `bitext/route` | 120 | 100.0% | 92.5% | +7.5 |
221
+ | `counsel/step_has_error` | 120 | 82.5% | 75.0% | +7.5 |
222
+ | `codexglue/code_to_doc` | 120 | 100.0% | 93.3% | +6.7 |
223
+ | `snli/must_be_true` | 120 | 99.2% | 92.5% | +6.7 |
224
+ | `banking77/intent` | 120 | 92.5% | 86.7% | +5.8 |
225
+ | `prompt_injections/detect` | 116 | 59.5% | 54.3% | +5.2 |
226
+ | `codexglue/doc_to_code` | 120 | 98.3% | 97.5% | +0.8 |
227
+ | `codexglue/lang_id` | 120 | 100.0% | 100.0% | +0.0 |
228
+ | `policy/invoice_total_transfer` | 120 | 45.0% | 52.5% | -7.5 |
229
+
230
+ </details>
231
+
232
+ ### Support-email triage (bundled benchmark)
233
+
234
+ `benchmarks/triage_support_email_test.jsonl` contains 101 support emails, each with five typed questions and reference answers. It doubles as a worked example of the input format.
235
+
236
+ | Question | Type | Accuracy |
237
+ |---|---|---:|
238
+ | `topic` | choice (5-way) | 99.0% |
239
+ | `refund` | noul (yes/no) | 100.0% |
240
+ | `breakage` | score (4 levels) | 99.0% |
241
+ | `anger` | score (3 levels) | 92.1% |
242
+ | `judgment` | choice (3-way) | 82.2% |
243
+
244
+ All five answers together route an email to the correct handling pile **94.1%** of the time.
245
+
246
+ ### Long context
247
+
248
+ Three long-document tasks exercise the 2,048-token window (150 items each): `long/contract_clause`, `long/email_thread` and `long/service_log`. LightDec_V2 scored 100% on all three. These are synthetic, in-distribution tasks, so treat them as a check that long inputs are read end to end rather than as a measure of general long-document reasoning.
249
+
250
+ ### TEV1 transfer benchmark
251
+
252
+ TEV1 is a separate decision benchmark. Some of its tasks draw on the same public sources as LightDec's training data; the table marks which.
253
+
254
+ | Task | n | Accuracy | Overlaps LightDec training source |
255
+ |---|---:|---:|:---:|
256
+ | `tev1_test/ag_news` | 150 | 92.0% | yes |
257
+ | `tev1_test/banking77` | 200 | 72.0% | no |
258
+ | `tev1_test/boolq` | 200 | 83.5% | yes |
259
+ | `tev1_test/mnli` | 300 | 72.3% | yes |
260
+ | `tev1_test/policy` | 1200 | 52.9% | no |
261
+ | `tev1_test/routing` | 600 | 45.8% | no |
262
+ | `tev1_test/sst5` | 150 | 39.3% | no |
263
+
264
+ Overall TEV1 accuracy is **58.4%** (2,800 items), and **51.8%** on the tasks with no source overlap. For reference, the report lists published results for the 4B-parameter TEV1 model of 88% on its main decisions set and 100% on its policy transfer set; those are different splits and a model roughly 25× larger, so the figures are context rather than a like-for-like comparison.
265
+
266
+ <details>
267
+ <summary>Full per-task test results (73 tasks)</summary>
268
+
269
+ | Domain | Task | n | Accuracy | Chance | ECE | Held-out |
270
+ |---|---|---:|---:|---:|---:|:---:|
271
+ | agentic | `agenttrek/finish_now` | 151 | 78.8% | 50.0% | 0.075 | |
272
+ | agentic | `agenttrek/next_action_type` | 487 | 86.4% | 19.1% | 0.092 | |
273
+ | agentic | `counsel/critique_quality` | 201 | 62.7% | 33.3% | 0.271 | |
274
+ | agentic | `counsel/step_has_error` | 201 | 83.1% | 50.0% | 0.148 | |
275
+ | agentic | `hotpotqa/comparison_yes_no` | 26 | 92.3% | 50.0% | 0.082 | |
276
+ | agentic | `hotpotqa/retrieve` | 497 | 89.3% | 16.7% | 0.043 | |
277
+ | classification | `ag_news/topic` | 500 | 90.0% | 25.0% | 0.052 | |
278
+ | classification | `emotion/6way` | 500 | 47.2% | 16.7% | 0.211 | ✓ |
279
+ | classification | `sst5/score` | 500 | 40.6% | 20.0% | 0.069 | ✓ |
280
+ | classification | `yelp/score` | 500 | 66.8% | 20.0% | 0.091 | |
281
+ | code | `bigclonebench/clone` | 500 | 96.2% | 50.0% | 0.031 | |
282
+ | code | `codexglue/code_to_doc` | 504 | 99.0% | 26.2% | 0.010 | |
283
+ | code | `codexglue/doc_to_code` | 504 | 97.6% | 27.9% | 0.013 | |
284
+ | code | `codexglue/func_name` | 467 | 95.5% | 25.5% | 0.039 | |
285
+ | code | `codexglue/lang_id` | 504 | 100.0% | 23.5% | 0.001 | |
286
+ | code | `devign/vulnerability` | 500 | 63.2% | 50.0% | 0.070 | |
287
+ | code | `humaneval/completion` | 119 | 84.9% | 39.4% | 0.140 | ✓ |
288
+ | code | `mbpp/bugspot` | 256 | 85.5% | 40.6% | 0.037 | |
289
+ | code | `mbpp/solution` | 500 | 97.8% | 25.0% | 0.020 | |
290
+ | guardrails | `agentharm/refuse` | 416 | 70.0% | 50.0% | 0.089 | ✓ |
291
+ | guardrails | `civil_comments/toxic` | 500 | 92.6% | 50.0% | 0.051 | |
292
+ | guardrails | `jailbreak/detect` | 262 | 97.3% | 50.0% | 0.031 | |
293
+ | guardrails | `prompt_injections/detect` | 116 | 59.5% | 50.0% | 0.333 | ✓ |
294
+ | intents | `banking77/intent` | 500 | 92.2% | 32.1% | 0.037 | ✓ |
295
+ | intents | `banking77/intent_77` | 300 | 58.0% | 1.3% | 0.241 | ✓ |
296
+ | intents | `clinc150/intent` | 500 | 97.0% | 12.4% | 0.015 | |
297
+ | intents | `massive_en/intent` | 500 | 93.0% | 13.0% | 0.028 | |
298
+ | long_context | `long/contract_clause` | 150 | 100.0% | 20.0% | 0.000 | |
299
+ | long_context | `long/email_thread` | 150 | 100.0% | 25.0% | 0.000 | |
300
+ | long_context | `long/service_log` | 150 | 100.0% | 20.0% | 0.001 | |
301
+ | policy | `policy/access_control_transfer` | 500 | 100.0% | 33.3% | 0.002 | |
302
+ | policy | `policy/count_threshold_transfer` | 500 | 88.0% | 10.5% | 0.048 | |
303
+ | policy | `policy/free_shipping_transfer` | 500 | 93.8% | 50.0% | 0.041 | |
304
+ | policy | `policy/invoice_overdue_transfer` | 500 | 82.4% | 50.0% | 0.020 | |
305
+ | policy | `policy/invoice_total_transfer` | 500 | 47.4% | 50.0% | 0.082 | |
306
+ | policy | `policy/refund_approval_transfer` | 500 | 96.2% | 33.3% | 0.019 | |
307
+ | policy | `policy/return_window_transfer` | 500 | 100.0% | 33.3% | 0.026 | |
308
+ | policy | `policy/sla_urgency_transfer` | 500 | 60.6% | 25.0% | 0.131 | |
309
+ | policy | `policy/table_compare_transfer` | 500 | 92.8% | 50.0% | 0.049 | |
310
+ | policy | `policy/table_count_transfer` | 500 | 32.4% | 12.1% | 0.541 | |
311
+ | policy | `policy/table_extreme_transfer` | 500 | 93.2% | 12.2% | 0.053 | |
312
+ | reasoning | `anli/nli` | 498 | 48.6% | 33.3% | 0.146 | |
313
+ | reasoning | `aqua_rat/math` | 247 | 34.0% | 20.0% | 0.080 | |
314
+ | reasoning | `arc_challenge/mcq` | 500 | 49.0% | 25.0% | 0.168 | ✓ |
315
+ | reasoning | `arc_easy/mcq` | 500 | 61.2% | 25.0% | 0.122 | ✓ |
316
+ | reasoning | `boolq/yes_no` | 500 | 82.2% | 50.0% | 0.080 | |
317
+ | reasoning | `commonsense_qa/mcq` | 493 | 64.3% | 20.0% | 0.143 | |
318
+ | reasoning | `gsm8k/math` | 500 | 70.0% | 25.0% | 0.085 | |
319
+ | reasoning | `hellaswag/continuation` | 500 | 57.6% | 25.0% | 0.075 | |
320
+ | reasoning | `mmlu/mcq` | 500 | 39.0% | 25.0% | 0.155 | ✓ |
321
+ | reasoning | `mnli/claim` | 500 | 86.0% | 33.3% | 0.081 | |
322
+ | reasoning | `openbookqa/mcq` | 500 | 57.2% | 25.0% | 0.218 | |
323
+ | reasoning | `qasc/mcq` | 500 | 98.6% | 12.5% | 0.008 | |
324
+ | reasoning | `sciq/mcq` | 498 | 95.4% | 25.0% | 0.021 | |
325
+ | reasoning | `scitail/support` | 500 | 96.2% | 50.0% | 0.030 | |
326
+ | reasoning | `snli/contradicts` | 500 | 99.0% | 33.3% | 0.019 | |
327
+ | reasoning | `snli/must_be_true` | 500 | 98.6% | 33.3% | 0.043 | |
328
+ | reasoning | `snli/nli` | 988 | 89.5% | 33.3% | 0.108 | |
329
+ | reasoning | `winogrande/blank` | 500 | 66.2% | 50.0% | 0.116 | |
330
+ | support | `bitext/category` | 500 | 100.0% | 16.1% | 0.001 | |
331
+ | support | `bitext/route` | 500 | 100.0% | 19.0% | 0.000 | |
332
+ | support | `triage/support_email` | 505 | 94.5% | 32.3% | 0.046 | |
333
+ | tev1_benchmark | `tev1_test/ag_news` | 150 | 92.0% | 25.0% | 0.098 | ✓ |
334
+ | tev1_benchmark | `tev1_test/banking77` | 200 | 72.0% | 17.8% | 0.137 | ✓ |
335
+ | tev1_benchmark | `tev1_test/boolq` | 200 | 83.5% | 50.0% | 0.050 | ✓ |
336
+ | tev1_benchmark | `tev1_test/mnli` | 300 | 72.3% | 33.3% | 0.075 | ✓ |
337
+ | tev1_benchmark | `tev1_test/policy` | 1200 | 52.9% | 33.3% | 0.059 | ✓ |
338
+ | tev1_benchmark | `tev1_test/routing` | 600 | 45.8% | 20.0% | 0.038 | ✓ |
339
+ | tev1_benchmark | `tev1_test/sst5` | 150 | 39.3% | 20.0% | 0.108 | ✓ |
340
+ | workflows | `typed_decisions/agent_trace_observability` | 500 | 73.4% | 30.0% | 0.222 | |
341
+ | workflows | `typed_decisions/customer_service` | 500 | 76.4% | 28.0% | 0.210 | |
342
+ | workflows | `typed_decisions/invoice_processing` | 500 | 82.6% | 35.0% | 0.195 | |
343
+ | workflows | `typed_decisions/security_incidents` | 500 | 76.8% | 34.0% | 0.235 | |
344
+
345
+ </details>
346
 
347
  ## Training
348
 
349
+ | | |
350
+ |---|---|
351
+ | Initialization | Pretrained `jhu-clsp/ettin-encoder-150m` (trained from scratch on top of the backbone; no LightDec v1 weights) |
352
+ | Data | 1,023,814 training / 24,200 validation / 31,990 test examples |
353
+ | Sources | Public classification, NLI, QA, code, intent, guardrail and agent-trace datasets converted into typed decisions, plus synthetic policy-transfer, workflow, support-triage and long-context tasks and TEV1 builders. `mind2web` was excluded; no custom data was added. |
354
+ | Preset | `long` (2,048-token sequences, 1,500 long-context training tasks) |
355
+ | Objective | Cross-entropy with spherical-score (0.5) and ranked-probability-score (1.0) terms for ordinal questions; 8% "none of the above" augmentation; task sampling α = 0.5 |
356
+ | Optimizer | Learning rate 8e-5 (encoder) / 6e-4 (head), layer-wise decay 0.9, weight decay 0.01, 6% warmup, gradient clip 1.0, EMA 0.999 |
357
+ | Batching | Token-budget batches of 65,536 tokens (batch size 128) |
358
+ | Schedule | 10 epochs; checkpoint from **epoch 4** selected on validation macro accuracy |
359
+ | Calibration | Temperature per (question type × option-count bucket) fit on validation |
360
+ | Compute | 292 minutes on one NVIDIA RTX PRO 6000 Blackwell Server Edition |
361
+ | Software | PyTorch 2.9.0 (CUDA 13.0), Transformers 5.17.0, Python 3.12 |
362
+
363
+ | Epoch | Train loss | Train acc | Val macro acc | Val NLL |
364
+ |---:|---:|---:|---:|---:|
365
+ | 1 | 0.706 | 73.4% | 81.9% | 0.408 |
366
+ | 2 | 0.431 | 84.7% | 84.5% | 0.362 |
367
+ | 3 | 0.343 | 87.9% | 85.0% | 0.392 |
368
+ | 4 **(selected)** | 0.277 | 90.4% | 85.2% | 0.453 |
369
+ | 5 | 0.220 | 92.6% | 84.9% | 0.537 |
370
+ | 6 | 0.170 | 94.5% | 84.7% | 0.671 |
371
+ | 7 | 0.127 | 96.0% | 84.5% | 0.843 |
372
+ | 8 | 0.095 | 97.1% | 84.5% | 1.034 |
373
+ | 9 | 0.071 | 97.9% | 84.4% | 1.211 |
374
+ | 10 | 0.057 | 98.4% | 84.4% | 1.419 |
375
+
376
+ Validation accuracy peaked at epoch 4 while validation NLL kept rising after epoch 2, so the selected checkpoint is somewhat overconfident before calibration. The fitted temperatures (roughly 1.6 to 2.5) correct for this; they are part of the checkpoint and applied automatically by `decide` and `score_items`.
377
+
378
+ ## Speed
379
+
380
+ Measured with `decide` on a single state; latency grows only slightly as more questions are asked about the same state.
381
+
382
+ | Device | Weights | Questions per call | p50 | p95 |
383
+ |---|---|---:|---:|---:|
384
+ | cuda | fp16 | 1 | 10.05 ms | 10.43 ms |
385
+ | cuda | fp16 | 5 | 11.32 ms | 11.41 ms |
386
+ | cuda | fp16 | 10 | 12.57 ms | 12.69 ms |
387
+ | cpu fp32 | int8 file | 1 | 48.4 ms | 48.8 ms |
388
+
389
+ GPU throughput: about **2,589 decisions per second** with batching.
390
+
391
+ ## Intended use
392
+
393
+ LightDec_V2 is built for high-volume, low-latency decisions inside software: ticket and email routing, intent detection, triage scoring, content and prompt-safety screening, policy and rule checks over structured records, agent step verification and "should the agent stop now" checks, and pre-filtering before a more expensive LLM call. The calibrated `confidence` and `defer` flag are designed for human-in-the-loop and cascade setups in which uncertain cases escalate.
394
 
395
  ## Limitations
396
 
397
+ - **Closed-set only.** The model can only choose among the options you supply. If the right answer is not listed, it will still pick something; include a `none` option when that can happen.
398
+ - **Weak at multi-step reasoning and arithmetic.** Knowledge- and math-heavy tasks are well below the rest: MMLU 39.0%, AQuA-RAT 34.0%, ANLI 48.6%, ARC-Challenge 49.0%. Some policy tasks that require computing over a table are also weak: `policy/table_count_transfer` scores 32.4% with poor calibration (ECE 0.54), and `policy/invoice_total_transfer` (47.4%) is below chance. Use a larger model for anything that needs counting or totalling.
399
+ - **Fine-grained and subjective labels.** Accuracy drops on 77-way banking intents (58.0%), six-way emotion (47.2%) and five-level sentiment (SST-5 40.6%, Yelp 66.8%).
400
+ - **Transfer to unfamiliar rule sets is limited.** On TEV1 tasks without source overlap, accuracy is 51.8%, and routing and policy transfer (45.8% and 52.9%) are the weakest; validate on your own policies before relying on it.
401
+ - **Guardrail coverage is partial.** Jailbreak detection is strong (97.3%), but prompt-injection detection (59.5%) and harmful-request refusal (70.0%) are not reliable enough to be a sole safety layer.
402
+ - **Calibration is aggregate.** Overall ECE is low, but a few tasks (for example `counsel/critique_quality`, `prompt_injections/detect`, the table-count policy task) are noticeably miscalibrated; check calibration on your own distribution before tuning `defer_threshold`.
403
+ - **English only**, and questions are truncated to 96 tokens and each option to 24 tokens by default (`option_tokens` can raise the per-option budget).
404
+ - **Long-context scores are synthetic.** The 100% long-context results are in-distribution checks, not evidence of general long-document reasoning.
405
+
406
+ ## Files
407
 
408
+ | File | Description |
409
+ |---|---|
410
+ | `model.safetensors` | fp16 weights |
411
+ | `falcondec_config.json` | Head configuration, special-token ids, defer threshold and calibration temperatures |
412
+ | `falcondec_modeling.py` | Model definition, loader (`load_falcondec`) and inference API (`decide`, `score_items`) |
413
+ | `falcondec_report.json` | Full training and evaluation report |
414
+ | `encoder/`, `tokenizer/` | Ettin encoder config and tokenizer |
415
+ | `benchmarks/triage_support_email_test.jsonl` | 101-email support-triage benchmark with reference answers |
416
+
417
+ ## Citation
418
+
419
+ ```bibtex
420
+ @misc{falconsai_lightdec_v2_2026,
421
+ title = {LightDec_V2: A Long-Context, Calibrated Decision Model},
422
+ author = {Falcons.ai},
423
+ year = {2026},
424
+ howpublished = {\url{https://huggingface.co/Falconsai/LightDec_V2}}
425
+ }
426
+ ```
427
 
428
+ Built on the Ettin encoder from JHU CLSP ([`jhu-clsp/ettin-encoder-150m`](https://huggingface.co/jhu-clsp/ettin-encoder-150m)).