model card: out-of-template test
Browse files
README.md
CHANGED
|
@@ -13,3 +13,19 @@ Trained on 4850 rule-generated sentences from 386 claim contracts (canonical sen
|
|
| 13 |
This model does not decide anything on financeratecalc.com — the gate does. It is a second reader for paraphrases the regexes miss, and a reproducible artefact of the record. It knows nothing about figures it was not shown; check the number with the receipt, not with this model.
|
| 14 |
|
| 15 |
Data: https://financeratecalc.com/eval/drift-training-set.jsonl · Spec: https://financeratecalc.com/rl-env/CITATION-DECISION-SPEC.md · Source: https://github.com/financeratecalc/financeratecalc.github.io
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
This model does not decide anything on financeratecalc.com — the gate does. It is a second reader for paraphrases the regexes miss, and a reproducible artefact of the record. It knows nothing about figures it was not shown; check the number with the receipt, not with this model.
|
| 14 |
|
| 15 |
Data: https://financeratecalc.com/eval/drift-training-set.jsonl · Spec: https://financeratecalc.com/rl-env/CITATION-DECISION-SPEC.md · Source: https://github.com/financeratecalc/financeratecalc.github.io
|
| 16 |
+
|
| 17 |
+
<!-- ood:start -->
|
| 18 |
+
## Out-of-template test (the number to quote)
|
| 19 |
+
|
| 20 |
+
The in-template figure (0.998) is measured on sentences from the same rule generator that wrote the training set. On 104 violating and 26 clean sentences written by hand outside those templates:
|
| 21 |
+
|
| 22 |
+
| | |
|
| 23 |
+
|---|---|
|
| 24 |
+
| violating sentence flagged with any code | 68% |
|
| 25 |
+
| flagged with the right code | 58% |
|
| 26 |
+
| clean sentence wrongly flagged | 12% |
|
| 27 |
+
|
| 28 |
+
On the held-out half (40 violating sentences): this model caught 23, the site's hand-written rule gate caught 27, the two together 30.
|
| 29 |
+
|
| 30 |
+
The test sentences were written by the publisher's assistant, not by independent annotators; small n. Weakest: same-applicant framing and whole-market conclusions. Data: https://financeratecalc.com/eval/drift-ood-set.jsonl, result: https://financeratecalc.com/eval/drift-model-ood.json
|
| 31 |
+
<!-- ood:end -->
|