FinanceRateCalc commited on
Commit
fb478f4
·
verified ·
1 Parent(s): 515bf2d

model card: out-of-template test

Browse files
Files changed (1) hide show
  1. README.md +16 -0
README.md CHANGED
@@ -13,3 +13,19 @@ Trained on 4850 rule-generated sentences from 386 claim contracts (canonical sen
13
  This model does not decide anything on financeratecalc.com — the gate does. It is a second reader for paraphrases the regexes miss, and a reproducible artefact of the record. It knows nothing about figures it was not shown; check the number with the receipt, not with this model.
14
 
15
  Data: https://financeratecalc.com/eval/drift-training-set.jsonl · Spec: https://financeratecalc.com/rl-env/CITATION-DECISION-SPEC.md · Source: https://github.com/financeratecalc/financeratecalc.github.io
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  This model does not decide anything on financeratecalc.com — the gate does. It is a second reader for paraphrases the regexes miss, and a reproducible artefact of the record. It knows nothing about figures it was not shown; check the number with the receipt, not with this model.
14
 
15
  Data: https://financeratecalc.com/eval/drift-training-set.jsonl · Spec: https://financeratecalc.com/rl-env/CITATION-DECISION-SPEC.md · Source: https://github.com/financeratecalc/financeratecalc.github.io
16
+
17
+ <!-- ood:start -->
18
+ ## Out-of-template test (the number to quote)
19
+
20
+ The in-template figure (0.998) is measured on sentences from the same rule generator that wrote the training set. On 104 violating and 26 clean sentences written by hand outside those templates:
21
+
22
+ | | |
23
+ |---|---|
24
+ | violating sentence flagged with any code | 68% |
25
+ | flagged with the right code | 58% |
26
+ | clean sentence wrongly flagged | 12% |
27
+
28
+ On the held-out half (40 violating sentences): this model caught 23, the site's hand-written rule gate caught 27, the two together 30.
29
+
30
+ The test sentences were written by the publisher's assistant, not by independent annotators; small n. Weakest: same-applicant framing and whole-market conclusions. Data: https://financeratecalc.com/eval/drift-ood-set.jsonl, result: https://financeratecalc.com/eval/drift-model-ood.json
31
+ <!-- ood:end -->