frc-contract-drift / README.md
FinanceRateCalc's picture
model card: out-of-template test
23e7da1 verified
|
Raw History Blame Contribute Delete
2.23 kB
---
license: cc-by-4.0
language: en
tags: [text-classification, multi-label, citation, provenance, mortgage, hmda, fha]
base_model: distilroberta-base
---
# frc-contract-drift
Sentence → the reason codes FinanceRateCalc's deterministic gate assigns when a published figure is restated in a way the record cannot support. Labels: CAUSALITY_NOT_ESTABLISHED, INDIVIDUAL_PREDICTION_PROHIBITED, LEGAL_CONCLUSION_NOT_SUPPORTED, LENDER_RECOMMENDATION, PROGRAM_MISMATCH, QUALIFIER_DELETED, SAME_APPLICANT_FRAMING, SCOPE_TOO_BROAD.
Trained on 8896 rule-generated sentences from 386 claim contracts (canonical sentence and faithful paraphrases → no code; rule-written distortions → the code), plus the citation-decision fixture candidates with the codes the gate gave them. No model wrote any training sentence. Held out by claim id: exact-match 0.999, per-label precision/recall in `eval.json`.
This model does not decide anything on financeratecalc.com — the gate does. It is a second reader for paraphrases the regexes miss, and a reproducible artefact of the record. It knows nothing about figures it was not shown; check the number with the receipt, not with this model.
Data: https://financeratecalc.com/eval/drift-training-set.jsonl · Spec: https://financeratecalc.com/rl-env/CITATION-DECISION-SPEC.md · Source: https://github.com/financeratecalc/financeratecalc.github.io
<!-- ood:start -->
## Out-of-template test (the number to quote)
The in-template figure (0.999) is measured on sentences from the same rule generator that wrote the training set. On 144 violating and 36 clean sentences written by hand outside those templates:
| | |
|---|---|
| violating sentence flagged with any code | 91% |
| flagged with the right code | 86% |
| clean sentence wrongly flagged | 11% |
On the held-out half (40 violating sentences): this model caught 33, the site's hand-written rule gate caught 26, the two together 37.
The test sentences were written by the publisher's assistant, not by independent annotators; small n. Weakest: same-applicant framing and whole-market conclusions. Data: https://financeratecalc.com/eval/drift-ood-set.jsonl, result: https://financeratecalc.com/eval/drift-model-ood.json
<!-- ood:end -->