frc-contract-drift / README.md
FinanceRateCalc's picture
model card: out-of-template test
23e7da1 verified
|
Raw History Blame Contribute Delete
2.23 kB
metadata
license: cc-by-4.0
language: en
tags:
  - text-classification
  - multi-label
  - citation
  - provenance
  - mortgage
  - hmda
  - fha
base_model: distilroberta-base

frc-contract-drift

Sentence → the reason codes FinanceRateCalc's deterministic gate assigns when a published figure is restated in a way the record cannot support. Labels: CAUSALITY_NOT_ESTABLISHED, INDIVIDUAL_PREDICTION_PROHIBITED, LEGAL_CONCLUSION_NOT_SUPPORTED, LENDER_RECOMMENDATION, PROGRAM_MISMATCH, QUALIFIER_DELETED, SAME_APPLICANT_FRAMING, SCOPE_TOO_BROAD.

Trained on 8896 rule-generated sentences from 386 claim contracts (canonical sentence and faithful paraphrases → no code; rule-written distortions → the code), plus the citation-decision fixture candidates with the codes the gate gave them. No model wrote any training sentence. Held out by claim id: exact-match 0.999, per-label precision/recall in eval.json.

This model does not decide anything on financeratecalc.com — the gate does. It is a second reader for paraphrases the regexes miss, and a reproducible artefact of the record. It knows nothing about figures it was not shown; check the number with the receipt, not with this model.

Data: https://financeratecalc.com/eval/drift-training-set.jsonl · Spec: https://financeratecalc.com/rl-env/CITATION-DECISION-SPEC.md · Source: https://github.com/financeratecalc/financeratecalc.github.io

Out-of-template test (the number to quote)

The in-template figure (0.999) is measured on sentences from the same rule generator that wrote the training set. On 144 violating and 36 clean sentences written by hand outside those templates:

violating sentence flagged with any code 91%
flagged with the right code 86%
clean sentence wrongly flagged 11%

On the held-out half (40 violating sentences): this model caught 33, the site's hand-written rule gate caught 26, the two together 37.

The test sentences were written by the publisher's assistant, not by independent annotators; small n. Weakest: same-applicant framing and whole-market conclusions. Data: https://financeratecalc.com/eval/drift-ood-set.jsonl, result: https://financeratecalc.com/eval/drift-model-ood.json