FormaFin

FormaFin is a LoRA adapter for Qwen2.5-Coder-7B-Instruct that translates a loan covenant into an SMT-LIB formula. An SMT solver such as Z3 then applies the formula to a borrower's compliance certificate to decide compliance, earnings cushion and debt capacity. The model is small enough to run on a single GPU, so confidential figures stay in house.

Quick start

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "bethelhall/formafin"
tok = AutoTokenizer.from_pretrained(repo)
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, repo).eval()

SYSTEM = ("Translate the real financial covenant into SMT-LIB2. Return exactly a set-logic command and one "
          "define-fun named covenant_holds that returns Bool and is true exactly when the borrower complies.")
PROMPT = """Covenant clause:

{clause}

Write (define-fun covenant_holds (<parameters>) Bool <body>) with EXACTLY these parameters, in this order and with
these types (do not add, rename or drop any):
{params}
Amounts are in US dollars as reported in the borrower's compliance certificate; test_date is the test date as an
integer YYYYMMDD (e.g. 20121231), so a threshold that changes over time is written with test_date comparisons.
covenant_holds is true exactly when every financial covenant in the clause that applies on test_date is satisfied.
Return only the SMT-LIB2 (set-logic QF_NRA or QF_LIA/QF_LRA as needed, then the define-fun)."""

clause = "The Borrower shall maintain at all times a Total Debt to EBITDA Ratio of not more than 3.0 to 1.0."
params = "(total_debt Real) ; Consolidated Total Debt\n(ebitda Real) ; Consolidated EBITDA\n(test_date Int) ; test date as YYYYMMDD"

messages = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": PROMPT.format(clause=clause, params=params)}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
formula = tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)

The parameter list must be passed exactly: one Real per certificate line item, then test_date as Int, then one Bool per condition.

Checking a certificate with Z3 (pip install z3-solver):

import re, z3

smt = re.sub(r"```\w*", "", formula)
smt = smt[smt.index("(define-fun"):]
decls = "(declare-const total_debt Real)(declare-const ebitda Real)(declare-const test_date Int)"
facts = "(assert (= total_debt 453900000.0))(assert (= ebitda 149800000.0))(assert (= test_date 20121231))"

s = z3.Solver()
s.from_string(smt + decls + facts + "(assert (covenant_holds total_debt ebitda test_date))")
print("comply" if s.check() == z3.sat else "breach")   # 3.03x > 3.0x, so breach

Adapters

The root adapter is rl_sched_s42. The other adapters load with subfolder="<folder>" on their own base model.

Folder Base model Accuracy (%)
rl_sched_s42 (default), _s43, _s44 Qwen2.5-Coder-7B-Instruct 93.8, 94.9, 95.9
llama_rl_sched_s42, _s43, _s44 Llama-3.1-8B-Instruct 96.9, 97.8, 95.7
mistral_rl_sched_s42, _s43, _s44 Mistral-7B-Instruct-v0.3 91.1, 93.2, 93.6
Other runs kept for reproducibility
Folder Description Accuracy (%)
sft789 Supervised finetuning only 85.8
rl_v2_s42, _s43, _s44 Earlier reward, tested at a single date 95.0, 96.2, 93.8
rl_v1 Reward without the parameter check 58.4

Each folder includes a training/ directory with the run configuration and reward logs.

Results

Accuracy on 143 held-out covenants (2,527 questions), averaged over covenants and three training seeds.

Model Accuracy (%)
Qwen2.5-Coder-7B, answering directly 64.8
Llama-3.3-70B (4-bit), answering directly 79.1
Qwen2.5-Coder-32B, answering directly 85.8
Qwen3.8-27B, answering directly 88.8
FormaFin (Qwen2.5-Coder-7B) + Z3 94.8 ยฑ 1.0

Training

  1. Distillation. Supervised finetuning on 789 formulas written by a teacher model and filtered for validity.
  2. Reinforcement learning. GRPO on 828 further clauses without reference formulas. The reward is 0.3 + 0.7 ร— (share of solver tests passed), and 0 for formulas that do not parse with the required parameters. The tests check each limit stated in the clause just below, at and just above the threshold, across dates and conditions.

Limitations

  • The benchmark certificates are constructed, and the references are agreeing readings by two LLMs rather than expert annotations.
  • The model takes each line item as a reported amount and does not apply an agreement's definitions to raw financial statements.
  • Formulas should be reviewed before use. This is a research artifact, not financial or legal advice.

License

The adapters are released under MIT. Base models keep their own terms (Apache 2.0 for Qwen2.5-Coder and Mistral, the Llama 3.1 Community License for Llama). Training clauses come from the loan agreement corpus of Ganglmair and Wardlaw (CC BY 4.0).

Downloads last month
12
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for bethelhall/formafin

Base model

Qwen/Qwen2.5-7B
Adapter
(667)
this model