Instructions to use bethelhall/formafin with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use bethelhall/formafin with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct") model = PeftModel.from_pretrained(base_model, "bethelhall/formafin") - Notebooks
- Google Colab
- Kaggle
FormaFin
FormaFin is a LoRA adapter for Qwen2.5-Coder-7B-Instruct that translates a loan covenant into an SMT-LIB formula. An SMT solver such as Z3 then applies the formula to a borrower's compliance certificate to decide compliance, earnings cushion and debt capacity. The model is small enough to run on a single GPU, so confidential figures stay in house.
Quick start
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "bethelhall/formafin"
tok = AutoTokenizer.from_pretrained(repo)
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, repo).eval()
SYSTEM = ("Translate the real financial covenant into SMT-LIB2. Return exactly a set-logic command and one "
"define-fun named covenant_holds that returns Bool and is true exactly when the borrower complies.")
PROMPT = """Covenant clause:
{clause}
Write (define-fun covenant_holds (<parameters>) Bool <body>) with EXACTLY these parameters, in this order and with
these types (do not add, rename or drop any):
{params}
Amounts are in US dollars as reported in the borrower's compliance certificate; test_date is the test date as an
integer YYYYMMDD (e.g. 20121231), so a threshold that changes over time is written with test_date comparisons.
covenant_holds is true exactly when every financial covenant in the clause that applies on test_date is satisfied.
Return only the SMT-LIB2 (set-logic QF_NRA or QF_LIA/QF_LRA as needed, then the define-fun)."""
clause = "The Borrower shall maintain at all times a Total Debt to EBITDA Ratio of not more than 3.0 to 1.0."
params = "(total_debt Real) ; Consolidated Total Debt\n(ebitda Real) ; Consolidated EBITDA\n(test_date Int) ; test date as YYYYMMDD"
messages = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": PROMPT.format(clause=clause, params=params)}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
formula = tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
The parameter list must be passed exactly: one Real per certificate line item, then test_date as Int, then one
Bool per condition.
Checking a certificate with Z3 (pip install z3-solver):
import re, z3
smt = re.sub(r"```\w*", "", formula)
smt = smt[smt.index("(define-fun"):]
decls = "(declare-const total_debt Real)(declare-const ebitda Real)(declare-const test_date Int)"
facts = "(assert (= total_debt 453900000.0))(assert (= ebitda 149800000.0))(assert (= test_date 20121231))"
s = z3.Solver()
s.from_string(smt + decls + facts + "(assert (covenant_holds total_debt ebitda test_date))")
print("comply" if s.check() == z3.sat else "breach") # 3.03x > 3.0x, so breach
Adapters
The root adapter is rl_sched_s42. The other adapters load with subfolder="<folder>" on their own base model.
| Folder | Base model | Accuracy (%) |
|---|---|---|
rl_sched_s42 (default), _s43, _s44 |
Qwen2.5-Coder-7B-Instruct | 93.8, 94.9, 95.9 |
llama_rl_sched_s42, _s43, _s44 |
Llama-3.1-8B-Instruct | 96.9, 97.8, 95.7 |
mistral_rl_sched_s42, _s43, _s44 |
Mistral-7B-Instruct-v0.3 | 91.1, 93.2, 93.6 |
Other runs kept for reproducibility
| Folder | Description | Accuracy (%) |
|---|---|---|
sft789 |
Supervised finetuning only | 85.8 |
rl_v2_s42, _s43, _s44 |
Earlier reward, tested at a single date | 95.0, 96.2, 93.8 |
rl_v1 |
Reward without the parameter check | 58.4 |
Each folder includes a training/ directory with the run configuration and reward logs.
Results
Accuracy on 143 held-out covenants (2,527 questions), averaged over covenants and three training seeds.
| Model | Accuracy (%) |
|---|---|
| Qwen2.5-Coder-7B, answering directly | 64.8 |
| Llama-3.3-70B (4-bit), answering directly | 79.1 |
| Qwen2.5-Coder-32B, answering directly | 85.8 |
| Qwen3.8-27B, answering directly | 88.8 |
| FormaFin (Qwen2.5-Coder-7B) + Z3 | 94.8 ยฑ 1.0 |
Training
- Distillation. Supervised finetuning on 789 formulas written by a teacher model and filtered for validity.
- Reinforcement learning. GRPO on 828 further clauses without reference formulas. The reward is 0.3 + 0.7 ร (share of solver tests passed), and 0 for formulas that do not parse with the required parameters. The tests check each limit stated in the clause just below, at and just above the threshold, across dates and conditions.
Limitations
- The benchmark certificates are constructed, and the references are agreeing readings by two LLMs rather than expert annotations.
- The model takes each line item as a reported amount and does not apply an agreement's definitions to raw financial statements.
- Formulas should be reviewed before use. This is a research artifact, not financial or legal advice.
License
The adapters are released under MIT. Base models keep their own terms (Apache 2.0 for Qwen2.5-Coder and Mistral, the Llama 3.1 Community License for Llama). Training clauses come from the loan agreement corpus of Ganglmair and Wardlaw (CC BY 4.0).
- Downloads last month
- 12