ashrafksalim/contract-clause-risk-lora

Contract Clause Risk Classifier (LoRA)

A LoRA-fine-tuned microsoft/deberta-v3-small that classifies contract clauses into their legal category, as part of an end-to-end contract risk review pipeline (fine-tuning → RAG over precedent clauses → agentic explanation generation → confidence-based human review routing).

Model Details

• Base model: microsoft/deberta-v3-small
• Fine-tuning method: LoRA (r=8, alpha=16, dropout=0.1, targeting query_proj, key_proj, value_proj)
• Task: Multi-class clause classification, 100 categories
• Training data: LEDGAR (lex_glue, ledgar config) — 60,000 train / 10,000 validation / 10,000 test examples, real clause text drawn from SEC contract filings
• Training regime: 3 epochs, learning rate 3e-4, batch size 16
• Hardware: single T4 GPU (Colab)

Results (test set, n=10,000)

Metric	Value
Accuracy	0.797
Macro F1	0.643
Eval loss	0.784
Throughput	~84.5 samples/sec

Read macro-F1 alongside accuracy, not instead of it. Accuracy is pulled up by a handful of common, easy categories; macro-F1 weights every category equally and exposes that performance is uneven across the full label set.

Per-category performance

Strongest categories (F1 > 0.96):

Category	Precision	Recall	F1	Support
Financial Statements	0.988	0.988	0.988	82
Counterparts	0.970	0.998	0.984	490
Use Of Proceeds	0.975	0.975	0.975	120
Base Salary	0.982	0.964	0.973	112
Waiver Of Jury Trials	0.964	0.973	0.969	111

Weakest categories (F1 = 0.0 — model never correctly predicted these on the test set):

Category	Support
Applicable Laws	53
Defined Terms	56
Indemnity	29
Modifications	55
Assigns	4
Costs	15
Books	2
Qualifications	5

25 of 100 categories fall below F1 = 0.5. The near-zero-support categories (Books, n=2; Assigns, n=4) are unsurprising — there's barely any signal to learn from. But Indemnity (n=29) and Applicable Laws (n=53) and Defined Terms (n=56) failing completely despite reasonable support is a real weakness, not a data-starvation artifact, and is exactly the kind of category a fairness/bias guardrail should catch before this model is trusted unsupervised on categories that matter for actual risk decisions.

Inference latency

Measured on 50 samples, single T4 GPU, classifier forward pass only (does not include RAG retrieval or LLM summary generation from the full pipeline):

	Value
Mean	25.97 ms
p50	26.44 ms
p95	35.02 ms

Estimated time savings

Based on an assumed 3 minutes of manual review per clause (adjust to your own team's real average — this is a placeholder, not measured data):

Clauses	Manual review	Automated	Time saved
100	5.0 hrs	~4 sec	~5.0 hrs
1,000	50.0 hrs	~7 sec	~50.0 hrs
10,000	500.0 hrs	~72 sec	~499.9 hrs

These numbers cover classification only. They do not include the LLM summary-generation step, which is the slower part of the full pipeline in practice.

Known limitation: risk-tier mapping needs correction

This model outputs a clause category (e.g. Indemnity, Governing Laws). A separate rule-based lookup maps categories to a risk tier (High/Medium/Low) for the downstream pipeline. As shipped, only 3 of 11 intended high-risk category names actually match LEDGAR's real label strings (e.g. the dataset uses Indemnity, not Indemnification), so the risk-tier output should not be trusted until that mapping is corrected against the model's actual id2label values. The classifier's category predictions above are accurate to what's reported; the risk-tier labeling built on top of them currently is not.

Intended Use

Prototype / portfolio component for automating a first pass of contract clause categorization ahead of legal review. Not a substitute for legal review — low-confidence and weak-category predictions should be routed to a human, which is what the confidence-threshold guardrail in the full pipeline does.

Bias, Risks, and Limitations

• Training labels come from LEDGAR (real SEC filing clauses), which is a reasonable proxy for clause type but was not curated by legal experts for this specific risk-review use case.
• Performance is uneven across categories (see per-category table above); 25/100 categories underperform, several with zero correct predictions on held-out data.
• Latency figures reflect classification only, not the full RAG + LLM pipeline.
• Risk-tier labels are currently unreliable — see limitation above — and should not be used until the category-name mapping is fixed.

Training Procedure

lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=8,
    lora_alpha=16,
    lora_dropout=0.1,
    target_modules=["query_proj", "key_proj", "value_proj"],
)

3 epochs, learning rate 3e-4, batch size 16, evaluated each epoch on the validation split, best checkpoint selected by macro-F1.

How to Use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

base = AutoModelForSequenceClassification.from_pretrained(
    "microsoft/deberta-v3-small", num_labels=100
)
model = PeftModel.from_pretrained(base, "ashrafksalim/contract-clause-risk-lora")
tokenizer = AutoTokenizer.from_pretrained("ashrafksalim/contract-clause-risk-lora")

inputs = tokenizer("Each party shall indemnify and hold harmless...", return_tensors="pt")
logits = model(**inputs).logits