qwen-contract-clause-extractor

QLoRA fine-tune of Qwen2.5-1.5B-Instruct that extracts 12 key clause types from commercial contracts into structured JSON.

Model Details

• Base model: Qwen/Qwen2.5-1.5B-Instruct
• Fine-tuning method: QLoRA (4-bit NF4 quantization + LoRA adapters)
• Task: Structured clause extraction (contract text → JSON)
• Language: English
• License: Apache 2.0 (inherited from base model)

Model Description

Given raw contract text, the model returns a single JSON object with 12 fields, using null for any clause category not present in the contract:

parties, agreement_date, effective_date, governing_law, termination_for_convenience, cap_on_liability, non_compete, exclusivity, expiration_date, license_grant, document_name, renewal_term

These map to 12 of the 41 clause categories in CUAD (Contract Understanding Atticus Dataset), chosen for being common across most commercial contract types.

Intended Uses

• Drafting-assistance / first-pass review tooling that flags key clauses in a contract for a lawyer to verify
• Structured-extraction portfolio/benchmark example for small-model QLoRA fine-tuning

Not intended for: standalone legal advice, or any use where extracted output is relied upon without human review. This model has a non-trivial hallucination rate on absent fields (see Limitations).

How to Use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_model_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "meharshh/qwen-contract-clause-extractor"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id, quantization_config=bnb_config, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)

instruction = (
    "Extract the following fields from this contract. "
    "Return valid JSON only, with null for any field not present.\n"
    "Fields: parties, agreement_date, effective_date, governing_law, "
    "termination_for_convenience, cap_on_liability, non_compete, exclusivity, "
    "expiration_date, license_grant, document_name, renewal_term\n\n"
    "Contract text:\n<your contract text here>"
)
messages = [{"role": "user", "content": instruction}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training Data

CUAD — 510 real commercial contracts with expert legal annotations, originally SQuAD-style span extraction. Reshaped into one instruction→JSON example per contract (285 train / 61 validation / 62 test).

Training Procedure

Quantization	4-bit NF4 (bitsandbytes), bf16 compute
LoRA rank / alpha / dropout	16 / 32 / 0.05
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Epochs	3
Learning rate	2e-4, cosine schedule
Effective batch size	8
Hardware	Single consumer GPU, 4GB VRAM

Evaluation

Evaluated on a 62-example held-out test set. Metric: per-field exact match (case-insensitive, whitespace-normalized) computed only over outputs that parsed as valid JSON, plus overall JSON validity rate.

Metric	Base model	Fine-tuned	Change
JSON validity	98.4%	95.2%	-3.2pp
parties	0%	24%	+24pp
agreement_date	41%	63%	+22pp
effective_date	33%	44%	+11pp
governing_law	10%	2%	-8pp
termination_for_convenience	57%	51%	-6pp
cap_on_liability	33%	19%	-14pp
non_compete	62%	85%	+23pp
exclusivity	48%	59%	+11pp
expiration_date	18%	10%	-8pp
license_grant	41%	53%	+12pp
document_name	49%	83%	+34pp
renewal_term	52%	69%	+17pp
Overall field accuracy	37.0%	46.8%	+9.8pp

Fine-tuning improved 8 of the 12 fields, with a net +9.8 percentage point gain in overall field accuracy.

Loss curve

Epoch	Train loss (end of epoch)	Eval loss
1	1.45	1.4133
2	1.28	1.3891
3	1.23	1.3875

Train loss decreases steadily across all 3 epochs, but eval loss is nearly flat after epoch 1 — the model is data-limited rather than undertrained; more epochs on this dataset size wouldn't meaningfully help, more/varied training examples would.

Example output

Input: a 1996 technology license agreement.

{
  "parties": "ENTRUST TECHNOLOGIES INC.",
  "agreement_date": "31 December, 1996",
  "effective_date": "31 December, 1996",
  "governing_law": "This Agreement will be governed by and construed in accordance with the laws of the Province of Ontario without regard to conflicts of law principles.",
  "termination_for_convenience": null,
  "cap_on_liability": "IN NO EVENT SHALL EITHER PARTY BE LIABLE TO THE OTHER FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES ARISING OUT OF THIS AGREEMENT, EVEN IF SUCH PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.",
  "non_compete": null,
  "exclusivity": null,
  ...
}

Qualitative error analysis

Manual review of the 4 regressed fields (governing_law, cap_on_liability, termination_for_convenience, expiration_date) shows the exact-match metric understates real quality:

• Paraphrase mismatches: e.g. ground truth "...without reference to conflict of law principles" vs. model output "...without giving effect to principles of conflicts of law" — same legal meaning, scored as wrong by exact string match.
• Valid-but-different clause selection: contracts often contain multiple sentences relevant to a category (e.g. cap_on_liability); the model sometimes extracts a different, equally valid sentence than the one CUAD's annotators chose as ground truth.
• Genuine failure mode: when a field is truly absent (null in ground truth), the model occasionally still generates a confident-sounding but fabricated clause instead of returning null.

Limitations

• Exact-match evaluation likely understates true quality: manual review found several "incorrect" predictions were paraphrases or alternate valid clause selections rather than genuine errors.
• Some fields (non_compete, exclusivity, renewal_term) had under 40% non-null coverage in training data, limiting their ceiling.
• The model occasionally generates a plausible-sounding but fabricated clause for fields that are genuinely absent, instead of returning null — outputs should always be reviewed by a human before use.
• Covers only 12 of CUAD's 41 labeled clause categories.