solankiom/llama-3.1-8b-contract-extractor

Llama 3.1 8B — Contract Clause Extractor (QLoRA adapter)

A LoRA adapter that fine-tunes meta-llama/Llama-3.1-8B-Instruct to extract 12 commercially-critical contract clauses as strict JSON, trained on the CUAD (Contract Understanding Atticus Dataset). Fine-tuning lifts schema-valid JSON output from 0% / 12% (naive / strong-prompt baselines) to 96% on a held-out test set.

• Base model: unsloth/llama-3.1-8b-instruct-unsloth-bnb-4bit (4-bit; identical weights to meta-llama/Llama-3.1-8B-Instruct)
• Method: QLoRA (Unsloth 4-bit base + LoRA) via TRL SFTTrainer, assistant-only loss
• Task: structured legal contract clause extraction (12 fields)
• Language: English
• License: MIT (adapter weights). CUAD data is CC BY 4.0 — see License & Data.
• Code: https://github.com/OmkumarSolanki/fine-tuned-contract-extractor

The 12 fields

document_name, parties, agreement_date, effective_date, expiration_date, governing_law, renewal_term, notice_period_to_terminate_renewal, exclusivity, non_compete, cap_on_liability, uncapped_liability.

All non-list fields are null when the contract doesn't address the topic; parties is a (possibly empty) list of strings.

How to use

This is a PEFT/LoRA adapter — load the base model, then apply the adapter. Use the exact training prompt (below); a different prompt degrades accuracy.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct", device_map="auto", load_in_4bit=True
)
model = PeftModel.from_pretrained(base, "solankiom/llama-3.1-8b-contract-extractor")
tokenizer = AutoTokenizer.from_pretrained("solankiom/llama-3.1-8b-contract-extractor")

SYSTEM_PROMPT = 'You are a legal contract analyst. Extract structured clauses from contracts.'
USER_PROMPT_TEMPLATE = 'Extract structured clauses from this contract:\n\n{contract_text}'

contract_text = "AGREEMENT made as of January 1, 2024, between Acme Corp and Beta Inc. ..."
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": USER_PROMPT_TEMPLATE.format(contract_text=contract_text)},
]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(input_ids=input_ids, max_new_tokens=2048, do_sample=False)
print(tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True))
# -> compact JSON with the 12 fields

Unsloth users can instead load this repo id directly with FastLanguageModel.from_pretrained(model_name="solankiom/llama-3.1-8b-contract-extractor", load_in_4bit=True).

Training

QLoRA on the 408/51/51 ChatML split (seed 42), assistant-only loss, on 1× 1x NVIDIA A100 80GB PCIe.

Hyperparameter	Value
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	7 projection modules
Trainable params	41,943,040 / 8,072,204,288 (0.52%)
Epochs / steps	3 / 153
Effective batch	8 (1 × grad-accum 8)
Optimizer / LR	adamw_8bit, 0.0002 (cosine)
Precision	bf16
Best val eval_loss	0.2127 (step 100, kept via load_best_model_at_end)
Final mean train_loss	0.1767
Runtime	~54 min

Evaluation

Held-out 51-contract test set, greedy decoding (deterministic). The reportable metric is JSON-validity — the fraction of outputs that parse as JSON and validate against the 12-field schema.

Model	JSON-validity (51 contracts)	overall_f1 (CAVEATED)
Naive baseline	0 / 51 (0%)	0.4069
Strong-prompt baseline	6 / 51 (12%)	0.4139
Fine-tuned (this adapter)	49 / 51 (96%)	0.7295

Read the per-field F1 with the validity rate, never alone. Schema-invalid predictions are scored as empty extractions; because many CUAD gold fields are null, an empty prediction scores "correct" on those sparse fields, which inflates the baselines' per-field numbers. The metric is an apples-to-apples extraction-quality measure only once a model mostly emits valid JSON — which is exactly what fine-tuning achieves here.

Fine-tuned per-field match rate (CAVEATED)

Field	Match rate
document_name	0.863
parties	0.774
agreement_date	0.882
effective_date	0.647
expiration_date	0.471
governing_law	0.686
renewal_term	0.804
notice_period_to_terminate_renewal	0.804
exclusivity	0.667
non_compete	0.745
cap_on_liability	0.667
uncapped_liability	0.745

Limitations

• English-only, trained on commercial contracts from CUAD; out-of-distribution documents (other languages, non-commercial agreements) will degrade.
• Long contracts are head+tail-truncated to an 8000-token budget at training time; extremely long inputs may still be truncated at inference.
• Not legal advice. Outputs must be reviewed by a qualified professional.
• No authentication is built into the reference serving layer — add it before any public deployment.

License & Data

• Adapter weights: MIT © 2026 Om Solanki.
• Base model: subject to the Llama 3.1 Community License.
• Training data: CUAD (CC BY 4.0), via the public theatticusproject/cuad-qa mirror. No CUAD-derived contract text is redistributed in this repo.

Acknowledgments

• The Atticus Project — for curating and releasing CUAD.
• Meta AI — for Llama 3.1 8B Instruct.
• Unsloth AI — for the 4-bit base mirror and fast QLoRA tooling.
• Hugging Face — for transformers, peft, trl, and the Hub.

@article{hendrycks2021cuad,
  title   = {CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
  author  = {Dan Hendrycks and Collin Burns and Anya Chen and Spencer Ball},
  journal = {arXiv preprint arXiv:2103.06268},
  year    = {2021}
}