| --- |
| base_model: microsoft/Phi-3-mini-4k-instruct |
| library_name: peft |
| tags: |
| - lora |
| - legal |
| - text-classification |
| - contract-analysis |
| license: mit |
| datasets: |
| - cuad |
| language: |
| - en |
| --- |
| |
| # contract-clause-phi3-lora |
|
|
| A QLoRA fine-tune of [microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) for legal contract clause classification, trained on the [CUAD](https://www.atticusprojectai.org/cuad) (Contract Understanding Atlas Dataset) dataset. Given a contract clause, it predicts which of 41 legal clause categories it belongs to (or "None"). |
|
|
| **Code, training pipeline, and full evaluation:** [github.com/D-L-C-S/contract-clause-qlora](https://github.com/D-L-C-S/contract-clause-qlora) |
|
|
| ## Results |
|
|
| Evaluated on a held-out test set of 2,784 clauses, compared against the zero-shot base model: |
|
|
| | Metric | Zero-shot base model | This adapter | |
| |---|---|---| |
| | Accuracy | 21.6% | **62.5%** | |
| | Invalid (unparseable) output rate | 58.5% | **0.5%** | |
|
|
| Full per-category metrics, error analysis, and known limitations are in the [eval notebook](https://github.com/D-L-C-S/contract-clause-qlora/blob/main/notebooks/eval.ipynb). |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig |
| from peft import PeftModel |
| |
| base_model = AutoModelForCausalLM.from_pretrained( |
| "microsoft/Phi-3-mini-4k-instruct", |
| quantization_config=BitsAndBytesConfig( |
| load_in_4bit=True, |
| bnb_4bit_quant_type="nf4", |
| bnb_4bit_compute_dtype=torch.float16, |
| ), |
| device_map="auto", |
| ) |
| tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct") |
| model = PeftModel.from_pretrained(base_model, "DLCS/contract-clause-phi3-lora") |
| model.eval() |
| |
| clause = "This Agreement shall be governed by and construed under the laws of the State of Delaware." |
| prompt_messages = [{ |
| "role": "user", |
| "content": f'Classify the following contract clause into its category, or respond with "None" if it does not match any category. Respond with only the category name and nothing else.\n\nClause:\n{clause}', |
| }] |
| prompt = tokenizer.apply_chat_template(prompt_messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| with torch.inference_mode(): |
| output = model.generate(**inputs, max_new_tokens=18) |
| print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| # "Governing Law" |
| ``` |
|
|
| ## Training details |
|
|
| - **Base model:** microsoft/Phi-3-mini-4k-instruct, loaded in 4-bit (NF4) |
| - **Method:** LoRA (r=16, alpha=32, dropout=0.05) on `qkv_proj`, `o_proj`, `gate_up_proj`, `down_proj` — ~0.57% of total parameters trainable |
| - **Data:** CUAD's annotated clause spans (positive examples) plus heuristically-constructed "None" examples from unannotated text gaps, split at the contract level to prevent leakage |
| - **Training run:** 2 epochs, 376 steps, full fp32 (see the [GitHub repo](https://github.com/D-L-C-S/contract-clause-qlora) for why — a Turing-GPU bf16 limitation combined with a `bitsandbytes`/`GradScaler` bug forced this) |
|
|
| ## Limitations |
|
|
| Single-label classification only; several of the 41 categories have very few test examples, so per-category metrics for those are directional, not statistically reliable; the "None" class is heuristically constructed and carries some inherent label noise. See the [eval notebook](https://github.com/D-L-C-S/contract-clause-qlora/blob/main/notebooks/eval.ipynb) for a full, honest discussion. |
|
|
