docs/FINE_TUNING.md

Fine-tuning the clause classifier (simple guide)

Goal: teach the DeBERTa model our exact clause-tagging task using the CUAD dataset, raising accuracy from a measured 0.61 to an expected 0.75–0.85 macro-F1.

You do NOT need to understand the internals. It's three commands.

One-time setup

# from the repo root
python3 -m venv .venv
.venv/bin/pip install -r backend/requirements.txt
.venv/bin/pip install -r backend/requirements-ml.txt   # torch, transformers, ...
.venv/bin/python backend/scripts/download_cuad.py        # downloads CUAD (~once)

The three steps

cd backend

# 1. Build training data from CUAD (QA spans -> labelled clauses)
../.venv/bin/python -m scripts.prepare_cuad
#   -> data/cuad/train.jsonl, val.jsonl, labels.json

# 2. Train (writes backend/models/clause-clf/)
../.venv/bin/python -m scripts.train_classifier --epochs 3
#   prints loss per epoch + a quick validation F1

# 3. Measure against CUAD, compare to the old number
../.venv/bin/python -m eval.run_eval --classifier finetuned --limit 50

Use the fine-tuned model in the app

CLASSIFIER=finetuned ../.venv/bin/uvicorn app.main:app --port 8000

If no trained model exists yet, the app safely falls back — set CLASSIFIER=zeroshot (DeBERTa, no training) or rules (no model at all).

Tips

• No GPU? It still runs on a Mac, just slower. For a quick smoke test: --device=cpu --batch=4 --max_len=256 --limit=800.
• Apple Silicon (MPS) out-of-memory? DeBERTa at batch 8 / seq 512 can exhaust unified memory. Use --device=cpu (most reliable), or keep MPS but lower --batch=4 --max_len=256.
• Fastest full run: upload prepare_cuad's output to Google Colab (free GPU) and run step 2 there, then copy backend/models/clause-clf/ back.
• Knobs: --model (default microsoft/deberta-v3-base), --epochs, --batch, --lr. Defaults are sensible; only change if you know why.

What gets produced

backend/models/clause-clf/      the trained model (loaded by app/finetuned.py)
data/cuad/train.jsonl, val.jsonl training data (git-ignored)

Why not LegalBERT?

We deliberately fine-tune DeBERTa, not LegalBERT — same effort, better result, cleaner licence. See DECISIONS.md §2.