docs/DECISIONS.md

Decisions, in plain English

This explains what we changed and why, without jargon. Read top to bottom.

1. Which branch do we build on?

We have three branches:

• main — the original working app.
• feature/complete-ui — main plus bug fixes and 5 new features (Risk Heatmap, Contract Comparison, Chat, Defined-Terms glossary). It also swapped the AI model that tags clauses to "LegalBERT" — that swap was a mistake (see below).
• dynamic — the one we use. Built on main, it adds the dynamic baseline, all the genuine bug/accuracy fixes, the 5 features, and the fine-tuning tools — while keeping main's better DeBERTa model (no LegalBERT).

Decision: use dynamic. It is the best of all worlds, and merges cleanly into main.

2. LegalBERT vs DeBERTa — why we did NOT use LegalBERT

The app tags each clause with a type ("this is a liability clause", "this is an auto-renewal clause"). It uses an AI model to do that.

• The name "LegalBERT" sounds better because it has "legal" in it. But LegalBERT, on its own, cannot classify anything — it only understands legal words. To make it classify, you must train it first.
• The feature/complete-ui branch tried to use LegalBERT without training, by comparing "how similar" a clause is to a description. That method is weak and was never measured. It also came with a stricter software licence.
• The original model (DeBERTa) can classify out of the box with no training, scored a real 0.61 in our tests, and has a clean licence.

Decision: keep DeBERTa. "Legal" in the name did not make it better.

3. The dynamic baseline — removing the hardcoded numbers

The app flags risky clauses by comparing them to "what's normal" (e.g. "a liability cap above 12 months is risky"). The problem: those numbers (12 months, 60 days, 99% uptime…) were typed directly into the code. If a judge asks "where does 12 come from?", the honest answer was "we made it up."

We fixed this:

• The numbers now live in a data file with a source, not in the code.
• A script (extract_cuad_baselines.py) can calculate those numbers from 500+ real contracts, so the answer becomes "it's the median of 500 real contracts" instead of a guess.
• As you upload your own contracts, the baseline can adapt toward them — but it never drops below the safe default, so nothing breaks.
• A safety test proves the new system produces the exact same flags as before on our demo contract, so accuracy did not drop.

Decision: baselines are now data, not hardcoded — and provably safe.

4. Fine-tuning — making the clause tagger smarter

"Fine-tuning" = taking the DeBERTa model and teaching it our exact task using thousands of example clauses that experts already labelled (the CUAD dataset, 13,000+ labels).

• Today (no training): 0.61 accuracy.
• After fine-tuning: expected 0.75–0.85 (a real, measurable jump).

We built the full toolkit so the team isn't stuck:

1. prepare_cuad.py — turns the raw dataset into training examples.
2. train_classifier.py — trains the model (one command).
3. eval/run_eval.py --classifier finetuned — measures the result.

Training the full model is faster on a GPU (e.g. Google Colab); the tools run anywhere. See FINE_TUNING.md.

Decision: fine-tune DeBERTa (not LegalBERT).

5. Do we train Qwen? No.

Qwen is the separate AI that writes out obligations in plain English. It is a different kind of model (it generates text, it doesn't tag).

• It already works without training.
• Training it would need a GPU, a special dataset we don't have, and lots of time — for little gain.
• The cheap, same-day way to improve it is better instructions/examples in the prompt, not training.

Decision: do NOT train Qwen. Improve it with prompting if needed.

Summary table

Question	Answer
Which branch?	dynamic
LegalBERT or DeBERTa?	DeBERTa (LegalBERT was worse + unmeasured)
Hardcoded baselines?	Replaced with data-derived, adaptive baselines
Fine-tune the tagger?	Yes — DeBERTa on CUAD (0.61 → ~0.75–0.85)
Train Qwen?	No — prompt-tune instead