docs/DECISIONS.md

# Decisions, in plain English

This explains *what we changed and why*, without jargon. Read top to bottom.

## 1. Which branch do we build on?

We have three branches:

- **`main`** — the original working app.
- **`feature/complete-ui`** — `main` plus bug fixes **and** 5 new features
  (Risk Heatmap, Contract Comparison, Chat, Defined-Terms glossary). It also
  swapped the AI model that tags clauses to "LegalBERT" — that swap was a
  mistake (see below).
- **`dynamic`** — **the one we use.** Built on `main`, it adds the dynamic
  baseline, all the genuine bug/accuracy fixes, the 5 features, and the
  fine-tuning tools — while keeping `main`'s better DeBERTa model (no LegalBERT).

**Decision: use `dynamic`.** It is the best of all worlds, and merges cleanly into main.

## 2. LegalBERT vs DeBERTa — why we did NOT use LegalBERT

The app tags each clause with a type ("this is a liability clause", "this is an
auto-renewal clause"). It uses an AI model to do that.

- The name "LegalBERT" *sounds* better because it has "legal" in it. But
  LegalBERT, on its own, **cannot classify anything** — it only understands
  legal words. To make it classify, you must *train* it first.
- The `feature/complete-ui` branch tried to use LegalBERT without training, by
  comparing "how similar" a clause is to a description. That method is weak and
  was **never measured**. It also came with a stricter software licence.
- The original model (**DeBERTa**) can classify out of the box with **no
  training**, scored a real **0.61** in our tests, and has a clean licence.

**Decision: keep DeBERTa.** "Legal" in the name did not make it better.

## 3. The dynamic baseline — removing the hardcoded numbers

The app flags risky clauses by comparing them to "what's normal" (e.g. "a
liability cap above 12 months is risky"). The problem: those numbers (12
months, 60 days, 99% uptime…) were **typed directly into the code**. If a judge
asks "where does 12 come from?", the honest answer was "we made it up."

We fixed this:

- The numbers now live in a **data file** with a source, not in the code.
- A script (`extract_cuad_baselines.py`) can **calculate** those numbers from
  500+ real contracts, so the answer becomes "it's the median of 500 real
  contracts" instead of a guess.
- As you upload your own contracts, the baseline can **adapt** toward them — but
  it never drops below the safe default, so nothing breaks.
- A **safety test** proves the new system produces the exact same flags as
  before on our demo contract, so accuracy did not drop.

**Decision: baselines are now data, not hardcoded — and provably safe.**

## 4. Fine-tuning — making the clause tagger smarter

"Fine-tuning" = taking the DeBERTa model and **teaching it our exact task**
using thousands of example clauses that experts already labelled (the CUAD
dataset, 13,000+ labels).

- Today (no training): **0.61** accuracy.
- After fine-tuning: expected **0.75–0.85** (a real, measurable jump).

We built the full toolkit so the team isn't stuck:

1. `prepare_cuad.py` — turns the raw dataset into training examples.
2. `train_classifier.py` — trains the model (one command).
3. `eval/run_eval.py --classifier finetuned` — measures the result.

Training the full model is faster on a GPU (e.g. Google Colab); the tools run
anywhere. See [FINE_TUNING.md](FINE_TUNING.md).

**Decision: fine-tune DeBERTa (not LegalBERT).**

## 5. Do we train Qwen? No.

Qwen is the *separate* AI that writes out obligations in plain English. It is a
different kind of model (it generates text, it doesn't tag).

- It already works without training.
- Training it would need a GPU, a special dataset we don't have, and lots of
  time — for little gain.
- The cheap, same-day way to improve it is **better instructions/examples in
  the prompt**, not training.

**Decision: do NOT train Qwen. Improve it with prompting if needed.**

## Summary table

| Question | Answer |
|---|---|
| Which branch? | `dynamic` |
| LegalBERT or DeBERTa? | **DeBERTa** (LegalBERT was worse + unmeasured) |
| Hardcoded baselines? | Replaced with **data-derived, adaptive** baselines |
| Fine-tune the tagger? | **Yes — DeBERTa on CUAD** (0.61 → ~0.75–0.85) |
| Train Qwen? | **No** — prompt-tune instead |