README.md

---
license: apache-2.0
language:
  - en
tags:
  - onnx
  - bert
  - cross-encoder
  - legal
  - contract-understanding
  - reranking
  - cuad
base_model: cross-encoder/ms-marco-MiniLM-L-6-v2
pipeline_tag: text-ranking
---

# cuad-cross-encoder-v11

A cross-encoder reranker fine-tuned for **legal clause retrieval** in contract review workflows. Built on [`cross-encoder/ms-marco-MiniLM-L-6-v2`](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2) and fine-tuned on a combination of CUAD, ACCORD, LEDGAR, ContractNLI, EDGAR-sourced contract pairs, and LLM-generated synthetic pairs targeting known failure patterns.

Deployed as **ONNX INT8** for in-browser inference via [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/).

---

## Intended Use

- **Primary:** Reranking retrieved contract chunks against natural-language clause queries (e.g. *"What are the governing law provisions?"*, *"What IP does each party retain?"*)
- **Domains covered:** Joint Venture, Intellectual Property, Non-Compete / Non-Solicit, Non-Disclosure Agreement (NDA)
- **Not intended for:** General-purpose document retrieval, non-legal domains, or as a standalone legal advisor

---

## Training Data

| Source | Description | Pairs |
|--------|-------------|-------|
| CUAD v1 | 510 contracts, 41 clause categories (Atticus Project) | ~30,000 |
| ACCORD | 3,931 annotated legal passages | ~6,000 |
| LEDGAR | SEC EDGAR provisions, 14 labels filtered for JV/NC/IP | ~4,000 |
| ContractNLI / LegalBench | 14 NLI tasks over contract text | ~3,000 |
| EDGAR default (re-run) | SC 13D + 8-K NC/IP exhibits | 6,680 pos |
| EDGAR JV (re-run) | 8-K joint venture exhibit filings | 1,054 pos |
| EDGAR Sino-JV (re-run) | 20-F chapter-format Sino-JV agreements | ~4,272 |
| EDGAR NDA *(new v11)* | EX-99 confidentiality exhibits from SC TO-T / 8-K filings | 4,417 pos |
| Synthetic spinoff-IP *(new v11)* | LLM-labeled pairs for IP spinoff format failures | 294 |
| Synthetic NDA *(new v11)* | LLM-labeled pairs for NDA section-dominance failures | 267 |
| Synthetic definitions-bleed *(new v11)* | LLM-labeled pairs for definitions-article bleed failures | 507 |
| Synthetic Armstrong-IP *(new v11)* | LLM-labeled pairs for irrevocable license confusion | 102 |
| Eval positives | Full-chunk positives extracted from passing eval cases | 84 |
| Pipeline hard negatives | Clause queries where prior model failed — reranked negatives | 254 |

**Total: ~72,101 training pairs · 5,612 validation pairs**

Pairs are `(query, positive_chunk, negative_chunk)` triplets. Negatives are a mix of hard negatives (wrong clause from same contract) and random negatives (chunks from other contracts).

---

## Training Details

| Hyperparameter | Value |
|----------------|-------|
| Base model | `cross-encoder/ms-marco-MiniLM-L-6-v2` |
| Epochs | 3 |
| Batch size | 32 |
| Learning rate | 2e-5 |
| Max sequence length | 512 tokens |
| Warmup steps | 10% of total steps |
| Loss | Cross-entropy (sentence-transformers `CrossEncoderTrainer`) |
| Hardware | NVIDIA RTX 3090 (RunPod) |
| Training time | ~50 min |

---

## Evaluation

Evaluated on a held-out set of 16 contracts across 4 clause domains. Each contract is queried with 3–8 clause-type questions; the top-ranked chunk is scored as **pass** (correct clause returned), **partial** (correct section but wrong chunk boundary), or **fail**.

| Suite | Contracts | Queries | Pass | Partial | Fail | vs v10 |
|-------|-----------|---------|------|---------|------|--------|
| Joint Venture | 9 | 51 | 20 (39%) | 15 (29%) | 16 (31%) | +11 pass 🚀 |
| Intellectual Property | 4 | 49 | 18 (37%) | 18 (37%) | 13 (27%) | +1 pass |
| Non-Compete / Non-Solicit | 3 | 13 | 6 (46%) | 7 (54%) | 0 (0%) | +1 pass |
| NDA | 3 | 19 | 9 (47%) | 7 (37%) | 3 (16%) | +1 pass |

**Test contracts (JV):** MightyCell Batteries, BorrowMoney.com, Galera Therapeutics, MINDA IMPCO Technologies, Kiromic Biopharma, Novo Integrated Sciences, Transphorm / Aizu Fujitsu, Valence Technology / Baoding Fengfan, Veoneer

**Test contracts (IP):** Armstrong Flooring, Cerence Inc, Garrett Motion, Rare Element Resources

**Test contracts (NDA):** Kite Pharma / Gilead Sciences, Fortune Brands / Norcraft Companies, Aspect Medical Systems / Tyco Healthcare

The JV improvement (+11 pass) is driven by new Sino-JV EDGAR data and synthetic definitions-bleed pairs targeting contracts where the model previously returned definitions articles for Governing Law and Non-Compete queries.

---

## Usage

### ONNX Runtime (recommended for browser / edge)

```python
import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("datgacon/cuad-cross-encoder-v11")
session = ort.InferenceSession("onnx/model_quantized.onnx")

query = "What governing law applies to this agreement?"
passage = "This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware."

inputs = tokenizer(query, passage, return_tensors="np", max_length=512, truncation=True, padding=True)
outputs = session.run(None, {k: v for k, v in inputs.items() if k in ["input_ids", "attention_mask", "token_type_ids"]})
score = outputs[0][0][0]
print(f"Relevance score: {score:.4f}")
```

### sentence-transformers (PyTorch)

```python
from sentence_transformers.cross_encoder import CrossEncoder

model = CrossEncoder("datgacon/cuad-cross-encoder-v11")

query = "What governing law applies to this agreement?"
passages = [
    "This Agreement shall be governed by the laws of the State of Delaware.",
    "Each party shall maintain the confidentiality of the other party's information.",
    "The term of this Agreement shall commence on the Effective Date.",
]

scores = model.predict([(query, p) for p in passages])
ranked = sorted(zip(scores, passages), reverse=True)
for score, passage in ranked:
    print(f"{score:.4f}  {passage[:80]}")
```

---

## Limitations

- Trained on US commercial contracts (CUAD corpus); may underperform on EU, UK, or public-sector agreements
- Partial matches are common at clause-boundary edges — chunk size and overlap in the retrieval pipeline significantly affect results
- Not a legal advisor — scores indicate retrieval relevance, not legal interpretation
- Performance on clause types outside the four trained domains (JV, IP, NC, NDA) is untested
- Token type IDs must be passed explicitly when using ONNX Runtime Web; omitting them collapses score spread

---

## Citation

If you use this model, please cite the underlying datasets:

```bibtex
@article{hendrycks2021cuad,
  title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
  author={Hendrycks, Dan and others},
  journal={arXiv preprint arXiv:2103.06268},
  year={2021}
}
```