cuad-cross-encoder-v11

A cross-encoder reranker fine-tuned for legal clause retrieval in contract review workflows. Built on cross-encoder/ms-marco-MiniLM-L-6-v2 and fine-tuned on a combination of CUAD, ACCORD, LEDGAR, ContractNLI, EDGAR-sourced contract pairs, and LLM-generated synthetic pairs targeting known failure patterns.

Deployed as ONNX INT8 for in-browser inference via ONNX Runtime Web.

---

Intended Use

• Primary: Reranking retrieved contract chunks against natural-language clause queries (e.g. "What are the governing law provisions?", "What IP does each party retain?")
• Domains covered: Joint Venture, Intellectual Property, Non-Compete / Non-Solicit, Non-Disclosure Agreement (NDA)
• Not intended for: General-purpose document retrieval, non-legal domains, or as a standalone legal advisor

---

Training Data

Source	Description	Pairs
CUAD v1	510 contracts, 41 clause categories (Atticus Project)	~30,000
ACCORD	3,931 annotated legal passages	~6,000
LEDGAR	SEC EDGAR provisions, 14 labels filtered for JV/NC/IP	~4,000
ContractNLI / LegalBench	14 NLI tasks over contract text	~3,000
EDGAR default (re-run)	SC 13D + 8-K NC/IP exhibits	6,680 pos
EDGAR JV (re-run)	8-K joint venture exhibit filings	1,054 pos
EDGAR Sino-JV (re-run)	20-F chapter-format Sino-JV agreements	~4,272
EDGAR NDA (new v11)	EX-99 confidentiality exhibits from SC TO-T / 8-K filings	4,417 pos
Synthetic spinoff-IP (new v11)	LLM-labeled pairs for IP spinoff format failures	294
Synthetic NDA (new v11)	LLM-labeled pairs for NDA section-dominance failures	267
Synthetic definitions-bleed (new v11)	LLM-labeled pairs for definitions-article bleed failures	507
Synthetic Armstrong-IP (new v11)	LLM-labeled pairs for irrevocable license confusion	102
Eval positives	Full-chunk positives extracted from passing eval cases	84
Pipeline hard negatives	Clause queries where prior model failed — reranked negatives	254

Total: ~72,101 training pairs · 5,612 validation pairs

Pairs are (query, positive_chunk, negative_chunk) triplets. Negatives are a mix of hard negatives (wrong clause from same contract) and random negatives (chunks from other contracts).

---

Training Details

Hyperparameter	Value
Base model	cross-encoder/ms-marco-MiniLM-L-6-v2
Epochs	3
Batch size	32
Learning rate	2e-5
Max sequence length	512 tokens
Warmup steps	10% of total steps
Loss	Cross-entropy (sentence-transformers CrossEncoderTrainer)
Hardware	NVIDIA RTX 3090 (RunPod)
Training time	~50 min

---

Evaluation

Evaluated on a held-out set of 16 contracts across 4 clause domains. Each contract is queried with 3–8 clause-type questions; the top-ranked chunk is scored as pass (correct clause returned), partial (correct section but wrong chunk boundary), or fail.

Suite	Contracts	Queries	Pass	Partial	Fail	vs v10
Joint Venture	9	51	20 (39%)	15 (29%)	16 (31%)	+11 pass 🚀
Intellectual Property	4	49	18 (37%)	18 (37%)	13 (27%)	+1 pass
Non-Compete / Non-Solicit	3	13	6 (46%)	7 (54%)	0 (0%)	+1 pass
NDA	3	19	9 (47%)	7 (37%)	3 (16%)	+1 pass

Test contracts (JV): MightyCell Batteries, BorrowMoney.com, Galera Therapeutics, MINDA IMPCO Technologies, Kiromic Biopharma, Novo Integrated Sciences, Transphorm / Aizu Fujitsu, Valence Technology / Baoding Fengfan, Veoneer

Test contracts (IP): Armstrong Flooring, Cerence Inc, Garrett Motion, Rare Element Resources

Test contracts (NDA): Kite Pharma / Gilead Sciences, Fortune Brands / Norcraft Companies, Aspect Medical Systems / Tyco Healthcare

The JV improvement (+11 pass) is driven by new Sino-JV EDGAR data and synthetic definitions-bleed pairs targeting contracts where the model previously returned definitions articles for Governing Law and Non-Compete queries.

---

Usage

ONNX Runtime (recommended for browser / edge)

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("datgacon/cuad-cross-encoder-v11")
session = ort.InferenceSession("onnx/model_quantized.onnx")

query = "What governing law applies to this agreement?"
passage = "This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware."

inputs = tokenizer(query, passage, return_tensors="np", max_length=512, truncation=True, padding=True)
outputs = session.run(None, {k: v for k, v in inputs.items() if k in ["input_ids", "attention_mask", "token_type_ids"]})
score = outputs[0][0][0]
print(f"Relevance score: {score:.4f}")

sentence-transformers (PyTorch)

from sentence_transformers.cross_encoder import CrossEncoder

model = CrossEncoder("datgacon/cuad-cross-encoder-v11")

query = "What governing law applies to this agreement?"
passages = [
    "This Agreement shall be governed by the laws of the State of Delaware.",
    "Each party shall maintain the confidentiality of the other party's information.",
    "The term of this Agreement shall commence on the Effective Date.",
]

scores = model.predict([(query, p) for p in passages])
ranked = sorted(zip(scores, passages), reverse=True)
for score, passage in ranked:
    print(f"{score:.4f}  {passage[:80]}")

---

Limitations

• Trained on US commercial contracts (CUAD corpus); may underperform on EU, UK, or public-sector agreements
• Partial matches are common at clause-boundary edges — chunk size and overlap in the retrieval pipeline significantly affect results
• Not a legal advisor — scores indicate retrieval relevance, not legal interpretation
• Performance on clause types outside the four trained domains (JV, IP, NC, NDA) is untested
• Token type IDs must be passed explicitly when using ONNX Runtime Web; omitting them collapses score spread

---

Citation

If you use this model, please cite the underlying datasets:

@article{hendrycks2021cuad,
  title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
  author={Hendrycks, Dan and others},
  journal={arXiv preprint arXiv:2103.06268},
  year={2021}
}