Update model card with v11 eval results

README.md

@@ -16,9 +16,9 @@ pipeline_tag: text-ranking
 
 # cuad-cross-encoder-v11
 
-A cross-encoder reranker fine-tuned for **legal clause retrieval** in contract review workflows. Built on top of [`cross-encoder/ms-marco-MiniLM-L-6-v2`](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2) and fine-tuned on a combination of CUAD, ACCORD, LEDGAR, ContractNLI, and EDGAR-sourced contract pairs.
+A cross-encoder reranker fine-tuned for **legal clause retrieval** in contract review workflows. Built on [`cross-encoder/ms-marco-MiniLM-L-6-v2`](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2) and fine-tuned on a combination of CUAD, ACCORD, LEDGAR, ContractNLI, EDGAR-sourced contract pairs, and LLM-generated synthetic pairs targeting known failure patterns.
 
-Deployed as **ONNX INT8** for in-browser inference via [WebAssembly / ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/).
+Deployed as **ONNX INT8** for in-browser inference via [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/).
 
 ---
 
@@ -38,13 +38,18 @@ Deployed as **ONNX INT8** for in-browser inference via [WebAssembly / ONNX Runti
 | ACCORD | 3,931 annotated legal passages | ~6,000 |
 | LEDGAR | SEC EDGAR provisions, 14 labels filtered for JV/NC/IP | ~4,000 |
 | ContractNLI / LegalBench | 14 NLI tasks over contract text | ~3,000 |
-| EDGAR scraped (default) | SC 13D + 8-K NC/IP exhibits, live EDGAR data | ~2,500 |
-| EDGAR JV | 8-K joint venture exhibit filings | ~1,500 |
-| EDGAR Sino-JV | 20-F chapter-format Sino-JV agreements | ~4,272 |
-| Pipeline hard negatives | Clause queries where v9 failed — reranked negatives | 254 |
-| Eval positives | Full-chunk positives extracted from passing eval cases | ~200 |
-
-**Total: ~72,101 training pairs** · **5,612 validation pairs**
+| EDGAR default (re-run) | SC 13D + 8-K NC/IP exhibits | 6,680 pos |
+| EDGAR JV (re-run) | 8-K joint venture exhibit filings | 1,054 pos |
+| EDGAR Sino-JV (re-run) | 20-F chapter-format Sino-JV agreements | ~4,272 |
+| EDGAR NDA *(new v11)* | EX-99 confidentiality exhibits from SC TO-T / 8-K filings | 4,417 pos |
+| Synthetic spinoff-IP *(new v11)* | LLM-labeled pairs for IP spinoff format failures | 294 |
+| Synthetic NDA *(new v11)* | LLM-labeled pairs for NDA section-dominance failures | 267 |
+| Synthetic definitions-bleed *(new v11)* | LLM-labeled pairs for definitions-article bleed failures | 507 |
+| Synthetic Armstrong-IP *(new v11)* | LLM-labeled pairs for irrevocable license confusion | 102 |
+| Eval positives | Full-chunk positives extracted from passing eval cases | 84 |
+| Pipeline hard negatives | Clause queries where prior model failed — reranked negatives | 254 |
+
+**Total: ~72,101 training pairs · 5,612 validation pairs**
 
 Pairs are `(query, positive_chunk, negative_chunk)` triplets. Negatives are a mix of hard negatives (wrong clause from same contract) and random negatives (chunks from other contracts).
 
@@ -61,8 +66,8 @@ Pairs are `(query, positive_chunk, negative_chunk)` triplets. Negatives are a mi
 | Max sequence length | 512 tokens |
 | Warmup steps | 10% of total steps |
 | Loss | Cross-entropy (sentence-transformers `CrossEncoderTrainer`) |
-| Hardware | NVIDIA RTX 3090 / A10 (RunPod) |
-| Training time | ~45–60 min |
+| Hardware | NVIDIA RTX 3090 (RunPod) |
+| Training time | ~50 min |
 
 ---
 
@@ -70,12 +75,12 @@ Pairs are `(query, positive_chunk, negative_chunk)` triplets. Negatives are a mi
 
 Evaluated on a held-out set of 16 contracts across 4 clause domains. Each contract is queried with 3–8 clause-type questions; the top-ranked chunk is scored as **pass** (correct clause returned), **partial** (correct section but wrong chunk boundary), or **fail**.
 
-| Suite | Contracts | Queries | Pass | Partial | Fail |
-|-------|-----------|---------|------|---------|------|
-| Joint Venture | 9 | 51 | 9 (18%) | 26 (51%) | 16 (31%) |
-| Intellectual Property | 4 | 49 | 17 (35%) | 20 (41%) | 12 (24%) |
-| Non-Compete / Non-Solicit | 3 | 13 | 5 (38%) | 8 (62%) | 0 (0%) |
-| NDA | 3 | 19 | 8 (42%) | 9 (47%) | 2 (11%) |
+| Suite | Contracts | Queries | Pass | Partial | Fail | vs v10 |
+|-------|-----------|---------|------|---------|------|--------|
+| Joint Venture | 9 | 51 | 20 (39%) | 15 (29%) | 16 (31%) | +11 pass 🚀 |
+| Intellectual Property | 4 | 49 | 18 (37%) | 18 (37%) | 13 (27%) | +1 pass |
+| Non-Compete / Non-Solicit | 3 | 13 | 6 (46%) | 7 (54%) | 0 (0%) | +1 pass |
+| NDA | 3 | 19 | 9 (47%) | 7 (37%) | 3 (16%) | +1 pass |
 
 **Test contracts (JV):** MightyCell Batteries, BorrowMoney.com, Galera Therapeutics, MINDA IMPCO Technologies, Kiromic Biopharma, Novo Integrated Sciences, Transphorm / Aizu Fujitsu, Valence Technology / Baoding Fengfan, Veoneer
 
@@ -83,6 +88,8 @@ Evaluated on a held-out set of 16 contracts across 4 clause domains. Each contra
 
 **Test contracts (NDA):** Kite Pharma / Gilead Sciences, Fortune Brands / Norcraft Companies, Aspect Medical Systems / Tyco Healthcare
 
+The JV improvement (+11 pass) is driven by new Sino-JV EDGAR data and synthetic definitions-bleed pairs targeting contracts where the model previously returned definitions articles for Governing Law and Non-Compete queries.
+
 ---
 
 ## Usage
@@ -95,7 +102,7 @@ from transformers import AutoTokenizer
 import numpy as np
 
 tokenizer = AutoTokenizer.from_pretrained("datgacon/cuad-cross-encoder-v11")
-session = ort.InferenceSession("model_quantized.onnx")
+session = ort.InferenceSession("onnx/model_quantized.onnx")
 
 query = "What governing law applies to this agreement?"
 passage = "This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware."
@@ -134,6 +141,7 @@ for score, passage in ranked:
 - Partial matches are common at clause-boundary edges — chunk size and overlap in the retrieval pipeline significantly affect results
 - Not a legal advisor — scores indicate retrieval relevance, not legal interpretation
 - Performance on clause types outside the four trained domains (JV, IP, NC, NDA) is untested
+- Token type IDs must be passed explicitly when using ONNX Runtime Web; omitting them collapses score spread
 
 ---