failure_report.md

# Failure Report

## Evaluation Results Summary (Template-Based System)

| Task | Contract Type | Position | Total Score | Key Weaknesses |
|------|--------------|----------|-------------|----------------|
| saas_pro_company_001 | SaaS | pro_company | ~0.52 | Placeholder clauses, business constraints not in text |
| nda_balanced_001 | NDA | balanced | ~0.75 | Good coverage, template quality acceptable |
| msa_pro_counterparty_001 | MSA | pro_counterparty | ~0.45 | Missing key terms, playbook compliance low |
| dpa_balanced_001 | DPA | balanced | ~0.70 | Good clause completeness, weak citation support |
| consulting_balanced_001 | Consulting | balanced | ~0.55 | Placeholder clauses hurt invented_terms score |

## Root Causes

### 1. Template Coverage Gaps (Impact: HIGH)
- Only 7 of 9 contract types have template clauses for common provisions
- SaaS-specific clauses (scope_of_services, subscription_term, SLA, assignment) have no template
- Fallback: "[Placeholder clause for X]" — this triggers invented_terms penalty

**Fix**: Add per-contract-type templates for ALL required clauses, not just the 7 common ones

### 2. Business Constraint Injection (Impact: HIGH)  
- Business constraints (e.g., "SOC 2 Type II", "99.9% uptime") are checked via string matching
- Template clauses don't incorporate constraints into generated text
- Score: business_usefulness ≈ 0.0 for most tasks since constraints aren't literally present in clause text

**Fix**: Template generation must inject business_constraints into clause text, not just reference them

### 3. Playbook Compliance Heuristics (Impact: MEDIUM)
- Compliance scoring checks for keywords like "cap", "mutual", "company" in clause text
- Pro-company position: checks for "cap" OR "company" — too permissive
- Pro-counterparty position: checks for "broad" OR "customer" — too narrow

**Fix**: Use structured fallback positions in scoring — check that fallback position terms appear in clause text

### 4. Risk Flag Detection (Impact: MEDIUM)
- Keyword-based detection misses semantic issues
- "NO_DPA" flag fires even when DPA is referenced in text ("DPA may be executed upon request")
- Detection doesn't distinguish between "referenced" and "attached"

**Fix**: Use ContractNLI-based entailment checking instead of keyword matching

### 5. Citation Support (Impact: MEDIUM)
- With seed clause library (~100 clauses), retrieval matches are sparse
- Many clause types (SLA, assignment, subscription_term) have no seed examples
- Score: citation_support ≈ 0.3-0.5

**Fix**: Expand clause corpus with CUAD-extracted clauses and ACORD training data

## Known Limitations

1. **Template-based generation** produces short, generic clauses — need LLM-based generation
2. **Risk flag detection** uses keyword matching, not semantic understanding — need NLI verification
3. **Playbook compliance** scoring is heuristic-based — need structured clause analysis
4. **Business constraints** checked via string matching — need semantic embedding comparison
5. **No LLM-based generation** when running without GPU/transformers
6. **Citation support** depends on retrieval corpus size — currently sparse for uncommon clause types

## Recommended Next Steps

1. Fine-tune Qwen2.5-3B on CUAD + ContractNLI for clause classification and generation
2. Build embedding index over ACORD clause corpus for dense retrieval
3. Add per-contract-type, per-position clause templates covering ALL required clauses
4. Replace keyword-based risk flag detection with NLI-based entailment scoring
5. Add business constraint injection into template/LLM generation prompts
6. Expand eval suite to 20+ tasks across all 9 contract types