failure_report.md

Failure Report

Evaluation Results Summary (Template-Based System)

Task	Contract Type	Position	Total Score	Key Weaknesses
saas_pro_company_001	SaaS	pro_company	~0.52	Placeholder clauses, business constraints not in text
nda_balanced_001	NDA	balanced	~0.75	Good coverage, template quality acceptable
msa_pro_counterparty_001	MSA	pro_counterparty	~0.45	Missing key terms, playbook compliance low
dpa_balanced_001	DPA	balanced	~0.70	Good clause completeness, weak citation support
consulting_balanced_001	Consulting	balanced	~0.55	Placeholder clauses hurt invented_terms score

Root Causes

1. Template Coverage Gaps (Impact: HIGH)

• Only 7 of 9 contract types have template clauses for common provisions
• SaaS-specific clauses (scope_of_services, subscription_term, SLA, assignment) have no template
• Fallback: "[Placeholder clause for X]" — this triggers invented_terms penalty

Fix: Add per-contract-type templates for ALL required clauses, not just the 7 common ones

2. Business Constraint Injection (Impact: HIGH)

• Business constraints (e.g., "SOC 2 Type II", "99.9% uptime") are checked via string matching
• Template clauses don't incorporate constraints into generated text
• Score: business_usefulness ≈ 0.0 for most tasks since constraints aren't literally present in clause text

Fix: Template generation must inject business_constraints into clause text, not just reference them

3. Playbook Compliance Heuristics (Impact: MEDIUM)

• Compliance scoring checks for keywords like "cap", "mutual", "company" in clause text
• Pro-company position: checks for "cap" OR "company" — too permissive
• Pro-counterparty position: checks for "broad" OR "customer" — too narrow

Fix: Use structured fallback positions in scoring — check that fallback position terms appear in clause text

4. Risk Flag Detection (Impact: MEDIUM)

• Keyword-based detection misses semantic issues
• "NO_DPA" flag fires even when DPA is referenced in text ("DPA may be executed upon request")
• Detection doesn't distinguish between "referenced" and "attached"

Fix: Use ContractNLI-based entailment checking instead of keyword matching

5. Citation Support (Impact: MEDIUM)

• With seed clause library (~100 clauses), retrieval matches are sparse
• Many clause types (SLA, assignment, subscription_term) have no seed examples
• Score: citation_support ≈ 0.3-0.5

Fix: Expand clause corpus with CUAD-extracted clauses and ACORD training data

Known Limitations

1. Template-based generation produces short, generic clauses — need LLM-based generation
2. Risk flag detection uses keyword matching, not semantic understanding — need NLI verification
3. Playbook compliance scoring is heuristic-based — need structured clause analysis
4. Business constraints checked via string matching — need semantic embedding comparison
5. No LLM-based generation when running without GPU/transformers
6. Citation support depends on retrieval corpus size — currently sparse for uncommon clause types

Recommended Next Steps

1. Fine-tune Qwen2.5-3B on CUAD + ContractNLI for clause classification and generation
2. Build embedding index over ACORD clause corpus for dense retrieval
3. Add per-contract-type, per-position clause templates covering ALL required clauses
4. Replace keyword-based risk flag detection with NLI-based entailment scoring
5. Add business constraint injection into template/LLM generation prompts
6. Expand eval suite to 20+ tasks across all 9 contract types