openspec/changes/fix-traced-bugs/design.md

Context

G.U.I.D.E. routes user complaints through a 4-layer pipeline: Presidio PII redaction → DL models (DomainClassifier + EvidenceNER) → Claude Managed Agent → FastAPI + Gradio UI. Trace analysis over 10 complaint flows surfaced 8 bugs. The fixes are additive and contained — no API contract breaks, no new external services, no schema migrations.

Current state of each broken path:

• HITL form: _parse_hitl_entities in ui/app.py scrapes the agent's reply text with 6 hardcoded regex patterns. Any phrasing variation leaves all form fields blank.
• Draft tab: _extract_draft requires ---\nSubject:...\n---; the fallback assigns the full reply string on "Subject:" in reply, not just the letter body.
• prior_contact: prior == "Yes" evaluates to False when prior is None or "", silently sending "prior_contact": false in the confirmed entity payload even when the user stated they had already contacted the company.
• Agent confirmation loop: Rule 4 says "present summary and pause" but does not explicitly prohibit an additional in-chat yes/no prompt before the user reaches the Verify Entities tab.
• Telecom classifier: CFPB training data contains zero telecom rows. Synthetic supplement (2,000/class) is outweighed by tens of thousands of CFPB banking rows. SIM-activation and prepaid-recharge scenarios are absent from the 20 telecom templates.
• NER ORG gap: Synthetic ORG list covers named banks, telcos, and e-commerce platforms but not generic descriptors ("Indian bank") or newer Indian companies ("Niva Bupa", "Agoda").
• PII over-redaction: Presidio's built-in US_BANK_NUMBER pattern fires on the numeric segment of REF-20260501-001, producing REF-<US_BANK_NUMBER>-001. No context filter or deny-list is applied.
• Tesseract: pytesseract is listed in requirements.txt (Python binding) but tesseract-ocr (system binary) is not validated at startup. The error surfaces only on the first document upload.

Goals / Non-Goals

Goals:

• UI form populated reliably from structured data, not fragile regex
• prior_contact value preserved from agent conversation to confirmed payload
• Draft letter always extracted correctly into the Complaint Draft tab
• Agent proceeds to HITL tab without an extra confirmation turn
• BSNL SIM-activation complaints classified as telecom
• "Indian bank" and similar generic ORG phrases extracted by EvidenceNER
• REF-, TRN-, TXN-, OD- prefixed IDs not mangled by Presidio
• Tesseract absence surfaced at startup with actionable install instructions

Non-Goals:

• Streaming the agent reply to Gradio in real time
• Cloud OCR fallback when Tesseract is absent
• Changing routing logic or the 6-domain taxonomy

Decisions

D1 — Agent emits a fenced JSON block at the HITL gate

Decision: Add a <!--ENTITIES:{...}--> HTML comment block immediately after the Rule 4 numbered summary. The UI extracts this with a single re.search(r'<!--ENTITIES:(\{.*?\})-->', reply, re.DOTALL) call and json.loads the match.

Why over regex scraping: The numbered list is human-readable prose; phrasing varies. A machine-readable comment is invisible to the user, deterministic, and survives any reordering of the summary lines.

Why HTML comment over fenced JSON block: A fenced block (```json) is rendered visibly in the Gradio chat. An HTML comment is stripped by the Markdown renderer and is invisible to the user.

Format:

<!--ENTITIES:{"provider":"Axis Bank","incident_date":"17 June 2026","amount":"₹1000","reference_id":"REF-20260501-001","prior_contact":"Yes","desired_resolution":"Full refund"}-->

Agent prompt change: Rule 4 gains an explicit instruction to emit this block with the exact key names, immediately after the numbered list, before the tab-navigation instruction.

UI change: _parse_hitl_entities is replaced by _parse_hitl_entities_json which extracts the comment block. The regex fallback is removed entirely — if the comment is absent, the form fields return _nu() (no-update), same as today's miss case.

---

D2 — prior_contact guard

Decision: In _handle_confirm, change prior == "Yes" to prior == "Yes" if prior else None, then exclude None from the entity dict (same filter already applied to other blank fields). The Verify Entities form's Radio widget initial value is None; if the user never touched it and the agent JSON provided "prior_contact": "Yes", that value pre-fills the widget via D1 and the guard is moot. The guard matters only if the JSON block is missing.

---

D3 — _extract_draft relaxation

Decision: Add a third extraction path before the fallback: match Subject: through the end of the last non-empty line that follows the letter body (without requiring a closing ---). The --- primary path is preserved. The "Subject:" in reply → full reply fallback is tightened to extract only the substring from Subject: to the end of string, not the entire reply.

---

D4 — Rule 4/5 prompt tightening

Decision: Append to Rule 4: "Do not ask 'Does this look correct?' or any yes/no follow-up. The numbered summary above IS the confirmation request. The user will confirm in the Verify Entities tab. Your next action after this message is to wait."
Remove ambiguity from Rule 5 by replacing "the most recent user message begins with" with "you have received a message that begins with" (eliminates confusion about turn ordering).

---

D5 — Classifier rebalancing (Option C)

Decision:

1. Add 10 new telecom templates focused on SIM card purchase, activation failure, and prepaid recharge — the exact scenario type from Trace 5.
2. Raise --supplement_per_class default from 2,000 to 5,000 in train.py.
3. Keep the --supplement_per_class CLI flag so it remains overridable.

Why 5,000: At 5,000 synthetic samples per class, telecom has 5K rows vs. CFPB banking's ~50K cap. The ratio improves from 1:25 to 1:10 — still imbalanced but within the range where DistilBERT fine-tuning generalises reliably with 3 epochs.

Requires checkpoint retrain: models/domain_classifier/ must be regenerated after this change.

---

D6 — NER ORG list extension

Decision: Add to _ENTITY_VALUES["ORG"] in src/ner/train.py:

• Generic descriptors: "Indian bank", "the bank", "my bank"
• Indian health insurance: "Niva Bupa", "Care Health Insurance", "Bajaj Allianz"
• Travel/hospitality OTAs: "Agoda", "OYO Rooms", "Booking.com India"
• Fintech: "Razorpay", "BharatPe", "CRED"

Requires checkpoint retrain: models/evidence_ner/ must be regenerated after this change.

---

D7 — Complaint ID deny-list recognizer

Decision: Register a new PatternRecognizer in PIIRedactor.__init__ that matches complaint ID patterns (REF-, TRN-, TXN-, OD-, BK-, CLM-, CASE-, CMP-, LN-, POL-, CR-) with a high-confidence score (0.99) and entity type COMPLAINT_REF_ID. This entity type is NOT included in _ENTITY_TYPES (the redaction list), so Presidio will detect the spans but the anonymizer will not replace them. This effectively reserves those spans from being claimed by US_BANK_NUMBER.

Why not just remove US_BANK_NUMBER: Real bank account numbers in complaint text should still be redacted. Removing the recognizer would be a privacy regression.

Why not a deny-list on US_BANK_NUMBER: Presidio's PatternRecognizer deny_list applies to the matched text value, not a prefix context. A higher-priority overlapping recognizer is the clean approach.

---

D8 — Tesseract pre-flight in start.py

Decision: In start.py, before the uvicorn server is launched, call shutil.which("tesseract"). If None, print a formatted error with platform-specific install commands and call sys.exit(1). Document processors that depend on Tesseract are not lazy-loaded until a document upload arrives, so this is the only reliable point to catch the missing binary early.

Install message format:

[GUIDE] ✗ Tesseract OCR not found. Install it before starting:
  macOS:   brew install tesseract
  Ubuntu:  sudo apt install tesseract-ocr
  Windows: https://github.com/UB-Mannheim/tesseract/wiki
Then re-run: python start.py

Risks / Trade-offs

• D1 — LLM output determinism: Claude may occasionally omit the <!--ENTITIES:--> block or malform the JSON if the conversation is long and the system prompt is partially evicted from the context window. Mitigation: The UI degrades gracefully to blank fields (same as today's miss case); no crash path. Monitor via LangSmith traces.

• D5 — Retrain required, checkpoint not auto-committed: models/domain_classifier/ is gitignored. Developers who pull this change must retrain locally. Mitigation: start.py auto-trains missing checkpoints by default; --no-train flag bypasses this. Document the retrain step in the task list.

• D7 — New entity type in Presidio registry: COMPLAINT_REF_ID is added to the analyzer registry but not to _ENTITY_TYPES. If a future developer adds it to _ENTITY_TYPES by mistake, complaint IDs will be redacted. Mitigation: Add a comment in redactor.py explaining this intentional exclusion.

Migration Plan

1. Apply code changes (D1–D4, D7, D8) — no model changes needed, safe to deploy immediately.
2. Apply training data changes (D5, D6) and retrain both checkpoints locally.
3. Verify retrained models with the 10 trace inputs before replacing production checkpoints.
4. Replace models/domain_classifier/ and models/evidence_ner/ checkpoints.
5. Restart servers (python start.py --no-train).

Rollback: restore prior checkpoint directories; no database or schema rollback needed.

Open Questions

• Should the <!--ENTITIES:--> parsing have a JSON-schema validation step (reject malformed JSON silently vs. raise)? Currently: silent _nu() on parse error.
• Should supplement_per_class=5000 become the new hardcoded default in _build_supplement(), or only in the CLI --supplement_per_class default? Keeping it only as a CLI default means start.py --train uses the new value but programmatic calls to _build_supplement() still use 2,000.