openspec/changes/fix-traced-bugs/design.md

## Context

G.U.I.D.E. routes user complaints through a 4-layer pipeline: Presidio PII redaction → DL models (DomainClassifier + EvidenceNER) → Claude Managed Agent → FastAPI + Gradio UI. Trace analysis over 10 complaint flows surfaced 8 bugs. The fixes are additive and contained — no API contract breaks, no new external services, no schema migrations.

Current state of each broken path:

- **HITL form**: `_parse_hitl_entities` in `ui/app.py` scrapes the agent's reply text with 6 hardcoded regex patterns. Any phrasing variation leaves all form fields blank.
- **Draft tab**: `_extract_draft` requires `---\nSubject:...\n---`; the fallback assigns the full reply string on `"Subject:" in reply`, not just the letter body.
- **prior_contact**: `prior == "Yes"` evaluates to `False` when `prior` is `None` or `""`, silently sending `"prior_contact": false` in the confirmed entity payload even when the user stated they had already contacted the company.
- **Agent confirmation loop**: Rule 4 says "present summary and pause" but does not explicitly prohibit an additional in-chat yes/no prompt before the user reaches the Verify Entities tab.
- **Telecom classifier**: CFPB training data contains zero telecom rows. Synthetic supplement (2,000/class) is outweighed by tens of thousands of CFPB banking rows. SIM-activation and prepaid-recharge scenarios are absent from the 20 telecom templates.
- **NER ORG gap**: Synthetic ORG list covers named banks, telcos, and e-commerce platforms but not generic descriptors ("Indian bank") or newer Indian companies ("Niva Bupa", "Agoda").
- **PII over-redaction**: Presidio's built-in `US_BANK_NUMBER` pattern fires on the numeric segment of `REF-20260501-001`, producing `REF-<US_BANK_NUMBER>-001`. No context filter or deny-list is applied.
- **Tesseract**: `pytesseract` is listed in `requirements.txt` (Python binding) but `tesseract-ocr` (system binary) is not validated at startup. The error surfaces only on the first document upload.

## Goals / Non-Goals

**Goals:**
- UI form populated reliably from structured data, not fragile regex
- `prior_contact` value preserved from agent conversation to confirmed payload
- Draft letter always extracted correctly into the Complaint Draft tab
- Agent proceeds to HITL tab without an extra confirmation turn
- BSNL SIM-activation complaints classified as `telecom`
- "Indian bank" and similar generic ORG phrases extracted by EvidenceNER
- `REF-`, `TRN-`, `TXN-`, `OD-` prefixed IDs not mangled by Presidio
- Tesseract absence surfaced at startup with actionable install instructions

**Non-Goals:**
- Streaming the agent reply to Gradio in real time
- Cloud OCR fallback when Tesseract is absent
- Changing routing logic or the 6-domain taxonomy

## Decisions

### D1 — Agent emits a fenced JSON block at the HITL gate

**Decision**: Add a `<!--ENTITIES:{...}-->` HTML comment block immediately after the Rule 4 numbered summary. The UI extracts this with a single `re.search(r'<!--ENTITIES:(\{.*?\})-->', reply, re.DOTALL)` call and `json.loads` the match.

**Why over regex scraping**: The numbered list is human-readable prose; phrasing varies. A machine-readable comment is invisible to the user, deterministic, and survives any reordering of the summary lines.

**Why HTML comment over fenced JSON block**: A fenced block (` ```json `) is rendered visibly in the Gradio chat. An HTML comment is stripped by the Markdown renderer and is invisible to the user.

**Format**:
```
<!--ENTITIES:{"provider":"Axis Bank","incident_date":"17 June 2026","amount":"₹1000","reference_id":"REF-20260501-001","prior_contact":"Yes","desired_resolution":"Full refund"}-->
```

**Agent prompt change**: Rule 4 gains an explicit instruction to emit this block with the exact key names, immediately after the numbered list, before the tab-navigation instruction.

**UI change**: `_parse_hitl_entities` is replaced by `_parse_hitl_entities_json` which extracts the comment block. The regex fallback is removed entirely — if the comment is absent, the form fields return `_nu()` (no-update), same as today's miss case.

---

### D2 — prior_contact guard

**Decision**: In `_handle_confirm`, change `prior == "Yes"` to `prior == "Yes" if prior else None`, then exclude `None` from the entity dict (same filter already applied to other blank fields). The Verify Entities form's Radio widget initial value is `None`; if the user never touched it and the agent JSON provided `"prior_contact": "Yes"`, that value pre-fills the widget via D1 and the guard is moot. The guard matters only if the JSON block is missing.

---

### D3 — _extract_draft relaxation

**Decision**: Add a third extraction path before the fallback: match `Subject:` through the end of the last non-empty line that follows the letter body (without requiring a closing `---`). The `---` primary path is preserved. The `"Subject:" in reply → full reply` fallback is tightened to extract only the substring from `Subject:` to the end of string, not the entire reply.

---

### D4 — Rule 4/5 prompt tightening

**Decision**: Append to Rule 4: *"Do not ask 'Does this look correct?' or any yes/no follow-up. The numbered summary above IS the confirmation request. The user will confirm in the Verify Entities tab. Your next action after this message is to wait."*  
Remove ambiguity from Rule 5 by replacing "the most recent user message begins with" with "you have received a message that begins with" (eliminates confusion about turn ordering).

---

### D5 — Classifier rebalancing (Option C)

**Decision**: 
1. Add 10 new telecom templates focused on SIM card purchase, activation failure, and prepaid recharge — the exact scenario type from Trace 5.
2. Raise `--supplement_per_class` default from 2,000 to 5,000 in `train.py`.
3. Keep the `--supplement_per_class` CLI flag so it remains overridable.

**Why 5,000**: At 5,000 synthetic samples per class, telecom has 5K rows vs. CFPB banking's ~50K cap. The ratio improves from 1:25 to 1:10 — still imbalanced but within the range where DistilBERT fine-tuning generalises reliably with 3 epochs.

**Requires checkpoint retrain**: `models/domain_classifier/` must be regenerated after this change.

---

### D6 — NER ORG list extension

**Decision**: Add to `_ENTITY_VALUES["ORG"]` in `src/ner/train.py`:
- Generic descriptors: `"Indian bank"`, `"the bank"`, `"my bank"`
- Indian health insurance: `"Niva Bupa"`, `"Care Health Insurance"`, `"Bajaj Allianz"`
- Travel/hospitality OTAs: `"Agoda"`, `"OYO Rooms"`, `"Booking.com India"`
- Fintech: `"Razorpay"`, `"BharatPe"`, `"CRED"`

**Requires checkpoint retrain**: `models/evidence_ner/` must be regenerated after this change.

---

### D7 — Complaint ID deny-list recognizer

**Decision**: Register a new `PatternRecognizer` in `PIIRedactor.__init__` that matches complaint ID patterns (`REF-`, `TRN-`, `TXN-`, `OD-`, `BK-`, `CLM-`, `CASE-`, `CMP-`, `LN-`, `POL-`, `CR-`) with a high-confidence score (0.99) and entity type `COMPLAINT_REF_ID`. This entity type is NOT included in `_ENTITY_TYPES` (the redaction list), so Presidio will detect the spans but the anonymizer will not replace them. This effectively reserves those spans from being claimed by `US_BANK_NUMBER`.

**Why not just remove US_BANK_NUMBER**: Real bank account numbers in complaint text should still be redacted. Removing the recognizer would be a privacy regression.

**Why not a deny-list on US_BANK_NUMBER**: Presidio's `PatternRecognizer` deny_list applies to the matched text value, not a prefix context. A higher-priority overlapping recognizer is the clean approach.

---

### D8 — Tesseract pre-flight in start.py

**Decision**: In `start.py`, before the `uvicorn` server is launched, call `shutil.which("tesseract")`. If `None`, print a formatted error with platform-specific install commands and call `sys.exit(1)`. Document processors that depend on Tesseract are not lazy-loaded until a document upload arrives, so this is the only reliable point to catch the missing binary early.

**Install message format**:
```
[GUIDE] ✗ Tesseract OCR not found. Install it before starting:
  macOS:   brew install tesseract
  Ubuntu:  sudo apt install tesseract-ocr
  Windows: https://github.com/UB-Mannheim/tesseract/wiki
Then re-run: python start.py
```

## Risks / Trade-offs

- **D1 — LLM output determinism**: Claude may occasionally omit the `<!--ENTITIES:-->` block or malform the JSON if the conversation is long and the system prompt is partially evicted from the context window. **Mitigation**: The UI degrades gracefully to blank fields (same as today's miss case); no crash path. Monitor via LangSmith traces.

- **D5 — Retrain required, checkpoint not auto-committed**: `models/domain_classifier/` is gitignored. Developers who pull this change must retrain locally. **Mitigation**: `start.py` auto-trains missing checkpoints by default; `--no-train` flag bypasses this. Document the retrain step in the task list.

- **D7 — New entity type in Presidio registry**: `COMPLAINT_REF_ID` is added to the analyzer registry but not to `_ENTITY_TYPES`. If a future developer adds it to `_ENTITY_TYPES` by mistake, complaint IDs will be redacted. **Mitigation**: Add a comment in `redactor.py` explaining this intentional exclusion.

## Migration Plan

1. Apply code changes (D1–D4, D7, D8) — no model changes needed, safe to deploy immediately.
2. Apply training data changes (D5, D6) and retrain both checkpoints locally.
3. Verify retrained models with the 10 trace inputs before replacing production checkpoints.
4. Replace `models/domain_classifier/` and `models/evidence_ner/` checkpoints.
5. Restart servers (`python start.py --no-train`).

Rollback: restore prior checkpoint directories; no database or schema rollback needed.

## Open Questions

- Should the `<!--ENTITIES:-->` parsing have a JSON-schema validation step (reject malformed JSON silently vs. raise)? Currently: silent `_nu()` on parse error.
- Should `supplement_per_class=5000` become the new hardcoded default in `_build_supplement()`, or only in the CLI `--supplement_per_class` default? Keeping it only as a CLI default means `start.py --train` uses the new value but programmatic calls to `_build_supplement()` still use 2,000.