reports/Phase10_Post_IVtoOOV_Audit.md

# Phase 10 Benchmark Audit — Post IVtoOOV Removal

> **Date**: 2026-06-24
> **Action**: Removed `IVtoOOV` filter and added advanced `camel-tools` POS tagging for dual/plural noun-adjective agreement.

## 1. Top-Level Aggregate Metrics

| Metric | Score | Notes |
|---|---|---|
| **Overall Pass Rate** | **56.2%** | Massive improvement (previously ~25%) |
| Total Tests | 320 | |
| True Positives (TP) | 95 | Successfully fixed real errors |
| True Negatives (TN) | 85 | Successfully ignored correct text |
| False Positives (FP) | 79 | Hallucinations or overcorrections |
| False Negatives (FN) | 61 | Failed to fix real errors |

### Root Cause Analysis (61 FN + 79 FP = 140 Failures)
- **Punctuation Model (`MODEL:punctuation`)**: 64 failures
- **Integration/Collisions (`PIPELINE:integration`)**: 32 failures
- **Spelling Model (`MODEL:spelling`)**: 21 failures
- **Grammar Model (`MODEL:grammar`)**: 18 failures

---

## 2. Per-Dataset Breakdown

### Grammar Dataset
* **Pass Rate:** 57.8% (up from 26.7%)
* **Recall:** 80.0% (up from 40.0%)
* **Analysis:** Removing `IVtoOOV` successfully unblocked valid grammatical structural changes. The recall doubled.
* **Remaining Issue:** High False Positive Rate on the `correct...` category. The model hallucinates changes on already perfect text.

### Spelling Dataset
* **Pass Rate:** 63.7% (up from 42.5%)
* **Recall:** 79.4%
* **Remaining Issue:** Still missing some Hamza errors and complex word splits (`عندالباب` -> `عند الباب`).

### Structured Content & Religious Datasets
* **Structured Pass Rate:** 82.9% (up from 5.7%)
* **Religious Pass Rate:** 90.0% (up from 10.0%)
* **Analysis:** The `DigitGuard` and punctuation bypass rules are working incredibly well to protect specialized text.

### Pipeline Collision Dataset
* **Pass Rate:** 16.0% (Terrible)
* **False Negative Rate:** 84.0%
* **Analysis:** When a spelling error is adjacent to a grammar error, `StageLocker` is locking the word and preventing the grammar model from seeing or fixing the grammatical context.

### Entities Dataset
* **Pass Rate:** 13.3%
* **Analysis:** The models (especially punctuation and spelling) are aggressively modifying named entities (people, places).

---

## 3. Strategic Action Plan for Enhancements

To push the pass rate from **56.2%** to **>80%**, we must address the following critical areas:

### A. Tame the "StageLocker" (Fix Pipeline Collisions)
The `StageLocker` in `app.py` enforces a rigid "Spelling locks word X, Grammar cannot touch word X" rule. This breaks multi-stage corrections.
**Solution:** Relax the `StageLocker`. Allow the grammar model to operate on tokens that were modified by spelling, provided the grammatical change doesn't completely revert the spelling correction (e.g., checking Jaccard distance or allowing suffix-only changes to locked words).

### B. Stop Punctuation Hallucinations
The punctuation model causes **64 failures**, mostly by adding periods `.` or question marks `؟` to the end of short sentences or entities where they don't belong.
**Solution:** Implement a strict `TerminalPunctuationGuard`. If the original text is < 5 words and doesn't end in punctuation, automatically strip any trailing punctuation added by the model.

### C. Implement Named Entity Recognition (NER) Bypass
Entities (Person names, Cities) are failing at an 86% rate.
**Solution:** Integrate `camel-tools` NER (Named Entity Recognition). Scan the input text for `LOC`, `PERS`, and `ORG`. If a word is an entity, add it to a dynamic whitelist so the Spelling and Grammar models skip it entirely.

### D. Tame Grammar Hallucinations on Correct Text
The grammar model hallucinates on perfectly correct text.
**Solution:** Use a POS-based confidence score. If the grammar model attempts to change a noun into a verb, or completely alters the POS structure of an already valid sentence, reject the change. Alternatively, enforce stricter `Jaccard_05` checks for non-structural changes.