NEW PIPELINE STEP after spelling, before grammar:
1. Trailing و removal (from legacy AraSpell):
- المصنعو→المصنع, الماهرينوومن→الماهرينوومن
- Catches PC004, PC008, PC010 benchmark failures
2. Edit-distance-1 OOV→IV correction:
- For remaining OOV words, find closest IV word in BERT vocab
- Only replaces when edit-1 candidate exists and first letter matches
- Catches: صممو→صمموا (PC001), حضرو→حضروا (PC042)
Also adds contextual_corrector.py module (MLM-based validation).
Tests: 39 passing.
Fix: TerminalPunctuationGuard for short phrases and comprehensive Nasb/Jazm grammar rules fixes
Root cause: Steps 8-10 in correct() compared edit distance to INPUT text,
which IS the error. The postprocessor's hamza fixes (انا→أنا) were being
reverted because 'انا' is closer to input 'انا' than 'أنا' is.
BUG #1 (P0): Grammar model was stripping trailing periods from correct text.
- Added punctuation-only diff filter in grammar stage
- Eliminates 25/31 false overcorrections
- Grammar and punctuation models no longer fight each other
P1: Expanded hamza whitelist with 30+ missing common words
- Added: اردت, الاطفال, لان, الامتحان, وانا, etc.
- Explicit prefixed forms: وانا→وأنا, فانت→فأنت, etc.
P2: Foreign word protection (CSS→CS destructive fix)
- Skip non-Arabic tokens in spelling filter
P2: Tanween preservation (جميلاً was being stripped to جميلا)
- Skip words ending in tanween from hamza normalization
P3: Space-before-punctuation fix in grammar rules
- Grammar model output حالك ؟ → now حالك؟
P3: Zero-Arabic-content guard
- Numbers-only, emojis-only text no longer gets punctuation added
revert: restore backend/model files to cf07939 state, keep UI/UX changes only
S1 (P0): WordAligner now prefers ة over ه at word end when both IV
- araspell_rules.py _select_best_word: ه→ة preference for feminine nouns
S2 (P0): Gender preservation — reject corrections that drop feminine marker
- app.py _is_small_spelling_change: block بارده→بارد, منخفظه→منخفض
S3 (P1): Hamza whitelist — 50+ common Arabic hamza corrections
- araspell_rules.py HAMZA_WHITELIST + fix_common_hamza()
- Fixes: الي→إلى, انت→أنت, لان→لأن, امس→أمس, الايام→الأيام, etc.
G1 (P1): Verb-subject agreement for SVO word order
- grammar_rules.py fix_subject_verb_agreement()
- Handles: الطلاب ذهب→ذهبوا, الطالبات ذهب→ذهبن
P1 (P2): Punctuation model now only adds marks, no spelling/grammar changes
- punctuation_service.py _strip_non_punctuation_changes()
- Reverts PuncAra's baked-in spelling/grammar corrections, keeps only marks
S4 (P2): Mitigated by S1+S2+S3 — spelling now makes better corrections,
fewer bad locks blocking grammar
31/31 tests passing
fix: PDF dropdown visible + verify spelling context + araspell audit - Fix summary export dropdown clipping (overflow: visible + opens upward) - Verify: use_contextual=True in araspell_service.py (line 83) ✓ - Verify: ContextualCorrector fully initialized and used in pipeline ✓ - Verify: All araspell_rules.py classes used: AraSpellPostProcessor, ErrorClassifier, RulesBasedCorrector, OutputValidator, VocabularyManager, WordAligner, SplitMergeSpecialist, EditDistanceCorrector, ContextualCorrector, ArabicSpellChecker ✓ - Fix misleading comment about contextual=False
feat: Add AraSpell spelling correction endpoint (POST /api/spelling) with lazy model loading - Extract AraSpell pipeline into src/nlp/spelling/ (araspell_rules.py + araspell_service.py) - Pre-download AraBERT encoder-decoder + checkpoint in Dockerfile - Lazy load on first request, contextual=False to save RAM - Update health check to report spelling status