The بال...ون/ان and ل...ون/ان patterns had no blocklist protection.
بالامتحان would become بالامتحين, للإنسان would become للإنسين.
Now all three preposition regexes use the same callback with:
- _PREP_BLOCKLIST check
- ان suffix → skip (almost always root noun form)
Tests: 39 passing (10 new for FIX-33b).
- FIX-35: Narrowed suffix set from {ن,ت,ا,ي,ة,و,ه} to {ن,ت} only.
Only verb conjugation suffixes are blocked. ة/ا/ي/و/ه are legitimate
typo corrections that should not be blocked.
- FIX-37: Raised word threshold from 2 to 4. Prevents noisy period
suggestions while user is still typing short phrases.
- FIX-32b: Removed duplicate .sort() in applyAllSuggestions — already
sorted at line 679, second sort at 695 was redundant.
- Tests updated to match narrowed suffix expectations.
FIX-32b: Add missing reverse sort in applyAllSuggestions
Found during commit audit: the text-based replacement loop iterated
suggestions in original order instead of reverse-sorted by offset.
Without descending sort, applying a fix at offset 5 shifts all text
after it, causing fixes at offset 20+ to corrupt the text.
FIX-37: Rule-based terminal period fallback for punctuation
The punctuation model (PuncAra-v1) often fails to add a period at the
end of longer sentences (10+ words). It only suggests mid-sentence
semicolons which get correctly rejected by PUNC-SAFETY.
FIX-37 adds a rule-based fallback: after punctuation model processing,
if the text still lacks terminal punctuation and has 2+ words, inject
a period suggestion for the last word automatically.
Also fixes FIX-37 indentation (was at 14-space, now correctly at
12-space to run outside the 'if corrected_punc != current_text' block).
FIX-36: Merge punctuation into grammar when they overlap same span
When grammar fixes المعلمون→المعلمين at [18:26] and punctuation
wants المعلمون→المعلمين. at the same [18:26], the overlap resolver
was dropping the punctuation entirely. Now detects that the punctuation
correction is grammar_correction + trailing punct char, and merges them
into a single suggestion: المعلمون→المعلمين.
This preserves both the grammar fix AND the terminal period.
FIX-33: Grammar rules fix_prepositions_advanced was corrupting
الامتحان→الامتحين. The regex blindly replaced ان→ين after prepositions,
but الامتحان is a root noun not a dual. Added blocklist of ~60 common
root-form nouns and skip all ال-prefixed words ending in ان.
FIX-34: Remove auto-re-analysis from Apply All in editor.js.
When spelling and grammar disagree (spelling strips ذهبن→ذهب,
grammar re-adds ذهبوا→ذهبن), the auto-re-analysis creates an
infinite loop of contradictory suggestions.
FIX-35: Spelling no longer strips valid conjugation suffixes.
The insertion fix was accepting ذهبن→ذهب (removing final ن) as
'extra char removed'. But ن is a feminine plural verb suffix.
Now blocks removal of suffix chars (ن,ت,ا,ي,ة,و,ه) from word
endings, preventing spelling from undoing grammar corrections.
FIX-32: Fix word duplication bug in website frontend
Root cause: applyAllSuggestions used a mixed DOM+text fallback approach.
After DOM-based fixes shifted the editor text, the text-based fallback
used stale offsets from the original API response, inserting corrections
at wrong positions. This caused 'المشروعالمشروع.' (word duplication).
Fix:
1. applyAllSuggestions now uses PURE text-based reverse-order replacement.
Since patches are sorted highest-offset-first, each offset is valid
because nothing before it has been modified yet.
2. _isApplyingSuggestion guard now stays active 400ms (past the 300ms
re-analysis timeout) to prevent normalize()/input events from
triggering duplicate analysis calls.
FIX-31: .rstrip() before checking trailing punct regex — model may
output trailing whitespace after hallucinated dots, breaking the
dollar-sign anchor.
Log fix: Grammar and Punctuation service logs were hardcoding '...'
after [:80] truncation even for short texts. This made it appear
that the models were adding ellipsis when they weren't. Now only
appends '...' when text is actually >80 chars.
Phase 14b: Fix spelling punct hallucination + cross-stage overlap
FIX-31: Strip trailing punctuation (... . etc) hallucinated by the
AraSpell model that wasn't in the input. This prevents dot
accumulation across pipeline runs.
PatchSet: Spelling + Punctuation patches from different stages now
always coexist (they're compatible corrections). Previously,
punctuation's higher priority caused spelling patches on
adjacent/overlapping words to be dropped ('فيالمشروع' phantom).
- StageLocker: Grammar now only blocked by spelling locks when >50%
of the grammar diff falls INSIDE the spelling lock. Previously,
any boundary touch (even 1 char) blocked grammar entirely.
This fixes the Collision Dataset bug (32% pass rate) where
spelling locks bled into adjacent words and blocked grammar.
- PatchSet: Relaxed overlap resolution — patches with <50% overlap
of the smaller patch are allowed to coexist. This prevents
punctuation patches from dropping spelling patches on adjacent words.
Both fixes verified with unit tests.
Merge punctuation_rules V1+V2: V2 threshold+fallback, V1 softened exclamation guard for short texts
FIX-28a: Digit guard in bidirectional spelling path
- Numbers (2020, 150) were OOV→IV hallucinated into Arabic words
- Root cause: bidirectional fix had no digit protection
FIX-28b: Prefix-swap guard in bidirectional path
- فبالتالي→وبالتالي leaked through (same stem, prefix swap)
FIX-29: Exclamation mark guard in punctuation rules
- PuncAra added ! to declarative sentences (التاريخ→التاريخ!)
- Now only allows !/؟ when text has exclamatory cue words
FIX-30: Prefix-stripping protection in spelling filter
- وبالمستشفيات→والمستشفيات (stripped ب from compound prefix)
- Blocks corrections that strip compound prefixes (وبال, فبال, etc.)
Adversarial diagnostic: 83.3% pass (30/36), 6 failures identified
Full benchmark: 320 tests across 8 datasets, 55% overall pass rate
Spelling filter improvements:
- Add adjacent character transposition detection (e.g., العصوبات→الصعوبات)
Transpositions have Levenshtein=2 but are a single adjacent swap.
Accepted with dampened confidence (0.6) when OOV→IV.
- Add single character insertion detection (extra letter typed)
Accepted with confidence 0.7 when OOV→IV.
- Add single character deletion detection (missed key)
Accepted with confidence 0.7 when OOV→IV.
- All three new edit types require OOV→IV validation to prevent
false positives (spelling precision remains 1.000).
Bidirectional fix patch creation:
- The bidirectional validation (Phase 12 A5) was silently correcting
text without creating patches, so users never saw the suggestion.
- Now creates spelling patches when rescuing OOV→IV words, making
corrections visible in the UI.
Eager model loading:
- All NLP models (spelling, grammar, punctuation, autocomplete, dialect)
now load at server startup instead of on first request.
- Eliminates cold-start…
- Add _tel_events collector list in analyze_text()
- Append events to _tel_events alongside [FILTER-TEL] logger lines
- Include 'telemetry' key in API response JSON
- Update telemetry_capture.py to read from response instead of HF logs
- NO correction logic changes
Previously Rule 0c only rejected rearrangement (same punct chars in
different positions). Now also rejects substitution (comma→colon,
comma→semicolon) when the alphabetic content is unchanged.
Live tested: Fixes P013 (أحمد، كيف→أحمد كيف:) and H026
(الأمم، وبدونه→الأمم وبدونه؛). P012 and P015 already fixed.
FIX-25: Block grammar punct spacing (حالك؟→حالك ؟)
Grammar model inserts spaces around punctuation marks which is wrong
in Arabic. This affects P012-P015, H026, SC015-SC017 and more.
Traced from full failure analysis: 10+ failures share this pattern.
Added normalize-then-compare filter to catch spacing-only diffs.
FIX-24: Block grammar model punct stripping (البلاد.→البلاد)
Root cause traced from benchmark results: ALL 30 Hallucination FP samples
had the grammar model stripping the sentence-final period. The grammar
model diff was: 'البلاد.'→'البلاد' (type=grammar).
Added filter to block grammar diffs that only differ by punctuation marks
and/or tanween marks. This should restore Hallucination to ~97%.
FIX-23: Block grammar model tanween removal + HF log fetcher
1. Tanween blocker: grammar model strips tanween (جداً→جدا) from correct text.
Added filter to block diffs that only differ by tanween marks.
2. HF log fetcher script for automated tracing.
The regex patterns (ون→ين after prepositions, ون→وا after nasb particles)
were matching correct text everywhere. CamelTools-based rules handle these
with POS-tag awareness. Keep only narrow five-nouns patterns.
CRITICAL FIX: _DIRECTIONAL_BLOCKS was local scope → NameError in grammar diff filter
Root cause found from production logs:
NameError: name '_DIRECTIONAL_BLOCKS' is not defined (line 1633)
_DIRECTIONAL_BLOCKS was defined inside _compute_spelling_confidence() but
referenced in analyze_text()'s grammar diff processing section. This caused
ALL grammar corrections to silently crash and be dropped.
Fix: Move _DIRECTIONAL_BLOCKS to module level so both sections can access it.
Critical: Isolate grammar rules in try/except so regex fallback ALWAYS runs
The camel-tools dependent rules (SV agreement, nasb, gender) were crashing
inside process() and the exception propagated up, causing the regex-based
fallback rules (preposition case, nasb regex, five nouns) to NEVER execute.
Now each rule runs independently - if one fails, the rest still apply.
Hotfix: Revert pronoun-h guard to ته-only (was breaking ta_marbuta), add directional blocks for عمله/لسانه/بيته/كتابه
- Expanded grammar pattern whitelist for SV suffix additions (ذهب→ذهبوا, يذهب→يذهبون)
- Added gender agreement suffix (جميل→جميلة) to whitelist
- Added preposition case (ون→ين after في/من) and nasb (ون→وا after لن/لم) regex fallbacks
- Smart pronoun-ه guard: uses IV check instead of hardcoded consonant list
Root cause: Steps 8-10 in correct() compared edit distance to INPUT text,
which IS the error. The postprocessor's hamza fixes (انا→أنا) were being
reverted because 'انا' is closer to input 'انا' than 'أنا' is.
BUG #1 (P0): Grammar model was stripping trailing periods from correct text.
- Added punctuation-only diff filter in grammar stage
- Eliminates 25/31 false overcorrections
- Grammar and punctuation models no longer fight each other
P1: Expanded hamza whitelist with 30+ missing common words
- Added: اردت, الاطفال, لان, الامتحان, وانا, etc.
- Explicit prefixed forms: وانا→وأنا, فانت→فأنت, etc.
P2: Foreign word protection (CSS→CS destructive fix)
- Skip non-Arabic tokens in spelling filter
P2: Tanween preservation (جميلاً was being stripped to جميلا)
- Skip words ending in tanween from hamza normalization
P3: Space-before-punctuation fix in grammar rules
- Grammar model output حالك ؟ → now حالك؟
P3: Zero-Arabic-content guard
- Numbers-only, emojis-only text no longer gets punctuation added
