Grammar removes period, punctuation re-adds it. Auto re-analyze after apply
creates an infinite cycle. Analysis already auto-triggers when user types
(input event handler), which is the correct behavior.
fix: prevent analyze loop - skip redundant API calls when text unchanged
P1: Integrate fix_common_hamza into /api/analyze pipeline
- Standalone hamza-fix pass runs AFTER spelling, catches انت→أنت,
الان→الآن, ام→أم, انا→أنا in sentence context
- Works even when AraSpell is skipped (>300 chars)
P1: Split multi-word grammar diffs into individual suggestions
- Grammar corrections like 'الي المدرسه الاستاذ'→'إلى المدرسة الأستاذ'
now appear as 3 separate suggestions, not 1 merged block
- Users can accept/reject each correction individually
P2: Auto re-analyze after applying any suggestion (frontend)
- After applying single, alternative, or all suggestions,
editor automatically re-analyzes with 300ms delay
- Shows 'جار التحليل' without user needing to type anything
P3: Word truncation guard
- Reject spelling corrections that shorten valid words by 2+ chars
- Prevents الدمث→الدم (rare word truncation)
P3: CI/CD - regression tests on every push
- Added test job to GitHub Actions workflow
- Runs 120 tests before lint-and-validate
Added is_locked_by() to StageLocker that returns (start, end, owner).
Logs now show: locked by spelling[59:63] or locked by grammar[73:81]
This helps identify WHY punctuation marks get blocked.
fix(punctuation): allow pure punctuation changes through stage lock
BEFORE: Spelling locks 'بشدة' → punctuation can't add period after it
AFTER: If diff only adds/moves punctuation marks (Arabic text unchanged),
allow it through the lock even on locked ranges
Lock still blocks any diff that changes actual Arabic words.
fix(autocomplete): only trigger after space - no partial words
- Added self.threshold=0.05 — suggestions with score < 0.05 are filtered
- Changed default n from 5 to 3 everywhere (service, API, frontend)
- Threshold applied in _bigram_predict, _hybrid_predict, and GPT-2 fallback
fix(autocomplete): replace beam search with sampling for diverse GPT-2 predictions
BEFORE: num_beams=10 → beam collapse → same 1-3 words
AFTER: do_sample=True, top_k=50, top_p=0.9, temp=0.8, 15 sequences
→ frequency scoring → 8-15 diverse contextual words
Score = how many of 15 samples picked this word (frequency-based)
fix(autocomplete): GPT-2 now generates complete words, not BPE subword tokens
BEFORE: GPT-2 predicted next BPE token → 'ال' instead of 'القاهرة'
AFTER: GPT-2 uses generate() with beam search → complete Arabic words
Changes:
- model.generate(max_new_tokens=6, num_beams=10, num_return_sequences=10)
- Extract first complete Arabic word from each generated sequence
- alpha changed from 0.6 bigram / 0.4 gpt2 → 0.4 / 0.6 (GPT-2 has context!)
- GPT-2 bonus candidates weight increased from 0.3 to 0.5
- Added [GPT2] log line showing generated words for debugging
fix(autocomplete): true context awareness - GPT-2 now adds own candidates + cache key uses 5 words
COMPLETELY INDEPENDENT from the correction pipeline. Zero interaction
with /api/analyze, StageLocker, OffsetMapper, PatchSet, or highlights.
Backend:
- src/nlp/autocomplete/ package (service, rules, init)
- HybridAutoComplete: lazy-loaded singleton, LRU cache, OOM fallback
- Bigram model from bayan10/AutoComplete (bigram_model_v4.pkl)
- GPT-2 from aubmindlab/aragpt2-base (optional, falls back to bigram)
- POST /api/autocomplete with {context, n} → {status, suggestions}
- /api/health now reports autocomplete status
Frontend:
- src/js/autocomplete.js — self-contained module
- Ghost text overlay (gray, semi-transparent, pointer-events:none)
- Dropdown with keyboard nav (↑/↓), TAB accept, ESC dismiss
- 400ms debounce, min 3 chars, cursor-aware context extraction
- CSS in components.css with dark/light theme support
59/59 existing pipeline tests passing
fix(critical): stop spelling from corrupting correct words + fix pronoun agreement + reject hallucinations
3 critical fixes:
1. IV PROTECTION: When both original AND correction are in-vocabulary (valid
Arabic words), ONLY accept the change if it's a known orthographic fix
(hamza whitelist or ه→ة). This blocks وكان→وكأن type corruption where
the model changes one correct word to a completely different correct word.
2. PRONOUN EXCLUSION: fix_subject_verb_agreement now excludes pronouns
(أنا, أنت, هو, etc.) from triggering plural verb agreement. Previously
it incorrectly changed أنا ذهبت → أنا ذهبتوا.
3. HALLUCINATION FILTER: Grammar diffs with Jaccard char similarity <0.3
are rejected (e.g. جالس→جاكسون). Prevents model hallucinations from
reaching the user.
Also adds [SPELLING] Accepted/Rejected debug logging for production tracing.
59/59 tests passing
fix: re-label grammar spelling fixes + handle prefix in hamza whitelist
S1 (P0): WordAligner now prefers ة over ه at word end when both IV
- araspell_rules.py _select_best_word: ه→ة preference for feminine nouns
S2 (P0): Gender preservation — reject corrections that drop feminine marker
- app.py _is_small_spelling_change: block بارده→بارد, منخفظه→منخفض
S3 (P1): Hamza whitelist — 50+ common Arabic hamza corrections
- araspell_rules.py HAMZA_WHITELIST + fix_common_hamza()
- Fixes: الي→إلى, انت→أنت, لان→لأن, امس→أمس, الايام→الأيام, etc.
G1 (P1): Verb-subject agreement for SVO word order
- grammar_rules.py fix_subject_verb_agreement()
- Handles: الطلاب ذهب→ذهبوا, الطالبات ذهب→ذهبن
P1 (P2): Punctuation model now only adds marks, no spelling/grammar changes
- punctuation_service.py _strip_non_punctuation_changes()
- Reverts PuncAra's baked-in spelling/grammar corrections, keeps only marks
S4 (P2): Mitigated by S1+S2+S3 — spelling now makes better corrections,
fewer bad locks blocking grammar
31/31 tests passing
- Grammar service: transient errors (rate limiting, timeout) no longer cached permanently — next request will retry loading instead of failing forever
- Editor: cursor now placed right after corrected text instead of jumping to start — applied to all 3 code paths (UUID span, text-match fallback, offset fallback)
fix: Green traffic light (fullscreen) now works - Wrong selector: .editor-main-layout → .editor-layout - Grid override: grid-template-columns: 1fr !important - Hide both docs-panel and sidebar-desktop
fix: New document starts empty instead of inheriting old content Previously createDocument() used getEditorText() to save current content into the new doc. Now it creates with empty string and calls loadDocumentText('') to clear the editor.
fix: Strip HTML formatting on paste into editor - Added paste event handler that converts clipboard to plain text - Prevents rich text styling (colors, opacity, fonts) from being carried over - Preserves paragraph structure with line breaks - Fixes ghost formatting when pasting from chat or web pages
fix: Re-index suggestion spans after dismiss (keep as-is) - After filtering dismissed suggestion from array, span data-suggestion-id indices became stale - Adjacent error words became unclickable because findSuggestionById used wrong index - Now re-indexes all remaining spans to match updated array positions
fix: Grammar rules false positive - الامتحان was incorrectly matched - Increased minimum stem length from 2 to 4 in fix_prepositions_advanced - Prevents root-level ان endings (الامتحان, الإنسان) from being treated as grammatical ون/ان
fix: Deduplicate spelling/grammar overlap - grammar wins - When spelling and grammar both flag the same word position, keep only grammar suggestion - Prevents double-highlighting on conjugation errors like يعملوا
fix: Remove double-inversion of summary length - Browser already reverses range slider in RTL - Sending raw value now matches correctly: طويل→3, قصير→1
fix: Invert summary length slider value for RTL - RTL slider: min=1 is RIGHT (طويل), max=3 is LEFT (قصير) - API expects: 1=short, 3=long - Fix: send (4 - value) to API so طويل→3 (long), قصير→1 (short)
feat: Summary UI improvements + PDF fix - Swap short/long labels on summary length slider (RTL: long left, short right) - Remove 'تلخيص النص كاملاً' checkbox and button - Replace single export button with dropdown (TXT/DOCX/PDF) in summary - Hide format toolbar when on summary tab, show on writing tab - Fix PDF export freeze: use foreignObjectRendering:false + lower scale - Add loading toast during PDF export - Add summary DOCX and PDF export functions
fix: Undo/Redo inside editor only + auto-focus after corrections - Moved listener back to editor element (capture:true) as user requested - Added editor.focus() after applySuggestionAtOffsets (line 432) - Added editor.focus() after applyAlternativeCorrection (line 474) - Added editor.focus() after dismissSuggestion (line 518) - After any correction/dismiss, editor auto-focuses so Ctrl+Z works immediately
fix: Undo/Redo now works after spelling corrections - Moved keydown listener from editor to document with capture:true - Uses stopImmediatePropagation to prevent browser native undo - Auto-focuses editor after undo/redo - Works even when tooltip click steals focus from editor
fix: Use capture phase on editor for Ctrl+Z/Y to intercept before browser native undo - Listener now on editor element with capture:true instead of document - Added stopPropagation to prevent browser native undo from also firing - Ensures corrections/dismissals can be properly undone
feat: Add custom Undo/Redo for corrections and dismissals - Ctrl+Z undoes correction/dismiss (restores previous editor state) - Ctrl+Y or Ctrl+Shift+Z redoes - pushUndoState() called before every applyCorrection, applyAlternative, dismissSuggestion, applyAll - 50-level undo stack - Only intercepts when custom stack has entries (falls through to browser native otherwise)
