docs/scoring-dialects-plan.md

Plan: dialect acceptance and Hebrew-speaker rules in the scorer

Status: implemented 2026-10-15, except where "Implementation notes" at the end says otherwise. Source rules: pronunciation-rules.md.

Two layers on top of today's scoring path:

• Layer A, dialect acceptance: decides what counts as correct. Runs inside GOP: every correct realisation of a letter is pooled into one class, and members of the class never count as competitors.
• Layer B, Hebrew-speaker rules: decides how strictly to judge real errors. Runs after GOP: per-pair margins, severity, feedback, exemptions.

The native baseline and the teacher calibration floors stay where they are, but are keyed on the abstract symbol instead of the sound actually said.

Final order of the pipeline:

1. Vocab.merge pools spelling variants (exists)
2. Layer A: class columns → alignment → GOP against the class
3. Layer B exemptions (6.3)
4. Native baseline normalisation (exists, keyed on the abstract symbol)
5. Layer B margins, severity, feedback
6. Teacher DB floors (exists, last, manual override)

Phase 0: Rule loader (app/rules.py, new)

• Pull the YAML blocks out of pronunciation-rules.md (fenced ```yaml, split on ---). Load them into:
  • archiphonemes, dialects_enabled, optional_realisations_enabled
  • scoring (margins, floors)
  • checks (target, heard, severity, feedback), collected from every rule block
  • heard_labels
• Pass every symbol through base() when loading, so ʁ is stored as ɣ, χ as x, and so on. Warn about rules that become undetectable:
  • target and heard end up the same sound after base();
  • a heard symbol is in the same accepted set as its target;
  • a symbol is missing from the model vocab.
• Add pyyaml to requirements.txt. It's already installed through huggingface_hub, but it should be listed explicitly.
• Consider moving the rules file to docs/ as the doc itself suggests. The student sections are served from static/assets/, so either keep it there or copy it at build time.
• Tests: every block parses; every heard: symbol has a label in section 8; the undetectable-rule check fires on a deliberately bad rule.

Phase 1: Layer A core, with Q only

1a. Converting the phrases at load time, not in the source files

app/workbook.py has ~930 ق and app/grammar.py has more, so rewriting the IPA by hand isn't realistic.

• Add archify(ar, ipa) in a new app/archi.py. For each word whose Arabic contains ق, replace the matching ʔ/q/k with Q.
  • Match by consonant order. word_letters in app/translit.py already lines up ق with its phoneme, so reuse that logic.
• Ambiguous words (both ء and ق, or ك next to ق) need an explicit marker written in the phrase. Add a test that fails on any ق word it can't resolve.
• After conversion, merge duplicate variants. Today's separate q and k variants (e.g. i l q u d s) become the same as the ʔ one.
• Run it inside PHRASES / workbook_phrases() so every consumer sees Q, including references and tokenisation at startup in load_scorer.

1b. Vocab classes (app/scorer.py, Vocab)

• Vocab(token_to_id, classes=rules.archiphonemes) creates one virtual id per abstract symbol, numbered after the real vocabulary (V, V+1, …). It also keeps members[vid], the real representative ids of the accepted realisations.
• merge() returns logp with extra columns: column vid = logaddexp over its members.
• resolve("Q") returns [vid]. i2t[vid] = "Q".
• Anything that ranges over "real" symbols has to ignore virtual columns: greedy_decode, _heard_hebrew, the heatmap's "heard" track, and edit_distance for the unrelated check. Easiest is a vocab.real(logp) view of logp[:, :V].

1c. GOP (app/align.py, score_tokens)

• score_tokens(..., accept: list[set[int]] | None). For a class token, mask all its members from the competitor search, and always mask every virtual column.
• Add TokenScore.realised: the member with the highest peak inside the window (None for plain tokens).
• The alignment itself doesn't change. It already runs on the virtual column.

1d. Results and display

• Each phoneme entry gets realised. heard comes from the competitor as before, so an accepted realisation never shows up as "heard".
• _heard_hebrew: when the frame is inside a class token's span and the greedy symbol is one of its members, write the realised letter (כּ, א or ק) uncoloured.
• to_hebrew / word_letters: Q is written ק in the answer (today's ق rule already does this).
• app/tips.py: add Q to NO_DELETION_TIP (6.3 #2). Remove the ʔ>k qaf tip.

1e. Baseline and calibrations

• References need no change. Reference.sounds now holds (w, "Q"), so the baseline is the same whatever realisation the TTS voice used. The cache is in memory and rebuilds on restart.
• Calibration lookup in score_logp: try (archi, heard) first, then fall back to (realised target, heard) so existing DB rows still apply. Add a teacher-panel note or a one-off SQL to re-key ʔ rows that came from ق phrases.

1f. Tests (tests/test_scoring.py)

• Rewrite test_k_for_urban_qaf_picks_closest_variant_and_tips: k for Q scores ≥ 90, has no tip, and realised == "k".
• New tests:
  • q for Q is accepted;
  • Q dropped between vowels is not flagged;
  • ʔ from ء said as k is flagged (e.g. اهلا → kahla);
  • a reference voiced with q doesn't penalise a learner who says ʔ;
  • archify on a sample of workbook phrases, plus the ambiguity test.

Phase 2: The other five abstract symbols

• Extend archify: ك→K, ث→TH, ذ→DH, ظ→ZH, ج→J.
• Per-word override syntax for learned words, e.g. TH{t,s} (ثقافة) or a plain t to lock it. Parse it in tokenize, which creates a one-off class for that position.
• Vocab gaps: if θ, ð, ðˤ or tʃ aren't in the model, the class quietly shrinks. Log it, and expose it in /api/health under rules (section 7 of the rules doc).
• Add tʃ (צ'), θ (ת'), ð (ד״) to CONSONANTS in app/translit.py so rural realisations can be written in the heard line.
• Add θ, ð, tʃ, ðˤ, dʒ to the test vocab SYMS. Tests for each class, plus one mixed-dialect sentence that isn't penalised.

Phase 3: Positional classes and diphthongs

• E for a final ة/ه vowel → {a, e, i}. Derived automatically from the Arabic ending (word_letters already detects it).
• O for the vowel of a pronoun ending → {o, u}. This one needs an explicit marker in the phrase, because it can't be told reliably from the Arabic.
• EY / OW: expand into extra variants (eː | a j, oː | a w) inside archify, before variants are merged. _fit already picks the best variant. These can't be a per-token class, because one token stands against two.
• near_vowels stays for other positions until the vowel-length decision is made (see "Open decisions").
• Sounds that natives also drop (ʔ at the start or between vowels, il- after a vowel) are not Layer A: an optional token would need a separate alignment with and without it. They stay as Layer B exemptions.

Phase 4: Layer B, the rule engine

4a. Competitor set

• Limit competitors to the Arabic sounds plus the Hebrew-interference sounds (every heard: symbol in checks). Report any other winner as competitor=None, other=True ("not clear").
• Watch out: this lowers comp_peak in the deletion test (deleted_comp). Re-check the deletion tests.

4b. Flagging by margin

• In score_logp, flag a sound when score < flag_below and the competitor is outside the target's accepted set and −gop ≥ margin(target, heard). gop is already log P(target) − log P(competitor).
• The issues list keeps its threshold (issue_below) but also respects the margin.

4c. Feedback

• tip_for looks up checks[(target, heard)] first, then falls back to TIPS. Keep the key format expected>heard so the front end doesn't change.
• Sort tips by severity, then by score. max_tips comes from the rules file.

4d. Exemptions (6.3)

• Move NO_DELETION_TIP, elided_article and near_vowels into a predicate registry in rules.py: (name, applies(toks, i, ar_words, sc) -> bool, action).
• Add the doc's cases 1–3, 4 (il- not assimilated) and 12 (mixed_dialect as info only).

4e. Letters that can only be heard, never expected

• A HEARD_ONLY table in translit.py: p→פּ, v→ב/ו, ts→צ. Include it in HEBREW_SOUNDS so _heard_hebrew stops dropping these sounds. Today a student who says pinjaan sees the first letter missing instead of פּ.
• sound_name / letter_reason labels come from heard_labels.
• Tests: f→p (פִנְג'אן), w→v, b→v, sˤ→ts, r→ʁ. Emphatic just under the margin is not flagged, just over it is.

Phase 5: Teacher-facing parts

• dialects_enabled / optional_realisations_enabled as teacher settings, stored wherever teacher settings live today. Pass them into Scorer.score per request, so different classes can differ.
• The taught dialect only changes hints and info, never what's accepted.
• Teacher panel: show realised and mixed_dialect in the diagnostics view. Show abstract symbols by their Hebrew letter, never as "Q".
• /api/health: a rules field listing each rule and whether it can be detected.

Phase 6: Rollout check

• If recordings are stored, add a scripts/rescore.py that compares old and new scores per phrase. Review the biggest changes before switching over, especially ק phrases and emphatics.
• Ship Phase 1 on its own: it fixes ק/א and tests the class mechanism end to end.

Open decisions (outside this plan)

1. Vowel length: the doc's vowel_length checks (severity 1, margin 1.1) vs. today's near_vowel_floor, which forgives length completely.
2. Sentence formula: the doc's mean(word_min) and severity weights vs. today's 0.5/0.5 blend. Keep it behind a config flag until decided.
3. Bedouin g for Q: on or off by default.
4. Extra sounds: inserted sounds aren't detected, since forced alignment only scores expected sounds.

Main risks

• The virtual columns change the logp width. Anything that indexes logp by a real id is fine, but anything that iterates over all columns (heatmap, greedy) must use the real-only view.
• archify mismatches in the workbook. That's why the ambiguity test fails the build instead of guessing.
• Old calibration rows keyed on ʔ until they're re-keyed. That's why the lookup falls back to the old key.

Implementation notes

Code: app/rules.py (loader, detectability, exemptions), app/archi.py (symbols, letter alignment, diphthong variants), app/phones.py (base(), moved out of the scorer), app/scorer.py / app/align.py (classes, GOP, margins, tips), scripts/rescore.py. Tests: tests/test_rules.py and the "dialects and Hebrew-speaker rules" section of tests/test_scoring.py.

Where it differs from the plan above:

• archify runs inside the scorer, not on PHRASES. Scorer._variants converts each variant when scoring (cached). Display, TTS, transliteration and the verb game keep the plain IPA. Symbols replace phonemes one for one, so positions in the IPA still line up.
• Token format is SYMBOL:written (Q:ʔ), or SYMBOL{a,b} for a hand-written set. A phrase that writes a symbol itself is left alone.
• Ambiguity: across the bank (~11.6k variants) no ق is unresolved. 5 × ث merged with ت into one geminate (بعثت) and 1 × ج (جوز) stay plain phonemes; only an unresolved ق fails the test.
• Learned-only realisations (TH as s, DH as z) are accepted where the phrase writes them, with no override syntax needed.
• Missing model symbols: a class member the model lacks uses the same FALLBACKS stand-in as a plain phoneme. With the base model, ZH is {dˤ, z}, so "ظ heard as z" shows as accepted (undetectable) in /api/health → rules.
• Competitor naming (4a) keeps the GOP against the strongest symbol of any kind, so scores and deletions don't change. Only the name is withheld (other) when that symbol isn't Arabic or a Hebrew-interference sound.
• Tips: max_tips is now 3 (from the rules file), sorted by severity then score. Every tip carries severity.
• Results gain phonemes[].sound (Q), phonemes[].realised (k), and dialect: urban|rural or mixed_dialect: true. None of them affect the score.
• Teacher view: the expected/heard phone rows compare an accepted realisation with what was said, and show the dialect.

Not done:

• Per-teacher dialect settings (Phase 5). The accepted set comes from the rules file for everyone (dialects_enabled, optional_realisations_enabled). Since the taught dialect was only meant to change hints, nothing in scoring depends on it yet.
• Re-keying old calibration rows. The lookup falls back to the written phoneme, so old rows keep working. Rows proposed from the page are still keyed on phonemes[].ipa.
• The open decisions above (vowel length, sentence formula, Bedouin g, insertions) are unchanged. Vowel-length rules show as undetectable in /api/health.