docs/grading-pipeline.md

How grading works: from a recording to an exercise grade

This document traces one recording through the scorer, from the audio bytes and the reference text to the numbers the student and the teacher see. It is written for a reader who knows CTC, forced alignment and GOP, and wants the exact formulas, thresholds and order of operations this codebase uses.

The pipeline has no ASR in it. The model never transcribes freely and then diffs against the text. Instead, the reference text is turned into phoneme sequences, the audio is force-aligned to each sequence, and each phoneme is scored by how strongly the model's posteriors support it inside its own time window.

Stage	Where
Audio decode, silence trim	app/scorer.py decode_audio
Phoneme posteriors (wav2vec2 CTC)	app/scorer.py HFEmitter
Vocab pooling, many-ways letters	app/scorer.py Vocab, app/phones.py, app/archi.py
CTC Viterbi, GOP, greedy decode, edit distance	app/align.py
Linguistic rules, exemptions, margins	app/rules.py, static/assets/pronunciation-rules.md
Native baseline from TTS	app/reference.py
Result assembly, word and sentence score	app/scorer.py Scorer.score_logp
HTTP endpoint	app/main.py POST /api/score
Exercise-level grades	static/index.html, app/speech_evaluation.py

---

1. The whole pipeline at a glance

flowchart TD
    A["Browser recording<br/>webm/opus, ogg, mp4, wav"] --> B["decode_audio<br/>ffmpeg → 16 kHz mono f32<br/>cap 32 s, energy trim"]
    R["Phrase from the bank<br/>Hebrew prompt + accepted variants<br/>(Arabic, IPA)"] --> V
    B --> E["wav2vec2-xlsr-53-espeak-cv-ft<br/>log-softmax → logp (T × V)"]
    E --> M["Vocab.merge<br/>pool spelling variants of one sound"]
    V["Expand variants<br/>archify: Q, K, TH, DH, ZH, J, E<br/>+ diphthong forms"] --> T["Tokenise each variant<br/>collapse geminates / repeats"]
    M --> X["Vocab.extend<br/>add pooled class columns"]
    T --> X
    X --> F["CTC Viterbi per variant<br/>keep max total log-prob"]
    F --> G["GOP per token<br/>in its window"]
    G --> H["Exemptions<br/>glottal drop · article vowel · near vowel"]
    H --> I["Vowel-insertion cap<br/>rule 3.10"]
    I --> J["Native baseline<br/>divide by TTS voice's score"]
    J --> K["Teacher calibration floors"]
    K --> L["Flags, tips, issues<br/>margins + rules"]
    L --> W["Word score<br/>0.5·min + 0.5·mean"]
    W --> S["Sentence score<br/>0.5·phone mean + 0.5·word mean"]
    F --> U["Greedy decode vs best variant<br/>edit distance ratio"]
    U -->|"over 0.65"| C["unrelated: cap at 30, no tips"]
    C --> S
    S --> O["JSON result 0–100"]
    O --> P["Exercise grades<br/>band, best score, topic, quiz, verb game"]

2. Request sequence

sequenceDiagram
    autonumber
    participant UI as Browser (index.html)
    participant API as FastAPI /api/score
    participant FF as ffmpeg
    participant Ref as References (TTS baseline)
    participant TTS as Sawtak TTS
    participant Sc as Scorer
    participant M as wav2vec2 CTC
    participant DB as Supabase attempts

    UI->>API: multipart {phrase_id, audio}
    API->>FF: decode to 16 kHz mono float32
    FF-->>API: wav (silence-trimmed)
    alt wav shorter than 0.3 s
        API-->>UI: {error: too_short}
    end
    API->>Ref: get(phrase_id, phrase), wait ≤ 8 s
    opt first time this phrase is scored
        Ref->>TTS: synthesise variant 0 (first voice)
        TTS-->>Ref: native audio
        Ref->>Sc: reference(wav, phrase)
        Sc->>M: emit
        M-->>Sc: logp
        Sc-->>Ref: per-sound native scores, or None if untrustworthy
    end
    Ref-->>API: list of References (possibly empty)
    API->>Sc: score(wav, phrase, refs, diagnostics=teacher)
    Sc->>M: emit(wav)
    M-->>Sc: logp (T × V)
    Sc->>Sc: merge → variants → Viterbi → GOP → rules → baseline → floors → aggregate
    Sc-->>API: result JSON
    API-->>UI: result
    API--)DB: background: {cat, phrase, score, unrelated, flagged issues}
    UI->>UI: band label, letter colours, keep best score per sentence

The reference text never arrives from the client: the client sends only phrase_id, and the server looks up the phrase (bank phrase, teacher text line, or a verb-game form held server-side). That matters for the verb game, where the expected answer must not be exposed to the browser.

---

3. Inputs

3.1 Audio

decode_audio pipes the upload through ffmpeg (-ac 1 -ar 16000 -f f32le -t 32). It then trims leading and trailing silence with a 20 ms RMS envelope (320 samples). A sample counts as voiced if its envelope is above max(0.01, 0.05 · peak). 200 ms of padding is kept on each side. The trim keeps CTC from being fed seconds of blank, which would otherwise widen the first and last tokens' GOP windows.

Anything shorter than 0.3 s after trimming is rejected before inference.

3.2 Reference text: a phrase and its accepted variants

A phrase in app/phrases.py (and the generated workbook, grammar and teacher texts) carries a list of accepted variants, each (arabic, ipa):

dict(cat="meet", he="אני מירושלים", variants=[
    ("انا من القدس", "ʔ a n a | m i n | i l ʔ u d s"),
    ("انا من القدس", "ʔ a n a | m i n | i l q u d s"),
])

IPA phonemes are space-separated, words are separated by |, and the Arabic and IPA must have the same word count. A variant is a genuinely different correct sentence (different lexical choice, a dropped word, a different vowel pattern). Dialectal realisations of single letters are handled by the next step instead.

3.3 Variant expansion: many-ways letters ("archiphonemes")

Six Arabic letters have more than one correct Palestinian realisation. app/archi.py archify() aligns each word's Arabic letters to its IPA consonants (a small edit-distance DP with cheap skips for weak letters like ا و ي ة ل). It then rewrites the phoneme that came from such a letter as SYMBOL:written:

Symbol	Letter	Accepted realisations (rules §5)
Q	ق	ʔ (urban), k (rural), q (also ok); g only if optional_realisations_enabled
K	ك	k, tʃ (rural)
TH	ث	t, θ; s only in learned words where the phrase writes s
DH	ذ	d, ð; z only in learned words where the phrase writes z
ZH	ظ	dˤ, zˤ, ðˤ
J	ج	dʒ, ʒ
E	final ة	a, e, i (positional)
O	pronoun ending	o, u (positional)

So ʔ a h w e for قهوة becomes Q:ʔ a h w E:e. The written phoneme is kept for display, tips and calibration keys. The class membership is what gets scored.

Two more expansions:

• Diphthongs (§6.3 #8). diphthong_variants() adds a copy of the variant with eː → a j and oː → a w where the Arabic writes ي / و. A diphthong is two tokens instead of one, so it cannot be a class. It becomes a separate variant that competes in the Viterbi step.
• Deduplication. Variants that differ only in how a many-ways letter is said collapse into one after archifying.

A teacher can switch off any of these (the dialect_*, final_vowel and diphthong toggles). The scorer then requires the phoneme as written.

---

4. Acoustic model and vocabulary handling

4.1 Emissions

HFEmitter runs facebook/wav2vec2-xlsr-53-espeak-cv-ft (or a fine-tuned checkpoint via MODEL_ID) and returns log_softmax(logits), shape (T, V). That is ~50 frames/s (20 ms hop), V ≈ 392 espeak IPA symbols plus <pad> as the CTC blank.

Nothing downstream depends on torch. Scorer takes any Emitter callable, which is how tests/test_scoring.py drives the full scorer with synthetic posteriors.

4.2 Pooling spelling variants (Vocab.merge)

The multilingual vocab spells one sound several ways (t / t̪ / tʰ / tː, r / ɾ, a / æ / ɐ, x / χ). Left alone, a correctly said t would lose its GOP to its own allophone t̪. app/phones.py base() maps each symbol to a canonical sound by stripping length, aspiration, dentality, nasality and so on, and folding a fixed equivalence table. Emphatics stay distinct.

merge() replaces each group's columns with one representative column holding the group's logsumexp. The other columns are set to −1e9.

A consequence: vowel length is invisible to the scorer, because aː and a share a base. Length errors can surface only through the margins and rules on the written phoneme, not through separate posteriors.

4.3 Class columns for many-ways letters (Vocab.add_class, Vocab.extend)

For a token like Q:ʔ, the scorer looks up each accepted realisation's model id (falling back via FALLBACKS, e.g. ðˤ → zˤ → z, when the vocab lacks a symbol). If more than one distinct id remains, it allocates a virtual id past the real vocab. extend() appends one column per virtual id: the logsumexp of its members' posteriors.

This implements rules §6.1. If P(ʔ)=0.35 and P(k)=0.40 in a frame, the Q column holds 0.75, and neither member is treated as a competitor of the other.

4.4 Tokenisation

tokenize() turns the archified IPA into a list of TargetToken(phoneme, id, word, sound). Adjacent tokens with the same (canonical) id collapse into one, as in CTC. That means geminates (m m) become one token, and so do two neighbouring phonemes that map to the same sound. Gemination is therefore not scored (§6.3 #10).

---

5. Choosing the variant: CTC forced alignment

For every expanded variant, app/align.py ctc_viterbi() runs the standard CTC Viterbi over the extended label sequence blank, y1, blank, y2, …, yN, blank (S = 2N+1 states). It allows stay, advance by one, and the skip s−2 → s when the label differs from s−2. The path must end on the last label or the trailing blank.

The result is total_logp (the best path's log-probability) and spans[i] = [start, end), the frames where token i is emitted.

stateDiagram-v2
    direction LR
    [*] --> b0
    [*] --> y1
    b0 --> b0
    b0 --> y1
    y1 --> y1
    y1 --> b1
    y1 --> y2: skip if y2 ≠ y1
    b1 --> b1
    b1 --> y2
    y2 --> y2
    y2 --> b2
    b2 --> b2
    y2 --> [*]
    b2 --> [*]

The variant with the highest total_logp wins (not length-normalised; all variants are scored against the same T frames). If T < N for every variant, the result is error: too_short.

Example: قديش said with k aligns better to Q a d d eː ʃ than to an alternative wording. Because Q pools ʔ/k/q, the k realisation costs nothing.

---

6. Per-token scoring

Each token goes through a fixed sequence of stages. The teacher's diagnostic view (trace.tokens[i].stages) reports the score after each one.

flowchart LR
    G["1 · GOP<br/>score_tokens"] --> X["2 · Exemption<br/>max(score, floor)"]
    X --> I["3 · Insertion cap<br/>min(score, 60)"]
    I --> B["4 · Native baseline<br/>100·s / b"]
    B --> C["5 · Calibration floor<br/>max(score, floor)"]
    C --> F{"6 · Flag?<br/>score under 50<br/>AND beyond margin<br/>AND consonant or flagged vowel"}
    F -->|yes| T["tip (by rule)"]
    F -->|no| N["maybe an 'issue' if under 90"]

6.1 Stage 1: GOP (score_tokens)

Window. Token i's window runs from the end of token i−1 to the start of token i+1 (clamped to include its own span). It therefore contains the blank frames around the emission, which is where a substituted phoneme typically peaks when the path was forced through the target.

Target peak. peak = max_t logp[t, target] inside the window. For a class token this is the pooled column. realised names the member with the highest peak (used for dialect detection and display).

Competitor. Over the real columns only (no virtual columns), take each column's max inside the window. Then mask:

• the blank,
• the labels of tokens i−1, i, i+1 and all their class members.

Neighbours are masked because coarticulation into the adjacent phoneme is not an error (§6.3 #11). The argmax of what remains is the competitor, and its max is comp_peak. If the competitor is outside the set of sounds the system can name (Arabic, Hebrew interference sounds, rule labels), it is reported as other instead of by name. Example: a click or a retroflex from the multilingual vocab.

Score.

$$\text{gop}=\log p_{\text{target}}-\max (\log p_{\text{target}},\log p_{\text{comp}})\le 0\backslash text\{gop\}\; =\; \backslash log\; p\_\{\backslash text\{target\}\}\; -\; \backslash max(\backslash log\; p\_\{\backslash text\{target\}\},\backslash \; \backslash log\; p\_\{\backslash text\{comp\}\})\; \backslash ;\backslash le\; 0$$

$$\text{confidence}=\min \text{\hspace{0.17em}⁣}(1,\frac{p_{\text{target}}}{0.30})\backslash text\{confidence\}\; =\; \backslash min\backslash !\backslash left(1,\backslash \; \backslash frac\{p\_\{\backslash text\{target\}\}\}\{0.30\}\backslash right)$$

$$\text{score}=100\cdot e^{\lambda \cdot \text{gop}}\cdot \sqrt{\text{confidence}},\lambda =0.5(\mathtt{\text{leniency}})\backslash text\{score\}\; =\; 100\; \backslash cdot\; e^\{\backslash ,\backslash lambda\backslash cdot\; \backslash text\{gop\}\}\; \backslash cdot\; \backslash sqrt\{\backslash text\{confidence\}\},\; \backslash qquad\; \backslash lambda\; =\; 0.5\backslash \; (\backslash texttt\{leniency\})$$

• With λ = 0.5 the ratio is square-rooted: a competitor twice as likely as the target gives 71, not 50. The zero-shot model is noisy on accented speech, so a linear ratio would punish the model's own uncertainty.
• The confidence term penalises a target that wins but only weakly. Below a peak posterior of 0.30, the score shrinks with √(p/0.3).
• A score of 100 requires the target to beat every non-neighbour competitor and reach p ≥ 0.30.

Deletion. deleted = p_target < 0.05 AND p_comp < 0.20: nothing clearly there, target or otherwise. A deleted token is reported as heard _.

Worked numbers. ħ said as a Hebrew ח (χ), with p(ħ) = 0.04 and p(χ) = 0.85 in the window:

quantity	value
gop	ln(0.04 / 0.85) = −3.056
e^(0.5·gop)	0.217
confidence	0.04 / 0.30 = 0.133, √ = 0.365
score	100 · 0.217 · 0.365 ≈ 7.9
deleted?	no (p_comp 0.85 ≥ 0.20)

6.2 Stage 2: exemptions (rules §6.3)

These are cases the model hears as a mistake that are ordinary native speech. The first rule that applies wins (app/rules.py EXEMPTIONS):

Exemption	Condition	Result
glottal_dropped	token deleted and its sound is ʔ or Q	100
article_vowel_elided	deleted vowel of ال right after a word ending in a vowel	100
near_vowel	vowel heard as same-group neighbour (i/e, u/o, a/ə/e) or other length	≥ 90 (near_vowel_floor)
unmarked_short_vowel	short vowel deliberately not judged by policy; excludes word-internal post-consonantal u (rule 30)	≥ 90 (near_vowel_floor)
final_b_devoicing	word-final b heard as p before a voiceless onset in the next word (rule 28)	100
word_final_vowel_linking	deleted word-final short vowel before a vowel-initial next word, including ʔ+V (rule 29)	≥ 90 (near_vowel_floor)

A glottal stop replaced by another consonant is not exempt (only a deletion is), unless the class accepts it (ق → k).

6.3 Stage 3: vowel insertion inside a cluster (rule 3.10)

Hebrew speakers often break a word-internal consonant cluster with an epenthetic vowel (katiir for ktiir). For each pair of adjacent consonants in the same word (not the sentence's first cluster, and not when either is w or j), the scorer looks at the greedy (argmax) frames between the two consonants' spans. If at least min_frames = 2 frames have a vowel as argmax with p ≥ min_peak = 0.5, the second consonant's score is capped at score_cap = 60, and a severity-1 tip is attached.

6.4 Stage 4: native baseline

The phoneme model mishears some sounds even from natives (ʕ in على heard as dropped, l as r). A learner should not lose points for the model's weaknesses.

app/reference.py synthesises the phrase's first variant with a Palestinian TTS voice and scores it with the same Scorer. It keeps the per-sound scores, unless the fit is unrelated or the mean phoneme score is under ref_min_mean = 60, in which case the voice is not trusted. One voice is generated per phrase. Further voices are used only if already cached from playback. The score request waits up to 8 s for a reference, then proceeds without one.

Learner tokens are matched to reference tokens with difflib.SequenceMatcher on (word, sound) pairs. Sounds compare by symbol, so a reference that said ق as q is a valid baseline for a learner who says ʔ. Per learner token, b is the minimum score over all references that share that sound. Then:

So a sound the TTS voice scores 40 on and the learner scores 40 on becomes 100. The baseline only ever raises scores. A word whose sounds were raised this way, and now all score ≥ 90, also shows the expected transliteration as "heard": the visible difference is one a native produces too.

6.5 Stage 5: teacher calibration floors

Confirmed tutor flags can become calibration rules (expected, heard, phrase?) → floor. A rule is looked up by sound symbol first, then by written phoneme, phrase-specific before global. It raises a matching token to floor and never lowers it. Applied floors are listed in result.calibrated.

The base multilingual model also has one conservative built-in calibration: when expected ʕ is classified as deleted (_), its score is raised to 60. This prevents a known false negative from producing a definite “ع was swallowed” warning, while retaining a meaningful penalty. An explicit teacher calibration for ʕ → _ overrides this floor.

6.6 Stage 6: is it a mistake? Flags, margins, rules

Final per-token scores now exist. The decision to flag a token (show "heard as X" and a tip) needs all three of:

1. score < flag_below (50), or the sound's own threshold in flag_below_sound. That is set per model (FLAG_BELOW_BY_MODEL): for the base model ح uses 20, because it scores a correct ح in the 20s–40s (measured with scripts/minerva_benchmark.py).
2. The competitor beats the target by at least the pair's margin (rules §6.2):
   • substitution: −gop ≥ margin(target, heard)
   • deletion: p_target ≤ deleted_peak − margin(target, "_") in log space
3. The token is a consonant, or a short vowel a rule explicitly flags. Currently that is only u after a consonant in the same word (rule 3.11). Other short vowels are never flagged (§6.3 #7).

Margins are natural-log ratios from the rules document:

Target	Heard	Margin	Meaning
sˤ tˤ dˤ zˤ ðˤ	s t d z ð	0.7	plain must be ≈ 2× likelier (emphasis is weak in the base model)
h	_	0.7	h is short and quiet; natives reduce it too
ħ	h	0.4	≈ 1.5×
long vowels	short	1.1	declared, but see the note below
everything else		0.0	the competitor merely has to win

Margins on long-vowel targets are skipped at parse time ("ː" not in t), because base() drops length and the margin would otherwise land on vowel-quality pairs it was not meant for.

The rule for a substitution is looked up as (target, heard) in the checks of static/assets/pronunciation-rules.md. Example:

id: pharyngeal_h
checks:
  - {target: ħ, heard: [χ, x], severity: 3, feedback: "…"}
  - {target: ħ, heard: [h, ɦ], severity: 2, feedback: "…"}
  - {target: ħ, heard: ["_"],  severity: 2, feedback: "…"}

Severity 3 means the error changes meaning (ħ/χ, ʕ/ʔ); 1 is accent. The rule chooses the Hebrew feedback text and the tip ordering. It does not change the number. The score is pure acoustics plus the stages above.

---

7. From tokens to words, letters and the sentence

flowchart BT
    P1["phoneme scores<br/>(vowels weight 0.5)"] --> WS["word score<br/>0.5 · min + 0.5 · mean"]
    P1 --> PM["phone mean<br/>weighted"]
    WS --> WM["word mean"]
    PM --> SS["sentence = 0.5 · phone mean + 0.5 · word mean"]
    WM --> SS
    SS --> UC{"unrelated?"}
    UC -->|yes| CAP["min(score, 30)"]
    UC -->|no| R["round → 0–100"]
    CAP --> R
    P1 --> LT["letter colour<br/>= weakest of its sounds"]

Word. word = 0.5 · min(phonemes) + 0.5 · mean(phonemes) (word_min_weight). One bad sound drags its word down hard, but not to zero.

Sentence.

The word term exists so that a single wrong consonant moves a sentence from "excellent" to "almost". A plain phoneme mean would dilute it across all the correct sounds.

Letters. For display, the Hebrew-letter transliteration of each word is mapped back to the tokens that carry each letter (_carriers, word_letters). Each letter gets the minimum score among its tokens. A letter under LETTER_GOOD = 75 also carries why (what the sound was heard as) and tip. The page colours letters ≥ 75 / ≥ 50 / below.

7.1 Worked example

A hypothetical phrase w eː n | i l ħ a m m aː m ("where is the bathroom"), with ħ said as χ as in §6.1. Geminate m m is one token.

word	tokens and scores	mean	min	word score
ween	w 100 · eː 95 · n 100	98.3	95	96.7
ilħammaam	i 100 · l 100 · ħ 7.9 · a 96 · m 100 · aː 92 · m 100	85.1	7.9	46.5

• phone_mean = (5·100 + 7.9 + 0.5·(95+100+96+92)) / (6 + 4·0.5) = 699.4 / 8 = 87.4
• word_mean = (96.7 + 46.5) / 2 = 71.6
• overall = 0.5·87.4 + 0.5·71.6 = 79.5 → 80, which the page labels "כמעט" (almost).

With phoneme averaging alone, the same recording would score 87 and be called excellent.

ħ is flagged: 7.9 < 50, the margin for (ħ, χ) is 0, so −gop = 3.06 ≥ 0, and ħ is a consonant. The rule pharyngeal_h gives a severity-3 tip.

---

8. "Different sentence?" detection

Forced alignment always produces a path, even for the wrong sentence. To catch this, the scorer greedy-decodes the posteriors (argmax per frame, collapse repeats, drop blanks; real columns only). It then computes the Levenshtein distance to the chosen variant's token ids, with class members counting as equal.

$$\text{unrelated}=\frac{\text{edit\_distance}}{N_{\text{tokens}}}>0.65\backslash text\{unrelated\}\; =\; \backslash frac\{\backslash text\{edit\backslash \_distance\}\}\{N\_\{\backslash text\{tokens\}\}\}\; >\; 0.65$$

If unrelated, the sentence score is capped at 30, no tips or issues are produced, letters get no reasons, and the page shows "משפט אחר?". It also covers hesitations and restarts (§6.3 #14). The greedy decode is returned as heard_ipa. The Hebrew-letter rendering of what was heard (heard_tr) is built by cutting the greedy sequence at the aligned word starts.

---

9. Feedback selection

flowchart TD
    A["Flagged tokens"] --> B["tip_for(sound, heard, rule)<br/>dedupe by tip key"]
    B --> C["sort: severity desc, then score asc"]
    C --> D["keep max_tips = 3"]
    E["All tokens under 100 without a tip"] --> F["keep those under 90 AND beyond margin"]
    F -->|none and no tips| G["the 3 weakest"]
    F --> H["issues, up to 8, weakest first"]
    G --> H

• Tips are the flagged substitutions with a rule, most severe first (meaning-changing before accent), then lowest score. At most 3 (4 with the tips_by_severity toggle off).
• Issues explain any score short of 100. Every imprecise sound not already covered by a tip, so the student never sees 94 with no reason.
• Dialect. If the accepted realisations used are all urban-only or all rural-only, the result carries dialect. If they disagree, it carries mixed_dialect. Either way this is informational and never affects the score (§6.3 #12).

---

10. The response

{
  "score": 80,                          // sentence, 0–100
  "variant": {"ar": "...", "tr": "...", "ipa": "..."},  // the chosen accepted variant
  "heard_ipa": "w e n i l χ a m a m",   // greedy decode
  "heard_tr": "...",                    // the same in Hebrew letters
  "unrelated": false,
  "words": [{
    "ar": "...", "tr": "...", "heard_tr": "...", "score": 46.5,
    "letters":  [{"t": "חַ", "score": 7.9, "why": "...", "tip": "..."}],
    "phonemes": [{"ipa": "ħ", "word": 1, "score": 7.9, "flagged": true, "heard": "χ"}]
  }],
  "tips":   [{"key": "...", "title": "...", "text": "...", "severity": 3, "word": "...", "score": 8}],
  "issues": [{"sound": "...", "word_tr": "...", "score": 92, "missing": false, "heard": "..."}],
  "references": 1,                       // native baselines used
  "dialect": "urban",                    // or "mixed_dialect": true
  "calibrated": [ ... ],                 // only if a teacher floor applied
  "phoneme_heatmap": { ... },            // teachers only: max-pooled posteriors, ≤ 180 columns
  "trace": { ... }                       // teachers only: every stage, threshold and intermediate value
}

trace is the place to start when debugging a surprising score. Its full schema is docs/trace-schema.ts, and the trace dashboard (static/trace.html, "לוח אבחון" in the lab and the phoneme dialog) draws it: the frames around a sound with its window, its score stage by stage, the candidate readings and the unrelated gauge, and the sentence with its aggregation. It holds the candidate variants with their path log-probs, the unrelated ratio, and per token: window, p_target, competitor, gop, confidence factor, each stage's score, margin needed vs. got, rule, and whether the token became a tip or an issue.

---

11. From a recording's score to exercise grades

The scorer grades one recording. Everything above that level is aggregation, mostly on the client.

flowchart TD
    R["Recording score 0–100"] --> BAND["Verdict on the card<br/>≥ 75 מצוין · ≥ 60 כמעט · else נסו שוב<br/>unrelated → משפט אחר?"]
    R --> BEST["Sentence best = max over attempts<br/>(device + attempts table)"]
    BEST --> TOPIC["Topic: done when every sentence has a best<br/>★ gold when all bests ≥ 75"]
    TOPIC --> LESSON["Lesson / homework progress"]
    BEST --> PANEL["Teacher panel: student grade = mean of bests<br/>class 'hard sentences' = class mean of bests under 70"]
    BEST --> DAILY["Daily session review pool: bests under 75"]
    R --> QUIZ["Story quiz: mean of line scores said<br/>+ top 3 mistakes by frequency"]
    R --> VERB["Verb game L4–5: ≥ 75 correct<br/>unrelated / too short → uncertain, not counted"]
    VERB --> GATE["Level gates: share correct ≥ 85–90 %"]

Exercise	Grade	Where
Single sentence	The recording's score; band 75 / 60	index.html (EXCELLENT = 75)
Sentence progress	Best score over all attempts. Any non-empty recording, including an unrelated one, marks the sentence as practised	keepScore
Topic	Practised count / total; gold if every best ≥ 75	tally()
Story quiz ("בוחן")	Arithmetic mean of the lines' scores. One recording per line, no retries	quizEnd()
Verb game, levels 4–5	correct if score ≥ 75. unrelated or too short → outcome: "uncertain", counts_as_attempt: false, never a grammar error	app/speech_evaluation.py
Verb game, unlocking	Correct ratio ≥ VERB_THRESH (0.85–0.90) over a minimum number of attempts	index.html
Arabic → Hebrew translation	Not scored by this pipeline: browser he-IL ASR, share of expected words heard in order, 0.8 = correct. Not stored	index.html (HE)
Teacher panel	Student grade = mean of their best scores; class difficulty per sentence = mean of students' bests	index.html, /api/class

Verb game levels 1–3 (choice and matching) are graded server-side by exact match against a stored answer, with no audio involved.

---

12. Configuration reference

The values live in code, not in the rules document. The scoring: block of pronunciation-rules.md is read for margins, margin_default and max_tips only. Its other keys (flag_below, bands, sentence_score, severity_weight, …) are documentation. Its sentence_score string (mean(word_min)) also describes an older word formula; the code uses 0.5·min + 0.5·mean.

Knob	Default	Effect
GopConfig.leniency	0.5	exponent on the target/competitor ratio
GopConfig.confident_peak	0.30	target posterior above which confidence stops lowering the score
GopConfig.deleted_peak / deleted_comp	0.05 / 0.20	deletion thresholds
GopConfig.consonant_window	4 frames	maximum context on each side of a consonant span; never crosses neighbours
ScorerConfig.flag_below	50	max score for a flag and tip
ScorerConfig.flag_below_sound	{ħ: 20} for the base model	per-sound replacement of flag_below, from FLAG_BELOW_BY_MODEL[MODEL_ID]
ScorerConfig.issue_below	90	max score to be listed as an issue
ScorerConfig.near_vowel_floor	90	floor for neighbouring vowel qualities
ScorerConfig.word_min_weight	0.5	weight of the weakest phoneme in a word
ScorerConfig.unrelated_ratio	0.65	edit-distance ratio for "different sentence"
ScorerConfig.ref_min_mean	60	minimum mean for a TTS reference to be trusted
vowel weight	0.5	in the sentence phone mean (hard-coded)
unrelated cap	30	hard-coded
LETTER_GOOD	75	letter colour / reason threshold
insertion min_peak, min_frames, score_cap	0.5, 2, 60	rule 3.10, from the rules document

Teachers can switch off each rule-derived behaviour at runtime (the ניטור tab, rules.TOGGLES). Switching one off restores the pre-rules behaviour for that aspect from the next recording on.

---

13. Known limitations

• Zero-shot model. The base model was not trained on Palestinian Arabic or Hebrew-accented speech. The native baseline and margins compensate partially; fine-tuning (train/finetune.py) on teacher and labelled learner recordings is the intended fix.
• Duration is not modelled. CTC posteriors are peaky and collapse repeats, so vowel length and gemination are effectively unscored.
• Fallback symbols. If the vocab lacks a phoneme (e.g. ðˤ), it is scored as its stand-in. /api/health → phoneme_fallbacks lists these, and the contrast they lose cannot be detected.
• GOP is max-pooled, not frame-summed. Using window peaks makes the score robust to alignment jitter, but a brief spurious competitor spike can still win the window.