RadeAI commited on
Commit
70129dd
Β·
verified Β·
1 Parent(s): b6dfe02

eval: add VisualEars golden test set results (3 datasets)

Browse files
Files changed (1) hide show
  1. README.md +2 -1
README.md CHANGED
@@ -61,9 +61,10 @@ Measured by Rade with greedy CTC decoding (fp16) on two standard Persian test se
61
  | Test set | Clips | WER | CER |
62
  |---|---|---|---|
63
  | **FLEURS** `fa_ir` β€” read speech | 871 | 19.6 % | **4.4 %** |
 
64
  | **Common Voice 17.0** `fa` β€” crowd-sourced | 10,355 | 21.8 % | 17.8 % |
65
 
66
- > **CER is the more faithful metric for Persian.** Persian **WER** is inflated by orthographic/spacing variation (Ω†ΫŒΩ…β€ŒΩΨ§Ψ΅Ω„Ω‡/ZWNJ, affix spacing, compound spelling) that doesn't reflect actual mis-recognition β€” note FLEURS sits at **19.6 % WER but only 4.4 % CER**, i.e. most "word errors" are one-character spelling differences. On clean read speech (FLEURS) the model reaches **CER β‰ˆ 4 %**; on noisier crowd-sourced audio (Common Voice β€” spontaneous speech, varied mics/accents, loan words) **CER β‰ˆ 18 %**.
67
 
68
  ## Speed & hardware
69
 
 
61
  | Test set | Clips | WER | CER |
62
  |---|---|---|---|
63
  | **FLEURS** `fa_ir` β€” read speech | 871 | 19.6 % | **4.4 %** |
64
+ | **VisualEars golden** `fa` β€” curated (clean/farfield/obstructed) | 6,669 | 22.9 % | **4.3 %** |
65
  | **Common Voice 17.0** `fa` β€” crowd-sourced | 10,355 | 21.8 % | 17.8 % |
66
 
67
+ > **CER is the more faithful metric for Persian.** Persian **WER** is inflated by orthographic/spacing variation (Ω†ΫŒΩ…β€ŒΩΨ§Ψ΅Ω„Ω‡/ZWNJ, affix spacing, compound spelling) that doesn't reflect actual mis-recognition β€” note FLEURS sits at **19.6 % WER but only 4.4 % CER**, i.e. most "word errors" are one-character spelling differences. On clean, well-curated speech (FLEURS, VisualEars) the model reaches **CER β‰ˆ 4 %**, and it stays robust across recording conditions (VisualEars far-field 4.5 % / obstructed 4.2 % CER). On noisier crowd-sourced audio (Common Voice β€” spontaneous speech, varied mics/accents, loan words) CER rises to **β‰ˆ 18 %**.
68
 
69
  ## Speed & hardware
70