AbhishekBarali commited on
Commit
aba45bd
verified
1 Parent(s): c3790da

Remove evaluation set details from the org card

Browse files
Files changed (1) hide show
  1. README.md +11 -12
README.md CHANGED
@@ -10,20 +10,19 @@ pinned: false
10
  desktop. Local speech to text, spoken answers, translation, and dictation cleanup, on
11
  Windows, macOS and Linux. Transcription runs on your own machine.
12
 
13
- This organisation publishes the models and evaluation sets behind it. Everything here is
14
- Apache-2.0 or CC-BY-4.0 and runs locally.
15
 
16
- - **speakoflow-mini-0.8b-GGUF**, a 0.8B dictation cleanup model that runs on a desktop CPU.
17
- 70.7% overall on the benchmark below, against 65.0% for a hosted frontier model and 50.0%
18
- for doing nothing at all.
19
- - **dictation-cleanup-bench**, 150 held-out rows, 68 of them no-ops, scoring restraint and
20
- edit accuracy as separate halves.
21
 
22
- Cleanup is the first piece published here because it is the part nobody measures.
23
- Grammatical error correction, ASR error correction and spoken-to-written conversion all
24
- score whether a required edit was *made*. None scores whether an unrequired edit was
25
- *withheld*, and that is the failure that ruins a dictation product. Four of the seven
26
- models measured score worse than leaving the transcript alone.
27
 
28
  Built by [Abhishek Barali](https://huggingface.co/AbhishekBarali) 路
29
  [speakoflow.com](https://speakoflow.com) 路
 
10
  desktop. Local speech to text, spoken answers, translation, and dictation cleanup, on
11
  Windows, macOS and Linux. Transcription runs on your own machine.
12
 
13
+ This organisation publishes the models behind it. Everything here is Apache-2.0 and runs
14
+ locally.
15
 
16
+ **speakoflow-mini-0.8b-GGUF** is a 0.8B dictation cleanup model that runs on a desktop CPU.
17
+ It applies the correction the speaker actually made and returns everything else byte for
18
+ byte. On a held-out evaluation it scores 70.7% overall, against 65.0% for a hosted frontier
19
+ model and 50.0% for doing nothing at all. Five quantisations are published and each one's
20
+ cost is measured rather than estimated.
21
 
22
+ That last number is the interesting one. Grammatical error correction, ASR error correction
23
+ and spoken-to-written conversion all score whether a required edit was *made*. None scores
24
+ whether an unrequired edit was *withheld*, and that is the failure that ruins a dictation
25
+ product. Four of the seven systems measured do worse than leaving the transcript alone.
 
26
 
27
  Built by [Abhishek Barali](https://huggingface.co/AbhishekBarali) 路
28
  [speakoflow.com](https://speakoflow.com) 路