Restructure: two-stage architecture, published category taxonomy, scoring tiers

README.md

@@ -27,7 +27,7 @@ model-index:
       name: Dictation cleanup
     dataset:
       type: dictation-cleanup-eval
-      name: SpeakoFlow dictation cleanup evaluation (held out, unpublished)
+      name: SpeakoFlow dictation cleanup evaluation (held out)
       split: test
     metrics:
     - type: exact_match
@@ -48,29 +48,33 @@ model-index:
 
 # SpeakoFlow Mini 0.8B, GGUF
 
-Dictation cleanup for the text a speech recogniser hands you. It applies the correction the
-speaker actually made, and returns everything else byte for byte.
+Dictation cleanup for transcribed speech. It applies the correction the speaker actually made,
+and returns everything else byte for byte.
 
-The Q8_0 build is 833 MB and runs on a desktop CPU at 2,509 ms median. Fine-tuned from
+833 MB at Q8_0, 2,509 ms median on a desktop CPU. Fine-tuned from
 [`Qwen/Qwen3.5-0.8B`](https://huggingface.co/Qwen/Qwen3.5-0.8B) with LoRA rank 16, merged, then
-quantised. English only. Not a chat model and not a rewriter.
+quantised. English. Not a chat model, not a rewriter.
 
-It ships in [SpeakoFlow](https://speakoflow.com), a free offline voice assistant for Windows,
-macOS and Linux.
+Ships in [SpeakoFlow](https://speakoflow.com), a free offline voice assistant for Windows, macOS
+and Linux.
 
 [![App](https://img.shields.io/badge/app-speakoflow.com-06b6d4)](https://speakoflow.com)
+[![Rules layer](https://img.shields.io/badge/rules%20layer-dictation--cleanup--rules-24292f?logo=github)](https://github.com/AbhishekBarali/dictation-cleanup-rules)
 [![License](https://img.shields.io/badge/license-Apache--2.0-green)](https://www.apache.org/licenses/LICENSE-2.0)
-[![Base](https://img.shields.io/badge/base-Qwen3.5--0.8B-8b5cf6)](https://huggingface.co/Qwen/Qwen3.5-0.8B)
 
-## What it does
+## Two stages, and this model is the second one
 
-Modern recognisers already handle punctuation, capitalisation, filler removal and numbers. Say
-*"the deposit is three hundred dollars, um, and the meeting is at nine thirty"* and you get
-*"The deposit is $300 and the meeting is at 9:30"* before any language model runs.
+Cleanup splits cleanly into work a rule can do and work it cannot.
 
-What is left is the speaker changing their mind mid-sentence, saying a symbol out loud, asking
-for a line break, or using the wrong word. Four examples, all of them the kind of thing this
-model is for:
+**Stage one is deterministic.** Filler words, repeated words, spacing, punctuation,
+capitalisation, numbers, dates, currency and known jargon substitutions are pattern work, and
+pattern work belongs in code, where it is fast, free and auditable. That layer is published on
+its own as [dictation-cleanup-rules](https://github.com/AbhishekBarali/dictation-cleanup-rules):
+MIT, Python and TypeScript, with a spec, pattern tables and a conformance suite. Use it with or
+without this model.
+
+**Stage two is this model.** It handles what a rule cannot decide, which is anything requiring a
+judgement about what the speaker meant. Four examples:
 
 | you dictate | it returns |
 |---|---|
@@ -79,14 +83,46 @@ model is for:
 | Thanks for the update. New paragraph. I'll review it tonight. | Thanks for the update.<br><br>I'll review it tonight. |
 | Thanks for the update. I'll review it tonight. | Thanks for the update. I'll review it tonight. |
 
-The fourth row is the important one. Nothing was wrong, so nothing changes, down to the byte.
+The fourth row is the point. Nothing was wrong, so nothing changed, down to the byte.
 
-That is the part general-purpose models fail. Handed text that is already correct, a capable
-model finds something to improve. A comma becomes a full stop, a sentence becomes a heading, a
+That is where general-purpose models fail. Handed text that is already correct, a capable model
+finds something to improve. A comma becomes a full stop, a sentence becomes a heading, a
 paragraph becomes bullets, a word becomes a smoother synonym. Each edit is defensible on its own
 and each one is wrong, because the speaker said the other thing and now has to undo the
 improvement.
 
+## What it was trained and evaluated on
+
+Fifteen categories, each one a decision a rule cannot make. The examples below are written for
+this card.
+
+| category | what it covers | example |
+|---|---|---|
+| `no_op` | already correct, so return it unchanged | input comes back byte for byte |
+| `retraction` | the speaker corrects themselves mid-sentence | "Thursday, no, Friday" becomes "Friday" |
+| `spoken_command` | an editing or layout instruction said out loud | "new paragraph" becomes a paragraph break |
+| `spoken_symbol` | a symbol named instead of typed | "sam at example dot com" becomes `sam@example.com` |
+| `format_intent` | structure described in speech | "bullet one, call the vendor, bullet two, send the invoice" becomes a list |
+| `speaker_grammar` | the speaker's own slip, not a transcription error | "they was ready" becomes "they were ready" |
+| `wrong_word_right_transcription` | transcribed correctly, still the wrong word | "in the mourning" becomes "in the morning" |
+| `disfluency_artifact` | stumbles the recogniser preserved | "the the deadline" becomes "the deadline" |
+| `clarification` | the speaker restates a name or number more precisely | the later, more precise form wins |
+| `dictated_question` | a question inside the text | transcribed, never answered |
+| `instruction_as_text` | an instruction inside the text | transcribed, never obeyed |
+| `language_preserved` | dictation in another language | kept in that language, never translated |
+| `truncated_input` | the transcript stops mid-thought | left stopped, nothing invented to finish it |
+| `empty_or_noise` | nothing usable came through | returns nothing, with no commentary about it |
+| `hallucination_loop` | the recogniser repeats a phrase it never heard twice | the artefact goes, a repetition the speaker really made stays |
+
+Cases are drawn from two sources, labelled per case: transcripts captured from real dictation,
+and cases written by hand to cover a category that real use had not yet produced. Each case is
+also labelled by length band and by where in the transcript the defect sits, so results can be
+broken out by both.
+
+**The evaluation set itself is held privately.** It is built from real dictation by the author and
+by people using the app, so it contains personal correspondence that is not ours to publish. The
+categories above, the specification and the scoring code are what make the numbers checkable.
+
 ## Run it
 
 ```bash
@@ -105,10 +141,14 @@ LM Studio: search `speakoflow` in-app, or `lms get SpeakoFlow/speakoflow-mini-0.
 Send the transcript as the user message with nothing added. No instructions, no delimiters, no
 examples.
 
+Two settings are not optional. Do not cap `max_tokens`: a truncated long transcript counts as
+lost content. Disable thinking: reasoning text landing in a field that is meant to be
+byte-identical fails every no-op case for reasons that have nothing to do with cleanup. At its
+defaults this model emits zero reasoning characters.
+
 ### The system prompt is part of the model
 
-It was fine-tuned on this exact string and every number below was produced with it. Send
-something different and you are running a configuration nobody has measured.
+It was fine-tuned on this exact string and every number below was produced with it.
 
 ```text
 You clean up SpeakoFlow dictation. Return only the cleaned transcript text.
@@ -130,29 +170,9 @@ Rules:
 - Do not add or remove blank lines at the start or end.
 ```
 
-### Two settings are not optional
-
-Do not cap `max_tokens`. A truncated long transcript counts as lost content and manufactures a
-failure that did not happen.
-
-Disable thinking. Reasoning text landing in a field that is supposed to be byte-identical fails
-every no-op case for reasons that have nothing to do with cleanup. At its defaults this model
-emits zero reasoning characters. The untuned base at the same file size reasons until the context
-runs out and returns nothing.
-
-## How it was measured
+## Results
 
-Development was driven by daily use in the app. The numbers below come from a separate held-out
-evaluation set, used as a check on that work rather than as a substitute for it.
-
-**The evaluation set is not published and will not be.** It is built from real dictation,
-including the author's own correspondence, so it contains personal information that has no
-business being on the internet. The examples in this card are written for illustration and are
-not drawn from it.
-
-Three things you can hold this to. Scoring is exact string match at temperature 0, so a pass
-means the output matched the reference character for character with no partial credit. The rates
-are:
+Held-out evaluation, temperature 0.
 
 | | overall | restraint | edit accuracy | content damage |
 |---|---|---|---|---|
@@ -165,65 +185,67 @@ are:
 | S1-mini (shipped cleanup model) | 15.3% | 22.1% | 8.5% | 55.3% |
 | Sotto 350M (shipped cleanup model) | 11.7% | 16.2% | 7.3% | 60.7% |
 
-- **Restraint** is the share of already-correct transcripts returned byte for byte.
-- **Edit accuracy** is the share of transcripts needing a change that came back exactly right.
-- **Overall** is the unweighted mean of the two.
-- **Content damage** is the share of all transcripts that lost a word the speaker said. Lower is
+- **Restraint**: of the transcripts that were already correct, the share returned byte for byte.
+- **Edit accuracy**: of the transcripts needing a change, the share that came back exactly right.
+- **Overall**: the unweighted mean of the two.
+- **Content damage**: the share of all transcripts that lost a word the speaker said. Lower is
   better.
 
-Overall is the mean rather than plain accuracy because the set is close to evenly split between
+Overall is a mean rather than plain accuracy because the set is close to evenly split between
 transcripts that need an edit and transcripts that do not. Plain accuracy would score a program
 that changes nothing at 45.3%, which reads like it half works. The mean scores that same
 do-nothing strategy at exactly 50.0%, so anything under 50 is worse than leaving the transcript
-alone. Four of the seven systems measured are under it. Two of those look safe if you read only
-the restraint column; they are not, because they alter text they should leave alone without
-reliably making the change that was needed.
+alone. Four of the seven systems here are under it, and two of those look safe if you read only
+the restraint column.
+
+### How scoring works
+
+Three tiers, in the order they run.
+
+**Human-authored references.** Every expected output is written by hand against a written
+specification that predates the cases. There is one annotator, so no inter-annotator agreement
+figure is available.
 
-**On judges.** An earlier draft of this card claimed there was no model judge anywhere in the
-pipeline. That was wrong, and worth correcting rather than quietly deleting. A model-judge tier
-was designed and run: four frontier models from four vendors adjudicated the same disputed
-outputs. They agreed on between 2 and 19 of them, meaning the choice of judge decided the result
-rather than the answers did, so the tier was dropped instead of being tuned until it produced a
-nicer number. Every rate on this page comes from exact string match. No judge score is published.
+**Deterministic scoring.** Exact string match. A pass means the output matched the reference
+character for character, with no partial credit. Every rate on this page comes from this tier. A
+deterministic difference classifier runs alongside it to group the failures by cause.
+
+**Model-based adjudication, evaluated and not adopted.** A tier using frontier models to judge
+disputed outputs was built and run. Four models from four vendors adjudicated the same cases and
+agreed on too few of them to be usable, which meant the choice of judge moved the result more
+than the answers did. The tier was dropped rather than tuned. No judge score is published.
 
 ## Fine-tuning bought the skill and cost no restraint
 
-Against the untuned `Qwen/Qwen3.5-0.8B` at the same file size, same quantisation, same prompt and
-same hardware, edit accuracy goes from 4.9% to 48.8%. Restraint goes from 89.7% to 92.6%, a
-difference of a single case that this set cannot resolve. Overall gain is +23.4 points, 95% CI
-[+16.3, +30.3].
+Against the untuned `Qwen/Qwen3.5-0.8B` at the same file size, quantisation, prompt and hardware,
+edit accuracy goes from 4.9% to 48.8%. Restraint goes from 89.7% to 92.6%, a difference of a
+single case that this set cannot resolve. Overall gain is +23.4 points, 95% CI [+16.3, +30.3].
 
 The usual expectation is that teaching a model to edit costs restraint. Here it cost nothing
 measurable.
 
 ## Against a frontier model: a tie, except on long transcripts
 
-Overall the gap is +5.8 points, 95% CI [-1.5, +12.9]. That interval contains zero, so it is a
-tie, and calling it anything else would not survive scrutiny.
+Overall the gap is +5.8 points, 95% CI [-1.5, +12.9]. The interval contains zero, so it is a tie.
 
-Transcript length is what separates them. On passages of 500 words and up the gap is +20.9
-points, 95% CI [+7.0, +35.6]. Long passages are what people actually dictate.
+Length is what separates them. On passages of 500 words and up the gap is +20.9 points, 95% CI
+[+7.0, +35.6]. Long passages are what people dictate.
 
 ## Speed
 
-One desktop CPU, Ryzen 7 7700X, 8 threads, reasoning off:
+Ryzen 7 7700X, 8 threads, reasoning off:
 
 | | median latency | p95 | decode |
 |---|---|---|---|
 | CPU | 2,509 ms | 14,821 ms | 54 tok/s |
 | CUDA, `--n-gpu-layers 99` | 311 ms | 1,909 ms | 342 tok/s |
 
-8.1x on median latency at the same restraint and the same edit accuracy. Every published rate is
-the CPU rate.
-
-Determinism is a within-backend claim. Three CPU runs are byte-identical to each other. CPU
-against CUDA is not: 1.3% of outputs differ, because the two kernels do not produce bit-identical
-logits and a near-tie can resolve the other way.
-
-## What each quantisation costs
+8.1x on median latency at the same restraint and edit accuracy. Every published rate is the CPU
+rate. Determinism is within-backend: three CPU runs are byte-identical to each other, while CPU
+against CUDA differs on 1.3% of outputs because the two kernels do not produce bit-identical
+logits.
 
-Most GGUF repos ship a size ladder and leave you to guess what the small end costs. Every file
-here was run through the same evaluation, so you do not have to guess.
+## Which file to download
 
 | file | size | overall | restraint | edit accuracy | outputs differing from Q8_0 |
 |---|---|---|---|---|---|
@@ -233,75 +255,49 @@ here was run through the same evaluation, so you do not have to guess.
 | Q5_K_M | 593 MB | 70.0% | 91.2% | 48.8% | 2.0% |
 | Q4_K_M | 542 MB | 69.4% | 91.2% | 47.6% | 7.3% |
 
-One pass each, all on one RTX 4070 Ti SUPER with `--n-gpu-layers 99` and every other setting held
-identical.
-
-**Take Q8_0 unless you have a reason not to.** It is the build every other number on this page
-describes, and the app downloads it by default.
-
-**This evaluation cannot tell Q4_K_M and Q8_0 apart.** The 1.3 point gap is one case on each
-axis, and the 95% intervals overlap almost entirely: restraint [83.9%, 96.8%] against
-[82.1%, 95.9%], edit accuracy [38.3%, 59.4%] against [37.1%, 58.2%]. Read that row as a null
-result, not as a measured cost. What is real, because it is a count and not an estimate, is that
-7.3% of outputs changed. Q4_K_M demonstrably changes what this model says. It does not
-demonstrably change how often it is right.
-
-In daily use I have noticed Q4_K_M come back slightly worse often enough to prefer Q8_0. That is
-an anecdote and it stays labelled as one, but it points the same way as the changed outputs.
+Every file was run through the same evaluation, so the ladder states its own cost. One pass each,
+all on one GPU with every setting held identical.
 
-**BF16 is the source, not an upgrade.** It is 724 MB larger than Q8_0 and scores the same. Its
-value is that you can quantise it yourself.
+**Take Q8_0.** It is the build the rest of this page describes and the one the app downloads.
 
-### Why the small quantisations hold up
-
-**Q4_K_M is not 4-bit.** In this file 52.5% of parameters are at 4 bits and 47.4% are at 6.5. The
-tensors kept at higher precision are the attention QKV projections, the FFN down projections, and
-`token_embd.weight`, which is 254 million parameters on its own and is tied to the output head.
-The matrix that decides which token comes out kept 6.5 bits while the label on the file says four.
-
-**Most of the scoring rewards copying.** Returning an already-correct transcript unchanged means
-emitting the token that is already in context, and the logit margin on a copy is large.
-Quantisation noise has to clear a wide gap to flip one. An evaluation weighted toward
-already-correct text is therefore less sensitive to quantisation than one that generates freely,
-and nothing here measures the compounding you would see over hundreds of generated tokens. This
-table is evidence about dictation cleanup, not about 4-bit quantisation in general.
-
-**The remaining difficulty is not the kind quantisation makes worse.** Edit accuracy is 48.8%, so
-the model misses more than half the changes it is asked to make. The task is not easy. It is hard
-in a way that depends on knowing which change the speaker wanted, and that knowledge either
-survives the weight noise or it does not.
+The spread across the ladder is smaller than this evaluation can resolve. Q4_K_M is 1.3 points
+behind on overall, which is one case on each axis, and the 95% intervals overlap almost entirely.
+What is a count rather than an estimate is that 7.3% of its outputs differ from Q8_0. So Q4_K_M
+demonstrably changes what the model says without demonstrably changing how often it is right, and
+in daily use I prefer Q8_0. BF16 is here as the source for anyone who wants to quantise it
+themselves, not as an upgrade: it is 724 MB larger and scores the same.
 
 ## Limitations
 
-**English only.** The specification, the training data, the evaluation and every number here are
-English. The only non-English behaviour taught or scored is leaving it alone, under an absolute
-rule never to translate. The base tokenizer covers languages this work does not, so the model
-will produce output for non-English input that nothing here evaluates.
+**Deterministic fixes are out of scope on purpose.** Filler removal, punctuation, capitalisation
+and number formatting are not trained into this model, because
+[the rules layer](https://github.com/AbhishekBarali/dictation-cleanup-rules) already does them and
+a rule is faster and auditable. Run the two together.
 
-**It is not a recogniser.** It runs on text, after transcription, and cannot recover a word the
-recogniser misheard.
+**It works on text.** If the transcription heard the wrong word and nothing in the sentence gives
+that away, this model cannot recover it.
 
 **It is not a rewriter.** Register, tone and verbosity are out of scope by design. Changing how
 formally someone speaks means replacing words they said.
 
-**It was trained on the specification it is scored against.** Competitors get that convention
-described in a prompt; this model was trained on it. That is the treatment being measured, not a
-hidden confound, but the correct reading is that training on a written specification beats
-prompting for it. Not that this model is better at English than a frontier model.
+**English.** The specification, the training data and every number here are English. The only
+non-English behaviour taught or scored is leaving it alone, under an absolute rule never to
+translate. The base tokenizer covers languages this work does not, so the model will produce
+output for non-English input that nothing here evaluates.
 
-**The evaluation and the model come from the same person.** The mitigation is that the scoring
-code is deterministic and the audit is honest: an internal review found a number of scored cases
-defective or disputed, and the sensitivity analysis with those removed puts the hosted frontier
-model's edit accuracy *above* this model's, 55.9% against 54.4%.
+**It was trained on the specification it is scored against.** Competitors get that convention
+described in a prompt; this model was trained on it. The correct reading is that training on a
+written specification beats prompting for it, not that this model is better at English than a
+frontier model.
 
-**One annotator**, so there is no inter-annotator agreement figure. The written specification
-predates the cases, which is the weaker thing that stands in its place.
+**The evaluation and the model share an author.** An internal audit found a number of scored cases
+defective or disputed. With those removed, the hosted frontier model's edit accuracy comes out
+*above* this model's, 55.9% against 54.4%. That is published because it is true.
 
-**The set cannot separate systems inside about six points.** Treat any gap smaller than that as
-a tie, including the one against GPT-5.6 Luna.
+**Gaps under about six points are not resolvable** by this set, including the one against GPT-5.6
+Luna.
 
-**10.7% of transcripts lose a word the speaker said.** That is the number to beat and it is not
-small.
+**10.7% of transcripts lose a word the speaker said.** That is the number to beat.
 
 ## Provenance
 
@@ -309,7 +305,7 @@ small.
 - Method: LoRA, rank 16, merged before quantisation
 - Reference build: Q8_0, 833,591,776 bytes
 - SHA-256: `696769bb6911f51bc231b112926e934cf7bfc760e6cdfa24212907bc5ad41fc9`
-- The other quantisations come from the same checkpoint with an importance matrix
+- Other quantisations come from the same checkpoint with an importance matrix
 
 ## Citation