- SonoJot Punctuation v4
SonoJot Punctuation v4
A 140M-parameter token classifier that repairs speech-recognizer output in one forward pass: punctuation, written-form spans, optional removal of fillers and repeats, and, new in v4, the case of each word. It runs on a desktop CPU inside SonoJot, a local dictation app, in about 0.1 s per 300 words on two threads, from a 147 MB file with 8-bit weights. It never adds, reorders or respells a word; only removal (which the app can switch off) and letter case change the words.
1. Task
Speech recognizers differ in how they write case. Some capitalise the first word of every separately decoded piece of audio ("a feature" + "Request for the export"); some write everything in lower case; some capitalise names well and others never do. Earlier SonoJot packages read every word lower-cased and left casing to rules, which could only lower capitals, never add them, and lowered real names that are also common words (Summit, Rust, Bill).
v4 makes four predictions per word:
- Punctuation. The mark after the word: none,
.,,,?or;. The recognizer's own marks are input evidence. - Written-form spans. BIO tags over 13 classes (27 labels) for spans a person would type differently from how
they are spoken (
four thousand dollarsβ$4,000). A deterministic renderer in SonoJot writes the value. - Removal (optional). KEEP/DELETE for fillers, repeats and restarts. SonoJot applies it only in its Clean mode and only above a calibrated threshold; Format mode and meetings keep every word.
- Casing (new). lower, Capitalised or ALL CAPS: the word's own case as it would appear inside a sentence.
Words are still read lower-cased. Two small side inputs carry what the recognizer knew: its own case of the word (unknown, lower, Capitalised, UPPER, mixed) and whether the word starts a separately decoded audio piece. During training the case input was hidden for about 30% of rows, so the model also works when the recognizer gives no case.
How SonoJot applies the casing output. The model proposes; fixed rules decide.
- A lower-case word gains a capital at P β₯ 0.80, or becomes ALL CAPS (2 to 6 letters) at P β₯ 0.70.
- An existing capital is lowered only at P β₯ 0.98, or P β₯ 0.80 when the capital is merely positional (after a sentence end the model removed, or at the start of an audio piece). Turning a capital into ALL CAPS needs 0.98.
- Never changed: vocabulary terms, code, URLs, protected literals, mixed-case words, words in a run of capitals ("Black Lives Matter", looking past up to two connector words), a single capital letter inside a sentence, "I", and sentence starts (always capitalised).
- By rule: a closed-class word (pronoun, conjunction, determiner, preposition, auxiliary, discourse word) loses a stray capital inside a sentence; a short list of mixed-case names (GitHub, iPhone, macOS) gets its official form.
- Only letter case may change; SonoJot rejects any output whose lower-cased words differ from its input.
The thresholds were chosen on public development data only (Section 5), and ship in manifest.json.
2. Model
| Architecture | mmBERT-small encoder; four linear heads on the hidden state of each word's last subword: punctuation (5), spans (27 BIO), delete (2), casing (3) |
| Side inputs | One embedding table (11 Γ 384, zero-initialised) added to the token embeddings: hint id = 1 + 2Β·(case β 1) + piece start for every subword of a word, 0 for special and mark tokens |
| Parameters | About 140M; v4 adds about 5k parameters to v3 |
| Initialization | SonoJot Cleanup v4 (unreleased; v3 plus the delete head), itself continued from v3 |
| Runtime file | model.int8.onnx, 147 MB: int8 token embeddings; every other weight matrix stored as 8-bit integers in blocks of 32 values along the input dimension with one fp32 scale per block (ONNX Runtime MatMulNBits, com.microsoft domain, bits=8, block_size=32, accuracy_level=4, so activations are also rounded to int8 per block of 32); inputs input_ids, attention_mask, case_hints |
| Runtime requirement | ONNX Runtime 1.22 or later on the CPU provider (8-bit MatMulNBits; 1.18 to 1.21 refuse the graph at load). Measured on 1.28; onnxruntime-web 1.31 (wasm) also runs it |
| Speed | Per 300 words warm (median of 30), ONNX Runtime 1.28, Ryzen 9 9900X: about 0.12 s on 2 threads and 0.07 s on 4, 1.2 to 1.3 times the full-precision file on the same machine (timings on this machine vary by up to 30% between sessions). Apple M4, 2 threads: 0.085 s (full precision 0.070 s) |
| Language | English |
Long texts are split into windows of 120 words with a hop of 100, so each word is read about 1.2 times instead of twice; each word takes its predictions from the window in which it is farthest from an edge. On the evaluation matrix the hop-100 output is level with hop 60 within noise, except Whisper sentence F1 on the dictations (β0.002 [β0.003, β0.000]). See Input encoding.
3. Training data
The punctuation, written-form and removal heads were held to cleanup v4: training replayed v4's own training windows with their labels, and every word of every row carried a distillation target from v4's predictions (Section 4). The casing head was trained on recognizer output aligned to cased published text:
| Source | Contents | Licence | Recognizers |
|---|---|---|---|
| LibriHeavy (k2-fsa) | 17,982 documents of up to six consecutive 17 s windows of LibriVox readings, book text as reference | Audio public domain; metadata and text Apache-2.0 | Qwen3-ASR 1.7B, Parakeet TDT 0.6B v2 |
| YODAS (espnet), manual captions | 14,314 documents from 3,041 videos; captions filtered to English, cased and punctuated | CC BY 3.0 (per video; list in yodas_videos.tsv) |
as above |
| AMI Meeting Corpus | 5,075 documents, headset and single distant microphone, word-timed transcripts | CC BY 4.0 | as above, plus Granite Speech 4.1 (lower case) on a subset |
| VoxPopuli | 8,475 single windows of European Parliament speech, raw cased transcription | CC0 1.0 | as above |
| Synthetic variants | Lower-cased copies without marks (35% of rows), capitals and false periods at random piece starts (30% of Qwen rows), hidden case input (30% of all rows) | Derived |
Evaluation groups (AMI test meetings, NOTSOFAR and YODAS evaluation videos) were excluded from training. The replayed v4 windows include Earnings-21/22 (CC BY-SA 4.0), AMI/ICSI (CC BY 4.0), synthetic written-form and correction text, and one consenting user's dictation (not distributed). Whisper large-v3-turbo output and the test dictations were never used in training.
4. Labels
- Projection. Each recognizer's words were aligned to the reference words (exact lower-cased match, the same alignment SonoJot uses for punctuation labels), and a matched word took the reference word's case.
- Masked. Reference sentence starts, the first word of every reference window, words after a quote or colon, "I" forms, words with digits, mixed case, ALL CAPS longer than six letters or on common words, and, in captions, meetings and parliament transcripts, an isolated capital on a common word (caption-line and utterance starts capitalise ordinary words). A capital inside a run of capitals is kept as a name even on a common word.
- Distillation. Loss = casing CE (class weights 1/2/3) + v4-label CE on replay rows (punctuation + 3Β·spans + 0.5Β·delete) + 4Β·TΒ²Β·KL(v4 β model) on punctuation, 3Β·spans and delete for every word, T = 2.
5. Evaluation protocol
Test sets.
- 172 dictations (22,190 words, 3.6 h) recorded by the in-domain speaker between 4 and 13 November 2026 with audio kept; test only. References: an automatic first pass drafted punctuation and case on the SonoJot Parakeet transcript without changing words; a second, independent automatic annotation decided case blind to that draft; the two agreed on 99.77% of 20,291 mid-sentence words; all 47 disagreements were adjudicated (43 kept the draft, 4 masked as undecidable), and 86 words whose case cannot be decided from the text are masked. The punctuation of these references was not independently checked, so punctuation is gated on the 88 adjudicated reserved dictations instead.
- Public: AMI headset test (133 segments), NOTSOFAR close-talk dev (300 sampled segments), held-out YODAS long-form videos (200 segments). Utterance-start capitals that these references give ordinary words without any sentence mark are masked (an isolated capitalised common word with no capitalised neighbour).
- Gate sets for the other heads: the 88 reserved dictations with adjudicated punctuation; the 88 dictation and 114 meeting documents with adjudicated removal labels; the 763-case written-form challenge.
Recognizers. Every test recording was decoded with Qwen3-ASR 1.7B (Q8, CUDA), SonoJot Parakeet (TDT 0.6B v2
fine-tune, with SonoJot's pause cutting), stock Parakeet TDT 0.6B v2, Moonshine medium, and Whisper large-v3-turbo,
the recognizer held out from training. A sixth row lower-cases the SonoJot Parakeet text and drops its marks, as a
recognizer without case or punctuation would give it. All text went through SonoJot's own cleanup path (rules, then
the package, Clean mode, written form on), replayed with fidelity_eval.
Metrics. Words are aligned to the reference; a word is scored for case when it is inside a sentence in both the reference and the output.
- Casing errors per 1,000 words: output spelling differs from the reference spelling.
- Harm per 1,000 words: the recognizer wrote the reference's capital and the output lost it; content harm excludes closed-class words.
- Proper-noun recall: reference capitals on words that are not common lower-case words.
- Sentence and comma F1 at word positions; words deleted per 1,000.
- 95% intervals: document bootstrap (500 draws for the matrix, 2,000 for the gate), paired against cleanup v4.
Selection. Three seeds; the released seed has the best development casing F1 (public dev rows), not the best test score. Casing thresholds: grid search on public development rows minimising errors with harm from threshold decisions at most 0.05 per 1,000 for every recognizer group (a quarter of the 0.2 budget, because development data under-measures harm on names made of common words). The delete threshold (0.955) matches cleanup v4's development wrong-deletion rate at its own threshold (0.975).
6. Results
Sections 6 and 7 were measured with full-precision compute (the 268 MB file of 13 and 14 November). The released 8-bit file is level with it on every measure; see Eight-bit release.
Casing, 172 dictations (per 1,000 words)
| Recognizer | Errors, v3 rules | cleanup v4 rules | v4 casing | Ξ vs cleanup v4 [95% CI] | Harm v4 rules β v4 casing | Proper-noun recall |
|---|---|---|---|---|---|---|
| Qwen3-ASR 1.7B | 2.60 | 2.56 | 2.02 | β0.54 [β1.05, β0.08] | 0.13 β 0.04 | 0.98 β 0.98 |
| SonoJot Parakeet | 3.99 | 3.90 | 2.96 | β0.94 [β1.56, β0.45] | 0.22 β 0.00 | 0.97 β 0.97 |
| Parakeet TDT 0.6B v2 | 2.03 | 2.03 | 1.49 | β0.54 [β0.96, β0.18] | 0.00 β 0.00 | 0.97 β 0.98 |
| Moonshine medium | 10.81 | 10.27 | 10.36 | +0.09 [β1.83, +1.94] | 0.36 β 0.04 | 0.81 β 0.91 |
| Whisper large-v3-turbo (held out) | 3.18 | 3.14 | 2.24 | β0.90 [β1.66, β0.29] | 0.18 β 0.04 | 0.95 β 0.98 |
| SonoJot Parakeet, lower-cased | 18.21 | 17.99 | 8.57 | β9.42 [β11.35, β7.64] | 0.00 β 0.00 | 0.09 β 0.72 |
Pooled over the six rows, casing errors fall 31% against cleanup v4 with its rules. Part of the dictation gain on "option A"-style labels comes from a rule change in SonoJot that applies to every package (a single capital A inside a sentence keeps its capital); cleanup v4 with that change scores 2.42, 3.72, 2.03, 11.30, 2.96 and 17.99.
Casing, public test sets (per 1,000 words; cleanup v4 β v4 casing)
| Recognizer | AMI headset | NOTSOFAR close-talk | YODAS videos |
|---|---|---|---|
| Qwen3-ASR 1.7B | 2.82 β 2.57 | 3.67 β 3.15 | 13.33 β 14.54 |
| SonoJot Parakeet | 3.49 β 2.64 | 3.15 β 3.25 | 13.57 β 14.54 |
| Parakeet TDT 0.6B v2 | 2.83 β 2.13 | 3.07 β 3.18 | 13.93 β 14.90 |
| Moonshine medium | 8.03 β 4.72 | 5.46 β 4.41 | 17.27 β 17.51 |
| Whisper large-v3-turbo (held out) | 2.51 β 1.96 | 2.83 β 2.41 | 11.99 β 11.87 |
| SonoJot Parakeet, lower-cased | 5.39 β 2.70 | 8.39 β 4.41 | 59.61 β 23.38 |
Content harm on the public sets is at most 0.12 per 1,000 for every cell. On YODAS, cased recognizers lose about one point to false capitals on adjectives next to names ("Republican states", "evangelical Christianity"); proper-noun recall rises on AMI (for example 0.69 β 0.83 for Qwen) and is level elsewhere.
The other heads, against cleanup v4 (released seed; second seed in brackets)
| Measure | cleanup v4 | v4 | Paired Ξ [95% CI] |
|---|---|---|---|
| Sentence F1, 88 reserved dictations | 0.910 | 0.909 (0.908) | β0.001 [β0.004, +0.002] |
| Comma F1, 88 reserved dictations | 0.897 | 0.898 (0.898) | +0.001 [β0.003, +0.004] |
| Removal recall, 88 dictations | 0.769 | 0.769 (0.765) | 0.000 [β0.009, +0.009] |
| Wrong deletions per 1,000, 88 dictations | 1.10 | 1.03 (1.13) | β0.07 [β0.28, +0.20] |
| Removal recall, 114 meeting documents | 0.684 | 0.693 (0.685) | +0.009 [β0.001, +0.020] |
| Wrong deletions per 1,000, meetings | 2.60 | 2.80 (2.60) | +0.21 [β0.20, +0.60] |
| Written-form challenge, exact of 763 | 662 | 659 (657) | (cleanup v4 three-seed mean 656) |
These cleanup-v4 numbers come from this evaluation's scorer, which reproduces the cleanup-v4 card (0.905/0.897, 1.03, 2.60, 661) within rounding of the scoring method.
7. Ablations and negative results
| Experiment | Result |
|---|---|
| First recipe, distillation weight 1 | Casing learned, but the removal head drifted cautious: meeting recall β0.040 (significant) |
| Masking every capital on a common word in captions and meetings | The model learned that common words are lower case and lowered title-case names ("Black Lives Matter"); fixed by keeping capitals inside runs |
| A capital run that crosses punctuation | A sentence-final name shielded the next sentence's stray capital ("Paris. And"); runs now stop at punctuation |
| Positional lowering at 0.80 instead of 0.90 | More harm on the in-domain speaker's product names; 0.90 chosen on development data with the quarter-budget rule |
| Two-epoch training on a shared 12 GB GPU | 2.4 GB peak with frozen token embeddings and gradient checkpointing; about 40 minutes per seed |
Revision of 14 November 2026
The released checkpoint was fine-tuned for one more epoch from the 13 November weights, distilled to them on all four outputs, with 3,000 invented sentences that mark colloquial hundreds ("twelve hundred dollars" β $1,200, "eighteen hundred chairs" β 1,800) and keep cases ("a few hundred", "hundreds of"). SonoJot's renderer gained the matching number form. Against the 13 November weights, on the gate and both matrices every row is level within noise: punctuation identical (sentence F1 0.911, comma 0.899), wrong deletions 0.99 against 1.03 per 1,000, written-form challenge 660 against 659. Thresholds were re-chosen on development data with the same rule.
Eight-bit release
The released model.int8.onnx is the 14 November checkpoint with 8-bit weights in blocks of 32 (Section 2): 147 MB
instead of 268 MB, so the package download falls from 303 MB to 181 MB. Same checkpoint, same thresholds. Against
the full-precision file of that checkpoint, paired (document bootstrap, 2,000 draws):
| Measure | Full precision | 8-bit | Paired Ξ [95% CI] |
|---|---|---|---|
| Sentence F1, 88 reserved dictations | 0.9115 | 0.9108 | β0.0007 [β0.003, +0.001] |
| Comma F1, 88 reserved dictations | 0.8985 | 0.8982 | β0.0003 [β0.002, +0.001] |
| Removal recall, 88 dictations | 0.769 | 0.767 | β0.0018 [β0.006, +0.002] |
| Wrong deletions per 1,000, 88 dictations | 0.96 | 0.99 | +0.03 [β0.08, +0.16] |
| Removal recall, 114 meeting documents | 0.694 | 0.691 | β0.0025 [β0.006, 0.000] |
| Wrong deletions per 1,000, meetings | 2.74 | 2.67 | β0.07 [β0.22, 0.00] |
| Written-form challenge, exact of 763 | 660 | 660 | |
| Casing label errors per 1,000 (the model's own label, before the rules), public development rows | 9.80 | 9.68 | β0.12 [β0.20, β0.04] |
Casing errors per 1,000 words on the 172 dictations, full precision β 8-bit: Qwen3-ASR 2.20 β 2.20, SonoJot Parakeet 3.09 β 3.05, Parakeet TDT 0.6B v2 1.43 β 1.43, Moonshine 9.91 β 9.87, Whisper large-v3-turbo (held out) 2.11 β 2.20, SonoJot Parakeet lower-cased 8.61 β 8.70. Across the whole evaluation matrix (six recognizer rows on the dictations and on AMI, NOTSOFAR and YODAS: 24 cells, each with casing errors, harm, sentence F1, comma F1 and words deleted), no measure is worse than full precision and two are better. On 102,242 words of public and synthetic rows, the 8-bit file takes the same punctuation decision as full precision for 99.91% of words, the same removal decision for 99.97% and the same casing label for 99.91%; with Apple Silicon's ONNX Runtime 1.28.2 the figures are 99.92%, 99.97% and 99.92%, and the full-precision file gives identical decisions on both platforms.
Plain dynamic int8 quantization (one activation range per tensor) was rejected: one channel of the input to one feed-forward down-projection reaches about 9,400, where the same input in other layers peaks near 10 to 40, so per-tensor rounding loses punctuation and removal accuracy. Rounding activations per block of 32, as this file does, keeps them.
8. Known failure cases
- Product names split into ordinary words ("Test Flight", "Said Frame", "T3 Code") stay lower case unless the recognizer wrote them capitalised; SonoJot's vocabulary is the fix.
- Adjectives next to names can gain a false capital ("Chinese goldfield").
- Moonshine output barely improves: its errors are mostly product names it writes in lower case.
- Reference conventions. Caption and meeting transcripts capitalise utterance starts; scores on them include that noise even after masking.
9. Intended use and limitations
For English speech-recognizer output before it is pasted as text, on local hardware, with any recognizer. Not a general truecaser for edited text, not for other languages, not a grammar corrector.
- Speakers. All in-domain dictation, training and test, is one consenting speaker's.
- Test size. 172 dictations and a few hundred public segments; differences of a few tenths per 1,000 words are within noise.
- Thresholds in
manifest.jsonare part of the model; other operating points are untested.
10. Reproducibility
- Code: the replay tool behind every matrix score (
fidelity_eval), the casing rules and the written-form renderer are in SonoJot's public repository, https://github.com/Cody-Stevens/sonojot (rewrite/src-tauri/src/bin/fidelity_eval.rs,rewrite/src-tauri/src/punctuation/casing.rs,rewrite/src-tauri/src/written_form/). The Python data, training and scoring scripts are not published. - Recipe: AdamW lr 2e-5, weight decay 0.01, OneCycle with 6% warm-up, batch 16, 2 epochs, bf16 autocast, gradient clipping 1.0, frozen token embeddings, 65% of casing rows and 50% of v4 windows sampled per seed. RTX 5070.
- Export: CPU, eager attention, opset 17, int8 Gather weights; then every weight MatMul converted to 8-bit
MatMulNBits(symmetric, blocks of 32 along the input dimension, zero point 128,accuracy_level=4). On authored test sentences the 8-bit file and the PyTorch weights agree on 491 of 492 punctuation decisions, every span and removal decision, and 490 of 492 casing labels (verification.json, which also records the parity figures above). - Integrity:
manifest.jsonlists sizes and SHA-256 values; SonoJot verifies them before loading.
Input encoding
As v3, plus case_hints: CLS, then for each word the tokens of " " + word (lower-cased except I-forms) followed by
the recognizer's mark tokens, then SEP. Every subword of a word carries hint id 1 + 2Β·(case β 1) + piece_start, case
being the recognizer's case of the word (1 unknown, 2 lower, 3 Capitalised, 4 UPPER, 5 mixed); CLS, SEP and mark
tokens carry 0. Read all four heads at each word's last subword token. Windows of 120 words, hop 100.
Files
model.int8.onnx (8-bit), tokenizer.json, punct-config.json (labels, encodings, casing thresholds for browser
use), manifest.json (sizes, SHA-256, delete and casing thresholds), LICENSE-CC-BY-SA-4.0.txt,
LICENSE-CC-BY-4.0.txt, NOTICE, yodas_videos.tsv, verification.json, and pytorch/ (the full-precision
PyTorch weights of the same checkpoint).
License and attribution
The weights are released under CC BY-SA 4.0, because training labels derive from the Earnings-21 and Earnings-22
transcripts (CC BY-SA 4.0, Β© Rev.com). Further sources: AMI and ICSI Meeting Corpora (CC BY 4.0), YODAS manual
captions (CC BY 3.0, per-video list in yodas_videos.tsv), LibriHeavy (Apache-2.0 text, public-domain LibriVox
audio), VoxPopuli (CC0). Base model chain: Horizon-Labs/punctuation-restoration-small (Apache-2.0) on
jhu-clsp/mmBERT-small (MIT). Keep this attribution and the share-alike terms when redistributing.
Citation
@misc{stevens2026sonojotpunctv4,
author = {Stevens, Cody},
title = {SonoJot Punctuation v4},
year = {2026},
publisher = {SonoJot},
howpublished = {\url{https://huggingface.co/SonoJot/sonojot-punct-v4}}
}
Model tree for SonoJot/sonojot-punct-v4
Base model
jhu-clsp/mmBERT-small