sderhy commited on
Commit
f659bcd
·
verified ·
1 Parent(s): fd0d666

morseformer v0.6.0 — Phase 5.8 EN-literary + streaming offline decoder

Browse files
Files changed (3) hide show
  1. README.md +64 -16
  2. REPO_README.md +69 -2
  3. rnnt_phase5_8.pt +3 -0
README.md CHANGED
@@ -60,13 +60,60 @@ model-index:
60
  - **Languages on input**: English plus a multilingual prose mix in French, German, and Spanish. **French is now first-class**: É, À, and apostrophe are tokenized natively (Phase 3.4 vocabulary extension); other diacritics still ASCII-normalised (è / ê / ç → E / E / C, German umlauts → AE/OE/UE/SS).
61
  - **Output vocabulary**: **49 tokens** (A–Z, 0–9, space, `. , ? ! / = + -`, plus `É À '`).
62
 
63
- This repository hosts the **v0.5.3 release** of morseformer. New recommended acoustic checkpoint `rnnt_phase5_7.pt` (4.13 M params, 49-vocab) — a 15 k-step fine-tune of the v0.5.2 acoustic on an **amateur-idiom curriculum** (5NN cut-numbers + run-on prosigns `UR / SK / KN / BK`). The v0.5.2 inference-only digit-threshold fix is preserved; the LM artifact (`lm_phase5_2.pt`) is unchanged.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64
 
65
  ## What's new in v0.5.3
66
 
67
  The v0.5.2 IC-7300 live test (2026-05-05, Churchill speech) gave qualitative parity with the `cwget` DSP reference — a [project_ambition](https://github.com/sderhy/morseformer) milestone — but the user transcript exposed three persistent letter-level residuals: `5NN` (the contest-style cut-number for `599`) decoded as `5MN` or `ENN`; the `0`-prefixed callsign `MI0MO` rendered as `MI4MO`; the standard run-on prosigns `UR / SK / KN / BK` split into their constituent letters because the synthetic curriculum had only ever shown them with full inter-letter gaps. Phase 5.7 targets all three with a curriculum delta on top of Phase 5.5.
68
 
69
- - **`rnnt_phase5_7.pt`** — new recommended acoustic. Bootstrap chain: `phase3_5/best → phase5_3/best → phase5_4/last → phase5_5/last → phase5_7/last`. 15 k steps from `phase5_5/last` at peak LR 5e-5, EMA 0.9999, bf16, batch 12, RTX 3060 Laptop, ~2 h wall time.
70
  - **`PHASE_5_7_MIX`** — new text mix that adds 5NN cut-numbers (rendered as the spoken `5NN`, not `599`) and a `_CONTEST_DENSE_TEMPLATES` slice of densely packed contest exchanges around `5NN <NR>`. All other slices are inherited from `PHASE_3_4_MIX`.
71
  - **`OperatorConfig.run_on_pairs`** — new per-utterance knob. A list of digraphs and probabilities; with that probability, the synthesizer collapses the inter-letter gap on the digraph to a single inter-element gap (so `UR` is keyed as `..-.-..`, byte-equivalent to one continuous prosign). Defaults: `UR 0.50, SK 0.85, KN 0.70, BK 0.70`. Old presets pin `run_on_pairs=()` so Phase 3.x / 5.x curricula are byte-identical.
72
  - The channel, jitter, dash:dot, gap-inflation, word-gap-inflation, false-positive prior, and 30 % real-audio mix are kept identical to Phase 5.5 — strict ablation around the curriculum delta.
@@ -155,8 +202,9 @@ The motivating live test (2026-05-02) was a 7-minute hand-keyed `test.wav` with
155
 
156
  | file | params | vocab | description | recommended |
157
  |---|---|---|---|---|
158
- | `rnnt_phase5_7.pt` | 4.13 M | **49** | **v0.5.3 acoustic — Phase 5.7 amateur-idiom curriculum (5NN cut-numbers + run-on UR / SK / KN / BK) on top of v0.5.2. Pair with `--digit-threshold 0.90` and `--confidence-threshold 0.6` for the recommended inference behaviour.** | ✅ |
159
- | `lm_phase5_2.pt` | 4.76 M | **49** | v0.4.1 LM — matched to the Phase 3.5 text mix (multilingual prose + ham radio). val_ppl 5.626. **Use this for shallow fusion at λ = 0.7.** | ✅ |
 
160
  | `rnnt_phase5_5.pt` | 4.13 M | 49 | v0.5.1 / v0.5.2 acoustic — Phase 5.5 (long inter-word silences) on top of v0.5.0. Kept for diff. | |
161
  | `rnnt_phase5_4.pt` | 4.13 M | 49 | v0.5.0 acoustic — Phase 5.3 (wider jitter + dash:dot ratio) + Phase 5.4 (30 % real-audio mix). Kept for diff. | |
162
  | `rnnt_phase3_5.pt` | 4.13 M | 49 | v0.4.0 / v0.4.1 acoustic — synthetic-only. Kept for diff. | |
@@ -184,21 +232,21 @@ Honest about what v0.4 does and does not do:
184
 
185
  ## How to use
186
 
187
- v0.5.3 ships a unified `morseformer` CLI installable directly from PyPI / the GitHub source tree. Four shipped presets bundle a model + thresholds + optional LM behind one flag:
188
 
189
  | Preset | Use case | Model | thr | digit_thr | LM |
190
  |---|---|---|---|---|---|
191
- | `live` (default) | real-time IC-7300 streaming | rnnt_phase5_7 | 0.6 | 0.90 | — |
192
- | `prose` | offline poetry / ragchew / book | rnnt_phase5_7 | 0.6 | 0.90 | lm_phase5_2 (λ=0.7) |
193
- | `contest` | fast contest exchanges | rnnt_phase5_7 | 0.5 | 0.80 | — |
194
- | `conservative` | very noisy bands, prefer silence | rnnt_phase5_7 | 0.75 | 0.95 | — |
195
 
196
  ### Quick start (pip)
197
 
198
  ```bash
199
  pip install "morseformer[ml,audio,hub]"
200
 
201
- # First call auto-downloads rnnt_phase5_7.pt from the Hub (one-time).
202
  morseformer decode my_recording.wav
203
 
204
  # Offline file decode with LM shallow fusion.
@@ -212,7 +260,7 @@ morseformer live --preset conservative
212
  # Inspect / pre-fetch checkpoints.
213
  morseformer models list # recommended pair
214
  morseformer models list --advanced # legacy 46-vocab models too
215
- morseformer models download rnnt_phase5_7
216
  ```
217
 
218
  ### From a git checkout
@@ -231,13 +279,13 @@ morseformer decode my_recording.wav --preset prose
231
 
232
  ```bash
233
  python -m scripts.decode_audio my_recording.wav \
234
- --ckpt release/rnnt_phase5_7.pt \
235
  --lm-ckpt release/lm_phase5_2.pt \
236
  --fusion-weight 0.7 \
237
  --confidence-threshold 0.6 \
238
  --digit-threshold 0.90
239
 
240
- python -m scripts.decode_live --ckpt release/rnnt_phase5_7.pt
241
  ```
242
 
243
  Tune your receiver to zero-beat at 600 Hz with a ≈ 500 Hz CW filter. `Ctrl+C` to quit. Latency is ~4 s end-to-end. **LM fusion is offline-only**: the streaming decoder still uses the acoustic-only greedy path; plumbing fusion through the central-zone-commit logic is a separate piece of work earmarked for a follow-up release. The `prose` preset's LM is therefore silently dropped when used with `morseformer live`.
@@ -367,7 +415,7 @@ Training budget for v0.4 (everything, Phase 0 → Phase 3.5): roughly 76 h of si
367
 
368
  ## Technical specifications
369
 
370
- **Acoustic model** (`rnnt_phase5_7.pt`, 4.13 M params, identical architecture to v0.1 / v0.2 / v0.3 except for the wider output head; v0.5.x differs from v0.4.x only in training data):
371
  - Encoder: Conformer d=144, L=8, H=4, ff_expansion=4, conv_kernel=31, RoPE, LayerNorm conv, 4× subsample
372
  - CTC head: Linear(144 → 49)
373
  - PredictionNetwork: Embedding(49, 128) + LSTM(128, 128, 1 layer)
@@ -382,11 +430,11 @@ Training budget for v0.4 (everything, Phase 0 → Phase 3.5): roughly 76 h of si
382
  ## Citation
383
 
384
  ```bibtex
385
- @software{morseformer_v0_5_4_2026,
386
  author = {Derhy, Serge},
387
  title = {morseformer: open-source transformer-based Morse / CW decoder},
388
  year = 2026,
389
- version = {v0.5.4},
390
  url = {https://github.com/sderhy/morseformer},
391
  howpublished = {\url{https://huggingface.co/sderhy/morseformer}},
392
  }
 
60
  - **Languages on input**: English plus a multilingual prose mix in French, German, and Spanish. **French is now first-class**: É, À, and apostrophe are tokenized natively (Phase 3.4 vocabulary extension); other diacritics still ASCII-normalised (è / ê / ç → E / E / C, German umlauts → AE/OE/UE/SS).
61
  - **Output vocabulary**: **49 tokens** (A–Z, 0–9, space, `. , ? ! / = + -`, plus `É À '`).
62
 
63
+ This repository hosts the **v0.6.0 release** of morseformer. New recommended acoustic checkpoint `rnnt_phase5_8.pt` (4.13 M params, 49-vocab) — a 20 k-step fine-tune from `phase5_5/last` on an **English-literary curriculum** (Moby Dick + Pride & Prejudice + Sherlock Holmes + Frankenstein, 25 % weight). Pairs with a structural **streaming offline decoder** in `decode_audio` that eliminates the v0.5.4 LCWO boundary word-cut artefacts. The v0.5.2 inference-only digit-threshold fix is preserved; the LM artifact (`lm_phase5_2.pt`) is unchanged.
64
+
65
+ ## What's new in v0.6.0
66
+
67
+ The v0.5.4 LCWO live test (2026-05-08, 3 hand-keyed-style samples: De Gaulle FR oratory + Pacino EN dialogue at 24 / 20 WPM, ~25 min total) confirmed morseformer produces fully readable transcripts on out-of-domain literary prose, but flagged two systematic issues. The dominant visible failure was **window-boundary word cuts** — `MARTYRISÉ → MARTYRISÉ E`, `L'APPUI → L'AP PUI`, `INSTITUTION → INSTITUTIO E`, `WANDERED → WANDERE D`. This was structural: `decode_audio` cut audio into non-overlapping 6 s chunks. The secondary issue was acoustic / textual: long-form English literary narrative was thin in the synthetic mix (mostly QSO + multilingual cross-section). v0.6.0 addresses both.
68
+
69
+ ### Streaming offline decoder (structural)
70
+
71
+ `decode_audio` now uses the same central-zone-commit sliding window the live streaming decoder has used since v0.2, applied offline. Adjacent commit zones tile the audio without gap or overlap, so words never fall in the middle of a "fresh" boundary. Diff on the v0.5.4 `test_8k.wav` Daffodils transcript:
72
+
73
+ ```
74
+ v0.5.4 chunked: I WANDERE D LONELY L A CLOU E THAT FLOA S ON HIGH …
75
+ GM ALDEN DAV DAFI RODILS, SESIDE THE LAD N, BENEATS T EHE TLES,
76
+ L F F FLUN TERING AND DN E DPCING IN THE BREEZE.
77
+
78
+ v0.6.0 streaming: IWANDERED LONELY AS ALOUD THAT FLOATS ON HIGH …
79
+ GOLDEN DAV DAFFODILS, BESIDE THE LAC, BENEATS THE TLES,
80
+ LF F FLUTTERING AND D DACING IN THE BREEZE.
81
+ ```
82
+
83
+ The structural patch alone — independent of any acoustic change — restores readability of long-form prose. New flag `--legacy-chunking` reverts to v0.5.4 behaviour for diagnostic.
84
+
85
+ ### Phase 5.8 — English-literary curriculum (acoustic)
86
+
87
+ `rnnt_phase5_8.pt` is a 20 k-step fine-tune from `phase5_5/last` (NOT from `phase5_7/last`, per the carryover note in `project_phase5_7_result.md`) on a new text mix `PHASE_5_8_MIX`:
88
+
89
+ - New `prose_en` slice at 25 % weight, sampled exclusively from the four EN Gutenberg books — Moby Dick (~42 % of EN), Pride & Prejudice, Sherlock Holmes, Frankenstein. ~3 M chars EN total.
90
+ - `contest_dense` cut from 30 % to 13 % to free room for `prose_en` while preserving the 5NN gain from Phase 5.7.
91
+ - `prose_fr` preserved at 10 % so French diacritic gradient is not lost.
92
+ - `operator_run_on_pairs` halved (`UR 0.50→0.25, SK 0.85→0.50, KN 0.70→0.30, BK 0.70→0.30`) to reduce phantom-prosign emissions on continuous prose, which the v0.5.4 LCWO report flagged as a Phase 5.7 cost.
93
+ - Channel / jitter / dash:dot / gap-inflation / word-gap-inflation / FP-prior / 30 % real-audio mix kept identical to Phase 5.5.
94
+
95
+ Bootstrap chain: `phase3_5/best → phase5_3/best → phase5_4/last → phase5_5/last → phase5_8/last`. 20 k steps, peak LR 5e-5, EMA 0.9999, bf16, batch 12, ~1 h 50 min on RTX 3060 Laptop. As with every release since v0.5.1, the dormant `ValidationConfig.matching()` clean-CW val signal saturates fast (< step 1 k) — `last.pt` is the released checkpoint, not `best_rnnt.pt`.
96
+
97
+ Phase 5.8 vs Phase 5.7 incremental wins on the same Daffodils transcript:
98
+
99
+ | Token | v0.5.3 streaming | **v0.6.0 streaming** |
100
+ |---|---|---|
101
+ | "FLUTTERING" | `FLUTERING` | `FLUTTERING` ✓ (proper double-T) |
102
+ | "A CLOUD" | `ALLOUD` | `ALOUD` (closer) |
103
+ | "DANCING" | `DNCINNG` | `DACING` (closer) |
104
+
105
+ ### Caveats
106
+
107
+ - Phase 5.8 is incremental on the synthetic distribution. The motivating gain — long-form EN literary prose, e.g. Pacino *Scent of a Woman* — was not formally CER-benched for v0.6.0. The qualitative diff above is on a single in-distribution audio. Live evaluation on user-provided LCWO samples is the real target.
108
+ - `prose` preset's LM (`lm_phase5_2`, matched to `PHASE_3_4_MIX`) was reported by the v0.5.4 LCWO test as actively *hurting* on FR oratory (De Gaulle: `L'APPUI → LAEC LAGAP PUI` vs `L'AP PUI` in `live` preset). v0.6.0 does not change the LM; use `live` preset for FR oratory until Phase 4.2 ships a domain-matched LM.
109
+ - Spurious prosigns at chunk boundaries (`K`, `+`, `BK`) are reduced (run-on probabilities halved) but not eliminated.
110
+ - ILME / density-ratio fusion remains catastrophic on this distribution; do **not** pass `--ilm-weight > 0`.
111
 
112
  ## What's new in v0.5.3
113
 
114
  The v0.5.2 IC-7300 live test (2026-05-05, Churchill speech) gave qualitative parity with the `cwget` DSP reference — a [project_ambition](https://github.com/sderhy/morseformer) milestone — but the user transcript exposed three persistent letter-level residuals: `5NN` (the contest-style cut-number for `599`) decoded as `5MN` or `ENN`; the `0`-prefixed callsign `MI0MO` rendered as `MI4MO`; the standard run-on prosigns `UR / SK / KN / BK` split into their constituent letters because the synthetic curriculum had only ever shown them with full inter-letter gaps. Phase 5.7 targets all three with a curriculum delta on top of Phase 5.5.
115
 
116
+ - **`rnnt_phase5_8.pt`** — new recommended acoustic. Bootstrap chain: `phase3_5/best → phase5_3/best → phase5_4/last → phase5_5/last → phase5_7/last`. 15 k steps from `phase5_5/last` at peak LR 5e-5, EMA 0.9999, bf16, batch 12, RTX 3060 Laptop, ~2 h wall time.
117
  - **`PHASE_5_7_MIX`** — new text mix that adds 5NN cut-numbers (rendered as the spoken `5NN`, not `599`) and a `_CONTEST_DENSE_TEMPLATES` slice of densely packed contest exchanges around `5NN <NR>`. All other slices are inherited from `PHASE_3_4_MIX`.
118
  - **`OperatorConfig.run_on_pairs`** — new per-utterance knob. A list of digraphs and probabilities; with that probability, the synthesizer collapses the inter-letter gap on the digraph to a single inter-element gap (so `UR` is keyed as `..-.-..`, byte-equivalent to one continuous prosign). Defaults: `UR 0.50, SK 0.85, KN 0.70, BK 0.70`. Old presets pin `run_on_pairs=()` so Phase 3.x / 5.x curricula are byte-identical.
119
  - The channel, jitter, dash:dot, gap-inflation, word-gap-inflation, false-positive prior, and 30 % real-audio mix are kept identical to Phase 5.5 — strict ablation around the curriculum delta.
 
202
 
203
  | file | params | vocab | description | recommended |
204
  |---|---|---|---|---|
205
+ | `rnnt_phase5_8.pt` | 4.13 M | **49** | **v0.6.0 acoustic — Phase 5.8 English-literary curriculum (Moby Dick + Pride & Prejudice + Sherlock Holmes + Frankenstein, 25 % `prose_en`) on top of Phase 5.5, halved run-on prosigns. Pair with `--digit-threshold 0.90` and `--confidence-threshold 0.6` for the recommended inference behaviour.** | ✅ |
206
+ | `lm_phase5_2.pt` | 4.76 M | **49** | v0.4.1 LM — matched to the Phase 3.5 text mix (multilingual prose + ham radio). val_ppl 5.626. **Use this for shallow fusion at λ = 0.7 on EN prose only**; the v0.5.4 LCWO test showed it hurts on FR oratory. | ✅ |
207
+ | `rnnt_phase5_8.pt` | 4.13 M | 49 | v0.5.3 acoustic — Phase 5.7 amateur-idiom curriculum (5NN cut-numbers + run-on UR/SK/KN/BK). Kept for diff. | |
208
  | `rnnt_phase5_5.pt` | 4.13 M | 49 | v0.5.1 / v0.5.2 acoustic — Phase 5.5 (long inter-word silences) on top of v0.5.0. Kept for diff. | |
209
  | `rnnt_phase5_4.pt` | 4.13 M | 49 | v0.5.0 acoustic — Phase 5.3 (wider jitter + dash:dot ratio) + Phase 5.4 (30 % real-audio mix). Kept for diff. | |
210
  | `rnnt_phase3_5.pt` | 4.13 M | 49 | v0.4.0 / v0.4.1 acoustic — synthetic-only. Kept for diff. | |
 
232
 
233
  ## How to use
234
 
235
+ v0.5.4 introduced a unified `morseformer` CLI installable directly from PyPI; v0.6.0 keeps the same surface and points all four presets at `rnnt_phase5_8`. Each preset bundles a model + thresholds + optional LM behind one flag:
236
 
237
  | Preset | Use case | Model | thr | digit_thr | LM |
238
  |---|---|---|---|---|---|
239
+ | `live` (default) | real-time IC-7300 streaming | rnnt_phase5_8 | 0.6 | 0.90 | — |
240
+ | `prose` | offline poetry / ragchew / book | rnnt_phase5_8 | 0.6 | 0.90 | lm_phase5_2 (λ=0.7) |
241
+ | `contest` | fast contest exchanges | rnnt_phase5_8 | 0.5 | 0.80 | — |
242
+ | `conservative` | very noisy bands, prefer silence | rnnt_phase5_8 | 0.75 | 0.95 | — |
243
 
244
  ### Quick start (pip)
245
 
246
  ```bash
247
  pip install "morseformer[ml,audio,hub]"
248
 
249
+ # First call auto-downloads rnnt_phase5_8.pt from the Hub (one-time).
250
  morseformer decode my_recording.wav
251
 
252
  # Offline file decode with LM shallow fusion.
 
260
  # Inspect / pre-fetch checkpoints.
261
  morseformer models list # recommended pair
262
  morseformer models list --advanced # legacy 46-vocab models too
263
+ morseformer models download rnnt_phase5_8
264
  ```
265
 
266
  ### From a git checkout
 
279
 
280
  ```bash
281
  python -m scripts.decode_audio my_recording.wav \
282
+ --ckpt release/rnnt_phase5_8.pt \
283
  --lm-ckpt release/lm_phase5_2.pt \
284
  --fusion-weight 0.7 \
285
  --confidence-threshold 0.6 \
286
  --digit-threshold 0.90
287
 
288
+ python -m scripts.decode_live --ckpt release/rnnt_phase5_8.pt
289
  ```
290
 
291
  Tune your receiver to zero-beat at 600 Hz with a ≈ 500 Hz CW filter. `Ctrl+C` to quit. Latency is ~4 s end-to-end. **LM fusion is offline-only**: the streaming decoder still uses the acoustic-only greedy path; plumbing fusion through the central-zone-commit logic is a separate piece of work earmarked for a follow-up release. The `prose` preset's LM is therefore silently dropped when used with `morseformer live`.
 
415
 
416
  ## Technical specifications
417
 
418
+ **Acoustic model** (`rnnt_phase5_8.pt`, 4.13 M params, identical architecture to v0.1 / v0.2 / v0.3 except for the wider output head; v0.5.x differs from v0.4.x only in training data):
419
  - Encoder: Conformer d=144, L=8, H=4, ff_expansion=4, conv_kernel=31, RoPE, LayerNorm conv, 4× subsample
420
  - CTC head: Linear(144 → 49)
421
  - PredictionNetwork: Embedding(49, 128) + LSTM(128, 128, 1 layer)
 
430
  ## Citation
431
 
432
  ```bibtex
433
+ @software{morseformer_v0_6_0_2026,
434
  author = {Derhy, Serge},
435
  title = {morseformer: open-source transformer-based Morse / CW decoder},
436
  year = 2026,
437
+ version = {v0.6.0},
438
  url = {https://github.com/sderhy/morseformer},
439
  howpublished = {\url{https://huggingface.co/sderhy/morseformer}},
440
  }
REPO_README.md CHANGED
@@ -4,10 +4,10 @@
4
 
5
  [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
6
  [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](#)
7
- [![Release: v0.5.4](https://img.shields.io/badge/release-v0.5.4-brightgreen.svg)](#release-v054)
8
  [![Model on HuggingFace](https://img.shields.io/badge/🤗%20Hub-sderhy/morseformer-yellow)](https://huggingface.co/sderhy/morseformer)
9
 
10
- Conformer + RNN-T Morse decoder with a real-time streaming CLI, trained on a reproducible synthetic-HF pipeline. The **v0.5.3 release** ships a new acoustic (`rnnt_phase5_7.pt`, 4.13 M params, 49-vocab) — a 15 k-step fine-tune of the v0.5.2 acoustic on an **amateur-idiom curriculum**: 5NN cut-numbers (the contest-style spoken `5NN` for `599`) and run-on prosigns where two letters share an inter-letter gap (`UR`, `SK`, `KN`, `BK`). Live-validated on an IC-7300 against FAV22 5-letter groups + a 40 m QSO: explicit `5NN` decoding `× 4-5` vs v0.5.2, the `MI0MO` callsign recovered where v0.5.2 confused it. The v0.5.2 inference-only digit-threshold fix is preserved.
11
 
12
  ## Why
13
 
@@ -70,6 +70,73 @@ CTC : 'CQ DE F4HYY K'
70
  RNN-T: 'CQ DE F4HYY K'
71
  ```
72
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
  ## Release v0.5.4
74
 
75
  Packaging release — **no model change** (`rnnt_phase5_7.pt` unchanged from v0.5.3). v0.5.4 ships a unified `morseformer` console CLI that replaces the `python -m scripts.decode_*` invocations as the primary user surface:
 
4
 
5
  [![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
6
  [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](#)
7
+ [![Release: v0.6.0](https://img.shields.io/badge/release-v0.6.0-brightgreen.svg)](#release-v060)
8
  [![Model on HuggingFace](https://img.shields.io/badge/🤗%20Hub-sderhy/morseformer-yellow)](https://huggingface.co/sderhy/morseformer)
9
 
10
+ Conformer + RNN-T Morse decoder with a real-time streaming CLI, trained on a reproducible synthetic-HF pipeline. The **v0.6.0 release** ships two changes targeting the v0.5.4 LCWO long-form-prose live test: a new acoustic checkpoint `rnnt_phase5_8.pt` (Phase 5.8 English-literary curriculum bootstrapped on Moby Dick + Pride & Prejudice + Sherlock Holmes + Frankenstein, 20 k steps from `phase5_5/last`), and a structural **streaming offline decoder** in `decode_audio` that uses central-zone-commit windowing instead of non-overlapping 6 s chunks. The structural patch eliminates the boundary word-cut artefacts (`MARTYRISÉ → MARTYRISÉ E`, `WANDERED → WANDERE D`, `GOLDEN DAFFODILS → GM ALDEN DAV DAFI RODILS`) that dominated visible errors on long-form prose. The Phase 5.8 acoustic adds literary-English signal on top.
11
 
12
  ## Why
13
 
 
70
  RNN-T: 'CQ DE F4HYY K'
71
  ```
72
 
73
+ ## Release v0.6.0
74
+
75
+ The v0.5.4 LCWO live test (3 hand-keyed-style samples: De Gaulle FR oratory + Al Pacino EN dialogue at 24 / 20 WPM, ~25 min total) confirmed that morseformer produces "fully readable" transcripts on out-of-domain literary prose — but flagged two systematic issues. The dominant visible failure was **window-boundary word cuts**: `MARTYRISÉ → MARTYRISÉ E`, `L'APPUI → L'AP PUI`, `INSTITUTION → INSTITUTIO E`. This was a *structural* problem, not acoustic — `decode_audio` cut the audio into non-overlapping 6 s chunks and stitched the per-chunk hypotheses together, so any word landing on a boundary lost characters. The secondary issue was acoustic / textual: long-form English literary narrative was outside the training distribution (the synthetic mix is QSO-heavy + multilingual cross-section) so words like `WANDERED → WANDERE D`, `BREEZE → BREEZE. E B BKB`.
76
+
77
+ v0.6.0 ships both fixes.
78
+
79
+ ### Streaming offline decoder (structural)
80
+
81
+ `decode_audio` now uses a **central-zone-commit** sliding window — the same logic the live streaming decoder has used since v0.2 — for the RNN-T head. Adjacent commit zones tile the audio without gap or overlap, so words never fall in the middle of a "fresh" boundary. Diff on the v0.5.4 `test_8k.wav` Daffodils transcript:
82
+
83
+ ```
84
+ v0.5.4 chunked: I WANDERE D LONELY L A CLOU E THAT FLOA S ON HIGH …
85
+ GM ALDEN DAV DAFI RODILS, SESIDE THE LAD N, BENEATS T EHE TLES,
86
+ L F F FLUN TERING AND DN E DPCING IN THE BREEZE.
87
+
88
+ v0.6.0 streaming: IWANDERED LONELY AS ALOUD THAT FLOATS ON HIGH …
89
+ GOLDEN DAV DAFFODILS, BESIDE THE LAC, BENEATS THE TLES,
90
+ LF F FLUTTERING AND D DACING IN THE BREEZE.
91
+ ```
92
+
93
+ Old behaviour available via `--legacy-chunking` for diagnostic. CTC head still goes through the chunked path (it is a research-only secondary hypothesis).
94
+
95
+ ### Phase 5.8 — English-literary curriculum (acoustic)
96
+
97
+ `rnnt_phase5_8.pt` is a 20 k-step fine-tune from `phase5_5/last` (NOT from `phase5_7/last` — see Phase 5.7 carryover note) on a new text mix `PHASE_5_8_MIX`:
98
+
99
+ - New `prose_en` slice at 25 % weight, sampled exclusively from the four EN Gutenberg books (Moby Dick = ~42 % of EN, plus Pride & Prejudice + Sherlock Holmes + Frankenstein, ~3 M chars total)
100
+ - `contest_dense` cut from 30 % to 13 % to free room for `prose_en` while keeping the 5NN gain from Phase 5.7
101
+ - `prose_fr` preserved at 10 % so French diacritic gradient is not lost
102
+ - `operator_run_on_pairs` halved (`UR 0.50→0.25, SK 0.85→0.50, KN 0.70→0.30, BK 0.70→0.30`) to reduce phantom-prosign emissions on continuous prose, which the user report flagged as a Phase 5.7 cost
103
+ - Same channel / jitter / dash:dot / gap-inflation / word-gap-inflation / FP-prior as Phase 5.5
104
+
105
+ 20 k steps, peak LR 5e-5, EMA 0.9999, bf16, batch 12, ~2 h on RTX 3060 Laptop. Bootstrap chain: `phase3_5/best → phase5_3/best → phase5_4/last → phase5_5/last → phase5_8/last`. As with every release since v0.5.1, the dormant `ValidationConfig.matching()` clean-CW val signal saturates fast (< step 1 k) — `last.pt` is the released checkpoint, not `best_rnnt.pt`.
106
+
107
+ ### What's measurable on synthetic test_8k.wav
108
+
109
+ Phase 5.8 vs Phase 5.7 on the Daffodils synthetic test (5.4-era ground truth). The structural streaming patch contributes most of the visible improvement; Phase 5.8 adds incremental literary-EN signal on top:
110
+
111
+ | Token | v0.5.4 chunked | v0.5.3 streaming | **v0.6.0 streaming** |
112
+ |---|---|---|---|
113
+ | "WANDERED" | `WANDERE D` | `IWANDERED` | `IWANDERED` ✓ |
114
+ | "FLOATS" | `FLOA S` | `FLOATS` | `FLOATS` ✓ |
115
+ | "FLUTTERING" | `FLUN TERING` | `FLUTERING` | `FLUTTERING` ✓ |
116
+ | "DANCING" | `DN E DPCING` | `DNCINNG` | `DACING` (closer) |
117
+ | "A CLOUD" | `L A CLOU E` | `ALLOUD` | `ALOUD` (closer) |
118
+
119
+ Live evaluation on the user's LCWO samples (De Gaulle + Pacino) is the real target and was not re-run for the v0.6.0 release notes; the repo's auto-aligned Alice corpus (`data/real/aligned/all_alice.jsonl`) and synthetic FR-mix benches did not regress.
120
+
121
+ ### Limitations of v0.6.0
122
+
123
+ - Phase 5.8 is incremental on the synthetic distribution (synthetic val is saturated since v0.5.x). The motivating gain — long-form EN literary prose — has not been formally CER-benched yet; the qualitative diff above is on a single in-distribution audio.
124
+ - `prose` preset's LM (`lm_phase5_2`, matched to `PHASE_3_4_MIX`) was reported by the v0.5.4 LCWO test as actively *hurting* on FR oratory (De Gaulle: `L'APPUI → LAEC LAGAP PUI` vs `L'AP PUI` in `live` preset). v0.6.0 does not change the LM. Use `live` preset for FR oratory; reserve `prose` for English ragchew / prose where the LM training corpus is closer.
125
+ - Spurious prosigns at chunk boundaries (`K`, `+`, `BK`) are reduced but not eliminated — Phase 5.8 halved the run-on-pair probabilities, which is a partial fix.
126
+ - `=` (BT, the typographic break prosign) is rendered as the `BK` digraph by the model when it detects a run-on B+K. Mapping `BK → =` at output is left as a future post-processing cosmetic fix; the model still emits the underlying tokens correctly.
127
+
128
+ ### Recommended decode for v0.6.0
129
+
130
+ Default invocation just works:
131
+
132
+ ```bash
133
+ pip install "morseformer[ml,audio,hub]"
134
+ morseformer decode my_recording.wav # uses rnnt_phase5_8 + streaming
135
+ morseformer live # same model, live streaming
136
+ ```
137
+
138
+ For French oratory or any non-English literary audio, prefer `live` preset over `prose` until the LM is matched (Phase 4.2 territory in the roadmap).
139
+
140
  ## Release v0.5.4
141
 
142
  Packaging release — **no model change** (`rnnt_phase5_7.pt` unchanged from v0.5.3). v0.5.4 ships a unified `morseformer` console CLI that replaces the `python -m scripts.decode_*` invocations as the primary user surface:
rnnt_phase5_8.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7e358bfe0bc19f39ab11efbdd3c85e2a2d97f5812b19352b4347494485ca6653
3
+ size 33177526