--- license: apache-2.0 library_name: coreml tags: - audio-classification - voice-activity-detection - radio - coreml - mlx - swift pipeline_tag: audio-classification --- # Skywave Segmenter — small **Is a person speaking over the music right now?** Nothing in a radio station's metadata answers that. It says which record is playing, never that a presenter has started talking over it — and no public dataset labels it either, which is why this exists. 159,058 parameters. **192 KB** as CoreML at int8. On an M5 Max it takes **3.1 ms** to label a forty-second window end to end, features included: about 13,000× faster than the audio arrives. ## What it does Log-mel in, two classes out — `music` and `speech` — at **12.5 verdicts a second**, for audio of any length. It is fully convolutional over time, so the same weights handle a three-second clip and a three-minute one. "Speech" here means *a human talking over or between records*, which is a narrower question than voice activity. A sung vocal is music. That distinction is the whole difficulty: the two things that most resemble a presenter are a rapper and a singer. ## Files | | | |---|---| | `SkywaveSegmenter.mlpackage` | CoreML, int8, what the app runs | | `skywave-segmenter.safetensors` | the trained weights, MLX layout | | `skywave-segmenter-norm.safetensors` | per-bin mean and standard deviation | | `skywave-segmenter.json` | the model card the runtime reads: classes, stride, front end | The normalisation is folded into the CoreML graph, so a caller feeds **raw log-mel** and the model normalises internally. The safetensors weights are unnormalised and expect it applied. ## The front end is part of the model ``` 16 kHz mono · 400-sample window · 320-sample hop · 64 mel bands · 50–7600 Hz log(energy + 1e-6) · 50 frames per second ``` These are not suggestions. The model learned on frames computed exactly this way, and frames computed any other way make it degrade quietly — which looks like a mediocre model rather than a mismatch. The reference implementation is [`SkywaveCore`](https://github.com/vdeturckheim/skywave-model), which is checked against committed fixtures to 1.5e-04 on every build. One detail that has cost real accuracy: decode to mono the way `ffmpeg -ac 1` does, which divides a stereo pair by **√2** rather than by 2 because it normalises to preserve power. The model reads absolute level, and averaging the channels instead is three decibels quieter — measured to be the difference between finding an announcement and reporting silence. ## Results Held out from training (four-hour chunks `h20` and `h12`), scored **end to end** — through the smoothing and the span merging, counting announcements a listener would notice rather than frames: | | | |---|---| | Announcements found | **33 of 33** | | Sung vocals wrongly flagged | 1 of 15 | | Ordinary music wrongly flagged | **0 of 40** | The last row is the one that matters: a false positive pulls a listener off a lossless track mid-song, and it does not happen. **Do not quote a per-frame number for this model.** They disagree with the end-to-end result badly enough to invert a ranking. A 4-bit quantization of this network agrees with its own float32 self on **81% of individual verdicts** while losing six spans of eight — it reports 1.36 s of speech where there is 7.28 s. A model can look four fifths right and detect almost nothing, because the frames it loses are the ones carrying the announcements. ## Size, and why this one Twelve architectures were trained, 41k to 2.5M parameters. **None of them can be told apart**: every one lands at 32–33 of 33. With 33 announcements to be right about, the benchmark cannot separate an architecture from a seed — three seeds of one architecture gave 32–33 found and 1–3 sung false positives. So `small` was chosen on what does not swing: it is small, and its sung false positives were 1 of 15 in all three seeds where the 41k variant gave 1 to 3. A `tiny` variant at 41,610 parameters measured equal and is published separately for anything where 41 KB matters more than margin. int8 is free here — identical spans to float16, verified through the app's own inference path. Below eight bits nothing survives. ## Training data, and its limits 28 hours of FIP (Radio France), labelled by an earlier transcribe-and-reason pipeline whose verdicts this model distils. The audio is not published and the derived corpus is private: it is a broadcaster's material, not ours. **This is a distillation of a teacher, not ground truth.** It inherits that pipeline's judgement, including its mistakes — three of eighteen "hard negatives" in the original labelling were real station dressing rejected for the wrong reason, because FIP puns on its own name and the transcriber wrote "flippe" and "Philippe". **It knows one station.** Trained on French-language presenters over predominantly anglophone music. On a station where the presenter and the songs share a language, the harder case, it has not been measured. The held-out split is also thin in one class: `h12` contains no presenter frames at all, only music and station idents. **It cannot report a station ident.** The corpus carries three classes — music, host, ident — and this model collapses the last two, so a jingle and a presenter both come back as `speech`. A three-class variant trained directly reaches only 25% ident recall: idents are 0.48% of frames, too thin for the class weighting used. ## Using it Swift, via [`skywave-model`](https://github.com/vdeturckheim/skywave-model): ```swift let segmenter = try Segmenter() // reads the card beside the model let spans = try segmenter.detail(samples) // mono 16 kHz Float for span in spans where span.label != "music" { print(span.start, span.end, span.confidence) } ``` Verdicts become final **2.8 s** after the audio: the receptive field is ±2.16 s, measured by perturbing one input frame and watching which outputs move, plus half the 1.25 s smoothing vote. Nothing beyond that window changes a verdict, so there is no reason to hand it more than about twelve seconds at a time.