seanll95/sidon-coreml

Sidon — Core ML

Core ML conversions of sarulab-speech/sidon-v0.1, a speech restoration model, rebuilt to run on an iPhone's Neural Engine. Used by Excerpt, an iOS audiobook player, to lift noise and muffling out of narration on the device.

16 kHz in, 48 kHz out, duration preserved.

Four versions, with the same file names, inputs and outputs:

• The repository root holds upstream's weights, unchanged. What changed is the graph around them, so that it compiles for the Neural Engine.
• v2/ holds the same conversion of a fine-tuned Sidon, trained to remove a warble upstream adds on some vowels. See v2: fine-tuned.
• v3/ holds a second fine-tune that targets the warble's cause directly, and removes more of it. See v3: phase-coherent.
• v4/ holds v3's recipe retrained on xHE-AAC audiobooks, with penalties for added hiss and clicks. See v4: less hiss.

Contents

File	Precision	Size	Fixed input
sidon_encoder_ane.mlpackage	fp16	468 MB (v2/ to v4/: 458 MB)	feats [1, 300, 160] -> hidden [1, 300, 1024]
sidon_decoder_ane.mlpackage	fp16	101 MB	feat [1, 1024, 68] -> [1, 1, 65280]
sidon_fbank.bin	float32	84 KB	Front-end constants: 400-tap Povey window and 257 x 80 mel filters; identical in both

.mlpackage files are uncompiled. Compile them on the device with MLModel.compileModel(at:) and cache the result; the first compile can take a minute or more. Load both with MLComputeUnits.cpuAndNeuralEngine.

How this differs from upstream

The feature extractor is not in the model. Upstream feeds raw audio through SeamlessM4TFeatureExtractor (from facebook/w2v-bert-2.0) in Python. Here the encoder takes the 160-dimensional stacked fbank features directly, and sidon_fbank.bin carries that extractor's exact window and mel filters so the features can be computed natively. Recomputing them from a formula instead shifts every frame. The extractor normalises per utterance, so compute features once over a long stretch of audio and slice the features, never the audio.

The encoder runs 300 frames (6 s) per call. Upstream has no fixed length. The Neural Engine compiler fails at 600 frames — it runs out of memory expanding the relative position tables, and on a phone it is killed. 300 frames compiles and runs 99.3 % on the Neural Engine. The conformer uses long-range context, so shorter windows cost accuracy: against 12 s windows this measures about 2 dB lower SNR, which was inaudible in blind listening tests. The encoder's dictionary output is also rewired to a plain tensor, since Core ML has no dictionary op; that surgery is checked against the original to 1e-5.

The decoder is rebuilt, and its transposed convolutions are rewritten. Upstream's DAC decoder is a frozen TorchScript module whose scripted Snake activations cannot be converted. It is reconstructed in Python from the repo config with the weights transplanted in, checked bit-exact against the original. Its ConvTranspose1d layers — which the Neural Engine computes wrongly, dropping SNR by 53 dB at a single layer — are replaced with an equivalent sub-pixel form: two convolutions whose outputs interleave. Because every layer has kernel == 2 * stride, this is exact, not an approximation (max|d| 0.0 in PyTorch). It is also 1.65x faster on the iPhone GPU.

The decoder runs 68 frames per call. The Neural Engine caps every tensor dimension at 65536, and the decoder's output is 960 samples per frame, so 68 frames is the most that fits. Run it in overlapping windows with a 12-frame halo on each side and keep only the middle; that reconstructs a single long call to 120 dB. Every window must hold real frames — slide the last one back rather than zero-padding it.

What it costs

Against the same models on the GPU, the Neural Engine output measures 1.7 dB lower, from fp16 hidden states and the shorter encoder window. The difference sits above 16 kHz, 52 dB down; across the speech band it is within 0.4 dB. In a blind, shuffled listening test, PyTorch, all-GPU, GPU + Neural Engine and all-Neural Engine renders could not be told apart.

On an iPhone, warm, this runs at 11.2x realtime, against 4.8x on the CPU alone. A backgrounded iOS app cannot use the GPU, so for long audio that is the comparison that matters.

v2: fine-tuned

Upstream Sidon adds a warble to some vowels, worst on "a" sounds, like a reverb whose pitch shifts. It does this on clean, uncompressed input too, so it is the model's own habit rather than a codec artefact.

v2/ is upstream with small trained adjustments to both the encoder and the decoder, folded into plain weights before conversion. It was trained on audiobook narration compressed through real audiobook codec chains (MP3, AAC, HE-AAC, Opus, Vorbis; one to three generations). Everywhere except the warbling vowels it was taught to reproduce upstream's own output, which keeps upstream's studio sound and the narrator's voice. On the vowels where upstream warbles, it was taught to reproduce the original recording instead.

Measured on 84 held-out clips from compressed input:

	Upstream	v2
NISQA quality score	4.74	4.77
Vowels over the warble threshold	20.3 %	9.3 %
Tone movement against the original (1.0 = as lively)	0.99	0.95

In blind listening on six held-out phrases, words heard as warbling fell from 10 to 4. Listening tests used one narrator, and some phrases still warble.

The Core ML conversion is the same as the root's. The encoder runs 99.4 % on the Neural Engine (root: 99.9 %), and both builds are equally close to their PyTorch models.

v3: phase-coherent

Measured against the original recordings, the warble is phasiness: a vowel's harmonics stay at the right pitch but lose their fixed phase relation to each other from one moment to the next, so each pitch period comes out a slightly different shape and the ear hears a room.

v3/ is trained like v2/, with three changes:

• 23 books chosen by ear for studio recording quality, so the recording the warbling vowels copy is itself clean;
• a harmonic phase-coherence loss: at each harmonic of the recording's pitch, the render's relative phases may step no more than the recording's do;
• a guard against any frame repeating less cleanly than the recording, so steadier harmonics are not bought with a rough vowel.

On clips from two books neither fine-tune trained on, compressed input, all three measured the same way:

	Upstream	v2	v3
Vowels over the warble threshold	35.4 %	23.6 %	11.4 %
Vowels far over it	17.3 %	9.0 %	3.4 %
Tone movement against the original (1.0 = as lively)	1.02	0.98	0.93
NISQA quality score	4.69	4.69	4.50

The automatic quality score and liveliness fall a little. By ear, a listener preferred v3/ to the earlier fine-tunes.

The Core ML conversion is the same as the root's. The encoder runs 99.4 % on the Neural Engine, and against its PyTorch model the Core ML output is as close as the root's is to upstream's (mel-cepstral distance 6.5 to 10.3 on three clips, against 6.9 to 10.6).

v4: less hiss

v4/ is trained like v3/, with two changes:

• 8 books whose own releases are xHE-AAC, a codec newer than any v3 trained on, one book held out;
• penalties for hiss and clicks the render adds that the recording does not have.

On 128 clips from 13 books neither model trained on, compressed through the same codec chains:

	Upstream	v3	v4
Added hiss, compressed input (mean)	0.78	0.57	0.30
Added hiss, clean input (mean)	0.12	0.12	0.05
Vowels over the warble threshold	29.3 %	10.4 %	10.3 %
Vowels far over it	9.6 %	2.4 %	2.1 %
Roughness against the original	0.070	0.040	0.047
Tone movement against the original (1.0 = as lively)	1.02	0.96	0.96

Hiss falls by about half and the warble stays where v3 left it; roughness rises a little. In a blind A/B against v3/ on six held-out clips, four were heard as the same and one each way, and a listener chose v4/.

The Core ML conversion is the same as the root's. The encoder runs 99.2 % on the Neural Engine; against its PyTorch model the Core ML output scores mel-cepstral distance 4.0 to 5.3 on three clips, every band within 0.1 dB.

What it does

Sidon re-synthesises speech from w2v-BERT semantic features rather than repairing the waveform. Running it on clean audio changes it about as much as running it on damaged audio. Expect cleaner, clearer narration in the same voice, not a sample-accurate restoration of the original recording.

Licence and attribution

MIT, inherited from the upstream model — see LICENSE. The model, its training and its weights are the work of the Sidon authors (Wataru Nakata, Yuki Saito); this repository contributes the Core ML conversion and, in v2/ to v4/, the fine-tunes.