aufklarer/Canary-180M-Flash-CoreML

Canary 180M Flash — Core ML

NVIDIA Canary 180M Flash exported to Core ML for on-device speech recognition on Apple platforms. FastConformer encoder with an autoregressive Transformer decoder: the encoder consumes a whole utterance, then tokens are decoded one at a time against a cached decoder state. Offline per utterance — there is no streaming mode.

English, German, Spanish and French, with punctuation and capitalisation. Source and target language are separate prompt tokens, so translation between those four is reachable from the same models.

Model

Parameters	182M (17-layer FastConformer encoder, 4-layer Transformer decoder)
Format	Core ML .mlmodelc (mlprogram, int8 weights)
Deployment target	iOS 17 / macOS 14
Sample rate	16 kHz mono
Features	128-bin log-mel, 512 FFT / 160 hop / 400 window
Encoder window	1000 mel frames (10 s), fixed — pad shorter audio
Vocabulary	5248 SentencePiece pieces
Languages	en, de, es, fr

Files

File	Size	Description
CanaryEncoder.mlmodelc	104 MB	FastConformer encoder + projection + mask
CanaryPrefill.mlmodelc	70 MB	Decoder over the prompt, empty cache
CanaryDecoder.mlmodelc	71 MB	Decoder step, one token against the cache
vocab.json	85 KB	Token id → SentencePiece piece
config.json	4 KB	Decode contract: prompt ids, cache dimensions, feature front end

Models ship compiled (.mlmodelc). Compiling on device has produced different results across runtimes, so it happens once at export.

Graph contract

CanaryEncoder  audio_signal [1,128,1000] float32, length [1] int32
            -> encoder_embeddings [1,125,1024], encoder_mask [1,125]

CanaryPrefill  input_ids [1,9] int32, encoder_embeddings, encoder_mask
            -> logits [1,1,5248], decoder_hidden_states [5,1,9,1024]

CanaryDecoder  input_ids [1,1] int32, encoder_embeddings, encoder_mask,
               decoder_mems [5,1,C,1024], start_pos [1] int32
            -> logits [1,1,5248], decoder_hidden_states [5,1,C+1,1024]

Decode is: encoder once, prefill once on the prompt, then the step model per token, feeding decoder_hidden_states back as decoder_mems and passing the cache length as start_pos. Stop when argmax hits the end-of-text id in config.json.

logits are log probabilities (the model's log-softmax head is kept), so exp(mean log p) over the emitted tokens is a usable confidence.

The encoder takes a fixed 10 s window: zero-pad shorter audio and pass the true frame count as length, which is what drives masking, so padding does not change the result. Audio longer than the window is truncated — segment with VAD before calling. On FLEURS English, 44% of utterances run longer than 10 s, so this bundle suits conversational turns rather than dictation.

Every shape is fixed on purpose. A flexible time axis on the conformer produces a graph that aborts the host process at load, and a flexible encoder length on the decoder graphs roughly doubles the cost of each decode step. The decoder is therefore pinned to this encoder's output length; a different window needs a matched pair.

Prompt

Nine tokens, published in config.json as promptIds, with per-language ids in languageTokenIds:

<|startofcontext|> <|startoftranscript|> <|emo:undefined|> <src> <tgt>
<|pnc|> <|noitn|> <|notimestamp|> <|nodiarize|>

Read the ids from config.json rather than resolving tokens by string: this vocabulary has no bare-space token (word boundaries are SentencePiece's ▁, U+2581), and a prompt token that silently resolves to -1 produces fluent text that stops after a couple of words or repeats a fragment.

Front end

Features follow the NeMo AudioToMelSpectrogramPreprocessor contract the model was trained with; config.json spells it out:

• pre-emphasis 0.97
• STFT n_fft 512, hop 160, window 400, centred, constant-padded
• symmetric Hann window
• Slaney-normalised mel bank, 128 bins
• log(x + 2^-24)
• per-feature normalisation using the sample (N−1) variance, epsilon 1e-5

Performance

FLEURS English, 364 utterances (those fitting the 10 s window), Apple silicon, .cpuAndNeuralEngine:

	WER	Speed
this bundle	7.40%	60.3× real time
previous float16 / 30 s bundle	7.48%	32.4× real time

Both scored on the identical subset, so the difference is not a truncation artefact. int8 weights cost no accuracy; the speed comes from the 10 s window.

Transcripts match the NeMo checkpoint's own greedy decode. .cpuAndNeuralEngine is the recommended setting and is measurably faster than CPU (75 ms vs 103 ms on a 2.9 s utterance). .all pays a large GPU planning cost on the first call and does not win afterwards.

Usage

let encoder = try MLModel(contentsOf: encoderURL, configuration: config)
let prefill = try MLModel(contentsOf: prefillURL, configuration: config)
let step    = try MLModel(contentsOf: stepURL,    configuration: config)
// config.computeUnits = .cpuAndNeuralEngine

// features: [1, 128, 3000] float16/32 log-mel, zero-padded, `length` = real frames
let encoded = try encoder.prediction(from: ...)
var state = try prefill.prediction(from: ...)      // prompt ids from config.json
// then loop the step model, feeding decoder_hidden_states back as decoder_mems
// and the cache length as start_pos, until argmax == specialTokenIds.eos

Source

Exported from nvidia/canary-180m-flash (CC-BY-4.0). Prompt ids, cache dimensions and the feature contract are read off the checkpoint at export time rather than hand-written.

Links

• speech-swift — Apple SDK
• Docs — install and CLI docs
• soniqo.audio — website
• blog — blog