README.md

---
license: cc-by-4.0
language:
- en
- bg
- hr
- cs
- da
- nl
- et
- fi
- fr
- de
- el
- hu
- it
- lv
- lt
- mt
- pl
- pt
- ro
- ru
- sk
- sl
- es
- sv
- uk
library_name: coreml
tags:
- coreml
- apple-silicon
- ane
- speech-translation
- automatic-speech-translation
- canary
- attention-encoder-decoder
- fluidaudio
base_model: nvidia/canary-1b-v2
base_model_relation: quantized
pipeline_tag: translation
---

# Canary-1B-v2 Speech Translation — CoreML (ANE)

**On-device speech-to-text translation (English ↔ 24 European languages) on the
Apple Neural Engine**, using the exact CoreML models published at
[`FluidInference/canary-1b-v2-coreml`](https://huggingface.co/FluidInference/canary-1b-v2-coreml).

No extra weights are needed: [`nvidia/canary-1b-v2`](https://huggingface.co/nvidia/canary-1b-v2)
is a multi-task attention encoder-decoder whose task is selected by two decoder
prompt slots. Transcription is the special case `source == target`; setting a
different target language token makes the same models translate the speech.
This repo documents that usage and hosts the benchmark artifacts; the model
files live in (and are auto-downloaded from) the main repo.

## Prompt contract

```
▁ <|startofcontext|> <|startoftranscript|> <|emo:undefined|> <SOURCE> <TARGET> <|pnc|> <|noitn|> <|notimestamp|> <|nodiarize|>
```

English→German is one token away from English transcription:

```
transcribe en:  [16053, 7, 4, 16, 64, 64, 5, 9, 11, 13]
translate en→de:[16053, 7, 4, 16, 64, 78, 5, 9, 11, 13]
```

`prompt_tokens.json` in this repo maps all 25 language codes to their token ids.

## Usage (FluidAudio)

Swift ([FluidAudio](https://github.com/FluidInference/FluidAudio), branch with
Canary translation support):

```swift
let manager = try await CanaryManager.load(source: .english, target: .german)
let german = try await manager.transcribe(audioURL: url)  // ≤ 15 s window
```

CLI:

```bash
swift run fluidaudiocli canary-transcribe audio.wav --translate-to de --source-lang en
```

## Benchmarks (FLEURS test, int4/ANE, Apple M5 Pro)

Scored with sacreBLEU / chrF2 / COMET (`Unbabel/wmt22-comet-da`). References are
the raw cased FLEURS transcriptions of the aligned target-language sentences.

| Direction | Pairs | BLEU | chrF2 | COMET | RTFx |
|---|---|---|---|---|---|
| en→de | 641 | 29.98 | 59.86 | 82.70 | 10.3× |
| de→en | 862 | 33.06 | 62.15 | 83.50 | 14.7× |

For reference, NVIDIA reports FLEURS 24-language fp16 averages of BLEU 29.4 /
COMET 84.56 (en→X) and BLEU 29.08 / COMET 79.30 (X→en) for the base model.

### The 15-second window matters

The CoreML contract is a fixed 15 s window; longer audio is chunked with an
overlap-merge designed for transcripts, which is unreliable for translation
(translated text is not token-stable across window seams). Splitting the
benchmark by clip length:

| Subset | en→de COMET | de→en COMET |
|---|---|---|
| ≤ 15 s (single window) | **83.44** | **85.67** |
| > 15 s (chunked) | 73.88 | 78.59 |

**Single-window quality matches the published fp16 numbers** — the int4
quantization penalty is ≤ ~1 COMET point. For long audio, segment at pauses
into ≤ 15 s pieces (e.g. with a VAD) instead of relying on the chunked path.

`fleurs_{en_de,de_en}_hypotheses.json` contain the parallel hypotheses and
references used for these scores.

## Files

| File | Contents |
|---|---|
| `prompt_tokens.json` | prompt layout + language-code → token-id map + example |
| `fleurs_en_de_hypotheses.json` | 641 en→de hypotheses/references (FLEURS test) |
| `fleurs_de_en_hypotheses.json` | 862 de→en hypotheses/references (FLEURS test) |

## Attribution

- Base model: [`nvidia/canary-1b-v2`](https://huggingface.co/nvidia/canary-1b-v2) (CC-BY-4.0)
- Benchmark references derive from [FLEURS](https://huggingface.co/datasets/google/fleurs) (CC-BY-4.0)
- CoreML conversion + Swift runtime: [FluidInference/FluidAudio](https://github.com/FluidInference/FluidAudio)