--- license: other license_name: desert-ant-labs-source-available-1.0 license_link: https://license.desertant.ai/1.0 language: - multilingual tags: - speech - word-timestamps - forced-alignment - speech-recognition - on-device - core-ml - multilingual pipeline_tag: automatic-speech-recognition --- # Align: on-device word-timestamp refinement for Apple SpeechAnalyzer Corrects the word-level timings that Apple's `SpeechTranscriber` and `SpeechAnalyzer` return, without replacing them. Align observes the same audio the analyzer already receives, runs a small Core ML cascade on the CPU and Neural Engine, and returns the familiar result surface with tightened `audioTimeRange` values. The models are tiny (**about 0.7 MB** compiled Core ML) and refine a typical result in a few milliseconds on device. > Apple: `"world"` 2.61-3.04s ➜ Align: `"world"` 2.57-2.98s ## Try it Ships as an Apple SwiftPM package: **[Desert-Ant-Labs/align](https://github.com/Desert-Ant-Labs/align)**. - **iOS / iPadOS / Mac Catalyst / macOS / tvOS / visionOS:** the Swift SDK (Swift Package Manager). It bundles the compiled Core ML models below, so it works fully offline. The package adds to apps with low deployment targets; the SpeechAnalyzer refinement APIs are gated with `@available` and run on the 26 releases those frameworks require. - Add one input modifier (`inputs.recordingAudio(for: refiner)`) and one result modifier (`transcriber.results.refiningTimestamps(with: refiner)`) to the standard Apple pipeline. ## Files | File | Format | Size | Contents | |---|---|---:|---| | `align_coarse.mlmodelc` | Compiled Core ML (FP16) | ~0.3 MB | Coarse stage: searches a 241-frame (2.4 s) context, fixed batch-16 | | `align_fine.mlmodelc` | Compiled Core ML (FP16) | ~0.3 MB | Fine stage: searches an 81-frame (0.8 s) crop centered on the coarse prediction | | `mel_filters.bin` | Float32 filter bank | ~40 KB | Log-mel filter bank the runtime frontend needs | | `calibrator.bin` | Gradient-boosted trees | ~70 KB | Correction calibrator over coarse/fine uncertainty features | | `refiner_config.json` | JSON | tiny | Frontend, lexical, and language config the runtime needs | | `coarse.pt` | PyTorch checkpoint | ~0.5 MB | Coarse-stage weights (for retraining / other runtimes) | | `fine.pt` | PyTorch checkpoint | ~0.5 MB | Fine-stage weights (for retraining / other runtimes) | The compiled `.mlmodelc` stages, `mel_filters.bin`, `calibrator.bin`, and `refiner_config.json` are exactly what the Swift SDK bundles. The `.pt` checkpoints are the training-run weights. ## Architecture A two-stage coarse-to-fine cascade over a log-mel spectrogram, refining one boundary at a time: - **Frontend**: an Accelerate/vDSP log-mel spectrogram of the same audio Apple transcribes. - **Coarse stage**: a compact convolutional model searches a 2.4 s context around Apple's proposed boundary and predicts a distribution over frames. - **Fine stage**: a second model re-searches a 0.8 s crop recentered on the coarse prediction for a tighter estimate. - **Lexical conditioning**: UTF-8 byte features of the neighboring words plus a language id let a single model cover all nine languages. - **Calibrator**: a small gradient-boosted-tree policy maps coarse/fine uncertainty features to a final correction, fit only on the validation split to reduce large regressions. - **Structural fallback**: boundaries whose correction would be invalid, hit the search-window edge, or lack streaming context keep Apple's original timestamp. Each stage runs fixed batch-16 on CPU + Neural Engine. Total parameters are about 117k per stage. ## Inputs and outputs - **Input:** mono audio plus Apple's recognized words with their proposed start/end times. - **Output:** the same words with corrected start/end times, or Apple's original time when a correction is not structurally safe. ## Accuracy Evaluated on the exact Swift runtime and these bundled Core ML models over 223 clean and 210 noisy group-held-out recordings across all nine languages, against forced-alignment references. | Condition | Apple raw error | Align error | Reduction | Median | Within 50 ms | |---|---:|---:|---:|---:|---:| | Clean | 113.5 ms | 44.9 ms | 60% | 28.2 ms | 75.1% | | Noisy | 124.4 ms | 50.1 ms | 60% | 32.0 ms | 69.4% | Error is mean absolute distance from the reference boundary. Align roughly halves Apple's typical error and removes most of its large mistakes. ## Languages English, Spanish, French, Italian, Portuguese, German, Japanese, Korean, and Chinese. A locale outside this set is passed through unchanged. ## Limitations - References are machine forced-alignment estimates, not human annotations, so the figures show a large, consistent reduction of Apple's timing error rather than sample-accurate ground truth. - A learned correction is not guaranteed to improve every boundary; the structural fallback keeps Apple's timestamp when a correction looks unsafe but cannot catch every plausible-looking error. - English, Italian, Japanese, and Korean are the weakest languages under the current reference convention. ## Built on - [FLEURS](https://huggingface.co/datasets/google/fleurs) (CC BY 4.0): multilingual training audio. - [Qwen3-ForcedAligner-0.6B](https://huggingface.co/Qwen/Qwen3-ForcedAligner-0.6B) (Apache-2.0): primary word-boundary references for all nine languages. - OWSM-CTC v4 1B (CC BY 4.0): gross alignment-outlier check where validation agreement is stable. - Genuine Apple `SpeechAnalyzer` proposals collected on macOS 26. See [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md). None of these systems are redistributed here. ## License [Desert Ant Labs Source-Available License](https://license.desertant.ai/1.0). Free for most apps; a commercial license is required at scale. Full terms are at the link. Licensing: . ## Citation ```bibtex @software{align_2026, title = {Align: on-device word-timestamp refinement for Apple SpeechAnalyzer}, author = {Desert Ant Labs}, year = {2026}, url = {https://huggingface.co/desert-ant-labs/align}, } ``` --- © 2026 Desert Ant Labs ·