passpoo/talyno-whisperkit-coreml

talyno-whisper-small-finance (WhisperKit CoreML)

WhisperKit CoreML build of talyno-whisper-small-finance, a LoRA fine-tune of openai/whisper-small specialized for earnings-call and investor-meeting speech. This is the on-device model used by Talyno — an iOS app that transcribes and translates multilingual earnings calls entirely on-device.

• Source (PyTorch) weights: passpoo/talyno-whisper-small-finance
• Base model: openai/whisper-small
• Format: WhisperKit CoreML (.mlmodelc), converted with whisperkittools
• Languages: English (en), Chinese — Simplified + Traditional (zh)
• Size on disk: ~466 MB
• License: Apache 2.0

What makes this different from stock Whisper

This is not a generic ASR model. It is fine-tuned specifically on earnings-call and investor- relations language:

1. Financial vocabulary & IR patterns — trained on real earnings-call audio (SPGISpeech) and financial-terminology TTS corpora, so it recognizes terms like EPS, EBITDA, guidance, revenue/margin figures, and analyst Q&A phrasing more reliably than a general-purpose model.
2. Focused language scope, intentionally — rather than a shallow "50+ language" claim, this model covers exactly the languages it was actually trained on: English and Chinese (Simplified/Traditional). See the training guide for why the scope was narrowed from an earlier 6-language generation.
3. On-device, offline-first — designed to run fully on iPhone Neural Engine via WhisperKit; no audio or transcript ever needs to leave the device.

Repo layout

talyno-whisper-small-finance/
├── AudioEncoder.mlmodelc/
├── TextDecoder.mlmodelc/
├── TextDecoderCoreML.mlmodelc/
└── config.json

Usage (WhisperKit / iOS)

import WhisperKit

let whisper = try await WhisperKit(
    model: "talyno-whisper-small-finance",
    modelRepo: "passpoo/talyno-whisperkit-coreml",
    downloadBase: FileManager.default
        .urls(for: .documentDirectory, in: .userDomainMask)[0]
        .appendingPathComponent("WhisperModels")
)

WhisperKit downloads the .mlmodelc bundle from this repo on first use and caches it locally. Talyno bundles the much smaller openai_whisper-tiny model for immediate offline use and offers this model as an optional "Accurate" upgrade in Settings → AI Models.

Training summary

Two-phase LoRA fine-tune on a frozen Whisper encoder:

Phase	Language	Data	LoRA rank
1	English	SPGISpeech (earnings-call audio, 200h subset)	r=8
2	Chinese (zh)	Google FLEURS cmn_hans_cn (real recordings) + TTS financial-terminology corpus (Traditional Chinese)	r=16

Limitations

• English and Chinese (Simplified/Traditional) only — other languages will not produce reliable transcripts with this model. Use the bundled openai_whisper-tiny or a stock Whisper model for broader language coverage.
• Optimized for earnings-call / IR register; performance on casual conversational speech is not specifically validated.
• Traditional Chinese coverage relies partly on synthesized (TTS) training audio for finance-specific vocabulary, since public real-recording Chinese speech corpora (FLEURS) do not include a Taiwan Mandarin locale.

License

Apache 2.0, consistent with the base openai/whisper-small model and the Talyno app.