docs/features/diarization.md

OmniVoice Studio — Speaker Diarization

Diarization splits a single audio stream into per-speaker tracks: who said what, and when. OmniVoice uses pyannote + WhisperX under the hood — the same stack the original WhisperX paper used.

What diarization buys you

• Multi-speaker dubbing: each detected speaker gets its own voice clone in the target language.
• Subtitle styling: speaker labels (SPEAKER_00:, SPEAKER_01:, …) on the exported SRT/VTT files.
• Audio editing: per-speaker tracks in the timeline view.

License acceptance flow

The diarization model — pyannote/speaker-diarization-3.1 — is gated on HuggingFace. A valid HF token alone is not enough: you also need to accept the model's license once.

1. Get a HF token if you don't have one — see docs/setup/huggingface-token.md.
2. Set the token via Settings → API Keys (or any of the other supported paths).
3. While signed in to HuggingFace with the same account, visit:
   • https://huggingface.co/pyannote/speaker-diarization-3.1 → "Agree and access repository".
   • https://huggingface.co/pyannote/segmentation-3.0 → same.
4. Restart the dub job. The first run downloads ~600 MB of model weights.

If you skip the license acceptance, the HF API returns 401 Unauthorized for the download — the same error class the in-app "Open docs for this error" button deeplinks to.

Fallback behaviour

When diarization is unavailable (no HF token, license not accepted, model download failed mid-run), OmniVoice's dub pipeline falls back to a silence-gap heuristic that splits speakers on long quiet stretches. You'll see a warning toast and the dub_core.py reason string surfaces in the job log:

• "diarization_skipped:no_token" — no token resolved from the cascade.
• "diarization_skipped:401" — token present but unauthorised on the gated model (license not accepted).
• "diarization_skipped:network" — model download interrupted.

The heuristic is not as accurate as pyannote — speakers with similar pitch or rapid turn-taking conversation get merged — but it lets the dub finish end-to-end instead of erroring.

HF token requirement

Diarization is the one OmniVoice feature where a HF token is required, not just recommended. See docs/setup/huggingface-token.md for the three-source cascade and how the in-app Settings → API Keys panel works.

Troubleshooting

• HF 401 → see troubleshooting.md#2-hf-401--pyannote-license-not-accepted.
• Model download stuck → check ~/.cache/huggingface/hub/models--pyannote--* size grows during the dub; if it stalls at 0 bytes, your token isn't being read — confirm in Settings → API Keys that the active source has a green checkmark.