Thorsten-Voice/CosyVoice3-Hessisch

Thorsten-Voice CosyVoice3 — Hessisch

A Hessian dialect fine-tune of Fun-CosyVoice3-0.5B-2512, trained on a dataset of Hessian-dialect recordings as part of the Thorsten-Voice project — free, openly licensed German speech data and TTS models.

This is a speaker fine-tune (SFT), not a zero-shot voice clone: it reproduces one specific voice speaking Hessian dialect, in the same spirit as Thorsten-Voice/CosyVoice3 (the standard German variant).

Model Details

• Base model: FunAudioLLM/Fun-CosyVoice3-0.5B-2512
• Language/Dialect: German — Hessian dialect (Hessisch)
• Training data: ~2,100 Hessian-dialect recordings, single speaker, 24kHz
• Fine-tuned components: LLM (language model) and Flow (mel-spectrogram decoder)
• Vocoder (HiFi-GAN): not fine-tuned for this dialect — this model uses the original base-model vocoder. In our testing, fine-tuning HiFi-GAN on this comparatively small dataset degraded audio quality (audible artifacts) rather than improving it, so we kept the original, better-generalizing vocoder instead.
• Speaker embedding key: thorsten_hessisch

Usage

This repository only contains the fine-tuned llm.pt, flow.pt, and spk2info.pt. You need the base model's remaining files (vocoder, tokenizers, configs) to run inference.

pip install -r https://raw.githubusercontent.com/FunAudioLLM/CosyVoice/main/requirements.txt

# 1. Download the base model
hf download FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --local-dir pretrained_models/CosyVoice3-0.5B

# 2. Download this fine-tune and overlay it onto a copy of the base model directory
hf download Thorsten-Voice/CosyVoice3-Hessisch --local-dir pretrained_models/CosyVoice3-Hessisch
cp -r pretrained_models/CosyVoice3-0.5B/* pretrained_models/CosyVoice3-Hessisch/
hf download Thorsten-Voice/CosyVoice3-Hessisch --local-dir pretrained_models/CosyVoice3-Hessisch --force-download

import sys
sys.path.insert(0, 'CosyVoice')
sys.path.insert(0, 'CosyVoice/third_party/Matcha-TTS')
import torch, torchaudio
from cosyvoice.cli.cosyvoice import CosyVoice3

model = CosyVoice3('pretrained_models/CosyVoice3-Hessisch')

# Instruct-prompt patch — prevents hallucination in SFT inference mode
original_frontend_sft = model.frontend.frontend_sft
def patched_frontend_sft(tts_text, spk_id):
    model_input = original_frontend_sft(tts_text, spk_id)
    instruct = 'You are a helpful assistant.<|endofprompt|>'
    prompt_token, prompt_token_len = model.frontend._extract_text_token(instruct)
    model_input['prompt_text'] = prompt_token
    model_input['prompt_text_len'] = prompt_token_len
    return model_input
model.frontend.frontend_sft = patched_frontend_sft

all_audio = []
for audio in model.inference_sft("Hä, wie meinst du das jetzt?", 'thorsten_hessisch', stream=False):
    all_audio.append(audio['tts_speech'])

final_audio = torch.cat(all_audio, dim=1)
torchaudio.save('output.wav', final_audio, model.sample_rate)

Known Limitations

• Trained on a comparatively small dataset (~2,100 utterances); generalization to unseen sentence structures or rare words may vary.
• Vocoder is not dialect-specific (see above).
• Single speaker only, no voice cloning support.

License

Apache-2.0 (inherited from the base model, FunAudioLLM/Fun-CosyVoice3-0.5B-2512).

Related

• Thorsten-Voice/CosyVoice3 — standard German (non-dialect) variant
• Thorsten-Voice Website — free German speech datasets and TTS models