Brand model card as FDSpeech

1c36a7e verified about 18 hours ago

5.63 kB

license: apache-2.0
language:
  - en
pipeline_tag: text-to-speech
base_model: openbmb/VoxCPM2
library_name: voxcpm
tags:
  - text-to-speech
  - flow-matching
  - frechet-distance
  - few-step-generation
  - lora
  - fdspeech

FDSpeech-VoxCPM2

FDSpeech-VoxCPM2 is the selected compact three-target LoRA adapter from the paper Fréchet Distance Loss on Speech Representations for Text-to-Speech Synthesis. It is trained with a Fréchet-distance loss on speech representations for intelligible four-step generation.

It is not a standalone TTS checkpoint. Use it with the external openbmb/VoxCPM2 base model. The FD loss is used only during fine-tuning; inference is the ordinary four-step base model with the LoRA adapter loaded.

Model details

Base model: openbmb/VoxCPM2, approximately 2B parameters
Artifact type: LoRA adapter
Adapter: rank 32, alpha 32, q/k/v/o projections in the language model and DiT
Inference sampler: four Euler steps, CFG 2.35 in the paper evaluation
Selected checkpoint: srfd_compact3/step_0001600
FDSpeech targets: low-step Whisper anchor, ten-step teacher CTC, real-speech CTC
Primary evaluation: Seed-TTS English test-en

Files

lora_config.json
lora_weights.safetensors
training_state.json
adapters/compact3_balanced/
ablations/
configs/
reports/

Optimizer, scheduler, reference statistics, and FD-loss feature-queue state are not included because they are not needed for inference. Base-model weights are downloaded separately.

Usage

pip install -U voxcpm huggingface_hub soundfile

import json
import os

import soundfile as sf
from huggingface_hub import snapshot_download
from voxcpm import VoxCPM
from voxcpm.model.voxcpm import LoRAConfig

adapter_dir = snapshot_download("voidful/FDSpeech-VoxCPM2")
with open(os.path.join(adapter_dir, "lora_config.json"), encoding="utf-8") as handle:
    adapter_info = json.load(handle)

model = VoxCPM.from_pretrained(
    hf_model_id="openbmb/VoxCPM2",
    load_denoiser=False,
    optimize=True,
    lora_config=LoRAConfig(**adapter_info["lora_config"]),
    lora_weights_path=adapter_dir,
)

wav = model.generate(
    text="The quick brown fox jumps over the lazy dog.",
    cfg_value=2.35,
    inference_timesteps=4,
    normalize=True,
    denoise=False,
    seed=0,
)
sf.write("fdspeech.wav", wav, model.tts_model.sample_rate)

The first run downloads the base model. A CUDA GPU is recommended. For continuation-style voice cloning, provide a consented prompt_wav_path and its exact prompt_text.

Training data and reference statistics

The paper fine-tunes on a 767-row manifest derived from LibriTTS voice-cloning material. Offline FDSpeech reference moments are computed from ASR-verified four-step generations, ten-step teacher generations, and real LibriTTS speech. The training manifest, source/reference audio, and precomputed moments are not redistributed in this repository.

See the training config for the released recipe and the integration guide for implementation details. The srfd path and config key are retained as compatibility identifiers for the released training artifacts.

Evaluation

Results use the upstream Seed-TTS English scorer over 1,088 prompts and 11,805 reference words.

System	Steps	Upstream WER ↓	SIM ↑	UTMOS / DNSMOS OVRL / P808 ↑
VoxCPM2	4	263/11805 = 2.2279%	0.7433	3.2974 / 2.8950 / 3.5296
VoxCPM2	10	205/11805 = 1.7366%	0.7610	3.8072 / 3.0866 / 3.6689
FDSpeech-VoxCPM2	4	167/11805 = 1.4147%	0.7613	3.7637 / 3.0711 / 3.6507

The WER reductions against both original baselines are significant under an utterance-level paired bootstrap. SIM, UTMOS, and DNSMOS are objective proxies, not human MOS. A blinded comparison with the ten-step baseline produced a near even decisive preference split, with equivalence supported within the paper's pre-specified 10-point margin. See the paper for the complete protocol and confidence intervals.

Intended use

Research on few-step flow-matching TTS and distributional regularization
Reproduction and analysis of the paper's four-step English setting
Evaluation of the released adapter on consented speech prompts

Limitations and risks

Evidence is concentrated on English Seed-TTS; multilingual gains are not established.
FDSpeech primarily targets intelligibility and is not a general perceptual-quality objective.
Aggregate WER improves, but individual prompts can still regress or contain substitutions.
Raw representation FD should not be used as a standalone quality or checkpoint-selection metric.
Voice cloning can enable impersonation and fraud. Use only consented voices, label synthetic audio, and do not use it for identity or access-control bypass.

License

The adapter and FDSpeech code are released under Apache-2.0. The base model, pretrained extractors, datasets, and evaluation tools remain subject to their own terms.

Citation

@article{chung2026fdspeech,
  title   = {Fr\'{e}chet Distance Loss on Speech Representations for Text-to-Speech Synthesis},
  author  = {Chung, Ho-Lam and Huang, Kuan-Po and Lu, Bo-Ru and Lee, Hung-yi},
  journal = {arXiv preprint arXiv:2607.06027},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.06027}
}