docs/method.md

SR-FD: Method

SR-FD (Speech Representation Fréchet Distance loss) is one extra training loss added to a standard few-step TTS fine-tuning recipe. During fine-tuning, the model synthesizes speech with the same few-step sampler used at deployment, and frozen speech encoders turn the generated audio into feature vectors. Each set of feature vectors is summarized by its mean and covariance (first- and second-order moments), and the moments of generated speech are pushed toward reference moments computed offline from desirable speech. The distance between the two moment sets is a Fréchet distance — the same quantity behind FID and FAD — used here as a differentiable loss. The loss needs no discriminator, adds no parameters, and is removed at test time, so inference is unchanged.

1. Base objective

The base model (an external, tokenizer-free flow-matching autoregressive TTS model — VoxCPM2) decodes continuous acoustic latents with a diffusion transformer trained by conditional flow matching. Given a real latent x1 and noise z ~ N(0, I), draw t ∈ [0,1], form y = (1-t) x1 + t z with constant velocity v = z - x1, and regress the predicted velocity:

L_fm = E ‖ u_θ(y, t) − v ‖²

At inference the decoder integrates the velocity field with a few Euler steps; with only four steps the integration is very coarse — exactly the regime SR-FD targets. The full fine-tuning objective is

L = w_fm · L_fm + w_stop · L_stop + L_aux + λ_srfd · L_srfd

L_stop is a stop-prediction loss; L_aux collects three small auxiliary losses inherited from the underlying recipe (teacher-endpoint, preference- feature, Whisper-text) with fixed small weights. SR-FD is the L_srfd term.

2. Matching the sampled-speech distribution

Standard fine-tuning supervises teacher-forced frames, so the few-step sampler never appears in the loss and training can look healthy while the four-step sampler drifts. SR-FD operates directly on sampled speech: during each update the model synthesizes a complete short utterance with the deployment-time four-step sampler, keeping the computation differentiable (see integration.md). Each frozen extractor φ_k maps the generated audio g_θ(x_b) to one utterance-level feature vector h_b^k = φ_k(g_θ(x_b)) ∈ R^{d_k}.

3. Two extractors, three targets

Target	Source	Extractor	Role
Low-step Whisper anchor	ASR-verified 4-step generations	Whisper	Low-step content anchor
Teacher CTC target	10-step teacher generations	CTC	Higher-step content transfer
Real-speech CTC target	Real LibriTTS speech	CTC	Natural-speech grounding

The two frozen extractors are a Whisper-large-v3 encoder (semantic content, srfd/extractors.py::WhisperEncoderAnchorExtractor) and a wav2vec2 CTC model (phonetic content, CTCPosteriorContentStatsExtractor). All three targets describe content, because content drift dominates the four-step failures: the audio stays speech-like but an ASR system no longer recovers the intended words. The Whisper anchor describes good low-step outputs, the teacher target imports higher-step behavior, and the real-speech target grounds everything in natural speech.

4. Reference and generated moments

Reference moments are precomputed offline from each target corpus (scripts/compute_reference_stats.py) and only the moments are stored — the reference audio is never used again. The CTC features are low-dimensional with a well-conditioned covariance; the Whisper features are high-dimensional relative to the sample count, so their covariance is rank-deficient and is regularized with a small ε I before the matrix square root. Because the absolute Whisper Fréchet value is biased by this rank deficiency, models are never selected by raw FD (see §7).

Generated moments are estimated from a feature queue. A covariance from a few utterances is meaningless, while generating hundreds of utterances per update is unaffordable, so for each extractor a queue Q_t^k of features from recent updates is kept. At step t the generated moments are computed over the queue together with the current mini-batch. Features from earlier steps are detached; only the current mini-batch keeps gradient — a large-sample moment estimate at the memory cost of a single batch. (srfd/loss.py::SRFDEmaLoss, stats_mode="queue".)

5. The SR-FD loss

For each extractor k and target j, SR-FD computes a Fréchet distance between the generated and reference Gaussian moment estimates (srfd/frechet.py):

FD = ‖ μ_g − μ_r ‖² + Tr(Σ_g + Σ_r − 2 (Σ_r^{1/2} Σ_g Σ_r^{1/2})^{1/2})

Different feature spaces have different natural scales, so each term is divided by its own detached value:

FD̃ = FD / stopgrad(FD + ε)

Each normalized term has magnitude near one, but its gradient still points in the FD-reducing direction, so targets are balanced by gradient scale rather than raw distance. The total loss is a weighted average of the normalized terms, first across targets within each extractor and then across extractors; with the paper weights the Whisper and CTC branches contribute equally and the two CTC targets split the CTC half.

A length gate admits a sample into the loss only when its generated-to-target duration ratio is close to one ([0.92, 1.08]), since strongly mismatched samples usually contain truncation or runaway speech and matching moments on them injects noise.

At test time the extractors, queues, reference moments, and Fréchet computation all disappear: the deployed model is a plain four-step model with LoRA adapters, with no added parameters and no added inference computation.

6. Hyperparameters

See configs/srfd_compact3.yaml. Key values: LoRA rank 32 / alpha 32 on the q,k,v,o projections of the LM and DiT; λ_srfd = 2e-4; raw target weights 1.0 / 0.5 / 0.5; both extractor weights 1.0; feature queue of 50,000 vectors; length gate [0.92, 1.08]; four-step Euler sampling with guidance 2.45 during training; 1600 fine-tuning steps with AdamW (weight decay 0.01, grad-norm clip 0.03), bf16, batch size 1, cosine LR 3e-8 → 0 with no warmup.

7. Is the Fréchet distance a good diagnostic?

No. SR-FD trains the model to reduce representation FD, but a smaller raw FD does not imply lower WER: across saved checkpoints the correlation between raw FD and WER is weak, and an external CTC FD pass confirms training moves generated features toward the reference while the raw value does not track WER. Reference targets are therefore validated through a WER ablation, not through absolute FD, and representation FD is not used to select checkpoints or as a standalone quality claim.