YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Evaluation

Evaluated on 50 randomly-sampled utterances from the training corpus. Reference recordings are the original human speakers.

⚠️ Training-set evaluation. The fine-tuned model saw all 1,924 utterances during training, so these metrics measure fit rather than generalisation. The base model has not seen them. A held-out evaluation requires retraining with a train/val/test split (see Future work).

Headline results β€” three alignment-independent metrics

These are the metrics that do not depend on DTW/spectrogram alignment, so they are not affected by reference-vs-generation duration differences.

Metric Fine-tuned (mean Β± std) Base (mean Β± std) Ξ” Direction
Speaker similarity 0.699 Β± 0.054 0.582 Β± 0.050 +0.117 (>2Οƒ) ↑ better
Duration ratio 0.924 Β± 0.375 0.693 Β± 0.230 +0.231 β‰ˆ1.0
Mel-L1 (log-mel MAE) 1.615 Β± 0.471 2.981 Β± 0.629 βˆ’1.366 ↓ better

The base checkpoint generates utterances 31% shorter than reference (mean duration ratio 0.69); the fine-tuned model reduces this to 8% (0.924). This directly explains the strong speaker-similarity gain β€” matched timing preserves speaker-level prosody that Resemblyzer captures.

Alignment-dependent metrics (relative improvement only)

These use DTW / time-stretch alignment, so their absolute values are sensitive to corpus characteristics (short utterances, cross-domain acoustic mismatch). Report only the relative change; do not compare the absolutes to literature.

Metric Fine-tuned Base Relative change
MCD (dB) 230.1 406.0 βˆ’43%
F0 RMSE (Hz) 80.8 90.8 βˆ’11%
PESQ 1.15 1.06 +8%
STOI 0.200 0.208 βˆ’4% (tied β€” see note)

Note on STOI: the fine-tuned model's intelligibility score is statistically tied with the base. The base model's score is depressed by DTW distortion caused by its 31%-short output; the fine-tuned model's score is depressed by residual pitch-reconstruction error on short utterances. Both will need a held-out evaluation to measure cleanly.

Artifacts

Method

  • Speaker similarity: Resemblyzer (2019) embeddings, cosine similarity.
  • Duration ratio: len(gen) / len(ref).
  • Mel-L1: DTW-aligned log-mel MAE (80 bands).
  • MCD: dB-scale MFCC-13, C0 excluded, DTW-aligned.
  • STOI/PESQ: computed after librosa.effects.time_stretch to equal duration.

Future work

  1. Held-out evaluation: retrain with an 80/10/10 split; report test metrics. Estimated cost: ~6 GPU-hours on a T4 for retrain + evaluation.
  2. Subjective MOS panel with native Fulfulde listeners (nβ‰₯10, 5-point scale).
  3. From-scratch VITS baseline on the same corpus (~100+ hours of speech would be required for convergence; current dataset is 1,924 utterances).

Metrics summary Spectrograms

Downloads last month
50
Safetensors
Model size
83M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support