YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Evaluation
Evaluated on 50 randomly-sampled utterances from the training corpus. Reference recordings are the original human speakers.
β οΈ Training-set evaluation. The fine-tuned model saw all 1,924 utterances during training, so these metrics measure fit rather than generalisation. The base model has not seen them. A held-out evaluation requires retraining with a train/val/test split (see Future work).
Headline results β three alignment-independent metrics
These are the metrics that do not depend on DTW/spectrogram alignment, so they are not affected by reference-vs-generation duration differences.
| Metric | Fine-tuned (mean Β± std) | Base (mean Β± std) | Ξ | Direction |
|---|---|---|---|---|
| Speaker similarity | 0.699 Β± 0.054 | 0.582 Β± 0.050 | +0.117 (>2Ο) | β better |
| Duration ratio | 0.924 Β± 0.375 | 0.693 Β± 0.230 | +0.231 | β1.0 |
| Mel-L1 (log-mel MAE) | 1.615 Β± 0.471 | 2.981 Β± 0.629 | β1.366 | β better |
The base checkpoint generates utterances 31% shorter than reference (mean duration ratio 0.69); the fine-tuned model reduces this to 8% (0.924). This directly explains the strong speaker-similarity gain β matched timing preserves speaker-level prosody that Resemblyzer captures.
Alignment-dependent metrics (relative improvement only)
These use DTW / time-stretch alignment, so their absolute values are sensitive to corpus characteristics (short utterances, cross-domain acoustic mismatch). Report only the relative change; do not compare the absolutes to literature.
| Metric | Fine-tuned | Base | Relative change |
|---|---|---|---|
| MCD (dB) | 230.1 | 406.0 | β43% |
| F0 RMSE (Hz) | 80.8 | 90.8 | β11% |
| PESQ | 1.15 | 1.06 | +8% |
| STOI | 0.200 | 0.208 | β4% (tied β see note) |
Note on STOI: the fine-tuned model's intelligibility score is statistically tied with the base. The base model's score is depressed by DTW distortion caused by its 31%-short output; the fine-tuned model's score is depressed by residual pitch-reconstruction error on short utterances. Both will need a held-out evaluation to measure cleanly.
Artifacts
evaluation/summary.csvβ full tableevaluation/results_per_sample.csvβ 50 rows Γ 17 columnsevaluation/metrics_summary.pngevaluation/spectrograms.png
Method
- Speaker similarity: Resemblyzer (2019) embeddings, cosine similarity.
- Duration ratio:
len(gen) / len(ref). - Mel-L1: DTW-aligned log-mel MAE (80 bands).
- MCD: dB-scale MFCC-13, C0 excluded, DTW-aligned.
- STOI/PESQ: computed after
librosa.effects.time_stretchto equal duration.
Future work
- Held-out evaluation: retrain with an 80/10/10 split; report test metrics. Estimated cost: ~6 GPU-hours on a T4 for retrain + evaluation.
- Subjective MOS panel with native Fulfulde listeners (nβ₯10, 5-point scale).
- From-scratch VITS baseline on the same corpus (~100+ hours of speech would be required for convergence; current dataset is 1,924 utterances).
- Downloads last month
- 50

