Download EVALUATION.md from owensong/Inflect-Mini-v3: direct link, hf CLI and curl.
- Browser
- Download file 3.3 kB
-
https://huggingface.co/owensong/Inflect-Mini-v3/resolve/main/EVALUATION.md
- Command line
-
hf download hf://owensong/Inflect-Mini-v3/EVALUATION.md
-
curl -L -o EVALUATION.md https://huggingface.co/owensong/Inflect-Mini-v3/resolve/main/EVALUATION.md
Mini Native24 Evaluation
Full corpus: 1,688 texts x six voices = 10,128 clips.
These are completed local reproductions on a repeatedly used known/development corpus, not an official submission, a new independent test, or certified training-disjoint evidence.
Configuration
Evaluated public revision: 9f539a24df04fe28e3d4507934b813d6654ddb2e.
Original/base speech, native 24 kHz, PCM16 recording, seed 42, all six fixed voices. No events, enhancer, VoiceTuner or pitch polish; core weights are unchanged.
Word Error Rate
WER is percent; lower is better. Qwen3-ASR-1.7B uses the pinned official-source normalizer, scorer and corpus units, without a reference-text prompt. This is not an official leaderboard run.
Local execution recipe
Each voice job calls the pinned official transcriber on consecutive blocks of up to 24 rows. Official duration sorting is applied within each block manifest, with batch size 8, at most 256 new tokens, a 30-second audio cap, and BF16 CUDA inference. This is not duration sorting of the entire job as one global manifest.
Execution used a local NVIDIA RTX 5090, not the upstream reference/default NVIDIA H200. The shared metric normalizer/source does not establish bit-for-bit equivalence to the official whole-manifest execution, a leaderboard submission, or a rank.
| Aggregate | WER % |
|---|---|
| Equal dataset, then equal six-voice mean | 0.739006 |
| Pooled word counts (separate) | 0.875043 |
Pooled counts: 1114 edits / 127308 reference words. Full and subset scopes are never mixed.
| Voice | CV3 % | Seed-TTS % | MiniMax % | Dataset mean % |
|---|---|---|---|---|
| hope | 1.220008 | 0.736957 | 0.263158 | 0.740041 |
| sophia | 1.152230 | 0.820702 | 0.210526 | 0.727819 |
| cassidy | 1.179341 | 0.762080 | 0.263158 | 0.734860 |
| james | 1.043785 | 0.862574 | 0.315789 | 0.740716 |
| jasper | 1.111563 | 0.812327 | 0.263158 | 0.729016 |
| mark | 1.098007 | 0.870949 | 0.315789 | 0.761582 |
Objective Quality Proxies
A separately frozen 180-clip panel (30 per voice), reused without resynthesis. These are objective model scores, not human MOS or a claim of superior audible quality. Scoring uses a 30-second per-clip cap; UTMOSv2 averages three seeded predictions per clip.
| Proxy | Six-voice mean | Clips |
|---|---|---|
| UTMOS22Strong | 4.478976 | 180 |
| UTMOSv2 | 3.916612 | 180 |
Evidence and Limits
Machine-readable counts, scope and receipt hashes: evaluation.json. Objective-quality proof includes each metric's reader-admission hash, independently pinned asset/source receipts and weight hashes; UTMOSv2 additionally records its verified runtime/source-tree receipt. These are saved pre-scoring checks, not a new remote-payload audit by this publication step. The known corpus has informed prior development. Reusing fixed voices and texts does not establish generalization, cloning quality, or all-platform compatibility. Saved receipts and arithmetic checked; this preparation does not rerun ASR, official normalization/alignment, quality models, or human listening. No official leaderboard submission.
Saved audio sample inputs and container identities are separate: audio provenance.