|
Download EVALUATION.md from owensong/Inflect-Mini-v3: direct link, hf CLI and curl.
- Browser
- Download file 3.3 kB
-
https://huggingface.co/owensong/Inflect-Mini-v3/resolve/main/EVALUATION.md
- Command line
-
hf download hf://owensong/Inflect-Mini-v3/EVALUATION.md
-
curl -L -o EVALUATION.md https://huggingface.co/owensong/Inflect-Mini-v3/resolve/main/EVALUATION.md
3.3 kB
| # Mini Native24 Evaluation | |
| Full corpus: 1,688 texts x six voices = 10,128 clips. | |
| These are completed local reproductions on a repeatedly used known/development corpus, not an official submission, a new independent test, or certified training-disjoint evidence. | |
| ## Configuration | |
| Evaluated public revision: `9f539a24df04fe28e3d4507934b813d6654ddb2e`. | |
| Original/base speech, native 24 kHz, PCM16 recording, seed 42, all six fixed voices. No events, enhancer, VoiceTuner or pitch polish; core weights are unchanged. | |
| ## Word Error Rate | |
| WER is **percent; lower is better**. Qwen3-ASR-1.7B uses the pinned official-source normalizer, scorer and corpus units, without a reference-text prompt. This is not an official leaderboard run. | |
| ### Local execution recipe | |
| Each voice job calls the pinned official transcriber on consecutive blocks of up to **24 rows**. Official duration sorting is applied **within each block manifest**, with batch size **8**, at most **256 new tokens**, a **30-second audio cap**, and **BF16 CUDA** inference. This is not duration sorting of the entire job as one global manifest. | |
| Execution used a local **NVIDIA RTX 5090**, not the upstream reference/default **NVIDIA H200**. The shared metric normalizer/source does not establish bit-for-bit equivalence to the official whole-manifest execution, a leaderboard submission, or a rank. | |
| | Aggregate | WER % | | |
| | --- | ---: | | |
| | Equal dataset, then equal six-voice mean | 0.739006 | | |
| | Pooled word counts (separate) | 0.875043 | | |
| Pooled counts: 1114 edits / 127308 reference words. Full and subset scopes are never mixed. | |
| | Voice | CV3 % | Seed-TTS % | MiniMax % | Dataset mean % | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | hope | 1.220008 | 0.736957 | 0.263158 | 0.740041 | | |
| | sophia | 1.152230 | 0.820702 | 0.210526 | 0.727819 | | |
| | cassidy | 1.179341 | 0.762080 | 0.263158 | 0.734860 | | |
| | james | 1.043785 | 0.862574 | 0.315789 | 0.740716 | | |
| | jasper | 1.111563 | 0.812327 | 0.263158 | 0.729016 | | |
| | mark | 1.098007 | 0.870949 | 0.315789 | 0.761582 | | |
| ## Objective Quality Proxies | |
| A separately frozen 180-clip panel (30 per voice), reused without resynthesis. These are objective model scores, **not human MOS** or a claim of superior audible quality. Scoring uses a 30-second per-clip cap; UTMOSv2 averages three seeded predictions per clip. | |
| | Proxy | Six-voice mean | Clips | | |
| | --- | ---: | ---: | | |
| | UTMOS22Strong | 4.478976 | 180 | | |
| | UTMOSv2 | 3.916612 | 180 | | |
| ## Evidence and Limits | |
| Machine-readable counts, scope and receipt hashes: [evaluation.json](evaluation.json). | |
| Objective-quality proof includes each metric's reader-admission hash, independently pinned asset/source receipts and weight hashes; UTMOSv2 additionally records its verified runtime/source-tree receipt. These are saved pre-scoring checks, not a new remote-payload audit by this publication step. | |
| The known corpus has informed prior development. Reusing fixed voices and texts does not establish generalization, cloning quality, or all-platform compatibility. | |
| Saved receipts and arithmetic checked; this preparation does not rerun ASR, official normalization/alignment, quality models, or human listening. No official leaderboard submission. | |
| Saved audio sample inputs and container identities are separate: [audio provenance](assets/audio-provenance.json). | |