|
Download EVALUATION.md from owensong/Inflect-Micro-v3: direct link, hf CLI and curl.
- Browser
- Download file 3.3 kB
-
https://huggingface.co/owensong/Inflect-Micro-v3/resolve/main/EVALUATION.md
- Command line
-
hf download hf://owensong/Inflect-Micro-v3/EVALUATION.md
-
curl -L -o EVALUATION.md https://huggingface.co/owensong/Inflect-Micro-v3/resolve/main/EVALUATION.md
3.3 kB
| # Micro Native24 Evaluation | |
| Full corpus: 1,688 texts x six voices = 10,128 clips. | |
| These are completed local reproductions on a repeatedly used known/development corpus, not an official submission, a new independent test, or certified training-disjoint evidence. | |
| ## Configuration | |
| Evaluated public revision: `47c4c0e465ede4a9e03f33204bc59750c01d1fcd`. | |
| Original/base speech, native 24 kHz, PCM16 recording, seed 42, all six fixed voices. No events, enhancer, VoiceTuner or pitch polish; core weights are unchanged. | |
| ## Word Error Rate | |
| WER is **percent; lower is better**. Qwen3-ASR-1.7B uses the pinned official-source normalizer, scorer and corpus units, without a reference-text prompt. This is not an official leaderboard run. | |
| ### Local execution recipe | |
| Each voice job calls the pinned official transcriber on consecutive blocks of up to **24 rows**. Official duration sorting is applied **within each block manifest**, with batch size **8**, at most **256 new tokens**, a **30-second audio cap**, and **BF16 CUDA** inference. This is not duration sorting of the entire job as one global manifest. | |
| Execution used a local **NVIDIA RTX 5090**, not the upstream reference/default **NVIDIA H200**. The shared metric normalizer/source does not establish bit-for-bit equivalence to the official whole-manifest execution, a leaderboard submission, or a rank. | |
| | Aggregate | WER % | | |
| | --- | ---: | | |
| | Equal dataset, then equal six-voice mean | 0.703185 | | |
| | Pooled word counts (separate) | 0.843623 | | |
| Pooled counts: 1074 edits / 127308 reference words. Full and subset scopes are never mixed. | |
| | Voice | CV3 % | Seed-TTS % | MiniMax % | Dataset mean % | | |
| | --- | ---: | ---: | ---: | ---: | | |
| | hope | 1.084452 | 0.778829 | 0.210526 | 0.691269 | | |
| | sophia | 1.206452 | 0.787204 | 0.210526 | 0.734728 | | |
| | cassidy | 1.206452 | 0.753706 | 0.210526 | 0.723562 | | |
| | james | 1.084452 | 0.762080 | 0.210526 | 0.685686 | | |
| | jasper | 1.070896 | 0.745331 | 0.210526 | 0.675585 | | |
| | mark | 1.152230 | 0.762080 | 0.210526 | 0.708279 | | |
| ## Objective Quality Proxies | |
| A separately frozen 180-clip panel (30 per voice), reused without resynthesis. These are objective model scores, **not human MOS** or a claim of superior audible quality. Scoring uses a 30-second per-clip cap; UTMOSv2 averages three seeded predictions per clip. | |
| | Proxy | Six-voice mean | Clips | | |
| | --- | ---: | ---: | | |
| | UTMOS22Strong | 4.392165 | 180 | | |
| | UTMOSv2 | 3.706836 | 180 | | |
| ## Evidence and Limits | |
| Machine-readable counts, scope and receipt hashes: [evaluation.json](evaluation.json). | |
| Objective-quality proof includes each metric's reader-admission hash, independently pinned asset/source receipts and weight hashes; UTMOSv2 additionally records its verified runtime/source-tree receipt. These are saved pre-scoring checks, not a new remote-payload audit by this publication step. | |
| The known corpus has informed prior development. Reusing fixed voices and texts does not establish generalization, cloning quality, or all-platform compatibility. | |
| Saved receipts and arithmetic checked; this preparation does not rerun ASR, official normalization/alignment, quality models, or human listening. No official leaderboard submission. | |
| Saved audio sample inputs and container identities are separate: [audio provenance](assets/audio-provenance.json). | |