Inflect-Mini-v3 / EVALUATION.md
owensong's picture
Document completed native24 evaluation and audio provenance
d133c91 verified
|
Raw History Blame Contribute Delete
3.3 kB
# Mini Native24 Evaluation
Full corpus: 1,688 texts x six voices = 10,128 clips.
These are completed local reproductions on a repeatedly used known/development corpus, not an official submission, a new independent test, or certified training-disjoint evidence.
## Configuration
Evaluated public revision: `9f539a24df04fe28e3d4507934b813d6654ddb2e`.
Original/base speech, native 24 kHz, PCM16 recording, seed 42, all six fixed voices. No events, enhancer, VoiceTuner or pitch polish; core weights are unchanged.
## Word Error Rate
WER is **percent; lower is better**. Qwen3-ASR-1.7B uses the pinned official-source normalizer, scorer and corpus units, without a reference-text prompt. This is not an official leaderboard run.
### Local execution recipe
Each voice job calls the pinned official transcriber on consecutive blocks of up to **24 rows**. Official duration sorting is applied **within each block manifest**, with batch size **8**, at most **256 new tokens**, a **30-second audio cap**, and **BF16 CUDA** inference. This is not duration sorting of the entire job as one global manifest.
Execution used a local **NVIDIA RTX 5090**, not the upstream reference/default **NVIDIA H200**. The shared metric normalizer/source does not establish bit-for-bit equivalence to the official whole-manifest execution, a leaderboard submission, or a rank.
| Aggregate | WER % |
| --- | ---: |
| Equal dataset, then equal six-voice mean | 0.739006 |
| Pooled word counts (separate) | 0.875043 |
Pooled counts: 1114 edits / 127308 reference words. Full and subset scopes are never mixed.
| Voice | CV3 % | Seed-TTS % | MiniMax % | Dataset mean % |
| --- | ---: | ---: | ---: | ---: |
| hope | 1.220008 | 0.736957 | 0.263158 | 0.740041 |
| sophia | 1.152230 | 0.820702 | 0.210526 | 0.727819 |
| cassidy | 1.179341 | 0.762080 | 0.263158 | 0.734860 |
| james | 1.043785 | 0.862574 | 0.315789 | 0.740716 |
| jasper | 1.111563 | 0.812327 | 0.263158 | 0.729016 |
| mark | 1.098007 | 0.870949 | 0.315789 | 0.761582 |
## Objective Quality Proxies
A separately frozen 180-clip panel (30 per voice), reused without resynthesis. These are objective model scores, **not human MOS** or a claim of superior audible quality. Scoring uses a 30-second per-clip cap; UTMOSv2 averages three seeded predictions per clip.
| Proxy | Six-voice mean | Clips |
| --- | ---: | ---: |
| UTMOS22Strong | 4.478976 | 180 |
| UTMOSv2 | 3.916612 | 180 |
## Evidence and Limits
Machine-readable counts, scope and receipt hashes: [evaluation.json](evaluation.json).
Objective-quality proof includes each metric's reader-admission hash, independently pinned asset/source receipts and weight hashes; UTMOSv2 additionally records its verified runtime/source-tree receipt. These are saved pre-scoring checks, not a new remote-payload audit by this publication step.
The known corpus has informed prior development. Reusing fixed voices and texts does not establish generalization, cloning quality, or all-platform compatibility.
Saved receipts and arithmetic checked; this preparation does not rerun ASR, official normalization/alignment, quality models, or human listening. No official leaderboard submission.
Saved audio sample inputs and container identities are separate: [audio provenance](assets/audio-provenance.json).