Inflect-Mini-v3 / EVALUATION.md
owensong's picture
Document completed native24 evaluation and audio provenance
d133c91 verified
|
Raw History Blame Contribute Delete
3.3 kB

Mini Native24 Evaluation

Full corpus: 1,688 texts x six voices = 10,128 clips.

These are completed local reproductions on a repeatedly used known/development corpus, not an official submission, a new independent test, or certified training-disjoint evidence.

Configuration

Evaluated public revision: 9f539a24df04fe28e3d4507934b813d6654ddb2e. Original/base speech, native 24 kHz, PCM16 recording, seed 42, all six fixed voices. No events, enhancer, VoiceTuner or pitch polish; core weights are unchanged.

Word Error Rate

WER is percent; lower is better. Qwen3-ASR-1.7B uses the pinned official-source normalizer, scorer and corpus units, without a reference-text prompt. This is not an official leaderboard run.

Local execution recipe

Each voice job calls the pinned official transcriber on consecutive blocks of up to 24 rows. Official duration sorting is applied within each block manifest, with batch size 8, at most 256 new tokens, a 30-second audio cap, and BF16 CUDA inference. This is not duration sorting of the entire job as one global manifest.

Execution used a local NVIDIA RTX 5090, not the upstream reference/default NVIDIA H200. The shared metric normalizer/source does not establish bit-for-bit equivalence to the official whole-manifest execution, a leaderboard submission, or a rank.

Aggregate WER %
Equal dataset, then equal six-voice mean 0.739006
Pooled word counts (separate) 0.875043

Pooled counts: 1114 edits / 127308 reference words. Full and subset scopes are never mixed.

Voice CV3 % Seed-TTS % MiniMax % Dataset mean %
hope 1.220008 0.736957 0.263158 0.740041
sophia 1.152230 0.820702 0.210526 0.727819
cassidy 1.179341 0.762080 0.263158 0.734860
james 1.043785 0.862574 0.315789 0.740716
jasper 1.111563 0.812327 0.263158 0.729016
mark 1.098007 0.870949 0.315789 0.761582

Objective Quality Proxies

A separately frozen 180-clip panel (30 per voice), reused without resynthesis. These are objective model scores, not human MOS or a claim of superior audible quality. Scoring uses a 30-second per-clip cap; UTMOSv2 averages three seeded predictions per clip.

Proxy Six-voice mean Clips
UTMOS22Strong 4.478976 180
UTMOSv2 3.916612 180

Evidence and Limits

Machine-readable counts, scope and receipt hashes: evaluation.json. Objective-quality proof includes each metric's reader-admission hash, independently pinned asset/source receipts and weight hashes; UTMOSv2 additionally records its verified runtime/source-tree receipt. These are saved pre-scoring checks, not a new remote-payload audit by this publication step. The known corpus has informed prior development. Reusing fixed voices and texts does not establish generalization, cloning quality, or all-platform compatibility. Saved receipts and arithmetic checked; this preparation does not rerun ASR, official normalization/alignment, quality models, or human listening. No official leaderboard submission.

Saved audio sample inputs and container identities are separate: audio provenance.