Inflect-Micro-v3 / EVALUATION.md
owensong's picture
Document completed native24 evaluation and audio provenance
5f86d39 verified
|
Raw History Blame Contribute Delete
3.3 kB

Micro Native24 Evaluation

Full corpus: 1,688 texts x six voices = 10,128 clips.

These are completed local reproductions on a repeatedly used known/development corpus, not an official submission, a new independent test, or certified training-disjoint evidence.

Configuration

Evaluated public revision: 47c4c0e465ede4a9e03f33204bc59750c01d1fcd. Original/base speech, native 24 kHz, PCM16 recording, seed 42, all six fixed voices. No events, enhancer, VoiceTuner or pitch polish; core weights are unchanged.

Word Error Rate

WER is percent; lower is better. Qwen3-ASR-1.7B uses the pinned official-source normalizer, scorer and corpus units, without a reference-text prompt. This is not an official leaderboard run.

Local execution recipe

Each voice job calls the pinned official transcriber on consecutive blocks of up to 24 rows. Official duration sorting is applied within each block manifest, with batch size 8, at most 256 new tokens, a 30-second audio cap, and BF16 CUDA inference. This is not duration sorting of the entire job as one global manifest.

Execution used a local NVIDIA RTX 5090, not the upstream reference/default NVIDIA H200. The shared metric normalizer/source does not establish bit-for-bit equivalence to the official whole-manifest execution, a leaderboard submission, or a rank.

Aggregate WER %
Equal dataset, then equal six-voice mean 0.703185
Pooled word counts (separate) 0.843623

Pooled counts: 1074 edits / 127308 reference words. Full and subset scopes are never mixed.

Voice CV3 % Seed-TTS % MiniMax % Dataset mean %
hope 1.084452 0.778829 0.210526 0.691269
sophia 1.206452 0.787204 0.210526 0.734728
cassidy 1.206452 0.753706 0.210526 0.723562
james 1.084452 0.762080 0.210526 0.685686
jasper 1.070896 0.745331 0.210526 0.675585
mark 1.152230 0.762080 0.210526 0.708279

Objective Quality Proxies

A separately frozen 180-clip panel (30 per voice), reused without resynthesis. These are objective model scores, not human MOS or a claim of superior audible quality. Scoring uses a 30-second per-clip cap; UTMOSv2 averages three seeded predictions per clip.

Proxy Six-voice mean Clips
UTMOS22Strong 4.392165 180
UTMOSv2 3.706836 180

Evidence and Limits

Machine-readable counts, scope and receipt hashes: evaluation.json. Objective-quality proof includes each metric's reader-admission hash, independently pinned asset/source receipts and weight hashes; UTMOSv2 additionally records its verified runtime/source-tree receipt. These are saved pre-scoring checks, not a new remote-payload audit by this publication step. The known corpus has informed prior development. Reusing fixed voices and texts does not establish generalization, cloning quality, or all-platform compatibility. Saved receipts and arithmetic checked; this preparation does not rerun ASR, official normalization/alignment, quality models, or human listening. No official leaderboard submission.

Saved audio sample inputs and container identities are separate: audio provenance.