Inflect v2 evaluation and benchmark report
This page is the detailed evidence record for Inflect-Micro-v2 and Inflect-Nano-v2. The model cards show the headline results; this document defines the datasets, systems, normalization, uncertainty, exclusions, runtime boundaries, and raw artifacts behind those results.
Read each metric separately: WER measures intelligibility, UTMOS22 predicts naturalness, listening tests record preference, and runtime measures deployment cost. None of them is an overall quality score.
Evidence at a glance
| Release | Complete parameters | FP32 weights | UTMOS22 ↑ | Two-ASR semantic WER ↓ | Community preference ↑ | 4-thread CPU ↑ |
|---|---|---|---|---|---|---|
| Inflect-Micro-v2 | 9,356,513 | 37.53 MB | 4.395 | 3.99% | 66.2% | 6.28× |
| Inflect-Nano-v2 | 3,966,721 | 15.97 MB | 4.386 | 4.21% | 63.9% | 10.72× |
Compared systems
The benchmark includes KittenTTS Nano, Piper Low, and Supertonic 3 because they are established compact or local TTS systems. In this package-level comparison, each has a larger deployable weight footprint than both Inflect releases. That provides useful size context, but no single result establishes overall superiority.
Voice variants are separate rows when voice identity can affect quality or intelligibility. Kitten and Piper may be shown as equal-weight two-voice means in family-level figures. Supertonic 3-step and 8-step are never pooled.
Complete automated result matrix
All systems received matched text within each benchmark. Lower WER is better; higher UTMOS22 is better.
| System / voice | Complete weights ↓ | UTMOS22 ↑ | Qwen3-ASR WER ↓ | Nemotron 3.5 WER ↓ | Whisper large-v3 WER ↓ | Two-ASR consensus ↓ |
|---|---|---|---|---|---|---|
| Inflect-Micro-v2 | 37.53 MB | 4.395 | 2.52% | 5.45% | 2.73% | 3.99% |
| Inflect-Nano-v2 | 15.97 MB | 4.386 | 2.78% | 5.63% | 2.65% | 4.21% |
| KittenTTS Nano · Bruno | 56.77 MB | 4.217 | 2.15% | 3.96% | 2.17% | 3.06% |
| KittenTTS Nano · Hugo | 56.77 MB | 4.191 | 2.39% | 3.80% | 2.11% | 3.09% |
| Piper Low · Danny | 63.10 MB | 4.195 | 2.62% | 5.60% | 2.55% | 4.11% |
| Piper Low · Ryan | 63.10 MB | 4.289 | 2.81% | 5.51% | 2.87% | 4.16% |
| Supertonic 3 · M2 · 8-step | 398.08 MB | 4.295 | 2.05% | 3.56% | 8.08% | 2.81% |
| Supertonic 3 · M2 · 3-step | 398.08 MB | 2.471 | 3.03% | 6.04% | 3.22% | 4.54% |
The two-ASR consensus is the unweighted mean of Qwen3-ASR and Nemotron 3.5 corpus-level semantic WER. Whisper remains visible but is excluded from the headline consensus for every system because its Supertonic 8-step result contains insertion-heavy outliers that conflict with the other recognizers and human listening. This rule is applied globally, not selectively to Inflect.
Uncertainty for the Inflect releases
Intervals use 10,000 matched-prompt bootstrap samples.
| Release | UTMOS22 mean (95% CI) | Qwen3-ASR WER (95% CI) | Nemotron WER (95% CI) | Whisper WER (95% CI) |
|---|---|---|---|---|
| Inflect-Micro-v2 | 4.395 (4.381–4.408) | 2.52% (1.88%–3.24%) | 5.45% (4.50%–6.50%) | 2.73% (2.05%–3.49%) |
| Inflect-Nano-v2 | 4.386 (4.372–4.399) | 2.78% (2.13%–3.52%) | 5.63% (4.72%–6.62%) | 2.65% (2.03%–3.35%) |
Modern400 intelligibility protocol
- 400 identical unseen English prompts per system
- 3,200 generated clips across eight system/voice configurations
- 200 manually constructed modern and stress prompts
- 200 deterministic Google FLEURS
en_ustest prompts - exact normalized-text exclusion against 87,362 training transcripts
- deterministic frontend and inference settings within each system
- explicit resampling to every recognizer's required sample rate
- corpus-level WER plus matched-prompt bootstrap intervals
The three independent recognizers are:
| Evaluator | Model |
|---|---|
| Qwen3-ASR | Qwen/Qwen3-ASR-1.7B |
| Nemotron 3.5 ASR | nvidia/nemotron-3.5-asr-streaming-0.6b |
| Whisper | openai/whisper-large-v3 |
Semantic normalization
Raw orthographic WER is not sufficient for TTS. A recognizer may transcribe
“twelve point six” as 12.6 or “July twenty-first” as July 21st without the
audio being wrong. The semantic normalizer canonicalizes equivalent number,
currency, date, time, unit, abbreviation, and punctuation forms before scoring.
The release retains both raw and semantic hypotheses so normalization cannot
silently hide errors.
Where Inflect errors remain
The following diagnostic uses the same two-ASR consensus as the headline and does not introduce a new score.
| Prompt category | Micro WER ↓ | Nano WER ↓ |
|---|---|---|
| Names Places | 16.67% | 17.54% |
| Short | 4.29% | 9.05% |
| Rare Words | 7.89% | 7.89% |
| Numbers | 6.25% | 4.69% |
| Homographs | 3.57% | 5.71% |
| External Heldout | 5.39% | 5.48% |
| Technical | 4.10% | 3.12% |
| Narrative | 0.57% | 1.56% |
| Punctuation | 0.00% | 1.54% |
| Long Range | 0.83% | 1.24% |
| Conversational | 0.49% | 1.13% |
| Descriptive | 0.44% | 0.88% |
| Long | 0.81% | 0.58% |
| Emotional | 0.15% | 0.30% |
| Question | 0.00% | 0.00% |
UTMOS22 predicted-naturalness protocol
- 500 identical unseen prompts per system
tarepan/SpeechMOSv1.2.0- one score per generated clip
- mean, median, standard deviation, percentiles, extrema, and 95% bootstrap CI
- Kitten and Piper family figures use equal-weight means across their two voices
UTMOS22 is a learned non-intrusive quality predictor. It is useful for large-scale screening but is not human MOS, can be sensitive to voice and recording characteristics, and must be interpreted beside intelligibility and listening evidence.
Community blind listening
The final community study stored anonymous pairwise choices rather than user identity. Model names were hidden during voting, left/right order was randomized, and a tie counts as half a win. The dataset contains 147 pairwise decisions; it does not claim a laboratory-controlled listener panel.
| System / voice | Wins | Losses | Ties | Appearances | Preference rate ↑ |
|---|---|---|---|---|---|
| KittenTTS Nano · Hugo | 22 | 8 | 3 | 33 | 71.2% |
| Inflect-Micro-v2 | 21 | 10 | 3 | 34 | 66.2% |
| Inflect-Nano-v2 | 22 | 12 | 2 | 36 | 63.9% |
| Supertonic 3 · James · 8-step | 16 | 12 | 3 | 31 | 56.5% |
| KittenTTS Nano · Bruno | 16 | 13 | 7 | 36 | 54.2% |
| Piper Low · Danny | 10 | 8 | 6 | 24 | 54.2% |
| Piper Low · Ryan | 14 | 13 | 6 | 33 | 51.5% |
| Inflect-Nano-v1 | 7 | 22 | 3 | 32 | 26.6% |
| Supertonic 3 · James · 3-step | 1 | 29 | 2 | 32 | 6.2% |
Preference rate is (wins + 0.5 × ties) / appearances. Unequal appearances
mean this table is descriptive; it should not be treated as a perfectly
balanced tournament or formal MOS.
Runtime and throughput
Managed CPU reference
The published CPU reference uses a Hugging Face CPU Upgrade instance (8 vCPU, 32 GB RAM) with four framework threads and end-to-end text-to-waveform timing. Inflect received 100 fixed Modern400 prompts across three complete passes. The first pass populated runtime caches; the published steady-state result pools passes two and three.
| Release | Steady-state RTF ↓ | Audio / wall time ↑ |
|---|---|---|
| Inflect-Micro-v2 | 0.1593 | 6.28× |
| Inflect-Nano-v2 | 0.0933 | 10.72× |
The earlier Ryzen 9 3900X CPU comparison is superseded because background host saturation and inconsistent framework thread policies made it unsuitable as release evidence.
Directional compact-system context
The same managed CPU and four-thread policy were used for a shorter comparator confirmation: the identical 50-prompt prefix repeated twice. KittenTTS and Piper are equal-work pooled across two voices.
| System | Audio / wall time ↑ |
|---|---|
| Piper Low | 31.37× |
| KittenTTS Nano | 13.33× |
| Inflect-Nano-v2 | 10.72× |
| Supertonic 3 · 3-step | 10.15× |
| Inflect-Micro-v2 | 6.28× |
| Supertonic 3 · 8-step | 4.37× |
This is deployment context, not a perfectly matched speed leaderboard: Inflect's figures come from the larger steady-state run, comparator rows use the shorter confirmation pass, and several comparator packages use optimized ONNX while the public Inflect package uses PyTorch.
The public runtime also collapses training-time weight normalization once, immediately after loading the checkpoint. A fixed-seed regression test produced bit-for-bit identical waveforms before and after this reparameterization. The downloaded checkpoints retain 9,356,513 and 3,966,721 complete parameters; the optimized live inference graphs contain 9,344,753 and 3,958,801 parameters respectively. This is a deployment optimization, not a smaller checkpoint or a changed acoustic model.
Retained CUDA engineering snapshot
The package retains a 50-prompt CUDA snapshot for the two Inflect releases. It is useful for regression checking but is not a cross-system leaderboard because the archived report does not identify the GPU model.
| Release | Prompts | RTF ↓ | Audio / wall time ↑ | Median latency | p95 latency |
|---|---|---|---|---|---|
| Inflect-Micro-v2 | 50 | 0.0174 | 57.49× | 78.6 ms | 82.7 ms |
| Inflect-Nano-v2 | 50 | 0.0171 | 58.41× | 80.2 ms | 85.3 ms |
Complete model footprint
“Complete” means the downloadable weight file required for text-to-waveform generation, including Inflect's integrated waveform decoder. Voice variants that share identical weights are counted once in family-level footprint figures. Package size is not the same as peak runtime memory.
Frozen artifacts and reproducibility
| Artifact | Purpose |
|---|---|
evaluation/final/modern400/prompts.jsonl |
Frozen Modern400 prompt manifest |
evaluation/final/modern400/summary.json |
Three-ASR summaries, intervals, and category results |
evaluation/final/modern400/exclusion_report.json |
Exact-text training exclusion audit |
evaluation/final/modern400/ |
Compressed row-level hypotheses and scoring reports |
evaluation/final/utmos22/report.json |
UTMOS22 clip scores and distributions |
evaluation/final/runtime_snapshot/ |
Retained engineering snapshots |
release_manifest.json |
File sizes and SHA-256 integrity hashes |
- Modern400 prompt SHA-256:
b7504ce2dce44a2da82770a6a5dfd2a034fe17e2113980f8a69663ade417a34c - UTMOS22 prompt SHA-256:
5b7aa33852e6c738eb0e1dbf9fcfcc90772b97ad219167e44aaf3917d063b056 - Modern400 bootstrap samples:
10,000 - UTMOS22 bootstrap samples:
10,000
Interpretation limits
- ASR WER measures recognizer agreement with intended text, not naturalness.
- UTMOS22 predicts perceived quality; it is not a substitute for human MOS.
- Community preference is valuable human evidence but is not a controlled lab study.
- One fixed voice per release limits conclusions about speaker diversity.
- Runtime depends on hardware, framework, provider, compilation, thread policy, and host load.
- Results apply to these frozen checkpoints, frontends, prompts, and inference settings.
- No benchmark result should be presented as proof that one model is best for every deployment.