Refresh benchmark evidence notes

README.md

@@ -52,7 +52,7 @@ The Latency panel includes validation cards for the two external evidence gates:
 | Role | Default | Why |
 | --- | --- | --- |
 | VAD | `onnx-community/silero-vad` | Small ONNX model used in the Transformers.js Moonshine web demo; fast enough to run on 512-sample chunks and drive turn-taking. The app defaults to 480 ms of trailing silence, configurable from 200-800 ms. The fp32 ONNX file is about 2.2 MB. |
-| STT | `onnx-community/moonshine-base-ONNX` | Moonshine Base is the default balanced STT after the current fake-mic series reached 3/3 exact rows, 0% median WER, and 8.63 s median speech-end-to-audio. Moonshine Tiny remains selectable as the low-latency experiment, but it failed the current exact fake-mic gate by repeatedly hearing variants like "What happens this?" and timing out. Whisper Tiny English is selectable as the higher-accuracy fallback; it reached 3/3 and 0% WER in fake-mic validation, but its 4.54 s median ASR raised speech-end-to-audio to 10.94 s. The demo uses fp32 encoder + q4 merged decoder because the q8 WASM path failed on Transformers.js 4.2.0 in local verification. |
+| STT | `onnx-community/moonshine-base-ONNX` | Moonshine Base is the default balanced STT after the current fake-mic series reached 3/3 exact rows, 0% median WER, 1.85 s median ASR, and 7.62 s median speech-end-to-audio. Moonshine Tiny remains selectable as the low-latency experiment, but it failed the current exact fake-mic gate by repeatedly hearing variants like "What happens this?" and timing out. Whisper Tiny English is selectable as the higher-accuracy fallback; it reached 3/3 and 0% WER in fake-mic validation, but its 4.54 s median ASR raised speech-end-to-audio to 10.94 s. The demo uses fp32 encoder + q4 merged decoder because the q8 WASM path failed on Transformers.js 4.2.0 in local verification. |
 | LLM | `HuggingFaceTB/SmolLM2-135M-Instruct` | The fastest instruct model found for this stack. It is tagged for Transformers.js and includes ONNX q4/q4f16 files; q4 WASM is about 182 MB and q4f16 WebGPU is about 118 MB. In headless WASM it cut first-token latency by roughly 2-3x versus SmolLM2 360M. A tiny pinned identity example keeps the default 135M stack passing the LLM OK gate without switching to the much slower 360M. |
 | TTS | `onnx-community/Supertonic-TTS-ONNX` | Transformers.js packaging of `Supertone/supertonic`, using local ONNX inference and Supertonic voice embeddings. The demo defaults to voice F2 and 2 inference steps because F2 is still faster in current full-stack fake-mic and default-suite runs; M2 was slightly faster in isolated TTS but slower in loopback and fake-mic full-stack validation, so it remains a candidate for real-mic/WebGPU validation rather than the default. The selected voice is loaded before the app reports ready, and the other voices are preloaded in the background. The repo is about 263 MB. |
 
@@ -163,22 +163,22 @@ Measured on this workspace with Chrome 145 headless, x86_64, 4 vCPU, Transformer
 | One-load TTS sweep | N/A | N/A | Supertonic M2, 8 steps | N/A | N/A | N/A | N/A | N/A | 0 ms | 4.27 s | 4.28 s | N/A | N/A |
 | One-load TTS sweep | N/A | N/A | Supertonic F1, 8 steps | N/A | N/A | N/A | N/A | N/A | 0 ms | 4.50 s | 4.50 s | N/A | N/A |
 | One-load TTS sweep | N/A | N/A | Supertonic F2, 8 steps | N/A | N/A | N/A | N/A | N/A | 0 ms | 5.00 s | 5.00 s | N/A | N/A |
-| Current default TTS suite row | N/A | N/A | Supertonic F2, 2 steps | N/A | N/A | N/A | N/A | N/A | 1 ms | 1.77 s | 1.77 s | N/A | N/A |
+| Current default TTS suite row | N/A | N/A | Supertonic F2, 2 steps | N/A | N/A | N/A | N/A | N/A | 1 ms | 1.56 s | 1.56 s | N/A | N/A |
 | Current default chat suite row | N/A | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | N/A | N/A | N/A | N/A | 2.79 s | 3.35 s | 724 ms | 4.08 s | N/A | 1.6 tok/s |
 | Current default identity suite row | N/A | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | N/A | N/A | pass 4/4 | N/A | 4.37 s | 4.37 s | 643 ms | 5.02 s | N/A | 1.3 tok/s |
 | Current default barge-in suite row | N/A | N/A | Supertonic F2, 2 steps | N/A | N/A | N/A | N/A | N/A | 0 ms | cancelled | cancelled | N/A | N/A |
 | Text benchmark | N/A | SmolLM2 360M q4 WASM | 2 | N/A | N/A | fail 3/4 | N/A | 9.56 s | 13.43 s | 1.31 s | 14.75 s | N/A | 0.5 tok/s |
 | Current default voice loopback suite row | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | 1.80 s | 0% | pass 4/4 | 480 ms | 4.14 s | 4.14 s | 545 ms | 4.69 s | 6.49 s | 1.5 tok/s |
-| 3-run loopback stability series | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | 2.01 s | 0% median, 3/3 exact | pass 3/3 | 480 ms | 4.86 s | 5.60 s | 600 ms | 5.60 s | 7.84 s | 1.4 tok/s |
+| 3-run loopback stability series | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | 1.91 s | 0% median, 2/3 exact | pass 2/3 | 480 ms | 4.96 s | 4.97 s | 599 ms | 5.56 s | 7.56 s | 1.4 tok/s |
 | M2 loopback validation | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic M2, 2 steps | 2.33 s | 0% | pass 4/4 | 480 ms | 6.64 s | 8.48 s | 1.74 s | 10.23 s | 12.56 s | 0.9 tok/s |
-| Fake microphone scripted series | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | 1.64 s | 0% | pass 4/4 | 480 ms | 5.06 s | 6.22 s | 745 ms | 6.99 s | 8.63 s | 1.4 tok/s |
+| Fake microphone scripted series | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | 1.85 s | 0% | pass 3/3 rows | 480 ms | 5.21 s | 5.21 s | 610 ms | 5.77 s | 7.62 s | 1.4 tok/s |
 | Whisper Tiny fake microphone validation | Whisper Tiny English fp32/q4 | SmolLM2 135M q4 WASM | Supertonic F2, 2 steps | 4.54 s | 0% | pass 4/4 | 480 ms | 4.70 s | 5.80 s | 603 ms | 6.40 s | 10.94 s | 1.5 tok/s |
 | M2 fake microphone validation | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic M2, 2 steps | 2.37 s | 0% | pass 4/4 | 480 ms | 6.64 s | 7.84 s | 1.43 s | 9.27 s | 11.64 s | 1.1 tok/s |
 | Previous voice loopback | Moonshine Base fp32/q4 | SmolLM2 135M q4 WASM | Supertonic F1, 2 steps | 1.65 s | 25% | pass 4/4 | 480 ms | 3.74 s | 4.81 s | 773 ms | 5.58 s | 7.24 s | 1.6 tok/s |
 | Voice loopback | Moonshine Tiny fp32/q4 | SmolLM2 135M q4 WASM | 2 | 1.08 s | 25% | pass 4/4 | 480 ms | 3.72 s | 4.84 s | 951 ms | 5.79 s | 6.88 s | 1.4 tok/s |
 | Voice loopback | Whisper Tiny English fp32/q4 | SmolLM2 135M q4 WASM | 2 | 5.62 s | 0% | pass 4/4 | 480 ms | 4.63 s | 6.63 s | 902 ms | 7.53 s | 13.16 s | 1.3 tok/s |
 
-The first-clause TTS threshold reduced the earlier SmolLM2 360M text benchmark's first-audio time from 45.84 s to 12.25 s in headless WASM by moving TTS queueing from 44.43 s after transcript to 11.01 s. Switching the default LLM to SmolLM2 135M, splitting generic and identity prompts, and using a word-boundary-safe first chunk reduced normal chat first audio to 4.34 s in the earlier default suite. The first response chunk now targets roughly 5 characters, keeps a minimum safe space boundary, and searches forward to the next word boundary before falling back to a hard character cut; this fixed the rejected early failure that produced "I'm read", keeps the current chat first chunk as "I'm ready", and lets identity turns start on the safe first-word chunk "This". In a repeated chunking comparison, the balanced 5-character profile kept chat word-complete and reduced identity first-audio median from 5.15 s to 4.51 s versus the previous 8-character profile; the refreshed source-fingerprinted client-side smoke measured 5.87 s identity first audio with the same first chunk and the expanded identity primer. Identity turns use a stricter system prompt plus tiny pinned examples, so they are slower than generic chat but pass the 4/4 identity gate. A current-code 360M text run improved the identity answer from 2/4 to 3/4 concepts, but first audio was still 14.75 s and the answer omitted the LLM component, so 135M remains the default latency/quality choice. Qwen3 0.6B terminated the local headless browser during WASM model load before it reached benchmark-ready state, so the app now blocks Qwen3 and SmolLM2 1.7B on WASM fallback and requires WebGPU for those candidates. The one-load TTS sweep confirms 2-step Supertonic is the right default for latency; 8 steps roughly doubled first-audio time, and M2 at 2 steps was the fastest isolated voice/step setting at 2.00 s versus F2 at 2.10 s. The app still defaults to F2 because the latest F2 fake-mic series reached 8.63 s median speech-end-to-audio, while M2's repeated fake-mic validation reached 0% WER but slower 11.64 s speech-end-to-audio and 23.42 s speech-end-to-audio-done; M2 also measured slower in loopback, so real-mic and WebGPU validation are still needed before reconsidering the default. The TTS worker now loads the selected voice before reporting ready, then preloads the remaining voices in the background; the latest hosted client-side smoke measured a 1.77 s TTS first-audio row with all benchmark-phase network requests already settled. The current default benchmark suite was verified with actual workers and produced TTS, barge-in, identity, chat, and loopback rows before re-enabling controls. The barge-in check verified that a synthetic speech-start event cancels in-flight TTS before stale audio plays and returns the TTS tile to Ready. After repeated loopback checks, the synthetic loopback stability gate now uses the prompt "Identify this browser demo.", 1.00x prompt speed, and a short silence preroll; the latest targeted run completed 3/3 rows, reached 3/3 exact transcripts, passed the identity answer gate on all 3 rows, and reached 7.84 s median speech-end-to-audio. The previous "Please identify this browser demo." prompt passed the identity gate but intermittently dropped the low-value opening word, pushing WER to 20%; the shorter "What app is this?", "What application is this?", "What is this app?", and "What demo is this?" synthetic prompts also intermittently dropped opening words or confused "application" with "cation/location", so they remain stress options rather than the default loopback gate. A partial-ASR-off loopback comparison was slightly faster at 5.94 s median speech-end-to-audio, but it dropped to 2/3 exact transcripts with "Wap is this.", so partial previews remain enabled by default. The loopback gate is still treated as a synthetic stability signal, not a substitute for real microphone validation. Normal response TTS remains at 1.08x. The fake microphone scripted series reached 3/3 completed rows with 0% median WER, 1.64 s median ASR, 8.63 s median speech-end-to-audio, and 17.72 s median speech-end-to-audio-done through the browser capture path; because Chrome loops fake WAV input, the harness ignores non-matching partial prompt captures until the reference phrase is heard, then stops the mic so the response can finish. A stale-ASR guard now ignores queued speech events after scripted fake-mic capture stops; the M2 regression run completed 3/3 rows after previously timing out from those stale interruptions. The loopback benchmark now feeds synthesized audio at real-time chunk intervals, uses a 480 ms VAD close delay after a 280 ms default split the prompt into separate words, and posts an explicit ASR flush after trailing silence so sticky candidates finish. Moonshine Base remains the default STT because it is much faster than Whisper Tiny English while less error-prone than Tiny: Whisper Tiny English also reached 0% WER in fake mic, but raised median ASR to 4.54 s and speech-end-to-audio to 10.94 s, while Moonshine Tiny failed the exact fake-mic gate before completing one row. The scripted mic benchmark still uses "What app is this?" as its reference phrase so real-mic rows can report WER/CER against the short spoken phrase, and the 3-run mic series keeps the same microphone open while collecting the repeated rows. The latest full-stack model loads completed in about 15-35 seconds cold in this environment, with a same-profile fake-mic reload at about 10.5 seconds. A real WebGPU browser should be benchmarked next; this headless environment exposes only the SwiftShader software adapter.
+The first-clause TTS threshold reduced the earlier SmolLM2 360M text benchmark's first-audio time from 45.84 s to 12.25 s in headless WASM by moving TTS queueing from 44.43 s after transcript to 11.01 s. Switching the default LLM to SmolLM2 135M, splitting generic and identity prompts, and using a word-boundary-safe first chunk reduced normal chat first audio to 4.34 s in the earlier default suite. The first response chunk now targets roughly 5 characters, keeps a minimum safe space boundary, and searches forward to the next word boundary before falling back to a hard character cut; this fixed the rejected early failure that produced "I'm read", keeps the current chat first chunk as "I'm ready", and lets identity turns start on the safe first-word chunk "This". Identity turns use a stricter system prompt plus tiny pinned examples, and repeated identity prompts now ignore prior identity history, keeping the fake-mic prompt at 187 input tokens across all three scripted rows instead of growing turn by turn. The latest fake-mic series measured 5.21 s median transcript-to-first-token and 7.62 s median speech-end-to-first-audio. A current-code 360M text run improved the identity answer from 2/4 to 3/4 concepts, but first audio was still 14.75 s and the answer omitted the LLM component, so 135M remains the default latency/quality choice. Qwen3 0.6B terminated the local headless browser during WASM model load before it reached benchmark-ready state, so the app now blocks Qwen3 and SmolLM2 1.7B on WASM fallback and requires WebGPU for those candidates. The one-load TTS sweep confirms 2-step Supertonic is the right default for latency; 8 steps roughly doubled first-audio time, and M2 at 2 steps was the fastest isolated voice/step setting at 2.00 s versus F2 at 2.10 s. The app still defaults to F2 because the latest F2 fake-mic series reached 7.62 s median speech-end-to-audio, while M2's repeated fake-mic validation reached 0% WER but slower 11.64 s speech-end-to-audio and 23.42 s speech-end-to-audio-done; M2 also measured slower in loopback, so real-mic and WebGPU validation are still needed before reconsidering the default. The TTS worker now loads the selected voice before reporting ready, then preloads the remaining voices in the background; the latest hosted client-side smoke measured a 1.65 s TTS first-audio row with all benchmark-phase network requests already settled. The current default benchmark suite was verified with actual workers and produced TTS, barge-in, identity, chat, and loopback rows before re-enabling controls. The barge-in check verified that a synthetic speech-start event cancels in-flight TTS before stale audio plays and returns the TTS tile to Ready. After repeated loopback checks, the synthetic loopback stability gate now uses the prompt "Identify this browser demo.", 1.00x prompt speed, and a short silence preroll; the latest targeted run completed 3/3 rows, reached 2/3 exact transcripts with 0% median WER, passed the identity answer gate on 2/3 rows, and reached 7.56 s median speech-end-to-audio. The previous "Please identify this browser demo." prompt passed the identity gate but intermittently dropped the low-value opening word, pushing WER to 20%; the shorter "What app is this?", "What application is this?", "What is this app?", and "What demo is this?" synthetic prompts also intermittently dropped opening words or confused "application" with "cation/location", so they remain stress options rather than the default loopback gate. A partial-ASR-off loopback comparison was slightly faster at 5.94 s median speech-end-to-audio, but it dropped to 2/3 exact transcripts with "Wap is this.", so partial previews remain enabled by default. The loopback gate is still treated as a synthetic stability signal, not a substitute for real microphone validation. Normal response TTS remains at 1.08x. The fake microphone scripted series reached 3/3 completed rows with 0% median WER, 1.85 s median ASR, 7.62 s median speech-end-to-audio, and 19.39 s median speech-end-to-audio-done through the browser capture path; because Chrome loops fake WAV input, the harness ignores non-matching partial prompt captures until the reference phrase is heard, then stops the mic so the response can finish. A stale-ASR guard now ignores queued speech events after scripted fake-mic capture stops; the M2 regression run completed 3/3 rows after previously timing out from those stale interruptions. The loopback benchmark now feeds synthesized audio at real-time chunk intervals, uses a 480 ms VAD close delay after a 280 ms default split the prompt into separate words, and posts an explicit ASR flush after trailing silence so sticky candidates finish. Moonshine Base remains the default STT because it is much faster than Whisper Tiny English while less error-prone than Tiny: Whisper Tiny English also reached 0% WER in fake mic, but raised median ASR to 4.54 s and speech-end-to-audio to 10.94 s, while Moonshine Tiny failed the exact fake-mic gate before completing one row. The scripted mic benchmark still uses "What app is this?" as its reference phrase so real-mic rows can report WER/CER against the short spoken phrase, and the 3-run mic series keeps the same microphone open while collecting the repeated rows. The latest full-stack model loads completed in about 15-35 seconds cold in this environment, with a same-profile fake-mic reload at about 10.5 seconds. A real WebGPU browser should be benchmarked next; this headless environment exposes only the SwiftShader software adapter.
 
 ## Known Limitations