# Model card — `dscnn_fan` int8 (voice-fan-controller) The deployed model: 18-class spoken-command recognition (16 fan commands + `_silence_` + `_unknown_`) running on a plain ESP32-WROOM-32. **28,584 bytes**, int8, no accelerator. Input is a (81, 32) log-mel plane = 2,592 floats; output is an 18-way softmax. The flashed image does **not** run TensorFlow Lite Micro. It runs a generated static-C engine (`dscnn_micro`) that reproduces the same graph — verified against the TFLite interpreter at **0 prediction flips over 200 clips**, pre-softmax logits within 1 LSB. A TFLM build still exists behind the same C API and is selectable at menuconfig, but it is not what ships. The hard requirement is **speaker independence** — it has to work for someone who never recorded a clip. Everything below is organised around how well it does that, and around not quoting numbers that measure something easier. > The retired `func_mlp` model card is at `archive/MODEL_CARD_func_mlp.md` in > this model repo (`docs/archive/MODEL_CARD.md` in the code repo). Its accuracy > numbers do not apply to this model — they are in-speaker. ## The number that matters **Leave-one-speaker-out (LOSO)** — score each speaker with a model that never heard them. It is the only honest metric here, and it belongs to a **recipe**, not to a weights file: the shipping model trains on everybody, so it has no held-out voice of its own left to be measured against. The **deployed** model's own recipe (`c48_rio15` — all voices, two-stage transfer with BN frozen, `domain_aug=0.5`, encoder widened to 48 channels; measured 2026-07-28) reaches **91.69% headline**, 5 seeds per fold: | anirudh | austin | francis | mishelle | headline | |---|---|---|---|---| | 91.34% | 93.86% | 95.55% | **86.00%** | **91.69%** | That is **+3.25pp** over the last `func_mlp`-era shipping recipe (88.44%), against 1 SE of 1.81pp — clear of the pre-registered accept bar (≥2pp *and* ≥1 SE). Against the *immediately* preceding recipe (`c48_deployable`, 91.37%) it is only **+0.32pp** — inside noise. The two differ by corpus, not by code: three more speakers' worth of recordings (+710 clips). It is reported as a corpus refresh, not as a lever that cleared a bar. Eleven further complete speakers are scored outside the headline. Averaged over all **fifteen** full-coverage speakers — the widest "what a stranger gets" view — it reads **90.05%** (SE 2.27pp), and its **worst voice is 66.85%** (aak). Read the worst, not the mean: this refresh raised the average while *lowering* the hardest voice by 1.98pp, which is precisely the failure mode the worst-voice rule exists to catch. The full spread, weakest first: ``` aak 66.9 · savio 74.0 · mishelle 86.0 · rio 86.5 · allen 90.5 anirudh 91.3 · adithyasharma 92.9 · alwin 94.1 · austin 93.9 · anon1 93.3 anon2 95.0 · danushkrishna 95.4 · francis 95.6 · adithya 97.0 · teresa 98.4 ``` **The width helped studio voices and hurt browser ones.** Widening the encoder raised every studio-corpus fold (+1.53 to +5.64pp) but *lowered* all three small browser-recorded folds (adithyasharma −5.93, anon1 −2.25, anon2 −6.22). That is roughly 14pp of fold-sum, and it is the whole reason this model lands just under 90%. A headline win does not automatically transfer to strangers on a laptop mic. A matched control run separates the two things that changed: the extra recorded data is worth **+0.88pp** on its own (a reject as a lever), and capacity alone is **+2.04pp**. The width is the real lever, and it clears the 2pp bar narrowly — not by a comfortable margin. (The fifteen-speaker average is not comparable to older fourteen- or seven-speaker figures — the corpus grew, so the same statistic now covers a different population. Only the frozen headline four are comparable across runs.) **In-speaker validation for the deployed model is 97.47% and is not a generalization estimate.** Every held-out clip there belongs to a speaker who is in training — by design, since nobody is held out of the shipping model. The gap between the two kinds of number is the entire point of this project: an earlier architecture scored ~95% in-speaker and **43.7%** LOSO. If you quote one number, quote a LOSO one. ## Limitations, in order of how likely they are to bite 1. **Pronunciation, not channel, is what sinks the worst voice.** aak sits at 66.85% — twenty-three points below the mean, and moving the wrong way across the last two recipes — and the errors are *within-word* confusions, not noise. A channel probe put that speaker at z=0.98 (unremarkable recording conditions), and cepstral mean normalisation made things worse by 2.8pp, so the gap is genuinely acoustic: those pronunciations land near the wrong class in feature space. More dense recordings of the weak voices is the only lever with evidence behind it; roughly eight code-side levers have been tried and spent. 2. **Vocal register is the second axis.** Nearly every training speaker sits at 107–149 Hz median F0; one sits at 255.7 Hz, roughly an octave up. That speaker is still the weakest headline fold (86.00%) and is the only one register augmentation helps. Expect degradation for voices outside the trained range, which includes most children. The headline averages three men and one woman, so it *understates* this failure mode. 3. **English only, one accent pool, 18 speakers.** The encoder is pretrained on 9,286 outside speakers, but fine-tuning corpus diversity is small — and four of the scored fifteen contributed only ~16–20 clips each, so their coverage is nominal rather than substantial. Those tiny folds are honest (genuinely held out) but noisy: one misheard clip moves such a fold by ~6pp. 4. **`timer_*` is the weakest command group.** Public pretraining corpora contain ten instances of the word "timer" in total, so transfer cannot teach it — it is learned only from the recorded set. 5. **Siblings dominate the errors, and the gate cannot catch them.** 66.7% of all command errors confuse two parameters of the *same* command (`power_on`↔`power_off`, `speed_2`↔`speed_3`, `timer_2h`↔`timer_4h`) — errors that produce a plausible-looking wrong action rather than an obvious one. This was priced on silicon (see *Downstream gate*): the wrong answers are often delivered with **high** confidence, so no threshold separates them from correct ones. 6. **`_unknown_` has zero human speakers** — it is sourced from YouTube audio. Rejection of *speech* that is not a command is therefore weaker than the negative-class accuracy suggests. ## Feature contract — not optional ``` sample_rate 20 kHz · window 25.6 ms (n_fft 512) · stride 20 ms mel_bins 32 · log-mel via power_to_db(ref=max, top_db=80) · input (81, 32, 1) ``` `ref=max` is a correctness requirement, not a tuning choice. The alternative (`log(mel + 1e-6)`) has no amplitude normalisation, so raw int16 PCM on the device shifts every feature by ~6σ of pure recording level — frozen BatchNorm cannot absorb it and the model runs while predicting nonsense. `window 25.6 ms` gives `n_fft = 512` so the device can use a radix-2 FFT. The C frontend is measured amplitude-invariant (0.25× gain moves features by 0.0016 dB) and agrees with the training frontend to **<0.002 dB**. ## Training DS-CNN (depthwise-separable conv stack, **672,672 MACs, 12,082 params**, 48 channels). The encoder is pretrained on Speech Commands v2 + MSWC — 112,049 clips from 9,286 speakers, 28.1% female — then fine-tuned on **5,542 recorded clips from 18 speakers**. The 48-channel width is a re-pretrain, not a knob: the encoder weights are shaped by `channels`, so widening it invalidates the previous encoder and forces a full pretraining run before any comparison is fair. The previous 32-channel model was 6,706 params / 441k MACs / 20,632 bytes. **Transfer is what makes this work: from-scratch on the same data scores 45.9%, against 67.4% pretrained.** That +21.5pp is the single largest gain on record here, and nothing else has come within 20pp of it. (That pair was measured with an *earlier* encoder — `dscnn_fan_scv2`, 11,376 speakers — not the 9,286-speaker `scv2_max` above. It is quoted as evidence that pretraining is what carries this model, which is encoder-independent; it is not a measurement of the encoder shipping today.) The fine-tune is two-stage: 15 epochs with the encoder frozen so the randomly initialised 18-class head cannot wash out the transferred filters, then unfreeze at `lr 3e-4` with **BatchNorm statistics frozen** — those were estimated over 9,286 speakers and re-estimating them from a handful of fold voices reintroduces the per-channel speaker signature the pretraining was for. Two train-time-only augmentations, neither of which costs anything on device: - **register augmentation** (Praat "Change gender", moving F0 and formants independently) — covers the register axis the corpus barely spans; - **domain augmentation** (reverb + bandlimit + µ-law codec round-trip) — added for the deployed model to close the browser-microphone gap. **It has still never been isolated in a controlled run**, so it remains a bet, and it is labelled as one. What *is* known is that the gap is not closed: the three browser-recorded folds were the only ones the wider encoder made worse. Training is seeded but **not bit-reproducible** — oneDNN varies float reduction order across threads, moving single folds by up to 5.1pp between identical runs. Compare 5-seed means, not folds. ## On device Measured on a real ESP32-D0WD-V3 from the **2026-07-27 flashed image**. That image carries the *previous* weights (`c48_deployable`); the model described by this card is built but **not yet flashed**. Every number in this section is a property of the graph, not of the weights — identical shapes, ops, parameter count and MACs — so it carries over unchanged. The accuracy numbers above do not: those belong to `c48_rio15`, and the board has not yet been asked to reproduce them. | | | previous (c32, TFLM, single-core) | |---|---|---| | frontend | **18.6 ms** | 45.3 ms | | invoke | **34.2 ms** | 49.75 ms | | total | **~53 ms** (after a 1,600 ms capture window) | ~95 ms | | tensor arena | **13,184 B** (static engine) | 17,324 B used / 20,480 reserved | | flash | **28,584 B** | 20,632 B | **Do not credit that speedup to the model.** This image is ~1.8× faster while computing **+53% more MACs**, because two other things changed at the same time: the static-C engine replaced TFLM, and both the STFT and the conv stack now use the second core. The wider model taken alone would have been *slower*. Two variables moved; the comparison is reported, not attributed. The arena is unchanged at 13,184 B despite the wider layers, which is arithmetic rather than luck: the peak simultaneously-live tensor pair is the input features plus the stem output, and `channels` does not touch `stem_channels=16`. Ops: CONV_2D, DEPTHWISE_CONV_2D, MAX_POOL_2D, RESHAPE, FULLY_CONNECTED, SOFTMAX. `MAX_POOL_2D` deliberately — `GlobalMaxPooling2D` lowers to `REDUCE_MAX`, which has no ESP-NN kernel and fails `AllocateTensors()` at boot. ### Validated on silicon against a true stranger This image was tested over UART with **180 clips from a speaker who is not in the corpus** — recorded after the model was built, so it had genuinely never heard him, and none of his audio influenced training: | | top-1 (argmax) | actually fires (after the gate) | |---|---|---| | all 180 clips | 83.3% | 76.7% | | excluding `speed_5` | **92.9%** | **89.0%** | | `speed_5` alone | 24.0% | 0.0% | Seven of his eight recorded classes land between 84% and 100%. One collapses: 16 of 25 `speed_5` clips are heard as `speed_3`. That is not a mislabelled recording — his `speed_5` runs 0.650 ± 0.044 s against `speed_3`'s 0.787 ± 0.135 s, a consistent and distinct utterance the model maps onto the wrong class. **Read the whole distribution, not the headline.** A single class failing this way is invisible in an averaged score and obvious to the person using the fan. (An earlier image was validated the same way against a different unseen speaker at 96.5% over 400 clips. She is an easy voice — her own held-out fold independently scores ~96% — so that number is evidence the deployment path is correct, not a generalization estimate.) An integer (Q15) build of the frontend also exists, verified accuracy-neutral (0/200 model predictions changed), which removes the FPU requirement for cheaper targets. It is staged, not flashed. ## Downstream gate The classifier output is not the device's decision. A confidence gate (`command_postprocessor`) suppresses low-confidence and ambiguous results before any UART frame is emitted. The shipped point is **`conf 0.60`**, and it is a real trade: at 0.60, 23.1 of every 100 commands from a new speaker are silently ignored against 4.2 wrong actions; at 0.40 that is 11.3 against 8.2. On held-out speakers saying *non-command* words, false-accept is 22.0% at 0.60 against 50.5% at 0.40. Pick the point on training speakers, never on the held-out set. **The gate cannot fix a misclassification, and this is now measured rather than argued.** All 12 of the stranger's wrongly-fired clips above were sibling confusions, with top-1-minus-top-2 margins of 0.348, 0.528, 0.574, 0.586, 0.598, 0.657, 0.711, 0.734, 0.742, 0.770, 0.773 and **0.949**. Tightening the sibling margin to 0.35 would catch 1 of the 12; catching half needs ~0.65; catching all needs >0.95, which would also reject legitimate commands. One clip was predicted `speed_3` at 0.969 confidence. A gate sees only the output distribution, and *wrong-and-certain* is indistinguishable from *right-and-certain*. Thresholds trade coverage against precision; they cannot recover information the model never had. One consequence worth stating plainly: because `CP_MIN_MARGIN` (0.15) is below `2 × 0.60 − 1 = 0.20`, and a softmax sums to 1, the global margin branch **cannot fire at this operating point**. It is inert by algebra, not by intent, and no compiler warns about a branch no input can reach. ## Intended use A hobby/research voice→UART bridge for one fan model. Not suitable for safety-, security- or accessibility-critical control, where a 5–20% miss rate on an unfamiliar voice is not acceptable.