| # Model card — `dscnn_fan` int8 (voice-fan-controller) | |
| The deployed model: 18-class spoken-command recognition (16 fan commands + | |
| `_silence_` + `_unknown_`) running on a plain ESP32-WROOM-32. **28,584 bytes**, | |
| int8, no accelerator. Input is a (81, 32) log-mel plane = 2,592 floats; output is | |
| an 18-way softmax. | |
| The flashed image does **not** run TensorFlow Lite Micro. It runs a generated | |
| static-C engine (`dscnn_micro`) that reproduces the same graph — verified against | |
| the TFLite interpreter at **0 prediction flips over 200 clips**, pre-softmax | |
| logits within 1 LSB. A TFLM build still exists behind the same C API and is | |
| selectable at menuconfig, but it is not what ships. | |
| The hard requirement is **speaker independence** — it has to work for someone who | |
| never recorded a clip. Everything below is organised around how well it does | |
| that, and around not quoting numbers that measure something easier. | |
| > The retired `func_mlp` model card is at `archive/MODEL_CARD_func_mlp.md` in | |
| > this model repo (`docs/archive/MODEL_CARD.md` in the code repo). Its accuracy | |
| > numbers do not apply to this model — they are in-speaker. | |
| ## The number that matters | |
| **Leave-one-speaker-out (LOSO)** — score each speaker with a model that never | |
| heard them. It is the only honest metric here, and it belongs to a **recipe**, | |
| not to a weights file: the shipping model trains on everybody, so it has no | |
| held-out voice of its own left to be measured against. | |
| The **deployed** model's own recipe (`c48_rio15` — all voices, two-stage | |
| transfer with BN frozen, `domain_aug=0.5`, encoder widened to 48 channels; | |
| measured 2026-07-28) reaches **91.69% headline**, 5 seeds per fold: | |
| | anirudh | austin | francis | mishelle | headline | | |
| |---|---|---|---|---| | |
| | 91.34% | 93.86% | 95.55% | **86.00%** | **91.69%** | | |
| That is **+3.25pp** over the last `func_mlp`-era shipping recipe (88.44%), | |
| against 1 SE of 1.81pp — clear of the pre-registered accept bar (≥2pp *and* | |
| ≥1 SE). | |
| Against the *immediately* preceding recipe (`c48_deployable`, 91.37%) it is only | |
| **+0.32pp** — inside noise. The two differ by corpus, not by code: three more | |
| speakers' worth of recordings (+710 clips). It is reported as a corpus refresh, | |
| not as a lever that cleared a bar. | |
| Eleven further complete speakers are scored outside the headline. Averaged over | |
| all **fifteen** full-coverage speakers — the widest "what a stranger gets" view — | |
| it reads **90.05%** (SE 2.27pp), and its **worst voice is 66.85%** (aak). Read | |
| the worst, not the mean: this refresh raised the average while *lowering* the | |
| hardest voice by 1.98pp, which is precisely the failure mode the worst-voice | |
| rule exists to catch. The full spread, weakest first: | |
| ``` | |
| aak 66.9 · savio 74.0 · mishelle 86.0 · rio 86.5 · allen 90.5 | |
| anirudh 91.3 · adithyasharma 92.9 · alwin 94.1 · austin 93.9 · anon1 93.3 | |
| anon2 95.0 · danushkrishna 95.4 · francis 95.6 · adithya 97.0 · teresa 98.4 | |
| ``` | |
| **The width helped studio voices and hurt browser ones.** Widening the encoder | |
| raised every studio-corpus fold (+1.53 to +5.64pp) but *lowered* all three small | |
| browser-recorded folds (adithyasharma −5.93, anon1 −2.25, anon2 −6.22). That is | |
| roughly 14pp of fold-sum, and it is the whole reason this model lands just under | |
| 90%. A headline win does not automatically transfer to strangers on a laptop mic. | |
| A matched control run separates the two things that changed: the extra recorded | |
| data is worth **+0.88pp** on its own (a reject as a lever), and capacity alone is | |
| **+2.04pp**. The width is the real lever, and it clears the 2pp bar narrowly — | |
| not by a comfortable margin. | |
| (The fifteen-speaker average is not comparable to older fourteen- or seven-speaker | |
| figures — the corpus grew, so the same statistic now covers a different | |
| population. Only the frozen headline four are comparable across runs.) | |
| **In-speaker validation for the deployed model is 97.47% and is not a | |
| generalization estimate.** Every held-out clip there belongs to a speaker who is | |
| in training — by design, since nobody is held out of the shipping model. The gap | |
| between the two kinds of number is the entire point of this project: an earlier | |
| architecture scored ~95% in-speaker and **43.7%** LOSO. If you quote one number, | |
| quote a LOSO one. | |
| ## Limitations, in order of how likely they are to bite | |
| 1. **Pronunciation, not channel, is what sinks the worst voice.** aak sits at | |
| 66.85% — twenty-three points below the mean, and moving the wrong way across the | |
| last two recipes — and the errors are *within-word* | |
| confusions, not noise. A channel probe put that speaker at z=0.98 (unremarkable | |
| recording conditions), and cepstral mean normalisation made things worse by | |
| 2.8pp, so the gap is genuinely acoustic: those pronunciations land near the | |
| wrong class in feature space. More dense recordings of the weak voices is the | |
| only lever with evidence behind it; roughly eight code-side levers have been | |
| tried and spent. | |
| 2. **Vocal register is the second axis.** Nearly every training speaker sits at | |
| 107–149 Hz median F0; one sits at 255.7 Hz, roughly an octave up. That speaker | |
| is still the weakest headline fold (86.00%) and is the only one register | |
| augmentation helps. Expect degradation for voices outside the trained range, | |
| which includes most children. The headline averages three men and one woman, | |
| so it *understates* this failure mode. | |
| 3. **English only, one accent pool, 18 speakers.** The encoder is pretrained on | |
| 9,286 outside speakers, but fine-tuning corpus diversity is small — and four | |
| of the scored fifteen contributed only ~16–20 clips each, so their coverage is | |
| nominal rather than substantial. Those tiny folds are honest (genuinely held | |
| out) but noisy: one misheard clip moves such a fold by ~6pp. | |
| 4. **`timer_*` is the weakest command group.** Public pretraining corpora | |
| contain ten instances of the word "timer" in total, so transfer cannot teach | |
| it — it is learned only from the recorded set. | |
| 5. **Siblings dominate the errors, and the gate cannot catch them.** 66.7% of all | |
| command errors confuse two parameters of the *same* command | |
| (`power_on`↔`power_off`, `speed_2`↔`speed_3`, `timer_2h`↔`timer_4h`) — errors | |
| that produce a plausible-looking wrong action rather than an obvious one. This | |
| was priced on silicon (see *Downstream gate*): the wrong answers are often | |
| delivered with **high** confidence, so no threshold separates them from correct | |
| ones. | |
| 6. **`_unknown_` has zero human speakers** — it is sourced from YouTube audio. | |
| Rejection of *speech* that is not a command is therefore weaker than the | |
| negative-class accuracy suggests. | |
| ## Feature contract — not optional | |
| ``` | |
| sample_rate 20 kHz · window 25.6 ms (n_fft 512) · stride 20 ms | |
| mel_bins 32 · log-mel via power_to_db(ref=max, top_db=80) · input (81, 32, 1) | |
| ``` | |
| `ref=max` is a correctness requirement, not a tuning choice. The alternative | |
| (`log(mel + 1e-6)`) has no amplitude normalisation, so raw int16 PCM on the | |
| device shifts every feature by ~6σ of pure recording level — frozen BatchNorm | |
| cannot absorb it and the model runs while predicting nonsense. `window 25.6 ms` | |
| gives `n_fft = 512` so the device can use a radix-2 FFT. | |
| The C frontend is measured amplitude-invariant (0.25× gain moves features by | |
| 0.0016 dB) and agrees with the training frontend to **<0.002 dB**. | |
| ## Training | |
| DS-CNN (depthwise-separable conv stack, **672,672 MACs, 12,082 params**, 48 | |
| channels). The encoder is pretrained on Speech Commands v2 + MSWC — 112,049 clips | |
| from 9,286 speakers, 28.1% female — then fine-tuned on **5,542 recorded clips | |
| from 18 speakers**. | |
| The 48-channel width is a re-pretrain, not a knob: the encoder weights are shaped | |
| by `channels`, so widening it invalidates the previous encoder and forces a full | |
| pretraining run before any comparison is fair. The previous 32-channel model was | |
| 6,706 params / 441k MACs / 20,632 bytes. | |
| **Transfer is what makes this work: from-scratch on the same data scores 45.9%, | |
| against 67.4% pretrained.** That +21.5pp is the single largest gain on record | |
| here, and nothing else has come within 20pp of it. (That pair was measured with | |
| an *earlier* encoder — `dscnn_fan_scv2`, 11,376 speakers — not the 9,286-speaker | |
| `scv2_max` above. It is quoted as evidence that pretraining is what carries this | |
| model, which is encoder-independent; it is not a measurement of the encoder | |
| shipping today.) | |
| The fine-tune is two-stage: 15 epochs with the encoder frozen so the randomly | |
| initialised 18-class head cannot wash out the transferred filters, then unfreeze | |
| at `lr 3e-4` with **BatchNorm statistics frozen** — those were estimated over | |
| 9,286 speakers and re-estimating them from a handful of fold voices reintroduces | |
| the per-channel speaker signature the pretraining was for. | |
| Two train-time-only augmentations, neither of which costs anything on device: | |
| - **register augmentation** (Praat "Change gender", moving F0 and formants | |
| independently) — covers the register axis the corpus barely spans; | |
| - **domain augmentation** (reverb + bandlimit + µ-law codec round-trip) — added | |
| for the deployed model to close the browser-microphone gap. **It has still | |
| never been isolated in a controlled run**, so it remains a bet, and it is | |
| labelled as one. What *is* known is that the gap is not closed: the three | |
| browser-recorded folds were the only ones the wider encoder made worse. | |
| Training is seeded but **not bit-reproducible** — oneDNN varies float reduction | |
| order across threads, moving single folds by up to 5.1pp between identical runs. | |
| Compare 5-seed means, not folds. | |
| ## On device | |
| Measured on a real ESP32-D0WD-V3 from the **2026-07-27 flashed image**. That | |
| image carries the *previous* weights (`c48_deployable`); the model described by | |
| this card is built but **not yet flashed**. Every number in this section is a | |
| property of the graph, not of the weights — identical shapes, ops, parameter | |
| count and MACs — so it carries over unchanged. The accuracy numbers above do | |
| not: those belong to `c48_rio15`, and the board has not yet been asked to | |
| reproduce them. | |
| | | | previous (c32, TFLM, single-core) | | |
| |---|---|---| | |
| | frontend | **18.6 ms** | 45.3 ms | | |
| | invoke | **34.2 ms** | 49.75 ms | | |
| | total | **~53 ms** (after a 1,600 ms capture window) | ~95 ms | | |
| | tensor arena | **13,184 B** (static engine) | 17,324 B used / 20,480 reserved | | |
| | flash | **28,584 B** | 20,632 B | | |
| **Do not credit that speedup to the model.** This image is ~1.8× faster while | |
| computing **+53% more MACs**, because two other things changed at the same time: | |
| the static-C engine replaced TFLM, and both the STFT and the conv stack now use | |
| the second core. The wider model taken alone would have been *slower*. Two | |
| variables moved; the comparison is reported, not attributed. | |
| The arena is unchanged at 13,184 B despite the wider layers, which is arithmetic | |
| rather than luck: the peak simultaneously-live tensor pair is the input features | |
| plus the stem output, and `channels` does not touch `stem_channels=16`. | |
| Ops: CONV_2D, DEPTHWISE_CONV_2D, MAX_POOL_2D, RESHAPE, FULLY_CONNECTED, SOFTMAX. | |
| `MAX_POOL_2D` deliberately — `GlobalMaxPooling2D` lowers to `REDUCE_MAX`, which | |
| has no ESP-NN kernel and fails `AllocateTensors()` at boot. | |
| ### Validated on silicon against a true stranger | |
| This image was tested over UART with **180 clips from a speaker who is not in the | |
| corpus** — recorded after the model was built, so it had genuinely never heard | |
| him, and none of his audio influenced training: | |
| | | top-1 (argmax) | actually fires (after the gate) | | |
| |---|---|---| | |
| | all 180 clips | 83.3% | 76.7% | | |
| | excluding `speed_5` | **92.9%** | **89.0%** | | |
| | `speed_5` alone | 24.0% | 0.0% | | |
| Seven of his eight recorded classes land between 84% and 100%. One collapses: | |
| 16 of 25 `speed_5` clips are heard as `speed_3`. That is not a mislabelled | |
| recording — his `speed_5` runs 0.650 ± 0.044 s against `speed_3`'s | |
| 0.787 ± 0.135 s, a consistent and distinct utterance the model maps onto the | |
| wrong class. | |
| **Read the whole distribution, not the headline.** A single class failing this | |
| way is invisible in an averaged score and obvious to the person using the fan. | |
| (An earlier image was validated the same way against a different unseen speaker | |
| at 96.5% over 400 clips. She is an easy voice — her own held-out fold | |
| independently scores ~96% — so that number is evidence the deployment path is | |
| correct, not a generalization estimate.) | |
| An integer (Q15) build of the frontend also exists, verified accuracy-neutral | |
| (0/200 model predictions changed), which removes the FPU requirement for cheaper | |
| targets. It is staged, not flashed. | |
| ## Downstream gate | |
| The classifier output is not the device's decision. A confidence gate | |
| (`command_postprocessor`) suppresses low-confidence and ambiguous results before | |
| any UART frame is emitted. The shipped point is **`conf 0.60`**, and it is a real | |
| trade: at 0.60, 23.1 of every 100 commands from a new speaker are silently ignored | |
| against 4.2 wrong actions; at 0.40 that is 11.3 against 8.2. On held-out speakers | |
| saying *non-command* words, false-accept is 22.0% at 0.60 against 50.5% at 0.40. | |
| Pick the point on training speakers, never on the held-out set. | |
| **The gate cannot fix a misclassification, and this is now measured rather than | |
| argued.** All 12 of the stranger's wrongly-fired clips above were sibling | |
| confusions, with top-1-minus-top-2 margins of 0.348, 0.528, 0.574, 0.586, 0.598, | |
| 0.657, 0.711, 0.734, 0.742, 0.770, 0.773 and **0.949**. Tightening the sibling | |
| margin to 0.35 would catch 1 of the 12; catching half needs ~0.65; catching all | |
| needs >0.95, which would also reject legitimate commands. One clip was predicted | |
| `speed_3` at 0.969 confidence. | |
| A gate sees only the output distribution, and *wrong-and-certain* is | |
| indistinguishable from *right-and-certain*. Thresholds trade coverage against | |
| precision; they cannot recover information the model never had. | |
| One consequence worth stating plainly: because `CP_MIN_MARGIN` (0.15) is below | |
| `2 × 0.60 − 1 = 0.20`, and a softmax sums to 1, the global margin branch **cannot | |
| fire at this operating point**. It is inert by algebra, not by intent, and no | |
| compiler warns about a branch no input can reach. | |
| ## Intended use | |
| A hobby/research voice→UART bridge for one fan model. Not suitable for safety-, | |
| security- or accessibility-critical control, where a 5–20% miss rate on an | |
| unfamiliar voice is not acceptable. | |