README.md

# Model card — `dscnn_fan` int8 (voice-fan-controller)



The deployed model: 18-class spoken-command recognition (16 fan commands +

`_silence_` + `_unknown_`) running on a plain ESP32-WROOM-32. **28,584 bytes**,

int8, no accelerator. Input is a (81, 32) log-mel plane = 2,592 floats; output is

an 18-way softmax.



The flashed image does **not** run TensorFlow Lite Micro. It runs a generated

static-C engine (`dscnn_micro`) that reproduces the same graph — verified against
the TFLite interpreter at **0 prediction flips over 200 clips**, pre-softmax
logits within 1 LSB. A TFLM build still exists behind the same C API and is
selectable at menuconfig, but it is not what ships.

The hard requirement is **speaker independence** — it has to work for someone who
never recorded a clip. Everything below is organised around how well it does
that, and around not quoting numbers that measure something easier.

> The retired `func_mlp` model card is at `archive/MODEL_CARD_func_mlp.md` in
> this model repo (`docs/archive/MODEL_CARD.md` in the code repo). Its accuracy

> numbers do not apply to this model — they are in-speaker.



## The number that matters



**Leave-one-speaker-out (LOSO)** — score each speaker with a model that never

heard them. It is the only honest metric here, and it belongs to a **recipe**,

not to a weights file: the shipping model trains on everybody, so it has no

held-out voice of its own left to be measured against.



The **deployed** model's own recipe (`c48_rio15` — all voices, two-stage
transfer with BN frozen, `domain_aug=0.5`, encoder widened to 48 channels;
measured 2026-07-28) reaches **91.69% headline**, 5 seeds per fold:

| anirudh | austin | francis | mishelle | headline |
|---|---|---|---|---|
| 91.34% | 93.86% | 95.55% | **86.00%** | **91.69%** |

That is **+3.25pp** over the last `func_mlp`-era shipping recipe (88.44%),
against 1 SE of 1.81pp — clear of the pre-registered accept bar (≥2pp *and*
≥1 SE).

Against the *immediately* preceding recipe (`c48_deployable`, 91.37%) it is only
**+0.32pp** — inside noise. The two differ by corpus, not by code: three more
speakers' worth of recordings (+710 clips). It is reported as a corpus refresh,
not as a lever that cleared a bar.

Eleven further complete speakers are scored outside the headline. Averaged over
all **fifteen** full-coverage speakers — the widest "what a stranger gets" view —
it reads **90.05%** (SE 2.27pp), and its **worst voice is 66.85%** (aak). Read
the worst, not the mean: this refresh raised the average while *lowering* the
hardest voice by 1.98pp, which is precisely the failure mode the worst-voice
rule exists to catch. The full spread, weakest first:

```

aak 66.9 · savio 74.0 · mishelle 86.0 · rio 86.5 · allen 90.5

anirudh 91.3 · adithyasharma 92.9 · alwin 94.1 · austin 93.9 · anon1 93.3

anon2 95.0 · danushkrishna 95.4 · francis 95.6 · adithya 97.0 · teresa 98.4

```

**The width helped studio voices and hurt browser ones.** Widening the encoder
raised every studio-corpus fold (+1.53 to +5.64pp) but *lowered* all three small
browser-recorded folds (adithyasharma −5.93, anon1 −2.25, anon2 −6.22). That is
roughly 14pp of fold-sum, and it is the whole reason this model lands just under
90%. A headline win does not automatically transfer to strangers on a laptop mic.

A matched control run separates the two things that changed: the extra recorded
data is worth **+0.88pp** on its own (a reject as a lever), and capacity alone is
**+2.04pp**. The width is the real lever, and it clears the 2pp bar narrowly —
not by a comfortable margin.

(The fifteen-speaker average is not comparable to older fourteen- or seven-speaker
figures — the corpus grew, so the same statistic now covers a different
population. Only the frozen headline four are comparable across runs.)

**In-speaker validation for the deployed model is 97.47% and is not a

generalization estimate.** Every held-out clip there belongs to a speaker who is
in training — by design, since nobody is held out of the shipping model. The gap
between the two kinds of number is the entire point of this project: an earlier
architecture scored ~95% in-speaker and **43.7%** LOSO. If you quote one number,
quote a LOSO one.

## Limitations, in order of how likely they are to bite

1. **Pronunciation, not channel, is what sinks the worst voice.** aak sits at
   66.85% — twenty-three points below the mean, and moving the wrong way across the
   last two recipes — and the errors are *within-word*
   confusions, not noise. A channel probe put that speaker at z=0.98 (unremarkable
   recording conditions), and cepstral mean normalisation made things worse by
   2.8pp, so the gap is genuinely acoustic: those pronunciations land near the
   wrong class in feature space. More dense recordings of the weak voices is the
   only lever with evidence behind it; roughly eight code-side levers have been
   tried and spent.
2. **Vocal register is the second axis.** Nearly every training speaker sits at
   107–149 Hz median F0; one sits at 255.7 Hz, roughly an octave up. That speaker
   is still the weakest headline fold (86.00%) and is the only one register
   augmentation helps. Expect degradation for voices outside the trained range,
   which includes most children. The headline averages three men and one woman,
   so it *understates* this failure mode.
3. **English only, one accent pool, 18 speakers.** The encoder is pretrained on
   9,286 outside speakers, but fine-tuning corpus diversity is small — and four
   of the scored fifteen contributed only ~16–20 clips each, so their coverage is
   nominal rather than substantial. Those tiny folds are honest (genuinely held
   out) but noisy: one misheard clip moves such a fold by ~6pp.
4. **`timer_*` is the weakest command group.** Public pretraining corpora

   contain ten instances of the word "timer" in total, so transfer cannot teach

   it — it is learned only from the recorded set.

5. **Siblings dominate the errors, and the gate cannot catch them.** 66.7% of all

   command errors confuse two parameters of the *same* command

   (`power_on`↔`power_off`, `speed_2`↔`speed_3`, `timer_2h`↔`timer_4h`) — errors

   that produce a plausible-looking wrong action rather than an obvious one. This

   was priced on silicon (see *Downstream gate*): the wrong answers are often

   delivered with **high** confidence, so no threshold separates them from correct

   ones.

6. **`_unknown_` has zero human speakers** — it is sourced from YouTube audio.

   Rejection of *speech* that is not a command is therefore weaker than the

   negative-class accuracy suggests.



## Feature contract — not optional



```

sample_rate 20 kHz · window 25.6 ms (n_fft 512) · stride 20 ms

mel_bins 32 · log-mel via power_to_db(ref=max, top_db=80) · input (81, 32, 1)

```



`ref=max` is a correctness requirement, not a tuning choice. The alternative

(`log(mel + 1e-6)`) has no amplitude normalisation, so raw int16 PCM on the

device shifts every feature by ~6σ of pure recording level — frozen BatchNorm

cannot absorb it and the model runs while predicting nonsense. `window 25.6 ms`

gives `n_fft = 512` so the device can use a radix-2 FFT.



The C frontend is measured amplitude-invariant (0.25× gain moves features by

0.0016 dB) and agrees with the training frontend to **<0.002 dB**.



## Training



DS-CNN (depthwise-separable conv stack, **672,672 MACs, 12,082 params**, 48

channels). The encoder is pretrained on Speech Commands v2 + MSWC — 112,049 clips

from 9,286 speakers, 28.1% female — then fine-tuned on **5,542 recorded clips
from 18 speakers**.



The 48-channel width is a re-pretrain, not a knob: the encoder weights are shaped

by `channels`, so widening it invalidates the previous encoder and forces a full

pretraining run before any comparison is fair. The previous 32-channel model was

6,706 params / 441k MACs / 20,632 bytes.



**Transfer is what makes this work: from-scratch on the same data scores 45.9%,
against 67.4% pretrained.** That +21.5pp is the single largest gain on record
here, and nothing else has come within 20pp of it. (That pair was measured with
an *earlier* encoder — `dscnn_fan_scv2`, 11,376 speakers — not the 9,286-speaker
`scv2_max` above. It is quoted as evidence that pretraining is what carries this
model, which is encoder-independent; it is not a measurement of the encoder
shipping today.)

The fine-tune is two-stage: 15 epochs with the encoder frozen so the randomly
initialised 18-class head cannot wash out the transferred filters, then unfreeze
at `lr 3e-4` with **BatchNorm statistics frozen** — those were estimated over
9,286 speakers and re-estimating them from a handful of fold voices reintroduces
the per-channel speaker signature the pretraining was for.

Two train-time-only augmentations, neither of which costs anything on device:

- **register augmentation** (Praat "Change gender", moving F0 and formants
  independently) — covers the register axis the corpus barely spans;
- **domain augmentation** (reverb + bandlimit + µ-law codec round-trip) — added
  for the deployed model to close the browser-microphone gap. **It has still

  never been isolated in a controlled run**, so it remains a bet, and it is
  labelled as one. What *is* known is that the gap is not closed: the three
  browser-recorded folds were the only ones the wider encoder made worse.

Training is seeded but **not bit-reproducible** — oneDNN varies float reduction
order across threads, moving single folds by up to 5.1pp between identical runs.
Compare 5-seed means, not folds.

## On device

Measured on a real ESP32-D0WD-V3 from the **2026-07-27 flashed image**. That
image carries the *previous* weights (`c48_deployable`); the model described by
this card is built but **not yet flashed**. Every number in this section is a
property of the graph, not of the weights — identical shapes, ops, parameter
count and MACs — so it carries over unchanged. The accuracy numbers above do
not: those belong to `c48_rio15`, and the board has not yet been asked to
reproduce them.

| | | previous (c32, TFLM, single-core) |
|---|---|---|
| frontend | **18.6 ms** | 45.3 ms |
| invoke | **34.2 ms** | 49.75 ms |
| total | **~53 ms** (after a 1,600 ms capture window) | ~95 ms |
| tensor arena | **13,184 B** (static engine) | 17,324 B used / 20,480 reserved |
| flash | **28,584 B** | 20,632 B |

**Do not credit that speedup to the model.** This image is ~1.8× faster while
computing **+53% more MACs**, because two other things changed at the same time:
the static-C engine replaced TFLM, and both the STFT and the conv stack now use
the second core. The wider model taken alone would have been *slower*. Two
variables moved; the comparison is reported, not attributed.

The arena is unchanged at 13,184 B despite the wider layers, which is arithmetic
rather than luck: the peak simultaneously-live tensor pair is the input features
plus the stem output, and `channels` does not touch `stem_channels=16`.

Ops: CONV_2D, DEPTHWISE_CONV_2D, MAX_POOL_2D, RESHAPE, FULLY_CONNECTED, SOFTMAX.
`MAX_POOL_2D` deliberately — `GlobalMaxPooling2D` lowers to `REDUCE_MAX`, which
has no ESP-NN kernel and fails `AllocateTensors()` at boot.

### Validated on silicon against a true stranger

This image was tested over UART with **180 clips from a speaker who is not in the

corpus** — recorded after the model was built, so it had genuinely never heard
him, and none of his audio influenced training:

| | top-1 (argmax) | actually fires (after the gate) |
|---|---|---|
| all 180 clips | 83.3% | 76.7% |
| excluding `speed_5` | **92.9%** | **89.0%** |
| `speed_5` alone | 24.0% | 0.0% |

Seven of his eight recorded classes land between 84% and 100%. One collapses:
16 of 25 `speed_5` clips are heard as `speed_3`. That is not a mislabelled
recording — his `speed_5` runs 0.650 ± 0.044 s against `speed_3`'s
0.787 ± 0.135 s, a consistent and distinct utterance the model maps onto the
wrong class.

**Read the whole distribution, not the headline.** A single class failing this
way is invisible in an averaged score and obvious to the person using the fan.

(An earlier image was validated the same way against a different unseen speaker
at 96.5% over 400 clips. She is an easy voice — her own held-out fold
independently scores ~96% — so that number is evidence the deployment path is
correct, not a generalization estimate.)

An integer (Q15) build of the frontend also exists, verified accuracy-neutral
(0/200 model predictions changed), which removes the FPU requirement for cheaper
targets. It is staged, not flashed.

## Downstream gate

The classifier output is not the device's decision. A confidence gate
(`command_postprocessor`) suppresses low-confidence and ambiguous results before
any UART frame is emitted. The shipped point is **`conf 0.60`**, and it is a real
trade: at 0.60, 23.1 of every 100 commands from a new speaker are silently ignored
against 4.2 wrong actions; at 0.40 that is 11.3 against 8.2. On held-out speakers
saying *non-command* words, false-accept is 22.0% at 0.60 against 50.5% at 0.40.
Pick the point on training speakers, never on the held-out set.

**The gate cannot fix a misclassification, and this is now measured rather than

argued.** All 12 of the stranger's wrongly-fired clips above were sibling
confusions, with top-1-minus-top-2 margins of 0.348, 0.528, 0.574, 0.586, 0.598,
0.657, 0.711, 0.734, 0.742, 0.770, 0.773 and **0.949**. Tightening the sibling
margin to 0.35 would catch 1 of the 12; catching half needs ~0.65; catching all
needs >0.95, which would also reject legitimate commands. One clip was predicted
`speed_3` at 0.969 confidence.

A gate sees only the output distribution, and *wrong-and-certain* is
indistinguishable from *right-and-certain*. Thresholds trade coverage against
precision; they cannot recover information the model never had.

One consequence worth stating plainly: because `CP_MIN_MARGIN` (0.15) is below
`2 × 0.60 − 1 = 0.20`, and a softmax sums to 1, the global margin branch **cannot

fire at this operating point**. It is inert by algebra, not by intent, and no
compiler warns about a branch no input can reach.

## Intended use

A hobby/research voice→UART bridge for one fan model. Not suitable for safety-,
security- or accessibility-critical control, where a 5–20% miss rate on an
unfamiliar voice is not acceptable.