README.md

---
license: apache-2.0
library_name: esp-idf
tags:
  - voice-activity-detection
  - vad
  - esp32-p4
  - embedded
  - audio
  - risc-v
  - dfsmn
  - quantization
---

# FireRedVAD Models for ESP32-P4

Converted [FireRedVAD](https://github.com/FireRedTeam/FireRedVAD) models optimized for ESP32-P4 microcontrollers.
All models use the custom `.frvd` binary format with native RISC-V PIE (Position Independent Execution) acceleration.

**Source Code:** https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

## Quick Start

```bash
cd examples/console_vad
pip install huggingface-hub
python download_models.py
```

```bash
idf.py build flash monitor
```

```
firevad> vad_model_list
firevad> vad_model_load stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd
firevad> vad_infer_mic
```

## Available Models

### stream-vad/ — Real-time streaming (causal, 10ms latency)

Best for real-time voice activity detection. Model is fully **causal** — no future context.
Runs in ~4.5ms per 10ms frame on ESP32-P4 @ 400MHz.

| Quantization | File | Size | Notes |
|---|---|---|---|
| **INT8-CH** | `stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd` | 576 KB | **Recommended.** Per-channel scale factors, near-FP32 accuracy |
| INT8 | `stream-vad/int8/firered-stream-vad-int8.frvd` | 556 KB | Global scale factor per layer, slightly lower accuracy |
| INT16 | `stream-vad/int16/firered-stream-vad-int16.frvd` | 1.1 MB | Higher precision, ~2x memory use |
| FP32 | `stream-vad/fp32/firered-stream-vad-fp32.frvd` | 2.2 MB | Development only. Too slow for real-time on P4 (~35ms/frame) |

### vad/ — Offline batch VAD (non-causal, 1-second chunks)

Uses bidirectional context. Higher accuracy than stream-vad, but adds latency.
Not suitable for real-time streaming.

| Quantization | File | Size | Notes |
|---|---|---|---|
| **INT8-CH** | `vad/int8-ch/firered-vad-int8-ch.frvd` | 597 KB | Recommended for batch processing |
| INT8 | `vad/int8/firered-vad-int8.frvd` | 576 KB | |
| INT16 | `vad/int16/firered-vad-int16.frvd` | 1.1 MB | |
| FP32 | `vad/fp32/firered-vad-fp32.frvd` | 2.3 MB | |

### aed/ — Audio Event Detection (Speech / Music / Singing)

Multi-class audio classifier. Identifies speech, music, and singing simultaneously.

| Quantization | File | Size | Notes |
|---|---|---|---|
| **INT8-CH** | `aed/int8-ch/firered-aed-int8-ch.frvd` | 598 KB | Recommended |
| INT8 | `aed/int8/firered-aed-int8.frvd` | 576 KB | |
| INT16 | `aed/int16/firered-aed-int16.frvd` | 1.1 MB | |
| FP32 | `aed/fp32/firered-aed-fp32.frvd` | 2.3 MB | |

## Quantization Explained

### Why INT8-CH (Per-Channel) is Recommended

Standard per-tensor INT8 quantization assigns **one** global scale factor per weight matrix.
DFSMN architectures have wide variance in weight distribution across output channels — a single
scale factor cannot capture this range accurately, causing silent accuracy loss.

**Per-Channel INT8 (`int8-ch`, Version 4 in the `.frvd` format)** assigns **one scale factor
per output channel**. This preserves near-FP32 accuracy at INT8 speed and memory cost.

| | int8 | int8-ch | int16 | fp32 |
|---|---|---|---|---|
| Format version | 2 | 4 | 3 | 1 |
| Inference time (P4) | ~4.47ms | ~4.54ms | ~6ms | ~35ms |
| Memory bandwidth | 4x less than FP32 | 4x less than FP32 | 2x less than FP32 | baseline |
| Accuracy vs FP32 | Lower | Near-identical | High | Reference |

## Benchmark Results (ESP32-P4, 400MHz, 10ms audio frame)

| Model | Avg Latency | Real-Time Load | Usable? |
|---|---|---|---|
| stream-fp32 | 35.2 ms | 352% | No — audio drops |
| stream-int8 | 4.47 ms | 44.7% | Yes |
| **stream-int8-ch** | **4.54 ms** | **45.4%** | **Yes — Recommended** |

Real-time budget for 10ms frames: 10ms. Anything above 10ms (>100% load) causes audio drops.

## Hardware Requirements

- MCU: ESP32-P4 (RISC-V dual-core, 400MHz)
- PSRAM: 32 MB
- Flash: 16–32 MB
- RAM at runtime: ~150 KB
- Microphone: INMP441 or equivalent I2S digital microphone @ 16kHz

**Note:** INT8 and INT8-CH models use ESP32-P4 PIE vector instructions
(`esp.vmulas.s8.xacc` etc.) with mandatory 16-byte memory alignment, handled automatically
by the runtime. FP32/INT16 models work on other ESP32 variants (S2, S3) but without
PIE acceleration.

## Known Limitations (Honest Assessment)

1. **Noise sensitivity:** Performance degrades in low-SNR environments (loud machinery, strong wind).
   False positive rate increases at SNR < 5dB.
2. **Microphone dependency:** Model was trained on clean 16kHz PCM. A high-quality I2S
   microphone with hardware PGA gain control is required for reliable results.
3. **No built-in noise suppression:** The ESP-IDF runtime does not include NS/AEC.
   Echo cancellation is available via the shared APLL (I2S0 + I2S1 synchronized clocking).
4. **APLL sharing warning:** When both TX and RX I2S ports are active, the ESP32-P4 APLL
   runs at 8,191,999 Hz instead of 8,192,000 Hz (1 Hz deviation). This is hardware-expected
   behavior, not a bug. Both ports share the same clock, which is ideal for AEC.

## .frvd File Format

Custom binary format, version-tagged in header byte [4..7]:

```
Header (32 bytes):
  [0..3]   Magic: "FRVD"
  [4..7]   Version: 1=fp32, 2=int8, 3=int16, 4=int8-per-channel
  [8..11]  Model type: 0=VAD, 1=Stream-VAD, 2=AED
  [12..15] Total parameter count
  [16..23] DFSMN block count + DNN layer count
  [24..31] Reserved

Architecture Metadata (32 bytes):
  Input dim, hidden size, projection size, output dim,
  lookback order/stride, lookahead order/stride

CMVN block:
  dim (uint32) + means[dim] (float32) + istd[dim] (float32)

Layer data (sequential):
  Per tensor: CRC32 name hash + element count + [scale per channel for int8-ch] + data
```

## Conversion Pipeline

Original FireRedVAD PyTorch checkpoints -> `.frvd`:

```bash
# Requirements
pip install torch kaldiio numpy

# Stream-VAD INT8-CH (recommended)
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8-ch \
    --model-type stream-vad \
    --quantize-int8-per-ch

# Stream-VAD INT8
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8 \
    --model-type stream-vad \
    --quantize-int8

# Verify conversion
python tools/converter/verify_conversion.py \
    --frvd examples/console_vad/converted_models/stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd
```

## License & Attribution

### Original Models
- **FireRedVAD** by Xiaohongshu (FireRedTeam) — Apache 2.0
- Source: https://github.com/FireRedTeam/FireRedVAD
- HuggingFace: https://huggingface.co/FireRedTeam/FireRedVAD

### ESP32-P4 Port
- **FireRedVAD-ESP32-P4** by Strg-Alt-Entf-0x00 — Apache 2.0
- Repository: https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

## Citation

```bibtex
@misc{fireredvad-esp32p4,
  title={FireRedVAD for ESP32-P4: Optimized Voice Activity Detection for Embedded Systems},
  author={Strg-Alt-Entf-0x00},
  year={2026},
  howpublished={\url{https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4}},
}
```