byosm-vad

Voice activity detection for a realtime voice agent. For every 10ms of microphone audio, say whether a human is speaking. Must ignore background noise, coughs, laughter, door slams, keyboard, crying babies and other non-speech human sounds. Runs on-device next to the agent, streaming, in well under 1ms per frame.

A single-purpose System 1 model built with BYOSM: a JEPA-style encoder pretrained with LeJEPA and trained only for this task, with a calibrated decision policy, shipped as a 0.10M-parameter ONNX bundle that runs on any CPU with onnxruntime.

task	audio / frame (one decision every 10 ms)
labels	non_speech, speech
parameters	101,458
backbone	tiny-crnn · embedding dim 96
objective	f1, primary suite ava
decision policy	hysteresis, threshold 0.425 (calibrated on ava's calib split)
CPU, streaming	0.186 ms per 30 ms chunk, 1 thread (PyTorch; the ONNX bundle is faster)

Labels

• non_speech: silence, noise, music, and non-verbal human sounds (coughing, laughing, breathing, crying)
• speech: a frame inside an utterance of human speech, including pauses up to ~300ms inside an utterance

Comparison

Every model below, this one and the baselines, was scored by the same code on the same frozen test splits, with its decision policy calibrated on the same calib splits. Bold = this model.

model	params	builtin f1	ava f1 (primary)	builtin ROC-AUC	ava ROC-AUC
v3 (this model)	0.10M	–	0.8816	–	0.942
v2 (BYOSM experiment)	0.10M	–	0.8763	–	0.939
silero (baseline)	0.55M	–	0.8653	–	0.923
v1 (BYOSM experiment)	0.10M	–	0.8485	–	0.900
energy (baseline)	–	–	0.7308	–	0.694

Test evaluation (this model)

suite	dataset	test items	f1	95% CI	precision	recall	ROC-AUC	avg precision	decision threshold
ava	nccratliri/vad-human-ava-speech	20	0.8816	0.863-0.899	0.8546	0.9103	0.9424	0.9519	0.425

Usage

pip install huggingface_hub numpy onnxruntime soundfile

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("bsye/byosm-vad")
sys.path.insert(0, path)
from byosm_rt import load

m = load(path)
import soundfile as sf
wav, sr = sf.read("clip.wav", dtype="float32")   # 16 kHz mono
print(m.predict(wav)["segments"])
emb = m.encode(...)                # the learned representation (JEPA-style), for probes / retrieval / other heads

Streaming

s = m.stream()
for chunk in mic_chunks():         # float32 mono, any size
    out = s.step(chunk)            # {"probs": [...per 10ms...], "speaking": bool, "events": [{"type": "start"|"end", "t_s": ...}]}

Local HTTP server

python byosm_rt.py serve . --port 8800
# GET /v1/models · POST /v1/vad/predict · POST /v1/vad/encode

How it was built

1. Spec: labels, definitions, objective and data plan in spec.yaml (included in this repo).
2. Data (8,400 training / 1,000 dev examples):

• openslr/librispeech_asr: role positive
• ashraq/esc50: role negative
• nccratliri/vad-human-ava-speech: role labeled, partition train

3. LeJEPA pretraining (800 steps): multi-view prediction + SIGReg (isotropic-Gaussian regulariser on random projections), no EMA teacher, no stop-gradient.
4. Supervised training: EMA of the weights, mixed precision; checkpoint chosen on dev (best epoch 16).
5. Evaluation: frozen eval suites; decision policy calibrated on each suite's calib split; baselines scored identically; promotion decided on calib, test only reported.
6. Improvement lineage (each step proposed by byosm improve from the previous board):

• v1: initial spec
• v2 ← real-data-ava (from v1): ava: objective 0.848 vs silero 0.855, 0.046 below builtin. The model has never seen this domain; train on the source's train partition (hash-disjoint from the suite's calib/test files).
• v2 ← utterance-labels (from v1): ava: calibration had to move the decision threshold to 0.20 (P=0.824, R=0.875) -> the suite's label convention differs from training's. Training labels only voiced frames (gaps > 60ms are negative); segment suites usually label whole utterances. Fill gaps up to 300ms in synthetic labels.
• v3 ← manual (from v2): train.ema=true train.holdout_test=auto

Limitations

• Trained on the train partition of the source behind ava (file-level hash split, disjoint from that suite's calib/test files). Results there are in-domain; baselines may not have seen that domain.
• Training data includes ESC-50 (CC BY-NC 3.0): non-commercial. Check licences before commercial use.
• Evaluated on ava (20 test items). Small suites have wide confidence intervals.

Reproduce

git clone https://github.com/bsye/BYOSM && cd BYOSM && pip install -e .
# put spec.yaml from this repo at specialists/vad/spec.yaml, then:
byosm build vad && byosm board vad && byosm export vad