| --- |
| license: cc-by-4.0 |
| tags: [audio-classification, vocal-bursts, non-verbal, voiceclap, multi-label] |
| pretty_name: VocalBurst Classifier (multi-label) |
| --- |
| |
| # VocalBurst Classifier — multi-label |
|
|
| A **presence classifier** for non-speech **vocal bursts** (laughs, sighs, gasps, coughs, groans, throat |
| sounds, …) over the **82-class** LAION VocalBurst taxonomy + `no_burst`. **Multi-label** (independent |
| sigmoid per class) — a clip can contain several bursts. Works on pure bursts and bursts embedded in speech. |
|
|
| ## Architecture |
| `laion/voiceclap-commercial` (768-d, frozen) → **MLP 2048-wide × 4 deep** (LayerNorm+GELU+dropout) → 83 sigmoids. ~14.4M trainable params. |
|
|
| ## Training |
| - **Positives:** Gemini-confirmed single bursts (DACVAE, 82 classes) + **real speech-with-bursts** (gemini_pro_25, multi-label). |
| - **Negatives / speech carriers:** Emilia clips filtered burst-free by `laion/vocalburst-locator`. |
| - **~97k samples**: pure bursts + composites (burst + speech carriers + a 2nd burst) + 25% overlays (burst mixed over speech) + neg+neg. The speech carriers carry no label, so the head must localise the burst, not memorise audio. Balanced (≤1000/class); 50 epochs, AdamW, BCE with per-class pos-weight, best-mAP checkpoint. |
|
|
| > **v2 (current):** retrained with **9,000 multilingual burst-free negatives** (FLEURS: Chinese, Hindi, Bengali, Arabic, Persian, Urdu, Tamil, Telugu, Vietnamese, Thai, Indonesian, Japanese, Korean, Swahili, Yoruba, Zulu, Turkish, Russian) — fixes hallucinated bursts on non-European speech. |
|
|
| ## Results (Gemini-mixed validation) |
| | metric | fine (82) | coarse (16 families) | |
| |---|---|---| |
| | macro mAP | 0.39 | **0.64** | |
| | top-1 exact | 49% | – | |
| | a true label in top-3 | 77% | – | |
| Best when you take the **top-1** prediction (which the two-stage combo demo uses). Much stronger at the coarse "which family of sound" level than the exact fine subtype. |
|
|
| ## How it works |
| Two stages: a **frozen** `laion/voiceclap-commercial` audio encoder turns a clip into a **768-d embedding** |
| (`encode_waveform`, auto-downloaded — the repo needs no extra setup), then this small trained **MLP head** |
| maps it to **83 outputs = 82 VocalBurst classes + `no_burst`** (taxonomy: [LAION-AI/voice-taxonomies · vocalburst](https://github.com/LAION-AI/voice-taxonomies/tree/main/vocalburst)). |
| A **no-burst gate**: if `P(no_burst) ≥ 0.5` the clip is declared burst-free (no false alarm); otherwise the top classes are returned. Clips are truncated to the first 30 s (the encoder's window). |
| |
| ## Usage |
| ```python |
| from inference import VocalBurstClassifier |
| clf = VocalBurstClassifier("laion/vocalburst-classifier-multilabel") # HF repo id, or a local checkout dir |
| print(clf.predict("clip.wav")) # -> {no_burst, p_no_burst, top1, predictions:[(class,prob)], group} |
| ``` |
| |
| ## Files |
| `model.pt` (MLP weights) · `config.json` (arch) · `classes.json` (83 labels, index order) · |
| `class_to_group.json` (fine→16 coarse families) · `inference.py` · `example.py` · `requirements.txt`. |
| |
| ## Interactive demos (audio + predictions) |
| - Multi-label predictions vs ground truth: https://projects.laion.ai/procedural-voice-captions/vocalburst-predictions/ |
| - Single-burst classifier: https://projects.laion.ai/procedural-voice-captions/vocalburst-single/ |
| - Two-stage detect→classify combo: https://projects.laion.ai/procedural-voice-captions/vocalburst-combo/ |
| - On MOSS character voices: https://projects.laion.ai/procedural-voice-captions/vocalburst-character/ |
| |
| Training data + embeddings: [laion/vocalburst-classification](https://huggingface.co/datasets/laion/vocalburst-classification). |
| Embedder: [laion/voiceclap-commercial](https://huggingface.co/laion/voiceclap-commercial). License: CC-BY-4.0. |
| |
