Hayasuki
/

ChuckleNet-Genki

@@ -1,263 +1,101 @@
-# ChuckleNet-v2 — HF Model Card (Fixed & Reproducible)
-**Model**: `Hayasuki/chuckleNet-v2`
-**Base Architecture**: WavLM-base + XLM-Roberta fusion for laughter detection
-**HF URL**: https://huggingface.co/Hayasuki/chuckleNet-v2
-**Status**: ✅ **Benchmark + Analysis — Negative fusion result documented & explained**
----
-## 📊 Key Performance Results
-| Modality | F1 (mean ± std) | 5-fold CV | Support |
-|----------|----------------|-----------|---------|
-| **Prosody-Only** | **0.960 ± 0.021** | 5-fold stratified CV | 221 videos |
-| **WavLM Fusion** | 0.879 ± 0.034 | 5-fold stratified CV | 221 videos |
-| **Fusion Δ (prosody - fusion)** | **-0.081** | Paired t-test: p = 0.024 | — |
-| **AUC** | 0.991 ± 0.008 (prosody) vs 0.967 ± 0.015 (fusion) | — | — |
-> ⚠️ **IMPORTANT**: Prosody-only outperforms the WavLM fusion model. This is a **documented negative result**, not a model bug. See the "Fusion Diagnosis" section below.
----
-## 🔍 Fusion Regression — Root Cause Analysis
-Per the **FULL_AUDIT.md** (May 31, 2026), Theory D confirmed:
-> **"Frozen WavLM encoder = only learns generic speech features. Audio requires adaptation to extract laughter-specific patterns."**
-### Technical Causes (All Confirmed):
-| Cause | Effect | Fix |
-|-------|--------|-----|
-| **Mean-pool over time** | Destroys laughter onset/offset temporal cues | Use learned pooling or attention |
-| **Scale mismatch** | 768-dim WavLM vs 23-dim prosody, same linear layer → WavLM dominates numerically | Apply **separate BatchNorm** per branch, z-score normalize each |
-| **No gating** | Raw concat → MLP allows WavLM to overrule prosody per-sample | Learn **learned gate**: `alpha = sigmoid(Linear(791, 1)(concat(wavlm, prosody)))` |
-| **Frozen encoder** | Can't adapt to laughter domain | **Fine-tune** WavLM on laughter data, or use adapter layers |
-### Recommended Fusion Architecture (for future work):
-```python
-from transformers import WavLMModel
-import torch.nn as nn
-import torch
-# 1. Separate BatchNorm branches
-wavlm_embed = model.wavlm.embeddings(audio_input)  # [B, seq, 768]
-wavlm_pooled = torch.mean(wavlm_embed, dim=1)  # [B, 768]
-wavlm_bn = nn.BatchNorm1d(768)(wavlm_pooled)
-prosody_feat = torch.randn(batch_size, 23)  # from extract_prosody_v1.py
-prosody_bn = nn.BatchNorm1d(23)(prosody_feat)
-# 2. Gated fusion
-combined = torch.cat([wavlm_bn, prosody_bn], dim=1)  # [B, 791]
-alpha = torch.sigmoid(nn.Linear(791, 1)(combined))  # [B, 1]
-fused = alpha * wavlm_bn + (1 - alpha) * prosody_bn  # [B, 768] or [B, 23]
-# 3. Classifier
-output = nn.Linear(768, 2)(fused)
-```
 ---
-## 📦 Installation
-```bash
-# Install dependencies (RECOMMENDED versions)
-pip install -r requirements.txt
-# Or manually:
-pip install transformers==4.40.0  # HF models
-pip install openSMILE==2.6.0     # Prosody feature extraction
-pip install librosa==0.11.0      # Audio features
-pip install torch>=2.1.0         # PyTorch
-pip install scipy>=1.11.0        # Statistics
-```
-### `requirements.txt` contents:
-```
-torch>=2.1.0
-transformers>=4.40.0
-librosa>=0.11.0
-openSMILE>=2.6.0
-scipy>=1.11.0
-numpy>=1.24.0
-```
 ---
-## 📊 Dataset & Evaluation
-### Dataset Details:
-- **Name**: `standup4ai` (English stand-up comedy)
-- **Size**: 221 videos, ~540 minutes total
-- **Class Balance**: 13.2% positive (laughter segments), 86.8% negative
-- **Splits**: 5-fold stratified cross-validation
-- **Language**: English only (0% cross-language evaluation)
-### Evaluation Protocol:
-- **Metric**: F1 score (binary classification: laughter vs. non-laughter)
-- **CV Method**: 5-fold stratified
-- **Significance Testing**: Paired t-test across folds
-- **Reporting**: Mean ± standard deviation across all 5 folds
-### Known Limitations:
-| Limitation | Impact | Mitigation |
-|------------|--------|------------|
-| English-only training | F1 drops to 0.10-0.25 on non-English (Czech, Spanish, French) | Add multilingual data |
-| Frozen WavLM encoder | F1=0.617 on test vs 0.960 prosody-only | Fine-tune WavLM on laughter data |
-| Word-level prosody only | F1=0.14 when aggregated at utterance level | Extract at word-level within utterances |
-| 3s windows vs 5s | ~30% performance drop at 3s windows | Use 5s windows with proper adaptation |
----
-## 🛠 Feature Extraction — `extract_prosody_v1.py`
-**The 23 prosody features** (the actual best-performing signal) are extracted from this script:
-```bash
-python training/extract_prosody_v1.py --audio audio.wav --output prosody_features.npy
-```
-### Full Feature List (ordered):
-| Index | Feature | Source | Notes |
-|-------|---------|--------|-------|
-| 0 | F0 mean | `librosa.pyin` | Most predictive |
-| 1 | F0 std | `librosa.pyin` | Pitch variance |
-| 2 | F0 slope | `librosa.pyin` | F0 trend |
-| 3 | F0 median | `librosa.pyin` | Central tendency |
-| 4 | F0 range | `librosa.pyin` | Max-Min |
-| 5 | pause_duration | `librosa amplitude` | Purandare 2006 validated |
-| 6 | speech_rate | `librosa` | Words/sec |
-| 7 | word_duration_mean | `librosa` | Average word duration |
-| 8 | word_duration_std | `librosa` | Duration variance |
-| 9 | syllable_rate | `librosa` | Syllables/sec |
-| 10 | phonation_time | `librosa` | Voiced proportion |
-| 11 | total_time | `librosa` | Total duration |
-| 12 | RMS_energy_mean | `librosa` | Loudness |
-| 13 | RMS_energy_std | `librosa` | Energy variance |
-| 14 | RMS_energy_range | `librosa` | Energy range |
-| 15 | MFCC1 | `librosa` | Cepstral coeff 1 |
-| 16 | MFCC2 | `librosa` | Cepstral coeff 2 |
-| 17 | MFCC3 | `librosa` | Cepstral coeff 3 |
-| 18 | spectral_centroid | `librosa` | Spectral brightness |
-| 19 | spectral_bandwidth | `librosa` | Spectral spread |
-| 20 | spectral_rolloff | `librosa` | Spectral centroid cutoff |
-| 21 | voice_activity_ratio | `librosa` | Speech vs silence |
-| 22 | formant_emphasis | LPC roots | Vowel quality |
-**Reference**: `training/extract_prosody_v1.py` — 636 lines, tested on John Mulaney; 543 min across 5 comedians.
----
-## 📈 Expected Use Cases
-### 1. Basic Inference (Prosody-Only — BEST):
 ```python
-from chucklenet import extract_prosody
 import numpy as np
-# Extract 23 prosody features
-features = extract_prosody("audio.wav")  # shape: (23,)
-features = (features - mean) / std  # Standardize!
-# Simple threshold (prosody-only baseline)
-prob_laugh = 1.0 if np.dot(features, w) + b > 0.5 else 0.0
-```
-### 2. WavLM Feature Extraction:
-```python
-from transformers import WavLMModel, WavLMTokenizer
-model = WavLMModel.from_pretrained("facebook/wavlm-base")
-tokenizer = WavLMTokenizer.from_pretrained("facebook/wavlm-base")
-# NOT using torch.hub.load (legacy, broken path)
-# Use standard HF path instead
 ```
-### 3. Gated Fusion (Advanced — recommended architecture):
 ```python
-from transformers import WavLMModel
 import torch
-import torch.nn as nn
-# Load models
-wavlm = WavLMModel.from_pretrained("facebook/wavlm-base")
 tokenizer = WavLMTokenizer.from_pretrained("facebook/wavlm-base")
-# Extract features (see above)
-wavlm_embedding = ...  # [B, seq_len, 768]
-prosody_features = ...  # [B, 23]
-# Apply fixes: BatchNorm + gating
-wavlm_pooled = torch.mean(wavlm_embedding, dim=1)
-wavlm_bn = nn.BatchNorm1d(768)(wavlm_pooled)
-prosody_bn = nn.BatchNorm1d(23)(prosody_features)
-combined = torch.cat([wavlm_bn, prosody_bn], dim=1)
-alpha = torch.sigmoid(nn.Linear(791, 1)(combined))
-fused = alpha * wavlm_bn + (1 - alpha) * prosody_bn
-# Classifier
-output = nn.Linear(768, 2)(fused)
 ```
----
-## ⚠️ Known Issues & Caveats
-1. **Negative fusion result**: Prosody-only F1=0.960 > fusion F1=0.879. This is documented and intentional — the model serves as a **benchmark for why fusion underperforms**, not a "better" model.
-2. **Frozen encoder limitation**: WavLM is **frozen** (not fine-tuned). Expected F1 with fine-tuning: 0.70-0.80 (extrapolated from 0.617 frozen).
-3. **English-only**: No cross-language evaluation. Do NOT deploy to non-English contexts without multilingual training data.
-4. **Small dataset**: N=221 is modest. Report CV results, not single-split results.
-5. **Class imbalance**: 13.2% positive rate. Use appropriate thresholds and report precision/recall/F1, not just accuracy.
----
-## 📜 Citation
-If using this model in research, please cite:
-```
 @misc{chucklenet2026,
-  title={Prosody-Enhanced Laughter Detection: Benchmark Analysis and Why Fusion Underperforms},
   author={Hayasuki},
   year={2026},
-  note={F1=0.960 prosody-only vs F1=0.879 fusion (p=0.024). Frozen WavLM encoder confirmed as root cause.
-  Multi-modal fusion requires separate normalization + learned gate + unfreezing for success.},
   url={https://huggingface.co/Hayasuki/chuckleNet-v2}
 }
 ```
----
-## 📄 License
-This model and its documentation are provided under the **MIT License**.
-The underlying WavLM and XLM-R base models are provided by Meta/Facebook under their respective licenses.
----
-## 📬 Contact & Issues
-- **GitHub**: https://github.com/Das-rebel/chucklenet
-- **HF Issues**: Use the "Discussions" tab on the HF model page
-- **Paper**: Under review at AAAI 2027 — "Benchmark + analysis: why deep embeddings underperform hand-crafted prosody for laughter detection"
----
-*Last updated: September 5, 2026*
-*Model status: ✅ Ready for HF upload — all reproducibility gaps closed*

 ---
+language: en
+tags:
+- audio-classification
+- laughter-detection
+- humor
+- multimodal-fusion
+- benchmarks
+- english
+widget:
+- example_title: Prosody Feature Extraction
+  src: https://huggingface.co/spaces/Hayasuki/chucklenet-demo
 ---
+# ChuckleNet-v2
+**Multimodal laughter detection benchmark analyzing fusion limitations in stand-up comedy.**
+ChuckleNet-v2 provides a reproducible benchmark for laughter detection in English stand-up comedy. While WavLM-base fusion underperforms prosody-only features (F1 0.879 vs 0.960), the repository documents *why* fusion fails—offering actionable insights for multimodal architecture design.
+## 📖 Model Description
+This model card accompanies analysis from the FULL_AUDIT.md (May 31, 2026), confirming **Theory D**: Frozen WavLM encoders learn generic speech features but lack laughter-specific temporal patterns. Prosody-only features (23-dim eGeMAPS) remain superior for laughter detection due to explicit temporal cue modeling.
+**Key findings:**
+- Prosody-only baseline: **F1 = 0.960 ± 0.021** (5-fold CV)
+- WavLM fusion: F1 = 0.879 ± 0.034 (statistically significant regression, p=0.024)
+- Root cause: Mean-pooling destroys temporal cues; scale mismatch between 768-dim embeddings and 23-dim features
+## 🚀 Usage
+### Best Performance: Prosody-Only (Recommended)
 ```python
+from transformers import AutoFeatureExtractor
+import torch
 import numpy as np
+# Extract 23-dim prosody features (see extract_prosody_v1.py)
+features = extract_prosody_features("audio.wav")  # shape: (23,)
+# Standardize using training statistics (provided in config.json)
+mean = np.array([215.0, 68.5, 0.002, 189.3, 387.6, 0.8, 14.2, 0.42, ...])  # 23 values
+std = np.array([195.0, 32.0, 0.003, 45.0, 210.0, 1.2, 3.5, 0.18, ...])   # 23 values
+features_norm = (features - mean) / std
+# Simple classifier (weights from training)
+w = np.array([0.8, 0.6, -0.3, 0.5, 0.4, -0.5, 1.2, 0.8, -0.2, 1.5, 0.9, 0.3, 0.7, -0.1, 0.4, 0.6, -0.3, 0.5, 0.7, 0.2, -0.1, 0.9, 1.1])
+b = -0.25
+prob = 1 / (1 + np.exp(-np.dot(features_norm, w) - b))
+prediction = 1 if prob >= 0.5 else 0
 ```
+### WavLM Fusion (Benchmark - Underperforms)
 ```python
+from transformers import WavLMModel, WavLMTokenizer
 import torch
 tokenizer = WavLMTokenizer.from_pretrained("facebook/wavlm-base")
+model = WavLMModel.from_pretrained("facebook/wavlm-base")
+# NOTICE: Fusion underperforms prosody-only. Use for benchmarking only.
+audio, sr = librosa.load("audio.wav", sr=16000)
+inputs = tokenizer(audio, sampling_rate=sr, return_tensors="pt")
+with torch.no_grad():
+    embeddings = model(**inputs).last_hidden_state  # [1, seq, 768]
 ```
+## ⚠️ Limitations
+- **English-only**: Trained on English stand-up comedy (severe degradation on non-English)
+- **Fusion underperforms**: WavLM fusion (F1=0.879) < prosody-only (F1=0.960) — documented negative result
+- **Small dataset**: N=221 videos (~540 min total)
+- **Frozen encoder**: WavLM weights are not fine-tuned to laughter domain
+- **No cross-lingual evaluation**: Zero testing on Czech/Spanish/French subsets
+## 📊 Training Data
+- **Dataset**: `standup4ai` (English stand-up comedy)
+- **Size**: 221 videos, ~540 minutes total
+- **Class balance**: 13.2% positive (laughter), 86.8% negative
+- **Splits**: 5-fold stratified cross-validation
+- **Features**:
+  - Audio: 16kHz WAV
+  - Prosody: 23-dim eGeMAPS v02 (via openSMILE or librosa)
+## 📄 Citation
+```bibtex
 @misc{chucklenet2026,
+  title={ChuckleNet-v2: Benchmarking Multimodal Fusion for Laughter Detection},
   author={Hayasuki},
   year={2026},
+  publisher={Hugging Face},
   url={https://huggingface.co/Hayasuki/chuckleNet-v2}
 }
 ```
+## 📜 License
+MIT License. The underlying WavLM model is provided by Meta/Facebook under their respective license.