matbee/whisper-to-vad

Whisper → V/A/D Regressor

A lightweight regressor that maps Whisper-large-v3-turbo encoder hidden states to continuous Valence / Arousal / Dominance coordinates in [-1, +1]^3.

Trained on CREMA-D under the Open Database License. Used as an external V/A/D auditor in the remotemedia-sdk affect-pipeline projects (LLM activation steering, audio→blendshape diffusion).

⭐ New: ridge_l12_neutral, cross-corpus model (2026-10-15)

ridge.onnx was trained and tested on CREMA-D only. On speech from other corpora it gets arousal roughly right, but scores everything, including happy speech, as strongly negative valence (bias ≈ −0.4). ridge_l12_neutral fixes that. It reads an earlier encoder layer and is applied to embeddings minus the speaker's own calm-voice baseline.

Scores below are on corpora excluded from training (leave-one-corpus-out, held-out speakers), with a 5-utterance calm warm-up. CCC is the primary metric: it penalises the offsets that Pearson r ignores.

held-out corpus	model	V CCC	A CCC	D CCC	AUC happy>neutral (valence)	neutral clips' mean V
RAVDESS (eval only, never trained on)	ridge.onnx	0.24	0.37	0.32	0.47	−0.45
	ridge_l12_neutral	0.66	0.59	0.65	0.91	−0.08
JL-Corpus (model trained without JL)	ridge.onnx	0.40	0.55	0.36	0.91	−0.21
	ridge_l12_neutral recipe	0.54	0.73	0.47	0.99	−0.07

• Opus round-trip (32 kbps, the browser path): changes every axis by ≤ 0.01 CCC.
• Cold start: with no speaker reference yet (population baseline only), RAVDESS still gets V CCC 0.60 and AUC 0.87, but neutral speech reads at valence ≈ −0.3. One or two calm utterances remove most of that.
• What the hand-off's earlier failure was: a corpus-level shift in the layer −2 embeddings, not a missing signal. More data alone improved ranking but not the offset.

Files

file	what
ridge_l12_neutral.onnx	Ridge, same I/O as ridge.onnx: whisper_embed (N, 1280) → vad (N, 3)
ridge_l12_neutral.json	Runtime contract (layer, normalisation, baseline gate) + provenance
ridge_l12_neutral.population_neutral.npy	(1280,) float32: population calm-voice embedding, used before the speaker's own is known

Runtime contract (different from ridge.onnx)

1. Layer −12 of the Whisper-large-v3-turbo encoder (hidden_states[-12], i.e. after block 21 of 32), mean-pooled over the processor's padded 30 s window, exactly like ridge.onnx's layer −2 input.
2. Subtract a per-speaker reference before the ONNX call: reference = (population + Σ accepted) / (1 + n_accepted).
   • Admit an utterance into the reference when the model scores its |arousal| < 0.3. Don't gate on valence: before the speaker's reference exists every valence carries the population offset.
   • After 20 accepted utterances, decay so the reference can follow drift.
   • Treat output as low confidence until 2 utterances have been accepted.
   • Reset per speaker.
   • A reference built from ≥ 40 % emotional speech brings the neutral offset back.

import json, numpy as np, onnxruntime as ort, torch, librosa
from transformers import AutoModel, AutoProcessor

cfg = json.load(open("ridge_l12_neutral.json"))["runtime"]
processor = AutoProcessor.from_pretrained("openai/whisper-large-v3-turbo")
encoder = AutoModel.from_pretrained("openai/whisper-large-v3-turbo").get_encoder().eval()
sess = ort.InferenceSession("ridge_l12_neutral.onnx", providers=["CPUExecutionProvider"])
population = np.load(cfg["population_baseline"])
accepted = []  # this speaker's low-arousal utterance embeddings

def predict(path):
    audio, _ = librosa.load(path, sr=16000)
    inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
    with torch.no_grad():
        hidden = encoder(**inputs, output_hidden_states=True).hidden_states[cfg["whisper_layer"]]
    embed = hidden.mean(dim=1)[0].numpy().astype(np.float32)
    reference = (population + np.sum(accepted, axis=0)) / (1 + len(accepted))
    vad = sess.run(["vad"], {"whisper_embed": (embed - reference)[None].astype(np.float32)})[0][0]
    if abs(vad[1]) < 0.3:
        accepted.append(embed)
    return vad, len(accepted) >= 2  # (valence, arousal, dominance), warmed up?

remotemedia-sdk's ProsodyVad reads ridge_l12_neutral.json and does all of this itself.

Training

• Ridge, α=100 on standardised features. Standardisation is folded into W/b, so the ONNX is still MatMul + Add.
• Trained on CREMA-D (7,442) + JL-Corpus (2,400) + EmoVoice-DB (24,923), every clip minus its speaker's mean neutral-clip embedding.
• Labels: the same categorical → V/A/D anchor table as ridge.onnx.
• Selected with a one-factor-at-a-time sweep (corpus mix, normalisation, α, encoder layer −1…−16, pooling, MLP vs ridge). Every row was scored leave-one-corpus-out on RAVDESS and JL.
• RAVDESS (CC BY-NC-SA) was used for evaluation only.
• Without EmoVoice the recipe still passes the same bars, with RAVDESS AUC happy>neutral 0.78 instead of 0.91.
• Evaluation code and the full tables: tools/affect_calibration/ in remotemedia-sdk (11_cross_corpus_eval.py, 12_vad_sweep.py, 13_online_baseline_sim.py, notes/whisper-to-vad-eval-2026-10-15.md).

Labels are acted categories mapped to fixed V/A/D points, not dimensional ratings, and every corpus here is acted or synthetic. Expect lower numbers on spontaneous conversation.

---

Original CREMA-D models

Two variants ship in this repo

Variant	File	Test Pearson r (V/A/D)	Notes
Ridge ⭐	ridge.onnx	0.58 / 0.66 / 0.69	Recommended. Linear projection, ~15 KB, faster.
MLP	mlp.onnx + mlp.onnx.data	0.28 / 0.62 / 0.55	Reference. 2-layer MLP, ~1.4 MB external weights.

Ridge wins on every axis on the test split. The MLP is shipped only as a comparison baseline; production use should default to ridge.onnx.

I/O signature

Both models share the same ONNX I/O:

Tensor	Name	Shape	dtype	Description
input	whisper_embed	(batch, 1280)	float32	Mean-pooled Whisper encoder hidden state
output	vad	(batch, 3)	float32	[valence, arousal, dominance] ∈ [-1, +1]^3

Critical: input is per-clip, not per-frame. Pool Whisper hidden states over time (mean across the time axis) before feeding to this regressor.

Inference snippet

import numpy as np
import onnxruntime as ort
from transformers import AutoModel, AutoProcessor

# 1. Load Whisper encoder + this regressor
processor = AutoProcessor.from_pretrained("openai/whisper-large-v3-turbo")
whisper = AutoModel.from_pretrained("openai/whisper-large-v3-turbo").get_encoder().eval()
sess = ort.InferenceSession("ridge.onnx", providers=["CPUExecutionProvider"])

# 2. Extract hidden states from layer -2 (penultimate); mean-pool over time
import torch, librosa
audio, _ = librosa.load("clip.wav", sr=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    out = whisper(**inputs, output_hidden_states=True)
embed = out.hidden_states[-2].mean(dim=1).numpy().astype(np.float32)  # (1, 1280)

# 3. Predict V/A/D
vad = sess.run(["vad"], {"whisper_embed": embed})[0]  # (1, 3)
print(f"valence={vad[0,0]:+.2f}  arousal={vad[0,1]:+.2f}  dominance={vad[0,2]:+.2f}")

Training

• Architecture (ridge): W ∈ R^(1280 × 3) + bias. Closed-form ridge regression with α=1.0, no regularization on bias. Total params: 3,843.
• Architecture (mlp): Linear(1280, 256) → GELU → Linear(256, 3), AdamW + cosine schedule, 100 epochs. Total params: 328,963.
• Audio frontend: Whisper-large-v3-turbo encoder, layer −2 (penultimate, pre-layernorm) hidden states, mean-pooled over time.
• Dataset: CREMA-D, speaker-disjoint split (train=1391, val=305, test=304 clips).
• Targets: continuous V/A/D coordinates derived from CREMA-D categorical labels via a fixed hand-authored anchor table (Russell circumplex + Mehrabian PAD + NRC-VAD style; see emotion_to_vad.py).

Detailed metrics

Ridge (recommended)

{
  "dataset": "crema_d",
  "whisper_model": "openai/whisper-large-v3-turbo",
  "whisper_layer": -2,
  "regressor": "ridge",
  "n_train": 1391,
  "n_val": 305,
  "n_test": 304,
  "speaker_disjoint": true,
  "axes": [
    "valence",
    "arousal",
    "dominance"
  ],
  "val": {
    "n": 305,
    "rmse": {
      "valence": 0.41578346490859985,
      "arousal": 0.27047833800315857,
      "dominance": 0.2509686052799225
    },
    "mae": {
      "valence": 0.3144280016422272,
      "arousal": 0.21582330763339996,
      "dominance": 0.20134007930755615
    },
    "pearson_r": {
      "valence": 0.6142750533217433,
      "arousal": 0.6183148626337297,
      "dominance": 0.7671747949052007
    }
  },
  "test": {
    "n": 304,
    "rmse": {
      "valence": 0.4229687750339508,
      "arousal": 0.2829131484031677,
      "dominance": 0.2870493531227112
    },
    "mae": {
      "valence": 0.3345738351345062,
      "arousal": 0.22188791632652283,
      "dominance": 0.2326992005109787
    },
    "pearson_r": {
      "valence": 0.5816889655848607,
      "arousal": 0.6575878679123864,
      "dominance": 0.6881453235310694
    }
  },
  "dry_run": false
}

MLP (reference)

{
  "dataset": "crema_d",
  "whisper_model": "openai/whisper-large-v3-turbo",
  "whisper_layer": -2,
  "regressor": "mlp",
  "n_train": 1391,
  "n_val": 305,
  "n_test": 304,
  "speaker_disjoint": true,
  "axes": [
    "valence",
    "arousal",
    "dominance"
  ],
  "val": {
    "n": 305,
    "rmse": {
      "valence": 0.4990813732147217,
      "arousal": 0.2946903705596924,
      "dominance": 0.299040824174881
    },
    "mae": {
      "valence": 0.42277732491493225,
      "arousal": 0.24514958262443542,
      "dominance": 0.25509873032569885
    },
    "pearson_r": {
      "valence": 0.339805625682777,
      "arousal": 0.518995070692627,
      "dominance": 0.672127424657303
    }
  },
  "test": {
    "n": 304,
    "rmse": {
      "valence": 0.5018822550773621,
      "arousal": 0.3011777400970459,
      "dominance": 0.32821124792099
    },
    "mae": {
      "valence": 0.4377283751964569,
      "arousal": 0.24337027966976166,
      "dominance": 0.28178495168685913
    },
    "pearson_r": {
      "valence": 0.2784075767971956,
      "arousal": 0.6199177507366342,
      "dominance": 0.5484364257265157
    }
  },
  "dry_run": false
}

License

MIT (this model). Upstream licenses:

• CREMA-D training data: Open Database License — commercial use allowed with attribution.
• Whisper-large-v3-turbo audio frontend: MIT.
• ridge_l12_neutral only: JL-Corpus (CC0, per the Kaggle upload by co-author Li Tian) and EmoVoice-DB (MIT; its audio was generated with OpenAI GPT-4o-audio).

If you publish work using this regressor, cite CREMA-D + the Whisper paper:

@article{cao2014cremad,
  title={CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset},
  author={Cao, Houwei and Cooper, David G and Keutmann, Michael K and Gur, Ruben C and Nenkova, Ani and Verma, Ragini},
  journal={IEEE Transactions on Affective Computing},
  year={2014}
}
@inproceedings{radford2023whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  booktitle={ICML},
  year={2023}
}