Release rc1

README.md

@@ -0,0 +1,125 @@
+---
+license: bsd-2-clause
+language: en
+library_name: onnx
+pipeline_tag: audio-classification
+base_model: laion/whisper-tiny-base-emotion-voice-burst
+tags:
+  - pipecat
+  - speech-emotion-recognition
+  - sarcasm
+  - prosody
+  - onnx
+---
+
+# Pipecat Voice Reader (rc1)
+
+A small CPU model that hears how a user *sounds*: tone of voice, sarcasm in the delivery, energy, laughter and
+speech rate. It reads the last 8 s of each user turn, once, when the user stops speaking, and Pipecat's
+`VoiceReaderProcessor` writes the reading as RCS perception at the head of the user's message, beside the text
+tone reader, which reads the words:
+
+```
+([emotion happy=0.62, neutral=0.30],[sarcasm value=0.08],[tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great.
+```
+
+The voice reader never writes `emotion` or `sarcasm`; it declares its own signals and the LLM weighs them with
+the words. Its best use is the case the text reader cannot see: words that read sincere, said in a voice that
+isn't.
+
+## Variants
+
+This release carries two sizes of the same model, one folder each, read by the same runtime:
+
+| Folder | `variant` | Params | int8 ONNX | Read time, p50 (M1 Max, 4 / 2 / 1 threads) | 1 thread, x86 server | Choose it for |
+|---|---|---|---|---|---|---|
+| `voice-reader-base/` | `"base"` (default) | 26M | 28.6 MB | 33 / 56 / 100 ms | ~100–150 ms | the best tone and sadness reading; adds the `sad` output |
+| `voice-reader-tiny/` | `"tiny"` | 12.5M | 13.3 MB | 16 / 26 / 47 ms | ~45 ms | one-vCPU deployments: sarcasm about as well, half the CPU |
+
+The read starts when the VAD says the user stopped and runs while the speech-to-text finalises, so it rarely
+delays the transcription; the cost to watch is CPU at the end of each turn, beside the VAD, the turn detector and
+the reply. Give `base` two threads where the machine allows.
+
+Each folder has `model.int8.onnx` (the default), `model.onnx` (fp32), `config.json` (calibration, threshold,
+output names) and `MANIFEST.json` (SHA-256 of every file, checked by the runtime on download).
+
+## Use
+
+```python
+from pipecat_voice.reader import VoiceReaderParams, VoiceReaderProcessor
+
+voice_reader = VoiceReaderProcessor(params=VoiceReaderParams(revision="rc1", variant="base", threads=2))
+
+pipeline = Pipeline([transport.input(), stt, tone_reader, voice_reader, user_aggregator, llm, tts, transport.output()])
+```
+
+It goes after the STT (it needs the transcriptions, and the STT passes the audio on) and after the text reader,
+whose signal declarations it merges with its own. The runtime is numpy and onnxruntime; the model downloads from
+this repository at the pinned revision the first time.
+
+## Outputs
+
+| Signal | Published to the LLM | Meaning |
+|---|---|---|
+| `tone` | yes, top two | how the voice sounds: positive, neutral, sad, frustrated (temperature-calibrated) |
+| `sarcasm_voice` | yes | the chance the delivery alone sounds sarcastic, Platt-calibrated on real conversation; below the release threshold (0.068) it is published under 0.5, as a guess; dropped under laughter |
+| `arousal` | yes | vocal energy, 0–1 |
+| `laughter` | yes | 0–1 |
+| `speech_rate` | yes | syllables per second |
+| `upset` | no, raw reading only | how clearly the speaker is upset, 0–1 |
+| `sad` (`base` only) | no, raw reading only | how likely listeners would hear sadness, 0–1 |
+
+Nothing is written for a turn with under 0.8 s of speech. The raw reading, including `upset` and `sad`, reaches
+the application through the processor's `on_reading` event.
+
+## Evaluation
+
+Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5) hearing
+each clip; a turn counts as sarcastic only when both say so (judge agreement on sarcasm: kappa 0.42). Test
+speakers never appear in training.
+
+| | `base` | `tiny` |
+|---|---|---|
+| **Sarcasm**, everyday conversation (4,366 turns, 203 sarcastic; Open Yap, otoSpeech, HarperValley), AUROC | 0.917 | 0.904 |
+| — at the release threshold: F1 / recall / false alarms on sincere turns | 0.575 / 0.50 / 1.3 % | 0.560 / 0.48 / 1.3 % |
+| — mean AUROC over four test sets (two conversational, Emolia, MUStARD++) | 0.791 | 0.788 |
+| — MUStARD++ (TV, audio only, evaluation only), AUROC | 0.656 | 0.654 |
+| **Tone** macro-F1: conversation / Emolia / CREMA-D (acted) / EmoNet-Voice (synthetic) | 0.706 / 0.661 / 0.887 / 0.563 | 0.712 / 0.627 / 0.872 / 0.457 |
+| **Sadness**: tone recall for sad, mean of four sets | 0.53 | 0.38 |
+| — `sad` output, AUROC: conversation (any judge heard sadness) / Emolia | 0.762 / 0.929 | – |
+| **Upset**, mean AUROC (three sets) | 0.867 | 0.856 |
+| **Arousal**, CCC against the training target | 0.89 | 0.88 |
+| **Laughter**, AUROC / F1 | 0.994 / 0.72 | 0.991 / 0.70 |
+| int8 vs fp32: tone agreement / sarcasm p95 \|Δp\| | 0.983 / 0.011 | 0.977 / 0.010 |
+
+## Limitations
+
+- **Sarcasm from the voice alone is a weak cue.** At the threshold it catches about half of the sarcasm both
+  judges heard, at about 1 false alarm in 75 sincere turns. Weigh it with the words, never alone. On synthetic
+  (TTS) sarcasm it is near chance (AUROC 0.60 on our 20 demo pairs).
+- **Sadness in conversation is rare** (2–6 % of turns), so even a good detector's per-turn flags are mostly
+  wrong (12–25 % precision at a 5 % false-alarm rate). Treat `sad` as a graded cue or average it over a
+  conversation. It is not a measure of depression or of anyone's mental health, and a single turn cannot show
+  either.
+- **Acted, theatrical sadness** is under-read: 23 of 34 CREMA-D sad clips.
+- English only; trained on conversational and podcast speech, 8 kHz telephone audio included.
+- Tone labels are LLM judgments of the audio, not human ratings.
+
+## Training
+
+- **Encoder**: Whisper encoder initialised from LAION's affect-tuned
+  [`laion/whisper-tiny-base-emotion-voice-burst`](https://huggingface.co/laion/whisper-tiny-base-emotion-voice-burst)
+  (CC BY 4.0), base and tiny encoders respectively, cut to an 8 s window, with masked statistics pooling and one
+  small head per output.
+- **Data** (about 119k training windows): Open Yap 1K and otoSpeech conversations, HarperValleyBank calls
+  (CC BY 4.0), the English YODAS part of Emilia from `laion/Emolia` (CC BY 4.0), and CREMA-D (ODbL / DbCL).
+  Augmented with gain, noise, telephone band-limiting and codecs; never speed or pitch, since rate and pitch are
+  the signal.
+- **Labels**: the two judges' verdicts on sarcasm, tone and upset; where they disagreed on tone, the mean of two
+  audio emotion models (emotion2vec+ large and the LAION model) at half weight; the `sad` head on the share of
+  judges who heard sadness.
+- **Tiny**: distilled from the base model's predictions at half weight, with base-labelled Emolia clips added.
+- **Calibration** on real conversational dev turns: temperature for tone, Platt for sarcasm and laughter; the
+  sarcasm threshold keeps false alarms at 2 % of sincere dev turns.
+
+Evaluation-only data, never trained on: MUStARD++ (CC BY-NC) and EmoNet-Voice Bench's clearly rated clips.

voice-reader-base/MANIFEST.json

@@ -0,0 +1,47 @@
+{
+ "release": "rc1",
+ "folder": "voice-reader-base",
+ "run": "v3-base-sad",
+ "staged": "2026-11-02T20:40:58+00:00",
+ "files": {
+  "model.onnx": {
+   "sha256": "99975890aa7f1b0648af82a480233fdf641cee04bfee84370d0bed9c39d63313",
+   "bytes": 109728612
+  },
+  "model.int8.onnx": {
+   "sha256": "5c25605ea019521853c92347b48ced6f7706c14f21183f5de9da2a607c9fae95",
+   "bytes": 28567980
+  },
+  "config.json": {
+   "sha256": "72ed694b3ca24b368baa0d68e47fa90ccf271bf0d74895bff2a20aefb53cd165",
+   "bytes": 658
+  }
+ },
+ "parity_int8": {
+  "windows": 2000,
+  "tone_argmax_agreement": 0.983,
+  "tone_max_abs_dp": 0.14630481600761414,
+  "sarcasm_p95_abs_dp": 0.011146726086735725,
+  "sarcasm_max_abs_dp": 0.053686708211898804,
+  "laughter_p95_abs_dp": 0.0013878615573048592,
+  "arousal_p95_abs_d": 0.026246830821037292,
+  "speech_rate_p95_rel_d": 0.04056451842188835,
+  "auroc_sarcasm": {
+   "reference": 0.8036348918022429,
+   "other": 0.8078700047385879
+  },
+  "auroc_sarcasm_both": {
+   "reference": 0.8481948120698782,
+   "other": 0.8499735309687665
+  },
+  "auroc_sarcasm_voice_only": {
+   "reference": 0.9116690614837782,
+   "other": 0.9187022612115253
+  },
+  "tone_macro_f1": {
+   "reference": 0.5146829335401363,
+   "other": 0.49449348337582855,
+   "rows": 1546
+  }
+ }
+}

voice-reader-base/config.json

@@ -0,0 +1,34 @@
+{
+ "window_secs": 8.0,
+ "tones": [
+  "positive",
+  "neutral",
+  "sad",
+  "frustrated"
+ ],
+ "outputs": [
+  "tone",
+  "sarcasm",
+  "laughter",
+  "arousal",
+  "speech_rate",
+  "upset",
+  "sad"
+ ],
+ "sarcasm_tau": 0.06803736669700389,
+ "calibration": {
+  "sarcasm": {
+   "a": 1.678951463802178,
+   "b": 0.12539543584903914
+  },
+  "laughter": {
+   "a": 0.8057154083813632,
+   "b": -0.09714794868282932
+  },
+  "tone_temperature": 0.8488899224103036
+ },
+ "run": "v3-base-sad",
+ "init": "laion-base",
+ "best_epoch": 7,
+ "note": "sarcasm_tau is the dev 2 % FPR threshold on the Platt-calibrated probability; tone temperature and laughter Platt curve fitted on dev"
+}

voice-reader-base/model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c25605ea019521853c92347b48ced6f7706c14f21183f5de9da2a607c9fae95
+size 28567980

voice-reader-base/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:99975890aa7f1b0648af82a480233fdf641cee04bfee84370d0bed9c39d63313
+size 109728612

voice-reader-tiny/MANIFEST.json

@@ -0,0 +1,47 @@
+{
+ "release": "rc1",
+ "folder": "voice-reader-tiny",
+ "run": "student-tiny-v3",
+ "staged": "2026-11-02T20:40:59+00:00",
+ "files": {
+  "model.onnx": {
+   "sha256": "873b5e5991f607483331ca7078b175dadaee3a1ec2046507ae65f008db64cad0",
+   "bytes": 50202707
+  },
+  "model.int8.onnx": {
+   "sha256": "1823e7d7e861586f3981ffa470150a5c3653f5d017df97a49e606ccc4e6c6c42",
+   "bytes": 13332966
+  },
+  "config.json": {
+   "sha256": "79582d3816d963a092dd4d71da61d4444b3127659f846076223d20f240fc821d",
+   "bytes": 655
+  }
+ },
+ "parity_int8": {
+  "windows": 2000,
+  "tone_argmax_agreement": 0.977,
+  "tone_max_abs_dp": 0.2388080656528473,
+  "sarcasm_p95_abs_dp": 0.010020742192864418,
+  "sarcasm_max_abs_dp": 0.04528002440929413,
+  "laughter_p95_abs_dp": 0.0021592299453914165,
+  "arousal_p95_abs_d": 0.019707761704921722,
+  "speech_rate_p95_rel_d": 0.03602607175707817,
+  "auroc_sarcasm": {
+   "reference": 0.7930224293160638,
+   "other": 0.7871979150213236
+  },
+  "auroc_sarcasm_both": {
+   "reference": 0.8221916357861302,
+   "other": 0.819777660137639
+  },
+  "auroc_sarcasm_voice_only": {
+   "reference": 0.8642516826741284,
+   "other": 0.8534371927701732
+  },
+  "tone_macro_f1": {
+   "reference": 0.45427391583654375,
+   "other": 0.46146068669445456,
+   "rows": 1546
+  }
+ }
+}

voice-reader-tiny/config.json

@@ -0,0 +1,33 @@
+{
+ "window_secs": 8.0,
+ "tones": [
+  "positive",
+  "neutral",
+  "sad",
+  "frustrated"
+ ],
+ "outputs": [
+  "tone",
+  "sarcasm",
+  "laughter",
+  "arousal",
+  "speech_rate",
+  "upset"
+ ],
+ "sarcasm_tau": 0.06590602517797733,
+ "calibration": {
+  "sarcasm": {
+   "a": 1.7731141168712765,
+   "b": 0.06321434642450725
+  },
+  "laughter": {
+   "a": 0.8303770152621825,
+   "b": -0.17392277961573782
+  },
+  "tone_temperature": 0.7673753909755566
+ },
+ "run": "student-tiny-v3",
+ "init": "laion-tiny",
+ "best_epoch": 10,
+ "note": "sarcasm_tau is the dev 2 % FPR threshold on the Platt-calibrated probability; tone temperature and laughter Platt curve fitted on dev"
+}

voice-reader-tiny/model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1823e7d7e861586f3981ffa470150a5c3653f5d017df97a49e606ccc4e6c6c42
+size 13332966

voice-reader-tiny/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:873b5e5991f607483331ca7078b175dadaee3a1ec2046507ae65f008db64cad0
+size 50202707