pipecat-ai/voice-reader

Pipecat Voice Reader (rc1)

⚠️ Experimental. This is a first release candidate, published for trying out and feedback. Its outputs, names and calibration may change between releases, and its readings are cues for an LLM to weigh, not verdicts to act on alone. Pin a revision, and read the limitations below before relying on it.

A small CPU model that hears how a user sounds: tone of voice, sarcasm in the delivery, energy, laughter and speech rate. It is the voice half of a pair: it is made to run beside Pipecat's Tone Reader (PipecatToneReader), which reads the words. It reads the last 8 s of each user turn, once, when the user stops speaking, and Pipecat's VoiceReaderProcessor writes the reading as RCS perception at the head of the user's message, after the Tone Reader's:

([emotion happy=0.62, neutral=0.30],[sarcasm value=0.08],[tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great.

Made to pair with the Tone Reader

The two readers split the work, and each declares its own signals so the LLM sees both side by side:

	Tone Reader (the words)	Voice Reader (the voice)
Reads	the final transcript, with the bot's previous line	the last 8 s of the user's audio
Writes	emotion (nine labels), sarcasm	tone (four labels), sarcasm_voice, arousal, laughter, speech_rate
Strong on	what the words say: emotion, and sarcasm the words give away	what the words don't: a flat or frustrated delivery, sadness behind neutral words, sarcasm said with sincere words

The voice reader never writes emotion or sarcasm. Its sarcasm_voice is a weak cue on its own, and its declared description tells the LLM to weigh it with the Tone Reader's sarcasm: the case it earns its place in is words that read sincere, said in a voice that isn't. For example, from our test clips, "He's the greatest, I'd say." said sarcastically reads as happy and 2 % sarcastic from the words, and 15 % (above the 6.8 % threshold) from the voice.

It does run without the Tone Reader: the LLM then gets tone instead of emotion, and no sarcasm from the words to weigh sarcasm_voice against, so we recommend running both.

Variants

This release carries two sizes of the same model, one folder each, read by the same runtime:

Folder	variant	Params	int8 ONNX	Read time, p50 (M1 Max, 4 / 2 / 1 threads)	1 thread, x86 server	Choose it for
voice-reader-base/	"base" (default)	26M	28.6 MB	33 / 56 / 100 ms	~100–150 ms	the best tone and sadness reading; adds the sad output
voice-reader-tiny/	"tiny"	12.5M	13.3 MB	16 / 26 / 47 ms	~45 ms	one-vCPU deployments: sarcasm about as well, half the CPU

The read starts when the VAD says the user stopped and runs while the speech-to-text finalises, so it rarely delays the transcription; the cost to watch is CPU at the end of each turn, beside the VAD, the turn detector and the reply. Give base two threads where the machine allows.

Each folder has model.int8.onnx (the default), model.onnx (fp32), config.json (calibration, threshold, output names) and MANIFEST.json (SHA-256 of every file, checked by the runtime on download).

Use

With the Tone Reader (pipecat-ai[tone]) before it, as intended:

from pipecat.processors.tone.reader import PipecatToneReader
from pipecat_voice.reader import VoiceReaderParams, VoiceReaderProcessor

pipeline = Pipeline([
    transport.input(),
    stt,
    PipecatToneReader(),  # the words: emotion, sarcasm
    VoiceReaderProcessor(params=VoiceReaderParams(revision="rc1", variant="base", threads=2)),  # the voice
    user_aggregator,
    llm,
    tts,
    transport.output(),
])

It goes after the STT (it needs the transcriptions, and the STT passes the audio on) and after the Tone Reader, whose signal declarations it merges with its own, so an LLM that keeps only the latest declaration still sees both. The runtime is numpy and onnxruntime; the model downloads from this repository at the pinned revision the first time.

Outputs

Signal	Published to the LLM	Meaning
tone	yes, top two	how the voice sounds: positive, neutral, sad, frustrated (temperature-calibrated)
sarcasm_voice	yes	the chance the delivery alone sounds sarcastic, Platt-calibrated on real conversation; below the release threshold (0.068) it is published under 0.5, as a guess; dropped under laughter
arousal	yes	vocal energy, 0–1
laughter	yes	0–1
speech_rate	yes	syllables per second
upset	no, raw reading only	how clearly the speaker is upset, 0–1
sad (base only)	no, raw reading only	how likely listeners would hear sadness, 0–1

Nothing is written for a turn with under 0.8 s of speech. The raw reading, including upset and sad, reaches the application through the processor's on_reading event.

Evaluation

Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5) hearing each clip; a turn counts as sarcastic only when both say so (judge agreement on sarcasm: kappa 0.42). Test speakers never appear in training.

	base	tiny
Sarcasm, everyday conversation (4,366 turns, 203 sarcastic; Open Yap, otoSpeech, HarperValley), AUROC	0.917	0.904
— at the release threshold: F1 / recall / false alarms on sincere turns	0.575 / 0.50 / 1.3 %	0.560 / 0.48 / 1.3 %
— mean AUROC over four test sets (two conversational, Emolia, MUStARD++)	0.791	0.788
— MUStARD++ (TV, audio only, evaluation only), AUROC	0.656	0.654
Tone macro-F1: conversation / Emolia / CREMA-D (acted) / EmoNet-Voice (synthetic)	0.706 / 0.661 / 0.887 / 0.563	0.712 / 0.627 / 0.872 / 0.457
Sadness: tone recall for sad, mean of four sets	0.53	0.38
— sad output, AUROC: conversation (any judge heard sadness) / Emolia	0.762 / 0.929	–
Upset, mean AUROC (three sets)	0.867	0.856
Arousal, CCC against the training target	0.89	0.88
Laughter, AUROC / F1	0.994 / 0.72	0.991 / 0.70
int8 vs fp32: tone agreement / sarcasm p95 |Δp|	0.983 / 0.011	0.977 / 0.010

Limitations

• Sarcasm from the voice alone is a weak cue. At the threshold it catches about half of the sarcasm both judges heard, at about 1 false alarm in 75 sincere turns. Weigh it with the words, never alone. On synthetic (TTS) sarcasm it is near chance (AUROC 0.60 on our 20 demo pairs).
• Sadness in conversation is rare (2–6 % of turns), so even a good detector's per-turn flags are mostly wrong (12–25 % precision at a 5 % false-alarm rate). Treat sad as a graded cue or average it over a conversation. It is not a measure of depression or of anyone's mental health, and a single turn cannot show either.
• Acted, theatrical sadness is under-read: 23 of 34 CREMA-D sad clips.
• English only; trained on conversational and podcast speech, 8 kHz telephone audio included.
• Tone labels are LLM judgments of the audio, not human ratings.

Training

• Encoder: Whisper encoder initialised from LAION's affect-tuned laion/whisper-tiny-base-emotion-voice-burst (CC BY 4.0), base and tiny encoders respectively, cut to an 8 s window, with masked statistics pooling and one small head per output.
• Data (about 119k training windows): Open Yap 1K and otoSpeech conversations, HarperValleyBank calls (CC BY 4.0), the English YODAS part of Emilia from laion/Emolia (CC BY 4.0), and CREMA-D (ODbL / DbCL). Augmented with gain, noise, telephone band-limiting and codecs; never speed or pitch, since rate and pitch are the signal.
• Labels: the two judges' verdicts on sarcasm, tone and upset; where they disagreed on tone, the mean of two audio emotion models (emotion2vec+ large and the LAION model) at half weight; the sad head on the share of judges who heard sadness.
• Tiny: distilled from the base model's predictions at half weight, with base-labelled Emolia clips added.
• Calibration on real conversational dev turns: temperature for tone, Platt for sarcasm and laughter; the sarcasm threshold keeps false alarms at 2 % of sincere dev turns.

Evaluation-only data, never trained on: MUStARD++ (CC BY-NC) and EmoNet-Voice Bench's clearly rated clips.