Release rc2
Browse files- README.md +66 -33
- config.json +64 -0
- voice-reader-base/MANIFEST.json +22 -22
- voice-reader-base/config.json +8 -8
- voice-reader-base/model.int8.onnx +1 -1
- voice-reader-base/model.onnx +1 -1
- voice-reader-tiny/MANIFEST.json +24 -24
- voice-reader-tiny/config.json +10 -9
- voice-reader-tiny/model.int8.onnx +2 -2
- voice-reader-tiny/model.onnx +2 -2
README.md
CHANGED
|
@@ -12,11 +12,11 @@ tags:
|
|
| 12 |
- onnx
|
| 13 |
---
|
| 14 |
|
| 15 |
-
# Pipecat Voice Reader (
|
| 16 |
|
| 17 |
-
> **⚠️ Experimental.** This is a
|
| 18 |
-
>
|
| 19 |
-
>
|
| 20 |
|
| 21 |
A small CPU model that hears how a user *sounds*: tone of voice, sarcasm in the delivery, energy, laughter and
|
| 22 |
speech rate. It is the voice half of a pair: it is made to run beside Pipecat's
|
|
@@ -28,6 +28,24 @@ writes the reading as RCS perception at the head of the user's message, after th
|
|
| 28 |
([emotion happy=0.62, neutral=0.30],[sarcasm value=0.08],[tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great.
|
| 29 |
```
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
## Made to pair with the Tone Reader
|
| 32 |
|
| 33 |
The two readers split the work, and each declares its own signals so the LLM sees both side by side:
|
|
@@ -41,7 +59,7 @@ The two readers split the work, and each declares its own signals so the LLM see
|
|
| 41 |
The voice reader never writes `emotion` or `sarcasm`. Its `sarcasm_voice` is a weak cue on its own, and its
|
| 42 |
declared description tells the LLM to weigh it with the Tone Reader's `sarcasm`: the case it earns its place in is
|
| 43 |
words that read sincere, said in a voice that isn't. For example, from our test clips, "He's the greatest, I'd
|
| 44 |
-
say." said sarcastically reads as happy and 2 % sarcastic from the words, and
|
| 45 |
from the voice.
|
| 46 |
|
| 47 |
It does run without the Tone Reader: the LLM then gets `tone` instead of `emotion`, and no `sarcasm` from the
|
|
@@ -53,15 +71,16 @@ This release carries two sizes of the same model, one folder each, read by the s
|
|
| 53 |
|
| 54 |
| Folder | `variant` | Params | int8 ONNX | Read time, p50 (M1 Max, 4 / 2 / 1 threads) | 1 thread, x86 server | Choose it for |
|
| 55 |
|---|---|---|---|---|---|---|
|
| 56 |
-
| `voice-reader-base/` | `"base"` (default) | 26M | 28.6 MB |
|
| 57 |
-
| `voice-reader-tiny/` | `"tiny"` | 12.5M |
|
| 58 |
|
| 59 |
The read starts when the VAD says the user stopped and runs while the speech-to-text finalises, so it rarely
|
| 60 |
delays the transcription; the cost to watch is CPU at the end of each turn, beside the VAD, the turn detector and
|
| 61 |
the reply. Give `base` two threads where the machine allows.
|
| 62 |
|
| 63 |
Each folder has `model.int8.onnx` (the default), `model.onnx` (fp32), `config.json` (calibration, threshold,
|
| 64 |
-
output names) and `MANIFEST.json` (SHA-256 of every file, checked by the runtime on download).
|
|
|
|
| 65 |
|
| 66 |
## Use
|
| 67 |
|
|
@@ -75,7 +94,7 @@ pipeline = Pipeline([
|
|
| 75 |
transport.input(),
|
| 76 |
stt,
|
| 77 |
PipecatToneReader(), # the words: emotion, sarcasm
|
| 78 |
-
VoiceReaderProcessor(params=VoiceReaderParams(revision="
|
| 79 |
user_aggregator,
|
| 80 |
llm,
|
| 81 |
tts,
|
|
@@ -93,16 +112,23 @@ this repository at the pinned revision the first time.
|
|
| 93 |
| Signal | Published to the LLM | Meaning |
|
| 94 |
|---|---|---|
|
| 95 |
| `tone` | yes, top two | how the voice sounds: positive, neutral, sad, frustrated (temperature-calibrated) |
|
| 96 |
-
| `sarcasm_voice` | yes | the chance the delivery alone sounds sarcastic, Platt-calibrated on real conversation; below the release threshold (0.
|
| 97 |
| `arousal` | yes | vocal energy, 0–1 |
|
| 98 |
| `laughter` | yes | 0–1 |
|
| 99 |
| `speech_rate` | yes | syllables per second |
|
| 100 |
| `upset` | no, raw reading only | how clearly the speaker is upset, 0–1 |
|
| 101 |
-
| `sad`
|
| 102 |
|
| 103 |
Nothing is written for a turn with under 0.8 s of speech. The raw reading, including `upset` and `sad`, reaches
|
| 104 |
the application through the processor's `on_reading` event.
|
| 105 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
## Evaluation
|
| 107 |
|
| 108 |
Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5) hearing
|
|
@@ -111,28 +137,33 @@ speakers never appear in training.
|
|
| 111 |
|
| 112 |
| | `base` | `tiny` |
|
| 113 |
|---|---|---|
|
| 114 |
-
| **Sarcasm**, everyday conversation (
|
| 115 |
-
| — at the release threshold: F1 / recall / false alarms on sincere turns | 0.
|
| 116 |
-
| —
|
| 117 |
-
| —
|
| 118 |
-
|
|
| 119 |
-
| **
|
| 120 |
-
|
|
| 121 |
-
|
|
|
|
|
| 122 |
| **Arousal**, CCC against the training target | 0.89 | 0.88 |
|
| 123 |
-
| **Laughter**, AUROC / F1 | 0.
|
| 124 |
-
| int8 vs fp32: tone agreement / sarcasm p95 \|Δp\| | 0.983 / 0.
|
| 125 |
|
| 126 |
## Limitations
|
| 127 |
|
| 128 |
- **Sarcasm from the voice alone is a weak cue.** At the threshold it catches about half of the sarcasm both
|
| 129 |
-
judges heard, at about 1 false alarm in
|
| 130 |
-
(TTS) sarcasm it is near chance (AUROC 0.
|
| 131 |
- **Sadness in conversation is rare** (2–6 % of turns), so even a good detector's per-turn flags are mostly
|
| 132 |
-
wrong (12–
|
| 133 |
conversation. It is not a measure of depression or of anyone's mental health, and a single turn cannot show
|
| 134 |
either.
|
| 135 |
-
- **
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
- English only; trained on conversational and podcast speech, 8 kHz telephone audio included.
|
| 137 |
- Tone labels are LLM judgments of the audio, not human ratings.
|
| 138 |
|
|
@@ -142,14 +173,16 @@ speakers never appear in training.
|
|
| 142 |
[`laion/whisper-tiny-base-emotion-voice-burst`](https://huggingface.co/laion/whisper-tiny-base-emotion-voice-burst)
|
| 143 |
(CC BY 4.0), base and tiny encoders respectively, cut to an 8 s window, with masked statistics pooling and one
|
| 144 |
small head per output.
|
| 145 |
-
- **Data** (about
|
| 146 |
-
(CC BY 4.0), the English YODAS part of Emilia from `laion/Emolia`
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
judges who heard sadness.
|
| 152 |
-
|
|
|
|
|
|
|
| 153 |
- **Calibration** on real conversational dev turns: temperature for tone, Platt for sarcasm and laughter; the
|
| 154 |
sarcasm threshold keeps false alarms at 2 % of sincere dev turns.
|
| 155 |
|
|
|
|
| 12 |
- onnx
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# Pipecat Voice Reader (rc2)
|
| 16 |
|
| 17 |
+
> **⚠️ Experimental.** This is a release candidate, published for trying out and feedback. Its outputs, names and
|
| 18 |
+
> calibration may change between releases, and its readings are cues for an LLM to weigh, not verdicts to act on
|
| 19 |
+
> alone. Pin a revision, and read the limitations below before relying on it.
|
| 20 |
|
| 21 |
A small CPU model that hears how a user *sounds*: tone of voice, sarcasm in the delivery, energy, laughter and
|
| 22 |
speech rate. It is the voice half of a pair: it is made to run beside Pipecat's
|
|
|
|
| 28 |
([emotion happy=0.62, neutral=0.30],[sarcasm value=0.08],[tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great.
|
| 29 |
```
|
| 30 |
|
| 31 |
+
## What changed since rc1
|
| 32 |
+
|
| 33 |
+
- **Licence-clean targets.** Every training target in rc2 is a judge's verdict or vote share, a cue from LAION's
|
| 34 |
+
CC BY 4.0 model, or a measurement of the audio. rc1 trained its tone head, where the two judges disagreed, on the
|
| 35 |
+
mean of two third-party audio emotion models; rc2 uses their vote share instead and has no such model in its
|
| 36 |
+
targets, so its lineage is Whisper (MIT), LAION's affect model (CC BY 4.0), the corpora under Training and the
|
| 37 |
+
judges. One thread remains: the pseudo-labelled windows below carry the rc1 base model's readings, so rc1's
|
| 38 |
+
lineage reaches rc2 second-hand through those labels.
|
| 39 |
+
- **More judged conversation.** A second round of about 24,000 conversational turns rated by both judges went into
|
| 40 |
+
training, and its held-out turns are a new test set (below).
|
| 41 |
+
- **Pseudo-labelled speech.** 159,000 windows read by the rc1 base model (Emolia clips no judge has rated, and
|
| 42 |
+
conversation turns) joined the training set. The sarcasm and conversation-tone gains come from them.
|
| 43 |
+
- **`tiny` now has the `sad` output** (int8 14.1 MB, from 13.3).
|
| 44 |
+
- Sarcasm in conversation AUROC 0.917 → 0.921 (base) and 0.904 → 0.908 (tiny); conversation tone macro-F1
|
| 45 |
+
0.706 → 0.754 and 0.712 → 0.739; acted sadness read 23 → 30 of 34 CREMA-D clips (base). Tone on EmoNet-Voice's
|
| 46 |
+
synthetic clips slips for base (0.563 → 0.510).
|
| 47 |
+
- The tone head leans neutral a little more on borderline turns than rc1 did: see `tone_bias` under Outputs.
|
| 48 |
+
|
| 49 |
## Made to pair with the Tone Reader
|
| 50 |
|
| 51 |
The two readers split the work, and each declares its own signals so the LLM sees both side by side:
|
|
|
|
| 59 |
The voice reader never writes `emotion` or `sarcasm`. Its `sarcasm_voice` is a weak cue on its own, and its
|
| 60 |
declared description tells the LLM to weigh it with the Tone Reader's `sarcasm`: the case it earns its place in is
|
| 61 |
words that read sincere, said in a voice that isn't. For example, from our test clips, "He's the greatest, I'd
|
| 62 |
+
say." said sarcastically reads as happy and 2 % sarcastic from the words, and 10 % (above the 6.1 % threshold)
|
| 63 |
from the voice.
|
| 64 |
|
| 65 |
It does run without the Tone Reader: the LLM then gets `tone` instead of `emotion`, and no `sarcasm` from the
|
|
|
|
| 71 |
|
| 72 |
| Folder | `variant` | Params | int8 ONNX | Read time, p50 (M1 Max, 4 / 2 / 1 threads) | 1 thread, x86 server | Choose it for |
|
| 73 |
|---|---|---|---|---|---|---|
|
| 74 |
+
| `voice-reader-base/` | `"base"` (default) | 26M | 28.6 MB | 28 / 49 / 92 ms | ~90–160 ms | the best tone and sadness reading |
|
| 75 |
+
| `voice-reader-tiny/` | `"tiny"` | 12.5M | 14.1 MB | 13 / 23 / 42 ms | ~45–70 ms | one-vCPU deployments: sarcasm about as well, half the CPU |
|
| 76 |
|
| 77 |
The read starts when the VAD says the user stopped and runs while the speech-to-text finalises, so it rarely
|
| 78 |
delays the transcription; the cost to watch is CPU at the end of each turn, beside the VAD, the turn detector and
|
| 79 |
the reply. Give `base` two threads where the machine allows.
|
| 80 |
|
| 81 |
Each folder has `model.int8.onnx` (the default), `model.onnx` (fp32), `config.json` (calibration, threshold,
|
| 82 |
+
output names) and `MANIFEST.json` (SHA-256 of every file, checked by the runtime on download). The `config.json` at
|
| 83 |
+
the root is the release index: the variants, their folders and files. The runtime reads it first on every load.
|
| 84 |
|
| 85 |
## Use
|
| 86 |
|
|
|
|
| 94 |
transport.input(),
|
| 95 |
stt,
|
| 96 |
PipecatToneReader(), # the words: emotion, sarcasm
|
| 97 |
+
VoiceReaderProcessor(params=VoiceReaderParams(revision="rc2", variant="base", threads=2)), # the voice
|
| 98 |
user_aggregator,
|
| 99 |
llm,
|
| 100 |
tts,
|
|
|
|
| 112 |
| Signal | Published to the LLM | Meaning |
|
| 113 |
|---|---|---|
|
| 114 |
| `tone` | yes, top two | how the voice sounds: positive, neutral, sad, frustrated (temperature-calibrated) |
|
| 115 |
+
| `sarcasm_voice` | yes | the chance the delivery alone sounds sarcastic, Platt-calibrated on real conversation; below the release threshold (0.061 for `base`, 0.063 for `tiny`) it is published under 0.5, as a guess; dropped under laughter |
|
| 116 |
| `arousal` | yes | vocal energy, 0–1 |
|
| 117 |
| `laughter` | yes | 0–1 |
|
| 118 |
| `speech_rate` | yes | syllables per second |
|
| 119 |
| `upset` | no, raw reading only | how clearly the speaker is upset, 0–1 |
|
| 120 |
+
| `sad` | no, raw reading only | how likely listeners would hear sadness, 0–1 |
|
| 121 |
|
| 122 |
Nothing is written for a turn with under 0.8 s of speech. The raw reading, including `upset` and `sad`, reaches
|
| 123 |
the application through the processor's `on_reading` event.
|
| 124 |
|
| 125 |
+
**`tone_bias`.** `tone` is calibrated on real conversation, where most turns are neutral, so on a borderline turn
|
| 126 |
+
it leans neutral: a sad or frustrated delivery a listener would notice can come out as neutral 0.45, sad 0.40.
|
| 127 |
+
`VoiceReaderParams(tone_bias={"sad": 0.5, "frustrated": 0.5})` adds to those tones' logits before the softmax
|
| 128 |
+
(+0.5 multiplies a tone's odds by about 1.65), so they are published more readily; our test harness runs with
|
| 129 |
+
exactly that. The graded `sad` and `upset` readings are not affected by it. Leave it unset to read the calibrated
|
| 130 |
+
probabilities.
|
| 131 |
+
|
| 132 |
## Evaluation
|
| 133 |
|
| 134 |
Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5) hearing
|
|
|
|
| 137 |
|
| 138 |
| | `base` | `tiny` |
|
| 139 |
|---|---|---|
|
| 140 |
+
| **Sarcasm**, everyday conversation (3,945 turns both judges rated, 203 sarcastic; Open Yap, otoSpeech, HarperValley), AUROC | 0.921 | 0.908 |
|
| 141 |
+
| — at the release threshold: F1 / recall / false alarms on sincere turns | 0.571 / 0.47 / 1.0 % | 0.548 / 0.45 / 1.1 % |
|
| 142 |
+
| — second conversational test, held out from the rc2 judging round (5,037 turns, 272 sarcastic), AUROC | 0.862 | 0.853 |
|
| 143 |
+
| — mean AUROC over four test sets (two conversational, Emolia, MUStARD++) | 0.798 | 0.797 |
|
| 144 |
+
| — MUStARD++ (TV, audio only, evaluation only), AUROC | 0.669 | 0.662 |
|
| 145 |
+
| **Tone** macro-F1: conversation / Emolia / CREMA-D (acted) / EmoNet-Voice (synthetic) | 0.754 / 0.667 / 0.902 / 0.510 | 0.739 / 0.647 / 0.873 / 0.477 |
|
| 146 |
+
| **Sadness**: tone recall for sad, mean of four sets | 0.54 | 0.45 |
|
| 147 |
+
| — `sad` output, AUROC: conversation (any judge heard sadness) / Emolia | 0.755 / 0.928 | 0.740 / 0.923 |
|
| 148 |
+
| **Upset**, mean AUROC (three sets) | 0.869 | 0.857 |
|
| 149 |
| **Arousal**, CCC against the training target | 0.89 | 0.88 |
|
| 150 |
+
| **Laughter**, AUROC / F1 | 0.993 / 0.64 | 0.991 / 0.64 |
|
| 151 |
+
| int8 vs fp32: tone agreement / sarcasm p95 \|Δp\| | 0.983 / 0.008 | 0.983 / 0.009 |
|
| 152 |
|
| 153 |
## Limitations
|
| 154 |
|
| 155 |
- **Sarcasm from the voice alone is a weak cue.** At the threshold it catches about half of the sarcasm both
|
| 156 |
+
judges heard, at about 1 false alarm in 100 sincere turns. Weigh it with the words, never alone. On synthetic
|
| 157 |
+
(TTS) sarcasm it is near chance (AUROC 0.57 on our 20 demo pairs).
|
| 158 |
- **Sadness in conversation is rare** (2–6 % of turns), so even a good detector's per-turn flags are mostly
|
| 159 |
+
wrong (12–24 % precision at a 5 % false-alarm rate). Treat `sad` as a graded cue or average it over a
|
| 160 |
conversation. It is not a measure of depression or of anyone's mental health, and a single turn cannot show
|
| 161 |
either.
|
| 162 |
+
- **Borderline tones lean neutral.** Calibrated on conversation, the tone head publishes neutral on turns a
|
| 163 |
+
listener might call sad or frustrated; `tone_bias` above is the knob. Acted sadness is read well (30 of 34
|
| 164 |
+
CREMA-D sad clips by `base`, 27 by `tiny`; the misses read as frustrated).
|
| 165 |
+
- **Laughter** ranks well but its published value is conservative: F1 0.64 at the calibrated 0.5 (rc1: 0.72), at
|
| 166 |
+
the same AUROC.
|
| 167 |
- English only; trained on conversational and podcast speech, 8 kHz telephone audio included.
|
| 168 |
- Tone labels are LLM judgments of the audio, not human ratings.
|
| 169 |
|
|
|
|
| 173 |
[`laion/whisper-tiny-base-emotion-voice-burst`](https://huggingface.co/laion/whisper-tiny-base-emotion-voice-burst)
|
| 174 |
(CC BY 4.0), base and tiny encoders respectively, cut to an 8 s window, with masked statistics pooling and one
|
| 175 |
small head per output.
|
| 176 |
+
- **Data** (about 290k training windows): about 131k judged windows from Open Yap 1K and otoSpeech conversations
|
| 177 |
+
(two judging rounds), HarperValleyBank calls (CC BY 4.0), the English YODAS part of Emilia from `laion/Emolia`
|
| 178 |
+
(CC BY 4.0) and CREMA-D (ODbL / DbCL); plus 159k windows pseudo-labelled by the rc1 base model, 112,996 Emolia
|
| 179 |
+
clips no judge has rated and 46,021 conversation turns. Augmented with gain, noise, telephone band-limiting and
|
| 180 |
+
codecs; never speed or pitch, since rate and pitch are the signal.
|
| 181 |
+
- **Labels**: the two judges' verdicts on sarcasm, tone and upset; where they disagreed on tone, their vote share
|
| 182 |
+
over the four tones; the `sad` head on the share of judges who heard sadness. No audio emotion model's output is
|
| 183 |
+
a direct target in rc2. The pseudo-labelled windows carry rc1's readings, and rc1 did use two such models
|
| 184 |
+
(emotion2vec+ large and the LAION model) for its disagreed turns.
|
| 185 |
+
- **Tiny**: distilled from the rc2 base model's predictions at half weight, on the same data.
|
| 186 |
- **Calibration** on real conversational dev turns: temperature for tone, Platt for sarcasm and laughter; the
|
| 187 |
sarcasm threshold keeps false alarms at 2 % of sincere dev turns.
|
| 188 |
|
config.json
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model": "voice-reader",
|
| 3 |
+
"release": "rc2",
|
| 4 |
+
"staged": "2026-11-04T17:14:31+00:00",
|
| 5 |
+
"window_secs": 8.0,
|
| 6 |
+
"sample_rate": 16000,
|
| 7 |
+
"tones": [
|
| 8 |
+
"positive",
|
| 9 |
+
"neutral",
|
| 10 |
+
"sad",
|
| 11 |
+
"frustrated"
|
| 12 |
+
],
|
| 13 |
+
"outputs": [
|
| 14 |
+
"tone",
|
| 15 |
+
"sarcasm",
|
| 16 |
+
"laughter",
|
| 17 |
+
"arousal",
|
| 18 |
+
"speech_rate",
|
| 19 |
+
"upset",
|
| 20 |
+
"sad"
|
| 21 |
+
],
|
| 22 |
+
"variants": {
|
| 23 |
+
"base": {
|
| 24 |
+
"folder": "voice-reader-base",
|
| 25 |
+
"run": "v4-base-all-sad",
|
| 26 |
+
"init": "laion-base",
|
| 27 |
+
"sarcasm_tau": 0.060534259700932624,
|
| 28 |
+
"files": {
|
| 29 |
+
"model.onnx": {
|
| 30 |
+
"sha256": "8a6aa25f4a0ce43d7e1289a15c0dfa894a8848ee1fbba682c38869782d19ded9",
|
| 31 |
+
"bytes": 109728612
|
| 32 |
+
},
|
| 33 |
+
"model.int8.onnx": {
|
| 34 |
+
"sha256": "8dfd1f1e271d4d64b0d001220ffb7ebd28935cef630377ac37fb9ce939de1218",
|
| 35 |
+
"bytes": 28567980
|
| 36 |
+
},
|
| 37 |
+
"config.json": {
|
| 38 |
+
"sha256": "e76664925738674626c87ae92d475a5dcf373d74ce87737ff61b869bef644a84",
|
| 39 |
+
"bytes": 661
|
| 40 |
+
}
|
| 41 |
+
}
|
| 42 |
+
},
|
| 43 |
+
"tiny": {
|
| 44 |
+
"folder": "voice-reader-tiny",
|
| 45 |
+
"run": "student-tiny-v4-sad",
|
| 46 |
+
"init": "laion-tiny",
|
| 47 |
+
"sarcasm_tau": 0.06329724596107583,
|
| 48 |
+
"files": {
|
| 49 |
+
"model.onnx": {
|
| 50 |
+
"sha256": "ad95fc8ae3370f98b91630bc4a12b71aec43d0d652f09b37dfed085e5238a833",
|
| 51 |
+
"bytes": 53366574
|
| 52 |
+
},
|
| 53 |
+
"model.int8.onnx": {
|
| 54 |
+
"sha256": "31d06f9a42656a3144997efa1f085abe29bbcbf20722e5711ba34d86f430dcbf",
|
| 55 |
+
"bytes": 14140364
|
| 56 |
+
},
|
| 57 |
+
"config.json": {
|
| 58 |
+
"sha256": "a32094d77c53f0fda50cd5328d4f4fc937535ffe1bfaefba760cb1771b1fb60d",
|
| 59 |
+
"bytes": 665
|
| 60 |
+
}
|
| 61 |
+
}
|
| 62 |
+
}
|
| 63 |
+
}
|
| 64 |
+
}
|
voice-reader-base/MANIFEST.json
CHANGED
|
@@ -1,46 +1,46 @@
|
|
| 1 |
{
|
| 2 |
-
"release": "
|
| 3 |
"folder": "voice-reader-base",
|
| 4 |
-
"run": "
|
| 5 |
-
"staged": "2026-11-
|
| 6 |
"files": {
|
| 7 |
"model.onnx": {
|
| 8 |
-
"sha256": "
|
| 9 |
"bytes": 109728612
|
| 10 |
},
|
| 11 |
"model.int8.onnx": {
|
| 12 |
-
"sha256": "
|
| 13 |
"bytes": 28567980
|
| 14 |
},
|
| 15 |
"config.json": {
|
| 16 |
-
"sha256": "
|
| 17 |
-
"bytes":
|
| 18 |
}
|
| 19 |
},
|
| 20 |
"parity_int8": {
|
| 21 |
"windows": 2000,
|
| 22 |
-
"tone_argmax_agreement": 0.
|
| 23 |
-
"tone_max_abs_dp": 0.
|
| 24 |
-
"sarcasm_p95_abs_dp": 0.
|
| 25 |
-
"sarcasm_max_abs_dp": 0.
|
| 26 |
-
"laughter_p95_abs_dp": 0.
|
| 27 |
-
"arousal_p95_abs_d": 0.
|
| 28 |
-
"speech_rate_p95_rel_d": 0.
|
| 29 |
"auroc_sarcasm": {
|
| 30 |
-
"reference": 0.
|
| 31 |
-
"other": 0.
|
| 32 |
},
|
| 33 |
"auroc_sarcasm_both": {
|
| 34 |
-
"reference": 0.
|
| 35 |
-
"other": 0.
|
| 36 |
},
|
| 37 |
"auroc_sarcasm_voice_only": {
|
| 38 |
-
"reference": 0.
|
| 39 |
-
"other": 0.
|
| 40 |
},
|
| 41 |
"tone_macro_f1": {
|
| 42 |
-
"reference": 0.
|
| 43 |
-
"other": 0.
|
| 44 |
"rows": 1546
|
| 45 |
}
|
| 46 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"release": "rc2",
|
| 3 |
"folder": "voice-reader-base",
|
| 4 |
+
"run": "v4-base-all-sad",
|
| 5 |
+
"staged": "2026-11-04T17:14:31+00:00",
|
| 6 |
"files": {
|
| 7 |
"model.onnx": {
|
| 8 |
+
"sha256": "8a6aa25f4a0ce43d7e1289a15c0dfa894a8848ee1fbba682c38869782d19ded9",
|
| 9 |
"bytes": 109728612
|
| 10 |
},
|
| 11 |
"model.int8.onnx": {
|
| 12 |
+
"sha256": "8dfd1f1e271d4d64b0d001220ffb7ebd28935cef630377ac37fb9ce939de1218",
|
| 13 |
"bytes": 28567980
|
| 14 |
},
|
| 15 |
"config.json": {
|
| 16 |
+
"sha256": "e76664925738674626c87ae92d475a5dcf373d74ce87737ff61b869bef644a84",
|
| 17 |
+
"bytes": 661
|
| 18 |
}
|
| 19 |
},
|
| 20 |
"parity_int8": {
|
| 21 |
"windows": 2000,
|
| 22 |
+
"tone_argmax_agreement": 0.9825,
|
| 23 |
+
"tone_max_abs_dp": 0.11225080490112305,
|
| 24 |
+
"sarcasm_p95_abs_dp": 0.007848226465284824,
|
| 25 |
+
"sarcasm_max_abs_dp": 0.05139094591140747,
|
| 26 |
+
"laughter_p95_abs_dp": 0.0011351238936185837,
|
| 27 |
+
"arousal_p95_abs_d": 0.01641533151268959,
|
| 28 |
+
"speech_rate_p95_rel_d": 0.024630198255181313,
|
| 29 |
"auroc_sarcasm": {
|
| 30 |
+
"reference": 0.8032301374190491,
|
| 31 |
+
"other": 0.8031314168377823
|
| 32 |
},
|
| 33 |
"auroc_sarcasm_both": {
|
| 34 |
+
"reference": 0.8373954473266279,
|
| 35 |
+
"other": 0.8395341450502911
|
| 36 |
},
|
| 37 |
"auroc_sarcasm_voice_only": {
|
| 38 |
+
"reference": 0.8842925206080314,
|
| 39 |
+
"other": 0.8799818498071542
|
| 40 |
},
|
| 41 |
"tone_macro_f1": {
|
| 42 |
+
"reference": 0.5149164346694854,
|
| 43 |
+
"other": 0.5091877352207718,
|
| 44 |
"rows": 1546
|
| 45 |
}
|
| 46 |
}
|
voice-reader-base/config.json
CHANGED
|
@@ -15,20 +15,20 @@
|
|
| 15 |
"upset",
|
| 16 |
"sad"
|
| 17 |
],
|
| 18 |
-
"sarcasm_tau": 0.
|
| 19 |
"calibration": {
|
| 20 |
"sarcasm": {
|
| 21 |
-
"a": 1.
|
| 22 |
-
"b": 0.
|
| 23 |
},
|
| 24 |
"laughter": {
|
| 25 |
-
"a": 0.
|
| 26 |
-
"b":
|
| 27 |
},
|
| 28 |
-
"tone_temperature": 0.
|
| 29 |
},
|
| 30 |
-
"run": "
|
| 31 |
"init": "laion-base",
|
| 32 |
-
"best_epoch":
|
| 33 |
"note": "sarcasm_tau is the dev 2 % FPR threshold on the Platt-calibrated probability; tone temperature and laughter Platt curve fitted on dev"
|
| 34 |
}
|
|
|
|
| 15 |
"upset",
|
| 16 |
"sad"
|
| 17 |
],
|
| 18 |
+
"sarcasm_tau": 0.060534259700932624,
|
| 19 |
"calibration": {
|
| 20 |
"sarcasm": {
|
| 21 |
+
"a": 1.7739810811041519,
|
| 22 |
+
"b": 0.311429803621114
|
| 23 |
},
|
| 24 |
"laughter": {
|
| 25 |
+
"a": 0.8804135276963087,
|
| 26 |
+
"b": 0.22502338628323926
|
| 27 |
},
|
| 28 |
+
"tone_temperature": 0.9473796186181745
|
| 29 |
},
|
| 30 |
+
"run": "v4-base-all-sad",
|
| 31 |
"init": "laion-base",
|
| 32 |
+
"best_epoch": 8,
|
| 33 |
"note": "sarcasm_tau is the dev 2 % FPR threshold on the Platt-calibrated probability; tone temperature and laughter Platt curve fitted on dev"
|
| 34 |
}
|
voice-reader-base/model.int8.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 28567980
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8dfd1f1e271d4d64b0d001220ffb7ebd28935cef630377ac37fb9ce939de1218
|
| 3 |
size 28567980
|
voice-reader-base/model.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 109728612
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8a6aa25f4a0ce43d7e1289a15c0dfa894a8848ee1fbba682c38869782d19ded9
|
| 3 |
size 109728612
|
voice-reader-tiny/MANIFEST.json
CHANGED
|
@@ -1,46 +1,46 @@
|
|
| 1 |
{
|
| 2 |
-
"release": "
|
| 3 |
"folder": "voice-reader-tiny",
|
| 4 |
-
"run": "student-tiny-
|
| 5 |
-
"staged": "2026-11-
|
| 6 |
"files": {
|
| 7 |
"model.onnx": {
|
| 8 |
-
"sha256": "
|
| 9 |
-
"bytes":
|
| 10 |
},
|
| 11 |
"model.int8.onnx": {
|
| 12 |
-
"sha256": "
|
| 13 |
-
"bytes":
|
| 14 |
},
|
| 15 |
"config.json": {
|
| 16 |
-
"sha256": "
|
| 17 |
-
"bytes":
|
| 18 |
}
|
| 19 |
},
|
| 20 |
"parity_int8": {
|
| 21 |
"windows": 2000,
|
| 22 |
-
"tone_argmax_agreement": 0.
|
| 23 |
-
"tone_max_abs_dp": 0.
|
| 24 |
-
"sarcasm_p95_abs_dp": 0.
|
| 25 |
-
"sarcasm_max_abs_dp": 0.
|
| 26 |
-
"laughter_p95_abs_dp": 0.
|
| 27 |
-
"arousal_p95_abs_d": 0.
|
| 28 |
-
"speech_rate_p95_rel_d": 0.
|
| 29 |
"auroc_sarcasm": {
|
| 30 |
-
"reference": 0.
|
| 31 |
-
"other": 0.
|
| 32 |
},
|
| 33 |
"auroc_sarcasm_both": {
|
| 34 |
-
"reference": 0.
|
| 35 |
-
"other": 0.
|
| 36 |
},
|
| 37 |
"auroc_sarcasm_voice_only": {
|
| 38 |
-
"reference": 0.
|
| 39 |
-
"other": 0.
|
| 40 |
},
|
| 41 |
"tone_macro_f1": {
|
| 42 |
-
"reference": 0.
|
| 43 |
-
"other": 0.
|
| 44 |
"rows": 1546
|
| 45 |
}
|
| 46 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"release": "rc2",
|
| 3 |
"folder": "voice-reader-tiny",
|
| 4 |
+
"run": "student-tiny-v4-sad",
|
| 5 |
+
"staged": "2026-11-04T17:14:31+00:00",
|
| 6 |
"files": {
|
| 7 |
"model.onnx": {
|
| 8 |
+
"sha256": "ad95fc8ae3370f98b91630bc4a12b71aec43d0d652f09b37dfed085e5238a833",
|
| 9 |
+
"bytes": 53366574
|
| 10 |
},
|
| 11 |
"model.int8.onnx": {
|
| 12 |
+
"sha256": "31d06f9a42656a3144997efa1f085abe29bbcbf20722e5711ba34d86f430dcbf",
|
| 13 |
+
"bytes": 14140364
|
| 14 |
},
|
| 15 |
"config.json": {
|
| 16 |
+
"sha256": "a32094d77c53f0fda50cd5328d4f4fc937535ffe1bfaefba760cb1771b1fb60d",
|
| 17 |
+
"bytes": 665
|
| 18 |
}
|
| 19 |
},
|
| 20 |
"parity_int8": {
|
| 21 |
"windows": 2000,
|
| 22 |
+
"tone_argmax_agreement": 0.983,
|
| 23 |
+
"tone_max_abs_dp": 0.21781721711158752,
|
| 24 |
+
"sarcasm_p95_abs_dp": 0.009449773468077183,
|
| 25 |
+
"sarcasm_max_abs_dp": 0.0660611093044281,
|
| 26 |
+
"laughter_p95_abs_dp": 0.003110053250566125,
|
| 27 |
+
"arousal_p95_abs_d": 0.019918425008654594,
|
| 28 |
+
"speech_rate_p95_rel_d": 0.023557225242257118,
|
| 29 |
"auroc_sarcasm": {
|
| 30 |
+
"reference": 0.8067347180540199,
|
| 31 |
+
"other": 0.8055303269625651
|
| 32 |
},
|
| 33 |
"auroc_sarcasm_both": {
|
| 34 |
+
"reference": 0.8388777130757015,
|
| 35 |
+
"other": 0.8375224986765485
|
| 36 |
},
|
| 37 |
"auroc_sarcasm_voice_only": {
|
| 38 |
+
"reference": 0.8949557589049384,
|
| 39 |
+
"other": 0.886863797927853
|
| 40 |
},
|
| 41 |
"tone_macro_f1": {
|
| 42 |
+
"reference": 0.47394276820506326,
|
| 43 |
+
"other": 0.4619192915760122,
|
| 44 |
"rows": 1546
|
| 45 |
}
|
| 46 |
}
|
voice-reader-tiny/config.json
CHANGED
|
@@ -12,22 +12,23 @@
|
|
| 12 |
"laughter",
|
| 13 |
"arousal",
|
| 14 |
"speech_rate",
|
| 15 |
-
"upset"
|
|
|
|
| 16 |
],
|
| 17 |
-
"sarcasm_tau": 0.
|
| 18 |
"calibration": {
|
| 19 |
"sarcasm": {
|
| 20 |
-
"a": 1.
|
| 21 |
-
"b": 0.
|
| 22 |
},
|
| 23 |
"laughter": {
|
| 24 |
-
"a": 0.
|
| 25 |
-
"b":
|
| 26 |
},
|
| 27 |
-
"tone_temperature": 0.
|
| 28 |
},
|
| 29 |
-
"run": "student-tiny-
|
| 30 |
"init": "laion-tiny",
|
| 31 |
-
"best_epoch":
|
| 32 |
"note": "sarcasm_tau is the dev 2 % FPR threshold on the Platt-calibrated probability; tone temperature and laughter Platt curve fitted on dev"
|
| 33 |
}
|
|
|
|
| 12 |
"laughter",
|
| 13 |
"arousal",
|
| 14 |
"speech_rate",
|
| 15 |
+
"upset",
|
| 16 |
+
"sad"
|
| 17 |
],
|
| 18 |
+
"sarcasm_tau": 0.06329724596107583,
|
| 19 |
"calibration": {
|
| 20 |
"sarcasm": {
|
| 21 |
+
"a": 1.778761607691192,
|
| 22 |
+
"b": 0.2934051131149342
|
| 23 |
},
|
| 24 |
"laughter": {
|
| 25 |
+
"a": 0.9323411247145696,
|
| 26 |
+
"b": 0.17735405909388247
|
| 27 |
},
|
| 28 |
+
"tone_temperature": 0.8733854440751974
|
| 29 |
},
|
| 30 |
+
"run": "student-tiny-v4-sad",
|
| 31 |
"init": "laion-tiny",
|
| 32 |
+
"best_epoch": 15,
|
| 33 |
"note": "sarcasm_tau is the dev 2 % FPR threshold on the Platt-calibrated probability; tone temperature and laughter Platt curve fitted on dev"
|
| 34 |
}
|
voice-reader-tiny/model.int8.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:31d06f9a42656a3144997efa1f085abe29bbcbf20722e5711ba34d86f430dcbf
|
| 3 |
+
size 14140364
|
voice-reader-tiny/model.onnx
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ad95fc8ae3370f98b91630bc4a12b71aec43d0d652f09b37dfed085e5238a833
|
| 3 |
+
size 53366574
|