File size: 6,021 Bytes
c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c c9b59be 08c9b8c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 | ---
license: bsd-2-clause
language: en
library_name: onnx
pipeline_tag: text-classification
base_model:
- jhu-clsp/ettin-encoder-150m
- jhu-clsp/ettin-encoder-32m
datasets:
- ACloudCenter/ami-addressee
tags:
- pipecat
- onnx
- addressee-detection
- multi-party
- turn-taking
---
# Pipecat Sidechat (experimental, rc0.2)
> **⚠️ Experimental.** Outputs, names and calibration may change between releases.
A small text classifier for a voice agent in a conversation with several people. For each line it gives the
probability that the agent should speak now, and whether the line was for it, for the room or about it. Speaker
labels can be anonymous ("Speaker 1"); the agent's name and the rhythm of the last turns carry the signal.
Code: [github.com/pipecat-ai/sidechat](https://github.com/pipecat-ai/sidechat)
Two builds:
| Build | Folder | For |
|---|---|---|
| Default, 150M (`side-r3-ettin-150m-s0`) | repo root | 1 vCPU or more; the more accurate one |
| `agent-1x`, 32M two-tower (`side-r6ttr3t-ettin-32m-s0`) | `agent-1x/` | 0.5 vCPU, 1 GiB (Pipecat Cloud `agent-1x`) |
The `agent-1x` build encodes the earlier lines while the person is still talking, then reads only the new line
when they stop. It was distilled from the default build.
## Use
```bash
pip install git+https://github.com/pipecat-ai/sidechat
```
```python
from huggingface_hub import snapshot_download
from sidechat.runtime import load
from sidechat.schema import Agent, SidechatRequest, Turn
model = load(snapshot_download("pipecat-ai/sidechat", allow_patterns=["*.json", "model.onnx"]))
a = model.answer(SidechatRequest(
agent=Agent(name="Atlas"),
speaker="Speaker 1",
text="Can you check that for us?",
history=[Turn(speaker="Atlas", text="Want me to look up flights?", agent=True)],
))
print(a.respond, model.decide(a)) # 0.99 respond
```
For `agent-1x`, load the folder and call `prepare` with the next line's request (its history is what counts)
as soon as the previous line ends; `answer` then encodes only the new line:
```python
path = snapshot_download("pipecat-ai/sidechat", allow_patterns=["agent-1x/*.json", "agent-1x/*.onnx"])
model = load(f"{path}/agent-1x", threads=1)
model.prepare(request) # while the person talks
a = model.answer(request) # when they stop
```
| Output | Meaning |
|---|---|
| `respond` | probability the agent should speak now; about 0.5 for a question left open to the room |
| `intent` | for the agent directly or indirectly, open to the room, about the agent, or other |
| `unclear` | the line fits the agent and someone else equally |
`decide` gives `respond`, `wait` (speak only if nobody else does) or `ignore`, at the thresholds in `config.json`.
Pass `int8=True` to `load` for the default build's `model-int8.onnx`, which has thresholds of its own.
## Files
| File | |
|---|---|
| `model.onnx`, `model-int8.onnx` | default build, fp32 598 MB, int8 267 MB |
| `agent-1x/context.onnx`, `agent-1x/line.onnx` | `agent-1x` build, fp32, 128 + 131 MB |
| `tokenizer.json`, `config.json` (each folder) | tokenizer, calibration and thresholds |
| `data/`, `agent-1x/data/` | the rows each build was trained on (train and dev) |
| `MANIFEST.json` (each folder) | sha256 of every file, row counts |
## Results against Jev
Anonymous speakers. Jev is asked as Pipecat's daily-audience demo asks it.
| Set | Default | `agent-1x` | Jev |
|---|---|---|---|
| [Duplex-MPE](https://huggingface.co/datasets/ChengqianMa/Duplex-MPE), 52,336 lines | 96.7% | 95.6% | 83.1% |
| … lines for others, answered outright | 0.22% | 0.26% | 0.41% |
| … lines for others, held for a pause | 0.9% | 0.8% | 16.0% |
| … request naming the agent, answered | 99.5% | 98.7% | 99.2% |
| … same request without the name, answered | 38.8% | 10.7% | 26.6% |
| [Social AI Ambient Bench](https://huggingface.co/datasets/text-ai/social-ai-ambient-bench), 15,793 lines | 98.7% | 98.4% | 85.4% |
| [AMI](https://huggingface.co/datasets/ACloudCenter/ami-addressee) labelled meetings, 8,875 lines | 99.9% | 100.0% | 85.2% |
| 270 hand-written lines | 87.4% | 87.4% | 77.8% |
These are baselines, not proper benchmarks: labels follow the project's own guide, and the choice between
training rounds looked at these sets.
## Speed
Median ms per line with 8 earlier lines, one thread, on containers capped like Pipecat Cloud's profiles. Jev is
an API call from the same container, so its time is mostly the network.
| | 0.5 vCPU, 1 GiB | 1 vCPU, 2 GiB | Memory |
|---|---|---|---|
| `agent-1x`, earlier lines encoded ahead | 10 to 15 | 9 | 488 MB |
| `agent-1x`, earlier lines not ready | 88 to 117 | 35 | |
| Default, int8 / fp32 | 532 / does not fit | 253 (fp32) | 571 MB (int8) |
| Jev | 56 to 163 | 58 to 149 | |
On an M5 Max the default build takes 10 ms with a short history and 52 ms with 8 earlier lines.
## Limitations
- A request without the agent's name, before the agent has spoken, is answered less than half the time by the
default build and about one time in ten by `agent-1x`, which mostly ignores it (Jev mostly holds for a pause).
- English text only. Most training lines are generated; only AMI is real speech.
## Training
Both use [Ettin](https://huggingface.co/jhu-clsp/ettin-encoder-150m) encoders (MIT) with intent, respond and
unclear heads. The default build: Ettin-150M, 2 epochs on 235,069 lines. `agent-1x`: Ettin-32M as one shared
encoder for the earlier lines and the new line, joined by a cross-attention layer, 3 epochs on 420,828 lines,
with the default build's probabilities as 80% of each target. The lines come from generated multi-party
conversations, labelled by kind under the project's guide, and from AMI meetings
([ACloudCenter/ami-addressee](https://huggingface.co/datasets/ACloudCenter/ami-addressee), CC BY 4.0) outside
every test meeting, all lines for people. Speakers were anonymous in 60% of training lines, named in 25% and
unlabelled in 15%. Duplex-MPE and the Social AI Ambient Bench were used for evaluation only.
|