Smart Turn for phone lines
Smart Turn v3.2 is Pipecat's model that listens to the last 8 seconds of someone speaking and says whether they have finished their turn. It gets worse on phone audio, so this is a fine-tune of it on simulated phone lines. On Pipecat's 31,527 test clips, accuracy goes from 91.7% to 93.1% on a simulated landline and from 90.5% to 92.3% on a GSM mobile codec, and stays at 93.7% on wideband audio.
smart-turn-telephony.onnx takes the same input and gives the same output as Pipecat's
smart-turn-v3.2-gpu.onnx, so one file can replace the other.
Summary
| Smart Turn v3.2 | This fine-tune | |
|---|---|---|
| Accuracy, wideband | 93.7% | 93.7% |
| Accuracy, simulated landline | 91.7% | 93.1% |
| Accuracy, simulated GSM codec | 90.5% | 92.3% |
Things to know before using it:
- Both lines are simulated. The GSM codec was never used in training, so it is the fairer number; the landline falls inside the family of lines used in training.
- Most of the test set is synthetic speech. On its 5,863 real recordings the gain is smaller: +0.8 points on the landline, +0.9 on GSM, and -0.2 on wideband audio, which is within noise.
- On 28 real phone calls an indirect end-of-turn test shows no change, better or worse (see "Real phone calls").
- Wideband accuracy is the same, but 1.9% of the wideband answers change.
- The comparison is against Pipecat's full-precision file. Their 8-bit CPU file was not tested, and there is no 8-bit version of this fine-tune yet.
This is an unofficial fine-tune. It is not made or endorsed by Pipecat or Daily.
Model
| Property | Value |
|---|---|
| Parameters | 8.0 M, same architecture as Smart Turn v3.2 (Whisper-tiny encoder cut to 8 s, attention pooling, classifier) |
| Input | input_features float32 [batch, 80, 800]: Whisper log-mel features of the last 8 s of 16 kHz audio, computed as Pipecat's inference.py computes them |
| Output | logits float32 [batch, 1]: despite the name, the probability that the turn is complete, as in Pipecat's file |
| Opset | 18 |
| Base | pipecat-ai/smart-turn-v3 at revision f766f81d3cfd, file smart-turn-v3.2-gpu.onnx |
Files
| File | Size | Description |
|---|---|---|
smart-turn-telephony.onnx |
32 MB | float32 ONNX, a drop-in replacement for smart-turn-v3.2-gpu.onnx |
smart-turn-telephony.safetensors |
32 MB | the same weights for the PyTorch rebuild in scripts/smartturn.py, for further training |
config.json |
β | input and output contract, base revision, fine-tuning settings |
scripts/ |
β | the rebuild, fine-tuning and test code that produced these files and numbers |
results/ |
β | the training log and the scores behind the tables below |
LICENSE, NOTICE |
β | Pipecat's BSD 2-Clause license, and what was changed |
Performance
Three conditions:
- Wideband: the test audio as published, 16 kHz.
- Landline: the same audio at 8 kHz, 300β3400 Hz, through G.711 mu-law.
- Mobile: the same band through the GSM 06.10 full-rate codec. No GSM audio was used in fine-tuning.
Both lines are simulated. Higher is better.
End of turn: all 31,527 clips of pipecat-ai/smart-turn-data-v3.2-test, each scored on its last
8 seconds as Pipecat's inference code does it, threshold 0.5. Accuracy, with AUC in brackets.
| Model | Wideband | Landline | Mobile |
|---|---|---|---|
smart-turn-v3.2-gpu.onnx (Pipecat) |
93.71% (0.9825) | 91.71% (0.9724) | 90.48% (0.9664) |
smart-turn-telephony.onnx |
93.73% (0.9826) | 93.12% (0.9799) | 92.31% (0.9760) |
Differences in accuracy, paired over clips, in points with 95% intervals:
| Wideband | Landline | Mobile | |
|---|---|---|---|
| This model minus Smart Turn v3.2 | +0.03 (-0.13 to +0.17) | +1.41 (+1.18 to +1.65) | +1.83 (+1.58 to +2.07) |
| The same, on the 5,863 real recordings only | -0.24 (-0.56 to +0.09) | +0.82 (+0.26 to +1.35) | +0.85 (+0.32 to +1.38) |
- A phone line now costs this model 0.6 points (landline) and 1.4 points (mobile), where it cost Smart Turn v3.2 2.0 and 3.2 points.
- The AUC rises on both lines too, so the gain is in the scores themselves, not a shift of the threshold.
- The intervals treat clips as independent. Many clips share a speaker or a source, so the true intervals are somewhat wider.
Real phone calls: 28 calls, 3.9 hours, from CallHome English, with hand-marked times of who spoke when. No test set we could use says whether a turn has really ended on a real call, so this test is indirect. At each of 737 pauses of at least 0.5 s, the score is read 0.4 s into the pause, and the question is whether it tells "the other person speaks next" (177 pauses) from "the same person goes on".
| Smart Turn v3.2 | This fine-tune | |
|---|---|---|
| AUC | 0.691 | 0.685 |
| Says "ended" when the other person speaks next | 37% | 32% |
| Says "ended" when the same person goes on | 19% | 17% |
Paired over calls, the AUC difference is -0.006 (-0.017 to +0.007): no measurable change. The fine-tune says "ended" a little less often at both kinds of pause. On wideband meeting recordings a phone line barely moves this test for Smart Turn v3.2, so it cannot show the gain measured above; it only shows that nothing broke on real calls.
Speed: one call (8 s of audio) on one CPU thread of an Apple M4 Max with ONNX Runtime 1.32 takes 19.7 ms, against 20.4 ms for Pipecat's file.
Usage
Wherever your code or your Pipecat setup loads smart-turn-v3.2-gpu.onnx, load
smart-turn-telephony.onnx instead. In Pipecat's own
inference.py, that is one line:
ONNX_MODEL_PATH = "smart-turn-telephony.onnx"
Phone audio arrives at 8 kHz. Decode it to linear PCM and upsample it to 16 kHz first; the
simulated lines this model was trained and tested on are upsampled with
scipy.signal.resample_poly(x, 2, 1).
How it was made
- Rebuild. Pipecat publishes Smart Turn as ONNX files only.
scripts/smartturn.pyrebuilds the model in PyTorch, followingSmartTurnV3Modelin Pipecat's training code, and copies the weights out ofsmart-turn-v3.2-gpu.onnx. It gives the same probabilities as that file to within 3e-6, and exporting the fine-tuned weights with it reproducessmart-turn-telephony.onnxexactly. - Fine-tuning. Two copies of the model. The frozen one hears each clip's last 8 seconds as published. The other hears the same clip through a random simulated phone line and is trained to give the frozen one's probability (the loss is the KL divergence between the two). Training uses no labels; the labels of a held-out file were only used to pick the checkpoint. One clip in four is left clean so the wideband behaviour stays put.
- Training lines. 8 kHz; band-pass with a low edge of 150β400 Hz and a high edge of 3000β3700 Hz; G.711 mu-law, A-law or no codec; input level 0 to -20 dB; white line noise on 30% of clips, at -70 to -30 dB relative to full scale. GSM was kept out of training to serve as the unseen line in the tests.
- Data. 29,385 clips from 9 of the 83 files of
pipecat-ai/smart-turn-data-v3.2-train, a set Smart Turn v3.2 was itself trained on, so no new speech data is involved. 1,000 clips of a tenth file, scored clean and through the test landline, chose the checkpoint: step 1,500 of 2,500. None of the test clips is in the training files. - Settings. Batch 32; AdamW with a peak learning rate of 1e-5, 100 warm-up steps and cosine decay; gradient norm clipped at 1.0; dropout off. Pipecat trains at 5e-5 with batch 384, and at batch 32 that rate made the loss climb. 19 minutes on an Apple M4 Max GPU, using about 3.6 GB of GPU memory.
- Checks.
scripts/test_smartturn.pychecks the rebuild and the export against Pipecat's file. Both models were scored by the same code,scripts/bench.py;scripts/compare.pyandscripts/compare_pairs.pyproduced the paired numbers.config.jsonhas the full settings andscripts/README.mdthe commands.
Limitations
- The end-of-turn gain is measured on simulated lines only. Real calls add packet loss, echo, noise suppression and codecs that were not simulated (AMR, Opus, EVS).
- On real voices the gain is about half of what it is on the synthetic speech that makes up most of the test set.
- The end-of-turn test set comes from the same collection as the fine-tuning audio, as it does for Smart Turn v3.2 itself.
- Fine-tuning copies Smart Turn's wideband answers, so it cannot make the model better than Smart Turn v3.2 on wideband audio, and everything Pipecat says about the model still applies.
- One training run, on 11% of the training set.
- Not tested as an 8-bit model, on CPUs other than Apple's, or on phones.
License and attribution
Smart Turn v3.2 and its training code are Β© 2024β2025 Daily, released under the BSD 2-Clause
License. This model is a fine-tuned form of those weights and is distributed under the same
license: LICENSE is Pipecat's license text and NOTICE lists what was changed. The fine-tuning
audio is from
pipecat-ai/smart-turn-data-v3.2-train
and the end-of-turn test set is
pipecat-ai/smart-turn-data-v3.2-test,
both CC BY 4.0.
The real-call test uses the first 28 calls of the English part of talkbank/callhome (CABank English CallHome Corpus, TalkBank, doi:10.21415/T5KP54, CC BY-NC-SA 4.0). It was used for testing only: the model was not trained on it, and none of its audio or annotation is in this repository.
- Downloads last month
- -
Model tree for matthiastl/smart-turn-telephony
Base model
pipecat-ai/smart-turn-v3