Smart Turn for phone lines

Smart Turn v3.2 is Pipecat's model that listens to the last 8 seconds of someone speaking and says whether they have finished their turn. It gets worse on phone audio, so this is a fine-tune of it on simulated phone lines. On Pipecat's 31,527 test clips, accuracy goes from 91.7% to 93.1% on a simulated landline and from 90.5% to 92.3% on a GSM mobile codec, and stays at 93.7% on wideband audio.

smart-turn-telephony.onnx takes the same input and gives the same output as Pipecat's smart-turn-v3.2-gpu.onnx, so one file can replace the other.

Summary

Smart Turn v3.2 This fine-tune
Accuracy, wideband 93.7% 93.7%
Accuracy, simulated landline 91.7% 93.1%
Accuracy, simulated GSM codec 90.5% 92.3%

Things to know before using it:

  • Both lines are simulated. The GSM codec was never used in training, so it is the fairer number; the landline falls inside the family of lines used in training.
  • Most of the test set is synthetic speech. On its 5,863 real recordings the gain is smaller: +0.8 points on the landline, +0.9 on GSM, and -0.2 on wideband audio, which is within noise.
  • On 28 real phone calls an indirect end-of-turn test shows no change, better or worse (see "Real phone calls").
  • Wideband accuracy is the same, but 1.9% of the wideband answers change.
  • The comparison is against Pipecat's full-precision file. Their 8-bit CPU file was not tested, and there is no 8-bit version of this fine-tune yet.

This is an unofficial fine-tune. It is not made or endorsed by Pipecat or Daily.

Model

Property Value
Parameters 8.0 M, same architecture as Smart Turn v3.2 (Whisper-tiny encoder cut to 8 s, attention pooling, classifier)
Input input_features float32 [batch, 80, 800]: Whisper log-mel features of the last 8 s of 16 kHz audio, computed as Pipecat's inference.py computes them
Output logits float32 [batch, 1]: despite the name, the probability that the turn is complete, as in Pipecat's file
Opset 18
Base pipecat-ai/smart-turn-v3 at revision f766f81d3cfd, file smart-turn-v3.2-gpu.onnx

Files

File Size Description
smart-turn-telephony.onnx 32 MB float32 ONNX, a drop-in replacement for smart-turn-v3.2-gpu.onnx
smart-turn-telephony.safetensors 32 MB the same weights for the PyTorch rebuild in scripts/smartturn.py, for further training
config.json β€” input and output contract, base revision, fine-tuning settings
scripts/ β€” the rebuild, fine-tuning and test code that produced these files and numbers
results/ β€” the training log and the scores behind the tables below
LICENSE, NOTICE β€” Pipecat's BSD 2-Clause license, and what was changed

Performance

Three conditions:

  • Wideband: the test audio as published, 16 kHz.
  • Landline: the same audio at 8 kHz, 300–3400 Hz, through G.711 mu-law.
  • Mobile: the same band through the GSM 06.10 full-rate codec. No GSM audio was used in fine-tuning.

Both lines are simulated. Higher is better.

End of turn: all 31,527 clips of pipecat-ai/smart-turn-data-v3.2-test, each scored on its last 8 seconds as Pipecat's inference code does it, threshold 0.5. Accuracy, with AUC in brackets.

Model Wideband Landline Mobile
smart-turn-v3.2-gpu.onnx (Pipecat) 93.71% (0.9825) 91.71% (0.9724) 90.48% (0.9664)
smart-turn-telephony.onnx 93.73% (0.9826) 93.12% (0.9799) 92.31% (0.9760)

Differences in accuracy, paired over clips, in points with 95% intervals:

Wideband Landline Mobile
This model minus Smart Turn v3.2 +0.03 (-0.13 to +0.17) +1.41 (+1.18 to +1.65) +1.83 (+1.58 to +2.07)
The same, on the 5,863 real recordings only -0.24 (-0.56 to +0.09) +0.82 (+0.26 to +1.35) +0.85 (+0.32 to +1.38)
  • A phone line now costs this model 0.6 points (landline) and 1.4 points (mobile), where it cost Smart Turn v3.2 2.0 and 3.2 points.
  • The AUC rises on both lines too, so the gain is in the scores themselves, not a shift of the threshold.
  • The intervals treat clips as independent. Many clips share a speaker or a source, so the true intervals are somewhat wider.

Real phone calls: 28 calls, 3.9 hours, from CallHome English, with hand-marked times of who spoke when. No test set we could use says whether a turn has really ended on a real call, so this test is indirect. At each of 737 pauses of at least 0.5 s, the score is read 0.4 s into the pause, and the question is whether it tells "the other person speaks next" (177 pauses) from "the same person goes on".

Smart Turn v3.2 This fine-tune
AUC 0.691 0.685
Says "ended" when the other person speaks next 37% 32%
Says "ended" when the same person goes on 19% 17%

Paired over calls, the AUC difference is -0.006 (-0.017 to +0.007): no measurable change. The fine-tune says "ended" a little less often at both kinds of pause. On wideband meeting recordings a phone line barely moves this test for Smart Turn v3.2, so it cannot show the gain measured above; it only shows that nothing broke on real calls.

Speed: one call (8 s of audio) on one CPU thread of an Apple M4 Max with ONNX Runtime 1.32 takes 19.7 ms, against 20.4 ms for Pipecat's file.

Usage

Wherever your code or your Pipecat setup loads smart-turn-v3.2-gpu.onnx, load smart-turn-telephony.onnx instead. In Pipecat's own inference.py, that is one line:

ONNX_MODEL_PATH = "smart-turn-telephony.onnx"

Phone audio arrives at 8 kHz. Decode it to linear PCM and upsample it to 16 kHz first; the simulated lines this model was trained and tested on are upsampled with scipy.signal.resample_poly(x, 2, 1).

How it was made

  • Rebuild. Pipecat publishes Smart Turn as ONNX files only. scripts/smartturn.py rebuilds the model in PyTorch, following SmartTurnV3Model in Pipecat's training code, and copies the weights out of smart-turn-v3.2-gpu.onnx. It gives the same probabilities as that file to within 3e-6, and exporting the fine-tuned weights with it reproduces smart-turn-telephony.onnx exactly.
  • Fine-tuning. Two copies of the model. The frozen one hears each clip's last 8 seconds as published. The other hears the same clip through a random simulated phone line and is trained to give the frozen one's probability (the loss is the KL divergence between the two). Training uses no labels; the labels of a held-out file were only used to pick the checkpoint. One clip in four is left clean so the wideband behaviour stays put.
  • Training lines. 8 kHz; band-pass with a low edge of 150–400 Hz and a high edge of 3000–3700 Hz; G.711 mu-law, A-law or no codec; input level 0 to -20 dB; white line noise on 30% of clips, at -70 to -30 dB relative to full scale. GSM was kept out of training to serve as the unseen line in the tests.
  • Data. 29,385 clips from 9 of the 83 files of pipecat-ai/smart-turn-data-v3.2-train, a set Smart Turn v3.2 was itself trained on, so no new speech data is involved. 1,000 clips of a tenth file, scored clean and through the test landline, chose the checkpoint: step 1,500 of 2,500. None of the test clips is in the training files.
  • Settings. Batch 32; AdamW with a peak learning rate of 1e-5, 100 warm-up steps and cosine decay; gradient norm clipped at 1.0; dropout off. Pipecat trains at 5e-5 with batch 384, and at batch 32 that rate made the loss climb. 19 minutes on an Apple M4 Max GPU, using about 3.6 GB of GPU memory.
  • Checks. scripts/test_smartturn.py checks the rebuild and the export against Pipecat's file. Both models were scored by the same code, scripts/bench.py; scripts/compare.py and scripts/compare_pairs.py produced the paired numbers. config.json has the full settings and scripts/README.md the commands.

Limitations

  • The end-of-turn gain is measured on simulated lines only. Real calls add packet loss, echo, noise suppression and codecs that were not simulated (AMR, Opus, EVS).
  • On real voices the gain is about half of what it is on the synthetic speech that makes up most of the test set.
  • The end-of-turn test set comes from the same collection as the fine-tuning audio, as it does for Smart Turn v3.2 itself.
  • Fine-tuning copies Smart Turn's wideband answers, so it cannot make the model better than Smart Turn v3.2 on wideband audio, and everything Pipecat says about the model still applies.
  • One training run, on 11% of the training set.
  • Not tested as an 8-bit model, on CPUs other than Apple's, or on phones.

License and attribution

Smart Turn v3.2 and its training code are Β© 2024–2025 Daily, released under the BSD 2-Clause License. This model is a fine-tuned form of those weights and is distributed under the same license: LICENSE is Pipecat's license text and NOTICE lists what was changed. The fine-tuning audio is from pipecat-ai/smart-turn-data-v3.2-train and the end-of-turn test set is pipecat-ai/smart-turn-data-v3.2-test, both CC BY 4.0.

The real-call test uses the first 28 calls of the English part of talkbank/callhome (CABank English CallHome Corpus, TalkBank, doi:10.21415/T5KP54, CC BY-NC-SA 4.0). It was used for testing only: the model was not trained on it, and none of its audio or annotation is in this repository.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for matthiastl/smart-turn-telephony

Finetuned
(3)
this model

Dataset used to train matthiastl/smart-turn-telephony