smart-turn-tamil

Semantic end-of-turn detection for Tamil — given the last 8 seconds of a speaker's audio, predicts whether they have finished their turn.

Fine-tune of Smart Turn v3. Trained on real Tamil telephone conversations (116 two-party calls, agent and customer sides). ONNX, CPU-only, drop-in for Pipecat and LiveKit.

Variants

Four graphs ship. tiny int8 is the default.

variant	precision	accuracy	ROC-AUC	FP/N	size	p50
tiny	int8	83.71%	0.905	7.94%	8.7 MB	83 ms
tiny	fp32	83.35%	0.904	7.73%	32 MB	133 ms
base	int8	86.13%	0.921	9.17%	21 MB	143 ms
base	fp32	86.23%	0.922	8.95%	81 MB	232 ms

tiny matches upstream's architecture, so it is a true drop-in replacement. base (whisper-base encoder) is +2.4 points for 2.4× the size and 1.7× the latency.

int8 is dynamic, not static. Static quantisation was measured and rejected: −4.03 points at tiny, −12.76 at base (AUC 0.922 → 0.792). Dynamic is lossless here.

Latency is 1 thread, batch 1, idle i5-12450H, inference only; add ~12 ms for mel extraction. Pipecat defaults to one thread, which is why that is quoted.

Usage

To use this model in LiveKit, install the smart-turn-livekit plugin:

pip install 'smart-turn-livekit[livekit]'

LiveKit Agents — weights download on first use and are cached.

from smart_turn_livekit import SmartTurnDetector
turn_detection = SmartTurnDetector(model="smart-turn-tamil-tiny")

smart-turn-livekit also runs upstream's smart-turn-v3.

Pipecat — no adapter needed; its built-in analyzer takes any Smart Turn ONNX:

from smart_turn_livekit import resolve_model
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3

turn_analyzer = LocalSmartTurnAnalyzerV3(
    smart_turn_model_path=str(resolve_model("smart-turn-tamil-tiny")),   # the file
)

Note Pipecat hardcodes the 0.5 threshold, and there the decision ends the turn rather than choosing a delay.

Direct ONNX — input input_features, shape (batch, 80, 800), log-mel at 16 kHz. Output is named logits but is already a sigmoid; applying another maps [0,1] onto [0.5, 0.73] and every clip predicts complete.

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("smart-turn-tamil-tiny/smart-turn-tamil-int8-dynamic.onnx")
p = sess.run(None, {"input_features": feats})[0].reshape(-1)   # (batch,) in [0,1]
complete = p > 0.5

Features must match WhisperFeatureExtractor(chunk_length=8) with do_normalize=True — that is not the Whisper default, and getting it wrong produces plausible but meaningless output.

Threshold

0.5 is inherited from the training loop; neither variant peaks there. Tuning it for accuracy on dev lost base 1.13 points on test, so 0.5 ships as the default. Targeting a false-positive rate transfers better than targeting accuracy:

operating point	tiny	base	effect
inherited (default)	0.50	0.50	figures above
polite	0.75	0.92	FP/N roughly halves, costs ~3 points of accuracy

Both picked on dev, then measured once on test.

Evaluation

Held-out test split: 4,168 clips from 30 calls, split by call. Threshold 0.5.

	accuracy	ROC-AUC	FP/N
majority class	63.08%	0.500	–
smart-turn-v3.2 zero-shot	70.30%	0.751	–
smart-turn-tamil, tiny int8	83.71%	0.905	7.94%
smart-turn-tamil, base int8	86.13%	0.921	9.17%

FP/N is Smart Turn's convention — FP/N + FN/N sums to the error rate — not the standard FP/(FP+TN). The two differ by roughly 3×.

For scale, Smart Turn v3.2's published per-language figures. These are from a different benchmark on TTS-generated audio and are not directly comparable:

	accuracy	FP/N
Hindi	90.11%	8.57%
Bengali	83.80%	10.90%
Marathi	82.43%	15.12%

In a live pipeline

Run end-to-end as a Tamil voice agent on LiveKit Agents 1.7, with an all-Sarvam stack around it:

turn detection	smart-turn-tamil tiny int8
VAD	Silero, min_silence_duration=0.25
STT	Sarvam saarika:v2.5
LLM	Sarvam sarvam-105b
TTS	Sarvam bulbul:v3

Also verified against Pipecat 1.7 via LocalSmartTurnAnalyzerV3. Adapter latency end-to-end is ~120–155 ms on live audio — mel plus ONNX plus the thread handoff — against the 83 ms inference-only figure above.

The accuracy cost of serving was measured separately, because a live call has no per-pause labels. Labelled test boundaries were replayed through the real Silero VAD and the real streaming adapter, then scored against the same clips cut the offline way:

	pre-cut clip	live window
accuracy	86.12%	83.51%
FP/N	5.21%	6.51%

−2.60 points, with the identical verdict on 90.9% of boundaries (n=461, McNemar p=0.09 — consistent in direction, not formally significant). The cause is a 90 ms window shift: training clips end 0.20 s past the speech offset, the VAD closes at 0.29 s.

Coverage 92.2%. LiveKit will not request a prediction below min_silence_duration + 50 ms, so the model is never consulted on the shortest pauses. Those are outside the product rather than errors — which is also why the pre-cut column above reads 86.12% against 83.35% for the full split: the boundaries a VAD surfaces are the easier ones.

Training data

santhosh-005/tamil-eot — 18,485 labelled turn boundaries from 116 Tamil telephone conversations, split by call. Labels validated at 97.5% agreement with a human listener before use.

whisper-tiny / whisper-base encoder, attention pooling, binary head, 6 epochs, lr 5e-5, batch 32, BCEWithLogitsLoss with per-batch pos_weight. Method and full ablations in the repository below.

Limitations

• Narrowband telephony. Trained on phone-quality audio (~8 kHz, narrowband). Wideband or close-mic speech is out of distribution.
• Tamil only. No claim on other South Indian languages yet.
• 8 s window. Only the last 8 seconds are read.
• No labelled live call. The headline figures are on pre-cut clips. The live path was measured by replay (above), which is deterministic and labelled; a real call carries no per-pause labels, so no accuracy number can come from one.
• Code-switching with English is frequent in this corpus and handled, but not measured as a separate slice.
• Run-to-run spread at identical config is ~0.9 points; treat differences below that as noise.

Other languages

The data pipeline is language-agnostic. Future releases may include other South Indian languages like Malayalam, Kannada, and Telugu.

Files

smart-turn-tamil-{tiny,base}/smart-turn-tamil-int8-dynamic.onnx	int8 dynamic
smart-turn-tamil-{tiny,base}/smart-turn-tamil.onnx	fp32 reference
smart-turn-tamil-{tiny,base}/best.pt	torch checkpoints, for further fine-tuning
mel_filters.npz	80-mel filterbank for the numpy feature path
config.json	thresholds and metadata

Licence and citation

Fine-tuned by Santhosh. BSD-2-Clause, inherited from pipecat-ai/smart-turn. Training data derives from SPRING_INX Tamil R1 (CC BY 4.0), SPRING Lab, IIT Madras.

@misc{smart-turn-tamil,
  author = {santhosh-005},
  title  = {smart-turn-tamil: semantic end-of-turn detection for Tamil},
  year   = {2026},
  url    = {https://github.com/santhosh-005/tamil-eot}
}

Method, benchmarks and ablations: https://github.com/santhosh-005/tamil-eot