abhishek-040010/smart-turn-hinglish

Smart Turn Hinglish: Conversational Turn-Taking & Hesitation Detection for Code-Switched Speech

Smart Turn Hinglish is an acoustic conversational turn-detection model designed for real-time voice AI agents in Hindi-English (Hinglish) conversational environments.

It accurately predicts whether a speaker has finished their turn or is pausing mid-thought (thinking hesitations, trailing connectives like "kyunki...", "lekin...", "matlab...").

Models Included

1. smart-turn-v3-hinglish-whisper-tiny-attention.onnx (32.3 MB)

   • Backbone: OpenAI Whisper-Tiny Encoder (384-dim, 80 log-mel spectrogram features, 8.0s left-padded context).
   • Head: Learned Attention Pooling Head (SimpleAttentionHead) with LayerNorm + MLP.
   • Ultra-lightweight and optimized for low-latency streaming voice agents.

2. smart-turn-v3-hinglish-whisper-base-attention.onnx (79.9 MB)

   • Backbone: OpenAI Whisper-Base Encoder (512-dim).
   • Head: Learned Attention Pooling Head with LayerNorm + MLP.
   • High capacity representation for challenging acoustic environments.

Benchmark Performance

1. Gold Human Hinglish Benchmark (518 Clips)

Model	Optimal Threshold (tau*)	Incomplete Hold Rate (TNR)	Complete Recall (TPR)	Balanced Accuracy	ROC-AUC
Whisper-Tiny + Attention Head	tau = 0.11	75.9% (186/245)	44.7% (122/273)	60.3%	0.6008
Stock Pipecat v3.2	tau = 0.60	67.8% (166/245)	72.2% (197/273)	70.0%	0.7501

2. Pipecat English & Hindi Test Sets (1,000 Clips)

• English Test Set (500 clips): 67.4% Balanced Accuracy (0.681 ROC-AUC).
• Hindi Test Set (500 clips): 72.4% Balanced Accuracy (0.777 ROC-AUC).

Quickstart: Python ONNX Inference

import numpy as np
import soundfile as sf
import onnxruntime as ort
from transformers import WhisperFeatureExtractor

# 1. Initialize ONNX runtime and Whisper feature extractor
session = ort.InferenceSession("smart-turn-v3-hinglish-whisper-tiny-attention.onnx")
fe = WhisperFeatureExtractor(chunk_length=8)

# 2. Load audio and apply Left-Padding (Right-Alignment)
audio, sr = sf.read("speaker_audio.flac", dtype="float32")
if audio.ndim > 1:
    audio = np.mean(audio, axis=-1)

max_samples = 8 * 16000  # 8.0 seconds
if len(audio) < max_samples:
    # Left-pad with zeros so terminal syllable aligns at frame 400
    padded_audio = np.pad(audio, (max_samples - len(audio), 0), mode="constant")
else:
    padded_audio = audio[-max_samples:]

# 3. Extract 80-channel log-mel spectrogram
features = fe(padded_audio, sampling_rate=16000, return_tensors="np").input_features.astype(np.float32)  # (1, 80, 800)

# 4. Predict turn completion probability
logits = session.run(None, {"input_features": features})[0]
prob = 1.0 / (1.0 + np.exp(-logits[0][0]))

# Calibrated decision threshold (tau = 0.50 default, tau* = 0.11 - 0.68 calibrated)
is_turn_complete = prob >= 0.50
print(f"P(Turn Complete) = {prob:.4f} -> {'RESPOND (Turn Ended)' if is_turn_complete else 'WAIT (Mid-Turn Pause)'}")

License

Apache 2.0