README.md

---
license: mit
language:
- en
library_name: pytorch
tags:
- audio
- turn-detection
- whisper
- conversation
- speech
- voice-assistant
pipeline_tag: audio-classification
datasets:
- pipecat-ai/smart-turn-data-v3.2-train
metrics:
- recall
- accuracy
- precision
- f1
base_model:
- openai/whisper-tiny
---

# Tiny Turn Detector

A lightweight real-time audio turn detection model that predicts whether a speaker is **DONE speaking** or **PAUSING/CONTINUING** in conversational audio.

## Model Description

This model addresses a critical challenge in building responsive voice assistants and conversation systems: determining when a speaker has actually finished their turn versus just pausing mid-sentence.

**Key Features:**
- ⚡ Fast inference (~100ms on CPU)
- 🎯 High accuracy (95.5% on training set, 73% on validation)
- 🔊 Works with 8-second audio clips
- 🚀 Easy to integrate with existing systems
- 📦 Small model size (~150MB with Whisper Tiny)

## Architecture

```
Audio (8 sec, 16kHz)
    ↓
Whisper Tiny Encoder (frozen/fine-tuned)
    ↓
Mean Pooling
    ↓
MLP Head (384 → 64 → 1)
    ↓
Sigmoid → P(end_turn)
    ↓
Binary Decision: END or CONTINUE
```

**Components:**
- **Encoder:** OpenAI Whisper Tiny (pretrained)
- **Classifier:** 2-layer MLP with ReLU activation
- **Input:** 8-second audio clips at 16kHz
- **Output:** Binary classification (0=CONTINUE, 1=END)

## Training Results

The model was trained on the [pipecat-ai/smart-turn-data-v3.2-train](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train) dataset.

### Final Metrics (Best Model)

| Split      | Loss   | Accuracy | Precision | Recall | F1 Score |
|------------|--------|----------|-----------|--------|----------|
| **Train**  | 0.1391 | 95.50%   | 95.90%    | 95.34% | **95.62%** |
| **Val**    | 4.9075 | 73.00%   | 70.00%    | 89.09% | **78.40%** |

**Training Configuration:**
- Epochs: Multiple epochs with early stopping
- Optimizer: AdamW
- Loss Function: Binary Cross-Entropy with Logits
- Best model selected based on validation F1 score (0.7840)

**Note:** The validation loss is higher due to the model being optimized for F1 score rather than loss. The high recall (89%) indicates the model is conservative about marking turn endings, which is desirable for real-time applications to avoid premature interruptions.

## Usage

### Download the Model

```python
from huggingface_hub import hf_hub_download
import torch

# Download model
model_path = hf_hub_download(
    repo_id="YOUR_USERNAME/tiny-turn-detector",
    filename="best_model.pt"
)

# Load model
model = torch.load(model_path, map_location='cpu')
model.eval()
```

### Run Inference

```python
import torch
import librosa
from transformers import WhisperProcessor

# Load processor
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")

# Load audio (8 seconds at 16kHz)
audio, sr = librosa.load("your_audio.wav", sr=16000, duration=8.0)

# Process audio
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")

# Predict
with torch.no_grad():
    outputs = model(inputs.input_features)
    probability = torch.sigmoid(outputs).item()
    
# Decision
threshold = 0.5
decision = "END" if probability > threshold else "CONTINUE"

print(f"Probability: {probability:.4f}")
print(f"Decision: {decision}")
```

### Full Inference Script

For complete inference code with audio loading, preprocessing, and visualization, see the [GitHub repository](YOUR_GITHUB_REPO_URL).

## Installation

```bash
pip install torch torchaudio transformers librosa huggingface_hub
```

## Use Cases

- 🎙️ Voice assistants and chatbots
- 📞 Real-time conversation systems
- 🎧 Meeting transcription tools
- 🤖 Interactive voice response (IVR) systems
- 💬 Voice-based interfaces
- 🎮 Voice-controlled applications

## Model Details

- **Model Type:** Audio Classification (Binary)
- **Base Model:** OpenAI Whisper Tiny
- **Language:** English (primarily)
- **Sampling Rate:** 16kHz
- **Input Duration:** 8 seconds
- **Framework:** PyTorch
- **Parameters:** ~39M (Whisper) + ~25K (Classifier)

## Training Data

**Dataset:** [pipecat-ai/smart-turn-data-v3.2-train](https://huggingface.co/datasets/pipecat-ai/smart-turn-data-v3.2-train)

The dataset contains conversational audio clips labeled with turn-taking information:
- `endpoint_bool`: Binary label (0=continue, 1=end)
- Audio clips of varying lengths (processed to 8 seconds)
- Real-world conversational scenarios

## Limitations

- **Validation Gap:** The model shows some overfitting (95.5% train vs 73% val accuracy). This could be improved with:
  - Data augmentation
  - Regularization techniques
  - More diverse training data
  
- **8-Second Window:** Requires exactly 8 seconds of audio context
  
- **English Focus:** Primarily trained on English conversations

- **VAD Dependency:** Works best when combined with Voice Activity Detection (VAD) for silence removal

## Future Improvements

- [ ] Add multi-language support
- [ ] Reduce validation gap through regularization
- [ ] Variable-length audio support
- [ ] Real-time streaming inference
- [ ] Integration with VAD systems
- [ ] Ensemble with acoustic features (pause duration, pitch)

## GitHub Repository

Full training code, evaluation scripts, and inference examples:  
🔗 **https://github.com/Nitin1613/Turn_detector/tree/main** 

The repository includes:
- Complete training pipeline
- Dataset preparation scripts
- Evaluation and benchmarking tools
- Inference examples
- Google Colab notebook for easy experimentation

## Citation

If you use this model in your research or application, please cite:

```bibtex
@misc{tiny-turn-detector-2026,
  title={Tiny Turn Detector: Real-time Audio Turn Detection with Whisper},
  author=Nitinbudania,
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/Nitinbudania/tiny-turn-detector}}
}
```

## License

MIT License - See [LICENSE](YOUR_GITHUB_REPO_URL/blob/main/LICENSE) file for details

## Acknowledgments

- OpenAI for the Whisper model
- Pipecat.ai for the training dataset
- Hugging Face for hosting and tools

---

**Model Card Authors:** YOUR_NAME  
**Contact:** YOUR_EMAIL or GitHub  
**Last Updated:** August 2026