Model Card: IndicTTS Deepfake Speech Detector
Model Details
- Base model:
ntu-spml/distilhubert - Architecture:
HubertForSequenceClassification— DistilHuBERT encoder (frozen CNN feature extractor + fine-tuned transformer layers) with a linear binary classification head - Task: Binary audio classification — authentic human speech vs. AI-generated (TTS) speech
- Language(s): Multilingual, Indic languages (see Training Data)
- License: Inherits
ntu-spml/distilhubert's license (Apache 2.0) for the base model; fine-tuned weights license is up to you to set - Fine-tuned by: [your name/org]
- Fine-tuning date: [fill in]
Intended Use
Intended for: demonstrating automated detection of synthetic speech for educational/portfolio purposes, and as a starting point for further research into speech deepfake detection.
Not intended for: any decision with real-world consequences for a person — content moderation, legal/forensic evidence, authentication systems, or any setting where a false result could cause harm. See Limitations below.
Training Data
SherryT997/IndicTTS-Deepfake-Challenge-Data —
Indic-language speech labeled is_tts (0 = authentic, 1 = synthetic).
- Trained on a 10,000-sample random subset (seed=37) of the full ~16GB training split, with a 90/10 train/validation split (seed=42)
- Audio standardized to 16kHz mono, 2-second (32,000 sample) windows, trimmed or zero-padded as needed
- Class balance of the training subset: [fill in — see
debug_check.py's printed class balance line]
Training Procedure
| Optimizer | AdamW |
| Learning rate | 1e-4, 1,000 warmup steps |
| Weight decay | 0.005 |
| Epochs | 7 (15,750 steps total) |
| Batch size | 4 |
| Precision | fp32 |
| Feature encoder | Frozen |
| Model selection | Best checkpoint by validation loss |
Evaluation
Held-out internal validation split (10% of the 10k training subsample), best checkpoint by validation loss (step 8,500 of 15,750):
| Metric | Score |
|---|---|
| Accuracy | 97.30% |
| Precision | 97.44% |
| Recall | 96.82% |
| F1 | 97.13% |
| ROC-AUC | 0.9960 |
Scored 0.9761 on the IndicTTS Deepfake Challenge public leaderboard (competition test set, disjoint from training data).
Limitations
- Training scale: fine-tuned on 10k samples out of a much larger dataset. Performance on the full data distribution, and on TTS systems/languages underrepresented in the subset, is unverified.
- Fixed input length: every clip is trimmed or padded to exactly 2 seconds. Longer clips are truncated rather than analyzed in full; very short clips are zero-padded, which may reduce reliability on edge cases.
- Distribution shift: trained and validated on audio from a single benchmark dataset. Real-world audio (background noise, compression artifacts, different microphones, TTS systems not represented in training) will likely perform worse than the reported metrics — observed anecdotally during demo testing, where out-of-distribution non-speech audio was classified with high but not necessarily meaningful confidence.
- Confidence calibration: the model was trained with cross-entropy loss and no explicit calibration step. High reported confidence (e.g. 99–100%) should not be read as a calibrated probability of correctness, especially on inputs unlike the training distribution.
- No quantization/compression applied — served as the raw fp32 fine-tuned checkpoint. See main README's "Known limitations" for planned optimization work.
Ethical Considerations
Speech deepfake detectors can be used both defensively (flagging synthetic media, protecting against voice-cloning fraud) and, if inverted, to help adversaries iterate against detection systems. This model is released for educational/demonstration purposes with modest training scale and unverified real-world robustness — it should not be relied on as a production safeguard against audio fraud without substantially more rigorous evaluation, adversarial testing, and domain-specific validation.
How to Use
from transformers import AutoModelForAudioClassification, AutoFeatureExtractor
model = AutoModelForAudioClassification.from_pretrained("path/to/model")
feature_extractor = AutoFeatureExtractor.from_pretrained("path/to/model")
See backend/predict.py in the main repo for the full inference pipeline
(audio decoding, resampling, and classification).
- Downloads last month
- -