Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
tags:
|
| 3 |
+
- turn-taking-detection
|
| 4 |
+
- audio-classification
|
| 5 |
+
- ecapa-tdnn
|
| 6 |
+
- whisper
|
| 7 |
+
- causal
|
| 8 |
+
license: mit
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# Turn-Taking Detection Models — Collection complète
|
| 12 |
+
|
| 13 |
+
Modeles de detection de fin de tour (HOLD/SHIFT), architecture causale, entraines sur un melange de Smart Turn v3.2 (synthetique) et AppTek (audio reel de centre d'appel, 13 accents anglophones).
|
| 14 |
+
|
| 15 |
+
## Protocole d'evaluation (versions "v3")
|
| 16 |
+
|
| 17 |
+
- **Train** : Smart Turn v3.2 train + AppTek train (13 accents, equilibre HOLD/SHIFT)
|
| 18 |
+
- **Validation** : tranche mixte (1% Smart Turn + 10% AppTek), utilisee uniquement pour la selection du meilleur epoch
|
| 19 |
+
- **Test** : Smart Turn v3.2 test (officiel) + AppTek test (20% du pool, jamais vu a l'entrainement)
|
| 20 |
+
- **OOD** : accent en-CN d'AppTek, totalement exclu de l'entrainement/validation/test
|
| 21 |
+
|
| 22 |
+
## Modeles disponibles
|
| 23 |
+
|
| 24 |
+
### phase1_v32_light.pt (deja depose precedemment)
|
| 25 |
+
|
| 26 |
+
Backbone acoustique causal (ECAPA-TDNN), 462K parametres, entraine uniquement sur Smart Turn v3.2, sans AppTek.
|
| 27 |
+
|
| 28 |
+
| Evaluation | Accuracy | ROC-AUC |
|
| 29 |
+
|---|---|---|
|
| 30 |
+
| Smart Turn v3.2 test (officiel) | 87.42% | 0.9438 |
|
| 31 |
+
|
| 32 |
+
### phase1-ecapa-tdnn-mixed-v3-light.pt / -full.pt
|
| 33 |
+
|
| 34 |
+
Meme architecture, entraine sur le mix Smart Turn + AppTek, protocole train/val/test propre (validation mixte).
|
| 35 |
+
|
| 36 |
+
| Evaluation | Accuracy | Balanced Acc | ROC-AUC |
|
| 37 |
+
|---|---|---|---|
|
| 38 |
+
| Test combine (ST+AppTek) | 84.89% | 84.89% | 0.9289 |
|
| 39 |
+
| — dont Smart Turn test seul | 87.42% | 87.42% | 0.9450 |
|
| 40 |
+
| — dont AppTek test seul (accents vus) | 68.54% | 68.59% | 0.7583 |
|
| 41 |
+
| OOD (en-CN, accent jamais vu) | 59.54% | 59.54% | 0.6587 |
|
| 42 |
+
|
| 43 |
+
### phase2-whisper-crossattn-mixed-v3-light.pt / -full.pt
|
| 44 |
+
|
| 45 |
+
Fusion acoustique-semantique : Phase 1 (gele) + Whisper-tiny encoder (gele) + cross-attention causale + TCN.
|
| 46 |
+
|
| 47 |
+
| Evaluation | Accuracy | Balanced Acc | ROC-AUC |
|
| 48 |
+
|---|---|---|---|
|
| 49 |
+
| Test combine (ST+AppTek) | 90.48% | 90.48% | 0.9672 |
|
| 50 |
+
| — dont Smart Turn test seul | 92.51% | 92.51% | 0.9772 |
|
| 51 |
+
| — dont AppTek test seul (accents vus) | 77.35% | 77.41% | 0.8541 |
|
| 52 |
+
| OOD (en-CN, accent jamais vu) | 71.07% | 71.07% | 0.7909 |
|
| 53 |
+
|
| 54 |
+
## Resultat cle
|
| 55 |
+
|
| 56 |
+
La fusion semantique ameliore la performance de facon croissante avec la difficulte du domaine : +5.09 points sur Smart Turn (synthetique familier), +8.81 points sur AppTek (reel familier), +11.53 points sur en-CN (reel inedit).
|
| 57 |
+
|
| 58 |
+
## Chargement
|
| 59 |
+
|
| 60 |
+
```python
|
| 61 |
+
import torch
|
| 62 |
+
ckpt = torch.load("phase2-whisper-crossattn-mixed-v3-light.pt", map_location="cpu")
|
| 63 |
+
model.load_state_dict(ckpt['best_state'])
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
Pour reprendre l'entrainement (versions "-full") :
|
| 67 |
+
```python
|
| 68 |
+
ckpt = torch.load("phase2-whisper-crossattn-mixed-v3-full.pt", map_location="cpu")
|
| 69 |
+
model.load_state_dict(ckpt['model'])
|
| 70 |
+
optimizer.load_state_dict(ckpt['opt'])
|
| 71 |
+
scheduler.load_state_dict(ckpt['sched'])
|
| 72 |
+
start_epoch = ckpt['epoch'] + 1
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
## Limites connues
|
| 76 |
+
|
| 77 |
+
- Entrainement majoritairement synthetique (Smart Turn ~94% du volume train)
|
| 78 |
+
- Ecart de performance significatif entre donnees synthetiques et reelles, attenue mais non resolu par la fusion semantique
|
| 79 |
+
- Couverture AppTek limitee a l'anglais (14 accents)
|
| 80 |
+
- Whisper-frozen (encodeur gele + tete simple) : run interrompu pour cause d'espace disque, a refaire
|