Commit ·
eeae125
1
Parent(s): 87bf58e
Implement parallel streaming: Orpheus audio + Wav2Lip frames
Browse files- Connect to Orpheus TTS and Wav2Lip in parallel
- Receive audio chunks from Orpheus WebSocket (high quality)
- Receive JPEG frames from Wav2Lip (lip sync with internal eSpeak)
- Discard Wav2Lip audio, use only Orpheus audio
- Mount and send chunks immediately as data arrives
- Add test_streaming.py client for testing
- Update CLAUDE.md with new architecture
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- interface/CLAUDE.md +179 -88
- interface/server.py +271 -221
- interface/test_streaming.py +232 -0
interface/CLAUDE.md
CHANGED
|
@@ -1,31 +1,58 @@
|
|
| 1 |
# Avatar Interface - Sistema de Streaming de Video com Lip Sync
|
| 2 |
|
| 3 |
-
##
|
| 4 |
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
-
|
| 8 |
-
- Wav2Lip (video) e Orpheus TTS (audio) DEVEM ser chamados em PARALELO
|
| 9 |
-
- O servidor deve iniciar ambas as conexoes WebSocket ao mesmo tempo
|
| 10 |
-
- Nao esperar um terminar para comecar o outro
|
| 11 |
-
- Isso reduz a latencia total pela metade
|
| 12 |
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
-
|
| 21 |
-
- O video deve parar exatamente quando o audio termina
|
| 22 |
-
- Usar audioExpectedEndTime para controlar fim dos frames
|
| 23 |
-
- Transicao suave (crossfade 300ms) para o video idle
|
| 24 |
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
|
|
|
| 29 |
|
| 30 |
---
|
| 31 |
|
|
@@ -47,63 +74,66 @@
|
|
| 47 |
|
| 48 |
---
|
| 49 |
|
| 50 |
-
##
|
| 51 |
-
|
| 52 |
-
Sistema de interface web que recebe texto via WebSocket e gera video sincronizado com audio em tempo real (lip sync). O sistema conecta dois servicos:
|
| 53 |
-
|
| 54 |
-
- **TTS (Orpheus)**: Gera audio a partir do texto (PCM 24kHz)
|
| 55 |
-
- **Wav2Lip**: Gera frames de video com sincronizacao labial (JPEG 25fps)
|
| 56 |
-
|
| 57 |
-
## Arquitetura
|
| 58 |
|
| 59 |
```
|
| 60 |
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────┐
|
| 61 |
│ Frontend │◄─────►│ Interface Server │◄─────►│ TTS (Orpheus) │
|
| 62 |
│ (Browser) │ WS │ (porta 8080) │ WS │ (porta 8081) │
|
| 63 |
│ │ │ │ │ PCM 24kHz │
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
|
|
|
| 69 |
└──────────────────────┘ │ JPEG 25fps │
|
|
|
|
| 70 |
└─────────────────┘
|
| 71 |
```
|
| 72 |
|
| 73 |
-
## Fluxo de
|
| 74 |
|
| 75 |
```
|
| 76 |
1. Frontend envia: {"action": "generate", "text": "Hello", "voice": "tara"}
|
| 77 |
│
|
| 78 |
▼
|
| 79 |
-
2. Interface conecta ao TTS (8081) e Wav2Lip (8082)
|
| 80 |
│
|
| 81 |
-
|
| 82 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
│
|
| 84 |
▼
|
| 85 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
│
|
| 87 |
▼
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
d. Mesclar audio + frames em chunk sincronizado
|
| 93 |
-
e. Enviar para frontend: {"type": "chunk", "data": "<base64>", ...}
|
| 94 |
│
|
| 95 |
▼
|
| 96 |
-
|
| 97 |
-
│
|
| 98 |
-
▼
|
| 99 |
-
7. Enviar para frontend: {"type": "done", "total_frames": N, "total_duration_ms": T}
|
| 100 |
```
|
| 101 |
|
| 102 |
-
## Protocolos WebSocket
|
| 103 |
|
| 104 |
### TTS Server (porta 8081) - Orpheus
|
| 105 |
|
| 106 |
-
**
|
| 107 |
|
| 108 |
**Requisicao:**
|
| 109 |
```json
|
|
@@ -136,7 +166,9 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 136 |
}
|
| 137 |
```
|
| 138 |
|
| 139 |
-
**Formato do audio:** PCM 24kHz, 16-bit signed
|
|
|
|
|
|
|
| 140 |
|
| 141 |
---
|
| 142 |
|
|
@@ -144,6 +176,9 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 144 |
|
| 145 |
**Arquivo:** `/home/marcosavatar/realtimeWav2lip/websocket_server.py`
|
| 146 |
|
|
|
|
|
|
|
|
|
|
| 147 |
**Enviar audio (chunk por chunk):**
|
| 148 |
```json
|
| 149 |
{
|
|
@@ -153,7 +188,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 153 |
}
|
| 154 |
```
|
| 155 |
|
| 156 |
-
**Resposta (frames):**
|
| 157 |
```json
|
| 158 |
{"type": "frame", "frame": "<base64 JPEG>", "index": 0}
|
| 159 |
{"type": "frame", "frame": "<base64 JPEG>", "index": 1}
|
|
@@ -179,7 +214,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 179 |
|
| 180 |
---
|
| 181 |
|
| 182 |
-
### Interface Server (porta 8080)
|
| 183 |
|
| 184 |
**Requisicao do frontend:**
|
| 185 |
```json
|
|
@@ -188,9 +223,14 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 188 |
{"action": "ping"}
|
| 189 |
```
|
| 190 |
|
| 191 |
-
**Respostas para o frontend:**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 192 |
```json
|
| 193 |
-
{"type": "
|
| 194 |
```
|
| 195 |
|
| 196 |
```json
|
|
@@ -200,8 +240,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 200 |
"audio_size": 4800,
|
| 201 |
"audio_duration_ms": 100,
|
| 202 |
"num_frames": 2,
|
| 203 |
-
"
|
| 204 |
-
"data": "<base64>"
|
| 205 |
}
|
| 206 |
```
|
| 207 |
|
|
@@ -210,7 +249,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 210 |
"type": "done",
|
| 211 |
"total_duration_ms": 5000,
|
| 212 |
"total_frames": 125,
|
| 213 |
-
"total_chunks":
|
| 214 |
"elapsed_ms": 3500
|
| 215 |
}
|
| 216 |
```
|
|
@@ -219,43 +258,83 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
|
|
| 219 |
{"type": "error", "message": "Descricao do erro"}
|
| 220 |
```
|
| 221 |
|
| 222 |
-
### Formato do Chunk (binario em base64)
|
| 223 |
|
| 224 |
```
|
| 225 |
[audio_size: 4 bytes big-endian]
|
| 226 |
-
[audio_data: PCM 24kHz, 16-bit, mono]
|
|
|
|
| 227 |
[frame_1_size: 4 bytes big-endian]
|
| 228 |
-
[frame_1_data: JPEG bytes]
|
| 229 |
[frame_2_size: 4 bytes big-endian]
|
| 230 |
-
[frame_2_data: JPEG bytes]
|
| 231 |
...
|
| 232 |
```
|
| 233 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 234 |
## Sincronizacao Audio/Video
|
| 235 |
|
| 236 |
### Calculos
|
| 237 |
|
| 238 |
```
|
| 239 |
-
TTS: 24000 Hz (24000 samples/segundo)
|
| 240 |
-
Wav2Lip: 16000 Hz (espera 16000 samples/segundo)
|
| 241 |
Video: 25 fps (1 frame a cada 40ms)
|
| 242 |
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
|
| 246 |
-
- Duracao (ms) = bytes / 2 / 24000 * 1000
|
| 247 |
-
- Frames esperados = duracao_ms / 40
|
| 248 |
```
|
| 249 |
|
| 250 |
-
### Exemplo
|
| 251 |
|
| 252 |
```
|
| 253 |
-
Chunk de
|
| 254 |
-
|
| 255 |
-
|
| 256 |
|
| 257 |
-
|
| 258 |
-
Novos bytes: 4800 * (16000/24000) = 3200 bytes
|
| 259 |
```
|
| 260 |
|
| 261 |
## Estrutura do Projeto
|
|
@@ -263,10 +342,10 @@ Novos bytes: 4800 * (16000/24000) = 3200 bytes
|
|
| 263 |
```
|
| 264 |
/workspace/interface/
|
| 265 |
├── CLAUDE.md # Esta documentacao
|
| 266 |
-
├── server.py # Servidor WebSocket
|
| 267 |
-
├── index.html # Frontend
|
| 268 |
-
├──
|
| 269 |
-
└──
|
| 270 |
```
|
| 271 |
|
| 272 |
## Configuracao
|
|
@@ -276,6 +355,7 @@ Novos bytes: 4800 * (16000/24000) = 3200 bytes
|
|
| 276 |
TTS_WS=ws://localhost:8081/ws
|
| 277 |
WAV2LIP_WS=ws://localhost:8082/ws
|
| 278 |
PORT=8080
|
|
|
|
| 279 |
```
|
| 280 |
|
| 281 |
## Como Executar
|
|
@@ -288,11 +368,12 @@ python3 server.py
|
|
| 288 |
**Output esperado:**
|
| 289 |
```
|
| 290 |
==================================================
|
| 291 |
-
Interface Server -
|
| 292 |
==================================================
|
| 293 |
Porta: 8080
|
| 294 |
-
TTS: ws://localhost:8081/ws (24kHz)
|
| 295 |
-
Wav2Lip: ws://localhost:8082/ws (16kHz)
|
|
|
|
| 296 |
Video: 25fps (40ms/frame)
|
| 297 |
==================================================
|
| 298 |
```
|
|
@@ -309,14 +390,24 @@ Video: 25fps (40ms/frame)
|
|
| 309 |
|
| 310 |
## Resumo das Portas
|
| 311 |
|
| 312 |
-
| Servico | Porta | Audio | Video
|
| 313 |
-
|------------------|-------|--------------|-----------|
|
| 314 |
-
| Interface Server | 8080 |
|
| 315 |
-
| TTS (Orpheus) | 8081 | PCM 24kHz | -
|
| 316 |
-
| Wav2Lip | 8082 |
|
| 317 |
|
| 318 |
## Dependencias Python
|
| 319 |
|
| 320 |
```
|
| 321 |
aiohttp>=3.9.0
|
| 322 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Avatar Interface - Sistema de Streaming de Video com Lip Sync
|
| 2 |
|
| 3 |
+
## ARQUITETURA PRINCIPAL
|
| 4 |
|
| 5 |
+
```
|
| 6 |
+
┌─────────────┐ ┌─────────────────────────────────┐ ┌─────────────────┐
|
| 7 |
+
│ Frontend │◄────►│ Interface Server │◄────►│ Orpheus TTS │
|
| 8 |
+
│ (Browser) │ WS │ (porta 8080) │ WS │ (porta 8081) │
|
| 9 |
+
└─────────────┘ │ │ │ chunks audio │
|
| 10 |
+
│ 1. Recebe texto do frontend │ └─────────────────┘
|
| 11 |
+
│ 2. Conecta Orpheus + Wav2Lip │
|
| 12 |
+
│ EM PARALELO │ ┌─────────────────┐
|
| 13 |
+
│ 3. Recebe chunks conforme │◄────►│ Wav2Lip │
|
| 14 |
+
│ chegam de ambos │ WS │ (porta 8082) │
|
| 15 |
+
│ 4. Monta: audio Orpheus + │ │ frames JPEG │
|
| 16 |
+
│ frames Wav2Lip │ │ (NAO MODIFICAR)│
|
| 17 |
+
│ 5. Envia chunk IMEDIATAMENTE │ └─────────────────┘
|
| 18 |
+
└─────────────────────────────────┘
|
| 19 |
+
```
|
| 20 |
|
| 21 |
+
## FLUXO DE STREAMING
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
+
```
|
| 24 |
+
1. Frontend envia: {"action": "generate", "text": "Hello", "voice": "tara"}
|
| 25 |
+
│
|
| 26 |
+
▼
|
| 27 |
+
2. Interface conecta EM PARALELO:
|
| 28 |
+
├── Orpheus WS (8081) → recebe chunks de audio PCM 24kHz
|
| 29 |
+
└── Wav2Lip WS (8082) → recebe frames JPEG (lip sync com eSpeak interno)
|
| 30 |
+
│
|
| 31 |
+
▼
|
| 32 |
+
3. Conforme dados chegam, acumula em buffers:
|
| 33 |
+
- audio_buffer: chunks PCM do Orpheus
|
| 34 |
+
- frame_buffer: frames JPEG do Wav2Lip
|
| 35 |
+
│
|
| 36 |
+
▼
|
| 37 |
+
4. Quando tem 1 frame + audio correspondente (~1920 bytes = 40ms):
|
| 38 |
+
- Monta chunk binario: [audio_orpheus + frame_wav2lip]
|
| 39 |
+
- Envia IMEDIATAMENTE para frontend
|
| 40 |
+
│
|
| 41 |
+
▼
|
| 42 |
+
5. Frontend reproduz em tempo real (nao espera tudo chegar)
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
## REGRAS CRITICAS
|
| 46 |
+
|
| 47 |
+
1. **NAO MODIFICAR O WAV2LIP** - Ele ja gera lip sync com eSpeak interno. So usar os frames JPEG.
|
| 48 |
|
| 49 |
+
2. **AUDIO VEM DO ORPHEUS** - O audio robotico do Wav2Lip e DESCARTADO. Audio final = Orpheus.
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
+
3. **CONEXOES EM PARALELO** - Orpheus e Wav2Lip devem ser chamados ao mesmo tempo.
|
| 52 |
+
|
| 53 |
+
4. **STREAMING IMEDIATO** - Montar e enviar chunks conforme dados chegam, nao esperar tudo.
|
| 54 |
+
|
| 55 |
+
5. **SINCRONIZACAO** - Calcular audio_por_frame = total_audio / total_frames para alinhar.
|
| 56 |
|
| 57 |
---
|
| 58 |
|
|
|
|
| 74 |
|
| 75 |
---
|
| 76 |
|
| 77 |
+
## Arquitetura de Streaming Progressivo
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
```
|
| 80 |
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────┐
|
| 81 |
│ Frontend │◄─────►│ Interface Server │◄─────►│ TTS (Orpheus) │
|
| 82 |
│ (Browser) │ WS │ (porta 8080) │ WS │ (porta 8081) │
|
| 83 |
│ │ │ │ │ PCM 24kHz │
|
| 84 |
+
│ Renderiza │ │ MONTA CHUNKS: │ │ (audio final) │
|
| 85 |
+
│ chunks │ │ audio Orpheus + │ └─────────────────┘
|
| 86 |
+
│ conforme │ │ frames Wav2Lip │
|
| 87 |
+
│ chegam │ │ │ ┌─────────────────┐
|
| 88 |
+
└─────────────────┘ │ DESCARTA audio │◄─────►│ Wav2Lip │
|
| 89 |
+
│ robotico! │ WS │ (porta 8082) │
|
| 90 |
└──────────────────────┘ │ JPEG 25fps │
|
| 91 |
+
│ (so frames!) │
|
| 92 |
└─────────────────┘
|
| 93 |
```
|
| 94 |
|
| 95 |
+
## Fluxo de Streaming Progressivo
|
| 96 |
|
| 97 |
```
|
| 98 |
1. Frontend envia: {"action": "generate", "text": "Hello", "voice": "tara"}
|
| 99 |
│
|
| 100 |
▼
|
| 101 |
+
2. Interface conecta ao TTS (8081) e Wav2Lip (8082) EM PARALELO
|
| 102 |
│
|
| 103 |
+
├──────────────────────────────────────────────────────────┐
|
| 104 |
+
│ │
|
| 105 |
+
▼ ▼
|
| 106 |
+
3a. Orpheus TTS gera audio 3b. Wav2Lip gera frames
|
| 107 |
+
em chunks (~100ms cada) baseado em audio interno
|
| 108 |
+
│ (eSpeak robotico - DESCARTAR)
|
| 109 |
+
│ │
|
| 110 |
+
▼ ▼
|
| 111 |
+
4. Interface Server acumula em buffers:
|
| 112 |
+
- audio_buffer: chunks do Orpheus
|
| 113 |
+
- frame_buffer: frames JPEG do Wav2Lip
|
| 114 |
│
|
| 115 |
▼
|
| 116 |
+
5. Quando tiver dados suficientes (~100ms):
|
| 117 |
+
a. Pegar audio do Orpheus (ex: 4800 bytes = 100ms)
|
| 118 |
+
b. Pegar frames do Wav2Lip (ex: 2-3 frames = 80-120ms)
|
| 119 |
+
c. Montar chunk: [audio_orpheus + frames]
|
| 120 |
+
d. Enviar para frontend IMEDIATAMENTE
|
| 121 |
│
|
| 122 |
▼
|
| 123 |
+
6. Frontend recebe chunk e renderiza:
|
| 124 |
+
- Decodifica audio e adiciona ao buffer de playback
|
| 125 |
+
- Decodifica frames JPEG e adiciona a fila de exibicao
|
| 126 |
+
- Inicia playback assim que tiver ~200ms de dados
|
|
|
|
|
|
|
| 127 |
│
|
| 128 |
▼
|
| 129 |
+
7. Repetir ate receber "done" de ambos os servicos
|
|
|
|
|
|
|
|
|
|
| 130 |
```
|
| 131 |
|
| 132 |
+
## Protocolos WebSocket
|
| 133 |
|
| 134 |
### TTS Server (porta 8081) - Orpheus
|
| 135 |
|
| 136 |
+
**Conexao:** `ws://localhost:8081/ws`
|
| 137 |
|
| 138 |
**Requisicao:**
|
| 139 |
```json
|
|
|
|
| 166 |
}
|
| 167 |
```
|
| 168 |
|
| 169 |
+
**Formato do audio:** PCM 24kHz, 16-bit signed little-endian, mono
|
| 170 |
+
|
| 171 |
+
**Vozes:** tara, leo, leah, jess, dan, mia, zac, zoe
|
| 172 |
|
| 173 |
---
|
| 174 |
|
|
|
|
| 176 |
|
| 177 |
**Arquivo:** `/home/marcosavatar/realtimeWav2lip/websocket_server.py`
|
| 178 |
|
| 179 |
+
**IMPORTANTE:** O Wav2Lip usa audio interno (eSpeak) para gerar lip sync.
|
| 180 |
+
Este audio robotico DEVE SER DESCARTADO - usar APENAS os frames JPEG!
|
| 181 |
+
|
| 182 |
**Enviar audio (chunk por chunk):**
|
| 183 |
```json
|
| 184 |
{
|
|
|
|
| 188 |
}
|
| 189 |
```
|
| 190 |
|
| 191 |
+
**Resposta (frames) - USAR APENAS OS FRAMES:**
|
| 192 |
```json
|
| 193 |
{"type": "frame", "frame": "<base64 JPEG>", "index": 0}
|
| 194 |
{"type": "frame", "frame": "<base64 JPEG>", "index": 1}
|
|
|
|
| 214 |
|
| 215 |
---
|
| 216 |
|
| 217 |
+
### Interface Server (porta 8080) - Frontend
|
| 218 |
|
| 219 |
**Requisicao do frontend:**
|
| 220 |
```json
|
|
|
|
| 223 |
{"action": "ping"}
|
| 224 |
```
|
| 225 |
|
| 226 |
+
**Respostas para o frontend (STREAMING PROGRESSIVO):**
|
| 227 |
+
|
| 228 |
+
```json
|
| 229 |
+
{"type": "status", "message": "Conectando aos servicos..."}
|
| 230 |
+
```
|
| 231 |
+
|
| 232 |
```json
|
| 233 |
+
{"type": "stream_start", "ttfb_ms": 150}
|
| 234 |
```
|
| 235 |
|
| 236 |
```json
|
|
|
|
| 240 |
"audio_size": 4800,
|
| 241 |
"audio_duration_ms": 100,
|
| 242 |
"num_frames": 2,
|
| 243 |
+
"data": "<base64 do chunk montado>"
|
|
|
|
| 244 |
}
|
| 245 |
```
|
| 246 |
|
|
|
|
| 249 |
"type": "done",
|
| 250 |
"total_duration_ms": 5000,
|
| 251 |
"total_frames": 125,
|
| 252 |
+
"total_chunks": 50,
|
| 253 |
"elapsed_ms": 3500
|
| 254 |
}
|
| 255 |
```
|
|
|
|
| 258 |
{"type": "error", "message": "Descricao do erro"}
|
| 259 |
```
|
| 260 |
|
| 261 |
+
### Formato do Chunk Montado (binario em base64)
|
| 262 |
|
| 263 |
```
|
| 264 |
[audio_size: 4 bytes big-endian]
|
| 265 |
+
[audio_data: PCM 24kHz do ORPHEUS, 16-bit, mono]
|
| 266 |
+
[num_frames: 4 bytes big-endian]
|
| 267 |
[frame_1_size: 4 bytes big-endian]
|
| 268 |
+
[frame_1_data: JPEG bytes do WAV2LIP]
|
| 269 |
[frame_2_size: 4 bytes big-endian]
|
| 270 |
+
[frame_2_data: JPEG bytes do WAV2LIP]
|
| 271 |
...
|
| 272 |
```
|
| 273 |
|
| 274 |
+
## Algoritmo de Montagem de Chunks
|
| 275 |
+
|
| 276 |
+
```python
|
| 277 |
+
CHUNK_DURATION_MS = 100 # Tamanho alvo de cada chunk
|
| 278 |
+
AUDIO_SAMPLE_RATE = 24000 # Orpheus
|
| 279 |
+
VIDEO_FPS = 25 # Wav2Lip
|
| 280 |
+
BYTES_PER_SAMPLE = 2 # 16-bit
|
| 281 |
+
|
| 282 |
+
# Buffers
|
| 283 |
+
audio_buffer = bytearray() # Audio do Orpheus
|
| 284 |
+
frame_buffer = [] # Frames do Wav2Lip
|
| 285 |
+
|
| 286 |
+
async def process_streaming():
|
| 287 |
+
# Conectar em PARALELO
|
| 288 |
+
orpheus_task = asyncio.create_task(connect_orpheus())
|
| 289 |
+
wav2lip_task = asyncio.create_task(connect_wav2lip())
|
| 290 |
+
|
| 291 |
+
while not done:
|
| 292 |
+
# Verificar se tem dados suficientes para montar chunk
|
| 293 |
+
audio_bytes_needed = int(CHUNK_DURATION_MS * AUDIO_SAMPLE_RATE * BYTES_PER_SAMPLE / 1000)
|
| 294 |
+
frames_needed = int(CHUNK_DURATION_MS * VIDEO_FPS / 1000)
|
| 295 |
+
|
| 296 |
+
if len(audio_buffer) >= audio_bytes_needed and len(frame_buffer) >= frames_needed:
|
| 297 |
+
# Extrair dados dos buffers
|
| 298 |
+
audio_chunk = audio_buffer[:audio_bytes_needed]
|
| 299 |
+
del audio_buffer[:audio_bytes_needed]
|
| 300 |
+
|
| 301 |
+
frames_chunk = frame_buffer[:frames_needed]
|
| 302 |
+
del frame_buffer[:frames_needed]
|
| 303 |
+
|
| 304 |
+
# Montar e enviar chunk
|
| 305 |
+
chunk_data = build_chunk(audio_chunk, frames_chunk)
|
| 306 |
+
await ws.send_json({
|
| 307 |
+
"type": "chunk",
|
| 308 |
+
"chunk_index": chunk_index,
|
| 309 |
+
"audio_duration_ms": CHUNK_DURATION_MS,
|
| 310 |
+
"num_frames": len(frames_chunk),
|
| 311 |
+
"data": base64.b64encode(chunk_data).decode()
|
| 312 |
+
})
|
| 313 |
+
chunk_index += 1
|
| 314 |
+
```
|
| 315 |
+
|
| 316 |
## Sincronizacao Audio/Video
|
| 317 |
|
| 318 |
### Calculos
|
| 319 |
|
| 320 |
```
|
| 321 |
+
TTS Orpheus: 24000 Hz (24000 samples/segundo)
|
| 322 |
+
Wav2Lip: 16000 Hz (espera 16000 samples/segundo para lip sync)
|
| 323 |
Video: 25 fps (1 frame a cada 40ms)
|
| 324 |
|
| 325 |
+
Para chunk de 100ms:
|
| 326 |
+
- Audio Orpheus: 100 * 24000 / 1000 * 2 = 4800 bytes
|
| 327 |
+
- Frames Wav2Lip: 100 / 40 = 2.5 ≈ 2-3 frames
|
|
|
|
|
|
|
| 328 |
```
|
| 329 |
|
| 330 |
+
### Exemplo de Chunk
|
| 331 |
|
| 332 |
```
|
| 333 |
+
Chunk de 100ms:
|
| 334 |
+
- Audio: 4800 bytes PCM 24kHz do Orpheus (voz de alta qualidade)
|
| 335 |
+
- Frames: 2-3 JPEGs do Wav2Lip (lip sync)
|
| 336 |
|
| 337 |
+
Total por chunk: ~50-80 KB (depende da compressao JPEG)
|
|
|
|
| 338 |
```
|
| 339 |
|
| 340 |
## Estrutura do Projeto
|
|
|
|
| 342 |
```
|
| 343 |
/workspace/interface/
|
| 344 |
├── CLAUDE.md # Esta documentacao
|
| 345 |
+
├── server.py # Servidor WebSocket com montagem de chunks
|
| 346 |
+
├── index.html # Frontend com playback progressivo
|
| 347 |
+
├── idle.mp4 # Video de idle loop
|
| 348 |
+
└── static/ # Arquivos estaticos
|
| 349 |
```
|
| 350 |
|
| 351 |
## Configuracao
|
|
|
|
| 355 |
TTS_WS=ws://localhost:8081/ws
|
| 356 |
WAV2LIP_WS=ws://localhost:8082/ws
|
| 357 |
PORT=8080
|
| 358 |
+
CHUNK_DURATION_MS=100
|
| 359 |
```
|
| 360 |
|
| 361 |
## Como Executar
|
|
|
|
| 368 |
**Output esperado:**
|
| 369 |
```
|
| 370 |
==================================================
|
| 371 |
+
Interface Server - Streaming Progressivo
|
| 372 |
==================================================
|
| 373 |
Porta: 8080
|
| 374 |
+
TTS (Orpheus): ws://localhost:8081/ws (24kHz)
|
| 375 |
+
Wav2Lip: ws://localhost:8082/ws (16kHz, apenas frames)
|
| 376 |
+
Chunk duration: 100ms
|
| 377 |
Video: 25fps (40ms/frame)
|
| 378 |
==================================================
|
| 379 |
```
|
|
|
|
| 390 |
|
| 391 |
## Resumo das Portas
|
| 392 |
|
| 393 |
+
| Servico | Porta | Audio | Video |
|
| 394 |
+
|------------------|-------|--------------|------------|
|
| 395 |
+
| Interface Server | 8080 | Monta chunks | - |
|
| 396 |
+
| TTS (Orpheus) | 8081 | PCM 24kHz | - |
|
| 397 |
+
| Wav2Lip | 8082 | DESCARTAR! | JPEG 25fps |
|
| 398 |
|
| 399 |
## Dependencias Python
|
| 400 |
|
| 401 |
```
|
| 402 |
aiohttp>=3.9.0
|
| 403 |
```
|
| 404 |
+
|
| 405 |
+
## Notas Importantes
|
| 406 |
+
|
| 407 |
+
1. **NUNCA usar audio do Wav2Lip no frontend** - O Wav2Lip usa eSpeak internamente para gerar lip sync. Este audio e robotico e deve ser DESCARTADO. Usar APENAS os frames JPEG.
|
| 408 |
+
|
| 409 |
+
2. **SEMPRE usar audio do Orpheus** - O audio final enviado ao frontend deve vir do Orpheus TTS, que gera voz de alta qualidade.
|
| 410 |
+
|
| 411 |
+
3. **Streaming progressivo e obrigatorio** - Nao esperar todo o audio/video ficar pronto. Montar e enviar chunks de ~100ms conforme os dados chegam.
|
| 412 |
+
|
| 413 |
+
4. **Frontend deve iniciar playback cedo** - Assim que receber ~200ms de dados (2 chunks), iniciar reproducao enquanto continua recebendo.
|
interface/server.py
CHANGED
|
@@ -1,13 +1,17 @@
|
|
| 1 |
"""
|
| 2 |
-
Interface Server -
|
| 3 |
Porta: 8080
|
| 4 |
|
| 5 |
-
|
| 6 |
-
1.
|
| 7 |
-
2.
|
| 8 |
-
3.
|
| 9 |
-
4.
|
| 10 |
-
5.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
"""
|
| 12 |
from aiohttp import web
|
| 13 |
import aiohttp
|
|
@@ -16,163 +20,216 @@ import json
|
|
| 16 |
import base64
|
| 17 |
import os
|
| 18 |
import time
|
| 19 |
-
import subprocess
|
| 20 |
-
import tempfile
|
| 21 |
import struct
|
| 22 |
|
| 23 |
-
# Configuracao
|
|
|
|
| 24 |
WAV2LIP_WS = os.getenv("WAV2LIP_WS", "ws://localhost:8082/ws")
|
| 25 |
PORT = int(os.getenv("PORT", "8080"))
|
| 26 |
|
| 27 |
-
#
|
| 28 |
-
|
| 29 |
-
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
routes = web.RouteTableDef()
|
| 32 |
|
| 33 |
|
| 34 |
-
|
| 35 |
-
"""
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
self.temp_dir = tempfile.mkdtemp()
|
| 45 |
-
self.frames = []
|
| 46 |
-
self.audio_data = None
|
| 47 |
-
|
| 48 |
-
def add_frame(self, jpeg_data: bytes):
|
| 49 |
-
self.frames.append(jpeg_data)
|
| 50 |
-
|
| 51 |
-
def set_audio(self, pcm_data: bytes, sample_rate: int = 24000):
|
| 52 |
-
self.audio_data = pcm_data
|
| 53 |
-
self.sample_rate = sample_rate
|
| 54 |
-
|
| 55 |
-
async def finalize(self) -> bytes:
|
| 56 |
-
if not self.frames or not self.audio_data:
|
| 57 |
-
return b""
|
| 58 |
-
|
| 59 |
-
# Salvar frames
|
| 60 |
-
for i, frame in enumerate(self.frames):
|
| 61 |
-
frame_path = os.path.join(self.temp_dir, f"frame_{i:05d}.jpg")
|
| 62 |
-
with open(frame_path, 'wb') as f:
|
| 63 |
-
f.write(frame)
|
| 64 |
-
|
| 65 |
-
# Salvar audio como WAV
|
| 66 |
-
audio_path = os.path.join(self.temp_dir, "audio.wav")
|
| 67 |
-
with open(audio_path, 'wb') as f:
|
| 68 |
-
f.write(b'RIFF')
|
| 69 |
-
f.write(struct.pack('<I', 36 + len(self.audio_data)))
|
| 70 |
-
f.write(b'WAVE')
|
| 71 |
-
f.write(b'fmt ')
|
| 72 |
-
f.write(struct.pack('<I', 16))
|
| 73 |
-
f.write(struct.pack('<H', 1))
|
| 74 |
-
f.write(struct.pack('<H', 1))
|
| 75 |
-
f.write(struct.pack('<I', self.sample_rate))
|
| 76 |
-
f.write(struct.pack('<I', self.sample_rate * 2))
|
| 77 |
-
f.write(struct.pack('<H', 2))
|
| 78 |
-
f.write(struct.pack('<H', 16))
|
| 79 |
-
f.write(b'data')
|
| 80 |
-
f.write(struct.pack('<I', len(self.audio_data)))
|
| 81 |
-
f.write(self.audio_data)
|
| 82 |
-
|
| 83 |
-
output_path = os.path.join(self.temp_dir, "output.webm")
|
| 84 |
-
|
| 85 |
-
cmd = [
|
| 86 |
-
'ffmpeg', '-y',
|
| 87 |
-
'-framerate', str(VIDEO_FPS),
|
| 88 |
-
'-i', os.path.join(self.temp_dir, 'frame_%05d.jpg'),
|
| 89 |
-
'-i', audio_path,
|
| 90 |
-
'-c:v', 'libvpx-vp9',
|
| 91 |
-
'-b:v', '1M',
|
| 92 |
-
'-c:a', 'libopus',
|
| 93 |
-
'-b:a', '128k',
|
| 94 |
-
'-shortest',
|
| 95 |
-
output_path
|
| 96 |
-
]
|
| 97 |
-
|
| 98 |
-
try:
|
| 99 |
-
proc = await asyncio.create_subprocess_exec(
|
| 100 |
-
*cmd,
|
| 101 |
-
stdout=asyncio.subprocess.PIPE,
|
| 102 |
-
stderr=asyncio.subprocess.PIPE
|
| 103 |
-
)
|
| 104 |
-
stdout, stderr = await proc.communicate()
|
| 105 |
-
|
| 106 |
-
if proc.returncode != 0:
|
| 107 |
-
print(f"FFmpeg error: {stderr.decode()[-500:]}")
|
| 108 |
-
return b""
|
| 109 |
-
|
| 110 |
-
with open(output_path, 'rb') as f:
|
| 111 |
-
return f.read()
|
| 112 |
-
|
| 113 |
-
except Exception as e:
|
| 114 |
-
print(f"Error creating WebM: {e}")
|
| 115 |
-
return b""
|
| 116 |
-
|
| 117 |
-
async def cleanup(self):
|
| 118 |
-
if self.temp_dir:
|
| 119 |
-
import shutil
|
| 120 |
-
try:
|
| 121 |
-
shutil.rmtree(self.temp_dir)
|
| 122 |
-
except:
|
| 123 |
-
pass
|
| 124 |
|
| 125 |
|
| 126 |
-
class
|
| 127 |
-
"""
|
| 128 |
|
| 129 |
def __init__(self, client_ws):
|
| 130 |
self.client_ws = client_ws
|
| 131 |
self.is_running = False
|
| 132 |
self.start_time = None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
self.total_frames = 0
|
| 134 |
self.chunks_sent = 0
|
| 135 |
|
| 136 |
-
async def
|
|
|
|
| 137 |
try:
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
self.
|
| 146 |
-
|
| 147 |
-
await self.send_to_client("status", message="Conectando ao Wav2Lip...")
|
| 148 |
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 152 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
try:
|
|
|
|
| 154 |
async with aiohttp.ClientSession() as session:
|
| 155 |
-
|
| 156 |
-
WAV2LIP_WS,
|
| 157 |
-
timeout=aiohttp.ClientWSTimeout(ws_close=120)
|
| 158 |
-
)
|
| 159 |
-
|
| 160 |
-
await self.send_to_client("status", message="Gerando avatar...")
|
| 161 |
|
| 162 |
-
# Enviar
|
| 163 |
-
await
|
| 164 |
-
"action": "
|
| 165 |
"text": text,
|
| 166 |
"voice": voice,
|
| 167 |
-
"
|
| 168 |
})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
|
| 170 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
|
| 172 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
|
| 174 |
-
|
| 175 |
-
async for msg in wav2lip_ws:
|
| 176 |
if not self.is_running:
|
| 177 |
break
|
| 178 |
|
|
@@ -180,105 +237,92 @@ class StreamingSession:
|
|
| 180 |
data = json.loads(msg.data)
|
| 181 |
msg_type = data.get("type", "")
|
| 182 |
|
| 183 |
-
if msg_type == "
|
| 184 |
-
|
| 185 |
-
print(f"[Wav2Lip] First chunk: {latency}ms")
|
| 186 |
-
await self.send_to_client("status", message="Processando frames...")
|
| 187 |
-
|
| 188 |
-
elif msg_type == "frame":
|
| 189 |
frame_b64 = data.get("frame", "")
|
| 190 |
frame_bytes = base64.b64decode(frame_b64)
|
| 191 |
-
frames.append(frame_bytes)
|
| 192 |
-
self.total_frames += 1
|
| 193 |
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 198 |
|
| 199 |
elif msg_type == "full_audio":
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
# Extrair PCM do WAV (pular header de 44 bytes)
|
| 203 |
-
if len(audio_wav) > 44 and audio_wav[:4] == b'RIFF':
|
| 204 |
-
audio_data = audio_wav[44:]
|
| 205 |
-
else:
|
| 206 |
-
audio_data = audio_wav
|
| 207 |
-
audio_sample_rate = 24000
|
| 208 |
-
duration = data.get("duration_ms", 0)
|
| 209 |
-
print(f"[Wav2Lip] Audio recebido: {len(audio_data)} bytes, {duration}ms")
|
| 210 |
|
| 211 |
elif msg_type == "done":
|
| 212 |
-
|
|
|
|
| 213 |
break
|
| 214 |
|
| 215 |
-
elif msg_type == "status":
|
| 216 |
-
print(f"[Wav2Lip] Status: {data.get('message')}")
|
| 217 |
-
|
| 218 |
elif msg_type == "error":
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
await self.send_to_client("error", message=error_msg)
|
| 222 |
break
|
| 223 |
|
| 224 |
elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
|
| 225 |
break
|
| 226 |
|
| 227 |
-
await
|
| 228 |
|
| 229 |
except Exception as e:
|
| 230 |
-
print(f"
|
| 231 |
import traceback
|
| 232 |
traceback.print_exc()
|
| 233 |
-
await self.send_to_client("error", message=str(e))
|
| 234 |
-
return
|
| 235 |
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
|
|
|
|
| 243 |
|
| 244 |
-
|
| 245 |
-
await self.send_to_client("status", message="Gerando video...")
|
| 246 |
|
| 247 |
-
|
| 248 |
-
|
|
|
|
|
|
|
| 249 |
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
webm.set_audio(audio_data, audio_sample_rate)
|
| 253 |
|
| 254 |
-
|
| 255 |
-
|
| 256 |
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
await self.
|
| 260 |
-
|
| 261 |
-
chunk_index=1,
|
| 262 |
-
size=len(webm_data),
|
| 263 |
-
data=base64.b64encode(webm_data).decode()
|
| 264 |
-
)
|
| 265 |
-
print(f"[WebM] Enviado: {len(webm_data)} bytes, {len(frames)} frames")
|
| 266 |
|
| 267 |
-
# Metricas finais
|
| 268 |
elapsed = time.time() - self.start_time
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
print(f"Sessao concluida em {elapsed:.2f}s")
|
| 272 |
-
print(f" Frames: {self.total_frames}")
|
| 273 |
-
print(f" Audio: {audio_duration_ms:.0f}ms")
|
| 274 |
|
| 275 |
-
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
| 281 |
-
|
|
|
|
|
|
|
|
|
|
| 282 |
|
| 283 |
def stop(self):
|
| 284 |
self.is_running = False
|
|
@@ -289,7 +333,7 @@ async def websocket_handler(request):
|
|
| 289 |
ws = web.WebSocketResponse()
|
| 290 |
await ws.prepare(request)
|
| 291 |
|
| 292 |
-
print("Cliente
|
| 293 |
session = None
|
| 294 |
|
| 295 |
try:
|
|
@@ -304,7 +348,7 @@ async def websocket_handler(request):
|
|
| 304 |
voice = data.get("voice", "tara")
|
| 305 |
|
| 306 |
if not text:
|
| 307 |
-
await ws.send_json({"type": "error", "message": "Text
|
| 308 |
continue
|
| 309 |
|
| 310 |
print(f"Gerando: '{text[:50]}...' voice={voice}")
|
|
@@ -312,7 +356,7 @@ async def websocket_handler(request):
|
|
| 312 |
if session:
|
| 313 |
session.stop()
|
| 314 |
|
| 315 |
-
session =
|
| 316 |
await session.run(text, voice)
|
| 317 |
|
| 318 |
elif action == "stop":
|
|
@@ -331,11 +375,11 @@ async def websocket_handler(request):
|
|
| 331 |
break
|
| 332 |
|
| 333 |
except Exception as e:
|
| 334 |
-
print(f"Erro
|
| 335 |
finally:
|
| 336 |
if session:
|
| 337 |
session.stop()
|
| 338 |
-
print("Cliente
|
| 339 |
|
| 340 |
return ws
|
| 341 |
|
|
@@ -345,18 +389,23 @@ async def index(request):
|
|
| 345 |
return web.FileResponse(os.path.join(os.path.dirname(__file__), "index.html"))
|
| 346 |
|
| 347 |
|
| 348 |
-
@routes.get("/
|
| 349 |
-
async def
|
| 350 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 351 |
|
| 352 |
-
try:
|
| 353 |
-
async with aiohttp.ClientSession() as session:
|
| 354 |
-
async with session.ws_connect(WAV2LIP_WS, timeout=aiohttp.ClientWSTimeout(ws_close=2)) as ws:
|
| 355 |
-
status["wav2lip"] = True
|
| 356 |
-
except:
|
| 357 |
-
pass
|
| 358 |
|
| 359 |
-
|
|
|
|
|
|
|
| 360 |
|
| 361 |
|
| 362 |
app = web.Application()
|
|
@@ -365,11 +414,12 @@ app.add_routes(routes)
|
|
| 365 |
|
| 366 |
if __name__ == "__main__":
|
| 367 |
print("=" * 50)
|
| 368 |
-
print("Interface Server -
|
| 369 |
print("=" * 50)
|
| 370 |
print(f"Porta: {PORT}")
|
|
|
|
| 371 |
print(f"Wav2Lip: {WAV2LIP_WS}")
|
| 372 |
-
print(f"
|
| 373 |
-
print(f"
|
| 374 |
print("=" * 50)
|
| 375 |
web.run_app(app, host="0.0.0.0", port=PORT)
|
|
|
|
| 1 |
"""
|
| 2 |
+
Interface Server - Streaming Paralelo
|
| 3 |
Porta: 8080
|
| 4 |
|
| 5 |
+
Arquitetura (ver CLAUDE.md):
|
| 6 |
+
1. Recebe texto do frontend via WebSocket
|
| 7 |
+
2. Conecta ao Orpheus (8081) e Wav2Lip (8082) EM PARALELO
|
| 8 |
+
3. Recebe chunks de audio do Orpheus e frames do Wav2Lip
|
| 9 |
+
4. Monta chunks (audio Orpheus + frames Wav2Lip) conforme chegam
|
| 10 |
+
5. Envia chunks IMEDIATAMENTE para o frontend
|
| 11 |
+
|
| 12 |
+
IMPORTANTE:
|
| 13 |
+
- NAO modificar Wav2Lip - ele gera lip sync com eSpeak interno
|
| 14 |
+
- Audio final = Orpheus (descartar audio do Wav2Lip)
|
| 15 |
"""
|
| 16 |
from aiohttp import web
|
| 17 |
import aiohttp
|
|
|
|
| 20 |
import base64
|
| 21 |
import os
|
| 22 |
import time
|
|
|
|
|
|
|
| 23 |
import struct
|
| 24 |
|
| 25 |
+
# Configuracao
|
| 26 |
+
ORPHEUS_WS = os.getenv("ORPHEUS_WS", "ws://localhost:8081/ws")
|
| 27 |
WAV2LIP_WS = os.getenv("WAV2LIP_WS", "ws://localhost:8082/ws")
|
| 28 |
PORT = int(os.getenv("PORT", "8080"))
|
| 29 |
|
| 30 |
+
# Constantes
|
| 31 |
+
AUDIO_SAMPLE_RATE = 24000 # Orpheus
|
| 32 |
+
VIDEO_FPS = 25 # Wav2Lip
|
| 33 |
+
BYTES_PER_SAMPLE = 2 # 16-bit
|
| 34 |
+
MS_PER_FRAME = 1000 / VIDEO_FPS # 40ms
|
| 35 |
+
AUDIO_BYTES_PER_FRAME = int(MS_PER_FRAME * AUDIO_SAMPLE_RATE * BYTES_PER_SAMPLE / 1000) # 1920 bytes
|
| 36 |
|
| 37 |
routes = web.RouteTableDef()
|
| 38 |
|
| 39 |
|
| 40 |
+
def build_chunk(audio_bytes: bytes, frames: list) -> bytes:
|
| 41 |
+
"""Monta chunk binario: [audio_size][audio][num_frames][frame_sizes][frames]"""
|
| 42 |
+
data = bytearray()
|
| 43 |
+
data.extend(struct.pack('>I', len(audio_bytes)))
|
| 44 |
+
data.extend(audio_bytes)
|
| 45 |
+
data.extend(struct.pack('>I', len(frames)))
|
| 46 |
+
for frame in frames:
|
| 47 |
+
data.extend(struct.pack('>I', len(frame)))
|
| 48 |
+
data.extend(frame)
|
| 49 |
+
return bytes(data)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
|
| 52 |
+
class ParallelStreamingSession:
|
| 53 |
+
"""Streaming paralelo: Orpheus (audio) + Wav2Lip (frames)"""
|
| 54 |
|
| 55 |
def __init__(self, client_ws):
|
| 56 |
self.client_ws = client_ws
|
| 57 |
self.is_running = False
|
| 58 |
self.start_time = None
|
| 59 |
+
|
| 60 |
+
# Buffers compartilhados
|
| 61 |
+
self.audio_buffer = bytearray()
|
| 62 |
+
self.frame_buffer = []
|
| 63 |
+
|
| 64 |
+
# Locks para acesso thread-safe
|
| 65 |
+
self.buffer_lock = asyncio.Lock()
|
| 66 |
+
|
| 67 |
+
# Estados
|
| 68 |
+
self.orpheus_done = False
|
| 69 |
+
self.wav2lip_done = False
|
| 70 |
+
self.first_chunk_sent = False
|
| 71 |
+
self.chunk_index = 0
|
| 72 |
+
|
| 73 |
+
# Estatisticas
|
| 74 |
+
self.total_audio_bytes = 0
|
| 75 |
self.total_frames = 0
|
| 76 |
self.chunks_sent = 0
|
| 77 |
|
| 78 |
+
async def send_status(self, message: str):
|
| 79 |
+
"""Envia status para o cliente."""
|
| 80 |
try:
|
| 81 |
+
if not self.client_ws.closed:
|
| 82 |
+
await self.client_ws.send_json({"type": "status", "message": message})
|
| 83 |
+
except:
|
| 84 |
+
pass
|
| 85 |
+
|
| 86 |
+
async def send_chunk(self, audio: bytes, frames: list):
|
| 87 |
+
"""Monta e envia chunk para o cliente."""
|
| 88 |
+
if self.client_ws.closed or not audio or not frames:
|
| 89 |
+
return
|
|
|
|
| 90 |
|
| 91 |
+
try:
|
| 92 |
+
chunk_data = build_chunk(audio, frames)
|
| 93 |
+
chunk_b64 = base64.b64encode(chunk_data).decode()
|
| 94 |
+
audio_ms = len(audio) / BYTES_PER_SAMPLE / AUDIO_SAMPLE_RATE * 1000
|
| 95 |
+
|
| 96 |
+
await self.client_ws.send_json({
|
| 97 |
+
"type": "chunk",
|
| 98 |
+
"chunk_index": self.chunk_index,
|
| 99 |
+
"audio_size": len(audio),
|
| 100 |
+
"audio_duration_ms": int(audio_ms),
|
| 101 |
+
"num_frames": len(frames),
|
| 102 |
+
"data": chunk_b64
|
| 103 |
+
})
|
| 104 |
+
|
| 105 |
+
self.chunk_index += 1
|
| 106 |
+
self.chunks_sent += 1
|
| 107 |
+
|
| 108 |
+
if not self.first_chunk_sent:
|
| 109 |
+
self.first_chunk_sent = True
|
| 110 |
+
ttfb = int((time.time() - self.start_time) * 1000)
|
| 111 |
+
print(f"[Stream] Primeiro chunk: TTFB={ttfb}ms")
|
| 112 |
+
await self.client_ws.send_json({"type": "stream_start", "ttfb_ms": ttfb})
|
| 113 |
|
| 114 |
+
except Exception as e:
|
| 115 |
+
print(f"[Stream] Erro enviando chunk: {e}")
|
| 116 |
+
|
| 117 |
+
async def try_send_chunks(self):
|
| 118 |
+
"""Tenta montar e enviar chunks com dados disponiveis."""
|
| 119 |
+
async with self.buffer_lock:
|
| 120 |
+
# Enquanto tiver 1 frame + audio correspondente
|
| 121 |
+
while len(self.frame_buffer) > 0 and len(self.audio_buffer) >= AUDIO_BYTES_PER_FRAME:
|
| 122 |
+
# Pega 1 frame
|
| 123 |
+
frame = self.frame_buffer.pop(0)
|
| 124 |
+
|
| 125 |
+
# Pega audio correspondente (~40ms = 1920 bytes)
|
| 126 |
+
audio = bytes(self.audio_buffer[:AUDIO_BYTES_PER_FRAME])
|
| 127 |
+
del self.audio_buffer[:AUDIO_BYTES_PER_FRAME]
|
| 128 |
+
|
| 129 |
+
await self.send_chunk(audio, [frame])
|
| 130 |
+
|
| 131 |
+
# Se ambos terminaram, enviar o que sobrou
|
| 132 |
+
if self.orpheus_done and self.wav2lip_done:
|
| 133 |
+
if self.frame_buffer and self.audio_buffer:
|
| 134 |
+
# Dividir audio restante pelos frames restantes
|
| 135 |
+
audio_per_frame = len(self.audio_buffer) // len(self.frame_buffer) if self.frame_buffer else 0
|
| 136 |
+
audio_per_frame = max(audio_per_frame, 2) # Minimo 2 bytes
|
| 137 |
+
audio_per_frame = audio_per_frame - (audio_per_frame % 2) # Alinhamento 16-bit
|
| 138 |
+
|
| 139 |
+
while self.frame_buffer and self.audio_buffer:
|
| 140 |
+
frame = self.frame_buffer.pop(0)
|
| 141 |
+
audio_size = min(audio_per_frame, len(self.audio_buffer))
|
| 142 |
+
if not self.frame_buffer: # Ultimo frame pega todo o resto
|
| 143 |
+
audio_size = len(self.audio_buffer)
|
| 144 |
+
audio = bytes(self.audio_buffer[:audio_size])
|
| 145 |
+
del self.audio_buffer[:audio_size]
|
| 146 |
+
await self.send_chunk(audio, [frame])
|
| 147 |
+
|
| 148 |
+
elif self.frame_buffer:
|
| 149 |
+
# Frames sem audio - enviar com audio vazio
|
| 150 |
+
for frame in self.frame_buffer:
|
| 151 |
+
await self.send_chunk(b'', [frame])
|
| 152 |
+
self.frame_buffer.clear()
|
| 153 |
+
|
| 154 |
+
async def stream_orpheus(self, text: str, voice: str):
|
| 155 |
+
"""Conecta ao Orpheus e recebe chunks de audio."""
|
| 156 |
try:
|
| 157 |
+
print(f"[Orpheus] Conectando a {ORPHEUS_WS}...")
|
| 158 |
async with aiohttp.ClientSession() as session:
|
| 159 |
+
ws = await session.ws_connect(ORPHEUS_WS, timeout=aiohttp.ClientWSTimeout(ws_close=120))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
|
| 161 |
+
# Enviar requisicao
|
| 162 |
+
await ws.send_json({
|
| 163 |
+
"action": "synthesize",
|
| 164 |
"text": text,
|
| 165 |
"voice": voice,
|
| 166 |
+
"stream": True
|
| 167 |
})
|
| 168 |
+
print(f"[Orpheus] Requisicao enviada")
|
| 169 |
+
|
| 170 |
+
async for msg in ws:
|
| 171 |
+
if not self.is_running:
|
| 172 |
+
break
|
| 173 |
+
|
| 174 |
+
if msg.type == aiohttp.WSMsgType.TEXT:
|
| 175 |
+
data = json.loads(msg.data)
|
| 176 |
+
msg_type = data.get("type", "")
|
| 177 |
+
|
| 178 |
+
if msg_type == "audio_chunk":
|
| 179 |
+
# Decodificar e adicionar ao buffer
|
| 180 |
+
audio_b64 = data.get("audio", "")
|
| 181 |
+
audio_bytes = base64.b64decode(audio_b64)
|
| 182 |
+
|
| 183 |
+
async with self.buffer_lock:
|
| 184 |
+
self.audio_buffer.extend(audio_bytes)
|
| 185 |
+
self.total_audio_bytes += len(audio_bytes)
|
| 186 |
+
|
| 187 |
+
# Tentar enviar chunks
|
| 188 |
+
await self.try_send_chunks()
|
| 189 |
+
|
| 190 |
+
chunk_idx = data.get("chunk_index", 0)
|
| 191 |
+
if chunk_idx == 1:
|
| 192 |
+
print(f"[Orpheus] Primeiro chunk de audio recebido")
|
| 193 |
+
|
| 194 |
+
elif msg_type == "done":
|
| 195 |
+
total = data.get("total_bytes", 0)
|
| 196 |
+
print(f"[Orpheus] Concluido: {total} bytes")
|
| 197 |
+
break
|
| 198 |
+
|
| 199 |
+
elif msg_type == "error":
|
| 200 |
+
print(f"[Orpheus] Erro: {data.get('message')}")
|
| 201 |
+
break
|
| 202 |
|
| 203 |
+
elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
|
| 204 |
+
break
|
| 205 |
+
|
| 206 |
+
await ws.close()
|
| 207 |
+
|
| 208 |
+
except Exception as e:
|
| 209 |
+
print(f"[Orpheus] Erro: {e}")
|
| 210 |
+
import traceback
|
| 211 |
+
traceback.print_exc()
|
| 212 |
+
|
| 213 |
+
finally:
|
| 214 |
+
self.orpheus_done = True
|
| 215 |
+
await self.try_send_chunks()
|
| 216 |
+
|
| 217 |
+
async def stream_wav2lip(self, text: str, voice: str):
|
| 218 |
+
"""Conecta ao Wav2Lip e recebe frames."""
|
| 219 |
+
try:
|
| 220 |
+
print(f"[Wav2Lip] Conectando a {WAV2LIP_WS}...")
|
| 221 |
+
async with aiohttp.ClientSession() as session:
|
| 222 |
+
ws = await session.ws_connect(WAV2LIP_WS, timeout=aiohttp.ClientWSTimeout(ws_close=120))
|
| 223 |
|
| 224 |
+
# Enviar requisicao
|
| 225 |
+
await ws.send_json({
|
| 226 |
+
"action": "generate",
|
| 227 |
+
"text": text,
|
| 228 |
+
"voice": voice
|
| 229 |
+
})
|
| 230 |
+
print(f"[Wav2Lip] Requisicao enviada")
|
| 231 |
|
| 232 |
+
async for msg in ws:
|
|
|
|
| 233 |
if not self.is_running:
|
| 234 |
break
|
| 235 |
|
|
|
|
| 237 |
data = json.loads(msg.data)
|
| 238 |
msg_type = data.get("type", "")
|
| 239 |
|
| 240 |
+
if msg_type == "frame":
|
| 241 |
+
# Decodificar e adicionar ao buffer
|
|
|
|
|
|
|
|
|
|
|
|
|
| 242 |
frame_b64 = data.get("frame", "")
|
| 243 |
frame_bytes = base64.b64decode(frame_b64)
|
|
|
|
|
|
|
| 244 |
|
| 245 |
+
async with self.buffer_lock:
|
| 246 |
+
self.frame_buffer.append(frame_bytes)
|
| 247 |
+
self.total_frames += 1
|
| 248 |
+
|
| 249 |
+
# Tentar enviar chunks
|
| 250 |
+
await self.try_send_chunks()
|
| 251 |
+
|
| 252 |
+
frame_idx = data.get("index", 0)
|
| 253 |
+
if frame_idx == 0:
|
| 254 |
+
print(f"[Wav2Lip] Primeiro frame recebido")
|
| 255 |
+
|
| 256 |
+
elif msg_type == "status":
|
| 257 |
+
print(f"[Wav2Lip] {data.get('message')}")
|
| 258 |
+
|
| 259 |
+
elif msg_type == "first_chunk":
|
| 260 |
+
print(f"[Wav2Lip] eSpeak latency: {data.get('latency_ms')}ms")
|
| 261 |
|
| 262 |
elif msg_type == "full_audio":
|
| 263 |
+
# Ignorar - usamos audio do Orpheus
|
| 264 |
+
print(f"[Wav2Lip] Audio ignorado (usando Orpheus)")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 265 |
|
| 266 |
elif msg_type == "done":
|
| 267 |
+
frames = data.get("frames", 0)
|
| 268 |
+
print(f"[Wav2Lip] Concluido: {frames} frames")
|
| 269 |
break
|
| 270 |
|
|
|
|
|
|
|
|
|
|
| 271 |
elif msg_type == "error":
|
| 272 |
+
print(f"[Wav2Lip] Erro: {data.get('message')}")
|
| 273 |
+
await self.send_status(f"Erro: {data.get('message')}")
|
|
|
|
| 274 |
break
|
| 275 |
|
| 276 |
elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
|
| 277 |
break
|
| 278 |
|
| 279 |
+
await ws.close()
|
| 280 |
|
| 281 |
except Exception as e:
|
| 282 |
+
print(f"[Wav2Lip] Erro: {e}")
|
| 283 |
import traceback
|
| 284 |
traceback.print_exc()
|
|
|
|
|
|
|
| 285 |
|
| 286 |
+
finally:
|
| 287 |
+
self.wav2lip_done = True
|
| 288 |
+
await self.try_send_chunks()
|
| 289 |
|
| 290 |
+
async def run(self, text: str, voice: str):
|
| 291 |
+
"""Executa streaming paralelo."""
|
| 292 |
+
self.is_running = True
|
| 293 |
+
self.start_time = time.time()
|
| 294 |
|
| 295 |
+
await self.send_status("Conectando aos servicos...")
|
|
|
|
| 296 |
|
| 297 |
+
try:
|
| 298 |
+
# Conectar Orpheus e Wav2Lip EM PARALELO
|
| 299 |
+
orpheus_task = asyncio.create_task(self.stream_orpheus(text, voice))
|
| 300 |
+
wav2lip_task = asyncio.create_task(self.stream_wav2lip(text, voice))
|
| 301 |
|
| 302 |
+
# Aguardar ambos terminarem
|
| 303 |
+
await asyncio.gather(orpheus_task, wav2lip_task)
|
|
|
|
| 304 |
|
| 305 |
+
# Enviar chunks restantes
|
| 306 |
+
await self.try_send_chunks()
|
| 307 |
|
| 308 |
+
except Exception as e:
|
| 309 |
+
print(f"[Stream] Erro: {e}")
|
| 310 |
+
await self.client_ws.send_json({"type": "error", "message": str(e)})
|
| 311 |
+
return
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
|
|
|
|
| 313 |
elapsed = time.time() - self.start_time
|
| 314 |
+
print(f"[Stream] Concluido: {elapsed:.2f}s, {self.chunks_sent} chunks, {self.total_frames} frames, {self.total_audio_bytes} bytes")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 315 |
|
| 316 |
+
try:
|
| 317 |
+
await self.client_ws.send_json({
|
| 318 |
+
"type": "done",
|
| 319 |
+
"total_chunks": self.chunks_sent,
|
| 320 |
+
"total_frames": self.total_frames,
|
| 321 |
+
"total_audio_bytes": self.total_audio_bytes,
|
| 322 |
+
"elapsed_ms": int(elapsed * 1000)
|
| 323 |
+
})
|
| 324 |
+
except:
|
| 325 |
+
pass
|
| 326 |
|
| 327 |
def stop(self):
|
| 328 |
self.is_running = False
|
|
|
|
| 333 |
ws = web.WebSocketResponse()
|
| 334 |
await ws.prepare(request)
|
| 335 |
|
| 336 |
+
print("Cliente conectado")
|
| 337 |
session = None
|
| 338 |
|
| 339 |
try:
|
|
|
|
| 348 |
voice = data.get("voice", "tara")
|
| 349 |
|
| 350 |
if not text:
|
| 351 |
+
await ws.send_json({"type": "error", "message": "Text required"})
|
| 352 |
continue
|
| 353 |
|
| 354 |
print(f"Gerando: '{text[:50]}...' voice={voice}")
|
|
|
|
| 356 |
if session:
|
| 357 |
session.stop()
|
| 358 |
|
| 359 |
+
session = ParallelStreamingSession(ws)
|
| 360 |
await session.run(text, voice)
|
| 361 |
|
| 362 |
elif action == "stop":
|
|
|
|
| 375 |
break
|
| 376 |
|
| 377 |
except Exception as e:
|
| 378 |
+
print(f"Erro: {e}")
|
| 379 |
finally:
|
| 380 |
if session:
|
| 381 |
session.stop()
|
| 382 |
+
print("Cliente desconectado")
|
| 383 |
|
| 384 |
return ws
|
| 385 |
|
|
|
|
| 389 |
return web.FileResponse(os.path.join(os.path.dirname(__file__), "index.html"))
|
| 390 |
|
| 391 |
|
| 392 |
+
@routes.get("/idle.mp4")
|
| 393 |
+
async def idle_video(request):
|
| 394 |
+
return web.FileResponse(os.path.join(os.path.dirname(__file__), "idle.mp4"))
|
| 395 |
+
|
| 396 |
+
|
| 397 |
+
@routes.get("/{filename}")
|
| 398 |
+
async def static_file(request):
|
| 399 |
+
filename = request.match_info["filename"]
|
| 400 |
+
filepath = os.path.join(os.path.dirname(__file__), filename)
|
| 401 |
+
if os.path.exists(filepath):
|
| 402 |
+
return web.FileResponse(filepath)
|
| 403 |
+
return web.Response(status=404)
|
| 404 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 405 |
|
| 406 |
+
@routes.get("/health")
|
| 407 |
+
async def health(request):
|
| 408 |
+
return web.json_response({"status": "ok", "mode": "parallel_streaming"})
|
| 409 |
|
| 410 |
|
| 411 |
app = web.Application()
|
|
|
|
| 414 |
|
| 415 |
if __name__ == "__main__":
|
| 416 |
print("=" * 50)
|
| 417 |
+
print("Interface Server - Streaming Paralelo")
|
| 418 |
print("=" * 50)
|
| 419 |
print(f"Porta: {PORT}")
|
| 420 |
+
print(f"Orpheus: {ORPHEUS_WS}")
|
| 421 |
print(f"Wav2Lip: {WAV2LIP_WS}")
|
| 422 |
+
print(f"Audio: {AUDIO_SAMPLE_RATE}Hz, {AUDIO_BYTES_PER_FRAME} bytes/frame")
|
| 423 |
+
print(f"Video: {VIDEO_FPS}fps, {MS_PER_FRAME}ms/frame")
|
| 424 |
print("=" * 50)
|
| 425 |
web.run_app(app, host="0.0.0.0", port=PORT)
|
interface/test_streaming.py
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
Test client para streaming progressivo.
|
| 4 |
+
Conecta via WebSocket, recebe chunks, monta e salva vídeo.
|
| 5 |
+
"""
|
| 6 |
+
import asyncio
|
| 7 |
+
import aiohttp
|
| 8 |
+
import json
|
| 9 |
+
import base64
|
| 10 |
+
import struct
|
| 11 |
+
import os
|
| 12 |
+
import sys
|
| 13 |
+
import wave
|
| 14 |
+
import subprocess
|
| 15 |
+
import tempfile
|
| 16 |
+
import time
|
| 17 |
+
|
| 18 |
+
# Configuracao
|
| 19 |
+
WS_URL = os.getenv("WS_URL", "ws://localhost:8090/ws")
|
| 20 |
+
TEXT = os.getenv("TEXT", "Hello! I am testing the progressive streaming avatar system.")
|
| 21 |
+
VOICE = os.getenv("VOICE", "tara")
|
| 22 |
+
OUTPUT_DIR = "/tmp/streaming_test"
|
| 23 |
+
|
| 24 |
+
async def test_streaming():
|
| 25 |
+
print("=" * 60)
|
| 26 |
+
print("Test Streaming Client")
|
| 27 |
+
print("=" * 60)
|
| 28 |
+
print(f"WebSocket: {WS_URL}")
|
| 29 |
+
print(f"Text: {TEXT[:50]}...")
|
| 30 |
+
print(f"Voice: {VOICE}")
|
| 31 |
+
print(f"Output: {OUTPUT_DIR}")
|
| 32 |
+
print("=" * 60)
|
| 33 |
+
|
| 34 |
+
# Criar diretorio de saida
|
| 35 |
+
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
| 36 |
+
|
| 37 |
+
# Buffers para acumular dados
|
| 38 |
+
all_audio = bytearray()
|
| 39 |
+
all_frames = []
|
| 40 |
+
chunks_received = 0
|
| 41 |
+
start_time = None
|
| 42 |
+
first_chunk_time = None
|
| 43 |
+
|
| 44 |
+
try:
|
| 45 |
+
async with aiohttp.ClientSession() as session:
|
| 46 |
+
print("\nConectando ao WebSocket...")
|
| 47 |
+
ws = await session.ws_connect(WS_URL, timeout=aiohttp.ClientWSTimeout(ws_close=120))
|
| 48 |
+
print("Conectado!")
|
| 49 |
+
|
| 50 |
+
# Enviar requisicao
|
| 51 |
+
print(f"\nEnviando requisicao...")
|
| 52 |
+
await ws.send_json({
|
| 53 |
+
"action": "generate",
|
| 54 |
+
"text": TEXT,
|
| 55 |
+
"voice": VOICE
|
| 56 |
+
})
|
| 57 |
+
start_time = time.time()
|
| 58 |
+
|
| 59 |
+
# Receber mensagens
|
| 60 |
+
async for msg in ws:
|
| 61 |
+
if msg.type == aiohttp.WSMsgType.TEXT:
|
| 62 |
+
data = json.loads(msg.data)
|
| 63 |
+
msg_type = data.get("type", "")
|
| 64 |
+
|
| 65 |
+
if msg_type == "status":
|
| 66 |
+
print(f" Status: {data.get('message')}")
|
| 67 |
+
|
| 68 |
+
elif msg_type == "stream_start":
|
| 69 |
+
ttfb = data.get("ttfb_ms", 0)
|
| 70 |
+
first_chunk_time = time.time()
|
| 71 |
+
print(f" Stream iniciado! TTFB: {ttfb}ms")
|
| 72 |
+
|
| 73 |
+
elif msg_type == "chunk":
|
| 74 |
+
chunks_received += 1
|
| 75 |
+
chunk_index = data.get("chunk_index", 0)
|
| 76 |
+
audio_size = data.get("audio_size", 0)
|
| 77 |
+
num_frames = data.get("num_frames", 0)
|
| 78 |
+
|
| 79 |
+
# Decodificar chunk
|
| 80 |
+
chunk_data = base64.b64decode(data.get("data", ""))
|
| 81 |
+
view = memoryview(chunk_data)
|
| 82 |
+
offset = 0
|
| 83 |
+
|
| 84 |
+
# Ler audio
|
| 85 |
+
audio_len = struct.unpack(">I", view[offset:offset+4])[0]
|
| 86 |
+
offset += 4
|
| 87 |
+
audio_bytes = bytes(view[offset:offset+audio_len])
|
| 88 |
+
offset += audio_len
|
| 89 |
+
all_audio.extend(audio_bytes)
|
| 90 |
+
|
| 91 |
+
# Ler frames
|
| 92 |
+
frames_count = struct.unpack(">I", view[offset:offset+4])[0]
|
| 93 |
+
offset += 4
|
| 94 |
+
for i in range(frames_count):
|
| 95 |
+
frame_len = struct.unpack(">I", view[offset:offset+4])[0]
|
| 96 |
+
offset += 4
|
| 97 |
+
frame_bytes = bytes(view[offset:offset+frame_len])
|
| 98 |
+
offset += frame_len
|
| 99 |
+
all_frames.append(frame_bytes)
|
| 100 |
+
|
| 101 |
+
print(f" Chunk {chunk_index}: {audio_size} bytes audio, {num_frames} frames (total: {len(all_frames)} frames, {len(all_audio)} bytes audio)")
|
| 102 |
+
|
| 103 |
+
elif msg_type == "done":
|
| 104 |
+
total_chunks = data.get("total_chunks", 0)
|
| 105 |
+
total_frames = data.get("total_frames", 0)
|
| 106 |
+
elapsed_ms = data.get("elapsed_ms", 0)
|
| 107 |
+
print(f"\n Done! {total_chunks} chunks, {total_frames} frames, {elapsed_ms}ms")
|
| 108 |
+
break
|
| 109 |
+
|
| 110 |
+
elif msg_type == "error":
|
| 111 |
+
print(f" ERRO: {data.get('message')}")
|
| 112 |
+
break
|
| 113 |
+
|
| 114 |
+
elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
|
| 115 |
+
print(" WebSocket fechado")
|
| 116 |
+
break
|
| 117 |
+
|
| 118 |
+
await ws.close()
|
| 119 |
+
|
| 120 |
+
except Exception as e:
|
| 121 |
+
print(f"Erro: {e}")
|
| 122 |
+
import traceback
|
| 123 |
+
traceback.print_exc()
|
| 124 |
+
return
|
| 125 |
+
|
| 126 |
+
total_time = time.time() - start_time if start_time else 0
|
| 127 |
+
|
| 128 |
+
print("\n" + "=" * 60)
|
| 129 |
+
print("Resultados:")
|
| 130 |
+
print("=" * 60)
|
| 131 |
+
print(f"Chunks recebidos: {chunks_received}")
|
| 132 |
+
print(f"Frames recebidos: {len(all_frames)}")
|
| 133 |
+
print(f"Audio recebido: {len(all_audio)} bytes")
|
| 134 |
+
print(f"Tempo total: {total_time:.2f}s")
|
| 135 |
+
|
| 136 |
+
if not all_frames or not all_audio:
|
| 137 |
+
print("\nNenhum dado recebido!")
|
| 138 |
+
return
|
| 139 |
+
|
| 140 |
+
# Salvar frames como JPEGs
|
| 141 |
+
print(f"\nSalvando {len(all_frames)} frames...")
|
| 142 |
+
frames_dir = os.path.join(OUTPUT_DIR, "frames")
|
| 143 |
+
os.makedirs(frames_dir, exist_ok=True)
|
| 144 |
+
for i, frame in enumerate(all_frames):
|
| 145 |
+
frame_path = os.path.join(frames_dir, f"frame_{i:05d}.jpg")
|
| 146 |
+
with open(frame_path, "wb") as f:
|
| 147 |
+
f.write(frame)
|
| 148 |
+
|
| 149 |
+
# Salvar audio como WAV
|
| 150 |
+
print(f"Salvando audio ({len(all_audio)} bytes)...")
|
| 151 |
+
audio_path = os.path.join(OUTPUT_DIR, "audio.wav")
|
| 152 |
+
|
| 153 |
+
# Detectar sample rate pelo tamanho do audio e frames
|
| 154 |
+
# 25 fps, cada frame = 40ms
|
| 155 |
+
# audio_duration = len(all_frames) * 40ms
|
| 156 |
+
audio_duration_ms = len(all_frames) * 40
|
| 157 |
+
samples = len(all_audio) // 2 # 16-bit = 2 bytes por sample
|
| 158 |
+
if audio_duration_ms > 0:
|
| 159 |
+
estimated_sample_rate = int(samples / (audio_duration_ms / 1000))
|
| 160 |
+
# Arredondar para sample rates comuns
|
| 161 |
+
if estimated_sample_rate > 20000:
|
| 162 |
+
sample_rate = 24000
|
| 163 |
+
else:
|
| 164 |
+
sample_rate = 22050
|
| 165 |
+
else:
|
| 166 |
+
sample_rate = 24000
|
| 167 |
+
|
| 168 |
+
print(f"Sample rate estimado: {sample_rate} Hz")
|
| 169 |
+
|
| 170 |
+
with wave.open(audio_path, "wb") as wf:
|
| 171 |
+
wf.setnchannels(1)
|
| 172 |
+
wf.setsampwidth(2) # 16-bit
|
| 173 |
+
wf.setframerate(sample_rate)
|
| 174 |
+
wf.writeframes(all_audio)
|
| 175 |
+
|
| 176 |
+
# Criar video com ffmpeg
|
| 177 |
+
print("\nCriando video com ffmpeg...")
|
| 178 |
+
video_path = os.path.join(OUTPUT_DIR, "output.mp4")
|
| 179 |
+
|
| 180 |
+
# Calcular fps para sincronizar com audio
|
| 181 |
+
audio_duration_s = len(all_audio) / 2 / sample_rate
|
| 182 |
+
if audio_duration_s > 0:
|
| 183 |
+
video_fps = len(all_frames) / audio_duration_s
|
| 184 |
+
else:
|
| 185 |
+
video_fps = 25
|
| 186 |
+
|
| 187 |
+
print(f"Video FPS: {video_fps:.2f}")
|
| 188 |
+
|
| 189 |
+
cmd = [
|
| 190 |
+
"ffmpeg", "-y",
|
| 191 |
+
"-framerate", str(video_fps),
|
| 192 |
+
"-i", os.path.join(frames_dir, "frame_%05d.jpg"),
|
| 193 |
+
"-i", audio_path,
|
| 194 |
+
"-vf", "pad=ceil(iw/2)*2:ceil(ih/2)*2", # Fix odd dimensions for libx264
|
| 195 |
+
"-c:v", "libx264",
|
| 196 |
+
"-preset", "fast",
|
| 197 |
+
"-crf", "23",
|
| 198 |
+
"-c:a", "aac",
|
| 199 |
+
"-b:a", "128k",
|
| 200 |
+
"-pix_fmt", "yuv420p",
|
| 201 |
+
"-shortest",
|
| 202 |
+
video_path
|
| 203 |
+
]
|
| 204 |
+
|
| 205 |
+
result = subprocess.run(cmd, capture_output=True, text=True)
|
| 206 |
+
if result.returncode != 0:
|
| 207 |
+
print(f"FFmpeg erro: {result.stderr[-500:]}")
|
| 208 |
+
else:
|
| 209 |
+
print(f"\nVideo salvo em: {video_path}")
|
| 210 |
+
# Mostrar info do video
|
| 211 |
+
probe = subprocess.run(
|
| 212 |
+
["ffprobe", "-v", "quiet", "-print_format", "json", "-show_format", video_path],
|
| 213 |
+
capture_output=True, text=True
|
| 214 |
+
)
|
| 215 |
+
if probe.returncode == 0:
|
| 216 |
+
import json as json_mod
|
| 217 |
+
info = json_mod.loads(probe.stdout)
|
| 218 |
+
duration = float(info.get("format", {}).get("duration", 0))
|
| 219 |
+
size = int(info.get("format", {}).get("size", 0))
|
| 220 |
+
print(f"Duracao: {duration:.2f}s")
|
| 221 |
+
print(f"Tamanho: {size / 1024:.1f} KB")
|
| 222 |
+
|
| 223 |
+
print("\n" + "=" * 60)
|
| 224 |
+
print("Arquivos gerados:")
|
| 225 |
+
print("=" * 60)
|
| 226 |
+
print(f" Frames: {frames_dir}/")
|
| 227 |
+
print(f" Audio: {audio_path}")
|
| 228 |
+
print(f" Video: {video_path}")
|
| 229 |
+
|
| 230 |
+
|
| 231 |
+
if __name__ == "__main__":
|
| 232 |
+
asyncio.run(test_streaming())
|