marcosremar2 Claude Opus 4.5 commited on
Commit
eeae125
·
1 Parent(s): 87bf58e

Implement parallel streaming: Orpheus audio + Wav2Lip frames

Browse files

- Connect to Orpheus TTS and Wav2Lip in parallel
- Receive audio chunks from Orpheus WebSocket (high quality)
- Receive JPEG frames from Wav2Lip (lip sync with internal eSpeak)
- Discard Wav2Lip audio, use only Orpheus audio
- Mount and send chunks immediately as data arrives
- Add test_streaming.py client for testing
- Update CLAUDE.md with new architecture

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Files changed (3) hide show
  1. interface/CLAUDE.md +179 -88
  2. interface/server.py +271 -221
  3. interface/test_streaming.py +232 -0
interface/CLAUDE.md CHANGED
@@ -1,31 +1,58 @@
1
  # Avatar Interface - Sistema de Streaming de Video com Lip Sync
2
 
3
- ## OBJETIVOS DO PROJETO
4
 
5
- ### Requisitos Obrigatorios (MUST HAVE)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
 
7
- 1. **Processamento Paralelo de Audio e Video**
8
- - Wav2Lip (video) e Orpheus TTS (audio) DEVEM ser chamados em PARALELO
9
- - O servidor deve iniciar ambas as conexoes WebSocket ao mesmo tempo
10
- - Nao esperar um terminar para comecar o outro
11
- - Isso reduz a latencia total pela metade
12
 
13
- 2. **Reproducao Progressiva (Streaming Real)**
14
- - O frontend DEVE comecar a reproduzir ANTES de receber todos os dados
15
- - Assim que tiver frames + audio suficientes, iniciar playback
16
- - Continuar recebendo dados enquanto reproduz
17
- - Meta: latencia < 500ms entre clique e inicio da reproducao
18
- - Atualmente: ~3 segundos (espera tudo chegar antes de reproduzir)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
- 3. **Sincronizacao Perfeita Audio/Video**
21
- - O video deve parar exatamente quando o audio termina
22
- - Usar audioExpectedEndTime para controlar fim dos frames
23
- - Transicao suave (crossfade 300ms) para o video idle
24
 
25
- 4. **Continuidade do Video Idle**
26
- - O video idle deve continuar do frame exato onde o Wav2Lip parou
27
- - Usar end_video_time_ms para posicionar o video idle
28
- - Video idle em loop infinito quando nao esta falando
 
29
 
30
  ---
31
 
@@ -47,63 +74,66 @@
47
 
48
  ---
49
 
50
- ## Visao Geral
51
-
52
- Sistema de interface web que recebe texto via WebSocket e gera video sincronizado com audio em tempo real (lip sync). O sistema conecta dois servicos:
53
-
54
- - **TTS (Orpheus)**: Gera audio a partir do texto (PCM 24kHz)
55
- - **Wav2Lip**: Gera frames de video com sincronizacao labial (JPEG 25fps)
56
-
57
- ## Arquitetura
58
 
59
  ```
60
  ┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────┐
61
  │ Frontend │◄─────►│ Interface Server │◄─────►│ TTS (Orpheus) │
62
  │ (Browser) │ WS │ (porta 8080) │ WS │ (porta 8081) │
63
  │ │ │ │ │ PCM 24kHz │
64
- └─────────────────┘ └─────────────────┘
65
-
66
- resample 24k→16k ┌─────────────────┐
67
- │ │─────►│ Wav2Lip │
68
- │ WS(porta 8082)
 
69
  └──────────────────────┘ │ JPEG 25fps │
 
70
  └─────────────────┘
71
  ```
72
 
73
- ## Fluxo de Dados Sincronizado
74
 
75
  ```
76
  1. Frontend envia: {"action": "generate", "text": "Hello", "voice": "tara"}
77
 
78
 
79
- 2. Interface conecta ao TTS (8081) e Wav2Lip (8082)
80
 
81
-
82
- 3. Interface envia para TTS: {"action": "synthesize", "text": "Hello", "voice": "tara", "stream": true}
 
 
 
 
 
 
 
 
 
83
 
84
 
85
- 4. TTS retorna chunks de audio (PCM 24kHz, 16-bit, mono)
 
 
 
 
86
 
87
 
88
- 5. Para CADA chunk de audio:
89
- a. Resample 24kHz 16kHz
90
- b. Enviar para Wav2Lip: {"action": "process_audio", "audio": "<base64>", "sample_rate": 16000}
91
- c. Receber frames correspondentes: {"type": "frame", "frame": "<base64 JPEG>"}
92
- d. Mesclar audio + frames em chunk sincronizado
93
- e. Enviar para frontend: {"type": "chunk", "data": "<base64>", ...}
94
 
95
 
96
- 6. Ao finalizar TTS, enviar para Wav2Lip: {"action": "end"}
97
-
98
-
99
- 7. Enviar para frontend: {"type": "done", "total_frames": N, "total_duration_ms": T}
100
  ```
101
 
102
- ## Protocolos WebSocket (Real)
103
 
104
  ### TTS Server (porta 8081) - Orpheus
105
 
106
- **Arquivo:** `/workspace/orpheus-streaming/streaming_api_server_ws.py`
107
 
108
  **Requisicao:**
109
  ```json
@@ -136,7 +166,9 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
136
  }
137
  ```
138
 
139
- **Formato do audio:** PCM 24kHz, 16-bit signed, mono, little-endian
 
 
140
 
141
  ---
142
 
@@ -144,6 +176,9 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
144
 
145
  **Arquivo:** `/home/marcosavatar/realtimeWav2lip/websocket_server.py`
146
 
 
 
 
147
  **Enviar audio (chunk por chunk):**
148
  ```json
149
  {
@@ -153,7 +188,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
153
  }
154
  ```
155
 
156
- **Resposta (frames):**
157
  ```json
158
  {"type": "frame", "frame": "<base64 JPEG>", "index": 0}
159
  {"type": "frame", "frame": "<base64 JPEG>", "index": 1}
@@ -179,7 +214,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
179
 
180
  ---
181
 
182
- ### Interface Server (porta 8080) Frontend
183
 
184
  **Requisicao do frontend:**
185
  ```json
@@ -188,9 +223,14 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
188
  {"action": "ping"}
189
  ```
190
 
191
- **Respostas para o frontend:**
 
 
 
 
 
192
  ```json
193
- {"type": "status", "message": "Gerando audio e video..."}
194
  ```
195
 
196
  ```json
@@ -200,8 +240,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
200
  "audio_size": 4800,
201
  "audio_duration_ms": 100,
202
  "num_frames": 2,
203
- "total_size": 65000,
204
- "data": "<base64>"
205
  }
206
  ```
207
 
@@ -210,7 +249,7 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
210
  "type": "done",
211
  "total_duration_ms": 5000,
212
  "total_frames": 125,
213
- "total_chunks": 10,
214
  "elapsed_ms": 3500
215
  }
216
  ```
@@ -219,43 +258,83 @@ Sistema de interface web que recebe texto via WebSocket e gera video sincronizad
219
  {"type": "error", "message": "Descricao do erro"}
220
  ```
221
 
222
- ### Formato do Chunk (binario em base64)
223
 
224
  ```
225
  [audio_size: 4 bytes big-endian]
226
- [audio_data: PCM 24kHz, 16-bit, mono]
 
227
  [frame_1_size: 4 bytes big-endian]
228
- [frame_1_data: JPEG bytes]
229
  [frame_2_size: 4 bytes big-endian]
230
- [frame_2_data: JPEG bytes]
231
  ...
232
  ```
233
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
234
  ## Sincronizacao Audio/Video
235
 
236
  ### Calculos
237
 
238
  ```
239
- TTS: 24000 Hz (24000 samples/segundo)
240
- Wav2Lip: 16000 Hz (espera 16000 samples/segundo)
241
  Video: 25 fps (1 frame a cada 40ms)
242
 
243
- Resample ratio: 16000 / 24000 = 0.666...
244
-
245
- Para cada chunk de audio:
246
- - Duracao (ms) = bytes / 2 / 24000 * 1000
247
- - Frames esperados = duracao_ms / 40
248
  ```
249
 
250
- ### Exemplo
251
 
252
  ```
253
- Chunk de audio: 4800 bytes (PCM 24kHz)
254
- Duracao: 4800 / 2 / 24000 * 1000 = 100ms
255
- Frames esperados: 100 / 40 = 2.5 ≈ 2-3 frames
256
 
257
- Apos resample para 16kHz:
258
- Novos bytes: 4800 * (16000/24000) = 3200 bytes
259
  ```
260
 
261
  ## Estrutura do Projeto
@@ -263,10 +342,10 @@ Novos bytes: 4800 * (16000/24000) = 3200 bytes
263
  ```
264
  /workspace/interface/
265
  ├── CLAUDE.md # Esta documentacao
266
- ├── server.py # Servidor WebSocket (aiohttp)
267
- ├── index.html # Frontend (HTML + JS)
268
- ├── static/ # Arquivos estaticos
269
- └── templates/ # Templates HTML
270
  ```
271
 
272
  ## Configuracao
@@ -276,6 +355,7 @@ Novos bytes: 4800 * (16000/24000) = 3200 bytes
276
  TTS_WS=ws://localhost:8081/ws
277
  WAV2LIP_WS=ws://localhost:8082/ws
278
  PORT=8080
 
279
  ```
280
 
281
  ## Como Executar
@@ -288,11 +368,12 @@ python3 server.py
288
  **Output esperado:**
289
  ```
290
  ==================================================
291
- Interface Server - WebSocket Streaming
292
  ==================================================
293
  Porta: 8080
294
- TTS: ws://localhost:8081/ws (24kHz)
295
- Wav2Lip: ws://localhost:8082/ws (16kHz)
 
296
  Video: 25fps (40ms/frame)
297
  ==================================================
298
  ```
@@ -309,14 +390,24 @@ Video: 25fps (40ms/frame)
309
 
310
  ## Resumo das Portas
311
 
312
- | Servico | Porta | Audio | Video |
313
- |------------------|-------|--------------|-----------|
314
- | Interface Server | 8080 | - | - |
315
- | TTS (Orpheus) | 8081 | PCM 24kHz | - |
316
- | Wav2Lip | 8082 | PCM 16kHz | JPEG 25fps|
317
 
318
  ## Dependencias Python
319
 
320
  ```
321
  aiohttp>=3.9.0
322
  ```
 
 
 
 
 
 
 
 
 
 
 
1
  # Avatar Interface - Sistema de Streaming de Video com Lip Sync
2
 
3
+ ## ARQUITETURA PRINCIPAL
4
 
5
+ ```
6
+ ┌─────────────┐ ┌─────────────────────────────────┐ ┌─────────────────┐
7
+ │ Frontend │◄────►│ Interface Server │◄────►│ Orpheus TTS │
8
+ │ (Browser) │ WS │ (porta 8080) │ WS │ (porta 8081) │
9
+ └─────────────┘ │ │ │ chunks audio │
10
+ │ 1. Recebe texto do frontend │ └─────────────────┘
11
+ │ 2. Conecta Orpheus + Wav2Lip │
12
+ │ EM PARALELO │ ┌─────────────────┐
13
+ │ 3. Recebe chunks conforme │◄────►│ Wav2Lip │
14
+ │ chegam de ambos │ WS │ (porta 8082) │
15
+ │ 4. Monta: audio Orpheus + │ │ frames JPEG │
16
+ │ frames Wav2Lip │ │ (NAO MODIFICAR)│
17
+ │ 5. Envia chunk IMEDIATAMENTE │ └─────────────────┘
18
+ └─────────────────────────────────┘
19
+ ```
20
 
21
+ ## FLUXO DE STREAMING
 
 
 
 
22
 
23
+ ```
24
+ 1. Frontend envia: {"action": "generate", "text": "Hello", "voice": "tara"}
25
+
26
+
27
+ 2. Interface conecta EM PARALELO:
28
+ ├── Orpheus WS (8081) recebe chunks de audio PCM 24kHz
29
+ └── Wav2Lip WS (8082) → recebe frames JPEG (lip sync com eSpeak interno)
30
+
31
+
32
+ 3. Conforme dados chegam, acumula em buffers:
33
+ - audio_buffer: chunks PCM do Orpheus
34
+ - frame_buffer: frames JPEG do Wav2Lip
35
+
36
+
37
+ 4. Quando tem 1 frame + audio correspondente (~1920 bytes = 40ms):
38
+ - Monta chunk binario: [audio_orpheus + frame_wav2lip]
39
+ - Envia IMEDIATAMENTE para frontend
40
+
41
+
42
+ 5. Frontend reproduz em tempo real (nao espera tudo chegar)
43
+ ```
44
+
45
+ ## REGRAS CRITICAS
46
+
47
+ 1. **NAO MODIFICAR O WAV2LIP** - Ele ja gera lip sync com eSpeak interno. So usar os frames JPEG.
48
 
49
+ 2. **AUDIO VEM DO ORPHEUS** - O audio robotico do Wav2Lip e DESCARTADO. Audio final = Orpheus.
 
 
 
50
 
51
+ 3. **CONEXOES EM PARALELO** - Orpheus e Wav2Lip devem ser chamados ao mesmo tempo.
52
+
53
+ 4. **STREAMING IMEDIATO** - Montar e enviar chunks conforme dados chegam, nao esperar tudo.
54
+
55
+ 5. **SINCRONIZACAO** - Calcular audio_por_frame = total_audio / total_frames para alinhar.
56
 
57
  ---
58
 
 
74
 
75
  ---
76
 
77
+ ## Arquitetura de Streaming Progressivo
 
 
 
 
 
 
 
78
 
79
  ```
80
  ┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────┐
81
  │ Frontend │◄─────►│ Interface Server │◄─────►│ TTS (Orpheus) │
82
  │ (Browser) │ WS │ (porta 8080) │ WS │ (porta 8081) │
83
  │ │ │ │ │ PCM 24kHz │
84
+ │ Renderiza │ MONTA CHUNKS: │ (audio final) │
85
+ chunks │ audio Orpheus + │ └─────────────────┘
86
+ conforme │ frames Wav2Lip │
87
+ chegam │ │─────────────────┐
88
+ └─────────────────┘ DESCARTA audio ◄─────►│ Wav2Lip
89
+ │ robotico! │ WS │ (porta 8082) │
90
  └──────────────────────┘ │ JPEG 25fps │
91
+ │ (so frames!) │
92
  └─────────────────┘
93
  ```
94
 
95
+ ## Fluxo de Streaming Progressivo
96
 
97
  ```
98
  1. Frontend envia: {"action": "generate", "text": "Hello", "voice": "tara"}
99
 
100
 
101
+ 2. Interface conecta ao TTS (8081) e Wav2Lip (8082) EM PARALELO
102
 
103
+ ├──────────────────────────────────────────────────────────┐
104
+ │ │
105
+ ▼ ▼
106
+ 3a. Orpheus TTS gera audio 3b. Wav2Lip gera frames
107
+ em chunks (~100ms cada) baseado em audio interno
108
+ │ (eSpeak robotico - DESCARTAR)
109
+ │ │
110
+ ▼ ▼
111
+ 4. Interface Server acumula em buffers:
112
+ - audio_buffer: chunks do Orpheus
113
+ - frame_buffer: frames JPEG do Wav2Lip
114
 
115
 
116
+ 5. Quando tiver dados suficientes (~100ms):
117
+ a. Pegar audio do Orpheus (ex: 4800 bytes = 100ms)
118
+ b. Pegar frames do Wav2Lip (ex: 2-3 frames = 80-120ms)
119
+ c. Montar chunk: [audio_orpheus + frames]
120
+ d. Enviar para frontend IMEDIATAMENTE
121
 
122
 
123
+ 6. Frontend recebe chunk e renderiza:
124
+ - Decodifica audio e adiciona ao buffer de playback
125
+ - Decodifica frames JPEG e adiciona a fila de exibicao
126
+ - Inicia playback assim que tiver ~200ms de dados
 
 
127
 
128
 
129
+ 7. Repetir ate receber "done" de ambos os servicos
 
 
 
130
  ```
131
 
132
+ ## Protocolos WebSocket
133
 
134
  ### TTS Server (porta 8081) - Orpheus
135
 
136
+ **Conexao:** `ws://localhost:8081/ws`
137
 
138
  **Requisicao:**
139
  ```json
 
166
  }
167
  ```
168
 
169
+ **Formato do audio:** PCM 24kHz, 16-bit signed little-endian, mono
170
+
171
+ **Vozes:** tara, leo, leah, jess, dan, mia, zac, zoe
172
 
173
  ---
174
 
 
176
 
177
  **Arquivo:** `/home/marcosavatar/realtimeWav2lip/websocket_server.py`
178
 
179
+ **IMPORTANTE:** O Wav2Lip usa audio interno (eSpeak) para gerar lip sync.
180
+ Este audio robotico DEVE SER DESCARTADO - usar APENAS os frames JPEG!
181
+
182
  **Enviar audio (chunk por chunk):**
183
  ```json
184
  {
 
188
  }
189
  ```
190
 
191
+ **Resposta (frames) - USAR APENAS OS FRAMES:**
192
  ```json
193
  {"type": "frame", "frame": "<base64 JPEG>", "index": 0}
194
  {"type": "frame", "frame": "<base64 JPEG>", "index": 1}
 
214
 
215
  ---
216
 
217
+ ### Interface Server (porta 8080) - Frontend
218
 
219
  **Requisicao do frontend:**
220
  ```json
 
223
  {"action": "ping"}
224
  ```
225
 
226
+ **Respostas para o frontend (STREAMING PROGRESSIVO):**
227
+
228
+ ```json
229
+ {"type": "status", "message": "Conectando aos servicos..."}
230
+ ```
231
+
232
  ```json
233
+ {"type": "stream_start", "ttfb_ms": 150}
234
  ```
235
 
236
  ```json
 
240
  "audio_size": 4800,
241
  "audio_duration_ms": 100,
242
  "num_frames": 2,
243
+ "data": "<base64 do chunk montado>"
 
244
  }
245
  ```
246
 
 
249
  "type": "done",
250
  "total_duration_ms": 5000,
251
  "total_frames": 125,
252
+ "total_chunks": 50,
253
  "elapsed_ms": 3500
254
  }
255
  ```
 
258
  {"type": "error", "message": "Descricao do erro"}
259
  ```
260
 
261
+ ### Formato do Chunk Montado (binario em base64)
262
 
263
  ```
264
  [audio_size: 4 bytes big-endian]
265
+ [audio_data: PCM 24kHz do ORPHEUS, 16-bit, mono]
266
+ [num_frames: 4 bytes big-endian]
267
  [frame_1_size: 4 bytes big-endian]
268
+ [frame_1_data: JPEG bytes do WAV2LIP]
269
  [frame_2_size: 4 bytes big-endian]
270
+ [frame_2_data: JPEG bytes do WAV2LIP]
271
  ...
272
  ```
273
 
274
+ ## Algoritmo de Montagem de Chunks
275
+
276
+ ```python
277
+ CHUNK_DURATION_MS = 100 # Tamanho alvo de cada chunk
278
+ AUDIO_SAMPLE_RATE = 24000 # Orpheus
279
+ VIDEO_FPS = 25 # Wav2Lip
280
+ BYTES_PER_SAMPLE = 2 # 16-bit
281
+
282
+ # Buffers
283
+ audio_buffer = bytearray() # Audio do Orpheus
284
+ frame_buffer = [] # Frames do Wav2Lip
285
+
286
+ async def process_streaming():
287
+ # Conectar em PARALELO
288
+ orpheus_task = asyncio.create_task(connect_orpheus())
289
+ wav2lip_task = asyncio.create_task(connect_wav2lip())
290
+
291
+ while not done:
292
+ # Verificar se tem dados suficientes para montar chunk
293
+ audio_bytes_needed = int(CHUNK_DURATION_MS * AUDIO_SAMPLE_RATE * BYTES_PER_SAMPLE / 1000)
294
+ frames_needed = int(CHUNK_DURATION_MS * VIDEO_FPS / 1000)
295
+
296
+ if len(audio_buffer) >= audio_bytes_needed and len(frame_buffer) >= frames_needed:
297
+ # Extrair dados dos buffers
298
+ audio_chunk = audio_buffer[:audio_bytes_needed]
299
+ del audio_buffer[:audio_bytes_needed]
300
+
301
+ frames_chunk = frame_buffer[:frames_needed]
302
+ del frame_buffer[:frames_needed]
303
+
304
+ # Montar e enviar chunk
305
+ chunk_data = build_chunk(audio_chunk, frames_chunk)
306
+ await ws.send_json({
307
+ "type": "chunk",
308
+ "chunk_index": chunk_index,
309
+ "audio_duration_ms": CHUNK_DURATION_MS,
310
+ "num_frames": len(frames_chunk),
311
+ "data": base64.b64encode(chunk_data).decode()
312
+ })
313
+ chunk_index += 1
314
+ ```
315
+
316
  ## Sincronizacao Audio/Video
317
 
318
  ### Calculos
319
 
320
  ```
321
+ TTS Orpheus: 24000 Hz (24000 samples/segundo)
322
+ Wav2Lip: 16000 Hz (espera 16000 samples/segundo para lip sync)
323
  Video: 25 fps (1 frame a cada 40ms)
324
 
325
+ Para chunk de 100ms:
326
+ - Audio Orpheus: 100 * 24000 / 1000 * 2 = 4800 bytes
327
+ - Frames Wav2Lip: 100 / 40 = 2.5 ≈ 2-3 frames
 
 
328
  ```
329
 
330
+ ### Exemplo de Chunk
331
 
332
  ```
333
+ Chunk de 100ms:
334
+ - Audio: 4800 bytes PCM 24kHz do Orpheus (voz de alta qualidade)
335
+ - Frames: 2-3 JPEGs do Wav2Lip (lip sync)
336
 
337
+ Total por chunk: ~50-80 KB (depende da compressao JPEG)
 
338
  ```
339
 
340
  ## Estrutura do Projeto
 
342
  ```
343
  /workspace/interface/
344
  ├── CLAUDE.md # Esta documentacao
345
+ ├── server.py # Servidor WebSocket com montagem de chunks
346
+ ├── index.html # Frontend com playback progressivo
347
+ ├── idle.mp4 # Video de idle loop
348
+ └── static/ # Arquivos estaticos
349
  ```
350
 
351
  ## Configuracao
 
355
  TTS_WS=ws://localhost:8081/ws
356
  WAV2LIP_WS=ws://localhost:8082/ws
357
  PORT=8080
358
+ CHUNK_DURATION_MS=100
359
  ```
360
 
361
  ## Como Executar
 
368
  **Output esperado:**
369
  ```
370
  ==================================================
371
+ Interface Server - Streaming Progressivo
372
  ==================================================
373
  Porta: 8080
374
+ TTS (Orpheus): ws://localhost:8081/ws (24kHz)
375
+ Wav2Lip: ws://localhost:8082/ws (16kHz, apenas frames)
376
+ Chunk duration: 100ms
377
  Video: 25fps (40ms/frame)
378
  ==================================================
379
  ```
 
390
 
391
  ## Resumo das Portas
392
 
393
+ | Servico | Porta | Audio | Video |
394
+ |------------------|-------|--------------|------------|
395
+ | Interface Server | 8080 | Monta chunks | - |
396
+ | TTS (Orpheus) | 8081 | PCM 24kHz | - |
397
+ | Wav2Lip | 8082 | DESCARTAR! | JPEG 25fps |
398
 
399
  ## Dependencias Python
400
 
401
  ```
402
  aiohttp>=3.9.0
403
  ```
404
+
405
+ ## Notas Importantes
406
+
407
+ 1. **NUNCA usar audio do Wav2Lip no frontend** - O Wav2Lip usa eSpeak internamente para gerar lip sync. Este audio e robotico e deve ser DESCARTADO. Usar APENAS os frames JPEG.
408
+
409
+ 2. **SEMPRE usar audio do Orpheus** - O audio final enviado ao frontend deve vir do Orpheus TTS, que gera voz de alta qualidade.
410
+
411
+ 3. **Streaming progressivo e obrigatorio** - Nao esperar todo o audio/video ficar pronto. Montar e enviar chunks de ~100ms conforme os dados chegam.
412
+
413
+ 4. **Frontend deve iniciar playback cedo** - Assim que receber ~200ms de dados (2 chunks), iniciar reproducao enquanto continua recebendo.
interface/server.py CHANGED
@@ -1,13 +1,17 @@
1
  """
2
- Interface Server - WebSocket Streaming com Wav2Lip
3
  Porta: 8080
4
 
5
- Fluxo simplificado:
6
- 1. Frontend conecta via WebSocket e envia texto
7
- 2. Server envia texto para Wav2Lip (action: speak)
8
- 3. Wav2Lip usa eSpeak interno para lip sync + busca audio do Orpheus
9
- 4. Server recebe frames + audio do Wav2Lip
10
- 5. Server muxa em WebM e envia chunks de ~64KB para frontend
 
 
 
 
11
  """
12
  from aiohttp import web
13
  import aiohttp
@@ -16,163 +20,216 @@ import json
16
  import base64
17
  import os
18
  import time
19
- import subprocess
20
- import tempfile
21
  import struct
22
 
23
- # Configuracao dos servicos
 
24
  WAV2LIP_WS = os.getenv("WAV2LIP_WS", "ws://localhost:8082/ws")
25
  PORT = int(os.getenv("PORT", "8080"))
26
 
27
- # Configuracao de video
28
- VIDEO_FPS = 25
29
- CHUNK_TARGET_SIZE = 64 * 1024 # 64KB por chunk
 
 
 
30
 
31
  routes = web.RouteTableDef()
32
 
33
 
34
- class WebMStreamer:
35
- """Gera WebM a partir de frames JPEG e audio PCM usando ffmpeg."""
36
-
37
- def __init__(self):
38
- self.temp_dir = None
39
- self.frames = []
40
- self.audio_data = None
41
- self.sample_rate = 24000
42
-
43
- async def start(self):
44
- self.temp_dir = tempfile.mkdtemp()
45
- self.frames = []
46
- self.audio_data = None
47
-
48
- def add_frame(self, jpeg_data: bytes):
49
- self.frames.append(jpeg_data)
50
-
51
- def set_audio(self, pcm_data: bytes, sample_rate: int = 24000):
52
- self.audio_data = pcm_data
53
- self.sample_rate = sample_rate
54
-
55
- async def finalize(self) -> bytes:
56
- if not self.frames or not self.audio_data:
57
- return b""
58
-
59
- # Salvar frames
60
- for i, frame in enumerate(self.frames):
61
- frame_path = os.path.join(self.temp_dir, f"frame_{i:05d}.jpg")
62
- with open(frame_path, 'wb') as f:
63
- f.write(frame)
64
-
65
- # Salvar audio como WAV
66
- audio_path = os.path.join(self.temp_dir, "audio.wav")
67
- with open(audio_path, 'wb') as f:
68
- f.write(b'RIFF')
69
- f.write(struct.pack('<I', 36 + len(self.audio_data)))
70
- f.write(b'WAVE')
71
- f.write(b'fmt ')
72
- f.write(struct.pack('<I', 16))
73
- f.write(struct.pack('<H', 1))
74
- f.write(struct.pack('<H', 1))
75
- f.write(struct.pack('<I', self.sample_rate))
76
- f.write(struct.pack('<I', self.sample_rate * 2))
77
- f.write(struct.pack('<H', 2))
78
- f.write(struct.pack('<H', 16))
79
- f.write(b'data')
80
- f.write(struct.pack('<I', len(self.audio_data)))
81
- f.write(self.audio_data)
82
-
83
- output_path = os.path.join(self.temp_dir, "output.webm")
84
-
85
- cmd = [
86
- 'ffmpeg', '-y',
87
- '-framerate', str(VIDEO_FPS),
88
- '-i', os.path.join(self.temp_dir, 'frame_%05d.jpg'),
89
- '-i', audio_path,
90
- '-c:v', 'libvpx-vp9',
91
- '-b:v', '1M',
92
- '-c:a', 'libopus',
93
- '-b:a', '128k',
94
- '-shortest',
95
- output_path
96
- ]
97
-
98
- try:
99
- proc = await asyncio.create_subprocess_exec(
100
- *cmd,
101
- stdout=asyncio.subprocess.PIPE,
102
- stderr=asyncio.subprocess.PIPE
103
- )
104
- stdout, stderr = await proc.communicate()
105
-
106
- if proc.returncode != 0:
107
- print(f"FFmpeg error: {stderr.decode()[-500:]}")
108
- return b""
109
-
110
- with open(output_path, 'rb') as f:
111
- return f.read()
112
-
113
- except Exception as e:
114
- print(f"Error creating WebM: {e}")
115
- return b""
116
-
117
- async def cleanup(self):
118
- if self.temp_dir:
119
- import shutil
120
- try:
121
- shutil.rmtree(self.temp_dir)
122
- except:
123
- pass
124
 
125
 
126
- class StreamingSession:
127
- """Gerencia uma sessao de streaming."""
128
 
129
  def __init__(self, client_ws):
130
  self.client_ws = client_ws
131
  self.is_running = False
132
  self.start_time = None
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
133
  self.total_frames = 0
134
  self.chunks_sent = 0
135
 
136
- async def send_to_client(self, msg_type: str, **kwargs):
 
137
  try:
138
- await self.client_ws.send_json({"type": msg_type, **kwargs})
139
- except Exception as e:
140
- print(f"Erro enviando para cliente: {e}")
141
-
142
- async def run(self, text: str, voice: str):
143
- """Executa a sessao usando Wav2Lip com action: speak."""
144
- self.is_running = True
145
- self.start_time = time.time()
146
-
147
- await self.send_to_client("status", message="Conectando ao Wav2Lip...")
148
 
149
- frames = []
150
- audio_data = None
151
- audio_sample_rate = 24000
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
152
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
153
  try:
 
154
  async with aiohttp.ClientSession() as session:
155
- wav2lip_ws = await session.ws_connect(
156
- WAV2LIP_WS,
157
- timeout=aiohttp.ClientWSTimeout(ws_close=120)
158
- )
159
-
160
- await self.send_to_client("status", message="Gerando avatar...")
161
 
162
- # Enviar texto para Wav2Lip (action: speak usa eSpeak + Orpheus interno)
163
- await wav2lip_ws.send_json({
164
- "action": "speak",
165
  "text": text,
166
  "voice": voice,
167
- "parallel": True
168
  })
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
169
 
170
- print(f"[Wav2Lip] Enviado texto: '{text[:50]}...'")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
171
 
172
- first_frame_time = None
 
 
 
 
 
 
173
 
174
- # Receber frames e audio do Wav2Lip
175
- async for msg in wav2lip_ws:
176
  if not self.is_running:
177
  break
178
 
@@ -180,105 +237,92 @@ class StreamingSession:
180
  data = json.loads(msg.data)
181
  msg_type = data.get("type", "")
182
 
183
- if msg_type == "first_chunk":
184
- latency = data.get("latency_ms", 0)
185
- print(f"[Wav2Lip] First chunk: {latency}ms")
186
- await self.send_to_client("status", message="Processando frames...")
187
-
188
- elif msg_type == "frame":
189
  frame_b64 = data.get("frame", "")
190
  frame_bytes = base64.b64decode(frame_b64)
191
- frames.append(frame_bytes)
192
- self.total_frames += 1
193
 
194
- if first_frame_time is None:
195
- first_frame_time = time.time() - self.start_time
196
- print(f"[Wav2Lip] Primeiro frame em {first_frame_time*1000:.0f}ms")
197
- await self.send_to_client("first_frame", latency_ms=int(first_frame_time*1000))
 
 
 
 
 
 
 
 
 
 
 
 
198
 
199
  elif msg_type == "full_audio":
200
- audio_b64 = data.get("audio", "")
201
- audio_wav = base64.b64decode(audio_b64)
202
- # Extrair PCM do WAV (pular header de 44 bytes)
203
- if len(audio_wav) > 44 and audio_wav[:4] == b'RIFF':
204
- audio_data = audio_wav[44:]
205
- else:
206
- audio_data = audio_wav
207
- audio_sample_rate = 24000
208
- duration = data.get("duration_ms", 0)
209
- print(f"[Wav2Lip] Audio recebido: {len(audio_data)} bytes, {duration}ms")
210
 
211
  elif msg_type == "done":
212
- print(f"[Wav2Lip] Concluido: {data.get('frames', 0)} frames")
 
213
  break
214
 
215
- elif msg_type == "status":
216
- print(f"[Wav2Lip] Status: {data.get('message')}")
217
-
218
  elif msg_type == "error":
219
- error_msg = data.get("message", "Erro")
220
- print(f"[Wav2Lip] Erro: {error_msg}")
221
- await self.send_to_client("error", message=error_msg)
222
  break
223
 
224
  elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
225
  break
226
 
227
- await wav2lip_ws.close()
228
 
229
  except Exception as e:
230
- print(f"Erro na sessao: {e}")
231
  import traceback
232
  traceback.print_exc()
233
- await self.send_to_client("error", message=str(e))
234
- return
235
 
236
- if not frames:
237
- await self.send_to_client("error", message="Nenhum frame recebido")
238
- return
239
 
240
- if not audio_data:
241
- await self.send_to_client("error", message="Nenhum audio recebido")
242
- return
 
243
 
244
- # Gerar WebM
245
- await self.send_to_client("status", message="Gerando video...")
246
 
247
- webm = WebMStreamer()
248
- await webm.start()
 
 
249
 
250
- for frame in frames:
251
- webm.add_frame(frame)
252
- webm.set_audio(audio_data, audio_sample_rate)
253
 
254
- webm_data = await webm.finalize()
255
- await webm.cleanup()
256
 
257
- if webm_data:
258
- self.chunks_sent = 1
259
- await self.send_to_client(
260
- "webm_chunk",
261
- chunk_index=1,
262
- size=len(webm_data),
263
- data=base64.b64encode(webm_data).decode()
264
- )
265
- print(f"[WebM] Enviado: {len(webm_data)} bytes, {len(frames)} frames")
266
 
267
- # Metricas finais
268
  elapsed = time.time() - self.start_time
269
- audio_duration_ms = len(audio_data) / 2 / audio_sample_rate * 1000
270
-
271
- print(f"Sessao concluida em {elapsed:.2f}s")
272
- print(f" Frames: {self.total_frames}")
273
- print(f" Audio: {audio_duration_ms:.0f}ms")
274
 
275
- await self.send_to_client(
276
- "done",
277
- total_duration_ms=int(audio_duration_ms),
278
- total_frames=self.total_frames,
279
- total_chunks=self.chunks_sent,
280
- elapsed_ms=int(elapsed * 1000)
281
- )
 
 
 
282
 
283
  def stop(self):
284
  self.is_running = False
@@ -289,7 +333,7 @@ async def websocket_handler(request):
289
  ws = web.WebSocketResponse()
290
  await ws.prepare(request)
291
 
292
- print("Cliente WebSocket conectado")
293
  session = None
294
 
295
  try:
@@ -304,7 +348,7 @@ async def websocket_handler(request):
304
  voice = data.get("voice", "tara")
305
 
306
  if not text:
307
- await ws.send_json({"type": "error", "message": "Text is required"})
308
  continue
309
 
310
  print(f"Gerando: '{text[:50]}...' voice={voice}")
@@ -312,7 +356,7 @@ async def websocket_handler(request):
312
  if session:
313
  session.stop()
314
 
315
- session = StreamingSession(ws)
316
  await session.run(text, voice)
317
 
318
  elif action == "stop":
@@ -331,11 +375,11 @@ async def websocket_handler(request):
331
  break
332
 
333
  except Exception as e:
334
- print(f"Erro no WebSocket: {e}")
335
  finally:
336
  if session:
337
  session.stop()
338
- print("Cliente WebSocket desconectado")
339
 
340
  return ws
341
 
@@ -345,18 +389,23 @@ async def index(request):
345
  return web.FileResponse(os.path.join(os.path.dirname(__file__), "index.html"))
346
 
347
 
348
- @routes.get("/health")
349
- async def health(request):
350
- status = {"server": True, "wav2lip": False}
 
 
 
 
 
 
 
 
 
351
 
352
- try:
353
- async with aiohttp.ClientSession() as session:
354
- async with session.ws_connect(WAV2LIP_WS, timeout=aiohttp.ClientWSTimeout(ws_close=2)) as ws:
355
- status["wav2lip"] = True
356
- except:
357
- pass
358
 
359
- return web.json_response(status)
 
 
360
 
361
 
362
  app = web.Application()
@@ -365,11 +414,12 @@ app.add_routes(routes)
365
 
366
  if __name__ == "__main__":
367
  print("=" * 50)
368
- print("Interface Server - Wav2Lip Streaming")
369
  print("=" * 50)
370
  print(f"Porta: {PORT}")
 
371
  print(f"Wav2Lip: {WAV2LIP_WS}")
372
- print(f"Video: {VIDEO_FPS}fps")
373
- print(f"Output: WebM (VP9 + Opus)")
374
  print("=" * 50)
375
  web.run_app(app, host="0.0.0.0", port=PORT)
 
1
  """
2
+ Interface Server - Streaming Paralelo
3
  Porta: 8080
4
 
5
+ Arquitetura (ver CLAUDE.md):
6
+ 1. Recebe texto do frontend via WebSocket
7
+ 2. Conecta ao Orpheus (8081) e Wav2Lip (8082) EM PARALELO
8
+ 3. Recebe chunks de audio do Orpheus e frames do Wav2Lip
9
+ 4. Monta chunks (audio Orpheus + frames Wav2Lip) conforme chegam
10
+ 5. Envia chunks IMEDIATAMENTE para o frontend
11
+
12
+ IMPORTANTE:
13
+ - NAO modificar Wav2Lip - ele gera lip sync com eSpeak interno
14
+ - Audio final = Orpheus (descartar audio do Wav2Lip)
15
  """
16
  from aiohttp import web
17
  import aiohttp
 
20
  import base64
21
  import os
22
  import time
 
 
23
  import struct
24
 
25
+ # Configuracao
26
+ ORPHEUS_WS = os.getenv("ORPHEUS_WS", "ws://localhost:8081/ws")
27
  WAV2LIP_WS = os.getenv("WAV2LIP_WS", "ws://localhost:8082/ws")
28
  PORT = int(os.getenv("PORT", "8080"))
29
 
30
+ # Constantes
31
+ AUDIO_SAMPLE_RATE = 24000 # Orpheus
32
+ VIDEO_FPS = 25 # Wav2Lip
33
+ BYTES_PER_SAMPLE = 2 # 16-bit
34
+ MS_PER_FRAME = 1000 / VIDEO_FPS # 40ms
35
+ AUDIO_BYTES_PER_FRAME = int(MS_PER_FRAME * AUDIO_SAMPLE_RATE * BYTES_PER_SAMPLE / 1000) # 1920 bytes
36
 
37
  routes = web.RouteTableDef()
38
 
39
 
40
+ def build_chunk(audio_bytes: bytes, frames: list) -> bytes:
41
+ """Monta chunk binario: [audio_size][audio][num_frames][frame_sizes][frames]"""
42
+ data = bytearray()
43
+ data.extend(struct.pack('>I', len(audio_bytes)))
44
+ data.extend(audio_bytes)
45
+ data.extend(struct.pack('>I', len(frames)))
46
+ for frame in frames:
47
+ data.extend(struct.pack('>I', len(frame)))
48
+ data.extend(frame)
49
+ return bytes(data)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
 
52
+ class ParallelStreamingSession:
53
+ """Streaming paralelo: Orpheus (audio) + Wav2Lip (frames)"""
54
 
55
  def __init__(self, client_ws):
56
  self.client_ws = client_ws
57
  self.is_running = False
58
  self.start_time = None
59
+
60
+ # Buffers compartilhados
61
+ self.audio_buffer = bytearray()
62
+ self.frame_buffer = []
63
+
64
+ # Locks para acesso thread-safe
65
+ self.buffer_lock = asyncio.Lock()
66
+
67
+ # Estados
68
+ self.orpheus_done = False
69
+ self.wav2lip_done = False
70
+ self.first_chunk_sent = False
71
+ self.chunk_index = 0
72
+
73
+ # Estatisticas
74
+ self.total_audio_bytes = 0
75
  self.total_frames = 0
76
  self.chunks_sent = 0
77
 
78
+ async def send_status(self, message: str):
79
+ """Envia status para o cliente."""
80
  try:
81
+ if not self.client_ws.closed:
82
+ await self.client_ws.send_json({"type": "status", "message": message})
83
+ except:
84
+ pass
85
+
86
+ async def send_chunk(self, audio: bytes, frames: list):
87
+ """Monta e envia chunk para o cliente."""
88
+ if self.client_ws.closed or not audio or not frames:
89
+ return
 
90
 
91
+ try:
92
+ chunk_data = build_chunk(audio, frames)
93
+ chunk_b64 = base64.b64encode(chunk_data).decode()
94
+ audio_ms = len(audio) / BYTES_PER_SAMPLE / AUDIO_SAMPLE_RATE * 1000
95
+
96
+ await self.client_ws.send_json({
97
+ "type": "chunk",
98
+ "chunk_index": self.chunk_index,
99
+ "audio_size": len(audio),
100
+ "audio_duration_ms": int(audio_ms),
101
+ "num_frames": len(frames),
102
+ "data": chunk_b64
103
+ })
104
+
105
+ self.chunk_index += 1
106
+ self.chunks_sent += 1
107
+
108
+ if not self.first_chunk_sent:
109
+ self.first_chunk_sent = True
110
+ ttfb = int((time.time() - self.start_time) * 1000)
111
+ print(f"[Stream] Primeiro chunk: TTFB={ttfb}ms")
112
+ await self.client_ws.send_json({"type": "stream_start", "ttfb_ms": ttfb})
113
 
114
+ except Exception as e:
115
+ print(f"[Stream] Erro enviando chunk: {e}")
116
+
117
+ async def try_send_chunks(self):
118
+ """Tenta montar e enviar chunks com dados disponiveis."""
119
+ async with self.buffer_lock:
120
+ # Enquanto tiver 1 frame + audio correspondente
121
+ while len(self.frame_buffer) > 0 and len(self.audio_buffer) >= AUDIO_BYTES_PER_FRAME:
122
+ # Pega 1 frame
123
+ frame = self.frame_buffer.pop(0)
124
+
125
+ # Pega audio correspondente (~40ms = 1920 bytes)
126
+ audio = bytes(self.audio_buffer[:AUDIO_BYTES_PER_FRAME])
127
+ del self.audio_buffer[:AUDIO_BYTES_PER_FRAME]
128
+
129
+ await self.send_chunk(audio, [frame])
130
+
131
+ # Se ambos terminaram, enviar o que sobrou
132
+ if self.orpheus_done and self.wav2lip_done:
133
+ if self.frame_buffer and self.audio_buffer:
134
+ # Dividir audio restante pelos frames restantes
135
+ audio_per_frame = len(self.audio_buffer) // len(self.frame_buffer) if self.frame_buffer else 0
136
+ audio_per_frame = max(audio_per_frame, 2) # Minimo 2 bytes
137
+ audio_per_frame = audio_per_frame - (audio_per_frame % 2) # Alinhamento 16-bit
138
+
139
+ while self.frame_buffer and self.audio_buffer:
140
+ frame = self.frame_buffer.pop(0)
141
+ audio_size = min(audio_per_frame, len(self.audio_buffer))
142
+ if not self.frame_buffer: # Ultimo frame pega todo o resto
143
+ audio_size = len(self.audio_buffer)
144
+ audio = bytes(self.audio_buffer[:audio_size])
145
+ del self.audio_buffer[:audio_size]
146
+ await self.send_chunk(audio, [frame])
147
+
148
+ elif self.frame_buffer:
149
+ # Frames sem audio - enviar com audio vazio
150
+ for frame in self.frame_buffer:
151
+ await self.send_chunk(b'', [frame])
152
+ self.frame_buffer.clear()
153
+
154
+ async def stream_orpheus(self, text: str, voice: str):
155
+ """Conecta ao Orpheus e recebe chunks de audio."""
156
  try:
157
+ print(f"[Orpheus] Conectando a {ORPHEUS_WS}...")
158
  async with aiohttp.ClientSession() as session:
159
+ ws = await session.ws_connect(ORPHEUS_WS, timeout=aiohttp.ClientWSTimeout(ws_close=120))
 
 
 
 
 
160
 
161
+ # Enviar requisicao
162
+ await ws.send_json({
163
+ "action": "synthesize",
164
  "text": text,
165
  "voice": voice,
166
+ "stream": True
167
  })
168
+ print(f"[Orpheus] Requisicao enviada")
169
+
170
+ async for msg in ws:
171
+ if not self.is_running:
172
+ break
173
+
174
+ if msg.type == aiohttp.WSMsgType.TEXT:
175
+ data = json.loads(msg.data)
176
+ msg_type = data.get("type", "")
177
+
178
+ if msg_type == "audio_chunk":
179
+ # Decodificar e adicionar ao buffer
180
+ audio_b64 = data.get("audio", "")
181
+ audio_bytes = base64.b64decode(audio_b64)
182
+
183
+ async with self.buffer_lock:
184
+ self.audio_buffer.extend(audio_bytes)
185
+ self.total_audio_bytes += len(audio_bytes)
186
+
187
+ # Tentar enviar chunks
188
+ await self.try_send_chunks()
189
+
190
+ chunk_idx = data.get("chunk_index", 0)
191
+ if chunk_idx == 1:
192
+ print(f"[Orpheus] Primeiro chunk de audio recebido")
193
+
194
+ elif msg_type == "done":
195
+ total = data.get("total_bytes", 0)
196
+ print(f"[Orpheus] Concluido: {total} bytes")
197
+ break
198
+
199
+ elif msg_type == "error":
200
+ print(f"[Orpheus] Erro: {data.get('message')}")
201
+ break
202
 
203
+ elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
204
+ break
205
+
206
+ await ws.close()
207
+
208
+ except Exception as e:
209
+ print(f"[Orpheus] Erro: {e}")
210
+ import traceback
211
+ traceback.print_exc()
212
+
213
+ finally:
214
+ self.orpheus_done = True
215
+ await self.try_send_chunks()
216
+
217
+ async def stream_wav2lip(self, text: str, voice: str):
218
+ """Conecta ao Wav2Lip e recebe frames."""
219
+ try:
220
+ print(f"[Wav2Lip] Conectando a {WAV2LIP_WS}...")
221
+ async with aiohttp.ClientSession() as session:
222
+ ws = await session.ws_connect(WAV2LIP_WS, timeout=aiohttp.ClientWSTimeout(ws_close=120))
223
 
224
+ # Enviar requisicao
225
+ await ws.send_json({
226
+ "action": "generate",
227
+ "text": text,
228
+ "voice": voice
229
+ })
230
+ print(f"[Wav2Lip] Requisicao enviada")
231
 
232
+ async for msg in ws:
 
233
  if not self.is_running:
234
  break
235
 
 
237
  data = json.loads(msg.data)
238
  msg_type = data.get("type", "")
239
 
240
+ if msg_type == "frame":
241
+ # Decodificar e adicionar ao buffer
 
 
 
 
242
  frame_b64 = data.get("frame", "")
243
  frame_bytes = base64.b64decode(frame_b64)
 
 
244
 
245
+ async with self.buffer_lock:
246
+ self.frame_buffer.append(frame_bytes)
247
+ self.total_frames += 1
248
+
249
+ # Tentar enviar chunks
250
+ await self.try_send_chunks()
251
+
252
+ frame_idx = data.get("index", 0)
253
+ if frame_idx == 0:
254
+ print(f"[Wav2Lip] Primeiro frame recebido")
255
+
256
+ elif msg_type == "status":
257
+ print(f"[Wav2Lip] {data.get('message')}")
258
+
259
+ elif msg_type == "first_chunk":
260
+ print(f"[Wav2Lip] eSpeak latency: {data.get('latency_ms')}ms")
261
 
262
  elif msg_type == "full_audio":
263
+ # Ignorar - usamos audio do Orpheus
264
+ print(f"[Wav2Lip] Audio ignorado (usando Orpheus)")
 
 
 
 
 
 
 
 
265
 
266
  elif msg_type == "done":
267
+ frames = data.get("frames", 0)
268
+ print(f"[Wav2Lip] Concluido: {frames} frames")
269
  break
270
 
 
 
 
271
  elif msg_type == "error":
272
+ print(f"[Wav2Lip] Erro: {data.get('message')}")
273
+ await self.send_status(f"Erro: {data.get('message')}")
 
274
  break
275
 
276
  elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
277
  break
278
 
279
+ await ws.close()
280
 
281
  except Exception as e:
282
+ print(f"[Wav2Lip] Erro: {e}")
283
  import traceback
284
  traceback.print_exc()
 
 
285
 
286
+ finally:
287
+ self.wav2lip_done = True
288
+ await self.try_send_chunks()
289
 
290
+ async def run(self, text: str, voice: str):
291
+ """Executa streaming paralelo."""
292
+ self.is_running = True
293
+ self.start_time = time.time()
294
 
295
+ await self.send_status("Conectando aos servicos...")
 
296
 
297
+ try:
298
+ # Conectar Orpheus e Wav2Lip EM PARALELO
299
+ orpheus_task = asyncio.create_task(self.stream_orpheus(text, voice))
300
+ wav2lip_task = asyncio.create_task(self.stream_wav2lip(text, voice))
301
 
302
+ # Aguardar ambos terminarem
303
+ await asyncio.gather(orpheus_task, wav2lip_task)
 
304
 
305
+ # Enviar chunks restantes
306
+ await self.try_send_chunks()
307
 
308
+ except Exception as e:
309
+ print(f"[Stream] Erro: {e}")
310
+ await self.client_ws.send_json({"type": "error", "message": str(e)})
311
+ return
 
 
 
 
 
312
 
 
313
  elapsed = time.time() - self.start_time
314
+ print(f"[Stream] Concluido: {elapsed:.2f}s, {self.chunks_sent} chunks, {self.total_frames} frames, {self.total_audio_bytes} bytes")
 
 
 
 
315
 
316
+ try:
317
+ await self.client_ws.send_json({
318
+ "type": "done",
319
+ "total_chunks": self.chunks_sent,
320
+ "total_frames": self.total_frames,
321
+ "total_audio_bytes": self.total_audio_bytes,
322
+ "elapsed_ms": int(elapsed * 1000)
323
+ })
324
+ except:
325
+ pass
326
 
327
  def stop(self):
328
  self.is_running = False
 
333
  ws = web.WebSocketResponse()
334
  await ws.prepare(request)
335
 
336
+ print("Cliente conectado")
337
  session = None
338
 
339
  try:
 
348
  voice = data.get("voice", "tara")
349
 
350
  if not text:
351
+ await ws.send_json({"type": "error", "message": "Text required"})
352
  continue
353
 
354
  print(f"Gerando: '{text[:50]}...' voice={voice}")
 
356
  if session:
357
  session.stop()
358
 
359
+ session = ParallelStreamingSession(ws)
360
  await session.run(text, voice)
361
 
362
  elif action == "stop":
 
375
  break
376
 
377
  except Exception as e:
378
+ print(f"Erro: {e}")
379
  finally:
380
  if session:
381
  session.stop()
382
+ print("Cliente desconectado")
383
 
384
  return ws
385
 
 
389
  return web.FileResponse(os.path.join(os.path.dirname(__file__), "index.html"))
390
 
391
 
392
+ @routes.get("/idle.mp4")
393
+ async def idle_video(request):
394
+ return web.FileResponse(os.path.join(os.path.dirname(__file__), "idle.mp4"))
395
+
396
+
397
+ @routes.get("/{filename}")
398
+ async def static_file(request):
399
+ filename = request.match_info["filename"]
400
+ filepath = os.path.join(os.path.dirname(__file__), filename)
401
+ if os.path.exists(filepath):
402
+ return web.FileResponse(filepath)
403
+ return web.Response(status=404)
404
 
 
 
 
 
 
 
405
 
406
+ @routes.get("/health")
407
+ async def health(request):
408
+ return web.json_response({"status": "ok", "mode": "parallel_streaming"})
409
 
410
 
411
  app = web.Application()
 
414
 
415
  if __name__ == "__main__":
416
  print("=" * 50)
417
+ print("Interface Server - Streaming Paralelo")
418
  print("=" * 50)
419
  print(f"Porta: {PORT}")
420
+ print(f"Orpheus: {ORPHEUS_WS}")
421
  print(f"Wav2Lip: {WAV2LIP_WS}")
422
+ print(f"Audio: {AUDIO_SAMPLE_RATE}Hz, {AUDIO_BYTES_PER_FRAME} bytes/frame")
423
+ print(f"Video: {VIDEO_FPS}fps, {MS_PER_FRAME}ms/frame")
424
  print("=" * 50)
425
  web.run_app(app, host="0.0.0.0", port=PORT)
interface/test_streaming.py ADDED
@@ -0,0 +1,232 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """
3
+ Test client para streaming progressivo.
4
+ Conecta via WebSocket, recebe chunks, monta e salva vídeo.
5
+ """
6
+ import asyncio
7
+ import aiohttp
8
+ import json
9
+ import base64
10
+ import struct
11
+ import os
12
+ import sys
13
+ import wave
14
+ import subprocess
15
+ import tempfile
16
+ import time
17
+
18
+ # Configuracao
19
+ WS_URL = os.getenv("WS_URL", "ws://localhost:8090/ws")
20
+ TEXT = os.getenv("TEXT", "Hello! I am testing the progressive streaming avatar system.")
21
+ VOICE = os.getenv("VOICE", "tara")
22
+ OUTPUT_DIR = "/tmp/streaming_test"
23
+
24
+ async def test_streaming():
25
+ print("=" * 60)
26
+ print("Test Streaming Client")
27
+ print("=" * 60)
28
+ print(f"WebSocket: {WS_URL}")
29
+ print(f"Text: {TEXT[:50]}...")
30
+ print(f"Voice: {VOICE}")
31
+ print(f"Output: {OUTPUT_DIR}")
32
+ print("=" * 60)
33
+
34
+ # Criar diretorio de saida
35
+ os.makedirs(OUTPUT_DIR, exist_ok=True)
36
+
37
+ # Buffers para acumular dados
38
+ all_audio = bytearray()
39
+ all_frames = []
40
+ chunks_received = 0
41
+ start_time = None
42
+ first_chunk_time = None
43
+
44
+ try:
45
+ async with aiohttp.ClientSession() as session:
46
+ print("\nConectando ao WebSocket...")
47
+ ws = await session.ws_connect(WS_URL, timeout=aiohttp.ClientWSTimeout(ws_close=120))
48
+ print("Conectado!")
49
+
50
+ # Enviar requisicao
51
+ print(f"\nEnviando requisicao...")
52
+ await ws.send_json({
53
+ "action": "generate",
54
+ "text": TEXT,
55
+ "voice": VOICE
56
+ })
57
+ start_time = time.time()
58
+
59
+ # Receber mensagens
60
+ async for msg in ws:
61
+ if msg.type == aiohttp.WSMsgType.TEXT:
62
+ data = json.loads(msg.data)
63
+ msg_type = data.get("type", "")
64
+
65
+ if msg_type == "status":
66
+ print(f" Status: {data.get('message')}")
67
+
68
+ elif msg_type == "stream_start":
69
+ ttfb = data.get("ttfb_ms", 0)
70
+ first_chunk_time = time.time()
71
+ print(f" Stream iniciado! TTFB: {ttfb}ms")
72
+
73
+ elif msg_type == "chunk":
74
+ chunks_received += 1
75
+ chunk_index = data.get("chunk_index", 0)
76
+ audio_size = data.get("audio_size", 0)
77
+ num_frames = data.get("num_frames", 0)
78
+
79
+ # Decodificar chunk
80
+ chunk_data = base64.b64decode(data.get("data", ""))
81
+ view = memoryview(chunk_data)
82
+ offset = 0
83
+
84
+ # Ler audio
85
+ audio_len = struct.unpack(">I", view[offset:offset+4])[0]
86
+ offset += 4
87
+ audio_bytes = bytes(view[offset:offset+audio_len])
88
+ offset += audio_len
89
+ all_audio.extend(audio_bytes)
90
+
91
+ # Ler frames
92
+ frames_count = struct.unpack(">I", view[offset:offset+4])[0]
93
+ offset += 4
94
+ for i in range(frames_count):
95
+ frame_len = struct.unpack(">I", view[offset:offset+4])[0]
96
+ offset += 4
97
+ frame_bytes = bytes(view[offset:offset+frame_len])
98
+ offset += frame_len
99
+ all_frames.append(frame_bytes)
100
+
101
+ print(f" Chunk {chunk_index}: {audio_size} bytes audio, {num_frames} frames (total: {len(all_frames)} frames, {len(all_audio)} bytes audio)")
102
+
103
+ elif msg_type == "done":
104
+ total_chunks = data.get("total_chunks", 0)
105
+ total_frames = data.get("total_frames", 0)
106
+ elapsed_ms = data.get("elapsed_ms", 0)
107
+ print(f"\n Done! {total_chunks} chunks, {total_frames} frames, {elapsed_ms}ms")
108
+ break
109
+
110
+ elif msg_type == "error":
111
+ print(f" ERRO: {data.get('message')}")
112
+ break
113
+
114
+ elif msg.type in (aiohttp.WSMsgType.CLOSED, aiohttp.WSMsgType.ERROR):
115
+ print(" WebSocket fechado")
116
+ break
117
+
118
+ await ws.close()
119
+
120
+ except Exception as e:
121
+ print(f"Erro: {e}")
122
+ import traceback
123
+ traceback.print_exc()
124
+ return
125
+
126
+ total_time = time.time() - start_time if start_time else 0
127
+
128
+ print("\n" + "=" * 60)
129
+ print("Resultados:")
130
+ print("=" * 60)
131
+ print(f"Chunks recebidos: {chunks_received}")
132
+ print(f"Frames recebidos: {len(all_frames)}")
133
+ print(f"Audio recebido: {len(all_audio)} bytes")
134
+ print(f"Tempo total: {total_time:.2f}s")
135
+
136
+ if not all_frames or not all_audio:
137
+ print("\nNenhum dado recebido!")
138
+ return
139
+
140
+ # Salvar frames como JPEGs
141
+ print(f"\nSalvando {len(all_frames)} frames...")
142
+ frames_dir = os.path.join(OUTPUT_DIR, "frames")
143
+ os.makedirs(frames_dir, exist_ok=True)
144
+ for i, frame in enumerate(all_frames):
145
+ frame_path = os.path.join(frames_dir, f"frame_{i:05d}.jpg")
146
+ with open(frame_path, "wb") as f:
147
+ f.write(frame)
148
+
149
+ # Salvar audio como WAV
150
+ print(f"Salvando audio ({len(all_audio)} bytes)...")
151
+ audio_path = os.path.join(OUTPUT_DIR, "audio.wav")
152
+
153
+ # Detectar sample rate pelo tamanho do audio e frames
154
+ # 25 fps, cada frame = 40ms
155
+ # audio_duration = len(all_frames) * 40ms
156
+ audio_duration_ms = len(all_frames) * 40
157
+ samples = len(all_audio) // 2 # 16-bit = 2 bytes por sample
158
+ if audio_duration_ms > 0:
159
+ estimated_sample_rate = int(samples / (audio_duration_ms / 1000))
160
+ # Arredondar para sample rates comuns
161
+ if estimated_sample_rate > 20000:
162
+ sample_rate = 24000
163
+ else:
164
+ sample_rate = 22050
165
+ else:
166
+ sample_rate = 24000
167
+
168
+ print(f"Sample rate estimado: {sample_rate} Hz")
169
+
170
+ with wave.open(audio_path, "wb") as wf:
171
+ wf.setnchannels(1)
172
+ wf.setsampwidth(2) # 16-bit
173
+ wf.setframerate(sample_rate)
174
+ wf.writeframes(all_audio)
175
+
176
+ # Criar video com ffmpeg
177
+ print("\nCriando video com ffmpeg...")
178
+ video_path = os.path.join(OUTPUT_DIR, "output.mp4")
179
+
180
+ # Calcular fps para sincronizar com audio
181
+ audio_duration_s = len(all_audio) / 2 / sample_rate
182
+ if audio_duration_s > 0:
183
+ video_fps = len(all_frames) / audio_duration_s
184
+ else:
185
+ video_fps = 25
186
+
187
+ print(f"Video FPS: {video_fps:.2f}")
188
+
189
+ cmd = [
190
+ "ffmpeg", "-y",
191
+ "-framerate", str(video_fps),
192
+ "-i", os.path.join(frames_dir, "frame_%05d.jpg"),
193
+ "-i", audio_path,
194
+ "-vf", "pad=ceil(iw/2)*2:ceil(ih/2)*2", # Fix odd dimensions for libx264
195
+ "-c:v", "libx264",
196
+ "-preset", "fast",
197
+ "-crf", "23",
198
+ "-c:a", "aac",
199
+ "-b:a", "128k",
200
+ "-pix_fmt", "yuv420p",
201
+ "-shortest",
202
+ video_path
203
+ ]
204
+
205
+ result = subprocess.run(cmd, capture_output=True, text=True)
206
+ if result.returncode != 0:
207
+ print(f"FFmpeg erro: {result.stderr[-500:]}")
208
+ else:
209
+ print(f"\nVideo salvo em: {video_path}")
210
+ # Mostrar info do video
211
+ probe = subprocess.run(
212
+ ["ffprobe", "-v", "quiet", "-print_format", "json", "-show_format", video_path],
213
+ capture_output=True, text=True
214
+ )
215
+ if probe.returncode == 0:
216
+ import json as json_mod
217
+ info = json_mod.loads(probe.stdout)
218
+ duration = float(info.get("format", {}).get("duration", 0))
219
+ size = int(info.get("format", {}).get("size", 0))
220
+ print(f"Duracao: {duration:.2f}s")
221
+ print(f"Tamanho: {size / 1024:.1f} KB")
222
+
223
+ print("\n" + "=" * 60)
224
+ print("Arquivos gerados:")
225
+ print("=" * 60)
226
+ print(f" Frames: {frames_dir}/")
227
+ print(f" Audio: {audio_path}")
228
+ print(f" Video: {video_path}")
229
+
230
+
231
+ if __name__ == "__main__":
232
+ asyncio.run(test_streaming())