setup/setup.md

Speech-X Setup Guide

Step-by-step install for the avatar conda environment. Adapted from the MuseTalk official docs for Python 3.12.

Or run the automated script from the repo root:

bash setup/setup.sh          # Linux / macOS
.\setup\setup.ps1            # Windows (PowerShell)

Stage 1: Create Environment

conda create -n avatar python=3.12
conda activate avatar

Stage 2: Install PyTorch

Python 3.12 requires PyTorch 2.5+ (2.0.1 doesn't support 3.12)

pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124

Stage 3: Install MMLab Packages

These are critical for MuseTalk - install one by one

pip install --no-cache-dir -U openmim
mim install mmengine
mim install mmcv==2.2.0
# If above fails, try: pip install mmcv-lite==2.2.0
mim install mmdet==3.3.0
# Note: mmpose not required — not present in the reference env

Stage 4: Install Musetalk Dependencies

These are from musetalk's official requirements.txt

pip install diffusers==0.30.2
pip install accelerate==0.28.0
pip install numpy==2.4.2
pip install opencv-python==4.13.0.92
pip install soundfile==0.12.1
pip install transformers==4.39.2
pip install huggingface-hub==0.36.2
pip install librosa==0.10.2
pip install einops==0.8.1

pip install gdown
pip install requests
pip install imageio==2.34.0
pip install imageio-ffmpeg

pip install omegaconf==2.3.0
pip install ffmpeg-python
pip install moviepy

Stage 5: Install Additional Dependencies

For this speech-to-video project

pip install fastapi>=0.115.0
pip install uvicorn[standard]>=0.30.0
pip install pydantic>=2.10.0
pip install python-dotenv>=1.0.1
pip install livekit>=0.10.0
pip install livekit-agents>=0.8.0
pip install kokoro-onnx>=0.5.0
pip install scipy>=1.13.0
pip install faster-whisper>=1.0.0
pip install sse-starlette>=2.0.0
pip install onnxruntime>=1.24.0
pip install sounddevice>=0.5.0
pip install tqdm>=4.65.0
pip install pyyaml>=6.0.0
pip install aiohttp>=3.9.0
pip install httpx>=0.27.0
pip install safetensors>=0.4.0
pip install pillow>=10.0.0

Quick Test

# Test MuseTalk imports
python -c "
import sys
sys.path.insert(0, 'backend')
from musetalk.processor import *
print('MuseTalk OK')
"

# Test TTS import
python -c "
import sys
sys.path.insert(0, 'backend')
from tts.kokoro_tts import KokoroTTS
print('KokoroTTS OK')
"

Avatar Creation

Run once per avatar before starting the server. Script reads from backend/config.py for model paths and writes assets to backend/avatars/<name>/.

Single portrait image:

conda activate avatar
python setup/avatar_creation.py --image frontend/public/Sophy.png --name sophy

Talking-head video:

python setup/avatar_creation.py --video /path/to/talking_head.mp4 --name harry_1

Batch (multiple avatars at once):

# Edit setup/avatars_config.yml first, then:
python setup/avatar_creation.py --config setup/avatars_config.yml

Options:

Flag	Default	Description
--name	required	Avatar folder name
--frames	50	Frame count for --image mode
--bbox-shift	5	Vertical bbox nudge (tune if face crop is off)
--device	cuda	cuda or cpu
--overwrite	off	Skip re-create prompt