|
|
| Each core returns the probability that the input is fake. The unified script combines |
| them as a weighted sum: |
|
|
| ``` |
| final_fakeness = 0.5·lipsync + 0.4·audio + 0.1·video |
| ``` |
|
|
| (weights are renormalised over whichever cores actually produced a score). |
|
|
| --- |
|
|
| ## How it routes input |
|
|
| ``` |
| ┌──────────── audio file ───────────► audio core ──► final = audio score |
| INPUT ──┤ |
| └──────────── video file ──┬─ extract wav ─► audio core ┐ |
| ├──────────────► video core ├─► weighted final |
| └──────────────► lipsync core ┘ |
| ``` |
|
|
| Audio-only input only the audio core runs; the final score *is* the audio score. |
| Video input audio is extracted with `ffmpeg` and sent to the audio core; the video |
| file goes to the video and lipsync cores. |
|
|
|
|
|
|
|
|
| ## Requirements |
|
|
| Linux `ffmpeg` / `ffprobe`** on `PATH` |
| (the scripts default to `/usr/bin/ffmpeg`, `/usr/bin/ffprobe`). |
| Three isolated Python environments, since the cores have mutually incompatible |
| dependencies, so they cannot share one environment: |
|
|
| | core | interpreter (default) | key pins | |
| |------|-----------------------|----------| |
| | audio | `/opt/dfdetect-envs/audio/bin/python` | torch 2.6.0, transformers 4.44.0, **scikit-learn 1.3.2**, scipy 1.13.1, joblib 1.4.2, librosa, soundfile | |
| | video | `/opt/dfdetect-envs/video/bin/python` | torch 2.2.2, torchvision 0.17.2, transformers 4.51.2, numpy 1.26.4, opencv-python 4.11.0.86 | |
| | lipsync | `/opt/conda/envs/avh/bin/python` | local fairseq + avhubert, dlib, skvideo, python_speech_features, numpy 1.25 | |
|
|
| Put the environments on LOCAL disk (e.g. `/opt`) |
|
|
|
|
| ### Recreating the environments |
|
|
| ```bash |
| # audio |
| python -m venv /opt/dfdetect-envs/audio |
| /opt/dfdetect-envs/audio/bin/python -m pip install -U pip wheel |
| /opt/dfdetect-envs/audio/bin/python -m pip install \ |
| torch==2.6.0 torchvision==0.21.0 transformers==4.44.0 \ |
| scikit-learn==1.3.2 scipy==1.13.1 joblib==1.4.2 \ |
| librosa soundfile accelerate "huggingface_hub<0.26" pandas numpy |
| |
| # video |
| python -m venv /opt/dfdetect-envs/video |
| /opt/dfdetect-envs/video/bin/python -m pip install -U pip wheel |
| /opt/dfdetect-envs/video/bin/python -m pip install \ |
| torch==2.2.2 torchvision==0.17.2 transformers==4.51.2 numpy==1.26.4 \ |
| opencv-python==4.11.0.86 huggingface-hub safetensors accelerate pillow scipy pandas |
| |
| # lipsync — a conda env (Python 3.10) named `avh` |
| conda create -n avh python=3.10 -y |
| /opt/conda/envs/avh/bin/python -m pip install -U pip wheel |
| /opt/conda/envs/avh/bin/python -m pip install -r avh-align_core/requirements.txt |
| |
| ``` |
|
|
| > The vendored `avh-align_core/fairseq/` is the *importable package only* (no `setup.py`), |
| > so it works via `sys.path` at runtime but is **not pip-installable** — that is why the |
| > recipe above installs fairseq from source. |
| |
| |
| |
| ## Inference — `detect.py` |
| |
| ```bash |
| python detect.py INPUT_FILE # text report |
| python detect.py INPUT_FILE --json # machine-readable JSON |
| |
| ``` |
| |
| |
| |
| --- |
| |
| ## Retraining |
| |
| Each script reuses the matching core's frozen feature extractor (so the feature space |
| stays identical to inference) and retrains only the lightweight head. They auto-relaunch |
| into the correct environment, so just run them with any Python. |
| |
| ### Label conventions (important — audio is inverted!) |
| |
| | script | metadata columns | label meaning | |
| |--------|------------------|---------------| |
| | `retrain_audio.py` | `file_path,label` | **`1` = real, `0` = fake** | |
| | `retrain_video.py` | `file_path,label` | **`1` = fake, `0` = real** | |
| | `retrain_lipsync.py` | `file_path` (or `path`) | **real videos only** (no label — unsupervised alingment network) | |
| |
| |
| |
| python retrain_audio.py --metadata audio_meta.csv |
| |
| |
| python retrain_video.py --metadata video_meta.csv --epochs 40 |
| |
| |
| python retrain_lipsync.py --metadata real_videos.csv --epochs 10 --skip-existing |
| |
| |
| Each script has `--help` for all options (epochs, lr, batch size, `--out`, etc.). |
| |
| ### Using a retrained model |
| |
| Point `detect.py` at it — the weights are swapped in after the core loads: |
| |
| ```bash |
| python detect.py clip.mp4 \ |
| --audio-model retrained_models/audio/logreg_retrained.joblib \ |
| --video-model retrained_models/video/cnn_retrained.pt \ |
| --lipsync-model retrained_models/lipsync/fusion_retrained.pt |
| ``` |
| |
| --- |
| |
| ## Troubleshooting |
| |
| A core reports `FAILED — its score is dropped and the remaining cores are |
| reweighted; rerun with `--verbose` to see that core's traceback (e.g. lipsync needs a |
| detectable face; very short clips may yield no segments). |
| No GPU / busy GPU — cores fall back to CPU (much slower). The box is shared, so |
| throughput depends on how many of the GPUs are free. |
| |
