| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - audio-to-video |
| - talking-head |
| - digital-human |
| - omnirt |
| - opentalking |
| --- |
| |
| # QuickTalk |
|
|
| This repository contains the QuickTalk model files used by the OmniRT `quicktalk` audio-to-video runtime. |
|
|
| ## Files |
|
|
| ```text |
| quicktalk.pth # QuickTalk PyTorch checkpoint |
| repair.npy # QuickTalk repair parameters used by the runtime post-processing stage |
| ``` |
|
|
| This repository intentionally does **not** include third-party dependency weights such as HuBERT or InsightFace `buffalo_l`. Download those dependencies from their original sources according to their own licenses and place them next to these files when running OmniRT. |
|
|
| Expected runtime layout: |
|
|
| ```text |
| $OMNIRT_MODEL_ROOT/quicktalk/ |
| quicktalk.pth |
| repair.npy |
| chinese-hubert-large/ |
| config.json |
| preprocessor_config.json |
| pytorch_model.bin |
| auxiliary/models/buffalo_l/ |
| <InsightFace model files> |
| ``` |
|
|
| ## Repair Parameters |
|
|
| `repair.npy` is a required QuickTalk runtime parameter file. It is not a standalone neural network checkpoint. The runtime loads it after the main QuickTalk model to apply fixed repair parameters during post-processing, helping map the model output back to the template face more consistently. |
|
|
| Keep `repair.npy` in the same QuickTalk model directory as `quicktalk.pth`. |
|
|
| ## Usage With OmniRT |
|
|
| Set the model root and start the QuickTalk runtime through OmniRT: |
|
|
| ```bash |
| export OMNIRT_MODEL_ROOT=/path/to/models |
| export OMNIRT_QUICKTALK_MODEL_ROOT="$OMNIRT_MODEL_ROOT/quicktalk" |
| export OMNIRT_QUICKTALK_RUNTIME=1 |
| export OMNIRT_QUICKTALK_DEVICE=cuda:0 |
| export OMNIRT_QUICKTALK_HUBERT_DEVICE=cuda:1 |
| |
| omnirt-serve --host 0.0.0.0 --port 9000 |
| ``` |
|
|
| OpenTalking can then connect through the unified OmniRT endpoint: |
|
|
| ```bash |
| export OMNIRT_ENDPOINT=http://127.0.0.1:9000 |
| ``` |
|
|
| The QuickTalk WebSocket route is: |
|
|
| ```text |
| /v1/audio2video/quicktalk |
| ``` |
|
|
| ## Related Projects |
|
|
| - OpenTalking: <https://github.com/datascale-ai/opentalking> |
| - OmniRT: <https://github.com/datascale-ai/omnirt> |
|
|
| ## Security Notes |
|
|
| `quicktalk.pth` is a PyTorch checkpoint. PyTorch checkpoint files are pickle-based, and unpickling arbitrary `.pth` files can execute code. For that reason, Hugging Face security scanners may flag this file as suspicious or unsafe, for example via Picklescan or generic Python obfuscation signatures. |
|
|
| This warning is expected for many pickle-based PyTorch checkpoints and does not by itself prove that the file is malicious. It does mean you should treat the file as executable input: |
|
|
| - only load it in a trusted environment; |
| - only use it with the intended OmniRT QuickTalk loader; |
| - do not load it with generic `torch.load` in an untrusted or multi-tenant environment; |
| - prefer isolated runtime users, containers, or virtual machines for production deployments. |
|
|
| The uploaded files in this repository are limited to the QuickTalk checkpoint and repair parameters: |
|
|
| ```text |
| quicktalk.pth |
| repair.npy |
| ``` |
|
|
| No runtime caches, template caches, HuBERT weights, or InsightFace weights are included. |
|
|
