Update QuickTalk model card
Browse files
README.md
CHANGED
|
@@ -1,3 +1,83 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- audio-to-video
|
| 6 |
+
- talking-head
|
| 7 |
+
- digital-human
|
| 8 |
+
- omnirt
|
| 9 |
+
- opentalking
|
| 10 |
---
|
| 11 |
+
|
| 12 |
+
# QuickTalk
|
| 13 |
+
|
| 14 |
+
This repository contains the QuickTalk model files used by the OmniRT `quicktalk` audio-to-video runtime.
|
| 15 |
+
|
| 16 |
+
## Files
|
| 17 |
+
|
| 18 |
+
```text
|
| 19 |
+
quicktalk.pth # QuickTalk PyTorch checkpoint
|
| 20 |
+
repair.npy # QuickTalk repair parameters
|
| 21 |
+
```
|
| 22 |
+
|
| 23 |
+
This repository intentionally does **not** include third-party dependency weights such as HuBERT or InsightFace `buffalo_l`. Download those dependencies from their original sources according to their own licenses and place them next to these files when running OmniRT.
|
| 24 |
+
|
| 25 |
+
Expected runtime layout:
|
| 26 |
+
|
| 27 |
+
```text
|
| 28 |
+
$OMNIRT_MODEL_ROOT/quicktalk/
|
| 29 |
+
quicktalk.pth
|
| 30 |
+
repair.npy
|
| 31 |
+
chinese-hubert-large/
|
| 32 |
+
config.json
|
| 33 |
+
preprocessor_config.json
|
| 34 |
+
pytorch_model.bin
|
| 35 |
+
auxiliary/models/buffalo_l/
|
| 36 |
+
*.onnx
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
## Usage With OmniRT
|
| 40 |
+
|
| 41 |
+
Set the model root and start the QuickTalk runtime through OmniRT:
|
| 42 |
+
|
| 43 |
+
```bash
|
| 44 |
+
export OMNIRT_MODEL_ROOT=/path/to/models
|
| 45 |
+
export OMNIRT_QUICKTALK_MODEL_ROOT="$OMNIRT_MODEL_ROOT/quicktalk"
|
| 46 |
+
export OMNIRT_QUICKTALK_RUNTIME=1
|
| 47 |
+
export OMNIRT_QUICKTALK_DEVICE=cuda:0
|
| 48 |
+
export OMNIRT_QUICKTALK_HUBERT_DEVICE=cuda:1
|
| 49 |
+
|
| 50 |
+
omnirt-serve --host 0.0.0.0 --port 9000
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
OpenTalking can then connect through the unified OmniRT endpoint:
|
| 54 |
+
|
| 55 |
+
```bash
|
| 56 |
+
export OMNIRT_ENDPOINT=http://127.0.0.1:9000
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
The QuickTalk WebSocket route is:
|
| 60 |
+
|
| 61 |
+
```text
|
| 62 |
+
/v1/audio2video/quicktalk
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
## Security Notes
|
| 66 |
+
|
| 67 |
+
`quicktalk.pth` is a PyTorch checkpoint. PyTorch checkpoint files are pickle-based, and unpickling arbitrary `.pth` files can execute code. For that reason, Hugging Face security scanners may flag this file as suspicious or unsafe, for example via Picklescan or generic Python obfuscation signatures.
|
| 68 |
+
|
| 69 |
+
This warning is expected for many pickle-based PyTorch checkpoints and does not by itself prove that the file is malicious. It does mean you should treat the file as executable input:
|
| 70 |
+
|
| 71 |
+
- only load it in a trusted environment;
|
| 72 |
+
- only use it with the intended OmniRT QuickTalk loader;
|
| 73 |
+
- do not load it with generic `torch.load` in an untrusted or multi-tenant environment;
|
| 74 |
+
- prefer isolated runtime users, containers, or virtual machines for production deployments.
|
| 75 |
+
|
| 76 |
+
The uploaded files in this repository are limited to the QuickTalk checkpoint and repair parameters:
|
| 77 |
+
|
| 78 |
+
```text
|
| 79 |
+
quicktalk.pth
|
| 80 |
+
repair.npy
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
No runtime caches, template caches, HuBERT weights, or InsightFace weights are included.
|