|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- KwaiVGI/LivePortrait |
|
|
pipeline_tag: image-to-video |
|
|
--- |
|
|
|
|
|
<p align="center"> |
|
|
<img src="./assets/showcase2.gif" alt="showcase"> |
|
|
</p> |
|
|
|
|
|
# LivePortrait |
|
|
|
|
|
This version of LivePortrait has been converted to run on the Axera NPU using **w8a16** quantization. |
|
|
|
|
|
This model has been optimized with the following: |
|
|
|
|
|
Compatible with Pulsar2 version: 3.4 |
|
|
|
|
|
## Convert tools links: |
|
|
|
|
|
For those who are interested in model conversion: |
|
|
|
|
|
- [the original repo](https://huggingface.co/KwaiVGI/LivePortrait) |
|
|
- [Github for LivePortrait](https://github.com/AXERA-TECH/LivePortrait.axera) |
|
|
|
|
|
## Support Platform |
|
|
- AX650/AX8850 |
|
|
- [M4N-Dock(η±θ―ζ΄ΎPro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) |
|
|
- [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) |
|
|
|
|
|
## How to use |
|
|
|
|
|
Download all files from this repository to the device. |
|
|
|
|
|
``` |
|
|
(py310) axera@dell:~/samples/LivePortrait$ tree -L 2 |
|
|
. |
|
|
βββ assets |
|
|
βΒ Β βββ examples |
|
|
βββ config.json |
|
|
βββ python |
|
|
βΒ Β βββ axmodels |
|
|
βΒ Β βββ cropper.py |
|
|
βΒ Β βββ infer_onnx.py |
|
|
βΒ Β βββ infer.py |
|
|
βΒ Β βββ pretrained_weights |
|
|
βΒ Β βββ requirements.txt |
|
|
βΒ Β βββ utils |
|
|
βββ README.md |
|
|
|
|
|
7 directories, 6 files |
|
|
``` |
|
|
|
|
|
### python env requirement |
|
|
|
|
|
#### pyaxengine |
|
|
|
|
|
https://github.com/AXERA-TECH/pyaxengine |
|
|
|
|
|
``` |
|
|
wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.3.rc1/axengine-0.1.3-py3-none-any.whl |
|
|
pip install axengine-0.1.3-py3-none-any.whl |
|
|
``` |
|
|
|
|
|
#### others |
|
|
|
|
|
``` |
|
|
pip install -r python/requirements.txt |
|
|
``` |
|
|
|
|
|
## Inference with AX650 or AX8850 Host, such as AX650 DEMO BOARD, M4N-DOCK(η±θ―ζ΄ΎPro) |
|
|
|
|
|
``` |
|
|
root@ax650 ~/yongqiang/LivePortrait.axera # python3 ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d8.jpg --models ./python/axmodels/ --output-dir ./axmodel_infer |
|
|
[INFO] Available providers: ['AxEngineExecutionProvider'] |
|
|
[INFO] Using provider: AxEngineExecutionProvider |
|
|
[INFO] Chip type: ChipType.MC50 |
|
|
[INFO] VNPU type: VNPUType.DISABLED |
|
|
[INFO] Engine version: 2.12.0s |
|
|
[INFO] Model type: 2 (triple core) |
|
|
[INFO] Compiler version: 3.3 144960ad |
|
|
[INFO] Using provider: AxEngineExecutionProvider |
|
|
[INFO] Model type: 2 (triple core) |
|
|
[INFO] Compiler version: 3.3 144960ad |
|
|
[INFO] Using provider: AxEngineExecutionProvider |
|
|
[INFO] Model type: 2 (triple core) |
|
|
[INFO] Compiler version: 3.3 0f7260e8 |
|
|
[INFO] Using provider: AxEngineExecutionProvider |
|
|
[INFO] Model type: 2 (triple core) |
|
|
[INFO] Compiler version: 3.3 144960ad |
|
|
FaceAnalysisDIY warmup time: 0.598s |
|
|
LandmarkRunner warmup time: 0.769s |
|
|
2025-05-30 09:56:12.247 | INFO | __main__:main:727 - Start making driving motion template... |
|
|
2025-05-30 09:56:14.770 | INFO | __main__:main:747 - Prepared pasteback mask done. |
|
|
2025-05-30 09:56:17.219 | INFO | __main__:main:787 - The output of image-driven portrait animation is an image. |
|
|
2025-05-30 09:56:30.701 | DEBUG | __main__:warp_decode:647 - warp time: 13.475s |
|
|
2025-05-30 09:56:31.118 | INFO | __main__:main:881 - Animated image: ./axmodel_infer/s0--d8.jpg |
|
|
2025-05-30 09:56:31.118 | INFO | __main__:main:882 - Animated image with concat: ./axmodel_infer/s0--d8_concat.jpg |
|
|
2025-05-30 09:56:31.167 | DEBUG | __main__:<module>:894 - LivePortrait axmodel infer time: 32.455s |
|
|
``` |
|
|
|
|
|
## Inference with M.2 Accelerator card |
|
|
|
|
|
[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on x86. |
|
|
|
|
|
### Image |
|
|
|
|
|
``` |
|
|
(py310) axera@dell:~/samples/LivePortrait$ python ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d8.jpg --models ./python/axmodels/ --output-dir ./axmodel_infer |
|
|
[INFO] Available providers: ['AXCLRTExecutionProvider'] |
|
|
[INFO] Using provider: AXCLRTExecutionProvider |
|
|
[INFO] SOC Name: AX650N |
|
|
[INFO] VNPU type: VNPUType.DISABLED |
|
|
[INFO] Compiler version: 3.3 144960ad |
|
|
[INFO] Using provider: AXCLRTExecutionProvider |
|
|
[INFO] SOC Name: AX650N |
|
|
[INFO] VNPU type: VNPUType.DISABLED |
|
|
[INFO] Compiler version: 3.3 144960ad |
|
|
[INFO] Using provider: AXCLRTExecutionProvider |
|
|
[INFO] SOC Name: AX650N |
|
|
[INFO] VNPU type: VNPUType.DISABLED |
|
|
[INFO] Compiler version: 3.3 0f7260e8 |
|
|
[INFO] Using provider: AXCLRTExecutionProvider |
|
|
[INFO] SOC Name: AX650N |
|
|
[INFO] VNPU type: VNPUType.DISABLED |
|
|
[INFO] Compiler version: 3.3 144960ad |
|
|
FaceAnalysisDIY warmup time: 0.024s |
|
|
[20:02:20] LandmarkRunner warmup time: 0.031s human_landmark_runner.py:95 |
|
|
2025-05-29 20:02:20.727 | INFO | __main__:main:727 - Start making driving motion template... |
|
|
2025-05-29 20:02:20.972 | INFO | __main__:main:747 - Prepared pasteback mask done. |
|
|
2025-05-29 20:02:21.449 | INFO | __main__:main:787 - The output of image-driven portrait animation is an image. |
|
|
2025-05-29 20:02:25.475 | DEBUG | __main__:warp_decode:647 - warp time: 4.017s |
|
|
2025-05-29 20:02:25.892 | INFO | __main__:main:881 - Animated image: ./axmodel_infer/s0--d8.jpg |
|
|
2025-05-29 20:02:25.892 | INFO | __main__:main:882 - Animated image with concat: ./axmodel_infer/s0--d8_concat.jpg |
|
|
2025-05-29 20:02:25.904 | DEBUG | __main__:<module>:894 - LivePortrait axmodel infer time: 8.165s |
|
|
(py310) axera@dell:~/samples/LivePortrait$ |
|
|
``` |
|
|
|
|
|
Here, `--models` specifies the storage path for the `*.axmodel model`. |
|
|
|
|
|
The output of axmodel-infer is as follows: |
|
|
|
|
|
 |
|
|
 |
|
|
|
|
|
### Video |
|
|
|
|
|
``` |
|
|
python3 ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d0.mp4 --models ./python/axmodels/ --output-dir ./axmodel_infer |
|
|
``` |
|
|
|
|
|
The output of `axmodel-infer` is as follows: |
|
|
|
|
|
 |
|
|
 |
|
|
|
