README.md

metadata

license: mit
language:
  - en
base_model:
  - KwaiVGI/LivePortrait
pipeline_tag: image-to-video

LivePortrait

This version of LivePortrait has been converted to run on the Axera NPU using w8a16 quantization.

This model has been optimized with the following:

Compatible with Pulsar2 version: 3.4

Convert tools links:

For those who are interested in model conversion:

• the original repo
• Github for LivePortrait

Support Platform

• AX650/AX8850
  • M4N-Dock(爱芯派Pro)
  • M.2 Accelerator card

How to use

Download all files from this repository to the device.

(py310) axera@dell:~/samples/LivePortrait$ tree -L 2
.
├── assets
│   └── examples
├── config.json
├── python
│   ├── axmodels
│   ├── cropper.py
│   ├── infer_onnx.py
│   ├── infer.py
│   ├── pretrained_weights
│   ├── requirements.txt
│   └── utils
└── README.md

7 directories, 6 files

python env requirement

pyaxengine

https://github.com/AXERA-TECH/pyaxengine

wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.3.rc1/axengine-0.1.3-py3-none-any.whl
pip install axengine-0.1.3-py3-none-any.whl

others

pip install -r python/requirements.txt

Inference with AX650 or AX8850 Host, such as AX650 DEMO BOARD, M4N-DOCK(爱芯派Pro)

root@ax650 ~/yongqiang/LivePortrait.axera # python3 ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d8.jpg --models ./python/axmodels/ --output-dir ./axmodel_infer
[INFO] Available providers:  ['AxEngineExecutionProvider']
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.12.0s
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 3.3 144960ad
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 3.3 144960ad
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 3.3 0f7260e8
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 3.3 144960ad
FaceAnalysisDIY warmup time: 0.598s
LandmarkRunner warmup time: 0.769s
2025-05-30 09:56:12.247 | INFO     | __main__:main:727 - Start making driving motion template...
2025-05-30 09:56:14.770 | INFO     | __main__:main:747 - Prepared pasteback mask done.
2025-05-30 09:56:17.219 | INFO     | __main__:main:787 - The output of image-driven portrait animation is an image.
2025-05-30 09:56:30.701 | DEBUG    | __main__:warp_decode:647 - warp time: 13.475s
2025-05-30 09:56:31.118 | INFO     | __main__:main:881 - Animated image: ./axmodel_infer/s0--d8.jpg
2025-05-30 09:56:31.118 | INFO     | __main__:main:882 - Animated image with concat: ./axmodel_infer/s0--d8_concat.jpg
2025-05-30 09:56:31.167 | DEBUG    | __main__:<module>:894 - LivePortrait axmodel infer time: 32.455s

Inference with M.2 Accelerator card

What is M.2 Accelerator card?, Show this DEMO based on x86.

Image

(py310) axera@dell:~/samples/LivePortrait$ python ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d8.jpg --models ./python/axmodels/ --output-dir ./axmodel_infer
[INFO] Available providers:  ['AXCLRTExecutionProvider']
[INFO] Using provider: AXCLRTExecutionProvider
[INFO] SOC Name: AX650N
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Compiler version: 3.3 144960ad
[INFO] Using provider: AXCLRTExecutionProvider
[INFO] SOC Name: AX650N
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Compiler version: 3.3 144960ad
[INFO] Using provider: AXCLRTExecutionProvider
[INFO] SOC Name: AX650N
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Compiler version: 3.3 0f7260e8
[INFO] Using provider: AXCLRTExecutionProvider
[INFO] SOC Name: AX650N
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Compiler version: 3.3 144960ad
FaceAnalysisDIY warmup time: 0.024s
[20:02:20] LandmarkRunner warmup time: 0.031s    human_landmark_runner.py:95
2025-05-29 20:02:20.727 | INFO     | __main__:main:727 - Start making driving motion template...
2025-05-29 20:02:20.972 | INFO     | __main__:main:747 - Prepared pasteback mask done.
2025-05-29 20:02:21.449 | INFO     | __main__:main:787 - The output of image-driven portrait animation is an image.
2025-05-29 20:02:25.475 | DEBUG    | __main__:warp_decode:647 - warp time: 4.017s
2025-05-29 20:02:25.892 | INFO     | __main__:main:881 - Animated image: ./axmodel_infer/s0--d8.jpg
2025-05-29 20:02:25.892 | INFO     | __main__:main:882 - Animated image with concat: ./axmodel_infer/s0--d8_concat.jpg
2025-05-29 20:02:25.904 | DEBUG    | __main__:<module>:894 - LivePortrait axmodel infer time: 8.165s
(py310) axera@dell:~/samples/LivePortrait$

Here, --models specifies the storage path for the *.axmodel model.

The output of axmodel-infer is as follows:

Video

python3 ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d0.mp4 --models ./python/axmodels/ --output-dir ./axmodel_infer

The output of axmodel-infer is as follows: