svjack/Joseon_Taizong_HunyuanVideo_lora

Joseon_Taizong (from 大王世宗 대왕세종 ) HunyuanVideo LoRA

This repository contains the necessary setup and scripts to generate videos using the HunyuanVideo model with a LoRA (Low-Rank Adaptation) fine-tuned for Joseon_Taizong. Below are the instructions to install dependencies, download models, and run the demo.

---

Installation

Step 1: Install System Dependencies

Run the following command to install required system packages:

sudo apt-get update && sudo apt-get install git-lfs ffmpeg cbm

Step 2: Clone the Repository

Clone the repository and navigate to the project directory:

git clone https://huggingface.co/svjack/Joseon_Taizong_HunyuanVideo_lora
cd Joseon_Taizong_HunyuanVideo_lora

Step 3: Install Python Dependencies

Install the required Python packages:

conda create -n py310 python=3.10
conda activate py310
pip install ipykernel
python -m ipykernel install --user --name py310 --display-name "py310"

pip install -r requirements.txt
pip install ascii-magic matplotlib tensorboard huggingface_hub
pip install moviepy==1.0.3
pip install sageattention==1.0.6

pip install torch==2.5.0 torchvision

---

Download Models

Step 1: Download HunyuanVideo Model

Download the HunyuanVideo model and place it in the ckpts directory:

huggingface-cli download tencent/HunyuanVideo --local-dir ./ckpts

Step 2: Download LLaVA Model

Download the LLaVA model and preprocess it:

cd ckpts
huggingface-cli download xtuner/llava-llama-3-8b-v1_1-transformers --local-dir ./llava-llama-3-8b-v1_1-transformers
wget https://raw.githubusercontent.com/Tencent/HunyuanVideo/refs/heads/main/hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py
python preprocess_text_encoder_tokenizer_utils.py --input_dir llava-llama-3-8b-v1_1-transformers --output_dir text_encoder

Step 3: Download CLIP Model

Download the CLIP model for the text encoder:

huggingface-cli download openai/clip-vit-large-patch14 --local-dir ./text_encoder_2

---

Demo

Generate Video 1: Joseon_Taizong bow

Run the following command to generate a video of Joseon_Taizong:

python hv_generate_video.py \
    --fp8 \
    --video_size 544 960 \
    --video_length 60 \
    --infer_steps 30 \
    --prompt "Joseon Taizong is captured mid-action, drawing a bow with intense focus. He wears a traditional red Korean hanbok adorned with intricate dragon designs, and a black hat with a white circular emblem. The background features a colorful tent with green and red patterns, and the sunlight filters through, casting a warm glow. Two other figures, one partially visible, are dressed in similar traditional attire." \
    --save_path . \
    --output_type both \
    --dit ckpts/hunyuan-video-t2v-720p/transformers/mp_rank_00_model_states.pt \
    --attn_mode sdpa \
    --vae ckpts/hunyuan-video-t2v-720p/vae/pytorch_model.pt \
    --vae_chunk_size 32 \
    --vae_spatial_tile_sample_min_size 128 \
    --text_encoder1 ckpts/text_encoder \
    --text_encoder2 ckpts/text_encoder_2 \
    --seed 1234 \
    --lora_multiplier 1.0 \
    --lora_weight Taizong_im_lora_dir/Taizong_im_lora-000012.safetensors

Generate Video 2: Joseon_Taizong talk

Run the following command to generate a video of Joseon_Taizong:

python hv_generate_video.py \
    --fp8 \
    --video_size 544 960 \
    --video_length 60 \
    --infer_steps 30 \
    --prompt "Joseon Taizong stands with commanding presence. He wears a resplendent golden dragon robe, a symbol of imperial authority, paired with a black hat featuring a white circular emblem. The background showcases a colorful tent with green and red patterns, as sunlight filters through, casting a warm glow." \
    --save_path . \
    --output_type both \
    --dit ckpts/hunyuan-video-t2v-720p/transformers/mp_rank_00_model_states.pt \
    --attn_mode sdpa \
    --vae ckpts/hunyuan-video-t2v-720p/vae/pytorch_model.pt \
    --vae_chunk_size 32 \
    --vae_spatial_tile_sample_min_size 128 \
    --text_encoder1 ckpts/text_encoder \
    --text_encoder2 ckpts/text_encoder_2 \
    --seed 1234 \
    --lora_multiplier 1.0 \
    --lora_weight Taizong_im_lora_dir/Taizong_im_lora-000016.safetensors

---

Notes

• Ensure you have sufficient GPU resources for video generation.
• Adjust the --video_size, --video_length, and --infer_steps parameters as needed for different output qualities and lengths.
• The --prompt parameter can be modified to generate videos with different scenes or actions.

---