Text-to-Video
VideoX Fun
MiniMax-H3-Acc-LoRAs / predict_ref2v.py
hkunzhe's picture
Upload MiniMax-H3 acceleration LoRAs and examples
d41cfc8 verified
Raw
History Blame Contribute Delete
7.7 kB
import sys
from pathlib import Path
import torch
from diffusers import ComponentsManager, ModularPipeline
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3ImageReference
from diffusers.utils.export_utils import encode_video
sys.path.insert(0, str(Path(__file__).resolve().parent))
from minimax_h3_pdd import apply_pdd_lora
here = Path(__file__).resolve().parent
model_path = "MiniMaxAI/MiniMax-H3"
pdd_lora_path = str(here / "MiniMax-H3-Ref2VA-Acc-8Step.safetensors")
reference_images = [here / "asset" / f"ref2va_test_5_{i}.jpg" for i in range(1, 5)]
prompt = "subject_definitions:\n<Subject 1> is the East Asian woman sword fighter shown in <Picture 1>, preserving her facial identity, black hair tied in a high bun, focused expression, cream martial-arts jacket, and black-handled katana.\n<Subject 2> is the pale-skinned woman fighter shown in <Picture 2>, preserving her facial identity, short layered black hair with bangs, dark eye makeup, black lipstick, black leather jacket, and black clothing. The motorcycle is not present in the target video.\n<Subject 3> is the outdoor competition swimming pool shown in <Picture 3>, preserving the bright turquoise water, blue-and-white lane dividers, pale pool deck, and strong daylight.\n<Subject 4> is the underwater pool environment shown in <Picture 4>, preserving the cyan water, tiled pool floor, dark lane marking, moving caustic light, suspended bubbles, and refracted surface light.\n\nsummary:\n[reference generation] In one continuous 10-second action shot, <Subject 1> and <Subject 2> sprint into a violent exchange beside <Subject 3>. A rolling dodge leads into a left-fist versus right-hand-sword collision, a jumping counter, and a connected two-body tumble across the wet deck. Their momentum throws both women into the pool, where the confrontation finishes underwater inside <Subject 4>.\n\nretention_analysis:\n<Subject 1> (appears throughout): fully_preserved - retain the same face, high bun, cream jacket, body proportions, and single black-handled katana before and after entering the water.\n<Subject 2> (appears throughout): fully_preserved - retain the same face, short black hair, dark makeup, black lipstick, black leather jacket, black clothing, and body proportions before and after entering the water.\n<Subject 3> (appears before the underwater transition): fully_preserved - retain the competition-pool geometry, turquoise water, blue-and-white lane ropes, sunlit deck, and bright daylight.\n<Subject 4> (appears after the underwater transition): fully_preserved - retain the tiled floor, dark lane marking, cyan water, caustic illumination, bubbles, and refracted sunlight.\n\ndetailed_description:\nPhotorealistic live-action action cinema, 16:9 composition, hard bright daylight above water and luminous cyan light below water. Use one continuous aggressively tracking shot with no cuts: low lateral sprint tracking, one fast camera roll during the deck tumble, then a seamless plunge through the waterline. Keep both adult women clearly distinguishable by face, hairstyle, clothing color, and silhouette. <Subject 1> holds exactly one katana in her right hand throughout. <Subject 2> remains unarmed. Any fist-to-sword contact strikes the flat near the guard, never the sharpened edge. No blood, wounds, severed contact, duplicate people, duplicate weapons, motorcycle, readable text, logos, or costume changes.\n\n[Shot 1, 00:00.000-00:02.000] A low handheld camera races parallel to the wet edge of <Subject 3>. <Subject 1> in the cream jacket and <Subject 2> in black leather sprint toward each other from opposite ends of the pool deck. <Subject 1> carries the single katana low in her right hand, blade trailing safely behind. She plants one foot on a low starting block, leaps forward, and brings the sword into a fast diagonal arc. <Subject 2> drops beneath it into a forward shoulder roll, slides across the wet tiles, and rises inside the sword's range as water sprays from both fighters' shoes.\n\n[Shot 1, 00:02.000-00:04.200] <Subject 2> drives upward from the roll with one explosive left punch. <Subject 1> snaps the katana across in her right hand so the punch collides with the flat of the blade immediately beside the guard. The impact produces one sharp metallic crack, a compact spray of sparks, and a ring of displaced droplets. Their faces pass within centimeters. <Subject 2> hooks the sword wrist with her right hand and whips a low leg sweep; <Subject 1> jumps over it, twists in the air, and lands one step from the pool edge without releasing the sword.\n\n[Shot 1, 00:04.200-00:06.800] <Subject 2> immediately tackles <Subject 1> around the waist. <Subject 1> braces her left forearm across <Subject 2>'s shoulder and extends her right sword arm away from both bodies. Locked together, they crash onto the wet deck and complete two fast connected shoulder-over-shoulder rolls. The camera rolls with them at tile level as sky, lane ropes, faces, black leather, and cream fabric whip across frame. <Subject 1> hooks <Subject 2>'s ankle during the second rotation; their combined momentum carries both bodies over the pool lip.\n\n[Shot 1, 00:06.800-00:07.900] Both women rotate once in open air and strike the turquoise surface together in an enormous splash. The camera plunges between them without a cut, crossing the violently folding waterline as daylight becomes refracted cyan and a dense wall of bubbles engulfs the frame.\n\n[Shot 1, 00:07.900-00:10.000] Inside <Subject 4>, their clothes and hair surge in opposite currents while both identities remain clear. <Subject 2> throws one slowed left punch through the bubbles. <Subject 1>, still gripping the single katana in her right hand, blocks with the hilt and her left forearm while keeping the blade pointed safely away. The impact rolls both women apart above the dark lane marking. They kick backward, regain facing guard positions, and end suspended on opposite sides of a rising column of bubbles as caustic light flashes across their faces and the tiled floor.\n\noverall_soundscape:\nRapid wet footfalls, starting-block impact, sword whoosh, rolling bodies, one metallic fist-to-blade crack, sparks, tile scrapes, a heavy synchronized splash, dense bubbles, muffled underwater impact, and filtered pool ambience.\n\nnon_diegetic_music:\nFast taiko-like percussion accelerates through the sprint and deck rolls, cuts completely at the splash, then returns underwater as two muffled bass pulses ending on one suspended metallic note.\n"
sample_size = [768, 1344]
video_length = 243
manager = ComponentsManager()
manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="12GB")
pipeline = ModularPipeline.from_pretrained(model_path, workflow="ref2va", components_manager=manager)
pipeline.load_components(dtype=torch.bfloat16, pretrained_model_name_or_path=model_path)
nfe = apply_pdd_lora(
pipeline.transformer_ref, pdd_lora_path, pipeline.scheduler.shift, pipeline.audio_scheduler.shift
)
generator = torch.Generator().manual_seed(42)
result = pipeline(
prompt=prompt,
references=[MiniMaxH3ImageReference.from_file(str(path)) for path in reference_images],
height=sample_size[0],
width=sample_size[1],
num_frames=video_length,
num_inference_steps=nfe + 1,
generator=generator,
output_type="np",
output=["videos", "audio", "sampling_rate"],
)
output_path = here / "samples" / "ref2v.mp4"
output_path.parent.mkdir(parents=True, exist_ok=True)
encode_video(
result["videos"][0],
fps=24,
output_path=str(output_path),
audio=result["audio"][0],
audio_sample_rate=int(result["sampling_rate"]),
)
print(f"Saved {output_path}", flush=True)