Image-to-Video
Diffusers
AlayaWorldPipeline
alayaworld
video-generation
world-model
distillation
lora
How to use from the
Use from the
Diffusers library
pip install -U diffusers transformers accelerate
import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image, export_to_video

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("fill-in-base-model", dtype=torch.bfloat16, device_map="cuda")
pipe.load_lora_weights("AlayaLab/AlayaWorld-v1.1-stage3")

prompt = "A man with short gray hair plays a red electric guitar."
input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png")

output = pipe(image=input_image, prompt=prompt).frames[0]
export_to_video(output, "output.mp4")

AlayaWorld v1.1 β€” Stage3 Few-Step Student (DMD LoRA)

4-step DMD-distilled student of AlayaWorld (real-time inference). LoRA on top of the stage2b AR teacher; LTX-2 Community License.

File Size Content
lora.safetensors 2.5 GB generator LoRA (rank 256)
history_encoder.pt 33 MB compressed frame-history encoder

Download β€” the student also needs the stage2b teacher as its base:

hf download AlayaLab/AlayaWorld-v1.1-stage2b --local-dir weights/alaya-world-ar    # AR teacher
hf download AlayaLab/AlayaWorld-v1.1-stage3 --local-dir weights/alaya-world-dmd    # this repo

Usage β€” with AlayaLab/AlayaWorld:

paths:
  resume_checkpoint: <stage2b download dir>            # teacher base
  dmd_resume:        <this repo download dir>          # lora.safetensors
  history_encoder:   <this repo download dir>/history_encoder.pt

Run: VALIDATE_ONLY=1 CONFIG_PATH=configs/infer_i2v_camera.yaml bash scripts/finetune/train.sh

Reports: intro Β· full Β· v1.1

Downloads last month
306
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Papers for AlayaLab/AlayaWorld-v1.1-stage3