AGENT.md

AGENT.md

This file provides guidance to AI coding agents when working with code in this repository.

Project Overview

This is a Gradio Space that implements "Next Scene" cinematic image generation using Qwen-Image-Edit-2509 with LoRA fine-tuning. The application generates visually progressive image sequences with natural cinematic transitions from frame to frame, optimized for fast 4-step inference.

Key Model Components:

• Base model: Qwen/Qwen-Image-Edit-2509 (image editing diffusion model)
• Accelerated transformer: linoyts/Qwen-Image-Edit-Rapid-AIO (4-step optimized variant)
• LoRA adapter: lovis93/next-scene-qwen-image-lora-2509 (cinematic progression fine-tune)
• Text encoder: Qwen2.5-VL-72B-Instruct (via Hugging Face InferenceClient for prompt enhancement)

Running the Application

Start the Gradio interface:

python app.py

Install dependencies:

pip install -r requirements.txt

The app requires GPU access. It uses the @spaces.GPU decorator for Hugging Face Spaces zero-GPU allocation.

Architecture

Pipeline Flow

1. Input Processing (app.py:infer):

   • Accepts input images via Gradio Gallery (filepath-based)
   • Optional prompt rewriting using Qwen2.5-VL-72B-Instruct API
   • Automatic "Next Scene" prompt generation from images

2. Image Generation (qwenimage/pipeline_qwenimage_edit_plus.py):

   • Custom pipeline extending DiffusionPipeline
   • Encodes images using VAE at 1024x1024 for latents
   • Encodes conditioning images at 384x384 for text encoder
   • Packs latents into 2x2 patches (latent dims must be divisible by 2)
   • Uses FlowMatchEulerDiscreteScheduler for denoising

3. Optimization (optimization.py):

   • Ahead-of-time (AOT) compilation using torch.export and spaces.aoti_compile
   • Dynamic shapes for variable sequence lengths
   • Custom inductor configs for performance (max_autotune, cudagraphs)
   • FlashAttention 3 integration via QwenDoubleStreamAttnProcessorFA3

4. Output Handling:

   • Saves outputs to outputs/ directory with unique timestamps
   • Maintains 20-image history gallery
   • Optional video generation via multimodalart/wan-2-2-first-last-frame Space

Custom QwenImage Components

Location: qwenimage/ package

• pipeline_qwenimage_edit_plus.py - Main diffusion pipeline with LoRA support
• transformer_qwenimage.py - Custom transformer model with cache management
• qwen_fa3_processor.py - FlashAttention 3 attention processor

Key architectural features:

• Latent packing/unpacking for 2x2 patch processing
• Multi-image conditioning support
• True CFG (classifier-free guidance) with separate pos/neg paths
• Dual-stream attention with rotary embeddings
• Cache contexts for conditional/unconditional forward passes

Prompt Engineering

Two-stage prompt system:

1. Edit Instruction Rewriter (SYSTEM_PROMPT):

   • Normalizes user prompts into professional editing instructions
   • Handles text replacement (requires quotes), object manipulation, style transfer
   • Used when rewrite_prompt=True checkbox is enabled

2. Next Scene Generator (NEXT_SCENE_SYSTEM_PROMPT):

   • Automatically suggests cinematic camera movements
   • Focus on visual progression (dolly, pan, zoom, framing changes)
   • Auto-triggers when input images change

Both use Qwen2.5-VL-72B-Instruct via Hugging Face InferenceClient with Nebius provider. Requires HF_TOKEN environment variable.

Important Implementation Details

Image Dimension Handling

Images are automatically resized based on calculate_dimensions() function:

• VAE images: resized to maintain 1024×1024 area (1,048,576 pixels)
• Condition images: resized to maintain 384×384 area (147,456 pixels)
• Output dimensions must be divisible by 16 (vae_scale_factor × 2)
• Height/width default to None which auto-calculates from input aspect ratio

LoRA Integration

The pipeline fuses the "next-scene" LoRA adapter at initialization:

pipe.load_lora_weights("lovis93/next-scene-qwen-image-lora-2509", ...)
pipe.set_adapters(["next-scene"], adapter_weights=[1.])
pipe.fuse_lora(adapter_names=["next-scene"], lora_scale=1.)
pipe.unload_lora_weights()

After fusion, the adapter weights are merged into the base model and cannot be unfused.

Video Generation Integration

The turn_into_video() function:

• Connects to external Gradio Space multimodalart/wan-2-2-first-last-frame
• Requires first input image and last output image
• Uses the original prompt (or "smooth cinematic transition" fallback)
• Returns video path for display

Gradio Gallery Format

Input/output galleries use type="filepath" (string paths) rather than PIL Image tuples. Helper functions handle format compatibility for legacy tuple support.

Environment Variables

• HF_TOKEN - Required for Qwen2.5-VL API access (prompt rewriting/generation)

File Outputs

Generated images are saved to outputs/ directory with format:

output_{seed}_{index}_{timestamp_ms}.png

Local Development and API Testing

The custom/ directory is fully gitignored and used for local development files. Specifically, it contains:

• API client scripts - For testing the Gradio Space remotely via API after deployment to Hugging Face
• API_GUIDE.txt - Auto-generated Gradio API documentation showing endpoint signatures and example usage
• Local testing environments - Virtual environments or test data that shouldn't be committed

API Integration Pattern: Once the Space is deployed to Hugging Face, you can interact with it programmatically using gradio_client:

from gradio_client import Client, handle_file

client = Client("Sneak-Moose/Qwen-Image-Edit-next-scene")
result = client.predict(
    images=[],
    prompt="Next Scene: Camera dollies forward...",
    seed=42,
    randomize_seed=False,
    true_guidance_scale=1.0,
    num_inference_steps=4,
    height=1024,
    width=1024,
    rewrite_prompt=False,
    api_name="/infer"
)

The custom/API_GUIDE.txt contains full documentation of all available endpoints including /infer, /turn_into_video, /suggest_next_scene_prompt, and utility functions.

Development Notes

• The model loads on startup and applies AOT compilation during first inference
• Compilation uses dynamic shapes to support variable text/image sequence lengths
• The transformer uses custom cache contexts ("cond"/"uncond") to optimize CFG passes
• True CFG applies norm-based rescaling: comb_pred * (cond_norm / noise_norm)
• FlashAttention 3 processor must be set before compilation