CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is a Gradio Space that implements "Next Scene" cinematic image generation using Qwen-Image-Edit-2511 with LoRA fine-tuning. The application generates visually progressive image sequences with natural cinematic transitions from frame to frame, optimized for fast 4-step inference.

Key Model Components:

• Base model: Qwen/Qwen-Image-Edit-2511 (image editing diffusion model)
• Accelerated transformer: Sneak-Moose/Qwen-Rapid-AIO-v18-NSFW-diffusers (extracted from Phr00t's v18, 4-step optimized)
• LoRA adapter: lovis93/next-scene-qwen-image-lora-2509 (cinematic progression fine-tune, trained on 2509)

Running the Application

Start the Gradio interface:

python app.py

Install dependencies:

pip install -r requirements.txt

The app requires GPU access. It uses the @spaces.GPU decorator for Hugging Face Spaces zero-GPU allocation.

Architecture

Pipeline Flow

1. Input Processing (app.py:infer):

   • Accepts input images via Gradio Gallery (filepath-based)
   • Uses user-provided prompts directly without modification

2. Image Generation (qwenimage/pipeline_qwenimage_edit_plus.py):

   • Custom pipeline extending DiffusionPipeline
   • Encodes images using VAE at 1024x1024 for latents
   • Encodes conditioning images at 384x384 for text encoder
   • Packs latents into 2x2 patches (latent dims must be divisible by 2)
   • Uses FlowMatchEulerDiscreteScheduler for denoising

3. Optimization (optimization.py):

   • Ahead-of-time (AOT) compilation using torch.export and spaces.aoti_compile
   • Dynamic shapes for variable sequence lengths
   • Custom inductor configs for performance (max_autotune, cudagraphs)
   • FlashAttention 3 integration via QwenDoubleStreamAttnProcessorFA3

4. Output Handling:

   • Saves outputs to outputs/ directory with unique timestamps
   • Maintains 20-image history gallery
   • Optional video generation via multimodalart/wan-2-2-first-last-frame Space

Custom QwenImage Components

Location: qwenimage/ package

• pipeline_qwenimage_edit_plus.py - Main diffusion pipeline with LoRA support
• transformer_qwenimage.py - Custom transformer model with cache management
• qwen_fa3_processor.py - FlashAttention 3 attention processor

Key architectural features:

• Latent packing/unpacking for 2x2 patch processing
• Multi-image conditioning support
• True CFG (classifier-free guidance) with separate pos/neg paths
• Dual-stream attention with rotary embeddings
• Cache contexts for conditional/unconditional forward passes

Prompt Handling

The application uses user-provided prompts directly without any preprocessing, rewriting, or AI-based enhancement. Users have full control over the exact prompt text that gets passed to the diffusion model.

Important Implementation Details

Image Dimension Handling

Images are automatically resized based on calculate_dimensions() function:

• VAE images: resized to maintain 1024×1024 area (1,048,576 pixels)
• Condition images: resized to maintain 384×384 area (147,456 pixels)
• Output dimensions must be divisible by 16 (vae_scale_factor × 2)
• Height/width default to None which auto-calculates from input aspect ratio

LoRA Integration

The pipeline fuses the "next-scene" LoRA adapter at initialization:

pipe.load_lora_weights("lovis93/next-scene-qwen-image-lora-2509", ...)
pipe.set_adapters(["next-scene"], adapter_weights=[1.])
pipe.fuse_lora(adapter_names=["next-scene"], lora_scale=1.)
pipe.unload_lora_weights()

After fusion, the adapter weights are merged into the base model and cannot be unfused.

Video Generation Integration

The turn_into_video() function:

• Connects to external Gradio Space multimodalart/wan-2-2-first-last-frame
• Requires first input image and last output image
• Uses the original prompt (or "smooth cinematic transition" fallback)
• Returns video path for display

Gradio Gallery Format

Input/output galleries use type="filepath" (string paths) rather than PIL Image tuples. Helper functions handle format compatibility for legacy tuple support.

Environment Variables

No environment variables are required for basic operation. The application runs entirely with local models.

File Outputs

Generated images are saved to outputs/ directory with format:

output_{seed}_{index}_{timestamp_ms}.png

Local Development and API Testing

The custom/ directory is fully gitignored and used for local development files. Specifically, it contains:

• API client scripts - For testing the Gradio Space remotely via API after deployment to Hugging Face
• API_GUIDE.txt - Auto-generated Gradio API documentation showing endpoint signatures and example usage
• Local testing environments - Virtual environments or test data that shouldn't be committed

API Integration Pattern: Once the Space is deployed to Hugging Face, you can interact with it programmatically using gradio_client:

from gradio_client import Client, handle_file

client = Client("Sneak-Moose/Qwen-Image-Edit-next-scene")
result = client.predict(
    images=[],
    prompt="Camera dollies forward, revealing more of the scene",
    seed=42,
    randomize_seed=False,
    true_guidance_scale=1.0,
    num_inference_steps=4,
    height=1024,
    width=1024,
    api_name="/infer"
)

The custom/API_GUIDE.txt contains full documentation of all available endpoints including /infer, /turn_into_video, and utility functions.

Development Notes

• The model loads on startup and applies AOT compilation during first inference
• Compilation uses dynamic shapes to support variable text/image sequence lengths
• The transformer uses custom cache contexts ("cond"/"uncond") to optimize CFG passes
• True CFG applies norm-based rescaling: comb_pred * (cond_norm / noise_norm)
• FlashAttention 3 processor must be set before compilation