CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is a Gradio Space that implements "Next Scene" cinematic image generation using Qwen-Image-Edit-2511 with LoRA fine-tuning. The application generates visually progressive image sequences with natural cinematic transitions from frame to frame, optimized for fast 4-step inference.

**Key Model Components:**
- Base model: `Qwen/Qwen-Image-Edit-2511` (image editing diffusion model)
- Accelerated transformer: `Sneak-Moose/Qwen-Rapid-AIO-v18-NSFW-diffusers` (extracted from Phr00t's v18, 4-step optimized)
- LoRA adapter: `lovis93/next-scene-qwen-image-lora-2509` (cinematic progression fine-tune, trained on 2509)

## Running the Application

**Start the Gradio interface:**
```bash
python app.py
```

**Install dependencies:**
```bash
pip install -r requirements.txt
```

The app requires GPU access. It uses the `@spaces.GPU` decorator for Hugging Face Spaces zero-GPU allocation.

## Architecture

### Pipeline Flow

1. **Input Processing** (`app.py:infer`):
   - Accepts input images via Gradio Gallery (filepath-based)
   - Uses user-provided prompts directly without modification

2. **Image Generation** (`qwenimage/pipeline_qwenimage_edit_plus.py`):
   - Custom pipeline extending `DiffusionPipeline`
   - Encodes images using VAE at 1024x1024 for latents
   - Encodes conditioning images at 384x384 for text encoder
   - Packs latents into 2x2 patches (latent dims must be divisible by 2)
   - Uses `FlowMatchEulerDiscreteScheduler` for denoising

3. **Optimization** (`optimization.py`):
   - Ahead-of-time (AOT) compilation using `torch.export` and `spaces.aoti_compile`
   - Dynamic shapes for variable sequence lengths
   - Custom inductor configs for performance (max_autotune, cudagraphs)
   - FlashAttention 3 integration via `QwenDoubleStreamAttnProcessorFA3`

4. **Output Handling**:
   - Saves outputs to `outputs/` directory with unique timestamps
   - Maintains 20-image history gallery
   - Optional video generation via `multimodalart/wan-2-2-first-last-frame` Space

### Custom QwenImage Components

**Location:** `qwenimage/` package

- `pipeline_qwenimage_edit_plus.py` - Main diffusion pipeline with LoRA support
- `transformer_qwenimage.py` - Custom transformer model with cache management
- `qwen_fa3_processor.py` - FlashAttention 3 attention processor

**Key architectural features:**
- Latent packing/unpacking for 2x2 patch processing
- Multi-image conditioning support
- True CFG (classifier-free guidance) with separate pos/neg paths
- Dual-stream attention with rotary embeddings
- Cache contexts for conditional/unconditional forward passes

### Prompt Handling

The application uses user-provided prompts directly without any preprocessing, rewriting, or AI-based enhancement. Users have full control over the exact prompt text that gets passed to the diffusion model.

## Important Implementation Details

### Image Dimension Handling

Images are automatically resized based on `calculate_dimensions()` function:
- VAE images: resized to maintain 1024×1024 area (1,048,576 pixels)
- Condition images: resized to maintain 384×384 area (147,456 pixels)
- Output dimensions must be divisible by 16 (vae_scale_factor × 2)
- Height/width default to `None` which auto-calculates from input aspect ratio

### LoRA Integration

The pipeline fuses the "next-scene" LoRA adapter at initialization:
```python
pipe.load_lora_weights("lovis93/next-scene-qwen-image-lora-2509", ...)
pipe.set_adapters(["next-scene"], adapter_weights=[1.])
pipe.fuse_lora(adapter_names=["next-scene"], lora_scale=1.)
pipe.unload_lora_weights()
```

After fusion, the adapter weights are merged into the base model and cannot be unfused.

### Video Generation Integration

The `turn_into_video()` function:
- Connects to external Gradio Space `multimodalart/wan-2-2-first-last-frame`
- Requires first input image and last output image
- Uses the original prompt (or "smooth cinematic transition" fallback)
- Returns video path for display

### Gradio Gallery Format

Input/output galleries use `type="filepath"` (string paths) rather than PIL Image tuples. Helper functions handle format compatibility for legacy tuple support.

## Environment Variables

No environment variables are required for basic operation. The application runs entirely with local models.

## File Outputs

Generated images are saved to `outputs/` directory with format:
```
output_{seed}_{index}_{timestamp_ms}.png
```

## Local Development and API Testing

The `custom/` directory is fully gitignored and used for local development files. Specifically, it contains:

- **API client scripts** - For testing the Gradio Space remotely via API after deployment to Hugging Face
- **`API_GUIDE.txt`** - Auto-generated Gradio API documentation showing endpoint signatures and example usage
- **Local testing environments** - Virtual environments or test data that shouldn't be committed

**API Integration Pattern:**
Once the Space is deployed to Hugging Face, you can interact with it programmatically using `gradio_client`:

```python
from gradio_client import Client, handle_file

client = Client("Sneak-Moose/Qwen-Image-Edit-next-scene")
result = client.predict(
    images=[],
    prompt="Camera dollies forward, revealing more of the scene",
    seed=42,
    randomize_seed=False,
    true_guidance_scale=1.0,
    num_inference_steps=4,
    height=1024,
    width=1024,
    api_name="/infer"
)
```

The `custom/API_GUIDE.txt` contains full documentation of all available endpoints including `/infer`, `/turn_into_video`, and utility functions.

## Development Notes

- The model loads on startup and applies AOT compilation during first inference
- Compilation uses dynamic shapes to support variable text/image sequence lengths
- The transformer uses custom cache contexts ("cond"/"uncond") to optimize CFG passes
- True CFG applies norm-based rescaling: `comb_pred * (cond_norm / noise_norm)`
- FlashAttention 3 processor must be set before compilation