AGENT.md

# AGENT.md

This file provides guidance to AI coding agents when working with code in this repository.

## Project Overview

This is a Gradio Space that implements "Next Scene" cinematic image generation using Qwen-Image-Edit-2509 with LoRA fine-tuning. The application generates visually progressive image sequences with natural cinematic transitions from frame to frame, optimized for fast 4-step inference.

**Key Model Components:**
- Base model: `Qwen/Qwen-Image-Edit-2509` (image editing diffusion model)
- Accelerated transformer: `linoyts/Qwen-Image-Edit-Rapid-AIO` (4-step optimized variant)
- LoRA adapter: `lovis93/next-scene-qwen-image-lora-2509` (cinematic progression fine-tune)
- Text encoder: `Qwen2.5-VL-72B-Instruct` (via Hugging Face InferenceClient for prompt enhancement)

## Running the Application

**Start the Gradio interface:**
```bash
python app.py
```

**Install dependencies:**
```bash
pip install -r requirements.txt
```

The app requires GPU access. It uses the `@spaces.GPU` decorator for Hugging Face Spaces zero-GPU allocation.

## Architecture

### Pipeline Flow

1. **Input Processing** (`app.py:infer`):
   - Accepts input images via Gradio Gallery (filepath-based)
   - Optional prompt rewriting using `Qwen2.5-VL-72B-Instruct` API
   - Automatic "Next Scene" prompt generation from images

2. **Image Generation** (`qwenimage/pipeline_qwenimage_edit_plus.py`):
   - Custom pipeline extending `DiffusionPipeline`
   - Encodes images using VAE at 1024x1024 for latents
   - Encodes conditioning images at 384x384 for text encoder
   - Packs latents into 2x2 patches (latent dims must be divisible by 2)
   - Uses `FlowMatchEulerDiscreteScheduler` for denoising

3. **Optimization** (`optimization.py`):
   - Ahead-of-time (AOT) compilation using `torch.export` and `spaces.aoti_compile`
   - Dynamic shapes for variable sequence lengths
   - Custom inductor configs for performance (max_autotune, cudagraphs)
   - FlashAttention 3 integration via `QwenDoubleStreamAttnProcessorFA3`

4. **Output Handling**:
   - Saves outputs to `outputs/` directory with unique timestamps
   - Maintains 20-image history gallery
   - Optional video generation via `multimodalart/wan-2-2-first-last-frame` Space

### Custom QwenImage Components

**Location:** `qwenimage/` package

- `pipeline_qwenimage_edit_plus.py` - Main diffusion pipeline with LoRA support
- `transformer_qwenimage.py` - Custom transformer model with cache management
- `qwen_fa3_processor.py` - FlashAttention 3 attention processor

**Key architectural features:**
- Latent packing/unpacking for 2x2 patch processing
- Multi-image conditioning support
- True CFG (classifier-free guidance) with separate pos/neg paths
- Dual-stream attention with rotary embeddings
- Cache contexts for conditional/unconditional forward passes

### Prompt Engineering

**Two-stage prompt system:**

1. **Edit Instruction Rewriter** (`SYSTEM_PROMPT`):
   - Normalizes user prompts into professional editing instructions
   - Handles text replacement (requires quotes), object manipulation, style transfer
   - Used when `rewrite_prompt=True` checkbox is enabled

2. **Next Scene Generator** (`NEXT_SCENE_SYSTEM_PROMPT`):
   - Automatically suggests cinematic camera movements
   - Focus on visual progression (dolly, pan, zoom, framing changes)
   - Auto-triggers when input images change

Both use `Qwen2.5-VL-72B-Instruct` via Hugging Face InferenceClient with Nebius provider. Requires `HF_TOKEN` environment variable.

## Important Implementation Details

### Image Dimension Handling

Images are automatically resized based on `calculate_dimensions()` function:
- VAE images: resized to maintain 1024×1024 area (1,048,576 pixels)
- Condition images: resized to maintain 384×384 area (147,456 pixels)
- Output dimensions must be divisible by 16 (vae_scale_factor × 2)
- Height/width default to `None` which auto-calculates from input aspect ratio

### LoRA Integration

The pipeline fuses the "next-scene" LoRA adapter at initialization:
```python
pipe.load_lora_weights("lovis93/next-scene-qwen-image-lora-2509", ...)
pipe.set_adapters(["next-scene"], adapter_weights=[1.])
pipe.fuse_lora(adapter_names=["next-scene"], lora_scale=1.)
pipe.unload_lora_weights()
```

After fusion, the adapter weights are merged into the base model and cannot be unfused.

### Video Generation Integration

The `turn_into_video()` function:
- Connects to external Gradio Space `multimodalart/wan-2-2-first-last-frame`
- Requires first input image and last output image
- Uses the original prompt (or "smooth cinematic transition" fallback)
- Returns video path for display

### Gradio Gallery Format

Input/output galleries use `type="filepath"` (string paths) rather than PIL Image tuples. Helper functions handle format compatibility for legacy tuple support.

## Environment Variables

- `HF_TOKEN` - Required for Qwen2.5-VL API access (prompt rewriting/generation)

## File Outputs

Generated images are saved to `outputs/` directory with format:
```
output_{seed}_{index}_{timestamp_ms}.png
```

## Local Development and API Testing

The `custom/` directory is fully gitignored and used for local development files. Specifically, it contains:

- **API client scripts** - For testing the Gradio Space remotely via API after deployment to Hugging Face
- **`API_GUIDE.txt`** - Auto-generated Gradio API documentation showing endpoint signatures and example usage
- **Local testing environments** - Virtual environments or test data that shouldn't be committed

**API Integration Pattern:**
Once the Space is deployed to Hugging Face, you can interact with it programmatically using `gradio_client`:

```python
from gradio_client import Client, handle_file

client = Client("Sneak-Moose/Qwen-Image-Edit-next-scene")
result = client.predict(
    images=[],
    prompt="Next Scene: Camera dollies forward...",
    seed=42,
    randomize_seed=False,
    true_guidance_scale=1.0,
    num_inference_steps=4,
    height=1024,
    width=1024,
    rewrite_prompt=False,
    api_name="/infer"
)
```

The `custom/API_GUIDE.txt` contains full documentation of all available endpoints including `/infer`, `/turn_into_video`, `/suggest_next_scene_prompt`, and utility functions.

## Development Notes

- The model loads on startup and applies AOT compilation during first inference
- Compilation uses dynamic shapes to support variable text/image sequence lengths
- The transformer uses custom cache contexts ("cond"/"uncond") to optimize CFG passes
- True CFG applies norm-based rescaling: `comb_pred * (cond_norm / noise_norm)`
- FlashAttention 3 processor must be set before compilation