README.md

---
license: openrail++
base_model: stabilityai/stable-diffusion-xl-base-1.0
tags:
  - stable-diffusion-xl
  - controlnet
  - temporal
  - video
  - diffusers
inference: true
---

# TemporalNet2 ControlNet for SDXL

This is a TemporalNet2 ControlNet model trained on SDXL (Stable Diffusion XL base 1.0).

## Model Description

TemporalNet2 is a ControlNet variant designed for temporal coherence in video generation. It takes two conditioning inputs:
- **Previous Frame**: The previous frame in the video sequence (3 channels)
- **Optical Flow**: The optical flow between the previous and current frame (3 channels)

Total conditioning channels: **6 channels**

This model was trained to generate temporally coherent frames by learning from both the visual content of the previous frame and the motion information encoded in optical flow.

## Usage

```python
from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, EulerDiscreteScheduler
from PIL import Image
import torch

# Load the ControlNet model
controlnet = ControlNetModel.from_pretrained(
    "YOUR_USERNAME/temporalnet2-sdxl-controlnet",
    torch_dtype=torch.float16
)

# Create the pipeline
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    torch_dtype=torch.float16
)
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")

# Load your conditioning images
prev_frame = Image.open("previous_frame.jpg")
optical_flow = Image.open("optical_flow.jpg")

# Concatenate conditioning images (they will be concatenated in the pipeline)
# Note: You'll need to prepare the 6-channel input by concatenating prev_frame and optical_flow
prompt = "your prompt describing the scene"

# Generate
image = pipe(
    prompt=prompt,
    image=[prev_frame, optical_flow],  # The pipeline will handle concatenation
    num_inference_steps=20,
    guidance_scale=7.5
).images[0]

image.save("output.jpg")
```

## Training Details

- **Base Model**: stabilityai/stable-diffusion-xl-base-1.0
- **Training Resolution**: Multi-resolution (512, 640, 768, 896, 1024px)
- **Conditioning Channels**: 6 (3 for previous frame + 3 for optical flow)
- **Training Steps**: 25,000
- **Mixed Precision**: bfloat16

## Limitations

This model requires specific conditioning inputs:
1. The previous frame from your video sequence
2. The optical flow computed between frames

For best results, ensure your optical flow visualization uses a consistent color scheme and magnitude representation.

## License

This model is released under the same license as SDXL (OpenRAIL++).