README.md

---
license: apache-2.0
base_model: madtune/pixeldit-diffusers
tags:
  - controlnet
  - ip-adapter
  - pixeldit
  - diffusers
  - image-to-image
  - style-transfer
---

# PixelDiT ControlNet + IP-Adapter

ControlNet scribble conditioning and IP-Adapter style transfer for [PixelDiT-1300M](https://huggingface.co/madtune/pixeldit-diffusers).

> **Note:** PixelDiT-1300M is a model by [NVIDIA Research](https://research.nvidia.com/). This repo contains trained adapters only — we are not affiliated with NVIDIA.

## Files

| File | Description |
|---|---|
| `controlnet.safetensors` | Combined ControlNet (7 blocks) + IP-Adapter weights |
| `ip_adapter.safetensors` | IP-Adapter weights only |
| `hed_detector.safetensors` | HED edge detector (Apache-2.0, VGG-based) |
| `config.json` | Model config |
| `train.py` | Joint ControlNet + IP-Adapter training script |
| `precompute_wd_tags.py` | Run WD tagger on dataset → `wd_tags.json` |
| `precompute_embeddings.py` | Encode images with SigLIP + Gemma → memmap files |
| `precompute_hed.py` | Precompute HED edge maps for a dataset |
| `control_maps.py` | Edge map post-processing utilities |
| `hed.py` | HED model definition |
| `convert_to_safetensors.py` | Convert .pt checkpoints to safetensors |

## Usage

```python
from diffusers.pipelines.pixeldit import PixelDiTStyledPipeline
from huggingface_hub import hf_hub_download
from PIL import Image
import torch

pipe = PixelDiTStyledPipeline.from_pretrained_styled(
    "madtune/pixeldit-diffusers",
    controlnet_path=hf_hub_download("madtune/pixeldit-controlnet", "controlnet.safetensors"),
    ip_adapter_path=hf_hub_download("madtune/pixeldit-controlnet", "ip_adapter.safetensors"),
    hed_ckpt_path=hf_hub_download("madtune/pixeldit-controlnet", "hed_detector.safetensors"),
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload(gpu_id=1)

out = pipe(
    image=Image.open("style_ref.jpg"),
    prompt="gothic pale woman, dramatic rim lighting",
    variation_strength=0.85,
    ctrl_strength=0.25,
    ip_strength=0.85,
    flow_shift=8.0,
    guidance_scale=4.5,
    num_inference_steps=50,
).images[0]
out.save("output.jpg")
```

## Recommended settings

| Mode | `ctrl_strength` | `ip_strength` | `variation_strength` |
|---|---|---|---|
| Pure variation | 0.0 | 0.0 | 0.65–0.85 |
| ControlNet only | 0.25 | 0.0 | 0.85 |
| IP-Adapter only | 0.0 | 0.85 | 0.85 |
| Full combo (best) | 0.25 | 0.35–0.85 | 0.85 |

`flow_shift=8.0` + `guidance_scale=3.0–3.5` works well at 768px+. `4.5` is valid but produces oversaturated colours.