redux_experiment_final/research/FLUX-REDUX-CONTROLNET-RESEARCH.md

# Flux Redux + ControlNet — research & implementation plan (Aug 2026)

Goal: workflows with **1 style-reference image (Redux)** + **1 composition-reference image
(depth/canny ControlNet)** + **optional text prompt**, plus simple Redux-only variants and a
faithful replica of fal-ai's `flux/dev/redux` endpoint. Maximum quality; pod is H100 80GB.

## Recommended stack (priority workflow)

```
flux1-dev (fp8 checkpoint, or bf16 unet for max quality)
├─ Style img  → CLIPVisionEncode (sigclip 384) → StyleModelApply (flux1-redux-dev,
│               strength ~0.5, strength_type=attn_bias)
├─ Prompt     → CLIPTextEncode → FluxGuidance 3.5 (empty string = no prompt; works fine)
├─ Comp img   → DepthAnythingV2Preprocessor (or native Canny node)
│             → ControlNetLoader (Union-Pro-2.0) → ControlNetApplySD3 ("Apply ControlNet
│               with VAE") strength 0.6–0.8, end_percent 0.8
└─ euler / simple / 20–28 steps / CFG 1.0
```

Conditioning order: text → StyleModelApply → ControlNetApplySD3 (matches all published
workflows; ControlNet applies to positive+negative, negative = ConditioningZeroOut).

### Why Union-Pro-2.0 for the ControlNet

- `Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0` (4.28 GB bf16) is the 2025–26 community
  default for FLUX structural control: one model covers **depth, canny, soft-edge, pose, gray**,
  mode inferred from the control image (no `SetUnionControlNetType` node, unlike v1).
- Independent `strength` / `start_percent` / `end_percent` knobs, and multiple applies **stack**
  (depth + canny simultaneously) — exactly what balancing against Redux needs.
- Official recommended settings: depth 0.8 / end 0.8, canny 0.7 / end 0.8, soft-edge 0.7 / 0.8,
  pose 0.9 / 0.65. Drop toward 0.4–0.6 if it fights the style.
- Alternatives considered:
  - **BFL FLUX.1-Depth/Canny-dev full models** (23.8 GB each): strongest raw structure adherence
    (trained by BFL alongside flux1-dev), combine fine with Redux (Redux conditioning feeds
    `InstructPixToPixConditioning`), but **no strength/timing knob at all**, one structure signal
    only (no depth+canny stack), and each replaces the base model. Tunability is what quality
    actually hinges on when Redux and structure must be balanced → not chosen as primary.
  - **BFL depth/canny LoRAs** (1.24 GB, `LoraLoaderModelOnly` on stock flux1-dev, adherence via
    LoRA strength): good middle ground, worth an A/B variant later, still no timing control and
    still latent-concat (no stacking).
  - **XLabs v3 controlnets / x-flux-comfyui**: legacy since early 2025, dormant repo,
    `XlabsSampler` doesn't compose with normal conditioning. Skip; retire the old
    `flux1-depth-controlnet-xlabs-redroom` workflows rather than port them.

### Redux strength control (critical — Redux leaks composition)

Redux appends 729 SigLIP patch tokens to the text conditioning; at full strength it imposes the
style image's layout/subjects, not just aesthetics. Two current mechanisms:

1. **Native** `StyleModelApply` `strength` + `strength_type` (in our ComfyUI 0.27.0,
   `nodes.py:1100`): use **`attn_bias` ~0.3–0.7 (start 0.5)** — down-weights attention to the
   Redux tokens; officially recommended since v0.3.8. `multiply` mode degrades style before it
   stops leaking composition — avoid.
2. **ReduxAdvanced** (`kaibioinfo/ComfyUI_AdvancedRefluxControl`): token **downsampling**
   (27×27 → 9×9 at factor 3) removes spatial information outright — qualitatively better at
   killing composition leakage while keeping style; also style masking + autocrop. Repo dormant
   (last commit Apr 2025) but functional. Plan: include as a selectable branch in the priority
   workflow (native attn_bias path active by default, ReduxAdvanced group bypassed) — the two
   mechanisms attack different problems (attention weight vs spatial info) and A/B-ing them is
   cheap once the node is installed.

Reviewed video workflows (user-supplied, all Nov–Dec 2024, pre-dating native attn_bias):
- Sebastian Kamph "How to use Flux Redux in ComfyUI" (YSJsejH5Viw) — official example graph;
  covered by workflows #2/#3, nothing new.
- Olivio Sarikas "REDUX Advanced for FLUX" (UrUDHSpmB90) — ComfyUI_AdvancedRefluxControl demo
  (style strength, image combining, masking). Patreon workflow ≈ repo example workflows.
- Code Crafters Corner "Flux Redux: Advanced Techniques" (kh3ikwEZQXk) — same node, focused on
  restoring positive-prompt authority over Redux (downsampling + strength), i.e. exactly our
  optional-text-prompt requirement. Simple + advanced graphs, rebuildable from the repo examples.
Net effect on plan: promote ReduxAdvanced from "fallback" to "installed + bypassed branch".

Inspected the node repo's own example workflows (what the Olivio / Code Crafters videos demo):
`simple_workflow.json` (StyleModelApplySimple preset "medium") and `advanced_workflow.json`
(ReduxAdvanced: downsampling 3, area, center-crop, weight 1.0) — both are the official-example
graph (UNETLoader flux1-dev + SamplerCustomAdvanced) with the apply node swapped. They load the
unet with `weight_dtype: fp8_e4m3fn` (runtime cast); on the H100 use `default` (bf16).

## Base models (final, no quantized checkpoints)

Single base across all workflows, loaded split (not all-in-one checkpoint):
- `diffusion_models/flux1-dev.safetensors` — bf16, 23.8 GB, `UNETLoader` weight_dtype `default`
- `text_encoders/t5xxl_fp16.safetensors` + `text_encoders/clip_l.safetensors` (`DualCLIPLoader`)
- `vae/ae.safetensors`
Adapters riding on it (not base models): `style_models/flux1-redux-dev.safetensors`,
`clip_vision/sigclip_vision_patch14_384.safetensors`, `controlnet/…Union-Pro-2.0.safetensors`.
The existing mirror `checkpoints/flux1-dev-fp8.safetensors` is NOT used in these workflows.

Swappability: any FLUX.1-**dev**-family finetune unet drops into `UNETLoader` unchanged (Redux
edits conditioning only; the ControlNet is an external module — both trained against flux1-dev,
adherence degrades gracefully with finetune drift). Schnell-family models are NOT compatible
(different distillation; Redux/Union-Pro trained on dev). LoRAs stack freely via
`LoraLoaderModelOnly` between UNETLoader and sampler — orthogonal to both Redux and ControlNet
(community precedent: pulid + redux + style-lora + depth-CN workflows).

With ControlNet owning composition, Redux can run stronger than in Redux-only workflows.
Starting balance: Redux attn_bias 0.5–0.7, depth CN 0.6–0.8/end 0.8; optional stacked canny
low (0.25–0.4) for hard contour lock.

## Workflows to implement

| # | File (proposed) | Graph | Notes |
|---|---|---|---|
| 1 | `flux-redux-style-composition.json` | Redux + Union-Pro-2.0 depth + optional prompt | **Priority.** Depth group active; bypassed canny group (native `Canny` node) usable instead of or stacked with depth |
| 2 | `flux-redux-fal-dev.json` | Official ComfyUI Redux example, fal `flux/dev/redux` defaults | bf16 flux1-dev via UNETLoader, t5 fp16, empty prompt, FluxGuidance 3.5, euler/simple/28, denoise 1.0, 768×1024, StyleModelApply strength 1.0 multiply. Doubles as the simple 1-image workflow; no safety checker (fal's black-image issue disappears). Seeds won't match fal's |
| 3 | `flux-redux-prompt.json` | Redux + text prompt, no ControlNet | Same graph as #2 but prompt filled and Redux restrained (attn_bias ~0.5) so the prompt has authority; fp8 checkpoint fine |
| 4 | `flux-redux-schnell.json` | Replicate `black-forest-labs/flux-redux-schnell` parity | bf16 flux1-schnell via UNETLoader (t5 fp16 + clip_l + ae shared), Redux strength 1.0 multiply, **no text prompt** (endpoint has none — redux_image replaces it), **no FluxGuidance** (schnell ignores guidance; none in endpoint schema), euler/simple/**4 steps**/denoise 1.0, 1024×1024 default (endpoint: aspect_ratio enum @ ~1MP, megapixels 1/0.25, num_outputs 1–4 → batch size). No conditioning LoRAs. Seeds won't match Replicate's. Schnell is Apache-2.0, HF repo NOT gated |
| 5 | (later, optional) `flux-redux-style-composition-bfl.json` | Redux + BFL depth LoRA via InstructPixToPixConditioning | A/B contender for max structural fidelity |

Also folded in: bypassed `LoraLoaderModelOnly` (`loras/flux1-turbo-alpha.safetensors`, already in
mirror) + low-step preset group in workflows #1–#3 as the fast-preview tier (better quality than
schnell at similar speed on dev).

## Models to add to the `aleph65/ComfyUI` mirror

Already there: `checkpoints/flux1-dev-fp8.safetensors`, `clip_vision/sigclip_vision_patch14_384.safetensors`,
`text_encoders/t5xxl_fp16.safetensors` + `clip_l.safetensors`, `vae/ae.safetensors`.

Needed:

| File → mirror path | Size | Source |
|---|---|---|
| `style_models/flux1-redux-dev.safetensors` | 129 MB | `black-forest-labs/FLUX.1-Redux-dev` (**gated** — accept license, download with your HF token) |
| `controlnet/FLUX.1-dev-ControlNet-Union-Pro-2.0.safetensors` | 4.28 GB | `Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0/diffusion_pytorch_model.safetensors` (rename) |
| `diffusion_models/flux1-dev.safetensors` | 23.8 GB | `black-forest-labs/FLUX.1-dev` (**gated**) — bf16 base for #1–#3 |
| `diffusion_models/flux1-schnell.safetensors` (workflow #4) | 23.8 GB | `black-forest-labs/FLUX.1-schnell` (not gated, Apache-2.0) |
| (optional, #5) `loras/flux1-depth-dev-lora.safetensors` | 1.24 GB | `black-forest-labs/FLUX.1-Depth-dev-lora` (**gated**) |

`depth_anything_v2_vitl.pth` auto-downloads at first run into
`custom_nodes/comfyui_controlnet_aux/ckpts/` (not a `models/` path).

## Custom nodes

- **`comfyui_controlnet_aux` (Fannovel16)** — `DepthAnythingV2Preprocessor`; still the standard,
  no native depth estimator in core. Canny needs nothing (core node).
- **`ComfyUI_AdvancedRefluxControl` (kaibioinfo)** — optional, only if native attn_bias proves
  insufficient for style-only isolation.
- Nothing else: Redux, StyleModelApply (with strength), ControlNetApplySD3, Canny are all core.
  Workflow JSONs must carry `cnr_id`/`aux_id` in node properties for `download_missing_models.sh`.

## Gotchas

- **Scaled-fp8 flux checkpoints produce black images with Redux** (ComfyUI #5849). Our mirror's
  `flux1-dev-fp8.safetensors` is the plain e4m3fn Comfy-Org repack (used fine by the USO
  workflow) — safe. If black outputs ever appear, that's the first suspect.
- Union-Pro-**2.0** must NOT get a `SetUnionControlNetType` node (that's v1-only; the old
  `flux-union-pro-controlnet-instantx-shakker-labs.json` workflow used v1 + that node).
- `CLIPVisionEncode` crop: `center` (default) discards edges of non-square style refs; fine for
  style, switch to `none` if edge content matters.
- fp8 vs bf16 base: fp8 = slight quality loss (official note), 2× faster load; H100 80GB runs
  bf16 + t5 fp16 comfortably → bf16 for #2 (fal parity), fp8 acceptable for #1/#3 per house
  convention (can flip to bf16 unet if A/B shows visible gain).
- Style-fidelity ranking (community, 2026): ByteDance **USO** (already have
  `uso-style-transfer.json`) ≥ Redux+downsampling/attn_bias > XLabs IP-Adapter. Redux remains
  the lightest and the easiest to combine with an independent ControlNet branch — the right
  choice for this use case; USO is the fallback if Redux style fidelity disappoints.

## Sources

- https://blog.comfy.org/p/day-1-support-for-flux-tools-in-comfyui
- https://comfyanonymous.github.io/ComfyUI_examples/flux/
- https://comfy.org/workflows/flux_redux_model_example-52dd3f09bb59/
- https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0
- https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev
- https://github.com/kaibioinfo/ComfyUI_AdvancedRefluxControl
- https://github.com/comfyanonymous/ComfyUI/releases/tag/v0.3.8 (attn_bias)
- https://github.com/comfyanonymous/ComfyUI/issues/5849 (scaled-fp8 black images)
- https://openart.ai/workflows/odam_ai/flux---style-transfer-controlnet-flux-tools-redux---beginner-friendly/LWMhfWmaku6tdDWjkM8D
- https://civitai.com/models/1312599/flux-advanced-redux-with-controlnet-flux-tool