Hugging Face's logo

aleph65
/
ComfyUI

ONNX
Safetensors
Model card Files
xet
 Community
ComfyUI / redux_experiment_final /research /FLUX-REDUX-CONTROLNET-RESEARCH.md
aleph65's picture
aleph65
Archive flux-redux experiment: final writeup, workflows, inputs, scripts, research, v4 outputs, comparison sheets, Claude memories
478cb8f verified
preview code
|
Raw
History Blame Contribute Delete
12.1 kB

Flux Redux + ControlNet — research & implementation plan (Aug 2026)

Goal: workflows with 1 style-reference image (Redux) + 1 composition-reference image (depth/canny ControlNet) + optional text prompt, plus simple Redux-only variants and a faithful replica of fal-ai's flux/dev/redux endpoint. Maximum quality; pod is H100 80GB.

Recommended stack (priority workflow) 

flux1-dev (fp8 checkpoint, or bf16 unet for max quality)
├─ Style img  → CLIPVisionEncode (sigclip 384) → StyleModelApply (flux1-redux-dev,
│               strength ~0.5, strength_type=attn_bias)
├─ Prompt     → CLIPTextEncode → FluxGuidance 3.5 (empty string = no prompt; works fine)
├─ Comp img   → DepthAnythingV2Preprocessor (or native Canny node)
│             → ControlNetLoader (Union-Pro-2.0) → ControlNetApplySD3 ("Apply ControlNet
│               with VAE") strength 0.6–0.8, end_percent 0.8
└─ euler / simple / 20–28 steps / CFG 1.0

Conditioning order: text → StyleModelApply → ControlNetApplySD3 (matches all published workflows; ControlNet applies to positive+negative, negative = ConditioningZeroOut).

Why Union-Pro-2.0 for the ControlNet

  • Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0 (4.28 GB bf16) is the 2025–26 community default for FLUX structural control: one model covers depth, canny, soft-edge, pose, gray, mode inferred from the control image (no SetUnionControlNetType node, unlike v1).
  • Independent strength / start_percent / end_percent knobs, and multiple applies stack (depth + canny simultaneously) — exactly what balancing against Redux needs.
  • Official recommended settings: depth 0.8 / end 0.8, canny 0.7 / end 0.8, soft-edge 0.7 / 0.8, pose 0.9 / 0.65. Drop toward 0.4–0.6 if it fights the style.
  • Alternatives considered:
    • BFL FLUX.1-Depth/Canny-dev full models (23.8 GB each): strongest raw structure adherence (trained by BFL alongside flux1-dev), combine fine with Redux (Redux conditioning feeds InstructPixToPixConditioning), but no strength/timing knob at all, one structure signal only (no depth+canny stack), and each replaces the base model. Tunability is what quality actually hinges on when Redux and structure must be balanced → not chosen as primary.
    • BFL depth/canny LoRAs (1.24 GB, LoraLoaderModelOnly on stock flux1-dev, adherence via LoRA strength): good middle ground, worth an A/B variant later, still no timing control and still latent-concat (no stacking).
    • XLabs v3 controlnets / x-flux-comfyui: legacy since early 2025, dormant repo, XlabsSampler doesn't compose with normal conditioning. Skip; retire the old flux1-depth-controlnet-xlabs-redroom workflows rather than port them.

Redux strength control (critical — Redux leaks composition)

Redux appends 729 SigLIP patch tokens to the text conditioning; at full strength it imposes the style image's layout/subjects, not just aesthetics. Two current mechanisms:

  1. Native StyleModelApply strength + strength_type (in our ComfyUI 0.27.0, nodes.py:1100): use attn_bias ~0.3–0.7 (start 0.5) — down-weights attention to the Redux tokens; officially recommended since v0.3.8. multiply mode degrades style before it stops leaking composition — avoid.
  2. ReduxAdvanced (kaibioinfo/ComfyUI_AdvancedRefluxControl): token downsampling (27×27 → 9×9 at factor 3) removes spatial information outright — qualitatively better at killing composition leakage while keeping style; also style masking + autocrop. Repo dormant (last commit Apr 2025) but functional. Plan: include as a selectable branch in the priority workflow (native attn_bias path active by default, ReduxAdvanced group bypassed) — the two mechanisms attack different problems (attention weight vs spatial info) and A/B-ing them is cheap once the node is installed.

Reviewed video workflows (user-supplied, all Nov–Dec 2024, pre-dating native attn_bias):

  • Sebastian Kamph "How to use Flux Redux in ComfyUI" (YSJsejH5Viw) — official example graph; covered by workflows #2/#3, nothing new.
  • Olivio Sarikas "REDUX Advanced for FLUX" (UrUDHSpmB90) — ComfyUI_AdvancedRefluxControl demo (style strength, image combining, masking). Patreon workflow ≈ repo example workflows.
  • Code Crafters Corner "Flux Redux: Advanced Techniques" (kh3ikwEZQXk) — same node, focused on restoring positive-prompt authority over Redux (downsampling + strength), i.e. exactly our optional-text-prompt requirement. Simple + advanced graphs, rebuildable from the repo examples. Net effect on plan: promote ReduxAdvanced from "fallback" to "installed + bypassed branch".

Inspected the node repo's own example workflows (what the Olivio / Code Crafters videos demo): simple_workflow.json (StyleModelApplySimple preset "medium") and advanced_workflow.json (ReduxAdvanced: downsampling 3, area, center-crop, weight 1.0) — both are the official-example graph (UNETLoader flux1-dev + SamplerCustomAdvanced) with the apply node swapped. They load the unet with weight_dtype: fp8_e4m3fn (runtime cast); on the H100 use default (bf16).

Base models (final, no quantized checkpoints)

Single base across all workflows, loaded split (not all-in-one checkpoint):

  • diffusion_models/flux1-dev.safetensors — bf16, 23.8 GB, UNETLoader weight_dtype default
  • text_encoders/t5xxl_fp16.safetensors + text_encoders/clip_l.safetensors (DualCLIPLoader)
  • vae/ae.safetensors Adapters riding on it (not base models): style_models/flux1-redux-dev.safetensors, clip_vision/sigclip_vision_patch14_384.safetensors, controlnet/…Union-Pro-2.0.safetensors. The existing mirror checkpoints/flux1-dev-fp8.safetensors is NOT used in these workflows.

Swappability: any FLUX.1-dev-family finetune unet drops into UNETLoader unchanged (Redux edits conditioning only; the ControlNet is an external module — both trained against flux1-dev, adherence degrades gracefully with finetune drift). Schnell-family models are NOT compatible (different distillation; Redux/Union-Pro trained on dev). LoRAs stack freely via LoraLoaderModelOnly between UNETLoader and sampler — orthogonal to both Redux and ControlNet (community precedent: pulid + redux + style-lora + depth-CN workflows).

With ControlNet owning composition, Redux can run stronger than in Redux-only workflows. Starting balance: Redux attn_bias 0.5–0.7, depth CN 0.6–0.8/end 0.8; optional stacked canny low (0.25–0.4) for hard contour lock.

Workflows to implement

# File (proposed) Graph Notes
1 flux-redux-style-composition.json Redux + Union-Pro-2.0 depth + optional prompt Priority. Depth group active; bypassed canny group (native Canny node) usable instead of or stacked with depth
2 flux-redux-fal-dev.json Official ComfyUI Redux example, fal flux/dev/redux defaults bf16 flux1-dev via UNETLoader, t5 fp16, empty prompt, FluxGuidance 3.5, euler/simple/28, denoise 1.0, 768×1024, StyleModelApply strength 1.0 multiply. Doubles as the simple 1-image workflow; no safety checker (fal's black-image issue disappears). Seeds won't match fal's
3 flux-redux-prompt.json Redux + text prompt, no ControlNet Same graph as #2 but prompt filled and Redux restrained (attn_bias ~0.5) so the prompt has authority; fp8 checkpoint fine
4 flux-redux-schnell.json Replicate black-forest-labs/flux-redux-schnell parity bf16 flux1-schnell via UNETLoader (t5 fp16 + clip_l + ae shared), Redux strength 1.0 multiply, no text prompt (endpoint has none — redux_image replaces it), no FluxGuidance (schnell ignores guidance; none in endpoint schema), euler/simple/4 steps/denoise 1.0, 1024×1024 default (endpoint: aspect_ratio enum @ ~1MP, megapixels 1/0.25, num_outputs 1–4 → batch size). No conditioning LoRAs. Seeds won't match Replicate's. Schnell is Apache-2.0, HF repo NOT gated
5 (later, optional) flux-redux-style-composition-bfl.json Redux + BFL depth LoRA via InstructPixToPixConditioning A/B contender for max structural fidelity

Also folded in: bypassed LoraLoaderModelOnly (loras/flux1-turbo-alpha.safetensors, already in mirror) + low-step preset group in workflows #1–#3 as the fast-preview tier (better quality than schnell at similar speed on dev).

Models to add to the aleph65/ComfyUI mirror

Already there: checkpoints/flux1-dev-fp8.safetensors, clip_vision/sigclip_vision_patch14_384.safetensors, text_encoders/t5xxl_fp16.safetensors + clip_l.safetensors, vae/ae.safetensors.

Needed:

File → mirror path Size Source
style_models/flux1-redux-dev.safetensors 129 MB black-forest-labs/FLUX.1-Redux-dev (gated — accept license, download with your HF token)
controlnet/FLUX.1-dev-ControlNet-Union-Pro-2.0.safetensors 4.28 GB Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0/diffusion_pytorch_model.safetensors (rename)
diffusion_models/flux1-dev.safetensors 23.8 GB black-forest-labs/FLUX.1-dev (gated) — bf16 base for #1–#3
diffusion_models/flux1-schnell.safetensors (workflow #4) 23.8 GB black-forest-labs/FLUX.1-schnell (not gated, Apache-2.0)
(optional, #5) loras/flux1-depth-dev-lora.safetensors 1.24 GB black-forest-labs/FLUX.1-Depth-dev-lora (gated)

depth_anything_v2_vitl.pth auto-downloads at first run into custom_nodes/comfyui_controlnet_aux/ckpts/ (not a models/ path).

Custom nodes

  • comfyui_controlnet_aux (Fannovel16) — DepthAnythingV2Preprocessor; still the standard, no native depth estimator in core. Canny needs nothing (core node).
  • ComfyUI_AdvancedRefluxControl (kaibioinfo) — optional, only if native attn_bias proves insufficient for style-only isolation.
  • Nothing else: Redux, StyleModelApply (with strength), ControlNetApplySD3, Canny are all core. Workflow JSONs must carry cnr_id/aux_id in node properties for download_missing_models.sh.

Gotchas

  • Scaled-fp8 flux checkpoints produce black images with Redux (ComfyUI #5849). Our mirror's flux1-dev-fp8.safetensors is the plain e4m3fn Comfy-Org repack (used fine by the USO workflow) — safe. If black outputs ever appear, that's the first suspect.
  • Union-Pro-2.0 must NOT get a SetUnionControlNetType node (that's v1-only; the old flux-union-pro-controlnet-instantx-shakker-labs.json workflow used v1 + that node).
  • CLIPVisionEncode crop: center (default) discards edges of non-square style refs; fine for style, switch to none if edge content matters.
  • fp8 vs bf16 base: fp8 = slight quality loss (official note), 2× faster load; H100 80GB runs bf16 + t5 fp16 comfortably → bf16 for #2 (fal parity), fp8 acceptable for #1/#3 per house convention (can flip to bf16 unet if A/B shows visible gain).
  • Style-fidelity ranking (community, 2026): ByteDance USO (already have uso-style-transfer.json) ≥ Redux+downsampling/attn_bias > XLabs IP-Adapter. Redux remains the lightest and the easiest to combine with an independent ControlNet branch — the right choice for this use case; USO is the fallback if Redux style fidelity disappoints.

Sources