| # FLUX Redux + ControlNet Style-Composition — Final Report & Replication Guide |
|
|
| **Date:** August 2026 · **Pod:** RunPod H100 80GB, ComfyUI 0.27.0 (`plx1029/comfyui-qwen` image) |
| **Goal:** 1 style-reference image + 1 composition-reference image (+ optional text prompt) → |
| style-transferred, composition-locked output on FLUX.1-dev. Maximum quality. |
|
|
| Everything here also lives on HF `aleph65/ComfyUI` (workflows in `workflows/`, models in |
| `models/`, this archive in `redux_experiment_final/`). |
|
|
| --- |
|
|
| ## TL;DR — the final production stack |
|
|
| ``` |
| FLUX.1-dev bf16 (UNETLoader, weight_dtype=default) |
| ├─ Style img → sigclip_vision_384 → CLIPVisionEncode(center) → StyleModelApply |
| │ (flux1-redux-dev, strength 0.35, strength_type=MULTIPLY — never attn_bias, see §Findings) |
| ├─ Prompt → handwritten per-run: target's content + style ref's aesthetic → FluxGuidance 3.0 |
| ├─ Comp img → ImageResize+ 896x1152 fill/crop → DepthAnythingV2(vitl) → Union-Pro-2.0 |
| │ via ControlNetApplySD3 (strength 0.7, start 0, end 0.8) — DEPTH ONLY, single CN |
| └─ KSampler euler/simple/32 steps/CFG 1.0/denoise 1.0, seed 777 for all test runs |
| └─ optional HIRES: LatentUpscaleBy bislerp 1.5x → 2nd pass denoise 0.30 (Redux kept, CN released) |
| ``` |
|
|
| A/B variant (`bfl-lora`): same graph but structure via `flux1-depth-dev-lora` **0.85** + |
| `InstructPixToPixConditioning`, **FluxGuidance 10.0** (BFL spec — the LoRA is distilled at |
| guidance 10). Competitive with the CN path since the v4 fixes; less tunable (no end_percent, |
| no stacking). |
| |
| ## The 7 workflow files (in `workflows/`) |
| |
| | file | purpose | |
| |---|---| |
| | `flux-v2-redux-style-composition.json` | **The main one.** Style+composition, all fixes baked. Bypassed groups: canny stack, ReduxAdvanced alt, turbo-LoRA fast preview, HIRES | |
| | `flux-v2-redux-style-composition-bfl-lora.json` | BFL depth-LoRA A/B variant, spec settings | |
| | `flux-redux-fal-dev.json` | fal `flux/dev/redux` endpoint replica (768×1024, empty prompt, strength 1.0 multiply, euler/simple/28, guidance 3.5). Doubles as simple 1-image workflow | |
| | `flux-redux-prompt.json` | 1 image + prompt with authority (attn_bias 0.5 — fine here, NO ControlNet in this graph) | |
| | `flux-redux-schnell.json` | Replicate `flux-redux-schnell` replica (4 steps, no guidance, no prompt; 1024² = their 1:1 default, set 896×1152 for portrait) | |
| | `flux-redux-style-composition.json` (v1) | Historical — superseded by flux-v2 | |
| | `flux-redux-style-composition-bfl-lora.json` (v1) | Historical — superseded | |
|
|
| `workflows/wip/` = every draft revision; `workflows/original-old-workflows/` = the pre-project |
| XLabs/Union-v1 workflows this started from (retired: XLabs is legacy, Union v1 needs |
| `SetUnionControlNetType` which v2.0 must NOT have). |
|
|
| ## Models (all mirrored on HF `aleph65/ComfyUI` under `models/`) |
|
|
| | file | dir | source | gated | |
| |---|---|---|---| |
| | `flux1-dev.safetensors` (bf16 23.8GB) | diffusion_models | black-forest-labs/FLUX.1-dev | yes | |
| | `flux1-schnell.safetensors` (bf16 23.8GB) | diffusion_models | black-forest-labs/FLUX.1-schnell | no | |
| | `flux1-redux-dev.safetensors` (129MB) | style_models | black-forest-labs/FLUX.1-Redux-dev | yes | |
| | `FLUX.1-dev-ControlNet-Union-Pro-2.0.safetensors` (4.28GB) | controlnet | Shakker-Labs (renamed from `diffusion_pytorch_model.safetensors`) | no | |
| | `flux1-depth-dev-lora.safetensors` (1.24GB) | loras | black-forest-labs/FLUX.1-Depth-dev-lora | yes | |
| | `t5xxl_fp16.safetensors` + `clip_l.safetensors` | text_encoders | Comfy-Org repack | no | |
| | `ae.safetensors` | vae | FLUX.1-dev repo | (yes) | |
| | `sigclip_vision_patch14_384.safetensors` | clip_vision | Comfy-Org/sigclip_vision_384 | no | |
| | `flux1-turbo-alpha.safetensors` | loras | alimama-creative (fast-preview group) | no | |
| | `depth_anything_v2_vitl.pth` | auto-downloads into `custom_nodes/comfyui_controlnet_aux/ckpts/` | | no | |
| |
| **Custom node packs (3):** `comfyui_controlnet_aux` (Fannovel16 — DepthAnythingV2), |
| `ComfyUI_essentials` (cubiq — ImageResize+), `ComfyUI_AdvancedRefluxControl` (kaibioinfo — |
| ReduxAdvanced alt branch). Everything else is ComfyUI core. |
| |
| ## Replication on a fresh pod |
| |
| 1. Pod from `plx1029/comfyui-qwen` image (or any ComfyUI ≥0.3.8 — that's when native |
| StyleModelApply strength landed). |
| 2. `HUGGING_FACE_ACCESS_TOKEN` env var; accept licenses for FLUX.1-dev, FLUX.1-Redux-dev, |
| FLUX.1-Depth-dev-lora on that account. |
| 3. `./download_missing_models.sh flux-v2-redux-style-composition.json ...` — resolves the node |
| packs from each node's `cnr_id`/`aux_id` and pulls all models from the mirror. (Or run |
| `scripts/download_models.py` which pulls from original sources.) |
| 4. Restart ComfyUI, load the workflow, drop images, go. Defaults are the calibrated ones. |
| 5. Batch runs: `scripts/run_matrix_v4.py` queues via the API (`POST /prompt` with API-format |
| graphs, poll `/history/<id>`). `scripts/prompts_v4.py` holds the handwritten prompt fragments. |
|
|
| ## Findings (chronological, what actually mattered) |
|
|
| 1. **Baseline worked but bodies deformed** (v1–v3): aesthetics great, frequent limb/torso |
| deformities in both two-image paths. Single-image workflows always clean. |
| 2. **ROOT CAUSE (code-verified, undocumented upstream): `attn_bias` + ControlNet is broken.** |
| `StyleModelApply(attn_bias)` appends all 729 Redux tokens at FULL strength and stores the |
| attenuation as an attention-bias mask. The base flux transformer honors the mask; the |
| ControlNet branch never receives it (`ControlNetFlux.forward_orig` has no attn_mask param; |
| not in its `extra_conds`). Base attends Redux @50%, CN attends @100% → two conditioning |
| streams steer different images every step → ghosting/deformities. Also: any attention mask |
| kicks flux off the flash-attention fast path. |
| **Fix: `strength_type=multiply` at 0.25–0.35 whenever a ControlNet is present.** attn_bias |
| remains fine in CN-free graphs. (8-run isolation diagnostic + two deep-research passes; |
| see `comparison_sheets/_diag_sheet.jpg` — column d is the smoking gun.) |
| 3. **BFL depth-LoRA wants its spec: guidance 10.0 + LoRA 0.85.** It is distilled at guidance 10; |
| running 3–4 gives mush. Earlier sweeps that said "guidance 4 looks better" were contaminated |
| by the attn_bias bug. The CN path conversely wants guidance ~3.0. **The two paths need |
| different FluxGuidance values.** |
| 4. **Prompts are the anatomy lever.** Flux resolves limb ambiguity from text. Empty prompt = |
| worst; generic anatomy boilerplate = marginal; **handwritten per-run prompts describing the |
| composition target's actual content fused with the style ref's actual aesthetic = best**. |
| Depth maps are semantically ambiguous at limb crossings; the prompt disambiguates. |
| Negative prompts are INERT at CFG 1.0 (comfy skips uncond entirely) — anatomy must be |
| handled positively. |
| 5. **Calibration verdicts** (30-run, `comparison_sheets/_C*.jpg`): multiply 0.35 > 0.25/0.30 |
| (style strength, all clean) > ReduxAdvanced ds3 (clean but weakens style); 896×1152 fine |
| despite Union-Pro-2.0's 512 training res (no penalty once the conditioning fight was fixed); |
| CN end 0.4 loses composition, 0.6–0.8 both hold → keep 0.8, use 0.6 for freedom; depth-only |
| single CN (canny stack changes scenes — keep off unless contour lock wanted). |
| 6. **Input preprocessing matters**: fill/crop everything to 896×1152 in-graph (portrait 3:4 |
| bucket), depth map == latent to the pixel, zero padding (padding becomes literal image |
| content in the latent-concat/bfl path). Save crops + depth maps for auditing. |
| 7. **Input spec** (what to feed it): 3:4 portrait, 1152×1536, sRGB 8-bit, no watermarks/text/ |
| borders, EXIF baked. Style refs: style-defining content in the vertical middle — sigclip |
| center-crops to a square, top/bottom ~12% discarded. Composition refs: clear fg/bg depth |
| separation. |
| 8. **Hires two-pass** (1.5× latent bislerp + denoise 0.30, Redux carried, CN released): |
| validated win — real detail gain, aesthetic preserved. Ships bypassed. |
| 9. **Known edge case, unsolved**: person-style × empty-room-target (our r×r4 combos) is a |
| contradictory ask; results inconsistent even with interior prompts. Needs its own treatment. |
| 10. **Redux mode compatibility notes**: scaled-fp8 flux checkpoints black-screen with Redux |
| (ComfyUI #5849). XLabs CN stack is legacy/dormant. Union-Pro-**2.0** must NOT have |
| `SetUnionControlNetType` (that's v1-only). |
| |
| ## Prompt-writing recipe (for new image pairs) |
| |
| Write one sentence of **content** from the composition target (subject, pose, clothing, |
| setting — what the depth map encodes), one sentence of **style** from the style ref (lighting |
| type, palette, finish), then `"Sharp focus, natural proportions, coherent anatomy."` |
| If the target has no person, say so explicitly and describe the space instead. |
| Fragments for the 10 experiment inputs are in `scripts/prompts_v4.py`. |
|
|
| ## What's in this archive |
|
|
| ``` |
| README.md ← this file |
| workflows/ ← all 7 finals + wip/ history + original-old-workflows/ |
| inputs/ ← inputs_v1 (first test set) + inputs_v2 (curated 1152×1536 3:4 set) |
| scripts/ ← graph builders, all matrix/sweep/calibration runners, prompts, model downloader |
| research/ ← FLUX-REDUX-CONTROLNET-RESEARCH.md (full research log w/ sources) |
| outputs_v4_final/ ← the final 84-run matrix: 36 style-comp (+hires/ +inputs/ audits), |
| 36 bfl-lora, 12 singles, calib/ (30 calibration runs) |
| comparison_sheets/ ← every verdict sheet: _diag_sheet (the attn_bias smoking gun), |
| _A* (v2 sweeps), _C* (v4 calibration), _v2_vs_v3, _v3_vs_v4 |
| claude_memory/ ← Claude Code memory files from this project |
| ``` |
|
|
| Run provenance: every test image used **seed 777**; matrices covered all 24 r×t + 12 ordered |
| r×r combos per two-image workflow. v1→v4 output sets remain in `/workspace/outputs*` on the pod |
| (not archived here except v4). |
|
|
| — Built with Claude Code (Fable 5), Aug 2026. It was fun, sir. 🫡 |
|
|
