darask0/Krea-2-color-depth-control

Krea 2 Color-Depth Control (unofficial)

A control adapter for Krea 2 that makes generation follow a color region map (where each object is) and a depth map (its 3D shape and front/back order), from the first denoising step to the last. It was made for posing scenes with 3D mannequins and for region-painted layouts, together with per-region prompts.

Unofficial: this is a community Derivative of Krea 2 and is not made or endorsed by Krea. It is distributed under the Krea 2 Community License; see NOTICE.

Held-out inputs, seed 1. Rows: two posed mannequins, a mannequin reaching toward a box, and a 12-region manga page map. Columns: the input, region prompts only, a training-free i2i baseline, then this adapter at 2,500 / 5,000 / 7,500 (released as krea2-control.safetensors) / 10,000 steps. Labels are in Japanese.

Files

File
krea2-control.safetensors	Recommended. Step 7,500. Best held-out results, including the manga page map.
krea2-control-10000.safetensors	Final step 10,000. Similar on 3D scenes; on flat panel maps it sometimes adds an extra person.
examples/	Two 3D scenes exported from a three.js mannequin editor: clay.png (shaded render), color.png (color pass), depth.png (depth pass), and regions.json for Hayate.
training/config.toml	Training configuration.
LICENSE.pdf, NOTICE	Krea 2 Community License and the attribution notice.

What it does

Region prompts alone tell Krea 2 what goes in each painted region, but the model drifts from the regions' positions and sizes (in the two-person scene both people move to the center). Image-to-image from a 3D render keeps the layout but carries the render's flat background and grey color into the result, and only works in a narrow strength band.

With this adapter the controls are conditions, not image content. Generation stays text-to-image from pure noise, and the output follows the map's placement, scale and pose, with a background described only by the prompt. In the examples the raised arm, the arm reaching toward the desk and the full-body scale of the mannequins are reproduced in a single pass.

Two people	Person and desk	Manga page map

Each grid (step 7,500, seeds 1 and 2): the inputs, a training-free baseline, then the adapter with full control and with early control (color end 0.4, depth end 0.6).

Inputs

• Color map: an RGB image at the output size where each object or region has its own flat color. The colors are IDs only and carry no class meaning. Black is fine for "unspecified" (for example the background of a 3D color pass). Training maps came from COCO panoptic segmentation (every thing and stuff segment in a random distinct color).
• Depth map: a grayscale image at the output size, near = white, normalized to the scene's range. Depth Anything V2 style relative depth, or a renderer's depth pass, both work.

Either input may be omitted: the adapter was trained with control dropout (both 20%, color only 35%, depth only 35%, none 10%).

Format

Weights are BF16 safetensors, keyed by the diffusers Krea2Transformer2DModel module names.

• img_in.weight [6144, 192] and img_in.bias [6144] replace the transformer's img_in (originally [6144, 64]). The input is the concatenation along channels of three 2×2-packed latents of 64 channels each:

  1. the noisy latent (as usual),
  2. the color map latent,
  3. the depth map latent.

  Control latents are the Krea 2 VAE (AutoencoderKLQwenImage) encoding of the control image, with depth replicated to RGB and pixels scaled to [-1, 1]. Use the posterior mean at inference, normalize with the VAE's latents_mean / latents_std, and pack 2×2 like the image latent. A zero latent means "no control" for that input. The output projection is unchanged: the model still predicts the usual target for the noisy latent only.

• *.lora_A.weight / *.lora_B.weight: rank 64, alpha 64, on all 263 linear layers of the transformer.

• In Hayate, strength scales the LoRA (img_in is never scaled). Per-input end fractions (color_end, depth_end) zero a control after that fraction of the steps, so the layout is fixed early and details stay free.

Trained on Krea 2 Raw, used with both Krea 2 Turbo (8 steps, as in all images here) and Raw.

Reference implementation: Hayate

Hayate (Rust/CUDA) runs it through --regions JSON, combined with region prompts, per-region LoRAs and region groups:

{
  "map": "color.png",
  "regions": [
    {"color": "#ff1a1a", "prompt": "a young man with short black hair, white shirt and jeans, raising his right hand and smiling"},
    {"color": "#1affff", "prompt": "a young woman with long brown hair in a red dress, walking toward the viewer"}
  ],
  "isolate_until": 0.2,
  "control": {"adapter": "krea2-control.safetensors", "color_map": "color.png", "depth": "depth.png",
              "strength": 1.0, "color_end": 1.0, "depth_end": 1.0}
}

hayate run krea2 --prompt "a photo of a sunny city street, daytime" --regions examples/two-people/regions.json --seed 1 --out out.png

Keep the base prompt to the scene only and describe each subject in its region. On an RTX PRO 6000 the 1024² DiT takes about 2.6 s with control (2.3 s without).

Training

• Base: Krea 2 Raw transformer, frozen in FP8; img_in widened to 192 inputs, with the control columns zero-initialized.
• Data: 10,000 Krea 2 Turbo images at 1024-class resolutions in five aspect ratios. Color maps come from Mask2Former Swin-L (COCO panoptic), depth from Depth Anything V2 Large (near = white, with light blur/quantization augmentation). Captions have 10% dropout.
• Optimization: AdamW, lr 1e-4, with 10× lr on the control columns of img_in. Batch 1 for 10,000 steps, at 0.73 s/step on one RTX PRO 6000 (peak 62 GB). An earlier run at 1× lr for the control columns barely learned to use the controls: their weights reached only 13–16% of the image columns' norm after 10,000 steps.

Limitations

• The training images are photographic Krea 2 Turbo outputs, so strong control can pull illustration styles toward realism.
• Flat, rectangular layouts such as manga panel maps are outside the training distribution. Step 7,500 followed the test page well, while steps 5,000 and 10,000 sometimes put a person into an empty panel. For multi-panel pages, combine it with region groups (panel isolation).
• Fine details follow the depth map only as far as the mannequin defines them; hands and fingers come from the model.

---

日本語の概要

Krea 2 用の非公式な制御アダプタです。色で塗り分けた領域マップと深度マップを、生成の最初から最後まで条件として与えます。3D マネキンでポーズを付けた構図や、領域ごとのプロンプトと組み合わせて使います。

• 推奨は krea2-control.safetensors（7,500 ステップ）です。学習に使っていない入力で、配置・全身の縮尺・手を上げる/腕を伸ばす姿勢が、txt2img 1 回で再現されました。
• 入力:
  • 色マップ: 物体ごとに別の色で塗ったもの。黒は「指定なし」です。
  • 深度: 近いほど白。
• どちらか片方だけでも使えます。
• Krea 2 Community License に従って配布しています（LICENSE.pdf、NOTICE）。Krea の公式製品ではありません。