| --- |
| license: other |
| license_name: experimental-research-only |
| tags: |
| - controlnet |
| - pose |
| - krea-2 |
| - experimental |
| - not-for-production |
| base_model: krea/Krea-2-Turbo |
| --- |
| |
| # Krea 2 Turbo — Pose ControlNet (S0 SPIKE checkpoints) |
|
|
| > ⚠️ **PURELY EXPERIMENTAL — DO NOT USE IN PRODUCTION.** |
| > These are **feasibility-spike** checkpoints from a GO/NO-GO experiment, **not** a finished |
| > model. They were trained on a tiny corpus (5,000 pairs) for a fixed step budget to answer a |
| > single yes/no question — *can* a Krea-native pose control branch be trained at all. They are |
| > intermediate artifacts kept only to bootstrap ControlNet **interface wiring**. The shippable |
| > overlay will be a **separate, retrained** model on the full corpus. Expect artifacts, |
| > incomplete pose fidelity, and no stability guarantees. |
|
|
| ## What this is |
|
|
| A from-scratch **pose ControlNet control branch** for the **Krea 2 Turbo** image model |
| (Qwen3‑VL‑4B TE + ~12B 28‑block single‑stream DiT + Qwen‑Image VAE; CFG‑free rectified‑flow, |
| 8‑step). Krea 2 has a bespoke backbone with **no transferable ControlNet**, so the control branch |
| was trained from scratch. This repo hosts two intermediate checkpoints from the S0 spike: |
|
|
| | File | Step | Notes | |
| |---|---|---| |
| | `control_step5000.safetensors` (+ `.json`) | 5000 | primary — best pose fidelity in the spike | |
| | `control_step4500.safetensors` (+ `.json`) | 4500 | fallback (clean pre‑resume checkpoint) | |
|
|
| Each `.safetensors` **requires its `.json` sidecar** — it carries the injection metadata |
| (`inject_offset`, `n_blocks`, residual‑clamp `tau`, step) the inference harness reads to |
| configure the branch. Do not use a checkpoint without its sidecar. |
|
|
| ## Architecture / recipe (spike) |
|
|
| - Branch = a trainable copy of the first **N=7** of 28 Krea single‑stream DiT blocks (~3.30B params). |
| - Reads the **VAE‑encoded pose‑skeleton latent**; a **zero‑initialised output projection** per block |
| adds a residual to the **frozen** main stream (base fully frozen: DiT + TE + VAE). |
| - **`inject_offset=1`** — branch injects into main block *i+1*, **skipping main block 0**. |
| - **Residual RMS clamp `τ=0.15`** — each injected residual is capped at `τ × RMS(main image tokens)`. |
| - Loss = Krea flow‑matching over the full noise schedule. Trained bf16 branch, lr 2e‑5, 512², batch 4. |
| |
| These three choices (skip block 0, clamp, and a low‑lr + weight‑decay optimizer group on the |
| zero‑init projections) are load‑bearing: without them the zero‑init projection grows under AdamW |
| until the branch overwrites the frozen stream and generation collapses to noise. |
| |
| ## Intended use |
| |
| **Wiring / integration testing only** — standing up and validating a Krea ControlNet inference |
| route (candle/CUDA `generate_krea_control_stream`, and later the MLX twin). Requires: |
| |
| - the **Krea 2 Turbo** base weights (`krea/Krea-2-Turbo`), and |
| - the `krea-control-infer` harness from the `candle-gen` spike branch |
| (`claudeanthropic/sc-8460/krea-2-pose-controlnet-s0-spike`). |
|
|
| Recommended `control_scale` band **0.5–0.75** (default ~0.6–0.7); `control_scale = 0` is bit‑exact |
| to the base model. Values ≥ ~0.9 over‑drive the branch and degrade image quality (haze / halftone). |
|
|
| ## Known limitations |
|
|
| - **Body pose‑lock** is strong and generalizes to unseen skeletons at 5k pairs; **head‑direction** |
| is *usable but scale‑sensitive* (best near `control_scale` 0.5), not rock‑solid. |
| - Image quality **degrades as `control_scale` rises** — a fundamental authority‑vs‑fidelity |
| tradeoff, sharper here because the base is frozen, CFG‑free, and few‑step. Not fully removable by |
| more data. |
| - Trained at **512²** only. Skeleton convention = OpenPose **body‑18 + head‑direction** |
| (nose/eyes/ears); **no hands, no dense face**. |
| |
| ## Training data & attribution |
| |
| Trained on ~5,000 pairs derived from **COCO train2017 `person_keypoints`** ground‑truth annotations |
| (skeletons rendered in the controlnet_aux OpenPose drawing convention) with **COCO Captions** as the |
| text conditioning. COCO annotations are **CC BY 4.0**; COCO images are Flickr mixed‑license. This |
| checkpoint is a derived model artifact for research/feasibility only. |
| |
| ## Provenance |
| |
| SceneWorks epic 8459 (Krea 2 pose ControlNet), story sc-8460 (S0 spike, verdict = GO). |
| Experimental — retained solely to bootstrap interface wiring; superseded by the production overlay |
| once the full‑corpus run (S3) lands. |
| |
