File size: 4,458 Bytes
210f5a4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
license: other
license_name: experimental-research-only
tags:
- controlnet
- pose
- krea-2
- experimental
- not-for-production
base_model: krea/Krea-2-Turbo
---
# Krea 2 Turbo — Pose ControlNet (S0 SPIKE checkpoints)
> ⚠️ **PURELY EXPERIMENTAL — DO NOT USE IN PRODUCTION.**
> These are **feasibility-spike** checkpoints from a GO/NO-GO experiment, **not** a finished
> model. They were trained on a tiny corpus (5,000 pairs) for a fixed step budget to answer a
> single yes/no question — *can* a Krea-native pose control branch be trained at all. They are
> intermediate artifacts kept only to bootstrap ControlNet **interface wiring**. The shippable
> overlay will be a **separate, retrained** model on the full corpus. Expect artifacts,
> incomplete pose fidelity, and no stability guarantees.
## What this is
A from-scratch **pose ControlNet control branch** for the **Krea 2 Turbo** image model
(Qwen3‑VL‑4B TE + ~12B 28‑block single‑stream DiT + Qwen‑Image VAE; CFG‑free rectified‑flow,
8‑step). Krea 2 has a bespoke backbone with **no transferable ControlNet**, so the control branch
was trained from scratch. This repo hosts two intermediate checkpoints from the S0 spike:
| File | Step | Notes |
|---|---|---|
| `control_step5000.safetensors` (+ `.json`) | 5000 | primary — best pose fidelity in the spike |
| `control_step4500.safetensors` (+ `.json`) | 4500 | fallback (clean pre‑resume checkpoint) |
Each `.safetensors` **requires its `.json` sidecar** — it carries the injection metadata
(`inject_offset`, `n_blocks`, residual‑clamp `tau`, step) the inference harness reads to
configure the branch. Do not use a checkpoint without its sidecar.
## Architecture / recipe (spike)
- Branch = a trainable copy of the first **N=7** of 28 Krea single‑stream DiT blocks (~3.30B params).
- Reads the **VAE‑encoded pose‑skeleton latent**; a **zero‑initialised output projection** per block
adds a residual to the **frozen** main stream (base fully frozen: DiT + TE + VAE).
- **`inject_offset=1`** — branch injects into main block *i+1*, **skipping main block 0**.
- **Residual RMS clamp `τ=0.15`** — each injected residual is capped at `τ × RMS(main image tokens)`.
- Loss = Krea flow‑matching over the full noise schedule. Trained bf16 branch, lr 2e‑5, 512², batch 4.
These three choices (skip block 0, clamp, and a low‑lr + weight‑decay optimizer group on the
zero‑init projections) are load‑bearing: without them the zero‑init projection grows under AdamW
until the branch overwrites the frozen stream and generation collapses to noise.
## Intended use
**Wiring / integration testing only** — standing up and validating a Krea ControlNet inference
route (candle/CUDA `generate_krea_control_stream`, and later the MLX twin). Requires:
- the **Krea 2 Turbo** base weights (`krea/Krea-2-Turbo`), and
- the `krea-control-infer` harness from the `candle-gen` spike branch
(`claudeanthropic/sc-8460/krea-2-pose-controlnet-s0-spike`).
Recommended `control_scale` band **0.5–0.75** (default ~0.6–0.7); `control_scale = 0` is bit‑exact
to the base model. Values ≥ ~0.9 over‑drive the branch and degrade image quality (haze / halftone).
## Known limitations
- **Body pose‑lock** is strong and generalizes to unseen skeletons at 5k pairs; **head‑direction**
is *usable but scale‑sensitive* (best near `control_scale` 0.5), not rock‑solid.
- Image quality **degrades as `control_scale` rises** — a fundamental authority‑vs‑fidelity
tradeoff, sharper here because the base is frozen, CFG‑free, and few‑step. Not fully removable by
more data.
- Trained at **512²** only. Skeleton convention = OpenPose **body‑18 + head‑direction**
(nose/eyes/ears); **no hands, no dense face**.
## Training data & attribution
Trained on ~5,000 pairs derived from **COCO train2017 `person_keypoints`** ground‑truth annotations
(skeletons rendered in the controlnet_aux OpenPose drawing convention) with **COCO Captions** as the
text conditioning. COCO annotations are **CC BY 4.0**; COCO images are Flickr mixed‑license. This
checkpoint is a derived model artifact for research/feasibility only.
## Provenance
SceneWorks epic 8459 (Krea 2 pose ControlNet), story sc-8460 (S0 spike, verdict = GO).
Experimental — retained solely to bootstrap interface wiring; superseded by the production overlay
once the full‑corpus run (S3) lands.
|