xocialize/refcontrol-FLUX.2-klein-4B-pose-lora

RefControl Pose LoRA — FLUX.2‑klein‑4B

A RefControl LoRA for FLUX.2‑klein‑base‑4B that re‑poses a character: give it a pose skeleton and a reference image of a subject, and it renders that subject in the skeleton's pose. Pose control and identity/reference are carried by the two control inputs of klein's edit path — this is not a separate ControlNet, just a low‑rank adapter on the DiT.

A generated character (T‑pose reference, image 2) re‑posed by a skeleton (image 1). Fully synthetic — no training data shown.

• image 1 — an OpenPose‑style skeleton (the target pose)
• image 2 — a reference image of the subject (identity / appearance)
• prompt — apply pose from image 1 with reference from image 2
• trigger word — refcontrol

Files

File	Notes
refcontrol-pose-klein-4b.safetensors	rank‑32 LoRA, ~88 MB, 160 tensors. BFL‑fused key layout (diffusion_model.double_blocks.N.*, single_blocks.*), 60 double + 40 single target modules. Loads at bf16 and int4.

Usage

Swift (MLX, Apple Silicon)

Via the open klein package xocialize/flux2-klein-swift (v0.6.0+):

klein-cli \
  --snapshot <FLUX.2-klein-4B> \
  --edit skeleton.png,reference.png \
  --lora refcontrol-pose-klein-4b.safetensors \
  --prompt "apply pose from image 1 with reference from image 2" \
  --size 1024 --steps 4 --seed 42 --out posed.png

Or through MLXEngine by pointing KleinConfiguration.loraPath at this file.

Tips

• Control order is load‑bearing: image 1 = skeleton, image 2 = reference. Swapping them swaps the roles.
• The skeleton must be an OpenPose‑style COCO‑18 render (colored limb sticks + joint disks on black). A self‑consistent extractor/renderer (Apple Vision → OpenPose) is available in the training tooling.
• A clean, plain‑background reference yields a clean output — the model preserves the reference's context, so a busy background carries through. For character sheets, use a white‑background reference.

Training

• Base: black-forest-labs/FLUX.2-klein-base-4B (Apache‑2.0), flux2_klein_4b arch, qfloat8.
• Recipe: ai‑toolkit, LoRA rank 32 / alpha 32, 4000 steps, lr 1e‑4, AdamW8bit, flowmatch, resolutions 512/768/1024. This checkpoint = the best of the run by a held‑out‑identity A/B (step 3500; later steps regressed slightly).
• Data: ~1,800 (skeleton, reference, target) triples across ~110 distinct human‑motion identities and ~25 activity types (dance, martial arts, gymnastics, fitness, everyday movement). Frames sourced from Wikimedia Commons (CC‑BY / CC‑BY‑SA) and Pexels (Pexels License). Real‑human data; generalizes to stylized subjects via the base model.

Attribution

Training frames are derived from Wikimedia Commons contributors (CC‑BY / CC‑BY‑SA) and Pexels videographers. The dataset itself (the source frames) is not redistributed here — only the trained LoRA weights. Thanks to Black Forest Labs for FLUX.2‑klein and to ostris/ai‑toolkit.

License

Apache‑2.0, matching the FLUX.2‑klein‑base‑4B base model. The weights are a derivative of the Apache base; the training media are credited above and not included.