README.md

---
license: other
license_name: circlestone-labs-non-commercial-license
license_link: LICENSE
base_model: circlestone-labs/Anima
library_name: diffusers
tags:
  - control-lora
  - pose
  - dwpose
  - anima
  - cosmos-predict2
  - comfyui
  - text-to-image
pipeline_tag: text-to-image
---

# Anima Control — Pose (Preview-2)

> ⚠️ **Preview-2 — still experimental.** Better than Preview-1, but not finished. It will still
> miss poses and produce deformed bodies, fused hands, and similar artifacts. Treat it as a
> work-in-progress preview, not a production tool. **Non-commercial use only** (inherits the Anima
> base model license). Behaviour and weights may still change.

A native **pose control adapter** for the Anima v1.0 image model: condition generation on a
skeleton pose map so the subject follows a target pose. Preview-2 is the same idea as Preview-1,
trained at higher resolutions on a larger corpus, and shipped with a friendlier ComfyUI node.

## What changed since Preview-1

- **Multi-resolution: 512, 768 and 1024.** Preview-1 was 512-only; this runs at all three, and the
  bodies hold together much better at the larger sizes. 1024 looks best.
- **Larger training corpus** than Preview-1's ~3,900 examples, which shows up as cleaner anatomy and
  steadier poses.
- **Better pose-following at every resolution**, with the biggest gains at 768 and 1024.
- **New ComfyUI node, "Anima Pose Control."** Drop in a reference photo; it detects the pose for you
  and renders the skeleton, and you can pick how the skeleton is drawn (thin, thick, puppet, heatmap,
  or with hands/face stripped). The plain thin skeleton is still the best default — it's the only one
  the model trained on — but a different render occasionally lands a stubborn pose when the seed
  alone won't.

## Method

Unchanged from Preview-1: a channel-concat control-LoRA on the frozen Anima DiT.

**Conditioning.** The base VAE encodes the skeleton pose map into a control latent, the same latent
space as the noisy image, so the control stays spatially aligned with the generation.

**Fusion (`ControlEmbedder` + `ControlInitialLayer`).** A zero-initialized `ControlEmbedder` produces
control tokens that are **added to the frozen base patch-embed output**. Zero-init means training
starts as an exact no-op (output == base) and the control contribution grows only as it earns loss,
so at `strength = 0` the adapter is exactly the base model.

**Trainable parameters.** The `ControlEmbedder` plus a rank-16 low-rank adapter on the transformer
blocks. The base transformer, text encoder, and VAE stay frozen.

```
skeleton ─▶ VAE ─▶ control latent ─┐
                                   ▼  (+ zero-init ControlEmbedder)
noisy latent ─▶ patch-embed ─▶ [ControlInitialLayer] ─▶ Block×N (+ rank-16 LoRA) ─▶ output
```

## Training

**Data.** (image, skeleton, caption) triples generated by Anima from a broad prompt distribution;
skeletons rendered from each image's detected keypoints (DWPose, COCO-WholeBody, black background).
Preview-2 uses a substantially larger corpus than Preview-1.

| Setting | Value |
|---|---|
| Resolution | **512 + 768 + 1024**, aspect-ratio bucketed |
| Adapter rank | 16 |
| Learning rate | 1e-4 |
| Epochs | 8 |
| Control dropout | 0.1 |
| Precision | bf16 |

> Final training loss ≈ 0.11 (denoising MSE, mean over the final 400 steps).

## Results

Measured on held-out full-body poses (fresh generations, not seen in training). Pose agreement is
body-PCK@0.1: re-detect keypoints on each output, compare to the target skeleton.

![Pose control across skeleton styles](samples/grid.png)

*Each grid: the BASE column shows the reference and the no-control generation (same prompt, different
seed — it ignores the pose); the remaining columns show the skeleton, drawn in each style, over the
pose-controlled output. Control follows the pose; no-control doesn't.*

| body-PCK@0.1 | control off | control on |
|---|---|---|
| 512 | ~0.33 | **~0.59** |
| 768 | ~0.38 | **~0.67** |
| 1024 | ~0.37 | **~0.83** |

Preview-1 reached ~0.59 at 512. Preview-2 matches that at 512 and pulls clearly ahead at 768 and
1024 — the gains grow with resolution.

**What's still off (honest):**
- It doesn't always follow the skeleton, even a clean, correct one.
- A thin stick figure on black isn't how anime is drawn, so the model only half-reads it; the worst
  artifacts (fused hands, mush) cluster where the skeleton is busiest.
- Dynamic poses — running, jumping, sitting — are the least reliable.
- On short or vague prompts the style flattens toward a samey default; a richer prompt fixes it.

## Usage (ComfyUI)

Preview-2 uses two small custom nodes: **Anima Control Apply** (`AnimaControlApply`) applies the
adapter, and **Anima Pose Control** (`AnimaPoseControl`) detects the pose from a photo and renders
the skeleton for you.

### Install

1. Download `anima_pose_preview2.safetensors` into `ComfyUI/models/loras/`.
2. Copy both folders from `comfyui/` in this repo into `ComfyUI/custom_nodes/`:
   `anima_control_lora/` and `ComfyUI-anima-pose-control/`. Restart ComfyUI. ComfyUI-Manager installs
   the second node's `requirements.txt` automatically; otherwise:
   `pip install -r ComfyUI-anima-pose-control/requirements.txt` (rtmlib, opencv-python, onnxruntime,
   numpy; torch and Pillow come with ComfyUI).
3. Load a workflow from the menu. `pose_control_demo.json` is the easiest — control vs no-control side
   by side. Also included: `pose_control.json` (simple), `pose_control_edit.json` (single node, pick
   the skeleton style), `pose_control_compare.json` (one pose across every style at once).

`anima_pose_preview2.safetensors` holds both the low-rank adapter (`lora.*` keys) and the control
embedder (`control_embedder.*` keys); `LoraLoaderModelOnly` reads the first set and Anima Control
Apply reads the second, both pointing at the same file.

**Strength.** `0.0` is the base model with no control; `1.0` follows the skeleton (range 0–2). Higher
tracks the pose more closely but can cost some image quality.

### Pose detector (first run)

The Anima Pose Control node needs a pose detector (rtmlib). On first use it downloads two ONNX
files (~316 MB) from this repo's `detector/` folder and caches them under
`~/.cache/rtmlib/hub/checkpoints/`; after that it runs offline. If you see
`urllib ... getaddrinfo failed`, your machine couldn't reach the download host — download
`detector/yolox_m_8xb8-300e_humanart-c2c7a14a.onnx` and
`detector/rtmw-dw-x-l_simcc-cocktail14_270e-256x192_20231122.onnx` from the Files tab by hand and
drop both into that cache folder (create it if missing), then restart ComfyUI.

## Roadmap

**Preview-3** targets the headline weakness: the model only half-reads the skeleton. The thin
lines-on-black signal seems foreign to it, so a representation bake-off is testing other renders
(thicker, puppet/segmentation, depth-mannequin) to find what Anima follows best, plus the smaller
fixes (default strength, dropping noisy hand points, more varied captions). The other half is the
detector: the one used here was built for photos, not anime, so on art it produces noisy skeletons.
Preview-3 will most likely wait on a purpose-built **anime pose detector** ("DWPose for anime"),
then retrain on the winning representation with a lot more dynamic-pose data.

**Version 1.0** comes after Preview-3, if it lands clean — the first non-preview release, trained on
a good deal more data again.

Pose is the first component on a shared control harness for Anima; planned siblings are image-prompt
(IP-Adapter) and face-identity conditioning.

## License

These weights are a derivative of the Anima base model
([`circlestone-labs/Anima`](https://huggingface.co/circlestone-labs/Anima)) and inherit its terms:
the **CircleStone Labs Non-Commercial License**, and — because Anima is itself a derivative of
Cosmos-Predict2 — the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).

The **model weights are for non-commercial use only.** Generated images (outputs) are not restricted
by these terms and may be used commercially. See the bundled [`LICENSE`](./LICENSE) for the full text.

## Support

Building these models means mining and labeling a lot of images and renting GPUs to train on them. If
they're useful to you and you want to chip in, it's appreciated and never expected:
https://ko-fi.com/claquasse

## Citation

```bibtex
@misc{anima_control_pose_preview2,
  title  = {Anima Control --- Pose (Preview-2)},
  author = {Claquasse},
  year   = {2026},
  note   = {Preview-2 multi-resolution pose control adapter for Anima v1.0},
  howpublished = {\url{https://huggingface.co/Claquasse/Anima-Control-Pose}}
}
```

Built on Anima (CircleStone Labs), the Cosmos-Predict2 transformer architecture, and the
diffusion-pipe training framework.