--- license: other license_name: circlestone-labs-non-commercial-license license_link: LICENSE base_model: circlestone-labs/Anima library_name: diffusers tags: - control-lora - pose - dwpose - anima - cosmos-predict2 - comfyui - text-to-image pipeline_tag: text-to-image --- # Anima Control — Pose (Preview-2) > ⚠️ **Preview-2 — still experimental.** Better than Preview-1, but not finished. It will still > miss poses and produce deformed bodies, fused hands, and similar artifacts. Treat it as a > work-in-progress preview, not a production tool. **Non-commercial use only** (inherits the Anima > base model license). Behaviour and weights may still change. A native **pose control adapter** for the Anima v1.0 image model: condition generation on a skeleton pose map so the subject follows a target pose. Preview-2 is the same idea as Preview-1, trained at higher resolutions on a larger corpus, and shipped with a friendlier ComfyUI node. ## What changed since Preview-1 - **Multi-resolution: 512, 768 and 1024.** Preview-1 was 512-only; this runs at all three, and the bodies hold together much better at the larger sizes. 1024 looks best. - **Larger training corpus** than Preview-1's ~3,900 examples, which shows up as cleaner anatomy and steadier poses. - **Better pose-following at every resolution**, with the biggest gains at 768 and 1024. - **New ComfyUI node, "Anima Pose Control."** Drop in a reference photo; it detects the pose for you and renders the skeleton, and you can pick how the skeleton is drawn (thin, thick, puppet, heatmap, or with hands/face stripped). The plain thin skeleton is still the best default — it's the only one the model trained on — but a different render occasionally lands a stubborn pose when the seed alone won't. ## Method Unchanged from Preview-1: a channel-concat control-LoRA on the frozen Anima DiT. **Conditioning.** The base VAE encodes the skeleton pose map into a control latent, the same latent space as the noisy image, so the control stays spatially aligned with the generation. **Fusion (`ControlEmbedder` + `ControlInitialLayer`).** A zero-initialized `ControlEmbedder` produces control tokens that are **added to the frozen base patch-embed output**. Zero-init means training starts as an exact no-op (output == base) and the control contribution grows only as it earns loss, so at `strength = 0` the adapter is exactly the base model. **Trainable parameters.** The `ControlEmbedder` plus a rank-16 low-rank adapter on the transformer blocks. The base transformer, text encoder, and VAE stay frozen. ``` skeleton ─▶ VAE ─▶ control latent ─┐ ▼ (+ zero-init ControlEmbedder) noisy latent ─▶ patch-embed ─▶ [ControlInitialLayer] ─▶ Block×N (+ rank-16 LoRA) ─▶ output ``` ## Training **Data.** (image, skeleton, caption) triples generated by Anima from a broad prompt distribution; skeletons rendered from each image's detected keypoints (DWPose, COCO-WholeBody, black background). Preview-2 uses a substantially larger corpus than Preview-1. | Setting | Value | |---|---| | Resolution | **512 + 768 + 1024**, aspect-ratio bucketed | | Adapter rank | 16 | | Learning rate | 1e-4 | | Epochs | 8 | | Control dropout | 0.1 | | Precision | bf16 | > Final training loss ≈ 0.11 (denoising MSE, mean over the final 400 steps). ## Results Measured on held-out full-body poses (fresh generations, not seen in training). Pose agreement is body-PCK@0.1: re-detect keypoints on each output, compare to the target skeleton. ![Pose control across skeleton styles](samples/grid.png) *Each grid: the BASE column shows the reference and the no-control generation (same prompt, different seed — it ignores the pose); the remaining columns show the skeleton, drawn in each style, over the pose-controlled output. Control follows the pose; no-control doesn't.* | body-PCK@0.1 | control off | control on | |---|---|---| | 512 | ~0.33 | **~0.59** | | 768 | ~0.38 | **~0.67** | | 1024 | ~0.37 | **~0.83** | Preview-1 reached ~0.59 at 512. Preview-2 matches that at 512 and pulls clearly ahead at 768 and 1024 — the gains grow with resolution. **What's still off (honest):** - It doesn't always follow the skeleton, even a clean, correct one. - A thin stick figure on black isn't how anime is drawn, so the model only half-reads it; the worst artifacts (fused hands, mush) cluster where the skeleton is busiest. - Dynamic poses — running, jumping, sitting — are the least reliable. - On short or vague prompts the style flattens toward a samey default; a richer prompt fixes it. ## Usage (ComfyUI) Preview-2 uses two small custom nodes: **Anima Control Apply** (`AnimaControlApply`) applies the adapter, and **Anima Pose Control** (`AnimaPoseControl`) detects the pose from a photo and renders the skeleton for you. ### Install 1. Download `anima_pose_preview2.safetensors` into `ComfyUI/models/loras/`. 2. Copy both folders from `comfyui/` in this repo into `ComfyUI/custom_nodes/`: `anima_control_lora/` and `ComfyUI-anima-pose-control/`. Restart ComfyUI. ComfyUI-Manager installs the second node's `requirements.txt` automatically; otherwise: `pip install -r ComfyUI-anima-pose-control/requirements.txt` (rtmlib, opencv-python, onnxruntime, numpy; torch and Pillow come with ComfyUI). 3. Load a workflow from the menu. `pose_control_demo.json` is the easiest — control vs no-control side by side. Also included: `pose_control.json` (simple), `pose_control_edit.json` (single node, pick the skeleton style), `pose_control_compare.json` (one pose across every style at once). `anima_pose_preview2.safetensors` holds both the low-rank adapter (`lora.*` keys) and the control embedder (`control_embedder.*` keys); `LoraLoaderModelOnly` reads the first set and Anima Control Apply reads the second, both pointing at the same file. **Strength.** `0.0` is the base model with no control; `1.0` follows the skeleton (range 0–2). Higher tracks the pose more closely but can cost some image quality. ### Pose detector (first run) The Anima Pose Control node needs a pose detector (rtmlib). On first use it downloads two ONNX files (~316 MB) from this repo's `detector/` folder and caches them under `~/.cache/rtmlib/hub/checkpoints/`; after that it runs offline. If you see `urllib ... getaddrinfo failed`, your machine couldn't reach the download host — download `detector/yolox_m_8xb8-300e_humanart-c2c7a14a.onnx` and `detector/rtmw-dw-x-l_simcc-cocktail14_270e-256x192_20231122.onnx` from the Files tab by hand and drop both into that cache folder (create it if missing), then restart ComfyUI. ## Roadmap **Preview-3** targets the headline weakness: the model only half-reads the skeleton. The thin lines-on-black signal seems foreign to it, so a representation bake-off is testing other renders (thicker, puppet/segmentation, depth-mannequin) to find what Anima follows best, plus the smaller fixes (default strength, dropping noisy hand points, more varied captions). The other half is the detector: the one used here was built for photos, not anime, so on art it produces noisy skeletons. Preview-3 will most likely wait on a purpose-built **anime pose detector** ("DWPose for anime"), then retrain on the winning representation with a lot more dynamic-pose data. **Version 1.0** comes after Preview-3, if it lands clean — the first non-preview release, trained on a good deal more data again. Pose is the first component on a shared control harness for Anima; planned siblings are image-prompt (IP-Adapter) and face-identity conditioning. ## License These weights are a derivative of the Anima base model ([`circlestone-labs/Anima`](https://huggingface.co/circlestone-labs/Anima)) and inherit its terms: the **CircleStone Labs Non-Commercial License**, and — because Anima is itself a derivative of Cosmos-Predict2 — the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). The **model weights are for non-commercial use only.** Generated images (outputs) are not restricted by these terms and may be used commercially. See the bundled [`LICENSE`](./LICENSE) for the full text. ## Support Building these models means mining and labeling a lot of images and renting GPUs to train on them. If they're useful to you and you want to chip in, it's appreciated and never expected: https://ko-fi.com/claquasse ## Citation ```bibtex @misc{anima_control_pose_preview2, title = {Anima Control --- Pose (Preview-2)}, author = {Claquasse}, year = {2026}, note = {Preview-2 multi-resolution pose control adapter for Anima v1.0}, howpublished = {\url{https://huggingface.co/Claquasse/Anima-Control-Pose}} } ``` Built on Anima (CircleStone Labs), the Cosmos-Predict2 transformer architecture, and the diffusion-pipe training framework.