anima
decoder
anima-pixel-decoder / README.md
sorryhyun's picture
Replace the head with realr_1119_2200 (original-image target + LPIPS at the base points, turbo point on v4_longer); previous head stays in old/
ed9da68 verified
|
Raw History Blame Contribute Delete
6.51 kB
---
base_model:
- circlestone-labs/Anima
license: other
license_name: circlestone-labs-non-commercial-license
license_link: LICENSE.md
tags:
- anima
- decoder
---
# Anima Pixel Decoder
A 25M-parameter decoder that reads the last-layer features of the
[Anima](https://huggingface.co/circlestone-labs/Anima) DiT at the final sampling step and
writes RGB pixels directly, in place of the VAE decode. The head and its training follow
[FastFrames](https://arxiv.org/html/2611.23226)' pixel head (a ConvNeXt readout on the diffusion transformer's features).
| | |
|---|---|
| File | `anima_pixel_head.safetensors` (fp32, 95 MB; shape in the file's metadata) |
| Input | the input of the DiT's `final_layer.linear` (2048-dim per token, after the adaLN-modulated LayerNorm) |
| Output | a 16×16 RGB patch per token, in [-1, 1] |
| Architecture | linear projection of the feature + linear embedding of the x_t patch, 2 ConvNeXt, one 2× pixel-shuffle upsample, 1 ConvNeXt, linear to 8×8×3 |
| DiT | `anima-base-v1.0` (28 blocks) |
| Target | base points: the original image, with LPIPS (weight 0.05); turbo point: VAE(x̂0) of the same forward, x̂0 = x_t − σ·v |
## Compatibility
Read the head only where it was trained:
- **Base DiT**: Euler, shift 3, 20 … 50 steps (last-step σ 0.058 … 0.136), read at CFG 4,
Spectrum included (its last step is an actual forward).
- **Turbo**: the 4-step turbo LoRA of
[anima-turbo-4step](https://huggingface.co/sorryhyun/anima-turbo-4step), Euler 4 steps,
shift 3, CFG 1 (last-step σ 0.5). Trained on `anima_turbo_4step_v4_longer.safetensors`.
The previous head is `old/anima_pixel_head.safetensors` (VAE(x̂0) target at every point, turbo
point on that repo's `old/anima_turbo_4step.safetensors`).
## Benchmark
The benchmarks, samples and held-out reads below were measured on the previous head
(`old/anima_pixel_head.safetensors`).
### In ComfyUI
Turbo 4 steps, CFG 1, RTX 5070 Ti, 16 prompts at ~1 MP, one seed
| | KSampler + VAE Decode | KSampler + Anima Pixel Decode |
|---|---|---|
| KSampler | 0.976 s | 0.998 s (with the tap) |
| decode | 151 ms | 7 ms |
| decode peak VRAM above the loaded models | 1.22 GB | 0.21 GB |
| PSNR against VAE Decode | — | 36.9 dB |
| **total per image** | **1.13 s** | **1.00 s (1.12×)** |
ComfyUI keeps the VAE on the card after its first decode, so the VAE column has no move cost.
### In anima_lora
The same turbo setting and prompts through anima_lora's `inference.py` (compiled DiT blocks);
medians. The VAE column is the usual interactive path: the VAE waits on the CPU and goes to the
card for the decode.
| | turbo + VAE | turbo + pixel head |
|---|---|---|
| DiT sampling | 0.97 s | 0.97 s |
| load VAE | 710 ms | — |
| VAE move to the card and back | 35 ms | — |
| decode | 141 ms | 7 ms |
| decode peak VRAM above the DiT | 1.38 GB | 0.24 GB |
| **total per image (DiT + decode)** | **1.86 s** | **0.98 s** |
## Samples
Turbo 4 steps, CFG 1, seed 42, 1024×1024, in ComfyUI (the workflow above); left VAE Decode,
right Anima Pixel Decode, decoded from the same latent. Prompt: `masterpiece, best quality,
score_7, safe. An anime girl wearing a black tank-top and denim shorts is standing outdoors.
She's holding a rectangular sign out in front of her that reads "ANIMA". She's looking at the
viewer with a smile. The background features some trees and blue sky with clouds.`; the
monochrome pair adds `monochrome, comic` after `safe`. Crops: 384×384 at native resolution,
x 192–576, y 512–896.
| | VAE Decode | pixel head |
|---|---|---|
| colour| ![colour, VAE](samples/color_vae.png) | ![colour, pixel head](samples/color_head.png) |
| crop | ![colour crop, VAE](samples/color_vae_crop.png) | ![colour crop, pixel head](samples/color_head_crop.png) |
| monochrome | ![mono, VAE](samples/mono_vae.png) | ![mono, pixel head](samples/mono_head.png) |
| crop | ![mono crop, VAE](samples/mono_vae_crop.png) | ![mono crop, pixel head](samples/mono_head_crop.png) |
On the monochrome crop the VAE keeps the screentone dots on the shorts; the head renders them as
flat grey (see Known limits).
## Held-out reads
Held-out reads during training (dB, colour / mono):
| decode point | colour | mono |
|---|---|---|
| turbo 4 steps | 35.90 | 29.01 |
| base 20 steps | 33.71 | 26.58 |
| base 28 steps | 33.83 | 26.38 |
| base 50 steps | 33.49 | 25.48 |
| base 28 + Spectrum | 34.37 | 26.00 |
## Known limits
- Screentone dots are smoothed into soft grey shading; textured regions (screentone,
dense hatching) come out about 1 – 2.5 levels darker than the VAE in linear light.
The lowest per-image PSNRs are manga pages and sketches (turbo 28.7 – 30.9 dB).
- Speed lines and footnote-size glyphs are slightly soft under zoom.
- One trained head, one seed; no run spread.
## Use
- ComfyUI: the [ComfyUI-Anima-PixelDecoder](https://github.com/sorryhyun/ComfyUI-Anima-PixelDecoder)
node pack (ComfyUI Manager: "Anima Pixel Decoder"). Put this file in `models/pixel_heads/`,
add **Anima Pixel Head Tap** on the MODEL before the KSampler (stock or Spectrum) and decode
the sampler's LATENT with **Anima Pixel Decode** instead of VAE Decode. Run the DiT on the
fp32 stream (Anima Adapter Loader, `stream = fp32`).
- [anima_lora](https://github.com/sorryhyun/anima_lora): `inference.py --decoder pixel_head`
(`make test PIXEL=1`, `make test-turbo PIXEL=1`); the module, loader and feature tap are in
`library/anima/pixel_head.py`.
- ×2 (test): `anima_pixel_head_x2_preview.safetensors` decodes and upscales 2× in one
readout (Anima Pixel Decode `scale = 2`, `inference.py --pixel_scale 2`). A test file; it will
be replaced.
## License
The weights in this repository are a **Derivative** of
[Anima](https://huggingface.co/circlestone-labs/Anima) and are distributed under the
**[CircleStone Labs Non-Commercial License](LICENSE.md)** (see [NOTICE.md](NOTICE.md)). Any
rights to use them are granted directly by CircleStone Labs LLC under that license.
- The model/weights may only be used for non-commercial purposes. Generated images (Outputs)
may be used commercially.
- Anima is itself a Derivative Model of NVIDIA Cosmos-Predict2-2B-Text2Image and is subject
to the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/)
insofar as it applies to Derivative Models.
- This is not an official CircleStone Labs product and is not endorsed by CircleStone Labs.
Built on NVIDIA Cosmos.