---
license: apache-2.0
base_model: zai-org/SCAIL-2
tags:
- mlx
- video
- character-animation
- image-to-video
- wan2.1
- work-in-progress
library_name: mlx
---

# SCAIL-2 — MLX (work in progress)

> ## ⚠️ WIP — pre-release conversion, expect changes
>
> These are Apple-MLX conversions of [zai-org/SCAIL-2](https://huggingface.co/zai-org/SCAIL-2)
> for the [xocialize/scail-2-mlx](https://github.com/xocialize/scail-2-mlx) port,
> published from our own namespace while the port is under active development.
> File formats, key layouts, and dtypes **may change without notice**. Quantized
> (q8/q4) variants, golden end-to-end validation against the PyTorch reference,
> and an mlx-community release are planned but not done. Use for
> experimentation, not production.

**SCAIL-2** (Zhipu AI, [arXiv 2512.05905](https://arxiv.org/abs/2512.05905)) is an
end-to-end controlled character-animation model: a reference character image +
a driving video → the character performing that motion. Cross-identity
replacement, multi-character scenes, and animal driving, with no intermediate
pose representations required. The backbone is a Wan2.1-I2V-14B fork with a
3-segment (reference / video / pose) RoPE design and dual mask conditioning.

## Files

| file | component | dtype | size |
|---|---|---|---|
| `dit.safetensors` | SCAIL2 DiT (14B, Wan2.1-I2V fork) | bf16 | 33 GB |
| `umt5.safetensors` | umT5-XXL text encoder | bf16 | 11 GB |
| `clip.safetensors` | open-clip xlm-roberta ViT-H/14 visual tower | fp16 | 1.2 GB |
| `vae.safetensors` | Wan2.1 VAE (16-ch) | fp32 | 0.5 GB |

Keys follow the [scail-2-mlx](https://github.com/xocialize/scail-2-mlx) module
tree (MLX `nn.Sequential` uses `.layers.N`; conv weights are NDHWC/NHWC).
Tokenizer: use `google/umt5-xxl` (or the `umt5-xxl/` directory bundled with the
original checkpoint).

## Usage

```bash
git clone https://github.com/xocialize/scail-2-mlx && cd scail-2-mlx
uv venv --python 3.12 .venv
uv pip install -e refs/mlx-video -e .
hf download xocialize/SCAIL-2-bf16 --local-dir weights/mlx

.venv/bin/python scripts/generate.py \
  --weights-dir weights/mlx \
  --image ref.jpg --mask-image ref_mask.jpg \
  --pose driving.mp4 --mask-video driving_mask.mp4 \
  --prompt "the girl is dancing" \
  --target-h 480 --target-w 832 --save-file out.mp4
```

Requires Apple Silicon with ≥ 64 GB unified memory at bf16 (active ~34 GB,
peak ~47 GB at 832×480×65 frames; ~3.7 min/step on an M5 Max — perf work
ongoing). Driving-input preprocessing (masks / pose renders) comes from the
upstream [SCAIL-Pose](https://github.com/zai-org/SCAIL-Pose) toolchain.

## Conversion provenance & fidelity

Converted by [`recipes/convert_scail2.py`](https://github.com/xocialize/scail-2-mlx/blob/main/recipes/convert_scail2.py)
from the original FSDP checkpoint via upstream `convert.py` key remapping
(1307/1307 strict key match). Component-level parity vs the PyTorch reference
(fp32, CPU): CLIP visual max_abs 2.7e-4 on real weights; chunked causal VAE
decode < 5e-4 per frame (canonical 1+(T−1)·4 frame mapping — see
[Blaizzy/mlx-video#38](https://github.com/Blaizzy/mlx-video/pull/38)); DiT
forward parity-locked at fp32 on the CPU oracle. End-to-end golden comparison
against the PyTorch pipeline is **pending**.

## License

Weights: converted from `zai-org/SCAIL-2` (model card: MIT; source repository:
Apache-2.0 — this card is marked Apache-2.0, the stricter of the two, pending
upstream clarification). Conversion code: Apache-2.0. Derived from SCAIL-2
(Zhipu AI), Wan2.1 (Alibaba), open-clip.