| --- |
| license: apache-2.0 |
| base_model: zai-org/SCAIL-2 |
| tags: |
| - mlx |
| - video |
| - character-animation |
| - image-to-video |
| - wan2.1 |
| - work-in-progress |
| library_name: mlx |
| --- |
| |
| # SCAIL-2 — MLX (work in progress) |
|
|
| > ## ⚠️ WIP — pre-release conversion, expect changes |
| > |
| > These are Apple-MLX conversions of [zai-org/SCAIL-2](https://huggingface.co/zai-org/SCAIL-2) |
| > for the [xocialize/scail-2-mlx](https://github.com/xocialize/scail-2-mlx) port, |
| > published from our own namespace while the port is under active development. |
| > File formats, key layouts, and dtypes **may change without notice**. Quantized |
| > (q8/q4) variants, golden end-to-end validation against the PyTorch reference, |
| > and an mlx-community release are planned but not done. Use for |
| > experimentation, not production. |
|
|
| **SCAIL-2** (Zhipu AI, [arXiv 2512.05905](https://arxiv.org/abs/2512.05905)) is an |
| end-to-end controlled character-animation model: a reference character image + |
| a driving video → the character performing that motion. Cross-identity |
| replacement, multi-character scenes, and animal driving, with no intermediate |
| pose representations required. The backbone is a Wan2.1-I2V-14B fork with a |
| 3-segment (reference / video / pose) RoPE design and dual mask conditioning. |
|
|
| ## Files |
|
|
| | file | component | dtype | size | |
| |---|---|---|---| |
| | `dit.safetensors` | SCAIL2 DiT (14B, Wan2.1-I2V fork) | bf16 | 33 GB | |
| | `umt5.safetensors` | umT5-XXL text encoder | bf16 | 11 GB | |
| | `clip.safetensors` | open-clip xlm-roberta ViT-H/14 visual tower | fp16 | 1.2 GB | |
| | `vae.safetensors` | Wan2.1 VAE (16-ch) | fp32 | 0.5 GB | |
|
|
| Keys follow the [scail-2-mlx](https://github.com/xocialize/scail-2-mlx) module |
| tree (MLX `nn.Sequential` uses `.layers.N`; conv weights are NDHWC/NHWC). |
| Tokenizer: use `google/umt5-xxl` (or the `umt5-xxl/` directory bundled with the |
| original checkpoint). |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/xocialize/scail-2-mlx && cd scail-2-mlx |
| uv venv --python 3.12 .venv |
| uv pip install -e refs/mlx-video -e . |
| hf download xocialize/SCAIL-2-bf16 --local-dir weights/mlx |
| |
| .venv/bin/python scripts/generate.py \ |
| --weights-dir weights/mlx \ |
| --image ref.jpg --mask-image ref_mask.jpg \ |
| --pose driving.mp4 --mask-video driving_mask.mp4 \ |
| --prompt "the girl is dancing" \ |
| --target-h 480 --target-w 832 --save-file out.mp4 |
| ``` |
|
|
| Requires Apple Silicon with ≥ 64 GB unified memory at bf16 (active ~34 GB, |
| peak ~47 GB at 832×480×65 frames; ~3.7 min/step on an M5 Max — perf work |
| ongoing). Driving-input preprocessing (masks / pose renders) comes from the |
| upstream [SCAIL-Pose](https://github.com/zai-org/SCAIL-Pose) toolchain. |
|
|
| ## Conversion provenance & fidelity |
|
|
| Converted by [`recipes/convert_scail2.py`](https://github.com/xocialize/scail-2-mlx/blob/main/recipes/convert_scail2.py) |
| from the original FSDP checkpoint via upstream `convert.py` key remapping |
| (1307/1307 strict key match). Component-level parity vs the PyTorch reference |
| (fp32, CPU): CLIP visual max_abs 2.7e-4 on real weights; chunked causal VAE |
| decode < 5e-4 per frame (canonical 1+(T−1)·4 frame mapping — see |
| [Blaizzy/mlx-video#38](https://github.com/Blaizzy/mlx-video/pull/38)); DiT |
| forward parity-locked at fp32 on the CPU oracle. End-to-end golden comparison |
| against the PyTorch pipeline is **pending**. |
| |
| ## License |
| |
| Weights: converted from `zai-org/SCAIL-2` (model card: MIT; source repository: |
| Apache-2.0 — this card is marked Apache-2.0, the stricter of the two, pending |
| upstream clarification). Conversion code: Apache-2.0. Derived from SCAIL-2 |
| (Zhipu AI), Wan2.1 (Alibaba), open-clip. |
| |
