File size: 3,564 Bytes
c5331df fe54a4b c5331df | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: apache-2.0
base_model: zai-org/SCAIL-2
tags:
- mlx
- video
- character-animation
- image-to-video
- wan2.1
- work-in-progress
library_name: mlx
---
# SCAIL-2 — MLX (work in progress)
> ## ⚠️ WIP — pre-release conversion, expect changes
>
> These are Apple-MLX conversions of [zai-org/SCAIL-2](https://huggingface.co/zai-org/SCAIL-2)
> for the [xocialize/scail-2-mlx](https://github.com/xocialize/scail-2-mlx) port,
> published from our own namespace while the port is under active development.
> File formats, key layouts, and dtypes **may change without notice**. Quantized
> (q8/q4) variants, golden end-to-end validation against the PyTorch reference,
> and an mlx-community release are planned but not done. Use for
> experimentation, not production.
**SCAIL-2** (Zhipu AI, [arXiv 2512.05905](https://arxiv.org/abs/2512.05905)) is an
end-to-end controlled character-animation model: a reference character image +
a driving video → the character performing that motion. Cross-identity
replacement, multi-character scenes, and animal driving, with no intermediate
pose representations required. The backbone is a Wan2.1-I2V-14B fork with a
3-segment (reference / video / pose) RoPE design and dual mask conditioning.
## Files
| file | component | dtype | size |
|---|---|---|---|
| `dit.safetensors` | SCAIL2 DiT (14B, Wan2.1-I2V fork) | bf16 | 33 GB |
| `umt5.safetensors` | umT5-XXL text encoder | bf16 | 11 GB |
| `clip.safetensors` | open-clip xlm-roberta ViT-H/14 visual tower | fp16 | 1.2 GB |
| `vae.safetensors` | Wan2.1 VAE (16-ch) | fp32 | 0.5 GB |
Keys follow the [scail-2-mlx](https://github.com/xocialize/scail-2-mlx) module
tree (MLX `nn.Sequential` uses `.layers.N`; conv weights are NDHWC/NHWC).
Tokenizer: use `google/umt5-xxl` (or the `umt5-xxl/` directory bundled with the
original checkpoint).
## Usage
```bash
git clone https://github.com/xocialize/scail-2-mlx && cd scail-2-mlx
uv venv --python 3.12 .venv
uv pip install -e refs/mlx-video -e .
hf download xocialize/SCAIL-2-bf16 --local-dir weights/mlx
.venv/bin/python scripts/generate.py \
--weights-dir weights/mlx \
--image ref.jpg --mask-image ref_mask.jpg \
--pose driving.mp4 --mask-video driving_mask.mp4 \
--prompt "the girl is dancing" \
--target-h 480 --target-w 832 --save-file out.mp4
```
Requires Apple Silicon with ≥ 64 GB unified memory at bf16 (active ~34 GB,
peak ~47 GB at 832×480×65 frames; ~3.7 min/step on an M5 Max — perf work
ongoing). Driving-input preprocessing (masks / pose renders) comes from the
upstream [SCAIL-Pose](https://github.com/zai-org/SCAIL-Pose) toolchain.
## Conversion provenance & fidelity
Converted by [`recipes/convert_scail2.py`](https://github.com/xocialize/scail-2-mlx/blob/main/recipes/convert_scail2.py)
from the original FSDP checkpoint via upstream `convert.py` key remapping
(1307/1307 strict key match). Component-level parity vs the PyTorch reference
(fp32, CPU): CLIP visual max_abs 2.7e-4 on real weights; chunked causal VAE
decode < 5e-4 per frame (canonical 1+(T−1)·4 frame mapping — see
[Blaizzy/mlx-video#38](https://github.com/Blaizzy/mlx-video/pull/38)); DiT
forward parity-locked at fp32 on the CPU oracle. End-to-end golden comparison
against the PyTorch pipeline is **pending**.
## License
Weights: converted from `zai-org/SCAIL-2` (model card: MIT; source repository:
Apache-2.0 — this card is marked Apache-2.0, the stricter of the two, pending
upstream clarification). Conversion code: Apache-2.0. Derived from SCAIL-2
(Zhipu AI), Wan2.1 (Alibaba), open-clip.
|