Image-to-Video
Diffusers
MLX
i2v
character-animation
video-generation
cross-identity-replacement
pose-driven
diffusion
apple-silicon
Instructions to use SceneWorks/scail2-mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use SceneWorks/scail2-mlx with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image, export_to_video # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("SceneWorks/scail2-mlx", dtype=torch.bfloat16, device_map="cuda") pipe.to("cuda") prompt = "A man with short gray hair plays a red electric guitar." image = load_image( "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/guitar-man.png" ) output = pipe(image=image, prompt=prompt).frames[0] export_to_video(output, "output.mp4") - MLX
How to use SceneWorks/scail2-mlx with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download SceneWorks/scail2-mlx --local-dir scail2-mlx
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
pipeline_tag: image-to-video
|
| 4 |
+
library_name: diffusers
|
| 5 |
+
tags:
|
| 6 |
+
- character-animation
|
| 7 |
+
- video-generation
|
| 8 |
+
- cross-identity-replacement
|
| 9 |
+
- pose-driven
|
| 10 |
+
- diffusion
|
| 11 |
+
- mlx
|
| 12 |
+
- apple-silicon
|
| 13 |
+
base_model: zai-org/SCAIL-2
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# SceneWorks/scail2-mlx
|
| 17 |
+
|
| 18 |
+
**Turnkey, SceneWorks-converted weights of [zai-org/SCAIL-2](https://huggingface.co/zai-org/SCAIL-2)** — an end-to-end controlled **character-animation / motion-transfer** video model — packaged for **native Apple-Silicon (MLX)** inference inside SceneWorks. This is **not** an original model; it is a format/dtype repackaging of the upstream release for first-class macOS use (no PyTorch at runtime).
|
| 19 |
+
|
| 20 |
+
> Capabilities (from upstream): character animation from a reference image + driving video, **cross-identity character replacement**, zero-shot animal-driving, end-to-end *and* pose-rendered driving, and (experimental) multi-reference. Image output is `num_frames == 1`.
|
| 21 |
+
|
| 22 |
+
## What changed vs. upstream
|
| 23 |
+
- The main DiT (`model/1/fsdp2_rank_0000_checkpoint.pt`, an FSDP2/SAT checkpoint) was key-remapped to the `SCAIL2Model` parameter naming using the upstream `convert.py` contract (fused `query_key_value`→`q`/`k`/`v`, `key_value`→`k`/`v`, `clip_feature_key_value_list`→`k_img`/`v_img`) and cast **fp32 → bf16**, then saved as a single safetensors. The remap is bit-faithful (validated: 987 source keys → 1307 model keys, an exact key+shape match against `SCAIL2Model.from_config(config-14b.json)`).
|
| 24 |
+
- The text encoder (UMT5-XXL), VAE (Wan2.1), and image encoder (open-CLIP XLM-RoBERTa ViT-H/14) are the **stock upstream files**, included here so the snapshot is self-contained.
|
| 25 |
+
|
| 26 |
+
## Contents
|
| 27 |
+
| file | source | notes |
|
| 28 |
+
|---|---|---|
|
| 29 |
+
| `dit.safetensors` | converted | SCAIL-2 14B DiT, **bf16**, 1307 tensors (~31 GB) |
|
| 30 |
+
| `config.json` | upstream `configs/config-14b.json` | `model_type: i2v`, `dim 5120`, `ffn 13824`, `40` layers/heads, `in_dim 20`, `mask_dim 28`, `out_dim 16` |
|
| 31 |
+
| `Wan2.1_VAE.pth` | upstream, stock | z16 VAE, stride (4,8,8) |
|
| 32 |
+
| `umt5-xxl/` | upstream, stock | UMT5-XXL encoder (bf16) + tokenizer |
|
| 33 |
+
| `models_clip_open-clip-xlm-roberta-large-vit-huge-14-onlyvisual.pth` | upstream, stock | reference-image visual encoder (1280-dim) |
|
| 34 |
+
| `bias-aware-dpo-lora.pt` | upstream, stock | optional Bias-Aware DPO refinement LoRA |
|
| 35 |
+
|
| 36 |
+
## Architecture (summary)
|
| 37 |
+
Wan2.1-14B **I2V** dense DiT. Conditioning is a **token-axis packed** stream — reference + video + pose patch-embedded (three Conv3d stems) with additive 28-channel color-coded mask embeddings, concatenated into one self-attention sequence — plus a **per-source RoPE** with integer T/H/W shifts (the `replace_flag` flips the reference H-shift, toggling animation vs. replacement). The reference image is encoded by the CLIP visual tower and injected via Wan-I2V image cross-attention. Sampling is plain CFG (guide 5.0), flow-matching UniPC/DPM++.
|
| 38 |
+
|
| 39 |
+
## License & attribution
|
| 40 |
+
This repackaging redistributes upstream weights under the license declared on the upstream model card (**MIT**); the upstream code repository is Apache-2.0. Please consult and cite the original:
|
| 41 |
+
- Model: https://huggingface.co/zai-org/SCAIL-2
|
| 42 |
+
- Code: https://github.com/zai-org/SCAIL-2
|
| 43 |
+
- Paper: *SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning* (arXiv:2606.10804)
|
| 44 |
+
- Built on Wan2.1 (Alibaba Wan team), UMT5-XXL, and OpenCLIP.
|
| 45 |
+
|
| 46 |
+
All credit for the model belongs to the original authors. This repo exists solely to make SCAIL-2 usable in SceneWorks on Apple Silicon.
|