SceneWorks commited on
Commit
1930fbd
·
verified ·
1 Parent(s): 1897f4b

Add model card

Browse files
Files changed (1) hide show
  1. README.md +46 -0
README.md ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: image-to-video
4
+ library_name: diffusers
5
+ tags:
6
+ - character-animation
7
+ - video-generation
8
+ - cross-identity-replacement
9
+ - pose-driven
10
+ - diffusion
11
+ - mlx
12
+ - apple-silicon
13
+ base_model: zai-org/SCAIL-2
14
+ ---
15
+
16
+ # SceneWorks/scail2-mlx
17
+
18
+ **Turnkey, SceneWorks-converted weights of [zai-org/SCAIL-2](https://huggingface.co/zai-org/SCAIL-2)** — an end-to-end controlled **character-animation / motion-transfer** video model — packaged for **native Apple-Silicon (MLX)** inference inside SceneWorks. This is **not** an original model; it is a format/dtype repackaging of the upstream release for first-class macOS use (no PyTorch at runtime).
19
+
20
+ > Capabilities (from upstream): character animation from a reference image + driving video, **cross-identity character replacement**, zero-shot animal-driving, end-to-end *and* pose-rendered driving, and (experimental) multi-reference. Image output is `num_frames == 1`.
21
+
22
+ ## What changed vs. upstream
23
+ - The main DiT (`model/1/fsdp2_rank_0000_checkpoint.pt`, an FSDP2/SAT checkpoint) was key-remapped to the `SCAIL2Model` parameter naming using the upstream `convert.py` contract (fused `query_key_value`→`q`/`k`/`v`, `key_value`→`k`/`v`, `clip_feature_key_value_list`→`k_img`/`v_img`) and cast **fp32 → bf16**, then saved as a single safetensors. The remap is bit-faithful (validated: 987 source keys → 1307 model keys, an exact key+shape match against `SCAIL2Model.from_config(config-14b.json)`).
24
+ - The text encoder (UMT5-XXL), VAE (Wan2.1), and image encoder (open-CLIP XLM-RoBERTa ViT-H/14) are the **stock upstream files**, included here so the snapshot is self-contained.
25
+
26
+ ## Contents
27
+ | file | source | notes |
28
+ |---|---|---|
29
+ | `dit.safetensors` | converted | SCAIL-2 14B DiT, **bf16**, 1307 tensors (~31 GB) |
30
+ | `config.json` | upstream `configs/config-14b.json` | `model_type: i2v`, `dim 5120`, `ffn 13824`, `40` layers/heads, `in_dim 20`, `mask_dim 28`, `out_dim 16` |
31
+ | `Wan2.1_VAE.pth` | upstream, stock | z16 VAE, stride (4,8,8) |
32
+ | `umt5-xxl/` | upstream, stock | UMT5-XXL encoder (bf16) + tokenizer |
33
+ | `models_clip_open-clip-xlm-roberta-large-vit-huge-14-onlyvisual.pth` | upstream, stock | reference-image visual encoder (1280-dim) |
34
+ | `bias-aware-dpo-lora.pt` | upstream, stock | optional Bias-Aware DPO refinement LoRA |
35
+
36
+ ## Architecture (summary)
37
+ Wan2.1-14B **I2V** dense DiT. Conditioning is a **token-axis packed** stream — reference + video + pose patch-embedded (three Conv3d stems) with additive 28-channel color-coded mask embeddings, concatenated into one self-attention sequence — plus a **per-source RoPE** with integer T/H/W shifts (the `replace_flag` flips the reference H-shift, toggling animation vs. replacement). The reference image is encoded by the CLIP visual tower and injected via Wan-I2V image cross-attention. Sampling is plain CFG (guide 5.0), flow-matching UniPC/DPM++.
38
+
39
+ ## License & attribution
40
+ This repackaging redistributes upstream weights under the license declared on the upstream model card (**MIT**); the upstream code repository is Apache-2.0. Please consult and cite the original:
41
+ - Model: https://huggingface.co/zai-org/SCAIL-2
42
+ - Code: https://github.com/zai-org/SCAIL-2
43
+ - Paper: *SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning* (arXiv:2606.10804)
44
+ - Built on Wan2.1 (Alibaba Wan team), UMT5-XXL, and OpenCLIP.
45
+
46
+ All credit for the model belongs to the original authors. This repo exists solely to make SCAIL-2 usable in SceneWorks on Apple Silicon.