| --- |
| license: apache-2.0 |
| pipeline_tag: text-to-video |
| tags: |
| - video-generation |
| - autoregressive |
| - self-forcing |
| - real-time |
| - wan2.1 |
| - mlx |
| - apple-silicon |
| base_model: krea/krea-realtime-video |
| --- |
| |
| # SceneWorks/krea-realtime-14b-mlx |
|
|
| **Turnkey, SceneWorks-converted weights of [krea/krea-realtime-video](https://huggingface.co/krea/krea-realtime-video)** (Krea Realtime 14B) — an **autoregressive** video generator distilled from Wan 2.1 T2V 14B via **Self-Forcing** — packaged for native **Apple-Silicon (MLX)** inference inside SceneWorks. This is **not** an original model; it is a format/dtype repackaging of the upstream release for first-class macOS use (no PyTorch at runtime). |
|
|
| > Upstream generates frame-chunks left-to-right with a persistent KV cache (~4 steps/chunk, causal attention, 24 fps), with KV-cache recompute and block-causal masking as the clip-length stability mechanisms. Modalities: text-to-video and video-to-video. |
|
|
| ## What changed vs. upstream |
|
|
| Krea Realtime 14B ships **transformer-only** — 1095 F16 tensors, 14,288,491,584 parameters, in two equivalent on-disk layouts (a single `krea-realtime-video-14b.safetensors` with every key `model.`-prefixed, and a sharded `transformer/` without the prefix). The text encoder, VAE and tokenizer are the **stock Wan 2.1** components and are bundled here unchanged so each tier is self-contained. |
|
|
| - **DiT** → key-normalized (the outer `model.` prefix stripped; `patch_embedding` conv → Linear reshape, `text/time_embedding` Sequential rename, `ffn.0/2` → `fc1/fc2`, the non-persistent `freqs` RoPE buffer dropped) and cast **F16 → bf16**, which is the dtype the upstream reference runs the DiT in. Emitted at three tiers (below). Tensor set and shapes are verified against the `wan21_t2v_14b` inventory — exactly 1095 tensors, no missing / extra / mis-shaped — *before* anything is written. |
| - **Text encoder** — stock **UMT5-XXL** (`t5_encoder.safetensors`, bf16), byte-identical to the encoder already published in [`SceneWorks/scail2-mlx`](https://huggingface.co/SceneWorks/scail2-mlx); loaded by `mlx_gen_wan::Umt5Encoder`. |
| - **VAE** — stock **Wan 2.1 z16** VAE (`vae.safetensors`, f32, stride 4×8×8); loaded by `mlx_gen_wan::WanVae`. |
| - **Tokenizer** — stock UMT5-XXL `tokenizer.json`. |
| - **`config.json`** — the Wan 2.1 14B geometry (`dim 5120`, `40` layers/heads, `ffn 13824`, `in/out_dim 16`, `patch (1,2,2)`, `text_len 512`, `text_dim 4096`) plus the autoregressive fields (`local_attn_size -1`, `sink_size 0`, `num_frames_per_block 3`, `kv_cache_num_frames 3`, `frame_seq_length 1560`, `seq_length 32760`, `denoising_step_list [1000, 937, 833, 625, 0]`, `timestep_shift 5.0`, `do_kv_recomp true`), and — on a packed tier — a `quantization` block. |
|
|
| ## Tiers |
|
|
| Each tier subdirectory is a **complete, self-contained snapshot** (DiT at that precision + the shared dense TE / VAE / tokenizer + `config.json`), so the shared components repeat per tier. Quantization is **group-wise affine, group size 64**, via MLX `quantize`, applied to exactly the per-block self/cross-attention `q`/`k`/`v`/`o` and FFN `fc1`/`fc2` Linears (400 of them); the patch/text/time embeddings, norms, modulation tables, `time_projection` and the output head stay dense bf16. The tiers ship **pre-quantized on disk** — the loader builds the quantized Linears straight from the packs, with no dense-bf16 load transient. |
|
|
| | tier | DiT | tier total | notes | |
| |---|---|---|---| |
| | `q4/` | 8.38 GB | **20.27 GB** | Q4 packed — the SceneWorks dense-video default | |
| | `q8/` | 15.40 GB | **27.29 GB** | Q8 packed | |
| | `bf16/` | 28.58 GB | **40.46 GB** | dense bf16, sharded ×7 | |
|
|
| ## Layout |
|
|
| ``` |
| q4/config.json q8/config.json bf16/config.json |
| q4/dit.safetensors q8/dit.safetensors bf16/transformer/dit-0000{1..7}-of-00007.safetensors |
| q4/t5_encoder.safetensors q8/t5_encoder.safetensors bf16/t5_encoder.safetensors |
| q4/vae.safetensors q8/vae.safetensors bf16/vae.safetensors |
| q4/tokenizer.json q8/tokenizer.json bf16/tokenizer.json |
| ``` |
|
|
| The `q4`/`q8` tiers ship the DiT as a single `dit.safetensors`. The dense `bf16` tier ships it as a **sharded `transformer/` directory** (7 shards, ~4 GiB each) — the loader accepts either layout, and sharding keeps the largest object well under the single-file sizes HF discourages while making the download resumable per shard. |
|
|
| ## License & attribution |
|
|
| This repackaging redistributes upstream weights under the upstream license (**Apache-2.0**). All credit for the model belongs to the original authors; this repo exists solely to make Krea Realtime usable in SceneWorks on Apple Silicon. |
|
|
| - **Model:** https://huggingface.co/krea/krea-realtime-video (Krea AI, Apache-2.0) |
| - **Code:** https://github.com/krea-ai/realtime-video |
| - **Stock Wan 2.1 components** (text encoder / VAE / tokenizer, bundled unchanged): https://huggingface.co/Wan-AI/Wan2.1-T2V-14B-Diffusers — Wan 2.1 by the Alibaba Wan team, Apache-2.0. Built on UMT5-XXL. |
|
|
