| --- |
| license: apache-2.0 |
| library_name: mlx |
| pipeline_tag: text-to-video |
| tags: |
| - mlx |
| - text-to-video |
| - video-editing |
| - video-to-video |
| - reference-to-video |
| - wan2.2 |
| - bernini |
| base_model: ByteDance/Bernini-R-Diffusers |
| --- |
| |
| # Bernini-R (MLX) |
|
|
| Apple MLX port of **[ByteDance/Bernini-R](https://huggingface.co/ByteDance/Bernini-R-Diffusers)** — |
| the open-sourced *Renderer* of ByteDance's Bernini: a Wan2.2-T2V-A14B-derived video |
| generator/editor with **Segment-Aware 3D RoPE** for multi-reference / editing tasks. |
|
|
| Runs on Apple Silicon via [MLX](https://github.com/ml-explore/mlx) + the |
| [mlx-video](https://github.com/Blaizzy/mlx-video) Wan2.2 backbone. |
|
|
| ## ⚠️ Scope: renderer only |
|
|
| Only the Renderer ("-R") is open-sourced upstream. The MLLM semantic **planner** (the |
| paper's headline "latent semantic planning", a Qwen2.5-VL-7B model) is **not released**. |
| This port therefore runs with **UMT5 text conditioning only** — the planner-feature |
| channel is absent (and carries no weights in the released checkpoint). You get the |
| renderer's editing / reference-to-video / subject-consistency behavior, not the full |
| planner-guided system. |
|
|
| ## Tasks |
|
|
| | Task | Description | |
| |---|---| |
| | `t2v` / `t2i` | text-to-video / image | |
| | `r2v` | reference-to-video — generate a subject from up to K reference images (chained APG) | |
| | `v2v` | prompt-based video editing (source video injected as conditioning) | |
| | `rv2v` | reference + video editing | |
|
|
| ## Variants |
|
|
| | Repo | Precision | Size / expert | |
| |---|---|---| |
| | `…-bf16` | bfloat16 | 28.6 GB | |
| | `…-int4` | 4-bit (group 64) | 8.4 GB | |
|
|
| Two experts (high/low-noise) + 16-ch Wan2.2 VAE (0.5 GB) + UMT5 (11 GB). |
|
|
| ## Usage |
|
|
| ```python |
| from bernini_r_mlx import pipeline_mlx as P |
| |
| # text-to-video |
| P.t2v("path/to/ckpt", "a red fox in a snowy forest", num_frames=49, output_path="out.mp4") |
| |
| # reference-to-video (subject consistency) |
| P.r2v("path/to/ckpt", "the fox running across a field", |
| reference_images=["fox.png"], output_path="r2v.mp4") |
| |
| # video editing |
| P.v2v("path/to/ckpt", "... autumn forest ...", source_video="in.mp4", output_path="v2v.mp4") |
| ``` |
|
|
| ## Provenance & validation |
|
|
| - Architecture: **stock Wan2.2-T2V-A14B** (verified — diffusers `WanTransformer3DModel` keys, |
| no extra tensors); Bernini knobs (`switch_dit_boundary 0.875`, `shift 3.0`, |
| `use_src_id_rotary_emb`) live in the wrapper config. SA-3D RoPE adds **no parameters**. |
| - Converted fp32 → bf16 from `ByteDance/Bernini-R-Diffusers`; VAE/UMT5 from `Wan-AI/Wan2.2-T2V-A14B`. |
| - Validated: SA-3D RoPE parity ~1e-7; VAE roundtrip MAD 2.1/255; multi-segment forward |
| bit-exact vs t2v; int4 per-pass cosine 0.9992 vs bf16; e2e t2v / r2v / v2v coherent. |
|
|
| ## License & attribution |
|
|
| Apache-2.0. Derived from ByteDance Bernini-R, Wan2.2 (Wan-AI), and mlx-video. See `NOTICE`. |
|
|
