FastFrames-S, MLX port (bf16)

An unofficial community port of FastFrames-S to MLX, for text-to-video on Apple silicon Macs. It is not made, reviewed or endorsed by Apple, and it has no relationship with Apple.

Research use only, non-commercial. The KV refinement network and the pixel decoder are released under the Apple Machine Learning Research Model License, which allows use, modification and redistribution only for non-commercial scientific research. That applies to this port as a whole. See LICENSE-APPLE-AMLR before you use it.

Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.

From a text prompt it makes a 5 second clip: 81 frames at 832x480, 16 fps. The pipeline has four parts:

  • the umT5-XXL text encoder from Wan2.1, run once per prompt;
  • the frame-wise causal Wan2.1-1.3B DiT, the two-step Causal Forcing++ generator, which denoises one latent frame (4 video frames) at a time in 2 steps against a KV cache of the frames before it;
  • the FastFrames KV refinement network, which repairs each finished frame's cache entry instead of running a second clean DiT pass;
  • the FastFrames causal pixel decoder, which turns the DiT features straight into RGB frames in place of the Wan VAE.

Samples

All were made with run.py at its default settings on an M3 Max. Click a preview for the full-quality mp4. The previews are 416 px wide at 12 fps.

corgi Big Sur coast
llama red fox

Prompts and seeds (also in the .json file next to each clip):

  1. corgi-surfing_s0, seed 0: "A corgi surfing a wave at sunset"
  2. big-sur-coast_s75000225, seed 75000225: "A drone view of waves crashing against the rugged cliffs along Big Sur's Garay Point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore, casting long shadows. In the distance, a small island with a lighthouse stands tall, its beam piercing the twilight. Green shrubbery covers the cliff's edge, and the steep drop from the road down to the beach is a dramatic feat, with the cliff's edges jutting out over the sea. The camera angle provides a bird's-eye view, capturing the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway. The scene is bathed in a warm, golden hue, highlighting the textures and details of the rocky terrain."
  3. llama-disco_s94000282, seed 94000282: "A realistic photo of a llama wearing colorful pajamas dancing energetically on a stage under vibrant disco lighting. The llama has large floppy ears and a playful expression, moving its legs in a lively dance. It wears a red and yellow striped pajama top and matching pajama pants, with a fluffy tail swaying behind it. The stage is adorned with glittering disco balls and colorful lights, casting a lively and joyful atmosphere. The background features blurred audience members and a backdrop with disco-themed decorations. A dynamic shot capturing the llama mid-dance from a slightly elevated angle."
  4. red-fox-snow_s3, seed 3: "A red fox trotting through fresh snow in a pine forest, morning light"

The corgi is a fair example of the model with a short prompt: the scene is right, but the dog is swimming rather than surfing. Long, descriptive prompts like the Big Sur and llama ones follow the text more closely.

Quick start

Needs an Apple silicon Mac with enough free memory for an 11 GB peak (tested on a 36 GB M3 Max), Python 3 (tested with 3.13) and ffmpeg on the PATH. The download is about 15 GB.

pip install -U huggingface_hub
hf download mlx-community/FastFrames-S-bf16-mlx --local-dir FastFrames-S-bf16-mlx
cd FastFrames-S-bf16-mlx
pip install -r requirements.txt     # brew install ffmpeg, if you don't have it

# one clip: prompt, seed, output file
python run.py "A red fox trotting through fresh snow in a pine forest, morning light" 3 fox.mp4

run.py looks for the weights next to itself, so it works from the downloaded folder as is. It also takes --weights DIR, or FASTFRAMES_MLX_WEIGHTS, and if it finds no weights it fetches this repo with snapshot_download. The umT5 tokenizer is downloaded from Wan-AI/Wan2.1-T2V-1.3B on first use (a few MB). A JSON line with the timings is printed and saved next to the mp4.

Useful options:

  • --frames N sets the number of latent frames (default 21, which gives 81 video frames). N latent frames give 4N - 3 video frames, so --frames 5 makes a 17-frame test clip.
  • --reference runs the plain port without the speed changes listed below, for A/B checks.
  • --opts xcollapse=0 turns off the cross-attention change, which makes the DiT bit-identical to the plain port. --opts attn=flash2 uses the custom Metal attention kernel (about 5 % slower).

Several clips at once: batch.py loads the text encoder once for the whole batch, caches each prompt's embedding, and writes each mp4 on a CPU thread while the GPU starts the next clip.

cat > prompts.txt <<'EOF'
A corgi surfing a wave at sunset
{"prompt": "A red fox trotting through fresh snow in a pine forest, morning light", "seeds": [3]}
EOF
python batch.py prompts.txt out/ --seeds 0,1

Text embeddings are cached in ~/.cache/fastframes-mlx/emb (set FASTFRAMES_MLX_CACHE to move it), so a repeated prompt skips the 11 GB text encoder.

Accuracy

  • In fp32, each MLX component (DiT pass, KV refiner, pixel decoder) matches Apple's PyTorch modules to about 1e-5 relative error.
  • The default bf16 path gives the same video as the plain bf16 port to within 1/255 per pixel (91-92 dB PSNR) when xcollapse is off. The DiT and refiner outputs are then bit-identical; the only difference is the fp32 rounding in the decoder kernel.
  • With xcollapse on (the default), the clips are no longer identical to the plain port, but they are as close to an fp32 run as the plain port is. PSNR of the decoded frames against the same clip with the DiT in fp32 (same noise), with the relative error of the DiT features in brackets:
clip plain bf16 port default (xcollapse) flash2 kernel
corgi, 25 frames 24.71 (0.116) 24.23 (0.118) 24.03 (0.123)
Big Sur, 25 frames 14.70 (0.253) 14.18 (0.259) 14.65 (0.255)
llama, 25 frames 14.98 (0.290) 14.95 (0.293) 14.58 (0.305)
corgi, 81 frames 24.62 (0.123) 24.33 (0.121) 24.02 (0.127)
Big Sur, 81 frames 19.30 (0.167) 20.99 (0.147) 19.50 (0.166)
llama, 81 frames 14.41 (0.329) 14.43 (0.329) 13.93 (0.334)
mean 18.79 dB (0.213) 18.85 dB (0.211) 18.45 dB (0.218)

These PSNR values look low because generation is chaotic. Each frame is generated from the cached frames before it, so a rounding difference in one frame changes the next one, and the bf16 and fp32 runs drift apart in motion detail while showing the same scene and subject. The table measures that spread, not a quality loss: all three bf16 variants sit at the same distance from fp32, and side by side the clips are equally clean, with no artefacts, blur or colour drift.

Performance

Measured on an Apple M3 Max with 36 GB of unified memory, one clip of 81 frames:

  • About 55-61 s per clip for the rollout on a quiet machine, and about 90 s when other GPU work is running. Run-to-run noise is about 20 %.
  • Peak memory about 11 GB: 11.1 GB while the text encoder runs, 10.7 GB during generation. The text encoder is unloaded before generation starts.
  • Text encoding about 7 s per new prompt. Embeddings are cached, so repeats and batches skip it.
  • Decoding the 81 frames takes about 1.8 s.
  • Self-attention over the growing KV cache is most of the late-frame cost. MLX's attention kernel runs at 8.2 TFLOP/s here, 85-89 % of the 9.2-9.5 TFLOP/s matrix-multiply peak measured on this GPU, so there is little left to gain from a faster kernel. A custom flash-attention kernel is included (--opts attn=flash2) but is about 5 % slower.

Limitations

  • Fine detail is soft, and faces and hands are often weak. This matches Apple's S model; it is not an effect of the port.
  • English prompts only were tested. Long, detailed prompts in the style of Wan's prompt examples work best; short prompts give looser results.
  • Clips are 5 seconds (81 frames at 16 fps). Longer clips are not supported.
  • Tested only on an M3 Max. Machines with 16 GB of memory or less may not have room for the 11 GB peak.

Model Derivative and modifications

This repository is a Model Derivative of the Apple Machine Learning Research Model FastFrames-S, as defined in its licence. It contains no retraining or fine-tuning: every weight value is the released one. The changes are:

  • Format. The PyTorch checkpoints were converted to MLX safetensors. The DiT, the KV refiner and the text encoder are bf16, as released; the pixel decoder is fp32, as released.
  • Inference code. The pipeline was re-implemented in MLX (ff_ref.py is the plain port, ff_mlx.py the faster default path, ff_kernels.py the Metal kernels). It follows the structure of Apple's reference code in ml-fastframes. Inference runs in bf16, with norms and RoPE computed in fp32, as in the reference.
  • Fused QKV. The DiT's three self-attention projections are concatenated at load time into one matrix multiply. The output is bit-identical.
  • Cross-attention padding collapse (xcollapse, on by default). The prompt embedding is zero-padded to 512 tokens, so every padding position has the same key and value. The port attends to the real tokens plus one padding key weighted by log of the padding count, in fp32. This is mathematically equal to the original softmax but changes the rounding, so clips differ slightly from the plain port (see Accuracy).
  • Skipped last cache repair (skiplast). The KV refinement for the final frame is never read by a later frame, so it is skipped. The output is bit-identical.
  • Decoder kernel. The pixel decoder's depthwise 3x3x3 causal convolution runs as one Metal kernel instead of 27 shifted adds (at most 1/255 per pixel difference).
  • Other speed changes that do not alter the output: mx.compile of elementwise chains and a per-prompt embedding cache.

Files

file size what
dit.safetensors 2.84 GB Causal Forcing++ frame-wise 2-step generator (Wan2.1-1.3B DiT), bf16
kv_refiner.safetensors 0.69 GB FastFrames-S KV refinement network, bf16
pixel_decoder.safetensors 0.26 GB FastFrames-S causal pixel decoder, fp32
umt5_xxl.safetensors 11.36 GB Wan2.1 umT5-XXL text encoder, bf16
run.py one clip from the command line
batch.py several prompts and seeds, text encoder loaded once
ff_mlx.py the default (fast) MLX pipeline
ff_ref.py the plain MLX port; ff_mlx.py builds on it
ff_kernels.py Metal kernels: decoder depthwise conv, flash attention
requirements.txt Python packages
LICENSE-APPLE-AMLR Apple Machine Learning Research Model License (KV refiner, pixel decoder, this derivative)
LICENSE-APACHE-2.0 Apache License 2.0 (Causal Forcing++ generator, Wan2.1 umT5 encoder and tokenizer)
samples/ the sample clips (mp4), previews (gif) and their prompts and seeds (json)

Licences

  • KV refinement network and pixel decoder: from apple/FastFrames-S, Apple Machine Learning Research Model License (LICENSE-APPLE-AMLR, included verbatim). Research purposes only; no commercial use.
  • DiT generator: the Causal Forcing++ frame-wise 2-step checkpoint from zhuhz22/Causal-Forcing, Apache License 2.0 (LICENSE-APACHE-2.0). Converted to MLX safetensors without other changes.
  • Text encoder and tokenizer: umT5-XXL from Wan-AI/Wan2.1-T2V-1.3B, Apache License 2.0 (LICENSE-APACHE-2.0). The encoder was converted to MLX safetensors without other changes; the tokenizer is downloaded from the Wan repo at run time.

Because the pipeline includes the Apple components, use of this repository as a whole is limited to the research purposes the Apple licence allows.

Credits

If you use this port in research, please cite the original work:

@article{chou2026fastframes,
  title   = {FastFrames: Video Generation at Interactive Latency},
  author  = {Chou, Gene and Yang, Guandao and Luan, Fujun and Mescheder, Lars and
             Dong, Wei and Richter, Stephan R. and Koltun, Vladlen},
  journal = {arXiv preprint},
  year    = {2026}
}

@article{zhu2026causal,
  title   = {Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality
             Real-Time Interactive Video Generation},
  author  = {Zhu, Hongzhou and Zhao, Min and He, Guande and Su, Hang and Li, Chongxuan and Zhu, Jun},
  journal = {arXiv preprint arXiv:2602.02214},
  year    = {2026}
}

@article{wan2.1,
  title   = {Wan: Open and Advanced Large-Scale Video Generative Models},
  author  = {Wan Team},
  year    = {2025}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mlx-community/FastFrames-S-bf16-mlx

Finetuned
(1)
this model

Paper for mlx-community/FastFrames-S-bf16-mlx