local-inference-lab/Step-5-Preview-FLARE

Step-5 Preview

This repository provides an alpha-quality prepared runtime package for Step-5 Preview, built exclusively for a community preview of Flare. It includes model weights, compiled CUDA kernels, and captured execution graphs. These files require Flare; they cannot be loaded directly by Transformers, vLLM, SGLang, or llama.cpp.

Flare’s Step-5 recipe uses NVFP4 routed experts, MXFP8 gate/up projections, and FP8 KV caches. It supports the model’s sliding-window and compressed-region sparse attention, image inputs, and causal multi-token prediction (MTP) for speculative decoding. Kernel tuning and graph capture are completed during preparation, so serving runs through Flare’s native Rust runtime without Python or runtime compilation.

Use the gb202 revision for four NVIDIA RTX PRO 6000 Blackwell GPUs with 96 GB VRAM each, on one host. This package is prepared for four-way tensor parallelism, up to 16 concurrent requests, up to three MTP draft tokens per round, and up to four images per request. Its prepared sequence limit is 466,048 tokens; available context also depends on the GPU memory available for KV storage. The package is approximately 350 GB; allow at least 360 GB of free disk space for the download.

Let Flare select the revision

flare download selects gb202 automatically for the selected GPUs. After hf auth login, its token is normally stored at ~/.cache/huggingface/token. Use the actual token-file path if your Hugging Face configuration changes it.

mkdir -p flare-cache flare-state
docker run --rm --user "$(id -u):$(id -g)" \
  --gpus '"device=0,1,2,3",capabilities=compute' \
  --mount "type=bind,src=$PWD/flare-cache,dst=/cache" \
  --mount "type=bind,src=$HOME/.cache/huggingface/token,dst=/hf-token,readonly" \
  ghcr.io/local-inference-lab/flare:a08efb0-amd64 \
  download local-inference-lab/Step-5-Preview-FLARE \
  --gpus 0,1,2,3 --cache-dir /cache --token-file /hf-token

Run the repository by name to use the cached runtime. This command also downloads the selected revision if it is missing, so the separate download step is optional:

mkdir -p flare-cache flare-state
docker run --rm -it --user "$(id -u):$(id -g)" \
  --gpus '"device=0,1,2,3",capabilities=compute' --ulimit memlock=-1:-1 \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/flare-cache,dst=/cache" \
  --mount "type=bind,src=$PWD/flare-state,dst=/state" \
  --mount "type=bind,src=$HOME/.cache/huggingface/token,dst=/hf-token,readonly" \
  ghcr.io/local-inference-lab/flare:a08efb0-amd64 \
  run local-inference-lab/Step-5-Preview-FLARE \
  --gpus 0,1,2,3 --cache-dir /cache --token-file /hf-token \
  --trust-store /state/trust.toml --bind 0.0.0.0:8000

Host GPUs 0,1,2,3 are examples; Docker's --gpus selects the host GPUs, and Flare uses their container-local indices. All four GPUs must be available to the container. Sampling defaults are temperature 1.0, top-p 0.95, and top-k disabled, matching the Step-5 model card. --max-concurrency and --mtp-draft-tokens can be reduced at launch.

Download the hardware revision with the Hugging Face CLI

Install the Hugging Face hf CLI, then download only the GB202 runtime:

hf auth login
mkdir -p models flare-state
hf download local-inference-lab/Step-5-Preview-FLARE \
  --revision gb202 --include '*.safetensors' --local-dir ./models

Run the downloaded file with Flare in Docker:

docker run --rm -it --user "$(id -u):$(id -g)" \
  --gpus '"device=0,1,2,3",capabilities=compute' --ulimit memlock=-1:-1 \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/models,dst=/models,readonly" \
  --mount "type=bind,src=$PWD/flare-state,dst=/state" \
  ghcr.io/local-inference-lab/flare:a08efb0-amd64 \
  run /models/step5-tp4-mtp3-c16-fp8kv-signed.safetensors \
  --gpus 0,1,2,3 --max-concurrency 16 --mtp-draft-tokens 3 \
  --trust-store /state/trust.toml --bind 0.0.0.0:8000

With Flare installed directly on the host, the equivalent command is:

flare run ./models/step5-tp4-mtp3-c16-fp8kv-signed.safetensors \
  --gpus 0,1,2,3 --max-concurrency 16 --mtp-draft-tokens 3 \
  --bind 127.0.0.1:8000

Advanced options

Disk prefix caching retains reusable prompt and conversation state after it leaves the GPU cache, allowing matching requests to resume without recomputing the entire prefix:

flare run ./models/step5-tp4-mtp3-c16-fp8kv-signed.safetensors \
  --gpus 0,1,2,3 \
  --prefix-cache-disk /path/to/flare-prefix-cache \
  --prefix-cache-disk-limit 100GiB

Prefix offload is disabled by default. This example allows up to 100 GiB of disk retention in addition to the model’s disk requirements; choose a limit appropriate for your storage. Reusing the directory preserves compatible checkpoints across restarts.

For Docker, mount the prefix-cache directory as writable persistent storage and allow Linux io_uring operations for disk I/O.