dllm-server (span-02 engine)

Rust + hand-written CUDA inference server for span-02 (Gemma-4 26B-A4B; NVFP4 experts, FP8 dense layers). It serves the TypeSafe System One / Decision Index /v1/systemone API (JSON, multipart assets, zstd / gzip bodies) and /health; the request and response format is in the model card (../README.md, section API).

Crates:

core: model config, safetensors loader, tokenizer and prompt construction, answer policy (gate, thought budget, blend).
kernels: the CUDA kernels (cuda/*.cu, compiled for sm_120a and embedded in the binary) and their launch glue.
model: the Gemma-4 decoder on the GPU (paged KV with a sliding-window pool, CUDA graphs), the MTP draft head, speculative decoding, the vision tower.
vision: image decode and the Gemma-4 image preprocessing (bit-exact with the HF processor).
runtime: the scheduler (continuous batching, radix prefix cache, earliest-deadline-first admission) and the readout / gate / thought / readout loop.
server: HTTP, wire format (wire.rs), body transport and TypeSafe image assets (assets.rs), command line.

Requirements

NVIDIA Blackwell GPU with compute capability 12.0 (sm_120a): RTX 5090 (32 GB) or RTX PRO 6000 Blackwell (96 GB). The kernels use sm_120a block-scaled FP4/FP8 tensor-core instructions; other architectures are not supported.
NVIDIA driver with CUDA 12.9 support (R575 or newer).
Build: CUDA toolkit 12.8+ (nvcc; 12.9 used for the published measurements), stable Rust (1.99.0 used), a C/C++ toolchain, cmake and nasm (for libjpeg-turbo), pkg-config, clang.

Build

cd engine
NVCC=/usr/local/cuda/bin/nvcc cargo build --release -p dllm-server
# -> target/release/dllm-server   (the sm_120a cubins are embedded; no GPU is needed at build time)

Without nvcc the crates still compile (for the CPU tests), but the binary has no device code and fails at start.

Run

From the repo root, bash launch.sh starts the server with the submission settings (see the model card). By hand:

engine/target/release/dllm-server --bind 0.0.0.0:8000 --model <repo dir> --spec-draft <dir of google/gemma-4-26B-A4B-it-assistant> \
  --max-len 16384 --budget-ms 0 --tau 0.8 --cap 1024 --think-ms 750 --think-cost <repo dir>/think_cost.json \
  --image-soft-tokens 560 --serial --no-prefix-cache --spec-max-branches 16

dllm-server --help lists every option. Throughput-oriented serving drops --serial --no-prefix-cache (cross-request batching and the prefix cache) and may use a wall-clock thought budget (--budget-ms) instead of --think-ms.

Tests

cargo test --workspace                                      # CPU: policy, prompts, wire format, scheduler, image preprocessing
# GPU (one CUDA device; the model tests need the weights and the MTP draft, ~24 GB of GPU memory):
NVCC=/usr/local/cuda/bin/nvcc cargo test --release -p dllm-kernels -- --ignored
DLLM_MODEL_DIR=<repo dir> DLLM_DRAFT_DIR=<draft dir> NVCC=/usr/local/cuda/bin/nvcc \
  cargo test --release -p dllm-model -p dllm-server -- --ignored --test-threads 1

The GPU tests check the kernels against analytic inputs and an f32 attention reference, that speculative thoughts equal one-row greedy decoding token for token (alone and batched), and that the server with the submission settings returns exactly the recorded responses of crates/server/tests/golden/ (choices, probabilities, per-answer reasoning tokens and traces).

License: Apache-2.0; see NOTICE at the repo root for the projects whose algorithms are ported.