# dllm-server (span-02 engine)

Rust + hand-written CUDA inference server for span-02 (Gemma-4 26B-A4B; NVFP4 experts, FP8 dense layers). It serves the
TypeSafe System One / Decision Index `/v1/systemone` API (JSON, multipart assets, zstd / gzip bodies)
and `/health`; the request and response format is in the model card (`../README.md`, section API).

Crates:
* `core`: model config, safetensors loader, tokenizer and prompt construction, answer policy (gate, thought budget, blend).
* `kernels`: the CUDA kernels (`cuda/*.cu`, compiled for sm_120a and embedded in the binary) and their launch glue.
* `model`: the Gemma-4 decoder on the GPU (paged KV with a sliding-window pool, CUDA graphs), the MTP draft head,
  speculative decoding, the vision tower.
* `vision`: image decode and the Gemma-4 image preprocessing (bit-exact with the HF processor).
* `runtime`: the scheduler (continuous batching, radix prefix cache, earliest-deadline-first admission) and the
  readout / gate / thought / readout loop.
* `server`: HTTP, wire format (`wire.rs`), body transport and TypeSafe image assets (`assets.rs`), command line.

## Requirements
* NVIDIA Blackwell GPU with compute capability 12.0 (sm_120a): RTX 5090 (32 GB) or RTX PRO 6000 Blackwell (96 GB).
  The kernels use sm_120a block-scaled FP4/FP8 tensor-core instructions; other architectures are not supported.
* NVIDIA driver with CUDA 12.9 support (R575 or newer).
* Build: CUDA toolkit 12.8+ (`nvcc`; 12.9 used for the published measurements), stable Rust (1.99.0 used),
  a C/C++ toolchain, `cmake` and `nasm` (for libjpeg-turbo), `pkg-config`, `clang`.

## Build
```
cd engine
NVCC=/usr/local/cuda/bin/nvcc cargo build --release -p dllm-server
# -> target/release/dllm-server   (the sm_120a cubins are embedded; no GPU is needed at build time)
```
Without `nvcc` the crates still compile (for the CPU tests), but the binary has no device code and fails at start.

## Run
From the repo root, `bash launch.sh` starts the server with the submission settings (see the model card). By hand:
```
engine/target/release/dllm-server --bind 0.0.0.0:8000 --model <repo dir> --spec-draft <dir of google/gemma-4-26B-A4B-it-assistant> \
  --max-len 16384 --budget-ms 0 --tau 0.8 --cap 1024 --think-ms 750 --think-cost <repo dir>/think_cost.json \
  --image-soft-tokens 560 --serial --no-prefix-cache --spec-max-branches 16
```
`dllm-server --help` lists every option. Throughput-oriented serving drops `--serial --no-prefix-cache` (cross-request
batching and the prefix cache) and may use a wall-clock thought budget (`--budget-ms`) instead of `--think-ms`.

## Tests
```
cargo test --workspace                                      # CPU: policy, prompts, wire format, scheduler, image preprocessing
# GPU (one CUDA device; the model tests need the weights and the MTP draft, ~24 GB of GPU memory):
NVCC=/usr/local/cuda/bin/nvcc cargo test --release -p dllm-kernels -- --ignored
DLLM_MODEL_DIR=<repo dir> DLLM_DRAFT_DIR=<draft dir> NVCC=/usr/local/cuda/bin/nvcc \
  cargo test --release -p dllm-model -p dllm-server -- --ignored --test-threads 1
```
The GPU tests check the kernels against analytic inputs and an f32 attention reference, that speculative thoughts equal
one-row greedy decoding token for token (alone and batched), and that the server with the submission settings returns
exactly the recorded responses of `crates/server/tests/golden/` (choices, probabilities, per-answer reasoning tokens and traces).

License: Apache-2.0; see `NOTICE` at the repo root for the projects whose algorithms are ported.