inference/server/README.md

Ripple on B200

BF16 Ripple-65B + native DFlash2 on two NVIDIA B200 GPUs, using SGLang 0.5.19. Four files here: this README, model.py, ripple_format.py, and serve.py. Both inference folders share chat_template.jinja at the repository root; keep that layout. Uses the tools-aware ChatML template from our evals.

Requires Linux, CUDA Toolkit 13, a C++ compiler, and uv. Use complete, unquantized BF16 target and matching DFlash2DraftModel checkpoints. Run the commands below from inference/server.

export CUDA_HOME=/usr/local/cuda-13.0
export PATH="$CUDA_HOME/bin:$PATH"

CUDA_VISIBLE_DEVICES=0,1 uv run --script serve.py \
  --model-path /path/to/target \
  --speculative-draft-model-path /path/to/draft

Dependencies install automatically. The server listens on 127.0.0.1:30000. It uses the checkpoint's tokenizer and the repository-root chat template.

curl http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"ripple","messages":[{"role":"user","content":"Hello"}],"max_tokens":128}'

Defaults: TP=2, BF16 weights and KV cache, TRT-LLM target attention, Triton draft attention, 16 draft tokens, and a 1Mi-token target context (--context-length 1048576, including prompt and generated tokens). Standard SGLang CLI flags apply. The DFlash2 draft's last layer is global attention without RoPE. It was trained on 16Ki-token rows, so the draft keeps a 16Ki-token window (--speculative-draft-window-size 16384); smaller windows are refused. The launcher defaults SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 so stock SGLang can extend the draft's positional cache beyond its 128Ki checkpoint limit. This does not increase the draft attention window. The target checkpoint supports up to 2Mi positions. For four GPUs, run a second copy with CUDA_VISIBLE_DEVICES=2,3 and --port 30001. This gives two independent replicas, each using two GPUs. Send eight concurrent requests to each port for sixteen total.

The launcher enables Ripple's parsers from ripple_format.py (--reasoning-parser ripple and --tool-call-parser ripple). Responses separate message.reasoning_content, final-answer message.content, and structured message.tool_calls. Streaming uses the corresponding delta fields. Allow enough output tokens for both reasoning and the answer.

Pass OpenAI-style tools and tool_choice: "auto" to request calls:

{
  "model": "ripple",
  "messages": [{"role": "user", "content": "Use add_numbers to calculate 23 + 19."}],
  "tools": [{"type": "function", "function": {
    "name": "add_numbers", "description": "Add two integers.", "strict": true,
    "parameters": {"type": "object", "properties": {
      "a": {"type": "integer"}, "b": {"type": "integer"}
    }, "required": ["a", "b"], "additionalProperties": false}
  }}],
  "tool_choice": "auto",
  "temperature": 0,
  "max_tokens": 1024
}

Your application executes the requested function. Append the returned assistant message unchanged, then a tool message containing its tool_call_id and a string content with the result, and send the history again. The template re-renders an unchanged message to exactly the tokens the model generated. Previous reasoning_content is retained by default (chat_template_kwargs.clear_thinking=false). The template does not implement a thinking on/off switch or reasoning-effort levels.

For tool_choice: "required", use function.strict=true and parameters.additionalProperties=false, as above. Non-strict required calls produced extra argument fields in validation with this SGLang version.