| # Ripple on B200 |
|
|
| BF16 Ripple-65B + native DFlash2 on two NVIDIA B200 GPUs, using **SGLang 0.5.19**. |
| Four files here: this README, `model.py`, `ripple_format.py`, and `serve.py`. Both |
| inference folders share `chat_template.jinja` at the repository root; keep that layout. |
| Uses the tools-aware ChatML template from our evals. |
|
|
| Requires Linux, CUDA Toolkit 13, a C++ compiler, and [uv](https://docs.astral.sh/uv/). |
| Use complete, unquantized BF16 target and matching `DFlash2DraftModel` checkpoints. |
| Run the commands below from `inference/server`. |
|
|
| ```bash |
| export CUDA_HOME=/usr/local/cuda-13.0 |
| export PATH="$CUDA_HOME/bin:$PATH" |
| |
| CUDA_VISIBLE_DEVICES=0,1 uv run --script serve.py \ |
| --model-path /path/to/target \ |
| --speculative-draft-model-path /path/to/draft |
| ``` |
|
|
| Dependencies install automatically. The server listens on `127.0.0.1:30000`. |
| It uses the checkpoint's tokenizer and the repository-root chat template. |
|
|
| ```bash |
| curl http://127.0.0.1:30000/v1/chat/completions \ |
| -H 'Content-Type: application/json' \ |
| -d '{"model":"ripple","messages":[{"role":"user","content":"Hello"}],"max_tokens":128}' |
| ``` |
|
|
| Defaults: TP=2, BF16 weights and KV cache, TRT-LLM target attention, Triton draft |
| attention, 16 draft tokens, and a **1Mi-token target context** (`--context-length 1048576`, |
| including prompt and generated tokens). Standard SGLang CLI flags apply. |
| The DFlash2 draft's last layer is global attention without RoPE. It was trained on |
| 16Ki-token rows, so the draft keeps a **16Ki-token window** (`--speculative-draft-window-size 16384`); |
| smaller windows are refused. |
| The launcher defaults `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1` so stock SGLang can |
| extend the draft's positional cache beyond its 128Ki checkpoint limit. This does not |
| increase the draft attention window. The target checkpoint supports up to 2Mi positions. |
| For four GPUs, run a second copy with `CUDA_VISIBLE_DEVICES=2,3` and `--port 30001`. |
| This gives two independent replicas, each using two GPUs. Send eight concurrent |
| requests to each port for sixteen total. |
|
|
| The launcher enables Ripple's parsers from `ripple_format.py` (`--reasoning-parser ripple` |
| and `--tool-call-parser ripple`). Responses separate `message.reasoning_content`, |
| final-answer `message.content`, and structured `message.tool_calls`. Streaming uses the |
| corresponding `delta` fields. Allow enough output tokens for both reasoning and the answer. |
|
|
| Pass OpenAI-style `tools` and `tool_choice: "auto"` to request calls: |
|
|
| ```json |
| { |
| "model": "ripple", |
| "messages": [{"role": "user", "content": "Use add_numbers to calculate 23 + 19."}], |
| "tools": [{"type": "function", "function": { |
| "name": "add_numbers", "description": "Add two integers.", "strict": true, |
| "parameters": {"type": "object", "properties": { |
| "a": {"type": "integer"}, "b": {"type": "integer"} |
| }, "required": ["a", "b"], "additionalProperties": false} |
| }}], |
| "tool_choice": "auto", |
| "temperature": 0, |
| "max_tokens": 1024 |
| } |
| ``` |
|
|
| Your application executes the requested function. Append the returned assistant |
| message unchanged, then a `tool` message containing its `tool_call_id` and a string |
| `content` with the result, and send the history again. The template re-renders an |
| unchanged message to exactly the tokens the model generated. Previous |
| `reasoning_content` is retained by default (`chat_template_kwargs.clear_thinking=false`). |
| The template does not implement a thinking on/off switch or reasoning-effort levels. |
|
|
| For `tool_choice: "required"`, use `function.strict=true` and |
| `parameters.additionalProperties=false`, as above. Non-strict required calls |
| produced extra argument fields in validation with this SGLang version. |
|
|
