Card: serve with --max-model-len 32768 in Run with vLLM

README.md

@@ -147,7 +147,7 @@ Images go before the text of the request, each read at up to 1.6 megapixels; vid
 
 ## Run with vLLM
 
-[vLLM](https://github.com/vllm-project/vllm) serves d3 natively: `vllm serve` loads this repository as it is, with no conversion and no extra flags, and answers on vLLM's System One endpoint, `/v1/systemone`. The support is on vLLM's main branch since 2026-10-14 ([vllm-project/vllm#61565](https://github.com/vllm-project/vllm/pull/61565), commit [`89cab55`](https://github.com/vllm-project/vllm/commit/89cab550862e4b7f08d70c19682121362d7dd0c5)), and no vLLM release includes it yet (the latest is v0.31.0). Until one does, install a current nightly build with [uv](https://docs.astral.sh/uv/); vLLM's nightly indexes do not work with plain `pip`:
+[vLLM](https://github.com/vllm-project/vllm) serves d3 natively: `vllm serve` loads this repository as it is, with no conversion, and answers on vLLM's System One endpoint, `/v1/systemone`. The support is on vLLM's main branch since 2026-10-14 ([vllm-project/vllm#61565](https://github.com/vllm-project/vllm/pull/61565), commit [`89cab55`](https://github.com/vllm-project/vllm/commit/89cab550862e4b7f08d70c19682121362d7dd0c5)), and no vLLM release includes it yet (the latest is v0.31.0). Until one does, install a current nightly build with [uv](https://docs.astral.sh/uv/); vLLM's nightly indexes do not work with plain `pip`:
 
 ```bash
 uv venv --python 3.12 --seed
@@ -165,9 +165,11 @@ uv pip install --pre vllm --extra-index-url https://wheels.vllm.ai/rocm/nightly/
 The ROCm wheels load OpenMPI and libdw at import, hence the `apt` line (Ubuntu 24.04 package names). Start the server; the first start downloads the model. Then ask questions about a request:
 
 ```bash
-vllm serve vllm-sr/d3
+vllm serve vllm-sr/d3 --max-model-len 32768
 ```
 
+`--max-model-len 32768` is plenty for d3's requests: a request with a full 32-frame video is about 6k tokens. Without it, vLLM takes the 262,144 positions of `config.json` as the limit. d3 reads its prompt with bidirectional attention, so vLLM serves it without chunked prefill and, at startup, profiles one forward pass of the whole limit; at 262,144 tokens that pass does not fit on an 80 GB GPU, even with [d3-FP8](https://huggingface.co/vllm-sr/d3-FP8).
+
 ```bash
 curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
   "model": "vllm-sr/d3",
@@ -196,7 +198,7 @@ The reply gives each question its pick and each option a probability (d3 on one
 
 The probabilities are this package's, as `model.system_one()` gives them, and `confidence` is the chosen option's probability; a yes / no answer's `noul` is the probability of yes. The image and video requests of the next section work unchanged on port 8000: vLLM places the media before the text and reads them with this package's budgets. A top-level `instructions` field returns 400, since the prompt is fixed. vLLM reserves 92 % of the GPU's memory by default; lower `--gpu-memory-utilization` to share the GPU.
 
-**Verified on AMD Instinct MI300X (ROCm).** These commands were run as written in a new Ubuntu 24.04 environment with ROCm 7.2 and the ROCm nightly of 2026-10-15 (`0.32.0rc1.dev190+g3f1bcfd82`). Against this package's runtime (AutoModel with the versions in `requirements.txt`), vLLM made the same decisions on the request above and on the image and video requests of the next section, and on 415 of 416 further questions (text, JSON, a 40-option choice, an image and a video); the other one is a near-tie whose two most likely answers are 0.002 apart. The median probability difference was 0.0025. The CUDA command installs a nightly that includes d3, but it has not been run on an NVIDIA GPU yet.
+**Verified on AMD Instinct MI300X (ROCm).** These commands were run as written, without `--max-model-len`, in a new Ubuntu 24.04 environment with ROCm 7.2 and the ROCm nightly of 2026-10-15 (`0.32.0rc1.dev190+g3f1bcfd82`). Against this package's runtime (AutoModel with the versions in `requirements.txt`), vLLM made the same decisions on the request above and on the image and video requests of the next section, and on 415 of 416 further questions (text, JSON, a 40-option choice, an image and a video); the other one is a near-tie whose two most likely answers are 0.002 apart. The median probability difference was 0.0025. The CUDA command installs a nightly that includes d3, but it has not been run on an NVIDIA GPU yet.
 
 In the measurements of the vLLM pull request, on one AMD Instinct MI325X, vLLM changed none of the 416 decisions and served 1.4× and 1.8× the requests per second of this package's own server (`d3_server.py`) at 8 and 64 concurrent clients; one request at a time it was 3 % slower. It starts more slowly than AutoModel: 215 s to the first answer with empty caches, against 40 s; `--max-model-len 32768` brings that to 139 s.