Set expandable_segments in the 2 x DGX Spark environment
Browse filesAdds `PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"` to the environment that both DGX Spark nodes share.
In two attempts at this section's manual launch on two GB10 machines (128 GB each), the second with the KV cache fixed as below, each host's available memory fell by 17 to 19 GiB within about 10 seconds after `Loading weights took`, before the server came up. With this setting on both nodes the same start completed and served (MTP 5, answered and called tools). The full report is in [discussion #7](https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local/discussions/7).
| Both nodes | Lowest available memory (two hosts) | Result |
|---|---|---|
| Without the setting | 7.8, 7.9 GiB, 5 to 7 s after the weights loaded | stopped by our 8 GiB floor |
| With it | 13.7, 14.5 GiB during the start; 13.9, 15.1 GiB while serving | served |
Both rows used `--gpu-memory-utilization 0.80` (0.87 here), `--max-model-len 131072` (500000) and `--kv-cache-memory-bytes 7650410496`, with the weights at `76348ddb` read offline from the pinned snapshot and `eugr/spark-vllm-b12x@sha256:03a95654…`. Apart from serving on 127.0.0.1 and adding `VLLM_HOST_IP` and `NCCL_IB_GID_INDEX=3` for our direct link, the rest followed this section. The 8 GiB floor is a watchdog of ours; vLLM had logged nothing after `Loading weights took` when the watchdog stopped the start. We have not run this section's own values, with or without the setting.
The RTX PRO 6000 section already sets `expandable_segments:True,large_segment_size_mb:12`; we ran only `expandable_segments:True`. The one-step launcher's recipe does not set it either: https://github.com/eugr/spark-vllm-docker/issues/441
Written with Weave (My Symbiotic AI)
|
@@ -213,6 +213,7 @@ export VLLM_ENABLE_PCIE_ALLREDUCE="0"
|
|
| 213 |
export B12X_POLICY_MODE="auto"
|
| 214 |
export VLLM_B12X_MOE_FP4_FORCE_A16=1
|
| 215 |
export B12X_W4A16_A4_PREFILL=1
|
|
|
|
| 216 |
```
|
| 217 |
|
| 218 |
Launch the head node:
|
|
|
|
| 213 |
export B12X_POLICY_MODE="auto"
|
| 214 |
export VLLM_B12X_MOE_FP4_FORCE_A16=1
|
| 215 |
export B12X_W4A16_A4_PREFILL=1
|
| 216 |
+
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
|
| 217 |
```
|
| 218 |
|
| 219 |
Launch the head node:
|