Bizuayeu commited on
Commit
f9a0877
·
verified ·
1 Parent(s): 76348dd

Set expandable_segments in the 2 x DGX Spark environment

Browse files

Adds `PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"` to the environment that both DGX Spark nodes share.

In two attempts at this section's manual launch on two GB10 machines (128 GB each), the second with the KV cache fixed as below, each host's available memory fell by 17 to 19 GiB within about 10 seconds after `Loading weights took`, before the server came up. With this setting on both nodes the same start completed and served (MTP 5, answered and called tools). The full report is in [discussion #7](https://huggingface.co/nvidia/GLM-5.3-Flash-FP4-Local/discussions/7).

| Both nodes | Lowest available memory (two hosts) | Result |
|---|---|---|
| Without the setting | 7.8, 7.9 GiB, 5 to 7 s after the weights loaded | stopped by our 8 GiB floor |
| With it | 13.7, 14.5 GiB during the start; 13.9, 15.1 GiB while serving | served |

Both rows used `--gpu-memory-utilization 0.80` (0.87 here), `--max-model-len 131072` (500000) and `--kv-cache-memory-bytes 7650410496`, with the weights at `76348ddb` read offline from the pinned snapshot and `eugr/spark-vllm-b12x@sha256:03a95654…`. Apart from serving on 127.0.0.1 and adding `VLLM_HOST_IP` and `NCCL_IB_GID_INDEX=3` for our direct link, the rest followed this section. The 8 GiB floor is a watchdog of ours; vLLM had logged nothing after `Loading weights took` when the watchdog stopped the start. We have not run this section's own values, with or without the setting.

The RTX PRO 6000 section already sets `expandable_segments:True,large_segment_size_mb:12`; we ran only `expandable_segments:True`. The one-step launcher's recipe does not set it either: https://github.com/eugr/spark-vllm-docker/issues/441

Written with Weave (My Symbiotic AI)

Files changed (1) hide show
  1. README.md +1 -0
README.md CHANGED
@@ -213,6 +213,7 @@ export VLLM_ENABLE_PCIE_ALLREDUCE="0"
213
  export B12X_POLICY_MODE="auto"
214
  export VLLM_B12X_MOE_FP4_FORCE_A16=1
215
  export B12X_W4A16_A4_PREFILL=1
 
216
  ```
217
 
218
  Launch the head node:
 
213
  export B12X_POLICY_MODE="auto"
214
  export VLLM_B12X_MOE_FP4_FORCE_A16=1
215
  export B12X_W4A16_A4_PREFILL=1
216
+ export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
217
  ```
218
 
219
  Launch the head node: