DGX Spark
What just happend?
Previous version (before RC2->main) was working fine, now 3 seconds after vllm start whole 128GB + swap is taken.
I don't know why they merged the branch, they knew there was a concern about the weight size of the RC2, 2 people including me made this remark, It's like loading the FP8/BF16 weights, maybe that's the only solution they've found to make the model work.
Starting to think the people at Poolside don't know what they are doing
I would second the initial comment here - the fact that this could run on a single machine with 128gb ram (with some headroom for OS) was the big reason to use this model (note that I also have a DGX Spark). Losing that reason means I'll look elsewhere for my daily driver. This model has a lot of promise, but we need the weights smaller for this to be viable.
For those who are not happy with the recent changes to main, you can still use the previous revision cache if you had downloaded the model weights using hf. If you don't have cache for some reason, you can download them (assuming you followed the "DGX Spark & Mac Studio" setup on the README page):
uv run hf download poolside/Laguna-S-2.1-NVFP4 --revision 07614121b31898586430f189d27a25a0be310843
uv run hf download poolside/Laguna-S-2.1-DFlash-NVFP4 --revision 4cdcc6e9b29105e8ff5790885cadccbeb4f33f54
Below is my updated script to use the specific revision that worked well for me.
#!/bin/bash
# Environment variables
export HF_HUB_OFFLINE=1
export CUTE_DSL_ARCH=sm_121a # arch string for FP4 kernel JIT
export PATH=/usr/local/cuda/bin:$PATH # nvcc for JIT
export MAX_JOBS=4 # cap JIT fan-out
# Activate virtual environment
source ~/venvs/vllm025/bin/activate
# Start vLLM server
vllm serve poolside/Laguna-S-2.1-NVFP4 \
--revision 07614121b31898586430f189d27a25a0be310843 \
--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","revision":"4cdcc6e9b29105e8ff5790885cadccbeb4f33f54","num_speculative_tokens":15}' \
--enable-auto-tool-choice \
--tool-call-parser poolside_v1 \
--reasoning-parser poolside_v1 \
--override-generation-config '{"temperature":0.7,"top_p":0.95}' \
--max-num-seqs 32 \
--max-model-len 262144 \
--gpu-memory-utilization 0.85 \
--host 0.0.0.0 --port 8000
I have the previous revision downloaded, but since it didn't work, it's not a solution. With how bad the previous revision was looping, this seems like the proverbial rock and a hard place - either it fits in our vram but it loops, or it doesn't fit. Neither is a solution for what I would imagine is the majority of people who are excited for this model - people who want to use it with 128gb of vram.
Quite a disappointment not being able to run the updated weights when the prior checkpoint ran great (for me; no looping). I've reverted to the 076...843 checkpoint as well. Seeing the reports all over X AI that DSV4-0731 XSS quant will run on a single GB10 and outperforms the quant running TP=2, maybe that's a viable alternative. Shame about Laguna, though.
I tried the DS4 XXS and honestly wasn't impressed. I have a test suite I run to try out models and it did way worse than the Qwen 122b. I didn't spend too much time trying to tweak it, but I'd rather be using vllm anyway so I stopped bothering once it didn't wow me.
Hope my script is ok. Was this on purpose?
Using prepopulated Laguna-S-2.1-NVFP4 snapshots:
OLD (July) 07614121b318
NEW (August) f8fdfcdc4e7b
==============================================================================
DIFF
OLD: /raid/models/cache/models--poolside--Laguna-S-2.1-NVFP4/snapshots/07614121b31898586430f189d27a25a0be310843
NEW: /raid/models/cache/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669
==============================================================================
INTEGRITY - OLD
------------------------------------------------------------------------------
shards parsed : 15 / 15
tensors : 145,153
manifest total_size: 66.96 GiB (71,898,733,760 bytes)
tensor payload sum : 66.96 GiB (71,898,733,760 bytes)
payload - manifest : +0.0 MB [OK]
actual file bytes : 66.98 GiB (+18.0 MB vs manifest)
-> within tolerance (72 MB). Expected shortfall
from shard headers alone: ~20 MB (145,153 tensors x ~140 B).
This is NOT a download problem.
INTEGRITY - NEW
------------------------------------------------------------------------------
shards parsed : 49 / 49
tensors : 126,625
manifest total_size: 92.85 GiB (99,697,287,856 bytes)
tensor payload sum : 92.84 GiB (99,681,730,048 bytes)
payload - manifest : -15.6 MB [OK]
actual file bytes : 92.85 GiB (-0.0 MB vs manifest)
-> within tolerance (100 MB). Expected shortfall
from shard headers alone: ~18 MB (126,625 tensors x ~140 B).
This is NOT a download problem.
STORAGE BY DTYPE
------------------------------------------------------------------------------
dtype old new delta
------------ -------------- -------------- --------------
U8 52.875 GiB 43.875 GiB -9.000 GiB
BF16 7.476 GiB 43.476 GiB +36.000 GiB
F8_E4M3 6.609 GiB 5.484 GiB -1.125 GiB
F32 329.0 KB 234.0 KB -95.0 KB
------------ -------------- -------------- --------------
TOTAL 66.961 GiB 92.836 GiB +25.875 GiB
DEQUANTIZATION CHECK
------------------------------------------------------------------------------
U8 (packed FP4) lost : 9.00 GiB
BF16 gained : 36.00 GiB
predicted (x4) : 36.00 GiB
observed / predicted : 1.000 [MATCH]
FP4 packs 2 values/byte; BF16 uses 2 bytes/value, so a pure
FP4 -> BF16 conversion multiplies those bytes by exactly 4.
Byte conservation
BF16 gain - U8 lost - scale lost : +25.87 GiB
actual total delta : +25.87 GiB
unexplained residual : +0.00 GiB [OK]
-> fully accounted for: nothing added or removed beyond
the dequantization itself.
Implied quantization group size
FP4 values removed / scales removed : 16.00
-> NVFP4 uses group size 16. A value near 16 independently
confirms these were genuine NVFP4 groups, not unrelated
tensor churn.
TENSOR COUNT
------------------------------------------------------------------------------
old 145,153 new 126,625 delta -18,528
A LARGER file with FEWER tensors is the signature of quantization
groups collapsing: (weight_packed + weight_scale + weight_global_scale)
-> a single BF16 'weight'.
GROUPS CHANGED IN PLACE (3073 groups, showing 30)
------------------------------------------------------------------------------
Present in BOTH revisions. This is where bytes moved FROM.
component old new delta dtype layers
-------------------------------------------------------------------------------------------------------------------
mlp.experts.84.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.255.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.215.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.174.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.239.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.227.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.58.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.242.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.140.down_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.136.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.104.down_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.214.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.212.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.129.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.245.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.225.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.190.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.26.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.163.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.103.down_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.244.down_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.172.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.209.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.85.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.99.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.120.up_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.241.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.13.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.142.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
mlp.experts.91.gate_proj.weight_packed 70.5 MB 58.5 MB -12.0 MB U8 1-47->1-39
-------------------------------------------------------------------------------------------------------------------
subtotal, all 3073 changed groups -10.125 GiB
GROUPS THAT APPEARED IN NEW (768)
------------------------------------------------------------------------------
Absent from OLD entirely. A renamed group lands here.
count suffix size tensors dtype layers
-------- ------------------------------ ------------ ---------- ---------- ----------
768 *.weight 36.000 GiB 6,144 BF16 40-47
GROUPS THAT VANISHED FROM OLD (2)
------------------------------------------------------------------------------
count suffix size tensors dtype layers
-------- ------------------------------ ------------ ---------- ---------- ----------
1 *.k_scale 96 B 48 BF16 0-47
1 *.v_scale 96 B 48 BF16 0-47
Read together: a '*.weight_packed' shrinking in CHANGED IN PLACE,
paired with a matching '*.weight' under APPEARED, is one
dequantized component.
==============================================================================
SIZE : 66.96 GiB -> 92.84 GiB (+25.87 GiB, +38.6%)
71.9 GB -> 99.7 GB (decimal, as HF reports)
AVG PRECISION: 4.41 -> 6.38 bits/param
==============================================================================
I'm tellen ya, they don't know what they are doing over there at @poolside