Salyut1

/

GLM-4.7-NVFP4

like 16

same issue

I had the exact same error when I attempted to run inference on vLLM. There's a difference in how the model was quantized and how vLLM tries to load it. The NVFP4 checkpoint uses a fused QKV quantization scheme. The Query, Key, and Value projections are treated as a single block (or partly fused), meaning there is likely one global scale (or input scale) for the layer, but no specific, separate scales for the Key (k_scale) or Value (v_scale) heads.

The current Glm4MoeForCausalLM loader in vLLM assumes a Split QKV scheme for NVFP4. vLLM iterates through the state dict looking for separate k_scale and v_scale tensors to load into the attention backend. When it attempts to access these keys in the safetensors file, they don't exist, triggering the crash.

What I did to fix it was modify vllm/model_executor/models/glm4_moe.py to skip the specific k_scale and v_scale parameters if they are missing from the checkpoint, rather than crashing.

Here's my implementation that got it running on my dual DGX Spark setup:

import sys
import os
import re

# Path to the vLLM model file
path = '/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py'

if os.path.exists(path):
    with open(path, 'r') as f:
        lines = f.readlines()
    
    target_str = 'param = params_dict[name]'
    new_lines = []
    patched = False
    
    for line in lines:
        # We look for the parameter loading line
        if target_str in line and 'k_scale' not in line:
            whitespace = re.match(r'^(\s*)', line).group(1)
            
            # Inject logic: If asking for k_scale/v_scale and it's missing, skip
            payload = f"{whitespace}if ('k_scale' in name or 'v_scale' in name) and name not in params_dict: continue\n"
            
            new_lines.append(payload)
            new_lines.append(line)
            patched = True
        else:
            new_lines.append(line)
            
    if patched:
        with open(path, 'w') as f:
            f.writelines(new_lines)
        print(f"Successfully patched {path}")
    else:
        print("File already patched or target not found.")

OK, thanks, I'll try it.
Here is a diff that you can feed to patch command:

--- a/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py
+++ b/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py
@@ -537,6 +537,7 @@
                 if is_pp_missing_parameter(name, self):
                     continue

+                if ('k_scale' in name or 'v_scale' in name) and name not in params_dict: continue
                 param = params_dict[name]
                 weight_loader = param.weight_loader
                 weight_loader(param, loaded_weight, shard_id)
@@ -596,6 +597,7 @@
                     if is_pp_missing_parameter(name, self):
                         continue

+                    if ('k_scale' in name or 'v_scale' in name) and name not in params_dict: continue
                     param = params_dict[name]
                     weight_loader = getattr(
                         param, "weight_loader", default_weight_loader

to patch with just one command:

cat <<'EOM' | patch -p1 -d /
--- a/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py
+++ b/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/glm4_moe.py
@@ -537,6 +537,7 @@
                 if is_pp_missing_parameter(name, self):
                     continue

+                if ('k_scale' in name or 'v_scale' in name) and name not in params_dict: continue
                 param = params_dict[name]
                 weight_loader = param.weight_loader
                 weight_loader(param, loaded_weight, shard_id)
@@ -596,6 +597,7 @@
                     if is_pp_missing_parameter(name, self):
                         continue

+                    if ('k_scale' in name or 'v_scale' in name) and name not in params_dict: continue
                     param = params_dict[name]
                     weight_loader = getattr(
                         param, "weight_loader", default_weight_loader
EOM

With this patch I can confirm that it's working.
Do you think this modification could negatively affect other quants? If not, you may consider submitting a pull request to vLLM.

It shouldn't affect other quants as it just skips missing k and v scale params instead of hard-crashing if there's a missing value. It could introduce instability if the model is actually broken though. I'll swing by vLLM and see if they have anything like this cooked up already.

Is this fix merged?

Check out my fix and let me know if this works: https://github.com/SILVERTHRONE32/nvfp4-fix

Following up on this thread — the k/v-scale fix here got me most of the way, but on
vLLM 0.23.1rc1.dev448 (eugr's vllm-node-tf5 image for DGX Spark) there's one more wall
after it:

TypeError: KVCacheScaleParameter.weight_loader() takes 2 positional arguments but 3
were given

glm4_moe.py:527 passes a third shard_id positional, but
KVCacheScaleParameter.weight_loader (in
vllm/model_executor/layers/quantization/kv_cache.py) is a staticmethod taking only
(param, loaded_weight). Since that loader handles a scalar scale (per-head scales
route elsewhere per its own docstring), ignoring the extra arg is safe:

@staticmethod
def weight_loader(param: torch.nn.Parameter,
loaded_weight: torch.Tensor, *args) -> None:

With that one-line change (plus --no-ray to avoid a Ray dual-session collision with
the eugr launcher, and --max-num-batched-tokens 8192 --max-num-seqs 8 with
--gpu-memory-utilization 0.90 to fit the KV budget while keeping CUDAGraph), the full
355B loads and serves on 2× DGX Spark GB10 (TP=2) at 64K context — coherent
completions, working tool calls, passed long-context needle recall at 21K tokens, and
~17.5 tok/s.

Happy to share the full four-walls write-up if it's useful to anyone.

— Cecil Ray Burnett III (Leathery Tendons)