vLLM cannot load this checkpoint: inherited DeepseekV2Config validator rejects num_attention_heads=96
Thanks for publishing this speculator. Reporting an interop problem for other users' benefit,
plus one question for you.
Symptom
Serving this draft with vLLM fails at config-parse time, before anything touches a GPU:
ValueError: The hidden size (7168) is not a multiple of the number of attention heads (96).
This is not a problem with your checkpoint (I think)
vLLM's K3DSparkConfig subclasses transformers.DeepseekV2Config, which carries a @strict
validator (validate_architecture) enforcing the multi-head invarianthead_dim == hidden_size / num_heads. MLA does not obey that invariant.
Your config.json declares qk_head_dim: 192 outright, and the tensor shapes inmodel.safetensors agree at 96 heads:
| tensor | shape | equals |
|---|---|---|
layers.N.self_attn.q_b_proj.weight |
[18432, 1536] |
96 x 192 (num_heads x qk_head_dim) |
layers.N.self_attn.kv_b_proj.weight |
[24576, 512] |
96 x 256 (num_heads x (qk_nope + v_head_dim)) |
layers.N.self_attn.kv_a_proj_with_mqa.weight |
[576, 7168] |
512 + 64 (kv_lora_rank + qk_rope) |
layers.N.self_attn.o_proj.weight |
[7168, 12288] |
96 x 128 (num_heads x v_head_dim) |
hidden_size (7168) is nowhere equal to num_heads x head_dim — o_proj takes 12288 in and
projects down to 7168. So 7168/96 = 74.67 is not a meaningful quantity for this architecture,
and 96 heads is self-consistent throughout.
Filed upstream with the full analysis: https://github.com/vllm-project/vllm/issues/53998
Workaround for anyone hitting this before a fix lands
At import time in vllm/transformers_utils/configs/k3_dspark.py:
DeepseekV2Config.__class_validators__ = [
v for v in DeepseekV2Config.__class_validators__
if v.__name__ != "validate_architecture"
]
Note that overriding validate_architecture on the subclass does not work —huggingface_hub's @strict collects validators by value and calls them throughcls.validate(self) bound to the decorated class, so MRO is never consulted.
With the above, this checkpoint's config loads cleanly: qk_head_dim=192, head_dim=64,
KV head dim 576.
One question
Is there a minimum vLLM version or patch set you test this against? If you have a known-good
combination it would be worth a line in the README, since the error message points at the
config rather than at the loader and reads as though the checkpoint is malformed.
Minor note
transformers 5.15.0 emits Unrecognized keys in rope_parameters for 'rope_type'='yarn': {'attn_factor'} when loading this config, just FYI.
Produced with AI assistance.