Add minimal quantization config to correct Hub parameter count

#3
by burtenshaw HF Staff - opened

Summary

Add the minimal top-level quantization metadata in config.json so the Hub reports approximately 594.05B parameters.

+  "quantization_config": {
+    "quant_method": "modelopt",
+    "quant_algo": "NVFP4"
+  }

The Hub's ModelOpt counting rules exclude *_weight_scale tensors and count two FP4 values per packed U8 weight. F8_E4M3 experts in layers 40–47 remain one value per element. This is the minimal counting config; the existing hf_quant_config.json still describes the NVFP4/MXFP8 mix.

Evidence

Ran computeNumOfParamsByDtypeSingleFile from @huggingface/hub@2.17.10 against the real safetensors headers of all 24 shards, pinned to aac974122acd33ddb0d3e1f54433a9c5dd2249ce. No weight payloads were downloaded.

  • Before: 383,072,987,216 parameters; exactly matches the current Hub count.
  • After: 594,050,672,720 parameters (594.05B).

The explicit 48-layer MIXED_PRECISION config produces the same count.

Checked that the JSON is valid and this block is the only semantic change.

Merge Danger

Door: two-way — remove this block to revert.

Blast Radius: config

This validates Hub parameter counting only. Transformers/vLLM inference loading has not been validated. The current Transformers auto-quantizer registry does not register modelopt; maintainers should confirm the Inkling2 loading path accepts this metadata before merging.

Thinking Machines Lab org

Thank you!

aurick changed pull request status to merged

Sign up or log in to comment