Salyut1

/

GLM-4.7-NVFP4

like 5

Can you confirm whether you calibrate all experts or not?

When looking at ModelOpt DeepSeek example, specific snippets of code are needed to ensure that all experts see the calibration samples: https://github.com/NVIDIA/Model-Optimizer/blob/0.40.0/examples/deepseek/ptq.py#L201-L219

    class CalibMoe(deekseep_model.MoE):
        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self._setup()

        def _setup(self):
            self._original_topk = self.gate.topk
            self._original_topk_groups = self.gate.topk_groups

        def forward(self, x: torch.Tensor) -> torch.Tensor:
            # Forward all tokens to all experts for calibration
            self.gate.topk = self.n_routed_experts
            self.gate.topk_groups = self.gate.n_groups
            super().forward(x)
            # Restore the original topk and topk_groups
            self.gate.topk = self._original_topk
            self.gate.topk_groups = self._original_topk_groups

            return super().forward(x)

Otherwise, I suspect some experts are quantized very naively, significantly impacting quality for use-cases not in the calibration data.