Spaces:

---

Intel

/

low_bit_open_llm_leaderboard

like 222

Running

Pipeline Failure Report

Model: deepseek-ai/DeepSeek-V4-Flash
Quantization Scheme: INT4 (W4A16)
Failed Phase: evaluate
Run ID: DeepSeek-V4-Flash-AutoRound-W4A16-RTN
Error Category: out_of_memory

Full Error Log

  batch_size=auto
  num_gpus=1
[evaluate] Running lm_eval with hf backend...
2026-08-03:16:00:35 INFO     [_cli.run:388] Selected Tasks: ['piqa', 'mmlu', 'hellaswag']
2026-08-03:16:00:36 INFO     [evaluator:214] Setting random seed to 42 | Setting numpy seed to 42 | Setting torch manual seed to 42 | Setting fewshot manual seed to 42
2026-08-03:16:00:36 INFO     [evaluator:239] Initializing hf model, with arguments: {'pretrained': '/root/_work/1/s/auto_quant/output/runs/DeepSeek-V4-Flash-AutoRound-W4A16-RTN/quantized_model', 'dtype': 'bfloat16', 'trust_remote_code': True}
2026-08-03:16:00:40 INFO     [models.huggingface:286] Using device 'cuda:0'
2026-08-03:16:00:41 INFO     [models.huggingface:579] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
/root/.venv/lib/python3.12/site-packages/transformers/quantizers/auto.py:271: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.However, loading attributes (e.g. ['backend']) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
  warnings.warn(warning_msg)
2026-08-03 16:00:42 INFO replace_modules.py L121: Experts (before replacement) [model.layers.0.mlp.experts] (DeepseekV4Experts):
DeepseekV4Experts(
  (act_fn): SiLUActivation()
)
2026-08-03 16:00:42 WARNING modeling_utils.py L4770: `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
2026-08-03 16:00:42 INFO device.py L1448: Before applying custom replacements 'peak_ram': 1.11GB
2026-08-03 16:00:49 INFO moe_experts_interface.py L655: [MoE Prep] Unfused 43 MOE experts modules
2026-08-03 16:00:49 INFO device.py L1448: After applying custom replacements 'peak_ram': 1.2GB
2026-08-03 16:00:49 INFO replace_modules.py L94: Prepared 43 MOE modules for quantization
2026-08-03 16:00:49 INFO replace_modules.py L121: Experts (after replacement/skip) [model.layers.0.mlp.experts] (DeepseekV4Experts):
DeepseekV4Experts(
  (act_fn): SiLUActivation()
  (0-255): 256 x _ExpertContainer(
    (down_proj): Linear(in_features=2048, out_features=4096, bias=False)
    (gate_proj): Linear(in_features=4096, out_features=2048, bias=False)
    (up_proj): Linear(in_features=4096, out_features=2048, bias=False)
  )
)
2026-08-03 16:01:09 WARNING backend.py L1176: Better backend is found, please install all the following requirements to enable it.
2026-08-03 16:01:09 WARNING backend.py L1176: `pip install -v "gptqmodel>=2.0" --no-build-isolation`
Traceback (most recent call last):
  File "/root/.venv/bin/lm_eval", line 10, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/__main__.py", line 10, in cli_evaluate
    parser.execute(args)
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/_cli/harness.py", line 60, in execute
    args.func(args)
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/_cli/run.py", line 391, in _execute
    results = simple_evaluate(
              ^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/utils.py", line 575, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/evaluator.py", line 242, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/api/model.py", line 169, in create_from_arg_obj
    return cls(**arg_dict, **additional_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 365, in __init__
    self._create_model(
  File "/root/.venv/lib/python3.12/site-packages/lm_eval/models/huggingface.py", line 814, in _create_model
    self._model = self.AUTO_MODEL_CLASS.from_pretrained(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 402, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4456, in from_pretrained
    loading_info, disk_offload_index = cls._load_pretrained_model(model, state_dict, checkpoint_files, load_config)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4590, in _load_pretrained_model
    loading_info, disk_offload_index = convert_and_load_state_dict_in_model(
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1695, in convert_and_load_state_dict_in_model
    realized_value = mapping.convert(
                     ^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 990, in convert
    collected_tensors = self.materialize_tensors()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 952, in materialize_tensors
    tensors = [future.result() for future in tensors if future.result() is not None]
                                                        ^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1239, in _job
    return _materialize_copy(tensor, device, dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/core_model_loading.py", line 1217, in _materialize_copy
    tensor = tensor.to(device=device, dtype=dtype)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 139.81 GiB of which 1.88 MiB is free. Process 16653 has 612.00 MiB memory in use. Including non-PyTorch memory, this process has 139.20 GiB memory in use. Of the allocated memory 2.05 GiB is allocated by PyTorch, and 136.64 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

Auto-generated by error_analysis pipeline. cc @lvkaokao