Spaces:

---

Intel

/

low_bit_open_llm_leaderboard

like 219

Running

Pipeline Failure Report

Model: google/gemma-4-12B
Quantization Scheme: INT4 (W4A16)
Failed Phase: quantize
Run ID: gemma-4-12B-AutoRound-W4A16-RTN
Error Category: model_forward_mismatch

Full Error Log

2026-06-30 02:19:00 INFO mllm.py L83: Using MLLM template: gemma4_unified
2026-06-30 02:19:00 INFO calib_dataset.py L977: Preprocessing calibration dataset in a subprocess to avoid memory leaks...
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/main/README.md "HTTP/1.1 307 Temporary Redirect"
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/datasets/NeelNanda/pile-10k/127bfedcd5047750df5ccf3a12979a47bfa0bafa/README.md "HTTP/1.1 200 OK"
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/127bfedcd5047750df5ccf3a12979a47bfa0bafa/pile-10k.py "HTTP/1.1 404 Not Found"
02:19:00 [INFO] HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/NeelNanda/pile-10k/NeelNanda/pile-10k.py "HTTP/1.1 404 Not Found"
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/127bfedcd5047750df5ccf3a12979a47bfa0bafa/.huggingface.yaml "HTTP/1.1 404 Not Found"
02:19:00 [INFO] HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=NeelNanda/pile-10k "HTTP/1.1 200 OK"
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/main/README.md "HTTP/1.1 307 Temporary Redirect"
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/api/resolve-cache/datasets/NeelNanda/pile-10k/127bfedcd5047750df5ccf3a12979a47bfa0bafa/README.md "HTTP/1.1 200 OK"
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/127bfedcd5047750df5ccf3a12979a47bfa0bafa/pile-10k.py "HTTP/1.1 404 Not Found"
02:19:00 [INFO] HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/NeelNanda/pile-10k/NeelNanda/pile-10k.py "HTTP/1.1 404 Not Found"
02:19:00 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/127bfedcd5047750df5ccf3a12979a47bfa0bafa/.huggingface.yaml "HTTP/1.1 404 Not Found"
02:19:00 [INFO] HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=NeelNanda/pile-10k "HTTP/1.1 200 OK"
2026-06-30 02:19:01 WARNING logging.py L340: Please note that 'shared_kv_states' key is not currently used in quantization fine-tuning.

  0%|          | 0/48 [00:00<?, ?it/s]
Quantizing model.language_model.layers.0:   0%|          | 0/48 [00:00<?, ?it/s]2026-06-30 02:19:15 WARNING logging.py L340: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0
2026-06-30 02:19:16 INFO offload.py L706: OffloadManager (compressor): tempdir = /root/_work/1/s/auto_quant/ar_work_space/offload/compressor_2s2ltbnk
2026-06-30 02:19:16 INFO device.py L1840: 'peak_ram': 8.81GB, 'peak_vram': 20.51GB

Quantizing model.language_model.layers.1:   2%|▏         | 1/48 [00:09<07:11,  9.18s/it]2026-06-30 02:19:24 INFO device.py L1840: 'peak_ram': 9.27GB, 'peak_vram': 20.51GB

Quantizing model.language_model.layers.2:   4%|▍         | 2/48 [00:17<06:34,  8.58s/it]2026-06-30 02:19:33 INFO device.py L1840: 'peak_ram': 9.64GB, 'peak_vram': 20.51GB

Quantizing model.language_model.layers.3:   6%|▋         | 3/48 [00:25<06:21,  8.47s/it]2026-06-30 02:19:41 INFO device.py L1840: 'peak_ram': 10.06GB, 'peak_vram': 20.51GB

Quantizing model.language_model.layers.4:   8%|▊         | 4/48 [00:33<06:09,  8.40s/it]2026-06-30 02:19:49 INFO device.py L1840: 'peak_ram': 10.51GB, 'peak_vram': 20.51GB

Quantizing model.language_model.layers.5:  10%|█         | 5/48 [00:42<06:02,  8.43s/it]02:19:50 [ERROR] Quantization failed: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3
Traceback (most recent call last):
  File "/root/_work/1/s/auto_quant/phases/quantize.py", line 282, in <module>
    quantize(args)
  File "/root/_work/1/s/auto_quant/phases/quantize.py", line 183, in quantize
    autoround.quantize()
  File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 1149, in quantize
    return self._quantize_impl()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 1175, in _quantize_impl
    self._quant_rtn_with_imatrix()
  File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 1109, in _quant_rtn_with_imatrix
    self._quantize_via_rtn_blockwise()
  File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 1027, in _quantize_via_rtn_blockwise
    input_ids = self.quantizer._get_block_outputs(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/auto_round/algorithms/quantization/base.py", line 452, in _get_block_outputs
    tmp_output = _bf(
                 ^^^^
  File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/utils.py", line 208, in block_forward
    output = block(**input_others)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_layers.py", line 93, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 516, in forward
    hidden_states, _ = self.self_attn(
                       ^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 421, in forward
    query_states = apply_rotary_pos_emb(query_states, cos, sin, unsqueeze_dim=2)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 304, in apply_rotary_pos_emb
    return (x * cos) + (rotate_half(x) * sin)
            ~~^~~~~
RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3

Quantizing model.language_model.layers.5:  10%|█         | 5/48 [00:42<06:06,  8.51s/it]

Auto-generated by error_analysis pipeline. cc @lvkaokao