google/gemma-4-12B-IT | INT4 (W4A16)
#27
by INC4AI - opened
Pipeline Failure Report
Model: google/gemma-4-12B-IT
Quantization Scheme: INT4 (W4A16)
Failed Phase: quantize
Run ID: gemma-4-12B-IT-AutoRound-W4A16-Tuning
Error Category: transformers_model_code_bug
Full Error Log
00:44:40 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/127bfedcd5047750df5ccf3a12979a47bfa0bafa/.huggingface.yaml "HTTP/1.1 404 Not Found"
00:44:40 [INFO] HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=NeelNanda/pile-10k "HTTP/1.1 200 OK"
00:44:40 [INFO] HTTP Request: GET https://huggingface.co/api/datasets/NeelNanda/pile-10k/tree/127bfedcd5047750df5ccf3a12979a47bfa0bafa/data?recursive=true&expand=false "HTTP/1.1 200 OK"
00:44:40 [INFO] HTTP Request: GET https://huggingface.co/api/datasets/NeelNanda/pile-10k/tree/127bfedcd5047750df5ccf3a12979a47bfa0bafa?recursive=false&expand=false "HTTP/1.1 200 OK"
[33;1m2026-06-30 00:44:40 WARNING logging.py L340: Please note that 'shared_kv_states' key is not currently used in quantization fine-tuning.[0m
[38;20m2026-06-30 00:44:42 INFO data_driven.py L685: caching done[0m
0%| | 0/48 [00:00<?, ?it/s]
Quantizing model.language_model.layers.0: 0%| | 0/48 [00:00<?, ?it/s]/root/.venv/lib/python3.12/site-packages/torch/autograd/graph.py:869: UserWarning: Memory Efficient attention defaults to a non-deterministic algorithm. To explicitly enable determinism call torch.use_deterministic_algorithms(True, warn_only=False). (Triggered internally at /pytorch/aten/src/ATen/native/transformers/cuda/attention_backward.cu:900.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[38;20mquantized 7/7 layers in the block, loss iter 0: 0.047937 -> iter 175: 0.009672[0m
[38;20m2026-06-30 00:46:01 INFO device.py L1840: 'peak_ram': 10.85GB, 'peak_vram': 20.06GB[0m
Quantizing model.language_model.layers.1: 2%|▏ | 1/48 [01:18<1:01:39, 78.71s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.012818 -> iter 197: 0.003223[0m
[38;20m2026-06-30 00:47:19 INFO device.py L1840: 'peak_ram': 11.58GB, 'peak_vram': 20.06GB[0m
Quantizing model.language_model.layers.1: 4%|▍ | 2/48 [02:37<1:00:11, 78.51s/it]
Quantizing model.language_model.layers.2: 4%|▍ | 2/48 [02:37<1:00:11, 78.51s/it]
Quantizing model.language_model.layers.2: 4%|▍ | 2/48 [02:48<1:00:11, 78.51s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.006628 -> iter 181: 0.001251[0m
[38;20m2026-06-30 00:48:38 INFO device.py L1840: 'peak_ram': 12.4GB, 'peak_vram': 20.06GB[0m
Quantizing model.language_model.layers.3: 6%|▋ | 3/48 [03:56<58:53, 78.51s/it] [38;20mquantized 7/7 layers in the block, loss iter 0: 0.010694 -> iter 198: 0.002502[0m
[38;20m2026-06-30 00:49:57 INFO device.py L1840: 'peak_ram': 13.25GB, 'peak_vram': 20.06GB[0m
Quantizing model.language_model.layers.3: 8%|▊ | 4/48 [05:15<57:48, 78.84s/it]
Quantizing model.language_model.layers.4: 8%|▊ | 4/48 [05:15<57:48, 78.84s/it]
Quantizing model.language_model.layers.4: 8%|▊ | 4/48 [05:28<57:48, 78.84s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.005018 -> iter 194: 0.001237[0m
[38;20m2026-06-30 00:51:16 INFO device.py L1840: 'peak_ram': 14.15GB, 'peak_vram': 20.06GB[0m
Quantizing model.language_model.layers.5: 10%|█ | 5/48 [06:34<56:30, 78.84s/it]00:51:16 [ERROR] Quantization failed: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3
Traceback (most recent call last):
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 282, in <module>
quantize(args)
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 183, in quantize
autoround.quantize()
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 722, in quantize
self._quantize_blocks(
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 529, in _quantize_blocks
reference_output = self.quantizer._get_block_outputs(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/algorithms/quantization/base.py", line 452, in _get_block_outputs
tmp_output = _bf(
^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/utils.py", line 208, in block_forward
output = block(**input_others)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_layers.py", line 93, in __call__
return super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 516, in forward
hidden_states, _ = self.self_attn(
^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 421, in forward
query_states = apply_rotary_pos_emb(query_states, cos, sin, unsqueeze_dim=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 304, in apply_rotary_pos_emb
return (x * cos) + (rotate_half(x) * sin)
~~^~~~~
RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3
Quantizing model.language_model.layers.5: 10%|█ | 5/48 [06:34<56:33, 78.93s/it]
Auto-generated by error_analysis pipeline. cc @lvkaokao