Spaces:
Runtime error
Runtime error
google/gemma-4-12b | INT4 (W4A16)
#28
by INC4AI - opened
Pipeline Failure Report
Model: google/gemma-4-12b
Quantization Scheme: INT4 (W4A16)
Failed Phase: quantize
Run ID: gemma-4-12b-AutoRound-W4A16-Tuning
Error Category: quantization_runtime_error
Full Error Log
00:35:42 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/127bfedcd5047750df5ccf3a12979a47bfa0bafa/.huggingface.yaml "HTTP/1.1 404 Not Found"
00:35:42 [INFO] HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=NeelNanda/pile-10k "HTTP/1.1 200 OK"
00:35:42 [INFO] HTTP Request: GET https://huggingface.co/api/datasets/NeelNanda/pile-10k/tree/127bfedcd5047750df5ccf3a12979a47bfa0bafa/data?recursive=true&expand=false "HTTP/1.1 200 OK"
00:35:42 [INFO] HTTP Request: GET https://huggingface.co/api/datasets/NeelNanda/pile-10k/tree/127bfedcd5047750df5ccf3a12979a47bfa0bafa?recursive=false&expand=false "HTTP/1.1 200 OK"
[33;1m2026-06-30 00:35:42 WARNING logging.py L340: Please note that 'shared_kv_states' key is not currently used in quantization fine-tuning.[0m
[38;20m2026-06-30 00:35:49 INFO data_driven.py L685: caching done[0m
0%| | 0/48 [00:00<?, ?it/s]
Quantizing model.language_model.layers.0: 0%| | 0/48 [00:01<?, ?it/s]/root/.venv/lib/python3.12/site-packages/torch/autograd/graph.py:823: UserWarning: Memory Efficient attention defaults to a non-deterministic algorithm. To explicitly enable determinism call torch.use_deterministic_algorithms(True, warn_only=False). (Triggered internally at /pytorch/aten/src/ATen/native/transformers/cuda/attention_backward.cu:683.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[38;20mquantized 7/7 layers in the block, loss iter 0: 0.053519 -> iter 175: 0.010533[0m
[38;20m2026-06-30 00:37:28 INFO device.py L1840: 'peak_ram': 10.54GB, 'peak_vram': 20.49GB[0m
Quantizing model.language_model.layers.1: 2%|▏ | 1/48 [01:39<1:17:43, 99.23s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.016605 -> iter 197: 0.003861[0m
[38;20m2026-06-30 00:39:05 INFO device.py L1840: 'peak_ram': 11.37GB, 'peak_vram': 20.49GB[0m
Quantizing model.language_model.layers.1: 4%|▍ | 2/48 [03:15<1:14:59, 97.81s/it]
Quantizing model.language_model.layers.2: 4%|▍ | 2/48 [03:15<1:14:59, 97.81s/it]
Quantizing model.language_model.layers.2: 4%|▍ | 2/48 [03:30<1:14:59, 97.81s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.006385 -> iter 167: 0.001268[0m
[38;20m2026-06-30 00:40:44 INFO device.py L1840: 'peak_ram': 12.29GB, 'peak_vram': 20.49GB[0m
Quantizing model.language_model.layers.3: 6%|▋ | 3/48 [04:55<1:13:21, 97.81s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.008098 -> iter 186: 0.001728[0m
[38;20m2026-06-30 00:42:25 INFO device.py L1840: 'peak_ram': 13.06GB, 'peak_vram': 20.49GB[0m
Quantizing model.language_model.layers.3: 8%|▊ | 4/48 [06:35<1:12:39, 99.08s/it]
Quantizing model.language_model.layers.4: 8%|▊ | 4/48 [06:35<1:12:39, 99.08s/it]
Quantizing model.language_model.layers.4: 8%|▊ | 4/48 [06:50<1:12:39, 99.08s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.008744 -> iter 194: 0.001764[0m
[38;20m2026-06-30 00:44:09 INFO device.py L1840: 'peak_ram': 13.9GB, 'peak_vram': 20.49GB[0m
Quantizing model.language_model.layers.5: 10%|█ | 5/48 [08:19<1:11:00, 99.08s/it]00:44:09 [ERROR] Quantization failed: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3
Traceback (most recent call last):
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 282, in <module>
quantize(args)
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 183, in quantize
autoround.quantize()
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 722, in quantize
self._quantize_blocks(
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 529, in _quantize_blocks
reference_output = self.quantizer._get_block_outputs(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/algorithms/quantization/base.py", line 452, in _get_block_outputs
tmp_output = _bf(
^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/utils.py", line 208, in block_forward
output = block(**input_others)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_layers.py", line 93, in __call__
return super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 516, in forward
hidden_states, _ = self.self_attn(
^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 421, in forward
query_states = apply_rotary_pos_emb(query_states, cos, sin, unsqueeze_dim=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 304, in apply_rotary_pos_emb
return (x * cos) + (rotate_half(x) * sin)
~~^~~~~
RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3
Quantizing model.language_model.layers.5: 10%|█ | 5/48 [08:20<1:11:40, 100.01s/it]
Auto-generated by error_analysis pipeline. cc @lvkaokao
In another discussion
lvkaokao changed discussion status to closed