Device assertion errors with transformers
#1
by hell0ks - opened
Hello,
I tried run the model with transformers library, but I'm getting error messages like below:
/pytorch/aten/src/ATen/native/cuda/IndexKernelUtils.cu:16: vectorized_gather_kernel: block: [219,0,0], thread: [192,0,0] Assertion
`ind >=0 && ind < ind_dim_size && "vectorized gather kernel index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernelUtils.cu:16: vectorized_gather_kernel: block: [219,0,0], thread: [193,0,0] Assertion
`ind >=0 && ind < ind_dim_size && "vectorized gather kernel index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernelUtils.cu:16: vectorized_gather_kernel: block: [219,0,0], thread: [194,0,0] Assertion
`ind >=0 && ind < ind_dim_size && "vectorized gather kernel index out of bounds"` failed.
....
│ /home/user01/transfr-env/lib/python3.12/site-packages/transformers/utils/generic.py:918 in │
│ wrapper │
│ │
│ 915 │ │ return_dict_passed = kwargs.pop("return_dict", return_dict) │
│ 916 │ │ if return_dict_passed is not None: │
│ 917 │ │ │ return_dict = return_dict_passed │
│ ❱ 918 │ │ output = func(self, *args, **kwargs) │
│ 919 │ │ if not return_dict and not isinstance(output, tuple): │
│ 920 │ │ │ output = output.to_tuple() │
│ 921 │ │ return output │
│ │
│ /home/user01/.cache/huggingface/modules/transformers_modules/tri21b_hyphen_think/modeling_trill │
│ ion.py:556 in forward │
│ │
│ 553 │ │ if position_ids is None: │
│ 554 │ │ │ position_ids = cache_position.unsqueeze(0) │
│ 555 │ │ │
│ ❱ 556 │ │ causal_mask = self._update_causal_mask( │
│ 557 │ │ │ attention_mask, inputs_embeds, cache_position, past_key_values, output_atten │
│ 558 │ │ ) │
│ 559 │
│ │
│ /home/user01/.cache/huggingface/modules/transformers_modules/tri21b_hyphen_think/modeling_trill │
│ ion.py:641 in _update_causal_mask │
│ │
│ 638 │ │ │
│ 639 │ │ # When output attentions is True, sdpa implementation's forward method calls the │
│ 640 │ │ if self.config._attn_implementation == "sdpa" and not using_static_cache and not │
│ ❱ 641 │ │ │ if AttentionMaskConverter._ignore_causal_mask_sdpa( │
│ 642 │ │ │ │ attention_mask, │
│ 643 │ │ │ │ inputs_embeds=input_tensor, │
│ 644 │ │ │ │ past_key_values_length=past_seen_tokens, │
│ │
│ /home/user01/transfr-env/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py: │
│ 294 in _ignore_causal_mask_sdpa │
│ │
│ 291 │ │ elif sliding_window is None or key_value_length < sliding_window: │
│ 292 │ │ │ if len(attention_mask.shape) == 4: │
│ 293 │ │ │ │ return False │
│ ❱ 294 │ │ │ elif not is_tracing and torch.all(attention_mask == 1): │
│ 295 │ │ │ │ if query_length == 1 or key_value_length == query_length: │
│ 296 │ │ │ │ │ # For query_length == 1, causal attention and bi-directional attenti │
│ 297 │ │ │ │ │ ignore_causal_mask = True
...
AcceleratorError: CUDA error: device-side assert triggered
Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Other model runs just fine with same environment and hardware, so I thought it's better to open an disscussion here.
Added:
It occurs on batch size > 2. Batch size = 1 seems work just fine.
Seems like vocab size is different for some reason
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Tri-21B-Think")
print(len(tokenizer))
124418
config.json
"vocab_size": 124416
Which one is correct?
Thanks, I think batch inference works well after re-download.
hell0ks changed discussion status to closed