ISSUE #44841: Processor fails for mistralai/Voxtral-Mini-3B-2507
State: closed  |  Labels: bug
Author: BhavyaShah1234  |  Created: 2026-03-19T02:26:39Z

### System Info

I am trying to run inference using `mistralai/Voxtral-Mini-3B-2507` on an audio (`np.ndarray`). On loading the processor using `processor = transformers.AutoProcessor.from_pretrained(MODEL, trust_remote_code=True)`, I am getting the following error:
```
/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning:
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
  warnings.warn(
Converting tekken.json to tokenizer.json: 100%|██████████| 150000/150000 [00:02<00:00, 69142.59it/s]
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
[/tmp/ipykernel_7845/2744582412.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in <cell line: 0>()
     12 # else:
     13 #     processor = MODEL_DICT[MODEL]['processor'].from_pretrained(MODEL, trust_remote_code=True)
---> 14 processor = MODEL_DICT[MODEL]['processor'].from_pretrained(MODEL, trust_remote_code=True)

13 frames[/usr/local/lib/python3.12/dist-packages/transformers/models/auto/processing_auto.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    409             )
    410         elif processor_class is not None:
--> 411             return processor_class.from_pretrained(
    412                 pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
    413             )

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, **kwargs)
   1402         processor_dict, instantiation_kwargs = cls.get_processor_dict(pretrained_model_name_or_path, **kwargs)
   1403         args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs)
-> 1404         return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
   1405
   1406     @classmethod

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in from_args_and_dict(cls, args, processor_dict, **kwargs)
   1171         processor = cls(*args, **valid_kwargs)
   1172
-> 1173         logger.info(f"Processor {processor}")
   1174         if return_unused_kwargs:
   1175             return processor, unused_kwargs

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in __repr__(self)
    777         attributes_repr = [f"- {name}: {repr(getattr(self, name))}" for name in self.get_attributes()]
    778         attributes_repr = "\n".join(attributes_repr)
--> 779         return f"{self.__class__.__name__}:\n{attributes_repr}\n\n{self.to_json_string()}"
    780
    781     def save_pretrained(self, save_directory, push_to_hub: bool = False, **kwargs):

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in to_json_string(self)
    759             `str`: String containing all the attributes that make up this feature_extractor instance in JSON format.
    760         """
--> 761         dictionary = self.to_dict()
    762
    763         return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in to_dict(self)
    697             `dict[str, Any]`: Dictionary of all the attributes that make up this processor instance.
    698         """
--> 699         output = copy.deepcopy(self.__dict__)
    700
    701         # Get the kwargs in `__init__`.

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    134     copier = _deepcopy_dispatch.get(cls)
    135     if copier is not None:
--> 136         y = copier(x, memo)
    137     else:
    138         if issubclass(cls, type):

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _deepcopy_dict(x, memo, deepcopy)
    219     memo[id(x)] = y
    220     for key, value in x.items():
--> 221         y[deepcopy(key, memo)] = deepcopy(value, memo)
    222     return y
    223 d[dict] = _deepcopy_dict

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    160                     y = x
    161                 else:
--> 162                     y = _reconstruct(x, memo, *rv)
    163
    164     # If is its own copy, don't memoize.

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    257     if state is not None:
    258         if deep:
--> 259             state = deepcopy(state, memo)
    260         if hasattr(y, '__setstate__'):
    261             y.__setstate__(state)

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    134     copier = _deepcopy_dispatch.get(cls)
    135     if copier is not None:
--> 136         y = copier(x, memo)
    137     else:
    138         if issubclass(cls, type):

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _deepcopy_dict(x, memo, deepcopy)
    219     memo[id(x)] = y
    220     for key, value in x.items():
--> 221         y[deepcopy(key, memo)] = deepcopy(value, memo)
    222     return y
    223 d[dict] = _deepcopy_dict

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    160                     y = x
    161                 else:
--> 162                     y = _reconstruct(x, memo, *rv)
    163
    164     # If is its own copy, don't memoize.

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    259             state = deepcopy(state, memo)
    260         if hasattr(y, '__setstate__'):
--> 261             y.__setstate__(state)
    262         else:
    263             if isinstance(state, tuple) and len(state) == 2:

Exception: Error while attempting to unpickle Tokenizer: Token `Ġ<SPECIAL_116>` out of vocabulary at line 1 column 9601910
```

I tested out the following standalone code provided in the model card of `mistralai/Voxtral-Mini-3B-2507` but it too does not work:
```
from transformers import VoxtralForConditionalGeneration, AutoProcessor
import torch

device = "cuda"
repo_id = "mistralai/Voxtral-Mini-3B-2507"

processor = AutoProcessor.from_pretrained(repo_id)
model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/winning_call.mp3",
            },
        ],
    }
]

inputs = processor.apply_chat_template(conversation)
inputs = inputs.to(device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=500)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("\nGenerated response:")
print("=" * 80)
print(decoded_outputs[0])
print("=" * 80)
```
Error given by above script:
```
Converting tekken.json to tokenizer.json: 100%|██████████| 150000/150000 [00:01<00:00, 116597.64it/s]
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
[/tmp/ipykernel_7845/3351359521.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in <cell line: 0>()
      5 repo_id = "mistralai/Voxtral-Mini-3B-2507"
      6
----> 7 processor = AutoProcessor.from_pretrained(repo_id)
      8 model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)
      9

13 frames[/usr/local/lib/python3.12/dist-packages/transformers/models/auto/processing_auto.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    409             )
    410         elif processor_class is not None:
--> 411             return processor_class.from_pretrained(
    412                 pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
    413             )

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, **kwargs)
   1402         processor_dict, instantiation_kwargs = cls.get_processor_dict(pretrained_model_name_or_path, **kwargs)
   1403         args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs)
-> 1404         return cls.from_args_and_dict(args, processor_dict, **instantiation_kwargs)
   1405
   1406     @classmethod

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in from_args_and_dict(cls, args, processor_dict, **kwargs)
   1171         processor = cls(*args, **valid_kwargs)
   1172
-> 1173         logger.info(f"Processor {processor}")
   1174         if return_unused_kwargs:
   1175             return processor, unused_kwargs

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in __repr__(self)
    777         attributes_repr = [f"- {name}: {repr(getattr(self, name))}" for name in self.get_attributes()]
    778         attributes_repr = "\n".join(attributes_repr)
--> 779         return f"{self.__class__.__name__}:\n{attributes_repr}\n\n{self.to_json_string()}"
    780
    781     def save_pretrained(self, save_directory, push_to_hub: bool = False, **kwargs):

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in to_json_string(self)
    759             `str`: String containing all the attributes that make up this feature_extractor instance in JSON format.
    760         """
--> 761         dictionary = self.to_dict()
    762
    763         return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

[/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in to_dict(self)
    697             `dict[str, Any]`: Dictionary of all the attributes that make up this processor instance.
    698         """
--> 699         output = copy.deepcopy(self.__dict__)
    700
    701         # Get the kwargs in `__init__`.

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    134     copier = _deepcopy_dispatch.get(cls)
    135     if copier is not None:
--> 136         y = copier(x, memo)
    137     else:
    138         if issubclass(cls, type):

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _deepcopy_dict(x, memo, deepcopy)
    219     memo[id(x)] = y
    220     for key, value in x.items():
--> 221         y[deepcopy(key, memo)] = deepcopy(value, memo)
    222     return y
    223 d[dict] = _deepcopy_dict

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    160                     y = x
    161                 else:
--> 162                     y = _reconstruct(x, memo, *rv)
    163
    164     # If is its own copy, don't memoize.

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    257     if state is not None:
    258         if deep:
--> 259             state = deepcopy(state, memo)
    260         if hasattr(y, '__setstate__'):
    261             y.__setstate__(state)

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    134     copier = _deepcopy_dispatch.get(cls)
    135     if copier is not None:
--> 136         y = copier(x, memo)
    137     else:
    138         if issubclass(cls, type):

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _deepcopy_dict(x, memo, deepcopy)
    219     memo[id(x)] = y
    220     for key, value in x.items():
--> 221         y[deepcopy(key, memo)] = deepcopy(value, memo)
    222     return y
    223 d[dict] = _deepcopy_dict

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in deepcopy(x, memo, _nil)
    160                     y = x
    161                 else:
--> 162                     y = _reconstruct(x, memo, *rv)
    163
    164     # If is its own copy, don't memoize.

[/usr/lib/python3.12/copy.py](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD#) in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    259             state = deepcopy(state, memo)
    260         if hasattr(y, '__setstate__'):
--> 261             y.__setstate__(state)
    262         else:
    263             if isinstance(state, tuple) and len(state) == 2:

Exception: Error while attempting to unpickle Tokenizer: Token `<SPECIAL_32><SPECIAL_32>` out of vocabulary at line 1 column 9847997
```
There are many issues with a lot of other APIs in `transformers==5.3.0`. Please resolve these issues ASAP.

### Who can help?

@eustlb @ebezzam @vasqu @Cyrilvallez

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the issue:
1. Connect to the correct GPU runtime (I tried on T4 GPU).
2. Upload a valid audio file with extension `.WAV`.
3. Ensure `MODEL = "mistralai/Voxtral-Mini-3B-2507"` and `AUDIO_PATH = "<valid audio path>.WAV"`.
4. Run the notebook at [https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD?usp=sharing](https://colab.research.google.com/drive/18tlO4aFg0RxV7hcBf04abb1wKW-B3zbD?usp=sharing).

[EMSCall (1).ipynb](https://github.com/user-attachments/files/26104503/EMSCall.1.ipynb)

### Expected behavior

Error on loading of `processor` for `mistralai/Voxtral-Mini-3B-2507` model.

--- Comment by eustlb at 2026-03-19T09:07:58Z ---
Hey @BhavyaShah1234, thank you for raising this issue 🤗

The standalone code provided in the model card runs fine for me on v5.3.0. Can you try again from a clean environment or provide a reproducer notebook (I see you're running your code from a notebook)?

--- Comment by themavik at 2026-03-19T10:21:16Z ---
Fix submitted in #44852.

--- Comment by BhavyaShah1234 at 2026-03-23T18:26:50Z ---
All the examples work. Sorry for raising a bug. Is there a special prompt for transcribing an audio verbatim from a `.WAV` file?

--- Comment by eustlb at 2026-03-24T10:07:36Z ---
Not that I know, you can likely play around this the prompts to test it 🤗


typo in code block in docs

ISSUE #44840: typo in code block in docs
State: closed  |  Labels:
Author: zhulinchng  |  Created: 2026-03-19T01:46:22Z

typo in code block:
https://github.com/huggingface/transformers/blob/16a5b0936dcaae6efbc03ab1a4fd98dc324bfb9e/docs/source/en/weightconverter.md?plain=1#L69


ISSUE #44829: AutoModelForSequenceClassification with attn_implementation="flash_attention_3" causes degenerate training (loss increases, model predicts all-one-class)
State: closed  |  Labels: bug
Author: Jantory  |  Created: 2026-03-18T13:56:33Z

### System Info

When fine-tuning `Qwen3ForSequenceClassification` (loaded via `AutoModelForSequenceClassification`) with `attn_implementation="flash_attention_3"`, training completely fails: loss increases instead of decreasing, and the model collapses to predicting all examples as one class. Removing `attn_implementation="flash_attention_3"` (falling back to default attention) fixes the issue immediately.

## Environment:

Hardware: NVIDIA H100 (Hopper)
transformers version: (your version)
flash-attn version: (your version)
Model: Qwen/Qwen3-Embedding-8B
PEFT / LoRA applied on top

**Observed**: loss increases (e.g. 0.35 → 0.41), eval_recall=1.0 with threshold≈0 (all predicted positive), F1 stuck at positive-class base rate.

**Note**: The issue appears specific to `Qwen3ForSequenceClassification + FA3`. The same model backbone with FA3 works correctly in other use cases (e.g. feature extraction / embedding), suggesting the problem lies in the last-token pooling or score head gradient path under FA3.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen/Qwen3-Embedding-8B",
    num_labels=2,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_3",  # remove this → works
)
# train with HF Trainer on binary classification task
```

### Expected behavior

normal convergence.

--- Comment by Rocketknight1 at 2026-03-19T11:54:13Z ---
Hi @Jantory, thanks for the bug report! Can you give us an entire reproducer script, using a small dataset from the Hub or something? It'll make it a lot easier for us to track down the problem.

--- Comment by github-actions[bot] at 2026-04-18T08:11:30Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44821: Unable to load `AutoImageProcessor` from URL
State: closed  |  Labels: bug
Author: BSchilperoort  |  Created: 2026-03-18T11:08:09Z

### System Info

Versions

- `transformers` version: 5.3.0
- Platform: Linux-6.8.0-106-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 1.7.1
- Safetensors version: 0.7.0
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (NA)

### Bug

I am trying to load a config from URL, to instatiate  the AutoImageProcessor. This worked before, but not anymore with new versions of transformers.

Bug starts occuring with `transformers>=5.3.0`. It worked with `transformers==4.57.6`

To reproduce;

```py
import transformers

url = "https://huggingface.co/jinfengxie/BFMS_1014/raw/main/config.json"
transformers.AutoImageProcessor.from_pretrained(url)  # fails on >=5.0, succeeds pre-5.0

local_file = "/path/to/config.json"  # download the file from the URL above
transformers.AutoImageProcessor.from_pretrained(local_file)  # succeeds on v4 and v5
```

traceback

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bart/git/streetscapes/.venv/lib/python3.12/site-packages/transformers/models/auto/image_processing_auto.py", line 514, in from_pretrained
    raise initial_exception
  File "/home/bart/git/streetscapes/.venv/lib/python3.12/site-packages/transformers/models/auto/image_processing_auto.py", line 501, in from_pretrained
    config_dict, _ = ImageProcessingMixin.get_image_processor_dict(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/git/streetscapes/.venv/lib/python3.12/site-packages/transformers/image_processing_base.py", line 282, in get_image_processor_dict
    resolved_processor_file = cached_file(
                              ^^^^^^^^^^^^
  File "/home/bart/git/streetscapes/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 277, in cached_file
    file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bart/git/streetscapes/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 469, in cached_files
    raise OSError(f"{e}") from e
OSError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'https://huggingface.co/jinfengxie/BFMS_1014/raw/main/config.json'. Use `repo_type` argument if needed.
```

--- Comment by zucchini-nlp at 2026-03-18T16:49:30Z ---
Oops, it got deleted accidentally

https://github.com/huggingface/transformers/blob/753d61104116eefc8ffc977327b441ee0c8d599f/src/transformers/image_processing_base.py#L326-L329

Do you want to submit a PR to fix that and add a small test as well?

Code agents, don't work on it unless a human is manually checking the generated code before submitting a PR!

--- Comment by he-yufeng at 2026-03-20T16:20:17Z ---
Taking a look at this. The `elif is_remote_url(...)` + `download_url(...)` block got dropped in the image processor refactor (#43514). I'll restore URL detection and add a regression test.

--- Comment by zucchini-nlp at 2026-03-23T12:03:04Z ---
That was an intended deprecation for v5, see the warning below 🤗

https://github.com/huggingface/transformers/blob/753d61104116eefc8ffc977327b441ee0c8d599f/src/transformers/utils/hub.py#L610-L616

--- Comment by BSchilperoort at 2026-03-23T12:10:08Z ---
> That was an intended deprecation for v5

Never had seen the warning while using this in v4.

Which makes sense as this warning was only in the 2026-01-16 release, while 5.0.0 was released on 2026-01-26 🤦♂

--- Comment by BSchilperoort at 2026-03-23T12:13:45Z ---
The docstring still states that it can be a URL. Please reopen this issue

https://github.com/huggingface/transformers/blob/b1527a32a1010cd94bfb4f937af247bb5871f6fd/src/transformers/models/auto/image_processing_auto.py#L591-L592

--- Comment by zucchini-nlp at 2026-03-23T12:17:20Z ---
Ah, left-over docs. Oke, will close after someone offers to clean up those


ISSUE #44811: Whisper processor.batch_decode() function ignoring skip_special_tokens params
State: closed  |  Labels: bug
Author: cfasana  |  Created: 2026-03-18T08:30:17Z

### System Info

- `transformers` version: 4.57.6
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.36.2
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu130 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 3080 Ti Laptop GPU

### Who can help?

@ArthurZucker

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running the code provided [here](https://huggingface.co/openai/whisper-small), it seems that _skip_special_tokens_ param is being ignored.

```
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset

# load model and processor
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model.config.forced_decoder_ids = None

# load dummy dataset and read audio files
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]
input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features

# generate token ids
predicted_ids = model.generate(input_features)
# decode token ids to text
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
print("[Skip special tokens=False] ", transcription)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print("[Skip special tokens=True] ", transcription)
```
The output is the following:
_[Skip special tokens=False]  [' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.']
[Skip special tokens=True]  [' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.']_

If however a dict output is required by running the following code:
```
# generate token ids
predicted_ids = model.generate(input_features, return_dict_in_generate=True)
# decode token ids to text
transcription = processor.batch_decode(predicted_ids.sequences, skip_special_tokens=False)
print("[Skip special tokens=False] ", transcription)
transcription = processor.batch_decode(predicted_ids.sequences, skip_special_tokens=True)
print("[Skip special tokens=True] ", transcription)
```

The output is (correctly) the following:
_[Skip special tokens=False]  ['<|startoftranscript|><|en|><|transcribe|><|notimestamps|> Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.<|endoftext|>']
[Skip special tokens=True]  [' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.']_

### Expected behavior

The _processor.batch_decode()_ function should output also special tokens if _skip_special_tokens=False_ is passed.

--- Comment by Rocketknight1 at 2026-03-18T15:52:50Z ---
cc @ebezzam @eustlb maybe?

--- Comment by eustlb at 2026-03-19T10:20:14Z ---
Hey, thanks for raising this issue! This isn't a bug but expected behavior 🤗 Let me break it down quickly:

The docs actually spell out two different return behaviours:

- **Default `generate()`** (see l. 550) → returns a `torch.LongTensor` with decoder input IDs (`<|startoftranscript|>`, `<|en|>`, etc.) **already stripped**. They're not in the tensor, so `skip_special_tokens=False` can't show what isn't there. That's why both settings give you identical output.

- **`return_dict_in_generate=True`** (see l. 548) → returns a `ModelOutput` where `.sequences` keeps **everything**, including decoder input IDs and `<|endoftext|>`. Now `skip_special_tokens` actually has something to filter, which is why you see the difference.

So the two paths are *intentionally* different. Hope that clears things up!

--- Comment by cfasana at 2026-03-20T08:25:44Z ---
Thanks for the clarification!
In this case, I would suggest updating the examples shown on the website, such as the one you can find [here](https://huggingface.co/openai/whisper-small) as they do not match the latest update. 🤗
Probably indicating the `transformers` version used when they were created is sufficient, if not already shown.

--- Comment by eustlb at 2026-03-20T10:25:19Z ---
Yes, the hub model card is a bit outdated. Thanks for flagging! Could you open a discussion there for tracking and close this issue 🙏

--- Comment by cfasana at 2026-03-20T10:59:59Z ---
I'll do it, thanks!


ISSUE #44810: Showcase / question: a board-proven offline language runtime on ESP32-C3, and whether some future language capability may move beyond general model definitions
State: closed  |  Labels:
Author: Alpha-Guardian  |  Created: 2026-03-18T07:09:16Z

Hi Transformers folks,

  I wanted to share a small but unusual language-runtime project that may still be relevant to a broader ecosystem question, even though it sits far outside the usual Python/GPU dense-model path.

  We built a public demo line called Engram and deployed it on a commodity ESP32-C3.

  Current public numbers:

  * Host-side benchmark capability
    * `LogiQA = 0.392523`
    * `IFEval = 0.780037`

  * Published board proof
    * `LogiQA 642 = 249 / 642 = 0.3878504672897196`
    * `host_full_match = 642 / 642`
    * runtime artifact size = `1,380,771 bytes`

  Important scope note:

  This is **not** presented as unrestricted open-input native LLM generation on MCU.

  The board-side path is closer to a flash-resident, table-driven runtime with:

  * packed token weights
  * hashed lookup structures
  * fixed compiled probe batches
  * streaming fold / checksum style execution over precompiled structures

  So this is not a standard dense model represented in a familiar inference stack. It is closer to a task-specialized language runtime whose behavior has been crystallized into a compact executable form under severe physical constraints.

  Repo:
  https://github.com/Alpha-Guardian/Engram

  Why I’m posting here is that Transformers sits at the center of how model definitions propagate across the open ecosystem.

  What I’d be curious about is whether systems like this should be thought of as:

  * completely outside the normal model-definition family
  * an extreme endpoint where some language capability is no longer best represented as a general dense model
  * or an early sign that future language systems may include both general model definitions and highly specialized executable forms for certain capability slices

  If this direction is relevant to the broader ecosystem conversation, I’d be glad to compare notes.

--- Comment by Rocketknight1 at 2026-03-18T15:47:59Z ---
This feels like AI psychosis to me - once you strip out the grandiose language and the extensive documentation, answers to questions have been stored in a lookup table and this has been compiled into a binary blob. This will obviously not be able to handle anything except emitting prewritten answers to known questions, which is probably not very useful!

--- Comment by Alpha-Guardian at 2026-03-18T15:53:17Z ---
> This feels like AI psychosis to me - once you strip out the grandiose language and the extensive documentation, answers to questions have been stored in a lookup table and this has been compiled into a binary blob. This will obviously not be able to handle anything except emitting prewritten answers to known questions, which is probably not very useful!

@Rocketknight1   I think we are disagreeing about category, not about the implementation being constrained.

  This repo does not claim open-input general LLM inference on MCU. It also is not intended as a cache of prewritten answers. The board-side path is a compiled execution form for a narrow, audited capability slice, and the repo tries to be explicit about that boundary.

  If your view is that anything outside a general dense model should also fall outside the Transformers conversation, that is a reasonable moderation decision. I would just distinguish it from a simple lookup table.


ISSUE #44805: IndexError: The shape of the mask [...] at index 0 does not match the shape of the indexed tensor [...] at index 0
State: closed  |  Labels: bug
Author: akowalsk  |  Created: 2026-03-18T00:58:28Z

### System Info

```
- `transformers` version: 5.3.0
- Platform: Linux-5.15.0-164-generic-x86_64-with-glibc2.35
- Python version: 3.12.13
- Huggingface_hub version: 1.7.1
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: 0.16.4
- PyTorch version (accelerator?): 2.10.0+cu129 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: yes
- GPU type: NVIDIA GeForce RTX 3090
```

### Who can help?

@SunMarc @zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I'm attempting to train a Lora adapter on a local copy of Qwen3-VL-4B-Instruct with a config like this:

```
warmup_steps = 0
max_seq_length = 2560

processor = AutoProcessor.from_pretrained(model_name)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_name,
    dtype=(
        torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
    ),
)

peft_config = LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules=[
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj",
        ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# ────────────────────────────────────────────────
#  DATASET (customize preprocessing!)
# ────────────────────────────────────────────────
train_dataset = load_dataset("train")
eval_dataset = load_dataset("test")

# ────────────────────────────────────────────────
#  TRAINING ARGUMENTS
# ────────────────────────────────────────────────
training_args = SFTConfig(
    output_dir=output_dir,
    num_train_epochs=num_epochs,
    per_device_train_batch_size=per_device_batch_size,
    per_device_eval_batch_size=per_device_batch_size,
    learning_rate=learning_rate,
    warmup_steps=warmup_steps,
    eval_strategy="steps",
    save_strategy="steps",
    save_steps=int(len(train_dataset) / 2),
    eval_steps=int(len(train_dataset) / 2),
    logging_steps=10,
    metric_for_best_model="eval_loss",
    load_best_model_at_end=True,
    max_grad_norm=1,
    ddp_find_unused_parameters=False,
    dataset_kwargs={"skip_prepare_dataset": True},
    remove_unused_columns=False,
    optim="adamw_torch",
    completion_only_loss=True,
    max_length=max_seq_length
)

# ────────────────────────────────────────────────
#  TRAIN
# ────────────────────────────────────────────────

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    peft_config=peft_config,
    processing_class=processor,
)

trainer.train()

```
As soon as training starts I get this error:

```
  0%|                                                                                | 0/117 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/opt/ml/models/train_hf_bug.py", line 184, in <module>
    trainer.train()
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1424, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1506, in _inner_training_loop
    self._run_epoch(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1734, in _run_epoch
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/trl/trainer/sft_trainer.py", line 1320, in training_step
    return super().training_step(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1906, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/trl/trainer/sft_trainer.py", line 1216, in compute_loss
    (loss, outputs) = super().compute_loss(
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1978, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/utils/operations.py", line 823, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/utils/operations.py", line 811, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/peft/peft_model.py", line 1923, in forward
    return self.base_model(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/peft/tuners/tuners_utils.py", line 311, in forward
    return self.model.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 843, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1522, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 843, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1348, in forward
    position_ids = self.compute_3d_position_ids(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1242, in compute_3d_position_ids
    position_ids, rope_deltas = self.get_rope_index(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1105, in get_rope_index
    input_token_type = input_token_type[attention_mask[batch_idx].bool()]
                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: The shape of the mask [2041] at index 0 does not match the shape of the indexed tensor [1010] at index 0
  0%|          | 0/117 [00:01<?, ?it/s]
```
If I downgrade to transformers 5.2.0, it works just fine.  I only see this error on 5.3.0 and I also see it with a model of a different architecture (GLM-4.6V-Flash).

My dataset is custom, but laid out in [prompt-completion](https://huggingface.co/docs/trl/dataset_formats#prompt-completion) format with an ["image" column](https://huggingface.co/docs/trl/dataset_formats#vision-datasets) with a single PIL image.  The dataset was working (as was the rest of the code) on earlier transformers version.

I attempted the workaround [here](https://github.com/huggingface/transformers/issues/41093#issuecomment-3476061753) but I still get the error with gradient checkpointing disabled and use cache enabled, but the stack trace is slightly different:

```
  0%|                                                                                | 0/117 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/opt/ml/models/train_hf_bug.py", line 185, in <module>
    trainer.train()
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1424, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1506, in _inner_training_loop
    self._run_epoch(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1734, in _run_epoch
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/trl/trainer/sft_trainer.py", line 1320, in training_step
    return super().training_step(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1906, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/trl/trainer/sft_trainer.py", line 1216, in compute_loss
    (loss, outputs) = super().compute_loss(
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1978, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/utils/operations.py", line 823, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/utils/operations.py", line 811, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/peft/peft_model.py", line 1923, in forward
    return self.base_model(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/peft/tuners/tuners_utils.py", line 311, in forward
    return self.model.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 843, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1522, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 843, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1348, in forward
    position_ids = self.compute_3d_position_ids(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1242, in compute_3d_position_ids
    position_ids, rope_deltas = self.get_rope_index(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1105, in get_rope_index
    input_token_type = input_token_type[attention_mask[batch_idx].bool()]
                       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: The shape of the mask [2041] at index 0 does not match the shape of the indexed tensor [1010] at index 0
```
I'm using the latest trl available (0.29.0).

### Expected behavior

Training should run without error.

--- Comment by zucchini-nlp at 2026-03-18T10:51:14Z ---
oh I love Qwen and position ids 🫠

I think the correct fix on TRL wasn't yet released for this and you might want to try again by installing from source @akowalsk . See related https://github.com/huggingface/transformers/issues/44384#issuecomment-3997014468

--- Comment by akowalsk at 2026-03-18T12:20:33Z ---
Installing TRL from source worked.  Thanks!


ISSUE #44792: Failed test case `test_model_generate_images` for janus model
State: closed  |  Labels: bug
Author: kaixuanliu  |  Created: 2026-03-17T10:40:18Z

### System Info

- `transformers` version: 5.3.0.dev0
- Platform: Linux-5.4.292-1.el8.elrepo.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 1.7.1
- Safetensors version: 0.5.3
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100 80GB PCIe

### Who can help?

multimodal models: @zucchini-nlp

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .
export RUN_SLOW=1
pytest -rA tests/models/janus/test_modeling_janus.py::JanusIntegrationTest::test_model_generate_images

### Expected behavior

test case pass. Although I added patch
```
--- a/src/transformers/models/janus/modeling_janus.py
+++ b/src/transformers/models/janus/modeling_janus.py
@@ -1285,7 +1285,7 @@ class JanusForConditionalGeneration(JanusPreTrainedModel, GenerationMixin):
         input_ids, model_kwargs = self._expand_inputs_for_generation(
             input_ids=input_ids,
             attention_mask=attention_mask,
-            expand_size=generation_config.num_return_sequences,
+            expand_size=generation_config.num_return_sequences or 1,
             **model_kwargs,
         )

@@ -1315,7 +1315,7 @@ class JanusForConditionalGeneration(JanusPreTrainedModel, GenerationMixin):
                 # batch_size should account for both conditional/unconditional input; hence multiplied by 2.
                 batch_size=batch_size * 2,
                 # we should have at least a cache len of seq_len + num_image_tokens.
-                max_cache_len=max(generation_config.max_length, num_image_tokens + seq_len),
+                max_cache_len=max(generation_config.max_length or 0, num_image_tokens + seq_len),
                 model_kwargs=model_kwargs,
             )
```
it still gets failed.

--- Comment by ydshieh at 2026-03-18T14:26:42Z ---
The `generate` in `JanusForConditionalGeneration`  seems to lack

```
        global_defaults = self.generation_config._get_default_generation_params()
        generation_config.update(**self.generation_config.to_dict(), defaults_only=True, allow_custom_entries=True)
        generation_config.update(**global_defaults, defaults_only=True)
```

that exists in `GeneratioinMixin.generate`.

I added that and it "kinda" works: no longer fail at `input_ids, model_kwargs = self._expand_inputs_for_generation(`, but at a later point with strange cuda device issue.

cc @zucchini-nlp I am not able to fix the ` torch.AcceleratorError: CUDA error: device-side assert triggered`, but maybe we can fix the first part about generation config first, so other team members can check the other issues maybe?

--- Comment by github-actions[bot] at 2026-04-17T08:29:45Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by kaixuanliu at 2026-04-17T08:32:33Z ---
Close this issue as it has been fixed.


ISSUE #44779: Deepseek tokenizer produces incorrect results as of v5 (works in v4)
State: closed  |  Labels: bug
Author: xenova  |  Created: 2026-03-17T00:30:56Z

### System Info

- `transformers` version: 5.3.0
- Platform: Linux-6.6.113+-x86_64-with-glibc2.35
- Python version: 3.12.12
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cpu (NA)
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('deepseek-ai/DeepSeek-R1')

text = "How are you doing?"
print(tokenizer.encode(text))
print(tokenizer.tokenize(text))
print(tokenizer.decode(tokenizer.encode(text)))
```

produces
```
[4117, 591, 12829, 62552, 33]
['How', 'are', 'you', 'doing', '?']
Howareyoudoing?
```

### Expected behavior

Downgrading to `transformers==4.57.6`, running the same code as above produces

```
[0, 4117, 477, 440, 4843, 33]
['How', 'Ġare', 'Ġyou', 'Ġdoing', '?']
<｜begin▁of▁sentence｜>How are you doing?
```

--- Comment by xenova at 2026-03-17T01:25:16Z ---
possibly related:
loading and saving again causes these differences between: original and re-saved.
```
diff original.json new.json
```

```diff
7369c7369
<   "normalizer": { "type": "Sequence", "normalizers": [] },
---
>   "normalizer": null,
7371,7399c7371,7374
<     "type": "Sequence",
<     "pretokenizers": [
<       {
<         "type": "Split",
<         "pattern": { "Regex": "\\p{N}{1,3}" },
<         "behavior": "Isolated",
<         "invert": false
<       },
<       {
<         "type": "Split",
<         "pattern": { "Regex": "[一-龥぀-ゟ゠-ヿ]+" },
<         "behavior": "Isolated",
<         "invert": false
<       },
<       {
<         "type": "Split",
<         "pattern": {
<           "Regex": "[!\"#$%&'()*+,\\-./:;<=>?@\\[\\\\\\]^_`{|}~][A-Za-z]+|[^\r\n\\p{L}\\p{P}\\p{S}]?[\\p{L}\\p{M}]+| ?[\\p{P}\\p{S}]+[\r\n]*|\\s*[\r\n]+|\\s+(?!\\S)|\\s+"
<         },
<         "behavior": "Isolated",
<         "invert": false
<       },
<       {
<         "type": "ByteLevel",
<         "add_prefix_space": false,
<         "trim_offsets": true,
<         "use_regex": false
<       }
<     ]
---
>     "type": "Metaspace",
>     "replacement": "▁",
>     "prepend_scheme": "always",
>     "split": false
7402,7419c7377,7380
<     "type": "TemplateProcessing",
<     "single": [
<       { "SpecialToken": { "id": "<｜begin▁of▁sentence｜>", "type_id": 0 } },
<       { "Sequence": { "id": "A", "type_id": 0 } }
<     ],
<     "pair": [
<       { "SpecialToken": { "id": "<｜begin▁of▁sentence｜>", "type_id": 0 } },
<       { "Sequence": { "id": "A", "type_id": 0 } },
<       { "SpecialToken": { "id": "<｜begin▁of▁sentence｜>", "type_id": 1 } },
<       { "Sequence": { "id": "B", "type_id": 1 } }
<     ],
<     "special_tokens": {
<       "<｜begin▁of▁sentence｜>": {
<         "id": "<｜begin▁of▁sentence｜>",
<         "ids": [0],
<         "tokens": ["<｜begin▁of▁sentence｜>"]
<       }
<     }
---
>     "type": "ByteLevel",
>     "add_prefix_space": true,
>     "trim_offsets": false,
>     "use_regex": true
7422,7425c7383,7389
<     "type": "ByteLevel",
<     "add_prefix_space": true,
<     "trim_offsets": true,
<     "use_regex": true
---
>     "type": "Sequence",
>     "decoders": [
>       { "type": "Replace", "pattern": { "String": "▁" }, "content": " " },
>       { "type": "ByteFallback" },
>       { "type": "Fuse" },
>       { "type": "Strip", "content": " ", "start": 1, "stop": 0 }
>     ]
7433,7434c7397,7398
<     "fuse_unk": false,
<     "byte_fallback": false,
---
>     "fuse_unk": true,
>     "byte_fallback": true,
```

--- Comment by sukhmansaran at 2026-03-17T09:08:25Z ---
I investigated this issue and can confirm it is caused by a tokenizer pipeline mismatch introduced when the tokenizer is re-saved in transformers v5.

### Root cause

The original DeepSeek tokenizer uses a ByteLevel-based pipeline:

- pre_tokenizer: `Sequence(Split..., ByteLevel)`
- decoder: `ByteLevel`
- post_processor: `TemplateProcessing` (with BOS token)

However, in transformers v5 this gets replaced with:

- pre_tokenizer: `Metaspace(split=False)`
- decoder: `Sequence(Replace + ByteFallback + Fuse + Strip)`
- post_processor: changed to `ByteLevel`

This change is incompatible with the existing vocabulary.

Specifically, the vocab does **not** contain metaspace-prefixed tokens:
- `'How'` → valid token
- `'▁How'` → not in vocab

Because of this, spaces are never encoded in the token stream, and the decoder cannot reconstruct them, leading to outputs like:

Howareyoudoing?

### Why decoder-only fixes don’t work

This is not just a decoding issue. Since the metaspace pre-tokenizer never produces tokens aligned with the vocab, the whitespace information is already lost before decoding. Any decoder-level fix will be insufficient.

### Fix

Restoring the original ByteLevel pipeline resolves the issue completely. The required changes are:

- Restore pre_tokenizer to `Sequence(Split..., ByteLevel)`
- Restore decoder to `ByteLevel`
- Restore `TemplateProcessing` (BOS handling)

After applying this, decoding becomes lossless again:

decode(encode(text)) ==text

across:
- punctuation
- whitespace
- mixed inputs

### Minimal patch

```python
from tokenizers import Regex
from tokenizers.pre_tokenizers import Sequence, Split, ByteLevel as BLPreTok
from tokenizers.processors import TemplateProcessing
from tokenizers.decoders import ByteLevel as BLDecoder

def fix_deepseek_tokenizer(tokenizer):
    backend = tokenizer.backend_tokenizer

    backend.pre_tokenizer = Sequence([
        Split(pattern=Regex(r"\p{N}{1,3}"), behavior="isolated"),
        Split(pattern=Regex(r"[一-龥぀-ゟ゠-ヿ]+"), behavior="isolated"),
        Split(
            pattern=Regex(
                r"""[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~][A-Za-z]+"""
                r"""|[^\r\n\p{L}\p{P}\p{S}]?[\p{L}\p{M}]+"""
                r"""| ?[\p{P}\p{S}]+[\r\n]*"""
                r"""|\s*[\r\n]+"""
                r"""|\s+(?!\S)"""
                r"""|\s+"""
            ),
            behavior="isolated",
        ),
        BLPreTok(add_prefix_space=False, trim_offsets=True, use_regex=False),
    ])

    bos = "<｜begin▁of▁sentence｜>"
    bos_id = tokenizer.convert_tokens_to_ids(bos)
    backend.post_processor = TemplateProcessing(
        single=f"{bos}:0 $A:0",
        pair=f"{bos}:0 $A:0 {bos}:1 $B:1",
        special_tokens=[(bos, bos_id)],
    )

    backend.decoder = BLDecoder(
        add_prefix_space=True,
        trim_offsets=True,
        use_regex=True,
    )

    return tokenizer

### Result

With this fix applied:

- decode(encode(text)) matches the original input (modulo BOS token)

- behavior matches transformers v4 and the original tokenizer.json

--- Comment by ArthurZucker at 2026-03-17T11:11:50Z ---
Yeah this is just a remote code model to be added to https://github.com/huggingface/transformers/blob/3491f116a0ec4def708c2fbc141ea08df16ba29b/src/transformers/models/auto/tokenization_auto.py#L342

--- Comment by ArthurZucker at 2026-03-17T11:12:10Z ---
cc @itazap as I think you are working on extanding the list with a test script>?


Disable mmap on Strix Halo to avoid OOM

ISSUE #44756: Disable mmap on Strix Halo to avoid OOM
State: open  |  Labels: bug
Author: woct0rdho  |  Created: 2026-03-16T14:28:04Z

### System Info

- `transformers` version: 5.3.0
- Platform: Linux-6.19.0-9-generic-x86_64-with-glibc2.42
- Python version: 3.13.12
- Huggingface_hub version: 1.7.1
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.11.0a0+rocm7.11.0a20260106 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: Radeon 8060S Graphics

Strix Halo APU has 128 GB unified memory. I've configured 125 GB TTM memory and 1 GB UMA buffer.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When I run
```python3
model = AutoModel.from_pretrained("Qwen/Qwen3.5-35B-A3B")
```
it errors like
```
  ...
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/models/auto/auto_factory.py", line 374, in from_pretrained
    return model_class.from_pretrained(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/modeling_utils.py", line 4137, in from_pretrained
    loading_info, disk_offload_index = cls._load_pretrained_model(model, state_dict, checkpoint_files, load_config)
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/modeling_utils.py", line 4256, in _load_pretrained_model
    loading_info, disk_offload_index = convert_and_load_state_dict_in_model(
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        model=model,
        ^^^^^^^^^^^^
    ...<3 lines>...
        disk_offload_index=disk_offload_index,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/core_model_loading.py", line 1212, in convert_and_load_state_dict_in_model
    realized_value = mapping.convert(
        first_param_name,
    ...<3 lines>...
        loading_info=loading_info,
    )
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/core_model_loading.py", line 678, in convert
    collected_tensors = self.materialize_tensors()
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/core_model_loading.py", line 657, in materialize_tensors
    tensors = [func() for func in tensors]
               ~~~~^^
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/core_model_loading.py", line 800, in _job
    return _materialize_copy(tensor, device, dtype)
  File "/home/wd/venv_torch/lib/python3.13/site-packages/transformers/core_model_loading.py", line 789, in _materialize_copy
    tensor = tensor.to(device=device, dtype=dtype)
torch.OutOfMemoryError: HIP out of memory. Tried to allocate 2.00 MiB. GPU 0 has a total capacity of 125.00 GiB of which 24.00 KiB is free. Of the allocated memory 58.06 GiB is allocated by PyTorch, and 4.94 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
```

### Expected behavior

The Qwen3.5 35B fp16 model takes 67 GB memory. With 125 GB TTM memory it should not OOM.

A proper fix would be in safetensors. For now I use the following trick to work around this issue:
```python3
def patch_transformers_disable_mmap():
    import transformers.core_model_loading as _cml

    def _materialize_copy_no_mmap(t, device=None, dtype=None):
        # The indexing materializes the tensor from safetensors.
        t = t[...]

        # If safetensors returned an mmapped tensor on CPU,
        # force a copy on CPU before any device/dtype conversion.
        if isinstance(t, torch.Tensor) and t.device.type == "cpu":
            t = t.to(device="cpu", copy=True)

        if dtype is not None or device is not None:
            t = t.to(device=device, dtype=dtype)
        return t

    assert hasattr(_cml, "_materialize_copy")
    _cml._materialize_copy = _materialize_copy_no_mmap
    print("Patched transformers to disable mmap.")

def main():
    patch_transformers_disable_mmap()
    model = AutoModel.from_pretrained("Qwen/Qwen3.5-35B-A3B")
    ...
```
It's inspired by how ComfyUI disables mmap:
https://github.com/Comfy-Org/ComfyUI/blob/593be209a45a8a306c26de550e240a363de405a7/comfy/utils.py#L137
and it's known that currently disabling mmap improves the model loading speed in ComfyUI on Strix Halo.

--- Comment by BillionClaw at 2026-03-17T07:43:47Z ---
I'm looking into implementing a fix for this Strix Halo mmap OOM issue. The workaround you provided is helpful - I'll investigate adding proper Strix Halo detection to automatically disable mmap when this hardware is detected. This should be more robust than manual patching.

--- Comment by woct0rdho at 2026-04-11T00:47:24Z ---
Update: I think the best way to fix it is in safetensors, see https://github.com/huggingface/safetensors/pull/728 . If the PR gets merged, we no longer need this workaround.

--- Comment by woct0rdho at 2026-05-01T23:56:05Z ---
The safetensors developers prefer to pass a parameter for the backend, so we need to pass the parameter in transformers after the next version of safetensors releases. For now I'm sticking to injecting an environment variable...


ISSUE #44754: I really need help regarding to meta device issue
State: closed  |  Labels:
Author: Hany138078  |  Created: 2026-03-16T14:03:28Z

I installed TRELLIS 2 and getting the error for meta device, I followed the instructions here for installing this release of transformers, but when I launch TRELLIS 2 I still get the same error. Can someone please help me Fix this step by step because I spend 5 days trying without any solution

--- Comment by sasmita016 at 2026-03-21T18:29:19Z ---
@Hany138078 This looks like the known TRELLIS.2 / BiRefNet issue with Transformers 5.x and meta tensors.

The error:
RuntimeError: Tensor.item() cannot be called on meta tensors

happens during model initialization (briaai/birefnet) when using newer Transformers versions. There are related reports showing this breaks with Transformers 5.x but works with 4.x.

A practical workaround that worked for me / others:

1. Create a clean environment (important — avoids mixed packages)
   conda create -n trellis2_clean python=3.10 -y
   conda activate trellis2_clean

2. Install the PyTorch version TRELLIS.2 expects (CUDA 12.4 example)
   pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124

3. Install dependencies but pin Transformers to <5
   pip install imageio imageio-ffmpeg tqdm easydict opencv-python-headless ninja trimesh gradio==6.0.1 tensorboard pandas lpips zstandard
   pip install "transformers<5"
   pip install git+https://github.com/EasternJournalist/utils3d.git@9a4eb15e4021b67b12c460c7057d642626897ec8
   pip install pillow-simd
   pip install kornia timm

4. Clear cached HF modules (important for this specific error)
   rm -rf ~/.cache/huggingface/modules/transformers_modules/briaai
   rm -rf ~/.cache/huggingface/hub/models--briaai*

5. Verify versions before launching
   python -c "import sys, torch, transformers; print(sys.executable); print(torch.__version__); print(transformers.__version__)"

   Expected:
   - torch: 2.6.0
   - transformers: 4.x

6. Launch again
   python app.py

---

If it still fails, it would help to share:
- OS (Linux / WSL / Windows)
- GPU + VRAM (output of nvidia-smi)
- Python / torch / transformers versions
- full traceback

This seems to be a compatibility issue rather than a user setup mistake, so pinning Transformers is currently the safest workaround.

--- Comment by github-actions[bot] at 2026-04-17T08:29:48Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44749: Transformer 从4.57.3 升级到5.3.0 后过滤数据时长变慢十倍以上
State: closed  |  Labels: bug
Author: chenjiaoAngel  |  Created: 2026-03-16T11:48:25Z

### System Info

H20

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

过滤函数实现：对应链接 https://github.com/verl-project/verl/blob/main/verl/utils/dataset/rl_dataset.py#L241
```
          def doc2len(doc) -> int:
                    try:
                        apply_kwargs = dict(**self.apply_chat_template_kwargs)
                        if self.tool_schemas is not None:
                            apply_kwargs["tools"] = self.tool_schemas

                        # Keep explicit tokenization to avoid transformers version default changes.
                        apply_kwargs.pop("tokenize", None)
                        apply_kwargs.pop("return_dict", None)
                        apply_kwargs.pop("return_tensors", None)

                        tokenized_prompt = tokenizer.apply_chat_template(
                            doc[prompt_key], add_generation_prompt=True, tokenize=True, **apply_kwargs
                        )
                        return len(normalize_token_ids(tokenized_prompt))
                    except Exception:
                        print("Error processing one of the samples, skipping...")
                        traceback.print_exc()
                        return self.max_prompt_length + 1

            dataframe = dataframe.filter(
                lambda doc: doc2len(doc) <= self.max_prompt_length,
                num_proc=self.num_workers,
                desc=f"Filtering prompts longer than {self.max_prompt_length} tokens",
            )
```
使用transformer 4.57.3 运行这段code大概十几分钟，如果升级到5.3.0 需要约两个小时；辛苦帮忙看看 模型=qwen3.5-35b-A22 数据集是aime-2024
4.57.3 结果

5.3.0 结果

### Expected behavior

希望transformer 升级后，其tokenizer 的速度没有变化

--- Comment by sparkingarthur at 2026-03-17T16:06:46Z ---
你应该检查一下，是不是因为v5 tokenize之后，input id变多了……我发觉v5会自动在所有中文前面加空格。
You should check if the increase in input ids after v5 tokenization. I find that v5 automatically adding spaces before all Chinese characters.

--- Comment by Rocketknight1 at 2026-03-18T15:36:51Z ---
cc @arthurzucker @itazap to the above comment since it seems important ^

--- Comment by chenjiaoAngel at 2026-03-19T03:43:12Z ---
> cc [@ArthurZucker](https://github.com/ArthurZucker) [@itazap](https://github.com/itazap) to the above comment since it seems important ^

我现在想到方法，区分文字和多模态任务，如果是纯文本话，走processor.tokenizer(**)处理，否则走processor(***) 处理；这样可以大大减少纯文本任务的数据处理；I’ve come up with a solution to distinguish between text-only and multi-modal tasks:
For text-only tasks, process data via processor.tokenizer(** kwargs);
For multi-modal tasks, process data via processor(** kwargs).
This approach will significantly reduce the data processing overhead of text-only tasks.

--- Comment by chenjiaoAngel at 2026-03-19T03:43:37Z ---
> 你应该检查一下，是不是因为v5 tokenize之后，input id变多了……我发觉v5会自动在所有中文前面加空格。 You should check if the increase in input ids after v5 tokenization. I find that v5 automatically adding spaces before all Chinese characters.

我看了应该不是的； It's not

--- Comment by itazap at 2026-03-19T13:09:46Z ---
> 我看了应该不是的； It's not

@chenjiaoAngel hello, just confirming if you're still facing an issue?

--- Comment by chenjiaoAngel at 2026-03-20T07:14:29Z ---
> > 我看了应该不是的； It's not
>
> [@chenjiaoAngel](https://github.com/chenjiaoAngel) hello, just confirming if you're still facing an issue?

I change some code, if task is task, use another api(processor.tokenizer) can save more latency. otherwise, the latency is still long;

--- Comment by ArthurZucker at 2026-03-23T08:28:15Z ---
I the model is `qwen3.5-35b-A22 ` on the hub, it might be that the underlying tokenizer changed, and has an extra pre tokenizer. Can you share the exact tokenizer on the hub ?

--- Comment by chenjiaoAngel at 2026-03-25T03:52:16Z ---
> I the model is `qwen3.5-35b-A22 ` on the hub, it might be that the underlying tokenizer changed, and has an extra pre tokenizer. Can you share the exact tokenizer on the hub ?

sorry, the model name is  `qwen3.5-35b-A3B `

--- Comment by github-actions[bot] at 2026-04-18T08:11:32Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44748: [Neuron] Auto-select StaticCache when device is Neuron
State: closed  |  Labels:
Author: dacorvo  |  Created: 2026-03-16T10:38:15Z

## Context

On Neuron devices, `StaticCache` is required for correct generation — dynamic tensor shapes trigger per-step recompilations. Currently, users must explicitly pass `past_key_values=StaticCache(...)` or `cache_implementation="static"` to `model.generate()`. Ideally, `model.generate()` should auto-select `StaticCache` when running on Neuron, providing zero-config DX.

**Depends on:** #44742 (StaticCache-friendly `_sample` must exist first — otherwise auto-selecting StaticCache still hits the dynamic-shape `_sample` path)

## Proposed change

In `_prepare_cache_for_generation` (around line 1879 of `src/transformers/generation/utils.py`), when no cache is specified:

```python
if generation_config.cache_implementation is None and self.device.type == "neuron":
    generation_config.cache_implementation = "static"
```

**Precedent:** The generation code already has device-aware logic:
- Line 2021: `self.device.type in ["cuda", "xpu"]` for compile criteria
- Line 1870: Forces `"dynamic_full"` for assisted generation

Device-aware cache selection fits the existing pattern.

## Expected behavior

```python
# On Neuron — auto-selects StaticCache, uses static-shape _sample path:
model.generate(input_ids)

# On CUDA — unchanged, uses DynamicCache + standard _sample:
model.generate(input_ids)

# Explicit override still works on any device:
model.generate(input_ids, cache_implementation="static")
model.generate(input_ids, past_key_values=my_static_cache)
```

## Related

- Parent: #44741
- Depends on: #44742

--- Comment by sasmita016 at 2026-03-21T18:19:03Z ---

Hi, I am willing to check on this, @maintainer could you please confirm

--- Comment by github-actions[bot] at 2026-04-16T08:29:46Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by dacorvo at 2026-04-17T14:05:08Z ---
Not entirely sure this is relevant, considering that we will probably need to prepare the StaticCache in advance to match hardware requirements (might not fit depending on tp_plan and maximum length).

If so, we will never pass the string "static", but rather the StaticCache instance all set with correct values.

--- Comment by github-actions[bot] at 2026-05-12T08:54:45Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44746: 最理想的Transformes二代生态环境的设想，100B的推理引擎模型会成为未来标准。
State: closed  |  Labels:
Author: QuickerStudio  |  Created: 2026-03-16T10:08:38Z

定律：
1.针对任务类型而优化的不到100B的“闭源”推理模型采用二进制加密部署到客户本地，后来以客户为中心形成的云计算生态将大幅度平摊算力和能源局限性。（增量更新型的大模型才是未来标准，虽然现在看起来很荒谬，但是人类意识就是在不停的增量更新，直到事实合理）

2.公共开放式地增量向量数据库收录系统。（每人抱着有各种缺陷的向量数据库，不如动用所有人的力量去维护一个完美的向量数据库）

3.小而精准的本地资料库负责搜索，管理的“开源”模型。（简单推理需求本地化）

4.基于开源合作下的功能研发类型的软件生态环境诞生的商业经济。（企业研发的试错场）

进入这个时代的人们向量数据库共享，知识共享，大量的人才会有更多的精力去校准推理能力和应用的功能开发与人类的推理能力和功能需求之间的偏差。

在大家收集向量数据库内容差距越来越小的情况下，会迎来推理引擎的高速增长。

推理模型和向量数据库分离才是未来。

公共向量数据库的增量更新反而节约了大量的能源，也能实现按需加载到本地。

模型参数100B的标准流行一段时间，硬件工程师才会有明确的目标设置，才会看到硬件根据100B参数的模型进行深度优化，大模型的普及率才会增长到千家万户。

世界和平。

--- Comment by Rocketknight1 at 2026-03-18T15:15:22Z ---
We appreciate your desire for world peace but Github issues are not the right place for manifestos!


推理模型的缺陷

ISSUE #44744: 推理模型的缺陷
State: closed  |  Labels:
Author: QuickerStudio  |  Created: 2026-03-16T09:22:56Z

这种架构的推理模型无法解决非线性推理模型下的问题。

例如：1+2，2+1=6，推理模型不会把目标设置成如何获得6缺失了什么条件，而是基于训练数据在疯狂地做小学数学。

第二个共性特征解析：在开发ASR,TTS的编程任务中，推理模型只会质疑代码文件历史中的潜在问题，并不会察觉到数据流中隐藏的线索是反向演绎法，和诊断问题的最短路径是：文本A→TTS→Wav→ASR→文本A。

如果推理模型没有实质性地研发进展，那我们只是再次造了一个用于搜索向量数据库的Google.甚至都没有Google提供的情报有参考性。

此架构下的数据库收录了大多数人类意识资产，却没有真正还原人类推理模型的一半能力。


ISSUE #44743: transformers modular_qwen3_5.py: Recurrent states always reset when using cache and seq_len>1
State: closed  |  Labels: bug
Author: yansongzhou  |  Created: 2026-03-16T08:47:36Z

### System Info

- `transformers` version: 5.3.0
- Platform: Linux-5.15.0-171-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 4090 D

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Run generation token by token, then use the list of output tokens somewhere in the middle as input_ids along with past_key_values.
For instance this prompt "随机生成三个运动员的得分记录，以jsonl格式输出为三行，每个字典包括运动员名字name，项目event，得分score。" gives something like this:
```
{"name": "Li Na", "event": "Tennis", "score": 98.5}
{"name": "Usain Bolt", "event": "Sprint", "score": 9.58}
{"name": "Simone Biles", "event": "Gymnastics", "score": 16.75}
```
But when you manually concat the exact tokens generated for 'Li Na' after '{"name": "', the response turns into an introduction of her:
```
{"name": "Li Na, a Chinese tennis player, has won 10 Grand Slam singles titles. The number of Grand Slam singles titles won by the top 10 players in the world is ...
```
I use the basic forward pass and greedy decode the output logits:
```
        forward_kwargs = {
            "input_ids": current_input_ids, # may include multiple tokens
            "past_key_values": past_key_values,
            "return_dict": True,
            "use_cache": True,
        }
        with torch.inference_mode():
            outputs = model(**forward_kwargs)
        next_token_logits = outputs.logits[:, -1, :]
        next_token = torch.argmax(next_token_logits, dim=-1)
```

### Expected behavior

## Source Code
https://github.com/huggingface/transformers/blob/d64a6d67d8c004a25570db4df5689e06caea6af7/src/transformers/models/qwen3_5/modular_qwen3_5.py#L345
When a past_key_values cache is present and input_ids contain multiple tokens, the recurrent state is always reset, causing loss of context.

## Possible Fix
My current fix is reusing the recurrent_state in the cache (initial_state=recurrent_state) but not reusing the conv_state. However, this fix only gives a reasonable output, it does not guarantee the same output as when you run the model token by token.
To be frank I'm not sure if this fix really makes sense, but it does work better than what is present now.

--- Comment by yansongzhou at 2026-03-16T09:27:24Z ---
A more reasonable fix would be to modify the forward function within Qwen3_5GatedDeltaNet to handle multiple tokens with past token by token.

--- Comment by yansongzhou at 2026-03-16T09:46:10Z ---
So I consulted the Qwen3.5 team and the fix is simple: skip the seq_len==1 check, as it's just a safety check in transformers.
He told me that transformers currently does not officially support mtp or similar features, so I guess this may not be planned. But, would it be better to give some instruction instead of letting the logits run wild?

Anyways, hope the solution helps!


ISSUE #44742: [Neuron] Compilation-friendly generation loop
State: open  |  Labels:
Author: dacorvo  |  Created: 2026-03-16T08:43:33Z

## Context

`GenerationMixin._sample` grows `input_ids`, `attention_mask`, and `position_ids` via `torch.cat` on every decode step. This is problematic for TPU backends where dynamic tensor shapes carry a cost: static shapes are required for graph caching — dynamic shapes cause retracing, even in eager mode.

`StaticCache` already exists in transformers and implies "I want fixed-size tensors in forward" — but `_update_model_kwargs_for_generation` uses dynamic tensors regardless of the cache type.
This makes sense since these tensors never reach the forward: they are just used by `prepare_inputs_for_generation` to precisely prepare the static shapes inputs.

This issue is to look at solutions to either:
- make sure all operations leading to growing tensors happen on CPU (could be a device specific rule),
- use static tensor shapes.

## Related

* Non-generation Neuron changes: [#44741](https://github.com/huggingface/transformers/issues/44741)
* Auto-StaticCache on Neuron device: [#44748](https://github.com/huggingface/transformers/issues/44748)

--- Comment by dacorvo at 2026-03-16T14:54:46Z ---
Auto-compilation gate fix for Neuron: #44757

--- Comment by github-actions[bot] at 2026-04-16T08:29:48Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by dacorvo at 2026-04-17T13:47:38Z ---
@Cyrilvallez would love to have your feedback on this subject. Is the extension of the StaticCache capabilities a way to improve the generation loop efficiency when dealing with static shapes ?

cc @tengomucho

--- Comment by Cyrilvallez at 2026-04-23T05:31:21Z ---
Hey @dacorvo! Thanks for opening the issue. First I'd like to perfectly understand the issue: Neuron needs static shapes all the time if I understand correctly right? Because currently, we do compile only the `forward` pass, and only after prefill, so we have static shapes but indeed only inside the `forward` and during the decoding step (i.e. not in.-between `forward` calls).

In general, as you said the critical point is the mask. It would be relatively straightforward to use fixed-shape `input_ids` and `position_ids`, but the mask would require more work. `position_ids` cannot be retrieved from `cache.cumulative_length`, as the latter does not track padding tokens (however as I said, the `position_ids` could be tracked independently, so this would not be an issue in general).

What I'm most concerned about here is maintainability. This would add significant code paths only for the benefit of 1 type of hardware. The advantage of the current approach is that it's agnostic to cache/no-cache and static/non-static. We then only slice according to what we need.

I believe the best approach is to instead use `generate_batch`, where all shapes are fixed by design. It will also provides better perfs, and only supports a subset of `generate` that matches yours (as your approach could not be used for e.g. speculative decoding for example). cc @remi-or here for viz/help.

Let me know if that answers you @dacorvo!

--- Comment by tengomucho at 2026-04-29T09:26:52Z ---
@Cyrilvallez I will let @dacorvo answer, but i think that without these changes, working with transformers on hardware platforms that trigger compilation for every shape change such as Trainium or TPUs will be very challenging.
`generate_batch` does not solve all problems, and it seems to me that the design is more to allow continuous batching (for example for `transformers serve`).

--- Comment by remi-or at 2026-05-04T06:46:09Z ---
It can be used for inference as well! What are the problems left if you switch to `generate_batch` ?

--- Comment by dacorvo at 2026-05-07T13:35:08Z ---
@Cyrilvallez thanks for the answer.
First, an apology, by rereading the code I realize I missed the fact that `prepare_inputs_for_generation` is already cache-aware (it already creates a causal mask tensor of fixed shape).

So the only issue is actually the growing input_ids, position_ids and attention_mask tensors (and a few others for different model types).
Since they are on device, every operations triggers a recompile (the Neuron eager mode is fake: it is actually a per-op compilation).
For instance this leads to a recompilation:
``
input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
``
Even if we eventually pass `input_ids[-1] to the `forward.

So if we look at this only from a neuron perspective (purely hypothetical, not saying we should do that), either we keep them on CPU and pay the sync cost every time we decode (update 4D causal mask is one sync + sliced input_ids is another one, both will probably collapse), or we make them to use static shapes to at least avoid the recompilation.
I understand the latter is something you are reluctant to maintain, could we consider the other option ?

--- Comment by dacorvo at 2026-05-07T13:43:31Z ---
I updated the issue and the description on this narrower scope.

--- Comment by dacorvo at 2026-05-07T14:58:18Z ---
Regarding `generate_batch`, it looks like:
- it supports a lot less features than generate,
- it has PagedAttention support as a prerequisite.

The neuron integration in transformers is not to provide the best inference performance (for that the target is rather vLLM), but to provide the widest set of models and features possible, so that it becomes appealing to machine learning engineer to prototype models and tools on Neuron.

--- Comment by remi-or at 2026-05-11T01:40:52Z ---
Yes, `generate_batch` supports a lot less features than generate, but I wonder what kind of models you would like to see added? We already support all texts models, and this could be the occasion to broaden the support of some models.
As for PagedAttention, it is not a requirement: you can use `eager`, `sdpa` or `flash`. It is more efficient to use Paged for decoding (a lot more actuallly) but not required at all.
Big picture, it might be a good thing to change `generate_batch` to support "most" use cases, but it will surely be easier to add some small features to generate, despite the fact that they could only be useful for Neuron. I think it depends on what direction we want to follow in the future.


ISSUE #44741: [Neuron] Improve transformers compatibility with AWS Neuron devices
State: closed  |  Labels:
Author: dacorvo  |  Created: 2026-03-16T08:38:23Z

## Context

AWS Neuron devices (Trainium/Inferentia) compile a separate **NEFF** (Neuron Executable File Format) for every unique tensor shape. Any code path that changes tensor shapes between iterations — growing masks, `torch.cat` on outputs, variable padding — triggers expensive recompilations (2–60s per NEFF depending on model size).

This umbrella issue tracks non-generation-loop changes needed in transformers for Neuron compatibility. Generation loop changes (`GenerationMixin._prefill`, `_sample`, `generate`) are tracked separately in #44742.

Performance numbers below are measured on Llama-3.1-8B at TP=2 on trn2.3xlarge.

---

## Tracked issues

### Kernel mask function dispatch
**Status:** PR open — #44680 (issue #44679)

`load_and_register_attn_kernel()` hardcodes the mask function to `flash_attention_2` for all custom kernels. Kernels that need SDPA-style 4D boolean masks get the wrong mask format. This is needed for an upcoming NKI (Neuron Kernel Interface) SDPA kernel that will be published as a HuggingFace Hub kernel. Fix: let kernel modules declare `MASK_FUNCTION = "sdpa"` (or any valid mask type), falling back to `flash_attention_2` for backward compatibility.

### `index_select` instead of fancy indexing in `batched_mm_experts_forward`
**Status:** Not required, advanced indexing now supported by torch-neuronx.

~~`batched_mm_experts_forward` uses fancy indexing (`self.gate_up_proj[expert_ids]`) which is ambiguous for non-CUDA compiler backends. `torch.index_select` is the explicit, unambiguous API. Semantically identical, no behavioral change.~~

### Auto-select StaticCache on Neuron device
**Status:** Issue open — #44748

When no cache is specified and `device.type == "neuron"`, auto-select `StaticCache` in `_prepare_cache_for_generation`. Depends on #44742 (StaticCache-friendly `_sample`).

### Auto-dispatch NKI SDPA kernel on Neuron device
**Status:** Open — needs design

`infer_device()` already returns `"neuron"` for Neuron devices, but `_check_and_adjust_attn_implementation()` has no auto-dispatch path for Neuron — unlike CUDA which has `FLASH_ATTN_KERNEL_FALLBACK`. When `device.type == "neuron"` and no explicit `attn_implementation` is set, the model falls back to `eager`, missing the NKI SDPA kernel that will be published as a Hub kernel.

Proposed: add a Neuron-specific fallback in `_check_and_adjust_attn_implementation()` (analogous to `FLASH_ATTN_KERNEL_FALLBACK`) that auto-calls `load_and_register_attn_kernel()` with the NKI SDPA Hub repo when running on a Neuron device. Depends on the mask dispatch fix (#44679) since the NKI kernel uses SDPA-style masks.

---

## Resolved issues (no upstream changes needed)

- **NKI flash attention backend** — resolved via HuggingFace `kernels` library ([kernels PR #285](https://github.com/huggingface/kernels/pull/285)). A dedicated NKI kernel for Neuron SDPA will be published as a HuggingFace Hub kernel.
- **StaticCache allocation** — resolved by passing pre-built `StaticCache` as `past_key_values=` (existing API)

## Related

- Generation loop changes: #44742
- OLMoE TP plan: #44677 / PR #44668 (general feature, not Neuron-specific)

--- Comment by github-actions[bot] at 2026-04-16T08:29:50Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by dacorvo at 2026-04-17T14:02:28Z ---
Closing this as most of the work will happen in the two other issues


ISSUE #44740: Run model linter on new models additions
State: closed  |  Labels:
Author: tarekziade  |  Created: 2026-03-16T08:04:16Z

- detect PRs with new models
- add a `new model` label
- run the linter and when failing comment the PR with errors

--- Comment by sasmita016 at 2026-03-21T18:16:44Z ---
hi can i work on this ?

--- Comment by github-actions[bot] at 2026-04-17T08:29:51Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44737: XLNet: relative_positional_encoding computes on CPU every forward pass (missing device= in torch.arange)
State: closed  |  Labels: bug
Author: mvstrauss  |  Created: 2026-03-16T06:30:52Z

### System Info

- `transformers` version: confirmed on 4.30.2, still present on `main` as of 2026-03-16
- Platform: Linux (tested on NVIDIA/AMD GPUs)
- PyTorch: 2.x
- Python: 3.10+

### Who can help?

@ArthurZucker @Rocketknight1

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`XLNetModel.relative_positional_encoding` (in `src/transformers/models/xlnet/modeling_xlnet.py`, lines ~940–976) creates all intermediate tensors on CPU because `torch.arange()` is called without `device=`. The entire sinusoidal positional encoding computation (arange → pow → einsum → sin/cos → cat → expand) runs on CPU every forward pass, with only the final result being copied to GPU via `.to(output_h.device)` on line ~1143.

There are four affected `torch.arange` calls:

```python
# Line 942 — freq_seq on CPU
freq_seq = torch.arange(0, self.d_model, 2.0, dtype=torch.int64).float()

# Line 955 — fwd_pos_seq on CPU (bi_data path)
fwd_pos_seq = torch.arange(beg, end, -1.0, dtype=torch.int64).float()

# Line 956 — bwd_pos_seq on CPU (bi_data path)
bwd_pos_seq = torch.arange(-beg, -end, 1.0, dtype=torch.int64).float()

# Line 971 — fwd_pos_seq on CPU (non-bi_data path)
fwd_pos_seq = torch.arange(beg, end, -1.0, dtype=torch.int64).float()
```

Since none of these specify `device=`, all downstream operations (`torch.pow`, `torch.einsum`, `torch.sin`, `torch.cos`, `torch.cat`, `.expand`) also execute on CPU. The `.to(output_h.device)` call in `forward()` only moves the *final tensor* to GPU — it does not retroactively move the computation.

This is called on every forward pass (not cached), causing:
- Unnecessary CPU computation that the GPU could perform faster
- A CPU→GPU memory copy and synchronization point per forward pass
- Measurable training/inference slowdown that scales with sequence length

### Expected behavior

All tensor creation inside `relative_positional_encoding` should happen on the model's device so the full computation stays on GPU.

### Suggested fix

Pass the model's device to all `torch.arange()` calls. The device can be obtained from `self.word_embedding.weight.device` (or equivalently `self.device`). Minimal diff:

```diff
 def relative_positional_encoding(self, qlen, klen, bsz=None):
     # create relative positional encoding.
-    freq_seq = torch.arange(0, self.d_model, 2.0, dtype=torch.int64).float()
+    freq_seq = torch.arange(0, self.d_model, 2.0, dtype=torch.int64, device=self.device).float()
     inv_freq = 1 / torch.pow(10000, (freq_seq / self.d_model))

     if self.attn_type == "bi":
@@ -954,8 +954,8 @@
         raise ValueError(f"Unknown `attn_type` {self.attn_type}.")

     if self.bi_data:
-        fwd_pos_seq = torch.arange(beg, end, -1.0, dtype=torch.int64).float()
-        bwd_pos_seq = torch.arange(-beg, -end, 1.0, dtype=torch.int64).float()
+        fwd_pos_seq = torch.arange(beg, end, -1.0, dtype=torch.int64, device=self.device).float()
+        bwd_pos_seq = torch.arange(-beg, -end, 1.0, dtype=torch.int64, device=self.device).float()

         if self.clamp_len > 0:
             fwd_pos_seq = fwd_pos_seq.clamp(-self.clamp_len, self.clamp_len)
@@ -968,7 +968,7 @@

         pos_emb = torch.cat([fwd_pos_emb, bwd_pos_emb], dim=1)
     else:
-        fwd_pos_seq = torch.arange(beg, end, -1.0, dtype=torch.int64).float()
+        fwd_pos_seq = torch.arange(beg, end, -1.0, dtype=torch.int64, device=self.device).float()
         if self.clamp_len > 0:
             fwd_pos_seq = fwd_pos_seq.clamp(-self.clamp_len, self.clamp_len)
         pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz)
```

After this fix, the existing `pos_emb = pos_emb.to(output_h.device)` in `forward()` becomes a no-op (same device), which could optionally be removed as well.

Note: `XLNetModel` inherits from `PreTrainedModel` which provides the `.device` property, so no additional device resolution logic is needed.

We are currently working around this with a runtime monkey-patch on our side but would prefer to see this fixed upstream.

--- Comment by BillionClaw at 2026-03-17T04:40:43Z ---
Looking into this — the missing device parameter in torch.arange is causing CPU computation. Happy to submit a fix.

--- Comment by JiwaniZakir at 2026-03-17T05:11:16Z ---
I think I see what's going on here. Going to put together a fix.

--- Comment by michaelanderson01826-glitch at 2026-03-17T14:11:33Z ---
Itight be useful to add this to the docs

On Tue, Mar 17, 2026, 5:11 AM Zakir Jiwani ***@***.***> wrote:

> *JiwaniZakir* left a comment (huggingface/transformers#44737)
> <https://github.com/huggingface/transformers/issues/44737#issuecomment-4072381248>
>
> I think I see what's going on here. Going to put together a fix.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/44737#issuecomment-4072381248>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEIFXTJ4DJJ6DLSYSTL4RDNBDAVCNFSM6AAAAACWS7EIHGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANZSGM4DCMRUHA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


ISSUE #44734: `transformers serve`: /v1/responses crashes on KV cache continuation due to wrong tensor indexing
State: closed  |  Labels:
Author: mango766  |  Created: 2026-03-16T04:08:09Z

### System Info

- transformers version: 5.3.0 (latest main)
- Platform: Linux / macOS
- Python version: 3.12

### Who can help?

@LysandreJik @Rocketknight1

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

In `src/transformers/cli/serve.py`, the `generate_response` method (streaming /v1/responses endpoint) crashes when the server attempts to reuse a KV cache from a previous request in the same session.

The issue is on [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/cli/serve.py#L1331):

```python
inputs = processor.apply_chat_template(
    inputs, add_generation_prompt=True, return_tensors="pt", return_dict=True
)["input_ids"]     # <-- extracts tensor from dict
inputs = inputs.to(model.device)

# ... later:
if self.is_continuation(req) and not must_discard_cache:
    seq_len = self.last_kv_cache.get_seq_length()
    if inputs["input_ids"].shape[-1] > seq_len:  # BUG: inputs is already a tensor, not a dict
        last_kv_cache = self.last_kv_cache
```

At this point `inputs` is already the raw `input_ids` tensor (extracted with `["input_ids"]` on line 1312), so `inputs["input_ids"]` will raise a `TypeError` because you can't index a tensor with a string key.

Steps to reproduce:
1. Run `transformers serve`
2. Send a /v1/responses request
3. Send a second request that continues the same conversation (triggering KV cache reuse)
4. The second request crashes

### Expected behavior

The continuation check should use `inputs.shape[-1]` instead of `inputs["input_ids"].shape[-1]`, matching the non-streaming counterpart `generate_response_non_streaming` which correctly uses `inputs.shape[-1]` on [line 1632](https://github.com/huggingface/transformers/blob/main/src/transformers/cli/serve.py#L1632).

--- Comment by LysandreJik at 2026-03-16T15:11:39Z ---
Hey @mango766, thanks for your PR, taking a look!