ISSUE #44589: TypeError: couldn't find storage object Float8_e4m3fnStorage
State: closed  |  Labels: bug
Author: xin3he  |  Created: 2026-03-11T02:16:53Z

### System Info

- `transformers` version: 5.3.0.dev0
- Platform: Linux-6.8.0-101-generic-x86_64-with-glibc2.35
- Python version: 3.12.13
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

@CyrilVallez

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

transformers serve --force-model Qwen/Qwen3.5-35B-A3B-FP8 --port 8000 --continuous-batching

```
  File "/home/xinhe/auto-round/.venv/lib/python3.12/site-packages/transformers/cli/serve.py", line 474, in __init__
    self.load_model_and_processor(model_id_and_revision)
  File "/home/xinhe/auto-round/.venv/lib/python3.12/site-packages/transformers/cli/serve.py", line 1876, in load_model_and_processor
    model, processor = self._load_model_and_data_processor(model_id_and_revision)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xinhe/auto-round/.venv/lib/python3.12/site-packages/transformers/cli/serve.py", line 1850, in _load_model_and_data_processor
    model = architecture.from_pretrained(model_id, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xinhe/auto-round/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4096, in from_pretrained
    with ContextManagers(model_init_context):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xinhe/auto-round/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 532, in __enter__
    self.stack.enter_context(context_manager)
  File "/home/xinhe/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 526, in enter_context
    result = _enter(cm)
             ^^^^^^^^^^
  File "/home/xinhe/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/xinhe/auto-round/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 240, in local_torch_dtype
    torch.set_default_dtype(dtype)
  File "/home/xinhe/auto-round/.venv/lib/python3.12/site-packages/torch/__init__.py", line 1358, in set_default_dtype
    _C._set_default_dtype(d)
TypeError: couldn't find storage object Float8_e4m3fnStorage
```

### Expected behavior

The FP8 model should be loaded correctly.

--- Comment by zeokin at 2026-03-11T03:02:54Z ---
Hi @xin3he
I'd love to work on this. Can I pick this up please?

--- Comment by xin3he at 2026-03-11T06:59:08Z ---
Sure, but I'm not a member of transformers. It would be helpful if you can help.

--- Comment by Anantingale at 2026-03-11T07:10:02Z ---
can i help you guys working on it ?

--- Comment by Rocketknight1 at 2026-03-12T12:25:28Z ---
Please don't run your code agent on this one! We'd like to investigate and confirm the issue before we just start monkey patching

--- Comment by DavidNemeskey at 2026-03-13T21:23:10Z ---
I got the same error when trying to load a Qwen 3.5 model.

CUDA_VISIBLE_DEVICES=0,2
GPU type=H200

```
In [1]: import torch
   ...: from transformers import Qwen3_5MoePreTrainedModel

In [2]: model = Qwen3_5MoePreTrainedModel.from_pretrained(
   ...:     'Qwen/Qwen3.5-122B-A10B-FP8',
   ...:     dtype="auto",
   ...:     device_map="auto"
   ...: )
Fetching 39 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39/39 [00:00<00:00, 12227.38it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                                                | 0/39 [00:00<?, ?it/s]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 model = Qwen3_5MoePreTrainedModel.from_pretrained(
      2     'Qwen/Qwen3.5-122B-A10B-FP8',
      3     dtype="auto",
      4     device_map="auto"
      5 )

File /home/user/miniconda3/envs/qwen3_5/lib/python3.13/site-packages/transformers/modeling_utils.py:4093, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, con
fig, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, weights_only, *model_args, **kwargs)
   4090 model_init_context = cls.get_init_context(dtype, is_quantized, _is_ds_init_called, allow_all_kernels)
   4092 config = copy.deepcopy(config)  # We do not want to modify the config inplace in from_pretrained.
-> 4093 with ContextManagers(model_init_context):
   4094     model = cls(config, *model_args, **model_kwargs)
   4095     patch_output_recorders(model)

File /home/user/miniconda3/envs/qwen3_5/lib/python3.13/site-packages/transformers/utils/generic.py:524, in ContextManagers.__enter__(self)
    522 def __enter__(self):
    523     for context_manager in self.context_managers:
--> 524         self.stack.enter_context(context_manager)

File /home/user/miniconda3/envs/qwen3_5/lib/python3.13/contextlib.py:530, in _BaseExitStack.enter_context(self, cm)
    527 except AttributeError:
    528     raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
    529                     f"not support the context manager protocol") from None
--> 530 result = _enter(cm)
    531 self._push_cm_exit(cm, _exit)
    532 return result

File /home/user/miniconda3/envs/qwen3_5/lib/python3.13/contextlib.py:141, in _GeneratorContextManager.__enter__(self)
    139 del self.args, self.kwds, self.func
    140 try:
--> 141     return next(self.gen)
    142 except StopIteration:
    143     raise RuntimeError("generator didn't yield") from None

File /home/user/miniconda3/envs/qwen3_5/lib/python3.13/site-packages/transformers/modeling_utils.py:240, in local_torch_dtype(dtype, model_class_name)
    238 original_dtype = torch.get_default_dtype()
    239 try:
--> 240     torch.set_default_dtype(dtype)
    241     yield
    242 finally:

File /home/user/miniconda3/envs/qwen3_5/lib/python3.13/site-packages/torch/__init__.py:1363, in set_default_dtype(d)
   1313 def set_default_dtype(d: "torch.dtype", /) -> None:
   1314     r"""
   1315
   1316     Sets the default floating point dtype to :attr:`d`. Supports floating point dtype
   (...)   1361
   1362     """
-> 1363     _C._set_default_dtype(d)

TypeError: couldn't find storage object Float8_e4m3fnStorage
```

--- Comment by Cyrilvallez at 2026-03-20T11:49:32Z ---
Thanks for the report @xin3he and @DavidNemeskey! Good try to the other and their agents, but wasn't the fix we wanted 😅 Will be fixed by https://github.com/huggingface/transformers/pull/44883


ISSUE #44574: Title: Clarification and consistency of "Transformers" terminology in README
State: closed  |  Labels:
Author: rabbierabbie  |  Created: 2026-03-10T14:40:08Z

While reading the **README as a new user**, I found the use of the term **"Transformers" slightly ambiguous**. At first glance it can refer either to the **Transformer architecture** or to the **Transformers Python library**.
For example, the sentence:
"Transformers acts as the model-definition framework for state-of-the-art machine learning..." appears to refer to the library, but the wording initially made it unclear whether it referred to the architecture or the 🤗 Transformers library.

Additionally, the README sometimes uses:
- "Transformers"
- "transformers" (the Python package)
- "🤗 Transformers"
which may be slightly confusing for beginners encountering the project for the first time.

**### Suggestion:**

Would it make sense to clarify references to the library in places where it may be ambiguous, for example by writing: **"the Transformers library" or using the existing project style such as "🤗 Transformers"**?

I would be happy to open a small documentation PR once the preferred wording/style is confirmed.

--- Comment by Rocketknight1 at 2026-03-11T13:24:42Z ---
We'd prefer to keep it as-is! I think it's clear from context which is meant, and it bloats the text a lot to continuously say "The Transformers library"

--- Comment by rabbierabbie at 2026-03-12T04:38:08Z ---
@Rocketknight1 Thanks for the clarification!

I completely understand the concern about avoiding repeated mentions of "the Transformers library" throughout the README.

I was wondering whether it might still make sense to clarify the **very first occurrence** in the introduction. As a new reader, I initially interpreted "Transformers" as referring to the Transformer architecture rather than the library, and it was only from the grammar later in the sentence that I realized it referred to the library.

Something like:

"The Transformers library acts as the model-definition framework..."

would clarify the first reference while keeping the rest of the document unchanged.

Of course, if this still goes against the preferred style, I’m happy to follow the current convention. Thanks again for the feedback!


<spam>

ISSUE #44573: <spam>
State: closed  |  Labels:
Author: aiqing20230305-bot  |  Created: 2026-03-10T14:37:42Z

<spam>


<spam>

ISSUE #44572: <spam>
State: closed  |  Labels:
Author: aiqing20230305-bot  |  Created: 2026-03-10T14:28:30Z

<spam>


ISSUE #44568: [BUG] add_special_tokens=True doesn't add BOS/EOS tokens for microsoft/mdeberta-v3-base tokenizer in transformers >=5.0
State: closed  |  Labels: bug
Author: Abdullahaml1  |  Created: 2026-03-10T11:43:59Z

### System Info

## Version Details
- Working version: transformers==4.48.0
- Broken versions: transformers==5.0.0, 5.1.0, 5.2.0, 5.3.0
## Environment
- transformers: 5.2.0
- tokenizers: 0.22.2
- Python: 3.12
- Platform: Linux

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

## Description
In transformers >=5.0, `add_special_tokens=True` doesn't add special tokens for `microsoft/mdeberta-v3-base` tokenizer. This is a regression from v4.x.
## Reproduction
```python
from transformers import AutoTokenizer
models = [
    "microsoft/mdeberta-v3-base",
    "FacebookAI/roberta-base",
    "bert-base-uncased"
]
for model_name in models:
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    result = tokenizer("hello", add_special_tokens=True)
    print(f"{model_name}: input_ids={result['input_ids']}")
```

## Additional Notes
- The issue is MODEL-SPECIFIC, not general to all tokenizers
- Only microsoft/mdeberta-v3-base is affected
- The tokenizer has correct bos_token_id=1 and eos_token_id=2 values
- This appears to be related to DeBERTa v3's SentencePiece-based tokenizer and the v5 tokenizer redesign
## Expected Behavior
The behavior should be consistent across v4.x and v5.x for backward compatibility.

### Expected behavior

```
Expected (v4.48.0 - Working correctly)
- microsoft/mdeberta-v3-base: [1, 124394, 2] (CLS, hello, SEP)
- FacebookAI/roberta-base: [0, 42891, 2] (<s>, hello, </s>)
- bert-base-uncased: [101, 7592, 102] (CLS, hello, SEP)
Actual (v5.2.0 - Broken for mdeberta only)
- microsoft/mdeberta-v3-base: [124394] ← NO special tokens!
- FacebookAI/roberta-base: [0, 42891, 2] ← Works
- bert-base-uncased: [101, 7592, 102] ← Works
```


ISSUE #44561: Removal of `is_torch_fx_available` in v5.0 breaks `trust_remote_code` models
State: closed  |  Labels:
Author: janbernloehr  |  Created: 2026-03-10T09:22:21Z

### System Info

- `transformers` version: 5.0.0
- PyTorch version: 2.10.0
- Python version: 3.12

### Who can help?

@ArthurZucker @Rocketknight1

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder
- [x] My own task or dataset (give details below)

### Reproduction

Several HuggingFace Hub models that use `trust_remote_code=True` import `is_torch_fx_available` from `transformers.utils.import_utils`. This function was removed in [#37234](https://github.com/huggingface/transformers/pull/37234) ("Remove old code for PyTorch, Accelerator and tokenizers"), which landed in transformers 5.0.0.

Since these models ship their own `modeling_*.py` on the Hub and are loaded dynamically, they break at import time with no way for users to fix it (short of monkey-patching or forking the model code).

**Example — `deepseek-ai/deepseek-moe-16b-base`:**

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/deepseek-moe-16b-base",
    trust_remote_code=True,
)
```

Traceback:

```
File ".../transformers/dynamic_module_utils.py", line 309, in get_class_in_module
  ...
File ".../modules/transformers_modules/deepseek.../modeling_deepseek.py", line 50
    from transformers.utils.import_utils import is_torch_fx_available
ImportError: cannot import name 'is_torch_fx_available' from 'transformers.utils.import_utils'
```

The offending line in the Hub model code ([`modeling_deepseek.py` line 50](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/main/modeling_deepseek.py#L50)):

```python
from transformers.utils.import_utils import is_torch_fx_available
```

This pattern was common in transformers v4.x and was used in many community models that copied from the official codebase at the time. The function was used to conditionally `torch.fx.wrap()` attention mask helpers.

### Affected models

At minimum:
- `deepseek-ai/deepseek-moe-16b-base`
- `deepseek-ai/deepseek-moe-16b-chat`
- Likely many other older community models that followed the same pattern

### Suggested fix

Add a backwards-compatibility shim in `transformers/utils/import_utils.py`:

```python
def is_torch_fx_available():
    """Deprecated: kept for backwards compatibility with trust_remote_code models."""
    return True
```

Since the function was removed because PyTorch >= 2.1 is now the minimum (where `torch.fx` is always available), returning `True` is correct. This would unbreak all `trust_remote_code` models that reference this symbol without any behavioral change.

Alternatively, a `DeprecationWarning` could be emitted to nudge model authors to update their code.

### Expected behavior

Loading models with `trust_remote_code=True` should not break due to internal API removals in transformers, especially for widely-used symbols. A compatibility shim (returning `True`) would maintain backwards compatibility at zero cost.

--- Comment by ArthurZucker at 2026-03-10T10:20:31Z ---
Hey!
1. `modeling_deepseek` is part of transformers you don't need remote code
2. this IS a major release, which is why there are breaking changes, this is probably not the only one.
3. the fix would be "wrong".
4. Happy to review a PR that proposes a way to monkey patch remote code / sanitizing them? But the best is to either open a PR to the repo or to transformers !


ISSUE #44560: Qwen3-vl-embedding Video Error "StopIteration" in transformers 5.3.0
State: closed  |  Labels: bug
Author: QYQTexas  |  Created: 2026-03-10T08:30:00Z

### System Info

transformers version: 5.3.0
Platform: Windows11, WSL2, uv, vscode
Python 3.12.13 (main, Mar  3 2026, 14:59:34) [Clang 21.1.4 ] on linux

### Who can help?

@zucchini-nlp

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Following README of https://github.com/[QwenLM/Qwen3-VL-Embedding](https://github.com/QwenLM/Qwen3-VL-Embedding/tree/main)/tree/main
2. Using video for embedding.
```python
import torch
from Qwen3_VL_Embedding.src.models.qwen3_vl_embedding import Qwen3VLEmbedder

model = Qwen3VLEmbedder(
    model_name_or_path="./models/Qwen3-VL-Embedding-2B",
    # flash_attention_2 for better acceleration and memory saving
    # torch_dtype=torch.bfloat16,
    # attn_implementation="flash_attention_2"
)

inputs = [
    ## Official example
#     {
#     "text": "A woman playing with her dog on a beach at sunset.",
#     "instruction": "Retrieve images or text relevant to the user's query.",
# }, {
#     "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust."
# }, {
#     "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
# }, {
#     "text": "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset, as the dog offers its paw in a heartwarming display of companionship and trust.",
#     "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
# }

    ## What I try
{
    "video":'./data/video/1602562103_1491434826.mp4' # 30min
},{
    "video":"./data/video/115814632003756_35105146583.mp4" # 3min
}
]

embeddings = model.process(inputs)
print(embeddings)
print(embeddings @ embeddings.T)
```
```text
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 625/625 [00:00<00:00, 1343.66it/s]
qwen-vl-utils using decord to read video.
Traceback (most recent call last):
  File "/root/graduation_project/test_video_embedding_local.py", line 23, in <module>
    embeddings = model.process(inputs)
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/Qwen3_VL_Embedding/src/models/qwen3_vl_embedding.py", line 387, in process
    outputs = self.forward(processed_inputs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/Qwen3_VL_Embedding/src/models/qwen3_vl_embedding.py", line 193, in forward
    outputs = self.model(**inputs)
              ^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/Qwen3_VL_Embedding/src/models/qwen3_vl_embedding.py", line 99, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 843, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1358, in forward
    position_ids = self.compute_3d_position_ids(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1252, in compute_3d_position_ids
    position_ids, rope_deltas = self.get_rope_index(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/root/graduation_project/.venv/lib/python3.12/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1135, in get_rope_index
    grid_thw = next(grid_iters[modality_type])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration
```

### Expected behavior

There is no problem with v4.57.6, v5.1.0, v5.2.0 but v5.3.0

--- Comment by zucchini-nlp at 2026-03-10T09:43:13Z ---
Fixed by https://github.com/huggingface/transformers/pull/44474, will be merging today

--- Comment by zucchini-nlp at 2026-03-10T09:57:21Z ---
Merged, closing as resolved!

--- Comment by QYQTexas at 2026-03-10T10:15:25Z ---
> Fixed by [#44474](https://github.com/huggingface/transformers/pull/44474), will be merging today

Thank you!


ISSUE #44559: flash-attn-4 (flash_attn.cute) is not supported by attn_implementation="flash_attention_2"
State: closed  |  Labels: Feature request
Author: DimensionSTP  |  Created: 2026-03-10T07:47:09Z

### Feature request

# Support `flash-attn-4` (`flash_attn.cute`) in Transformers attention backend selection

## System Info
- `transformers==5.3.0`
- `torch==2.10.0+cu128`
- `flash-attn-4==4.0.0b4`
- `accelerate==1.13.0`
- `trl==0.29.0`
- `peft==0.18.0`
- `deepspeed==0.18.7`
- `tokenizers==0.22.2`
- `huggingface_hub==1.6.0`
- Python 3.12
- CUDA 12.8
- GPU: NVIDIA Blackwell (`sm120`)

## Information
- [ ] The official example scripts
- [x] My own modified scripts
- [ ] I am willing to open a PR

## Reproduction

I am testing a Blackwell environment with:
- PyTorch 2.10
- CUDA 12.8
- `flash-attn-4`
- `transformers` 5.3.0

Model loading fails before training starts when I pass:

```python
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

## Actual behavior

Transformers appears to still expect the FlashAttention v2-style top-level import:

```python
from flash_attn import flash_attn_func, flash_attn_varlen_func
```

But `flash-attn-4` exposes the relevant API under:

```python
from flash_attn.cute import flash_attn_func, flash_attn_varlen_func
```

As a result, model initialization fails with:

```text
ImportError: cannot import name 'flash_attn_func' from 'flash_attn' (unknown location)
```

This is the traceback I get during training:

```text
Traceback (most recent call last):
  File "/home/joshua/llm-fine-tune-hf/main.py", line 68, in <module>
    main()
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/joshua/llm-fine-tune-hf/main.py", line 54, in main
    return train(config)
  File "/home/joshua/llm-fine-tune-hf/src/pipelines/pipeline.py", line 55, in train
    model = setup.get_model()
  File "/home/joshua/llm-fine-tune-hf/src/utils/setup.py", line 116, in get_model
    model = AutoModelForCausalLM.from_pretrained(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 374, in from_pretrained
    return model_class.from_pretrained(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4094, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 461, in __init__
    super().__init__(config)
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/transformers/modeling_utils.py", line 1260, in __init__
    self.config._attn_implementation_internal = self._check_and_adjust_attn_implementation(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/transformers/modeling_utils.py", line 1893, in _check_and_adjust_attn_implementation
    lazy_import_flash_attention(applicable_attn_implementation)
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/transformers/modeling_flash_attention_utils.py", line 171, in lazy_import_flash_attention
    _flash_fn, _flash_varlen_fn, _pad_fn, _unpad_fn = _lazy_imports(
  File "/home/joshua/anaconda3/envs/joshpp/lib/python3.12/site-packages/transformers/modeling_flash_attention_utils.py", line 96, in _lazy_imports
    from flash_attn import flash_attn_func, flash_attn_varlen_func
ImportError: cannot import name 'flash_attn_func' from 'flash_attn' (unknown location)
```

## Expected behavior

One of the following would solve this cleanly:

1. Detect `flash-attn-4` and import from `flash_attn.cute` when that package is installed.
2. Introduce an explicit backend such as `attn_implementation="flash_attention_4"`.
3. Document that `flash-attn-4` is not yet supported by the current attention backend selection logic.

## Why this matters

Blackwell users moving to newer CUDA / PyTorch stacks are likely to try `flash-attn-4`, but the current import path fails before training begins. This makes the newer FA4 stack unusable from stock Transformers attention selection even though the FA4 functions are present and importable from `flash_attn.cute`.

## Additional notes

In the same machine, an older stack works:
- `transformers==4.57.3`
- `flash_attn==2.8.3`

That older stack exports `flash_attn_func` at the package top level, so the current Transformers import path works there.

By contrast, in the newer environment:

```python
from flash_attn.cute import flash_attn_func, flash_attn_varlen_func
```

works, while:

```python
from flash_attn import flash_attn_func
```

does not.

### Motivation

I am trying to use a newer Blackwell training stack with `torch==2.10.0+cu128`, CUDA 12.8, and `flash-attn-4==4.0.0b4`.

At the moment, `transformers==5.3.0` appears to assume the FlashAttention v2-style top-level API when `attn_implementation="flash_attention_2"` is selected. However, `flash-attn-4` exposes its functions under `flash_attn.cute` instead of the older top-level import path. Because of this, model loading fails before training even starts with:

```text
ImportError: cannot import name 'flash_attn_func' from 'flash_attn'
```

This makes the newer FA4 stack unusable from stock Transformers attention backend selection, even though the FA4 functions themselves are present and importable.

This seems related in spirit to earlier flash-attn compatibility/import issues, for example:
- #35899
- #27002

My main motivation is to make newer Blackwell-oriented FlashAttention stacks usable from Transformers without requiring users to patch library internals locally.

### Your contribution

Yes, I can help with a PR.

I can test proposed changes on a local Blackwell environment using:
- `torch==2.10.0+cu128`
- `flash-attn-4==4.0.0b4`
- `transformers==5.3.0`

If the maintainers agree on the intended direction, I can help with:
- validating a fix for `flash-attn-4` detection/import
- testing whether `flash_attn.cute` can be supported safely
- verifying that the change does not break the existing `flash_attn` v2 path

I have read the contribution guidance and can prepare a focused PR once the preferred approach is confirmed.

--- Comment by Rocketknight1 at 2026-03-10T11:37:32Z ---
PR is already open here! https://github.com/huggingface/transformers/pull/42435

--- Comment by DimensionSTP at 2026-03-11T06:53:38Z ---
This issue is due to the lack of FlashAttention-4 support in the current Transformers release.

Initial FA4 support is being added in PR #42435:
https://github.com/huggingface/transformers/pull/42435

Closing this issue for now since it should be resolved once that PR is merged. Please feel free to reopen if the problem persists afterward.


ISSUE #44556: Loading checkpoint trained on v4.57 cannot be reload after upgraded to v5.2 & v5.3
State: closed  |  Labels: bug
Author: ihungalexhsu  |  Created: 2026-03-10T04:56:49Z

### System Info

I have trained models using Qwen3 using v4.57 but the ckpt loading will hang forever after Loading weights: 100%|██████████| 708/708 [00:09<00:00, 78.04it/s] .

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using from_pretrained function, provided with local_path.

The program hangs there forever.

### Expected behavior

The model should be able to load

--- Comment by Rocketknight1 at 2026-03-10T11:32:00Z ---
Is the checkpoint uploaded to the Hub anywhere? This will be easier to debug if we can reproduce the issue!

--- Comment by ihungalexhsu at 2026-03-11T20:27:04Z ---
Found that the bug is in our server end. Sorry for the false alarm.


ISSUE #44554: [MPS] Upstream correctness issue in attention when value head dim differs from query
State: open  |  Labels: WIP, bug
Author: hvaara  |  Created: 2026-03-10T01:13:32Z

### System Info

There is a correctness issue with attention in PyTorch when using the MPS backend with value head dims different from the query head (see https://github.com/pytorch/pytorch/issues/176767).

Consider the following reproducer

```python
import torch
import torch.nn.functional as F

q = torch.rand(1, 1, 8, 4, device="mps")
k = torch.rand(1, 1, 8, 4, device="mps")
v = torch.rand(1, 1, 8, 2, device="mps")

y_mps1 = F.scaled_dot_product_attention(q, k, v)
y_mps2 = F.scaled_dot_product_attention(q, k, v)

print(f"{y_mps1-y_mps2 = }")

y_cpu1 = F.scaled_dot_product_attention(q.cpu(), k.cpu(), v.cpu())
y_cpu2 = F.scaled_dot_product_attention(q.cpu(), k.cpu(), v.cpu())

print(f"{y_cpu1-y_cpu2 = }")

print(f"{y_mps1.shape = }")
print(f"{y_cpu1.shape = }")

# Output:
# y_mps1-y_mps2 = tensor([[[[ 0.0000,  0.0000,  0.0000,  0.0000],
#           [ 0.0000,  0.0000,  0.0000,  0.0000],
#           [-0.0957, -0.1705, -0.1087, -0.1648],
#           [-0.1155, -0.1248, -0.0946, -0.1255],
#           [-0.1089, -0.1662, -0.1261, -0.1559],
#           [-0.0904, -0.1262, -0.0927, -0.1335],
#           [-0.1080, -0.1644, -0.1226, -0.1611],
#           [-0.0849, -0.1324, -0.0984, -0.1281]]]], device='mps:0')
# y_cpu1-y_cpu2 = tensor([[[[0., 0.],
#           [0., 0.],
#           [0., 0.],
#           [0., 0.],
#           [0., 0.],
#           [0., 0.],
#           [0., 0.],
#           [0., 0.]]]])
# y_mps1.shape = torch.Size([1, 1, 8, 4])
# y_cpu1.shape = torch.Size([1, 1, 8, 2])
```

The numeric mismatch above is due to uninitialized memory being part of the return tensor. Notice that there is also a shape mismatch from the expected result.

The issue has been fixed in PyTorch (https://github.com/pytorch/pytorch/pull/176843), but won't be released until PyTorch 2.12.

I don't know if different dims for query and value is a common use-case for Hugging Face models, but I wanted to notify you of the issue regardless. I'm happy to provide a workaround if you think this issue should be fixed in transformers. Potential workaround is to resize the output tensor and wrangle the strides, as the calculation itself is good for the first (expected) numels.

The issue is similar to #44247, which is another correctness issue in MPS SDPA needing workaroud for specific PyTorch versions.

### Who can help?

@vasqu @Cyrilvallez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See reproducer above. I'm not sure if this is an issue in transformers.

### Expected behavior

See reproducer above.

--- Comment by hvaara at 2026-03-10T03:07:26Z ---
I did a bit of testing to see if I could get a clean workaround, but I think I spoke too soon. I'm afraid it's not possible to produce a workaround using the output from attention as-is.

Another option would be to pad the value tensor. Something like

```python
Ev = v.size(-1)
E = q.size(-1)
v_pad = F.pad(v, (0, E - Ev))
y = F.scaled_dot_product_attention(q, k, v_pad)
y = y[..., :Ev]
```

--- Comment by vasqu at 2026-03-10T17:07:57Z ---
Sorry but what is up with MPS 😢 all up for a workaround in sdpa integration then. I wouldn't look into old models who don't support the attention interface, too much work too brittle.

Atp, it might be worth to add a decorator that handles all these edge cases if possible

--- Comment by Cyrilvallez at 2026-03-11T09:13:30Z ---
Humm, actually I'm not super sure we need any workaround for this one! I think we never have qk head dim different from v head dim! Only exceptions may be the deepseek models with mqa, I need to check. But tagging @vasqu here, as this one may not need any workaround rn and avoid some precious bloat of fixing broken upstream torch!

--- Comment by Cyrilvallez at 2026-03-11T09:48:08Z ---
Arf, the deepseeks indeed have different head dims between qk and v 😭 So we cannot avoid this one 😭 The best IMO is to create a new function in the sdpa integration, such as `_fixes_for_mps` or something, that would fix both linked issues for mps.  In this function, let's clearly document the issues and the affected versions. Then in `sdpa_attention_forward`, simply do something such as `if q.device.type == "mps": -> _fixes_for_mps(...)`

--- Comment by hvaara at 2026-03-11T12:48:55Z ---
I quite like @vasqu's idea of making a decorator. It works for both issues (this and #44247). The MPS workarounds are not relevant for any other backend, so a decorator would allow us to apply the fix without cluttering the code. It also puts everything in one place so it's easier to understand and makes it easier to remove when that time comes.

@Cyrilvallez Does a decorator SGTY?

--- Comment by hvaara at 2026-03-11T13:04:40Z ---
Since it was not clear in #44591: I am already working on this issue.

--- Comment by Cyrilvallez at 2026-03-12T10:39:28Z ---
Well, I don't really mind if you apply it as a decorator on top of the `sdpa_attention_forward` or if you simply call it in the body itself haha! I think in the body it may be a bit clearer/simple to implement, but both work - as long as you make sure the decorator does not change at all the behavior when the device is not "mps", i.e. you exit early to call the base function

--- Comment by github-actions[bot] at 2026-04-09T08:27:05Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by hvaara at 2026-04-17T11:52:17Z ---
Sorry I've been sleeping on this. I'll prioritize this and the other issue today. Is there a way to mark this not stale and reopen it?

--- Comment by vasqu at 2026-04-17T11:56:26Z ---
Just added the labels for that, hopefully shouldnt reclose this :D


ISSUE #44545: Qwen2_5_VLProcessor.apply_chat_template crashes on batched input when padding=False
State: closed  |  Labels:
Author: Anakintano  |  Created: 2026-03-09T12:37:17Z

## Bug Description

`Qwen2_5_VLProcessor.apply_chat_template` raises `ValueError: setting an array element with a sequence` when processing a batch of ≥2 conversations that include images, under the default `padding=False` setting.

**Root cause:** `mm_token_type_ids` was built by calling `np.array(text_inputs["input_ids"])` on a ragged list (variable-length sequences when `padding=False`). NumPy ≥ 1.24 rejects inhomogeneous shapes for this operation.

This is distinct from #44521, which concerns `assistant_masks` being all zeros for multimodal inputs.

## Reproduction

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

batch_messages = [
    [{"role": "user", "content": [{"type": "image", "image": "img1.jpg"}, {"type": "text", "text": "Describe."}]}],
    [{"role": "user", "content": [{"type": "image", "image": "img2.jpg"}, {"type": "text", "text": "What is this? Give a detailed answer."}]}],
]

processor.apply_chat_template(batch_messages, padding=False, tokenize=True, return_dict=True)
# raises ValueError: setting an array element with a sequence
```

## Expected Behavior

The processor should handle batched inputs without crashing when `padding=False`.

## Fix

A fix is implemented in PR #44535.

--- Comment by zucchini-nlp at 2026-03-09T13:13:18Z ---
Duplicate of https://github.com/huggingface/transformers/issues/44514 and it's not a Qwen problem only. I will fix it for all models, it's much easier than opening a separate PR for each processor

--- Comment by Anakintano at 2026-03-09T13:26:46Z ---
Thanks for the clarification! I'll follow the fix in #44514. I'm currently exploring the codebase and learning the contribution workflow.😀


Can not deploy SFT Qwen3.5-9B model

ISSUE #44541: Can not deploy SFT Qwen3.5-9B model
State: closed  |  Labels: bug
Author: zouYC2021  |  Created: 2026-03-09T09:39:19Z

### System Info

I SFT Qwen3.5-9B model with transformers==5.2.0

### Reproduction

When I try to deploy my model with vllm==0.17.0, it reports:

TypeError: Invalid type of HuggingFace config. Expected type: <class 'vllm.transformers_utils.configs.qwen3_5.Qwen3_5Config'>, but found type: <class 'transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5TextConfig'>

I wonder how can I fix it.

### Expected behavior

vllm==0.17.0 requires transformers<5, but transformers>5 is required to support Qwen3.5, kind of strange.

--- Comment by Rocketknight1 at 2026-03-09T13:33:13Z ---
I'm not sure we can fix bugs on older versions of Transformers - have you tried the latest version?

--- Comment by zouYC2021 at 2026-03-09T13:36:17Z ---
> I'm not sure we can fix bugs on older versions of Transformers - have you tried the latest version?

Yep, I got same error with transformers==5.3.0

--- Comment by Rocketknight1 at 2026-03-09T14:54:47Z ---
cc @zucchini-nlp maybe, or check with Harry?

--- Comment by hmellor at 2026-03-10T09:30:21Z ---
vLLM vendored everything for Qwen 3.5 because they couldn't wait for Transformers v5 support.

So there's a mismatch with the vendored config and the upstream config.

--- Comment by DorBernsohn at 2026-03-17T18:49:32Z ---
@hmellor Is there an existing solution or suggested workaround for this issue?

--- Comment by hmellor at 2026-03-18T08:25:11Z ---
I don't actually see this error (after fixing a couple of things which changed in the last day).

Can you try https://github.com/vllm-project/vllm/pull/37398? I am able to serve Qwen 3.5 with it.

--- Comment by arkerwu at 2026-03-20T02:08:32Z ---
This is an issue with the SFT tool or the quantization tool; you can share the model's config file, which should differ from the original model.

--- Comment by zouYC2021 at 2026-03-20T04:31:20Z ---
config.json in origin Qwen3.5-9B :
```
{
    "architectures": [
        "Qwen3_5ForConditionalGeneration"
    ],
    "image_token_id": 248056,
    "model_type": "qwen3_5",
    "text_config": {
        "attention_bias": false,
        "attention_dropout": 0.0,
        "attn_output_gate": true,
        "dtype": "bfloat16",
        "eos_token_id": 248044,
        "full_attention_interval": 4,
        "head_dim": 256,
        "hidden_act": "silu",
        "hidden_size": 4096,
        "initializer_range": 0.02,
        "intermediate_size": 12288,
        "layer_types": [
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention",
            "linear_attention",
            "linear_attention",
            "linear_attention",
            "full_attention"
        ],
        "linear_conv_kernel_dim": 4,
        "linear_key_head_dim": 128,
        "linear_num_key_heads": 16,
        "linear_num_value_heads": 32,
        "linear_value_head_dim": 128,
        "max_position_embeddings": 262144,
        "mlp_only_layers": [],
        "model_type": "qwen3_5_text",
        "mtp_num_hidden_layers": 1,
        "mtp_use_dedicated_embeddings": false,
        "num_attention_heads": 16,
        "num_hidden_layers": 32,
        "num_key_value_heads": 4,
        "rms_norm_eps": 1e-06,
        "use_cache": true,
        "vocab_size": 248320,
        "mamba_ssm_dtype": "float32",
        "rope_parameters": {
            "mrope_interleaved": true,
            "mrope_section": [
                11,
                11,
                10
            ],
            "rope_type": "default",
            "rope_theta": 10000000,
            "partial_rotary_factor": 0.25
        }
    },
    "tie_word_embeddings": false,
    "transformers_version": "4.57.0.dev0",
    "video_token_id": 248057,
    "vision_config": {
        "deepstack_visual_indexes": [],
        "depth": 27,
        "hidden_act": "gelu_pytorch_tanh",
        "hidden_size": 1152,
        "in_channels": 3,
        "initializer_range": 0.02,
        "intermediate_size": 4304,
        "model_type": "qwen3_5",
        "num_heads": 16,
        "num_position_embeddings": 2304,
        "out_hidden_size": 4096,
        "patch_size": 16,
        "spatial_merge_size": 2,
        "temporal_patch_size": 2
    },
    "vision_end_token_id": 248054,
    "vision_start_token_id": 248053
}
```

config.json after SFT:
```
{
  "architectures": [
    "Qwen3_5ForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "attn_output_gate": true,
  "bos_token_id": null,
  "dtype": "bfloat16",
  "eos_token_id": 248046,
  "full_attention_interval": 4,
  "head_dim": 256,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 12288,
  "layer_types": [
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention",
    "linear_attention",
    "linear_attention",
    "linear_attention",
    "full_attention"
  ],
  "linear_conv_kernel_dim": 4,
  "linear_key_head_dim": 128,
  "linear_num_key_heads": 16,
  "linear_num_value_heads": 32,
  "linear_value_head_dim": 128,
  "mamba_ssm_dtype": "float32",
  "max_position_embeddings": 262144,
  "mlp_only_layers": [],
  "model_type": "qwen3_5_text",
  "mtp_num_hidden_layers": 1,
  "mtp_use_dedicated_embeddings": false,
  "num_attention_heads": 16,
  "num_hidden_layers": 32,
  "num_key_value_heads": 4,
  "pad_token_id": 248044,
  "partial_rotary_factor": 0.25,
  "rms_norm_eps": 1e-06,
  "rope_parameters": {
    "mrope_interleaved": true,
    "mrope_section": [
      11,
      11,
      10
    ],
    "partial_rotary_factor": 0.25,
    "rope_theta": 10000000,
    "rope_type": "default"
  },
  "tie_word_embeddings": false,
  "transformers_version": "5.2.0",
  "use_cache": false,
  "vocab_size": 248320
}
```

--- Comment by arkerwu at 2026-03-20T05:49:11Z ---
> config.json in origin Qwen3.5-9B :
>
> ```
> {
>     "architectures": [
>         "Qwen3_5ForConditionalGeneration"
>     ],
>     "image_token_id": 248056,
>     "model_type": "qwen3_5",
>     "text_config": {
>         "attention_bias": false,
>         "attention_dropout": 0.0,
>         "attn_output_gate": true,
>         "dtype": "bfloat16",
>         "eos_token_id": 248044,
>         "full_attention_interval": 4,
>         "head_dim": 256,
>         "hidden_act": "silu",
>         "hidden_size": 4096,
>         "initializer_range": 0.02,
>         "intermediate_size": 12288,
>         "layer_types": [
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention",
>             "linear_attention",
>             "linear_attention",
>             "linear_attention",
>             "full_attention"
>         ],
>         "linear_conv_kernel_dim": 4,
>         "linear_key_head_dim": 128,
>         "linear_num_key_heads": 16,
>         "linear_num_value_heads": 32,
>         "linear_value_head_dim": 128,
>         "max_position_embeddings": 262144,
>         "mlp_only_layers": [],
>         "model_type": "qwen3_5_text",
>         "mtp_num_hidden_layers": 1,
>         "mtp_use_dedicated_embeddings": false,
>         "num_attention_heads": 16,
>         "num_hidden_layers": 32,
>         "num_key_value_heads": 4,
>         "rms_norm_eps": 1e-06,
>         "use_cache": true,
>         "vocab_size": 248320,
>         "mamba_ssm_dtype": "float32",
>         "rope_parameters": {
>             "mrope_interleaved": true,
>             "mrope_section": [
>                 11,
>                 11,
>                 10
>             ],
>             "rope_type": "default",
>             "rope_theta": 10000000,
>             "partial_rotary_factor": 0.25
>         }
>     },
>     "tie_word_embeddings": false,
>     "transformers_version": "4.57.0.dev0",
>     "video_token_id": 248057,
>     "vision_config": {
>         "deepstack_visual_indexes": [],
>         "depth": 27,
>         "hidden_act": "gelu_pytorch_tanh",
>         "hidden_size": 1152,
>         "in_channels": 3,
>         "initializer_range": 0.02,
>         "intermediate_size": 4304,
>         "model_type": "qwen3_5",
>         "num_heads": 16,
>         "num_position_embeddings": 2304,
>         "out_hidden_size": 4096,
>         "patch_size": 16,
>         "spatial_merge_size": 2,
>         "temporal_patch_size": 2
>     },
>     "vision_end_token_id": 248054,
>     "vision_start_token_id": 248053
> }
> ```
>
> config.json after SFT:
>
> ```
> {
>   "architectures": [
>     "Qwen3_5ForCausalLM"
>   ],
>   "attention_bias": false,
>   "attention_dropout": 0.0,
>   "attn_output_gate": true,
>   "bos_token_id": null,
>   "dtype": "bfloat16",
>   "eos_token_id": 248046,
>   "full_attention_interval": 4,
>   "head_dim": 256,
>   "hidden_act": "silu",
>   "hidden_size": 4096,
>   "initializer_range": 0.02,
>   "intermediate_size": 12288,
>   "layer_types": [
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention",
>     "linear_attention",
>     "linear_attention",
>     "linear_attention",
>     "full_attention"
>   ],
>   "linear_conv_kernel_dim": 4,
>   "linear_key_head_dim": 128,
>   "linear_num_key_heads": 16,
>   "linear_num_value_heads": 32,
>   "linear_value_head_dim": 128,
>   "mamba_ssm_dtype": "float32",
>   "max_position_embeddings": 262144,
>   "mlp_only_layers": [],
>   "model_type": "qwen3_5_text",
>   "mtp_num_hidden_layers": 1,
>   "mtp_use_dedicated_embeddings": false,
>   "num_attention_heads": 16,
>   "num_hidden_layers": 32,
>   "num_key_value_heads": 4,
>   "pad_token_id": 248044,
>   "partial_rotary_factor": 0.25,
>   "rms_norm_eps": 1e-06,
>   "rope_parameters": {
>     "mrope_interleaved": true,
>     "mrope_section": [
>       11,
>       11,
>       10
>     ],
>     "partial_rotary_factor": 0.25,
>     "rope_theta": 10000000,
>     "rope_type": "default"
>   },
>   "tie_word_embeddings": false,
>   "transformers_version": "5.2.0",
>   "use_cache": false,
>   "vocab_size": 248320
> }
> ```

This is an incorrect configuration file, and it's obvious that multimodal-related data is missing; you can go to the project of the SFT tool to submit an issue.

--- Comment by peleg-yair at 2026-03-20T16:52:13Z ---
Hi! @zouYC2021, thank you for this issue.
Could you please share which SFT tool you used and how you ran it?
Thanks!

--- Comment by zouYC2021 at 2026-03-21T07:51:29Z ---
I use trl==0.29.0 as SFT tool. My train_sft.py is as follows:
```
import os
import argparse
import torch
from pathlib import Path
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig
from transformers.trainer_utils import get_last_checkpoint

def parse_args():
    ap = argparse.ArgumentParser()
    ap.add_argument("--model_dir", required=True, type=str)
    ap.add_argument("--train_file", required=True, type=str)
    ap.add_argument("--val_file", required=True, type=str)
    ap.add_argument("--out_dir", required=True, type=str)
    ap.add_argument("--epochs", type=int, default=2)
    ap.add_argument("--lr", type=float, default=2e-5)
    ap.add_argument("--grad_accum", type=int, default=16)
    ap.add_argument("--max_len", type=int, default=4096)
    ap.add_argument("--seed", type=int, default=42)
    ap.add_argument("--deepspeed_config", type=str, default=None)
    return ap.parse_args()

def collect_jsonl_files(path_str: str):
    p = Path(path_str)
    files = []
    if p.is_dir():
        files = sorted(str(f) for f in p.rglob("*.jsonl"))
    elif p.is_file():
        if p.suffix == ".jsonl":
            files = [str(p)]
        else:
            raise ValueError(f"Expected a .jsonl file, got: {p}")
    else:
        files = sorted(str(f) for f in Path().glob(path_str))

    if not files:
        raise FileNotFoundError(f"No .jsonl files found under: {path_str}")
    return files

def main():
    args = parse_args()
    torch.manual_seed(args.seed)

    # ---- Tokenizer / Model ----
    tok = AutoTokenizer.from_pretrained(
        args.model_dir, use_fast=False, trust_remote_code=True
    )
    if tok.pad_token is None and tok.eos_token is not None:
        tok.pad_token = tok.eos_token

    model = AutoModelForCausalLM.from_pretrained(
        args.model_dir,
        torch_dtype=torch.bfloat16,
        trust_remote_code=True
    )
    model.gradient_checkpointing_enable()
    model.config.use_cache = False
    if tok.pad_token_id is not None:
        model.config.pad_token_id = tok.pad_token_id
    if tok.eos_token_id is not None:
        model.config.eos_token_id = tok.eos_token_id

    data_files = {
        "train": args.train_file,
        "validation": args.val_file,
    }
    raw_datasets = load_dataset("json", data_files=data_files)

    train_ds = raw_datasets["train"]
    eval_ds = raw_datasets["validation"]

    # ---- Qwen3.5 only ----
    train_ds = train_ds.add_column(
        "chat_template_kwargs",
        [{"enable_thinking": False} for _ in range(len(train_ds))]
    )
    eval_ds = eval_ds.add_column(
        "chat_template_kwargs",
        [{"enable_thinking": False} for _ in range(len(eval_ds))]
    )

    cfg = SFTConfig(
        output_dir=args.out_dir,
        deepspeed=args.deepspeed_config,     # ZeRO-3
        completion_only_loss=True,
        assistant_only_loss=False,

        max_length=args.max_len,
        packing=False,
        eval_packing=False,

        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=args.grad_accum,
        num_train_epochs=args.epochs,
        bf16=True,
        gradient_checkpointing=True,

        learning_rate=args.lr,
        lr_scheduler_type="cosine",
        warmup_ratio=0.03,
        weight_decay=0.1,
        max_grad_norm=1.0,

        save_strategy="steps", save_steps=100,
        eval_strategy="steps", eval_steps=100,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,

        logging_steps=100,
        dataset_num_proc=max(1, (os.cpu_count() or 4) // 2),
        seed=args.seed,

        report_to=["swanlab"],
        run_name="xxx",
    )

    trainer = SFTTrainer(
        model=model,
        args=cfg,
        train_dataset=train_ds,
        eval_dataset=eval_ds,
        processing_class=tok,
    )

    print("[INFO] Start SFT training ...")
    resume_from_checkpoint = False
    if not resume_from_checkpoint:
        trainer.train()
    else:
        print("loading ckpt...")
        trainer.train(resume_from_checkpoint=get_last_checkpoint(args.model_dir))
    trainer.save_model(args.out_dir)
    tok.save_pretrained(args.out_dir)

if __name__ == "__main__":
    main()
```

--- Comment by LuisVasquezBSC at 2026-03-22T09:33:56Z ---
@zouYC2021
Came across this bug today, after training a Qwen3.5 model with `trl`.

### Quick and dirty fix:

Replace new `config.json` produced after training with the one before that:

```bash
mv $TRL_CHECKPOINT_PATH/config.json $TRL_CHECKPOINT_PATH/config.json.old # backup
cp $ORIGINAL_CHECKPOINT_PATH/config.json $TRL_CHECKPOINT_PATH/config.json

```

Same for `tokenizer_config.json`.

### Wait for the bug to be really fixed in future `vllm` release

--- Comment by hmellor at 2026-03-23T12:16:38Z ---
You are doing SFT on a model loaded with `AutoModelForCausalLM`, which will naturally result in a text only model.

If you want to preserve the config structure since this is a multimodal model you should use `AutoModelForMultimodalLM`.

--- Comment by HaithemH at 2026-04-08T13:02:49Z ---
**Got same issue**
simplejson                               3.20.2
six                                      1.17.0
sniffio                                  1.3.1
sortedcontainers                         2.4.0
sse-starlette                            3.3.4
starlette                                0.52.1
supervisor                               4.3.0
sympy                                    1.14.0
tabulate                                 0.10.0
tensorboard                              2.20.0
tensorboard-data-server                  0.7.2
tiktoken                                 0.12.0
tokenizers                               0.22.2
tomlkit                                  0.13.3
torch                                    2.10.0+cu128
torch-c-dlpack-ext                       0.1.5
torchaudio                               2.10.0+cu128
torchvision                              0.25.0+cu128
tqdm                                     4.67.3
transformers                             5.3.0
transformers-stream-generator            0.0.5
triton                                   3.6.0
trl                                      0.29.1
typer                                    0.24.1
typer-slim                               0.24.0
typing-extensions                        4.15.0
typing-inspection                        0.4.2
tzdata                                   2026.1
urllib3                                  2.6.3
uvicorn                                  0.44.0
uvloop                                   0.22.1
vllm                                     0.19.0

--- Comment by zouYC2021 at 2026-04-09T17:48:36Z ---
> You are doing SFT on a model loaded with `AutoModelForCausalLM`, which will naturally result in a text only model.
>
> If you want to preserve the config structure since this is a multimodal model you should use `AutoModelForMultimodalLM`.

Thanks, you are right. I successfully trained Qwen3.5-9B and deployed SFT model with two modification:

1. Load Qwen3.5-9B with AutoModelForMultimodalLM in train.py. The config.json would remain unchanged before and after SFT.
```
model = AutoModelForMultimodalLM.from_pretrained(
    args.model_dir,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)
```

2. Deploy SFT model with text-only config
```
CUDA_VISIBLE_DEVICES=${GPU_ID} python -m vllm.entrypoints.openai.api_server \
        --model ${SFT_MODEL_PATH} \
        --tensor-parallel-size 1 \
        --port ${PORT} \
        --max_model_len 10000 \
        --enable_prefix_caching \
        --trust-remote-code
        --language-model-only # important
```

--- Comment by Roger-G at 2026-04-10T10:55:44Z ---
use my code to transfer original class name to new name, where vllm0.18 version supported now
`#!/usr/bin/env python3
"""
将 Qwen3.5 CausalLM (纯文本SFT) 模型转换为 ConditionalGeneration (多模态) 格式
只改权重前缀和 config.json，不改模型能力

转换内容:
  - 权重 key: "model.xxx" → "language_model.model.xxx"
  - 权重 key: "lm_head.xxx" → "language_model.lm_head.xxx"
  - config.json: model_type "qwen3_5_text" → "qwen3_5"
  - config.json: architectures "Qwen3_5ForCausalLM" → "Qwen3_5ForConditionalGeneration"

转换后可直接用 vLLM 0.18 原生部署（无需 patch）:
  vllm serve <model_path> --language-model-only --trust-remote-code
"""

import json
import shutil
import sys
from pathlib import Path

from safetensors.torch import load_file, save_file

def convert_model(src_dir, dst_dir=None):
    src_dir = Path(src_dir)
    if dst_dir is None:
        dst_dir = src_dir.parent / (src_dir.name + "-vl-format")
    dst_dir = Path(dst_dir)

    if dst_dir.exists():
        print(f"[ERROR] 目标目录已存在: {dst_dir}")
        sys.exit(1)

    dst_dir.mkdir(parents=True)
    print(f"[INFO] 源目录: {src_dir}")
    print(f"[INFO] 目标目录: {dst_dir}")

    # 1. 转换 safetensors 权重
    safetensor_files = list(src_dir.glob("*.safetensors"))
    if not safetensor_files:
        print("[ERROR] 未找到 .safetensors 文件")
        sys.exit(1)

    for sf in safetensor_files:
        if sf.name.endswith(".bak"):
            continue
        print(f"\n[CONVERT] {sf.name}")
        tensors = load_file(str(sf))

        new_tensors = {}
        for key, value in tensors.items():
            if key.startswith("model."):
                new_key = "language_model." + key
            elif key.startswith("lm_head."):
                new_key = "language_model." + key
            else:
                new_key = key
            new_tensors[new_key] = value
            if new_key != key:
                print(f"  {key} → {new_key}")

        save_file(new_tensors, str(dst_dir / sf.name))
        print(f"  saved: {dst_dir / sf.name}")

    # 2. 转换 config.json
    print(f"\n[CONVERT] config.json")
    config_path = src_dir / "config.json"
    with open(config_path) as f:
        config = json.load(f)

    config["model_type"] = "qwen3_5"
    config["architectures"] = ["Qwen3_5ForConditionalGeneration"]
    print(f"  model_type: qwen3_5_text → qwen3_5")
    print(f"  architectures: Qwen3_5ForCausalLM → Qwen3_5ForConditionalGeneration")

    with open(dst_dir / "config.json", "w") as f:
        json.dump(config, f, indent=2, ensure_ascii=False)

    # 3. 复制其他文件
    for f in src_dir.iterdir():
        if f.name.endswith(".safetensors") or f.name == "config.json" or f.name.endswith(".bak"):
            continue
        print(f"[COPY] {f.name}")
        if f.is_file():
            shutil.copy2(f, dst_dir / f.name)

    print(f"\n[DONE] 转换完成!")
    print(f"  目标目录: {dst_dir}")
    print(f"\n  部署命令:")
    print(f"  vllm serve {dst_dir} --language-model-only --trust-remote-code")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print(f"用法: python3 {sys.argv[0]} <源模型目录> [目标目录]")
        print(f"示例: python3 {sys.argv[0]} ./qwen35-sft-merged ./qwen35-sft-merged-vl")
        sys.exit(1)

    src = sys.argv[1]
    dst = sys.argv[2] if len(sys.argv) > 2 else None
    convert_model(src, dst)
`

--- Comment by Oxi84 at 2026-04-19T01:15:17Z ---
I tried the code, thanks, itt gets past the error but now  get a new one: OSError: Can't load image processor for '/home/user/Desktop/qwen25base_4bit_vllm'. If you were trying to load it from

--- Comment by amanichopra at 2026-04-27T20:30:49Z ---
@zouYC2021 @hmellor does this mean that even if I'm training on text only, we need to load the model with `AutoModelForMultimodalLM`? If this is the case, shouldn't the library enforce that multimodal models need to be loaded with `AutoModelForMultimodalLM` to prevent these types of issues?

--- Comment by hmellor at 2026-05-01T12:21:37Z ---
> does this mean that even if I'm training on text only, we need to load the model with AutoModelForMultimodalLM?

You could use `AutoModelForCausalLM` but you would then need to transplant the text model into a `Qwen3_5ForConditionalGeneration` before saving.

> If this is the case, shouldn't the library enforce that multimodal models need to be loaded with AutoModelForMultimodalLM to prevent these types of issues?

It is perfectly valid to load a multimodal model using `AutoModelForCausalLM` if you don't intend to use the multi-model part.

--- Comment by amanichopra at 2026-05-07T14:05:43Z ---
> > does this mean that even if I'm training on text only, we need to load the model with AutoModelForMultimodalLM?
>
> You could use `AutoModelForCausalLM` but you would then need to transplant the text model into a `Qwen3_5ForConditionalGeneration` before saving.
>
> > If this is the case, shouldn't the library enforce that multimodal models need to be loaded with AutoModelForMultimodalLM to prevent these types of issues?
>
> It is perfectly valid to load a multimodal model using `AutoModelForCausalLM` if you don't intend to use the multi-model part.

@hmellor Thanks for the reply! Just to be clear, by "transplant", you mean doing what @LuisVasquezBSC suggested, right?

```
Replace new config.json produced after training with the one before that:
```

--- Comment by hmellor at 2026-05-07T17:19:45Z ---
I meant copying the weights from the `ForCausalLM` model into the `text_model` of the `ForMultimodalLM`

--- Comment by amanichopra at 2026-05-13T15:16:40Z ---
@hmellor So do you mean that if I use `AutoModelForCausalLM`, after training, I must edit the `.safetensor` files so that they contain the vision weights from the base model (obtained from the base `.safetensor` files on [HF](https://huggingface.co/Qwen/Qwen3.5-27B/blob/main/model.safetensors-00011-of-00011.safetensors))? After this transplant, I can deploy to vLLM as normally. This seems a bit tedious.

As an alternative, I went with this approach `Replace new config.json produced after training with the one before that:
` and it worked quite well. I just needed to replace the `config.json` after training with the base model's `config.json` (https://huggingface.co/Qwen/Qwen3.5-27B/blob/main/config.json).

I imagine both these approaches are equivalent, but please let me know if not. Thanks!

--- Comment by hmellor at 2026-05-20T08:44:48Z ---
```py
# You trained this:
llm: Qwen3_5ForCausalLM
# But vLLM wants to load this:
vlm: Qwen3_5ForConditionalGeneration
# So you transplant it like this:
vlm.model.language_model = llm.model
vlm.save_pretrained(...)


ISSUE #44537: About check_model_inputs after version 5.2.0, Where is it!?
State: closed  |  Labels:
Author: QYQTexas  |  Created: 2026-03-09T08:56:37Z

Hei bros, I found the func transformers.utils.generic.check_model_inputs in version <=5.1.0 turn to transformers.utils.generic.merge_with_config_defaults in version >=5.2.0, but I did not found any notes about it, or I am a blind person......TAT

--- Comment by Rocketknight1 at 2026-03-09T15:35:59Z ---
Hi @QYQTexas, did this cause a bug for you, or an issue with model remote code?

--- Comment by QYQTexas at 2026-03-10T02:04:28Z ---
> Hi [@QYQTexas](https://github.com/QYQTexas), did this cause a bug for you, or an issue with model remote code?
Thanks for your reply.

I have resolved the problem.

Maybe the problem was caused by Qwen3-vl-embedding but not transformers......

I ran the example of Qwen3-vl-embedding.

Then I got the error "check_model_inputs is not found", so I trace codes and found that the func check_model_inputs has been rename to merge_with_config_defaults.

There is without any problem after I changed the import-func to merge_with_config_defaults.

But there is a funny thing that I found check_model_inputs was imported by  qwen3_vl_embedding.py, but it did not work.

emmm......


ISSUE #44534: Transformers v5 fills non-persistent buffers with junk
State: closed  |  Labels: bug
Author: umarbutler  |  Created: 2026-03-09T03:43:59Z

### System Info

- `transformers` version: 5.3.0
- Platform: Linux-6.17.0-14-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.7.0
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: N/A
- Using GPU in script?: N/A
- GPU type: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition

### Who can help?

@ArthurZucker @CyrilVallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am recreating issue #43644 because I do not believe it was fairly closed.

Right now, in version 5 of Transformers, if you define a custom model that has a non-persistent PyTorch buffer, when you go to load that model again, all of the weights in your non-persistent buffer will turn into junk.

While I understand that the way model loading now works is such that this occurs, this behavior is nevertheless:
1. a major breaking change to any model leveraging non-persistent buffers.
2. extremely unituitive, diverging from reasonably expectable behavior.
3. dangerous in that there is no warning that this occurs, causing users not to realize why their models suddenly aren't working (I ended up spending multiple days trying to debug this before realizing the issue was with version 5 of Transformers).

To illustrate how unituitive and damaging this behavior is, see the below example:
```python
import torch

from transformers import PreTrainedModel, PretrainedConfig

class MyModelConfig(PretrainedConfig):
    model_type = "my_model"

    def __init__(
        self,
        temperature: float = 1.0,
        **kwargs,
    ) -> None:
        super().__init__(**kwargs)
        self.temperature = temperature

class MyModel(PreTrainedModel):
    config_class = MyModelConfig

    def __init__(self, config: MyModelConfig) -> None:
        super().__init__(config)

        self.classifier = torch.nn.Linear(1024, 1)

        self.temperature: torch.Tensor
        self.register_buffer("temperature", torch.tensor(config.temperature), persistent=False)

        self.post_init()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        logits = self.classifier(x)

        return logits / self.temperature

model = MyModel(MyModelConfig(temperature=0.5))

print(f"Original model temperature: {model.temperature}") # It will print 0.5 as expected

model.save_pretrained("my_model")
model = MyModel.from_pretrained("my_model")

print(f"Loaded model temperature: {model.temperature}") # It will print a random junk value like 1.983314177778084e-05 instead of the original 0.5 value
```

It is very common to define static variables in a model that are stored in a non-persistent buffer. When you go to load your model again, those variables will be overwritten entirely without you knowing that happened. Now, all of the logits in the example model would end up being scaled incorrectly.

I would like to strongly reiterate my request that this issue be resolved.

### Expected behavior

Non-persistent PyTorch buffers do not get filled in with random junk values when they are reloaded.

--- Comment by ArthurZucker at 2026-03-09T08:04:01Z ---
cc @Cyrilvallez probably a case for `__init_weight` to be module specific? + AST enforcing the init sets it to `config.temperature`

--- Comment by Cyrilvallez at 2026-03-09T12:56:33Z ---
Hey @umarbutler! As already explained in https://github.com/huggingface/transformers/issues/43644, we unfortunately cannot enforce BC for custom models... If you are not convincing by the benefits of the new approach, please have a look at the numbers I carefully benchmarked here https://github.com/huggingface/transformers/pull/42941.
As @ArthurZucker said, we will however be refactoring the init mechanism so that it will be at the module-level, which will be much more natural for everyone, including custom modules/models.


ISSUE #44530: [CB] PagedAttentionCache crashes with "Invalid group type: linear_attention" on Qwen3.5 models
State: closed  |  Labels: bug, Code agent slop
Author: mxchampagne  |  Created: 2026-03-08T18:49:55Z

### System Info

- `transformers` version: 5.2.0
- Platform: Windows-11-10.0.26200-SP0
- Python version: 3.13.3
- Huggingface_hub version: 1.5.0
- Safetensors version: 0.5.3
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 4060 Ti

### Who can help?

@remi-or @ArthurZucker @McPatate

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Reproduces with Qwen/Qwen3.5-0.8B using the official continuous batching example script.
Run with:
   python examples/pytorch/continuous_batching_simple.py --samples 5

The error occurs inside PagedAttentionCache.__init__() when building the KV cache group map.
Qwen3.5 uses a hybrid architecture with linear attention layers alongside standard attention layers.
The cache.py group type handler does not recognize "linear_attention" as a valid group type, causing an immediate crash before any generation begins.

## Full traceback:
```
C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\torch\cuda\__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Loading weights: 100%|█████████████████████████████████████████████| 320/320 [00:00<00:00, 553.59it/s, Materializing param=model.norm.weight]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Error in generation loop: Invalid group type: linear_attention
Traceback (most recent call last):
  File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\continuous_api.py", line 767, in _run_generation_loop
    paged_attention_cache = PagedAttentionCache(
        self.model.config,
    ...<4 lines>...
        allow_block_sharing=self._allow_block_sharing,
    )
  File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\cache.py", line 240, in __init__
    raise ValueError(f"Invalid group type: {group_type}")
ValueError: Invalid group type: linear_attention
Generation thread terminated unexpectedly.
Solving 5 requests:   0%|                                                                                | 0/5 [00:01<?, ?request/s]Returning results of generate_batch despite unexpected termination.
Solving 5 requests:   0%|                                                                                | 0/5 [00:01<?, ?request/s]

Batch processor was not initialized.
Request req_0 not found in results.
Request req_1 not found in results.
Request req_2 not found in results.
Request req_3 not found in results.
Request req_4 not found in results.
--- Running CB Generation Example ---
Error in generation loop: Invalid group type: linear_attention
Traceback (most recent call last):
  File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\continuous_api.py", line 767, in _run_generation_loop
    paged_attention_cache = PagedAttentionCache(
        self.model.config,
    ...<4 lines>...
        allow_block_sharing=self._allow_block_sharing,
    )
  File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\cache.py", line 240, in __init__
    raise ValueError(f"Invalid group type: {group_type}")
ValueError: Invalid group type: linear_attention
Generation thread terminated unexpectedly.
Solving 5 requests:   0%|                                                                                | 0/5 [00:01<?, ?request/s]Returning results of generate_batch despite unexpected termination.
Solving 5 requests:   0%|                                                                                | 0/5 [00:01<?, ?request/s]

Batch processor was not initialized.
Request req_0 not found in results.
Request req_1 not found in results.
Request req_2 not found in results.
Request req_3 not found in results.
Request req_4 not found in results.
Done with batch generation.
--------------------
--- Finished CB Generation Example ---

CB generation took: 1.01 seconds for 0 tokens. 0.00tok/s
```

### Expected behavior

Running `examples/pytorch/continuous_batching_simple.py --samples 5` (with `Qwen/Qwen3.5-0.8B`) should complete successfully and return generated text for all 5 requests, consistent with how it behaves for `Qwen/Qwen3-4B-Instruct-2507` (the model used in the official docs example).

`PagedAttentionCache` should handle `linear_attention` as a known group type. The crash prevents `generate_batch` from being usable with any hybrid linear-attention model.

--- Comment by remi-or at 2026-03-09T12:29:12Z ---
Hey @mxchampagne ! This is a feature missing from continuous batching, along other forms of attention. Feel free to open a PR adding support of `linear_attention` as is done in `flash_paged.py` if you want! I will be happy to review it.

--- Comment by mxchampagne at 2026-03-09T19:06:53Z ---
After digging into the architecture, I wanted to share a full breakdown of the technical challenge before opening a PR, in case the maintainers have context I'm missing.

---

**What causes the crash**

`PagedAttentionCache.__init__()` iterates over the layer groups produced by `group_layers_by_attn_type()` and builds a `CacheAllocator` per group. Qwen3.5 uses a hybrid architecture with `"linear_attention"` groups (handled by `Qwen3_5GatedDeltaNet`), and the `elif` chain in `cache.py` has no branch for that type, raising `ValueError: Invalid group type: linear_attention`. This is the immediate crash.

---

**Why a simple `pass` is insufficient**

`Qwen3_5GatedDeltaNet.forward()` does not go through `ALL_ATTENTION_FUNCTIONS` and never calls any of the three paged attention forwards (`eager_paged.py`, `sdpa_paged.py`, `flash_paged.py`). Instead, it accesses the cache object directly via three attributes:

```python
cache_params.has_previous_state   # property
cache_params.conv_states[layer_idx]
cache_params.recurrent_states[layer_idx]
```

So `PagedAttentionCache` needs to expose those attributes for linear attention layers to function at all. The proposed fix is:

1. Add a `LinearAttentionCacheAllocator` to `cache_allocator.py` that satisfies the `CacheAllocator` interface but consumes zero blocks from `BlockManager` (linear attention state is fixed-size, not paged).
2. In `cache.py`, add the missing `elif group_type == "linear_attention"` branch instantiating the new allocator, and expose `conv_states`, `recurrent_states`, and `has_previous_state` on `PagedAttentionCache` directly, mirroring the `Qwen3_5DynamicCache` interface that `GatedDeltaNet` expects.

---

**The deeper multi-request correctness problem**

Even with the above fix, there is a fundamental incompatibility between CB's packed sequence format and `GatedDeltaNet`'s decode path. The decode branch is gated by:

```python
use_precomputed_states = cache_params is not None and cache_params.has_previous_state and seq_len == 1
```

`seq_len` here is `hidden_states.shape[1]`, which under CB is the **total packed query token count** across all requests in the batch — never 1 except in the degenerate single-request case. This means:

- For any batch with more than one request in decode phase, `use_precomputed_states` is always `False`
- `chunk_gated_delta_rule` runs with `initial_state=None` on every step, discarding accumulated recurrent state
- Each decode step is treated as a fresh prefill for linear attention layers

For a single-request workload, correctness is achievable. For multi-request batches, the recurrent state cannot be correctly threaded through without modifying `GatedDeltaNet.forward()` to process each request's tokens independently using per-request states fetched from the cache.

---

**Proposed scope split**

- **Phase 1 PR** (this issue): Add `LinearAttentionCacheAllocator`, fix the `cache.py` branch, expose the required state attributes. Unblocks CB for Qwen3.5 with correct single-request behavior. Multi-request decode works but without recurrent state continuity — worth a warning in the docs.
- **Phase 2 PR**: Modify `GatedDeltaNet.forward()` to accept per-request states from the cache and process packed sequences request-by-request, enabling correct multi-request decode.

Happy to open the Phase 1 PR if this scope makes sense. Wanted to flag the Phase 2 issue upfront so it doesn't get missed.

--- Comment by remi-or at 2026-03-10T22:42:08Z ---
Closing this issue because it looks like code agent slop. If you want to open a human-looking PR, not entirely AI generated, feel free to do so! Thanks.

--- Comment by mxchampagne at 2026-03-11T03:26:46Z ---
Hey @remi-or, sorry if my previous message looked AI generated. I used it to help organize my notes while investigating the issue, but the debugging and proposed fix are mine.

I believe I am close to a fix locally, but it still needs more testing. The change involves adding support for the `linear_attention` group type in cache.py, and a small adjustment in the Qwen3.5 modular file so the model can use the paged attention cache properly with continuous batching.

I will clean it up and open a PR shortly so it is easier to review.

Thank you for your time.

--- Comment by mxchampagne at 2026-03-11T16:04:40Z ---
Hey @remi-or, after digging further into this I have to be honest, it's more complex than I initially thought. The Phase 1 fix is straightforward, but properly supporting continuous batching for GatedDeltaNet requires adapting the forward path to handle per-sequence recurrent state updates under the packed sequence format used by CB.

I don't want to open a PR I'm not 100% confident in, so I'll leave this one for someone with deeper familiarity with the CB internals. Thanks for the context and for taking the time to respond.

--- Comment by remi-or at 2026-03-13T11:26:25Z ---
No problem, thank you for bringing this to my attention. This is not a top priority atm, but I will make sure to ping you once that's done.

--- Comment by codybarrett108-pixel at 2026-03-13T11:58:36Z ---
Ping me too please

On Fri, Mar 13, 2026, 11:26 AM Rémi Ouazan ***@***.***> wrote:

> *remi-or* left a comment (huggingface/transformers#44530)
> <https://github.com/huggingface/transformers/issues/44530#issuecomment-4054423880>
>
> No problem, thank you for bringing this to my attention. This is not a top
> priority atm, but I will make sure to ping you once that's done.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/44530#issuecomment-4054423880>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HX4LWSYRFNF4PRD55C3DL4QPV75AVCNFSM6AAAAACWLDG5ICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANJUGQZDGOBYGA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


ISSUE #44521: apply_chat_template returns all-zero assistant_masks for multimodal inputs
State: open  |  Labels: WIP, bug
Author: renhouxing  |  Created: 2026-03-08T05:03:12Z

### System Info

transformers==5.3.0

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoProcessor

messages = [
    dict(
        role="user",
        content=[
            dict(type="image", image="test.jpg"),
            dict(type="text", text="Describe the image above."),
        ],
    ),
    dict(
        role="assistant",
        content=[
            dict(type="text", text="The image above shows a cat sitting on a table."),
        ],
    ),
]

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-0.8B")
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    return_assistant_tokens_mask=True,
)

print(inputs["assistant_masks"])
```

### Actual behavior

assistant_masks is all zeros.

### Expected behavior

The tokens corresponding to the assistant response should be marked with 1 in assistant_masks.

### Suspected cause
I think the issue comes from a mismatch between generation_indices and the final tokenized prompt in multimodal cases.

In AutoProcessor.apply_chat_template, the assistant mask is computed roughly like this:
```python
prompt, generation_indices = render_jinja_template(
    conversations=conversations,
    chat_template=chat_template,
    **template_kwargs,
    **special_tokens_map,
)

...

out = self(
    text=prompt,
    images=batch_images if images_exist else None,
    videos=batch_videos if videos_exist else None,
    audio=batch_audios if batch_audios else None,
    **kwargs,
)

...

offset_mapping = out.pop("offset_mapping")
input_ids = out["input_ids"]

for assistant_start_char, assistant_end_char in generation_indices[i]:
    start_pos = bisect.bisect_left(offset_starts, assistant_start_char)
    end_pos = bisect.bisect_left(offset_starts, assistant_end_char)
```
My understanding is:
- generation_indices is computed from the rendered text prompt returned by render_jinja_template
- but in multimodal processing, extra placeholder tokens such as <|image_pad|> are inserted later by the processor/tokenizer path
- therefore offset_mapping corresponds to the expanded multimodal text, while generation_indices still refers to the pre-expanded text
- this makes the character spans misaligned, so the assistant span lookup fails and assistant_masks ends up all zeros

--- Comment by umbilnm at 2026-03-08T19:38:32Z ---
Hi! I've investigated this and confirmed the root cause - generation_indices are computed from the original (unexpanded) prompt, but offset_mapping comes from the expanded text after placeholder token expansion (e.g. <|image_pad|> → N copies), so bisect_left misses the assistant span entirely.

Will open a PR shortly.

--- Comment by renhouxing at 2026-03-09T05:39:16Z ---
Thanks for digging into this.

Looking forward to the PR!

--- Comment by Rocketknight1 at 2026-03-09T14:01:59Z ---
@zucchini-nlp this is a chat template / multimodal processor crossover issue, so let me know if you want me to take this one. The assistant mask generator can be a bit weird!

--- Comment by zucchini-nlp at 2026-03-10T12:47:04Z ---
I am working on it, seems like adding image placeholder tokens messed up indexing, so I'll take advantage of another PR I had on mind and fix this along the way

--- Comment by Liao-YiHsiu at 2026-03-18T12:55:59Z ---
I'm also hitting this issue. Is it possible to merge the solution PRs?

--- Comment by Liao-YiHsiu at 2026-03-24T14:33:31Z ---
Hi all, any plans regarding this bug? How can we move forward from here?

--- Comment by zucchini-nlp at 2026-03-24T14:46:59Z ---
@Liao-YiHsiu it might take a bit more time since the fix would be part of a bigger PR. There is a workaround by @umbilnm  which seems to work, so you can adopt it in your project imo.

I don't really wish to merge a "workaround code" when there is a bigger refactor planned tbh

--- Comment by github-actions[bot] at 2026-04-18T08:11:39Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44514: `Qwen2_5_VLProcessor.apply_chat_template` crashes on batched input when `padding=False`
State: closed  |  Labels: bug
Author: qgallouedec  |  Created: 2026-03-07T17:03:20Z

### System Info

transformers 5.3.0

### Who can help?

@zucchini-nlp

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`Qwen2_5_VLProcessor.apply_chat_template` raises a `ValueError` when called with a batch of conversations (different prompt lengths) and `padding=False` (the default). The crash originates from `processing_qwen2_5_vl.py:148` which does `np.array(text_inputs["input_ids"])`, this fails when the tokenized sequences have different lengths because NumPy cannot create a homogeneous array from ragged lists.

Single-message inputs and batched inputs with `padding=True` both work fine.

```python
from transformers import AutoProcessor
from PIL import Image

processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

img = Image.new("RGB", (64, 64), color="red")

messages = [
    [{"role": "user", "content": [{"type": "image", "image": img}, {"type": "text", "text": "Describe this"}]}],
    [{"role": "user", "content": [{"type": "image", "image": img}, {"type": "text", "text": "What is this?"}]}],
]

# Crashes
result = processor.apply_chat_template(messages, tokenize=True, return_dict=True)
```

## Traceback

```
Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "transformers/processing_utils.py", line 1829, in apply_chat_template
    out = self(text=prompt, **kwargs)
  File "transformers/models/qwen2_5_vl/processing_qwen2_5_vl.py", line 148, in __call__
    array_ids = np.array(text_inputs["input_ids"])
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape
after 1 dimensions. The detected shape was (2,) + inhomogeneous part.
```

### Expected behavior

to work with `padding=False`

--- Comment by KartikPawade at 2026-03-07T19:27:18Z ---
**Root cause:** The crash happens in the `mm_token_type_ids` block where `np.array(text_inputs["input_ids"])`
is called on the full batch at once. With `padding=False` (the default), sequences have different lengths
producing a ragged list-of-lists that NumPy cannot convert to a 2D array.

**Fix:** Process each sequence individually instead of the whole batch at once, avoiding the ragged array issue entirely.


ISSUE #44512: Docs still mention transformers run command which was removed in v5
State: closed  |  Labels: bug
Author: math-hiyoko  |  Created: 2026-03-07T16:10:16Z

### System Info

Transformers version: 5.x

The issue is related to the documentation rather than a specific runtime environment.

### Who can help?

@stevhliu

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This is a documentation inconsistency in Transformers v5.
`transformers run` command was removed in v5.
Despite this, the official documentation still contains multiple references to this command.

- https://huggingface.co/docs/transformers/main/en/model_doc/apertus?usage=transformers+CLI#overview
- https://huggingface.co/docs/transformers/main/en/model_doc/barthez?usage=transformers+CLI#barthez
- https://huggingface.co/docs/transformers/main/en/model_doc/xlm?usage=transformers+CLI#xlm
- https://huggingface.co/docs/transformers/main/en/model_doc/bertweet?usage=transformers+CLI#bertweet
- https://huggingface.co/docs/transformers/main/en/model_doc/eurobert?usage=transformers+CLI#overview
- https://huggingface.co/docs/transformers/main/en/model_doc/olmo3?usage=transformers+CLI#olmo3
- https://huggingface.co/docs/transformers/main/en/model_doc/nanochat?usage=transformers+CLI#nanochat
- https://huggingface.co/docs/transformers/main/en/model_doc/bert?usage=transformers+CLI#bert
- https://huggingface.co/docs/transformers/main/en/model_doc/t5?usage=transformers+CLI#t5
- https://huggingface.co/docs/transformers/main/en/model_doc/vaultgemma?usage=transformers+CLI#overview
- https://huggingface.co/docs/transformers/main/en/model_doc/roberta?usage=transformers+CLI#roberta
- https://huggingface.co/docs/transformers/main/en/model_doc/olmo?usage=transformers+CLI#olmo
- https://huggingface.co/docs/transformers/main/en/model_doc/bart?usage=transformers+CLI#bart
- https://huggingface.co/docs/transformers/main/en/model_doc/roformer?usage=transformers+CLI#roformer
- https://huggingface.co/docs/transformers/main/en/model_doc/canine?usage=transformers+CLI#canine
- https://huggingface.co/docs/transformers/main/en/model_doc/opt?usage=transformers+CLI#opt
- https://huggingface.co/docs/transformers/main/en/model_doc/byt5?usage=transformers#byt5
- https://huggingface.co/docs/transformers/main/en/model_doc/roc_bert?usage=transformers+CLI#rocbert
- https://huggingface.co/docs/transformers/main/en/model_doc/deberta?usage=transformers+CLI#deberta
- https://huggingface.co/docs/transformers/main/en/model_doc/phi?usage=transformers+CLI#phi
- https://huggingface.co/docs/transformers/main/en/model_doc/gpt_neo?usage=transformers+CLI#gpt-neo
- https://huggingface.co/docs/transformers/main/en/model_doc/mt5?usage=transformers+CLI#mt5
- https://huggingface.co/docs/transformers/main/en/model_doc/olmo2?usage=transformers+CLI#olmo2
- https://huggingface.co/docs/transformers/main/en/model_doc/bartpho?usage=transformers+CLI#bartpho
- https://huggingface.co/docs/transformers/main/en/model_doc/t5gemma?usage=transformers+CLI#t5gemma
- https://huggingface.co/docs/transformers/main/en/model_doc/mobilebert?usage=transformers+CLI#mobilebert
- https://huggingface.co/docs/transformers/main/en/model_doc/mamba?usage=transformers+CLI#mamba
- https://huggingface.co/docs/transformers/main/en/model_doc/gpt2?usage=transformers+CLI#gpt-2
- https://huggingface.co/docs/transformers/main/en/model_doc/gemma2?usage=transformers+CLI#gemma2
- https://huggingface.co/docs/transformers/main/en/model_doc/distilbert?usage=transformers+CLI#distilbert
- https://huggingface.co/docs/transformers/main/en/model_doc/granite?usage=transformers+CLI#granite
- https://huggingface.co/docs/transformers/main/en/model_doc/llama?usage=transformers+CLI#llama
- https://huggingface.co/docs/transformers/main/en/model_doc/flex_olmo?usage=transformers+CLI#flexolmo
- https://huggingface.co/docs/transformers/main/en/model_doc/big_bird?usage=transformers+CLI#bigbird
- https://huggingface.co/docs/transformers/main/ko/model_doc/gpt2?usage=transformers+CLI#gpt-2
- https://huggingface.co/docs/transformers/main/en/model_doc/bamba?usage=transformers+CLI#bamba
- https://huggingface.co/docs/transformers/main/en/model_doc/nllb?usage=transformers+CLI#overview
- https://huggingface.co/docs/transformers/main/en/model_doc/electra?usage=transformers+CLI#electra
- https://huggingface.co/docs/transformers/main/en/model_doc/gemma?usage=transformers+CLI#gemma
- https://huggingface.co/docs/transformers/main/en/model_doc/camembert?usage=transformers+CLI#camembert
- https://huggingface.co/docs/transformers/main/en/model_doc/deberta-v2?usage=transformers+CLI#deberta-v2
- https://huggingface.co/docs/transformers/main/en/model_doc/pegasus?usage=transformers+CLI#pegasus
- https://huggingface.co/docs/transformers/main/en/model_doc/mamba2?Usage=transformers+CLI#mamba-2
- https://huggingface.co/docs/transformers/main/en/model_doc/jamba?usage=transformers+CLI#jamba
- https://huggingface.co/docs/transformers/main/ko/model_doc/jamba?usage=transformers+CLI#jamba
- https://huggingface.co/docs/transformers/main/en/model_doc/modernbert?usage=transformers+CLI#modernbert
- https://huggingface.co/docs/transformers/main/en/model_doc/gemma3n?usage=transformers+CLI#overview
- https://huggingface.co/docs/transformers/main/en/model_doc/led?usage=transformers#led
- https://huggingface.co/docs/transformers/main/en/model_doc/albert?usage=transformers+CLI#albert
- https://huggingface.co/docs/transformers/main/ko/model_doc/gemma3n?usage=transformers+CLI#overview
- https://huggingface.co/docs/transformers/main/en/model_doc/xlm-roberta-xl?usage=transformers+CLI#xlm-roberta-xl
- https://huggingface.co/docs/transformers/main/en/model_doc/longformer?usage=transformers+CLI#longformer
- https://huggingface.co/docs/transformers/main/en/model_doc/gemma3?usage=transformers+CLI#gemma-3
- https://huggingface.co/docs/transformers/main/ko/model_doc/gemma3?usage=transformers+CLI#gemma3
- https://huggingface.co/docs/transformers/main/en/model_doc/code_llama?usage=transformers+CLI#codellama
- https://huggingface.co/docs/transformers/main/en/model_doc/switch_transformers?usage=transformers+CLI#switch-transformers
- https://huggingface.co/docs/transformers/main/en/model_doc/bert-generation?usage=transformers+CLI#bertgeneration
- https://huggingface.co/docs/transformers/main/en/model_doc/pegasus_x?usage=transformers#pegasus-x
- https://huggingface.co/docs/transformers/main/ko/model_doc/albert?usage=transformers+CLI#albert
- https://huggingface.co/docs/transformers/main/ko/model_doc/code_llama?usage=transformers+CLI#codellama
- https://huggingface.co/docs/transformers/main/en/model_doc/gpt_neox_japanese?usage=transformers+CLI#gpt-neox-japanese
- https://huggingface.co/docs/transformers/main/en/model_doc/xlm-roberta?usage=transformers+CLI#xlm-roberta
- https://huggingface.co/docs/transformers/main/en/model_doc/modernbert-decoder?usage=transformers+CLI#modernbert-decoder
- https://huggingface.co/docs/transformers/main/en/model_doc/encoder-decoder?usage=transformers+CLI#encoder-decoder-models
- https://huggingface.co/docs/transformers/main/en/model_doc/bigbird_pegasus?usage=transformers#bigbirdpegasus
- https://huggingface.co/docs/transformers/main/en/model_doc/openai-gpt?usage=transformers+CLI#gpt
- https://huggingface.co/docs/transformers/main/en/model_doc/ernie?usage=transformers+CLI#ernie
- https://huggingface.co/docs/transformers/main/en/model_doc/biogpt?usage=transformers+CLI#biogpt
- https://huggingface.co/docs/transformers/main/zh/model_doc/bert?usage=transformers#bert

### Expected behavior

References to `transformers run` command should be removed.