ISSUE #45507: GraniteMoEHybrid Model Calls Invalid Method
State: closed  |  Labels: bug
Author: rnowling  |  Created: 2026-04-18T17:07:36Z

### System Info

Linux: Ubuntu 24.04.4 LTS / 6.8.0-107-generic-64k / aarch64
Python: 3.12.12
Transformers: 5.5.4
Cuda: 12.9

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Run the following script as follows:

```bash
$ CUDA_VISIBLE_DEVICES=0 python generate_text.py --model-name ibm-granite/granite-4.0-350m-base --n-samples 256 --output-fl granite-350m-base.feather
```

```python
import argparse

from datasets import Dataset
import numpy as np
import pandas as pd
from transformers import AutoTokenizer, DataCollatorForLanguageModeling
from transformers import AutoModelForCausalLM, TrainingArguments, Trainer

BATCH_SIZE = 32
MAX_LENGTH = 1024
PREFIX = "Your horoscope is: "

def parse_args():
    parser = argparse.ArgumentParser()

    parser.add_argument("--output-fl", type=str, required=True)
    parser.add_argument("--model-name", type=str, required=True)
    parser.add_argument("--n-samples", type=int, required=True)

    return parser.parse_args()

if __name__ == "__main__":
    args = parse_args()

    tokenizer = AutoTokenizer.from_pretrained(args.model_name, padding_side="left")
    model = AutoModelForCausalLM.from_pretrained(args.model_name)

    print(model.generation_config)

    # see tips here: https://huggingface.co/docs/transformers/en/model_doc/llama3
    if tokenizer.pad_token is None:
        print("Needed to add padding token")
        tokenizer.add_special_tokens({"pad_token":"<pad>"})
        model.resize_token_embeddings(len(tokenizer))
        model.config.pad_token_id = tokenizer.pad_token_id

    model.eval()

    # why is this necessary?
    model.to("cuda")

    batch = [PREFIX] * BATCH_SIZE
    model_inputs = tokenizer(batch, return_tensors="pt").to(model.device)

    generated_samples = []
    n_batches = args.n_samples // BATCH_SIZE + 1
    for _ in range(n_batches):
        generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=MAX_LENGTH)
        batch_samples = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

        generated_samples.extend(batch_samples)

    if len(generated_samples) > args.n_samples:
        generated_samples = generated_samples[:args.n_samples]

    df = pd.DataFrame({"generated_sample" : generated_samples,
                       "model" : [args.model_name] * args.n_samples })

    df.head()

    df.to_feather(args.output_fl, compression="zstd")
```

It produces the following stack trace:

```
Traceback (most recent call last):
  File "/home/rnowling/Projects/robust-llm-data-generators/sequence_token_probabilities/generate_text.py", line 48, in <module>
    generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=MAX_LENGTH)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2543, in generate
    result = decoding_method(
             ^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2736, in _sample
    outputs = self._prefill(
              ^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 3768, in _prefill
    return self(**model_inputs, return_dict=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 876, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py", line 1365, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 952, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 248, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py", line 1183, in forward
    mamba_mask = self._update_mamba_mask(attention_mask, past_key_values)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py", line 1217, in _update_mamba_mask
    if (past_key_values is not None and past_key_values.has_previous_state()) or (
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rnowling/pytorch-venv/lib/python3.12/site-packages/transformers/cache_utils.py", line 1057, in has_previous_state
    raise ValueError(
ValueError: `has_previous_state` can only be called on LinearAttention layers, and the current Cache seem to only contain Attention layers.
```

### Expected behavior

I expect the script to be able to generate text from the model.  :)


Add V-JEPA 2.1 inference support

ISSUE #45496: Add V-JEPA 2.1 inference support
State: open  |  Labels: Feature request
Author: davevanveen  |  Created: 2026-04-17T20:59:54Z

### Feature request

Meta released [V-JEPA 2.1](https://github.com/facebookresearch/vjepa2) on 2026-03-16 with four pretrained video encoders at 384 resolution (ViT-B 80M, ViT-L 300M, ViT-g 1B, ViT-G 2B). The existing `vjepa2` model family in transformers supports V-JEPA 2.0 but not 2.1.

V-JEPA 2.1 introduces several architectural changes over 2.0: corrected RoPE implementation, learnable modality embeddings, hierarchical feature extraction with per-layer norms, separate image patch embedding, RoPE position interpolation, and a predictor context token projection. These require config and modeling extensions beyond what the current `VJEPA2Model` supports.

Paper: https://huggingface.co/papers/2603.14482
Code: https://github.com/facebookresearch/vjepa2 (see `app/vjepa_2_1/`)

### Motivation

V-JEPA 2.1 checkpoints are currently only loadable through Meta's torch.hub interface. Adding transformers support would let users load these models via `from_pretrained` with standard HF APIs, consistent with the existing V-JEPA 2.0 integration.

There is also an open request from the HF team to Meta to upload the 2.1 weights to the Hub: facebookresearch/vjepa2#137.

### Your contribution

I have a working implementation on a branch that extends the existing `vjepa2` model family with backward-compatible config fields and modeling changes. Verified end-to-end against Meta's reference (ViT-B/384 checkpoint, encoder max diff 0.0001, predictor max diff 0.008). All existing tests pass. I will open a PR shortly.


ISSUE #45491: [Gemma3] NaN embeddings on GPU when batching sequences of mixed length (sliding window attention + all-padding windows)
State: closed  |  Labels: bug
Author: RiccardoTOTI  |  Created: 2026-04-17T13:10:25Z

### System Info

**System Info**

- `transformers`: 4.45.1
- `sentence-transformers`: 5.1.2
- `tokenizers`: 0.20.0
- `safetensors`: 0.4.5
- `PyTorch`: ≥ 2.6.0
- Serving runtime: `pytorch/torchserve-kfs:0.12.0` (Python 3.9, Linux x86_64)
- GPU: NVIDIA (CUDA, via KServe / TorchServe on Kubernetes)
- CPU inference: **not affected**

---

**Description**

When encoding a batch of sentences of **mixed lengths** with `google/EmbeddingGemma` (300M) via `SentenceTransformer.encode_query()`, short sentences receive an **all-NaN embedding vector** on GPU. The same inputs encoded individually (`batch_size=1`) return correct, fully-normalized embeddings.

**Trigger condition:** the shortest sequence in the batch must be short enough that, when padded to the length of the longest sequence, one or more sliding-window attention windows fall **entirely on padding tokens**.

In the reported case:
- Chunk 0: **142 tokens** → padded to 728 tokens → **NaN=768/768** in the batch
- Chunks 1–3: 452 / 555 / 728 tokens → valid embeddings

The model's sliding window size is **512**. With 728 − 142 = 586 padding tokens appended, the first attention windows of Chunk 0 are 100% padding, causing `softmax([-inf, -inf, …]) = 0/0 = NaN` in the GPU kernel — a silent numerical error (no exception raised, no warning).

**Actual behavior:** Chunk 0 (shortest, ~142 tokens) returns a vector of 768 NaN values when batched with longer sequences. No exception or warning is raised.

**Workaround:** detect NaN after encoding and re-encode the affected inputs with `batch_size=1`.

### Who can help?

@ArthurZucker

@Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

**Steps to reproduce**

```python
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("google/EmbeddingGemma-300M")  # GPU required

chunks = [
    "ipso literam...",  # ~142 tokens
    "ipso literam...",                      # ~452 tokens
    "ipso literam...",           # ~555 tokens
    "ipso literam...",                # ~728 tokens
]

# Batch — Chunk 0 produces all-NaN on GPU
embs = model.encode_query(chunks, batch_size=64)
for i, emb in enumerate(embs):
    print(f"Chunk {i}: NaN={np.any(np.isnan(emb))}")
# Output:
# Chunk 0: NaN=True   ← 768/768 NaN values
# Chunk 1: NaN=False
# Chunk 2: NaN=False
# Chunk 3: NaN=False

# Individual — all correct
for chunk in chunks:
    emb = model.encode_query([chunk], batch_size=1)
    print(f"NaN={np.any(np.isnan(emb))}")  # False for all
```

### Expected behavior

**Expected behavior:** all embeddings in the batch are valid (no NaN).

--- Comment by Kabir08 at 2026-04-19T08:31:06Z ---
I believe the root cause is Sliding window attention + dynamic padding in batched processing → all-padding windows → numerical instability in softmax.

I have added pr fixing the same

--- Comment by vasqu at 2026-04-20T13:17:12Z ---
Honestly, this is not a `transformers` issue itself but a padding side issue. Regardless of the size of the chunk, having only pad input is invalid in itself (i.e. what values would you expect on all padding input --> NaNs, 0s, ...). So this doesn't need a fix by us but you need to force left padding side instead - not sure why right is the default padding side here tbh

cc @tomaarsen for ST

--- Comment by tomaarsen at 2026-04-20T13:25:18Z ---
Many encoders use right padding, and it seems that https://huggingface.co/google/embeddinggemma-300m is not an exception. Left padding is a bit of a newer development from decoders. I'll try and reproduce this myself locally, but if right-padding with sliding window attention breaks if there's more than `window_size` padding tokens, then that might reasonably be a bug. I'll try and reproduce this in a bit.

- Tom Aarsen

--- Comment by tomaarsen at 2026-04-20T14:20:33Z ---
@RiccardoTOTI embeddinggemma-300m its architecture (`gemma3_text`) was added in https://github.com/huggingface/transformers/releases/tag/v4.56.0-Embedding-Gemma-preview, but you report being on 4.45.1. I get this error when loading with your exact environment:
```
...
KeyError: 'gemma3_text'

During handling of the above exception, another exception occurred:
...
ValueError: The checkpoint you are trying to load has model type `gemma3_text` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
```

Could you share a bit more details? Also, on the latest Transformers/Sentence Transformers, I'm unable to get any NaNs even with inputs of the same shape:
```python
from sentence_transformers import SentenceTransformer, __version__ as st_version
from transformers import __version__ as transformers_version
import numpy as np

print(f"ST version: {st_version}")
print(f"Transformers version: {transformers_version}")

model = SentenceTransformer("google/embeddinggemma-300m")  # GPU required

chunks = [
    "and " * 139,  # ~142 tokens
    "and " * 449,  # ~452 tokens
    "and " * 552,  # ~555 tokens
    "and " * 725,  # ~728 tokens
]

inputs = model.preprocess(chunks)
print(inputs["attention_mask"].sum(1))

# Batch — Chunk 0 produces all-NaN on GPU
embs = model.encode_query(chunks, batch_size=64)
for i, emb in enumerate(embs):
    print(f"Chunk {i}: NaN={np.any(np.isnan(emb))}")  # False for all

# Individual — all correct
for chunk in chunks:
    emb = model.encode_query([chunk], batch_size=1)
    print(f"NaN={np.any(np.isnan(emb))}")  # False for all
```

- Tom Aarsen

--- Comment by RiccardoTOTI at 2026-04-20T14:59:02Z ---
> [@RiccardoTOTI](https://github.com/RiccardoTOTI) embeddinggemma-300m its architecture (`gemma3_text`) was added in https://github.com/huggingface/transformers/releases/tag/v4.56.0-Embedding-Gemma-preview, but you report being on 4.45.1. I get this error when loading with your exact environment:
>
> ```
> ...
> KeyError: 'gemma3_text'
>
> During handling of the above exception, another exception occurred:
> ...
> ValueError: The checkpoint you are trying to load has model type `gemma3_text` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
> ```
>
> Could you share a bit more details? Also, on the latest Transformers/Sentence Transformers, I'm unable to get any NaNs even with inputs of the same shape:
>
> from sentence_transformers import SentenceTransformer, __version__ as st_version
> from transformers import __version__ as transformers_version
> import numpy as np
>
> print(f"ST version: {st_version}")
> print(f"Transformers version: {transformers_version}")
>
> model = SentenceTransformer("google/embeddinggemma-300m")  # GPU required
>
> chunks = [
>     "and " * 139,  # ~142 tokens
>     "and " * 449,  # ~452 tokens
>     "and " * 552,  # ~555 tokens
>     "and " * 725,  # ~728 tokens
> ]
>
> inputs = model.preprocess(chunks)
> print(inputs["attention_mask"].sum(1))
>
> # Batch — Chunk 0 produces all-NaN on GPU
> embs = model.encode_query(chunks, batch_size=64)
> for i, emb in enumerate(embs):
>     print(f"Chunk {i}: NaN={np.any(np.isnan(emb))}")  # False for all
>
> # Individual — all correct
> for chunk in chunks:
>     emb = model.encode_query([chunk], batch_size=1)
>     print(f"NaN={np.any(np.isnan(emb))}")  # False for all
>
>     * Tom Aarsen

The pip list | grep transformers output showing 4.45.1 is from the base image before model dependencies are installed. At model load time, requirements.txt upgrades transformers to 4.50.0 into an isolated directory, so the actual runtime version is 4.50.0. The model loads fine on 4.50.0 — gemma3_text is already present there. The NaN occurs in its GPU attention kernel when running on that version

--- Comment by tomaarsen at 2026-04-20T15:04:16Z ---
embeddinggemma-300m requires the aforementioned version also for the inclusion of the `use_bidirectional_attention` parameter: https://huggingface.co/google/embeddinggemma-300m/blob/main/config.json#L57

More details in https://github.com/huggingface/sentence-transformers/issues/3725

- Tom Aarsen

--- Comment by RiccardoTOTI at 2026-04-20T15:08:48Z ---
> embeddinggemma-300m requires the aforementioned version also for the inclusion of the `use_bidirectional_attention` parameter: https://huggingface.co/google/embeddinggemma-300m/blob/main/config.json#L57
>
> More details in [huggingface/sentence-transformers#3725](https://github.com/huggingface/sentence-transformers/issues/3725)
>
>     * Tom Aarsen

Thank you — this is a key clarification. Our config.json does include use_bidirectional_attention: true. If that parameter is ignored on 4.50.0, the model was running with causal attention, which would explain the abnormal behaviour. We will upgrade to transformers ≥ 4.56 and re-test. We may need to revisit this issue entirely after the upgrade — the NaN under bidirectional attention on a correct runtime may or may not reproduce.

--- Comment by tomaarsen at 2026-04-20T15:10:30Z ---
Yes, please give it another try with a newer version and let us know! Good luck.

- Tom Aarsen


ISSUE #45488: LlamaTokenizer in v5 overrides tokenizer.json's ByteLevel pre-tokenizer with Metaspace, silently breaks DeepSeek V3/R1 family
State: open  |  Labels:
Author: dc3671  |  Created: 2026-04-17T08:50:48Z

### System info

- `transformers`: 5.3.0
- `tokenizers`: 0.22.2
- Python: 3.12 / Linux

### Who can help?

@ArthurZucker

### Reproduction

Tokenizer-only, ~7 MB download:

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3")
print(repr(tok.decode(tok.encode("hello world", add_special_tokens=False))))
# 'helloworld'  — expected 'hello world'
```

### Expected vs actual

| | v4.x / raw `tokenizers.Tokenizer.from_file` | v5.x `AutoTokenizer` |
|---|---|---|
| round-trip of `"hello world"` | `"hello world"` | `"helloworld"` |
| `backend_tokenizer.pre_tokenizer` | `Sequence([..., ByteLevel])` (from `tokenizer.json`) | `Metaspace(replacement="▁", split=False)` |

### Root cause

DeepSeek's `tokenizer_config.json` sets `"tokenizer_class": "LlamaTokenizerFast"`, but `tokenizer.json` is ByteLevel BPE (GPT-2 style, not SentencePiece). In v5, `LlamaTokenizer.__init__` unconditionally installs a `Metaspace` pre-tokenizer, overwriting whatever `tokenizer.json` declared. Since the vocab has no `▁`-prefixed tokens, the metaspace marker drops and spaces vanish.

Loading the same file via `tokenizers.Tokenizer.from_file(...)` or `PreTrainedTokenizerFast.from_pretrained(...)` works — the bug is specifically in `LlamaTokenizer`'s `__init__`.

### Affected models

All models shipping `tokenizer_class: Llama*Tokenizer` + a ByteLevel `tokenizer.json`. Confirmed broken on all 5 DeepSeek V3/R1 upstream repos:
`DeepSeek-V3`, `DeepSeek-V3.1`, `DeepSeek-V3.2-Exp`, `DeepSeek-R1`, `DeepSeek-R1-0528`.

Note: `LlamaTokenizer is LlamaTokenizerFast` in v5, so either class name in `tokenizer_config.json` triggers the bug.

### Downstream impact

Catastrophic silent accuracy loss on any HF-based inference stack. For example, GSM8K on DeepSeek-V3-Lite drops ~63.7% → ~26.1% purely from this tokenizer regression.

### Suggested fix

In `LlamaTokenizer.__init__`, only install the default `Metaspace` pre-tokenizer when `tokenizer.json` is absent (the "train from scratch" path). When loading an existing `tokenizer.json`, trust its `pre_tokenizer`.

### Workaround

```python
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("deepseek-ai/DeepSeek-V3")
```

--- Comment by Rocketknight1 at 2026-04-17T11:43:10Z ---
cc @arthurzucker @itazap

--- Comment by itazap at 2026-04-20T08:45:28Z ---
hey @dc3671 it is fixed in the latest version! We load from `TokenziersBackend` (aka `PreTrainedTokenizerFast`) for deepseek models: https://github.com/huggingface/transformers/blob/4f302caf4c8186c0352cf82ed1306c786928ee41/src/transformers/models/auto/tokenization_auto.py#L350-L354

--- Comment by dc3671 at 2026-04-20T08:47:07Z ---
@itazap thanks~ maybe you can link the PR here and close the issue.

--- Comment by itazap at 2026-04-20T08:52:48Z ---
https://github.com/huggingface/transformers/pull/44801 🤗

--- Comment by myym0 at 2026-04-22T03:18:09Z ---
Hello，the problem persists even after I updated the transformers using the method below.
pip install git+https://github.com/huggingface/transformers.git@main

--- Comment by ArthurZucker at 2026-04-22T06:25:04Z ---
@myym0 can you provide a reproducer please?

--- Comment by trevor-m at 2026-05-15T19:12:11Z ---
This appears to be broken again after https://github.com/huggingface/transformers/pull/45078

Repro in sglang: https://github.com/sgl-project/sglang/issues/25315

Other transformers issue for this same bug: #45812 #45920

--- Comment by itazap at 2026-05-18T02:03:02Z ---
```python
Testing model: deepseek-ai/DeepSeek-V3
Transformers version: 5.8.0.dev0
'hello world'
```

v3 is working on main

--- Comment by nvpohanh at 2026-05-18T02:30:12Z ---
@itazap Does HF Transformers have any CI to cover this? It feels that this issue has showed up many times...

--- Comment by nvpohanh at 2026-05-18T02:30:30Z ---
@trevor-m could you check if the main branch is okay now? thanks!

--- Comment by itazap at 2026-05-18T05:58:52Z ---
Sorry for the hassle! ☹

yep we test v3 in `tests/models/deepseek_v3/test_modeling_deepseek_v3.py` and `deepseek-ai/DeepSeek-R1` in `tests/test_tokenizers_backend_mixin.py`. However, some checkpoints like `https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B` incorrectly map to Llama. So we have to add a workaround or fix the config (opened a hub PR here: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B/discussions/36) and workaround PR here https://github.com/huggingface/transformers/pull/45936)

--- Comment by myym0 at 2026-05-19T09:03:15Z ---
> [@myym0](https://github.com/myym0) can you provide a reproducer please?

Sorry for the late reply. I updated transformers to the latest version (5.8.1), but the issue still occurs.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3")
print(repr(tok.decode(tok.encode("hello world", add_special_tokens=False))))
# 'helloworld'  — expected 'hello world'

--- Comment by trevor-m at 2026-05-19T18:05:27Z ---
I tested the main branch today (https://github.com/huggingface/transformers/commit/b797f0d4c74ded2e368bbc971c4aaca9004318cc) and it appears to be fixed:
```
python3 -c "from transformers import AutoTokenizer; tok = AutoTokenizer.from_pretrained('deepseek-ai/DeepSeek-R1-0528'); print(repr(tok.decode(tok.encode('hello world', add_special_tokens=False))))"
'hello world'
```

I tested a few more versions:
5.6.0: bad
5.6.1: bad
5.6.2: bad
5.7.0: good
5.8.0: good
5.8.1: good

@myym0 Not sure why we got different results


ISSUE #45482: Gemma4 26B-A4B: cross-device errors with CPU offload (RoPE, inputs, layer_scalar, SDPA mask, mm_token_type_ids)
State: open  |  Labels:
Author: sirfyyn  |  Created: 2026-04-16T15:57:28Z

# Bug: Gemma4 cross-device tensor errors with accelerate CPU offload

## Environment

- transformers latest (Gemma4 support, `modeling_gemma4.py`)
- Gemma4 26B-A4B-it (MoE, 4B active params)
- `accelerate` device_map with CPU offload (layers overflow to RAM)
- BnB INT8 + PEFT LoRA + Gradient Checkpointing
- RTX 4090 (24GB) + 60GB CPU RAM

## Problems

### P4: RoPE embeddings on wrong device

`apply_rotary_pos_emb` computes `cos`/`sin` on CPU (from the offloaded embedding table) but `q`/`k` are already on CUDA. Fix: `.to(q.device)` on cos/sin before application.

### P5: Input tensors not following device

In `Gemma4TextModel.forward`, `position_ids` and `attention_mask` sometimes stay on CPU when the layer has moved to CUDA mid-forward. Fix: explicit `.to(hidden_states.device)` at layer entry.

### P7: `layer_scalar` cross-device

`Gemma4DecoderLayer` applies `layer_scalar` (a float scalar or 1-element tensor) to the output. When the layer is on CUDA but `layer_scalar` is on CPU, this raises a device mismatch. Fix: `.to(hidden_states.device)` before multiplication.

### P6: SDPA `attention_mask` cross-device — `integrations/sdpa_attention.py`

`scaled_dot_product_attention` receives an `attention_mask` computed on CPU. SDPA requires mask and Q/K/V on the same device. Fix: `.to(query_states.device)` before the SDPA call.

### P10: `mm_token_type_ids` required even for text-only training

`Gemma4ForConditionalGeneration.forward` unconditionally accesses `mm_token_type_ids` from the batch. For text-only fine-tuning (no images), this key is absent → KeyError or None-dereference.

Fix: guard with `if mm_token_type_ids is not None:` and skip the multimodal routing path when absent.

## Workaround / patches

All 5 patches + a full training example (Gemma4 26B on RTX 4090, ~6.25s/step at 512 tokens):

https://github.com/sirfyyn/consumer-llm-patches

Patches are currently applied at runtime via `apply_patches.py`. Happy to contribute upstream PRs for any of these — particularly P10 (text-only training guard) and P6 (SDPA mask device) which seem most likely to affect others.

## Benchmark context

With all patches applied, Gemma4 26B-A4B trains on a single RTX 4090 with BnB INT8 + LoRA + GC + CPU offload. Step time is nearly flat across seq lengths (64→512 tok = 1.06× difference), indicating CPU→GPU transfer dominates, not compute.

--- Comment by zucchini-nlp at 2026-04-17T10:33:47Z ---
@sirfyyn hey, can you provide a small code snippet to reproduce?

--- Comment by sirfyyn at 2026-04-17T17:12:08Z ---
> @sirfyyn hey, can you provide a small code snippet to reproduce?

Yes, thanks for looking. Quick context on me: I'm not an ML researcher — I started on this a few months ago from a living room, learning as I go. I'm contributing what I can and corrections are genuinely welcome.

The five symptoms in the issue come from slightly different paths, so I've separated them. They all reproduce against `transformers @ main` (with Gemma4 support) when the LM overflows to CPU via `device_map="auto"`. The 26B-A4B variant reliably triggers P4/P5/P7 because ~layers 20+ land on CPU; a smaller Gemma4 in a squeezed `max_memory` should hit the same pattern.

---

### Shared setup

```python
# pip install transformers accelerate bitsandbytes peft torch
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

MODEL = "google/gemma-4-26B-A4B-it"   # any Gemma4 that can overflow to CPU works

tok   = AutoTokenizer.from_pretrained(MODEL)
bnb   = BitsAndBytesConfig(load_in_8bit=True, llm_int8_enable_fp32_cpu_offload=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    quantization_config=bnb,
    device_map="auto",
    max_memory={0: "22GiB", "cpu": "60GiB"},   # forces CPU overflow for later layers
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    attn_implementation="sdpa",                 # for P6
)
model.eval()
```

A plain forward on a short prompt is enough to surface P4/P5/P6/P7. P10 triggers when `Gemma4ForConditionalGeneration` is called without `mm_token_type_ids`.

---

### P4 — RoPE cos/sin on CPU while x is on CUDA

Signature in `models/gemma4/modeling_gemma4.py`:

```python
def apply_rotary_pos_emb(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor, unsqueeze_dim: int = 1):
    ...
```

When Q/K are computed on CUDA by an earlier layer but the rotary embedding cache was populated on a CPU-hosted parent module, `cos`/`sin` stay on CPU and the call fails.

Micro-repro, no 26B model needed:

```python
import torch
from transformers.models.gemma4.modeling_gemma4 import apply_rotary_pos_emb

x   = torch.randn(1, 4, 8, 64, device="cuda")
cos = torch.randn(1, 8, 64, device="cpu")    # stuck on CPU after offload
sin = torch.randn(1, 8, 64, device="cpu")

apply_rotary_pos_emb(x, cos, sin)
# RuntimeError: Expected all tensors to be on the same device,
# but found at least two devices, cuda:0 and cpu!
```

(In the real code path `apply_rotary_pos_emb` is called once for Q and once for K — same pattern, same crash.)

The patch I'm using does `cos = cos.to(x.device); sin = sin.to(x.device)` at the top of `apply_rotary_pos_emb` before unsqueeze.

---

### P5 — `position_ids` / `attention_mask` stay on CPU while hidden_states migrated

Inside `Gemma4TextModel.forward`, when `accelerate` dispatches a layer to CUDA but `position_ids` / `attention_mask` came from CPU inputs earlier in the pass, the attention path sees a device mismatch.

Full-model repro (using the setup above):

```python
inp = tok("Hallo Welt", return_tensors="pt").input_ids.to("cuda")
with torch.no_grad():
    out = model(inp)    # RuntimeError on the first CPU-hosted layer
```

Patch normalizes `attention_mask` / `position_ids` / `position_embeddings` to `hidden_states.device` at layer entry.

---

### P6 — SDPA attention mask not on query device (`integrations/sdpa_attention.py`)

Same forward as P5 surfaces first inside `scaled_dot_product_attention` if P5's normalization isn't applied:

```
RuntimeError: ... expected all tensors to be on the same device, cuda:0 and cpu
  at torch.nn.functional.scaled_dot_product_attention(..., attn_mask=...)
```

Patch does `attention_mask = attention_mask.to(query.device)` in the SDPA integration right before the `F.scaled_dot_product_attention` call. This is the single most portable fix — it helps any MoE / offloaded model routing through the SDPA integration, not just Gemma4.

---

### P7 — `layer_scalar` (residual scale) on cuda:0 while layer lives on CPU

```
RuntimeError: expected scalar and tensor on same device (cuda:0 vs cpu)
  at: hidden_states = hidden_states * self.layer_scalar
```

Happens during the backward of a CPU-hosted `Gemma4DecoderLayer` — the parameter buffer wasn't moved with the layer.

Patch does `self.layer_scalar.to(hidden_states.device)` before the multiply.

---

### P10 — text-only training raises on `mm_token_type_ids`

Trigger:

```python
from transformers import Gemma4ForConditionalGeneration
mm_model = Gemma4ForConditionalGeneration.from_pretrained(MODEL, ...)
batch = tok("nur text, keine bilder", return_tensors="pt").to("cuda")
# text-only tokenizer output has no mm_token_type_ids
mm_model(**batch)
# KeyError: 'mm_token_type_ids'   or   TypeError on None
```

Current `forward` unconditionally accesses `mm_token_type_ids`. For any text-only fine-tune that still goes through `Gemma4ForConditionalGeneration` (common with PEFT + the `...-it` checkpoint), this path fails.

Patch guards with `if mm_token_type_ids is not None:` and skips the multimodal routing when absent. This is the one I'd most like to upstream — it's a one-line guard and every text-only trainer will hit it.

---

### Which ones I can PR

Happy to open upstream PRs — order I'd suggest from most general to narrowest:

1. **P6 — SDPA mask device** (helps every offloaded model through SDPA, not just Gemma4)
2. **P10 — text-only `mm_token_type_ids` guard** (one-line, unblocks all text-only Gemma4 fine-tunes)
3. **P4 — RoPE cos/sin device** (Gemma4-specific but pattern may exist in related variants)
4. **P5 / P7 — narrow `.to(device)` fixes** inside Gemma4's decoder layer. I'm happy to submit these as-is; if a maintainer later decides a broader `accelerate`-level fix is cleaner, the narrow patches can be removed without cost. Just say the word and I'll open the PR.

Let me know which would be most useful first and I'll open it with a focused test. See also [consumer-llm-patches](https://github.com/sirfyyn/consumer-llm-patches) for the broader context and honest status per patch.

This isn't a one-shot report either — I'm actively continuing on the consumer-GPU Gemma4 stack, will share new insights as they come up, and I'll revise or retract whenever new findings contradict what I said earlier. If any of the five symptoms above turn out to be my environment rather than upstream, I'll come back and say so plainly.

— Max

--- Comment by zucchini-nlp at 2026-04-20T12:19:31Z ---
Interesting, can't reproduce it on `main` branch. Can you update transformers and again?

If it fails, pls share installed pkg version

--- Comment by github-actions[bot] at 2026-05-17T08:39:51Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #45479: `problem_type="single_label_classification"` with `num_labels=1` leads to degenerate zero loss across multiple sequence-classification models
State: closed  |  Labels: bug
Author: BohdanBabii  |  Created: 2026-04-16T14:58:54Z

### System Info

Hi, I found what looks like a library-wide issue in `transformers` affecting multiple `ForSequenceClassification` models, not just ModernBERT.

If a model is initialized with:
```python
num_labels=1
problem_type="single_label_classification"
```
the forward pass uses `CrossEntropyLoss()` with only one output logit. This leads to a degenerate zero loss during training/evaluation instead of performing binary classification meaningfully.

I first observed this with `ModernBertForSequenceClassification`, but the same logic appears in other sequence-classification models as well (for example RoBERTa and others using the same loss-selection pattern).

In `modeling_modernbert.py`, the relevant part is:

```python
loss = None
if labels is not None:
    if self.config.problem_type is None:
        if self.num_labels == 1:
            self.config.problem_type = "regression"
        elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
            self.config.problem_type = "single_label_classification"
        else:
            self.config.problem_type = "multi_label_classification"

    if self.config.problem_type == "regression":
        loss_fct = MSELoss()
        if self.num_labels == 1:
            loss = loss_fct(logits.squeeze(), labels.squeeze())
        else:
            loss = loss_fct(logits, labels)
    elif self.config.problem_type == "single_label_classification":
        loss_fct = CrossEntropyLoss()
        loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
    elif self.config.problem_type == "multi_label_classification":
        loss_fct = BCEWithLogitsLoss()
        loss = loss_fct(logits, labels)
```
With `num_labels=1 and problem_type="single_label_classification", this becomes:
```python
CrossEntropyLoss()(logits.view(-1, 1), labels.view(-1))
```

which produces a degenerate zero loss because there is only one class dimension.

Why I think this is a bug:
This setup naturally suggests binary classification with labels like:

- 0 -> class 0
- 1 -> class 1

So from a user perspective, this looks like it should be a valid single-label binary classification setup.
Right now, however, `num_labels=1` is effectively treated as if there were only one possible class in the loss computation, which makes the classification loss meaningless.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Minimal reproduction
```python
from transformers import AutoModelForSequenceClassification
import torch

model = AutoModelForSequenceClassification.from_pretrained(
    "answerdotai/ModernBERT-base",
    num_labels=1,
    problem_type="single_label_classification",
)

input_ids = torch.tensor([[101, 102]])
attention_mask = torch.tensor([[1, 1]])
labels = torch.tensor([0])

outputs = model(
    input_ids=input_ids,
    attention_mask=attention_mask,
    labels=labels,
)

print(outputs.logits.shape)
print(outputs.loss)
```

Observed result
`outputs.loss` is `0` (or degenerate), and the same behavior also shows up during training.

### Expected behavior

Expected behavior
I would expect `num_labels=1` with `problem_type="single_label_classification"` to support binary classification meaningfully for labels `{0, 1}`, instead of silently producing a degenerate zero loss.
For example, this could be implemented with a single-logit binary objective such as `BCEWithLogitsLoss`, or by internally mapping this configuration to an equivalent binary-classification setup.
In any case, the current behavior of silently returning zero loss seems incorrect.

Actual behavior
The model runs, but training/eval loss becomes degenerate `(0)` because `CrossEntropyLoss` is applied to logits with shape `[..., 1]`.

--- Comment by Mati0kez at 2026-04-16T18:14:58Z ---
I've been looking into this — the root cause appears to be CrossEntropyLoss receiving logits.view(-1, 1) when num_labels=1. Happy to submit a fix that maps this to BCEWithLogitsLoss for meaningful binary classification.

--- Comment by Mati0kez at 2026-04-16T18:19:35Z ---
I've been looking into this — the root cause appears to be CrossEntropyLoss receiving logits.view(-1, 1) when num_labels=1. Happy to submit a fix that handles this as binary classification.

--- Comment by Rocketknight1 at 2026-04-17T11:48:17Z ---
If you have two classes, just use `num_labels=2`. Binary classification with a single sigmoid output or regular classification with two softmaxed outputs are interchangeable, so we don't have special binary classification code for `num_labels=1`.

There's nothing really for us to fix here - but maybe we could raise a clearer error if users set `num_labels=1` and `problem_type="single_label_classification"` to tell them not to do that?


ISSUE #45478: [BUG] transformers>=5.4.0, Qwen3.5 Moe from_pretrained error
State: closed  |  Labels: Should Fix, bug
Author: Jintao-Huang  |  Created: 2026-04-16T14:48:15Z

### System Info

https://github.com/huggingface/transformers/issues/45310

This issue has not been fixed in the main branch.

```
import os
os.environ['CUDA_VISIBLE_DEVICS'] = '0'

from transformers import Qwen3_5ForConditionalGeneration, AutoTokenizer

model = Qwen3_5ForConditionalGeneration.from_pretrained('Qwen/Qwen3.5-35B-A3B')
model.save_pretrained('/root/Qwen3.5-35B-A3B', max_shard_size='10GB')
model = Qwen3_5ForConditionalGeneration.from_pretrained('/root/Qwen3.5-35B-A3B')
```

### Who can help?

-

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

--- Comment by Jintao-Huang at 2026-04-16T14:48:45Z ---
<img width="781" height="247" alt="Image" src="https://github.com/user-attachments/assets/76df3ba1-9715-4145-b49d-00ec47aef4f9" />

--- Comment by JJJYmmm at 2026-04-17T07:28:58Z ---
I’m not fully caught up on the context, but qwen3.5 moe models (35a3, 122a10) should use `Qwen3_5MoeForConditionalGeneration`, whereas `Qwen3_5ForConditionalGeneration` is for dense variants (0.8b, 9b, 27b)

--- Comment by zucchini-nlp at 2026-04-17T10:30:36Z ---
cc @Cyrilvallez, same as in the other issue with DeepSpeed

--- Comment by Cyrilvallez at 2026-04-20T01:31:00Z ---
This is not a bug, `Qwen/Qwen3.5-35B-A3B` is `Qwen3_5MoeForConditionalGeneration`, not `Qwen3_5ForConditionalGeneration`. If you don't use AutoModels, please make sure to use the correct class.


ISSUE #45468: [BUG] Gemma-4 Gemma4AudioRelPositionalEncoding
State: closed  |  Labels: bug
Author: foldl  |  Created: 2026-04-16T06:52:43Z

### System Info

N/A.

### Who can help?

The hard coded numbers **12** and **-1** seem to be related to `attention_context_left` and `attention_context_right`.

https://github.com/huggingface/transformers/blob/8426e7e63d49d9c3b5f0c09d43e792a59c75c62c/src/transformers/models/gemma4/modular_gemma4.py#L160

@eustlb @ebezzam @vasqu

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code review.

### Expected behavior

Use params from `config`.

--- Comment by mathceo at 2026-04-16T09:30:36Z ---
I’d like to work on this. I’ll inspect how the positional range should be derived from the config, replace the hard coded values if appropriate, and add a regression test.

--- Comment by Aravind-11 at 2026-04-22T18:57:03Z ---
it'd be better if you describe what the issue is .

--- Comment by vasqu at 2026-04-23T12:08:54Z ---
Yea, I don't really get what the issue is 😅 could you clarify the issue, is this a feature request / bug report etc. Best to have some code for reproducability along with it

--- Comment by mathceo at 2026-04-23T13:24:18Z ---
@Aravind-11 @vasqu
I checked the code path more closely.
This does look like a real inconsistency: Gemma4AudioRelPositionalEncoding computes self.context_size from config and its docstring says it produces [1, 2 * context_size - 1, hidden_size], but forward() ignores that and always uses torch.arange(12, -1, -1)
That said, the attention path currently pads position_length up to context_size + 1 in _rel_shift, so the current hard-coded length of 13 is not obviously random and may be intentional for the blocked attention formulation. So I think the issue is real, but the correct fix is not yet proven to be simply 2 * context_size - 1.
Given that all public Gemma 4 audio configs currently use the same 12 / 13 / 0 defaults, this may not be a user-visible bug on released checkpoints today, but it is definitely a config/implementation mismatch and likely breaks or misspecifies nondefault configs.
I think the next step is to confirm what the intended positional range should be here (full relative range vs chunked-attention-specific range), then either:
1. derive it from config properly, or
2. fix the docstring/comments if the current shorter range is actually intended.

--- Comment by eustlb at 2026-04-23T14:43:10Z ---
This is not a bug per say, but as stated above this should not be hardcoded but rather inferred from config.
The correct value for it should be context_size // 2 (so 13 values). If you want to dive more deeply into it, you can check section 3.3 of https://huggingface.co/papers/1901.02860, and appendix B that explains the rel_shift trick 🤗

opened a PR to fix this: #45606


ISSUE #45464: chat/completions API fail on Qwen3.5-0.8B for streaming inference
State: open  |  Labels: bug
Author: zhangwei217245  |  Created: 2026-04-15T16:36:31Z

### System Info

- `transformers` version: 5.5.0 - 5.5.4
- Platform: macOS-26.4.1-arm64-arm-64bit-Mach-O
- Python version: 3.14.3
- Huggingface_hub version: 1.10.2
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.11.0 (NA)
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

as long as this is a streaming request, it will fail.
```
curl -X 'POST' \
  'http://localhost:8000/v1/chat/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model":"Qwen/Qwen3.5-0.8B",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What'\''s in this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://people.com/thmb/--y1mYUxVOMtWv0-z9xrjVHOfzc=/1500x0/filters:no_upscale():max_bytes(150000):strip_icc():focal(777x0:779x2)/endangered-3-2000-f2e1bd4fe1024e4f8ce0cc4945efd46f.jpg",
            "detail": "auto"
          }
        }
      ]
    }
  ],
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "temperature": 0.7
}'
```

Error in the log:

```
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/uvicorn/protocols/http/httptools_impl.py", line 420, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/applications.py", line 1163, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/applications.py", line 90, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 191, in __call__
    with recv_stream, send_stream, collapse_excgroups():
                                   ~~~~~~~~~~~~~~~~~~^^
  File "/opt/homebrew/Cellar/python@3.14/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
    raise exc
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 193, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/server.py", line 83, in request_id_middleware
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 168, in call_next
    raise app_exc from app_exc.__cause__ or app_exc.__context__
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 144, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/routing.py", line 660, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/routing.py", line 680, in app
    await route.handle(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 134, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 120, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 674, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 328, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/server.py", line 91, in chat_completions
    return await chat_handler.handle_request(body, request.state.request_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/chat_completion.py", line 138, in handle_request
    return self._streaming(
           ~~~~~~~~~~~~~~~^
        request_id,
        ^^^^^^^^^^^
    ...<6 lines>...
        tool_format=tool_format,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/chat_completion.py", line 174, in _streaming
    queue, streamer = gen_manager.generate_streaming(model, processor, inputs, gen_config, request_id=request_id)
                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/utils.py", line 565, in generate_streaming
    streamer = DirectStreamer(processor._tokenizer, loop, queue, skip_special_tokens=True)
                              ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Qwen3VLProcessor' object has no attribute '_tokenizer'. Did you mean: 'tokenizer'?
```

### Expected behavior

It should work as it would in version 5.4.0.

--- Comment by Brianzhengca at 2026-04-15T22:02:59Z ---
Potentially already covered by this PR: https://github.com/huggingface/transformers/pull/45368

--- Comment by Rocketknight1 at 2026-04-16T10:49:08Z ---
@zhangwei217245 can you check if it's fixed on `main`?

--- Comment by github-actions[bot] at 2026-05-16T08:31:39Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #45459: `except import_protobuf_decode_error()` hides real tokenizer errors when protobuf isn't installed
State: closed  |  Labels: bug
Author: jw9603  |  Created: 2026-04-15T12:48:42Z

### System Info

transformers 5.5.4 (latest release) and 5.6.0.dev0 (main).

`PreTrainedTokenizerBase._from_pretrained` has `except import_protobuf_decode_error():` at `src/transformers/tokenization_utils_base.py:1919` (line 1933 on main). The helper raises `ImportError` when protobuf isn't installed. The except-class expression is evaluated lazily when the try block raises, so that `ImportError` gets raised instead of the original exception.

protobuf isn't in `install_requires` (it only comes in through the `sentencepiece` extra), so a plain `pip install transformers` triggers this for any exception the tokenizer constructor raises. The `RuntimeError` and `OSError` handlers below never get a chance either, because the except evaluator itself raising terminates handler dispatch.

### Who can help?

@ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

With protobuf not installed (`pip install transformers` without any extras):

```python
from transformers.tokenization_utils_base import import_protobuf_decode_error

try:
    raise ValueError("real error")
except import_protobuf_decode_error():
    pass
except ValueError as e:
    print("caught real:", e)
# ImportError: ... requires the protobuf library ...
```

**Suggested fix**

Return `()` when protobuf is missing so the except matches nothing:

```python
def import_protobuf_decode_error(error_message=""):
    if is_protobuf_available():
        from google.protobuf.message import DecodeError

        return DecodeError
    return ()
```
`DecodeError` can't be raised without protobuf imported, so `()` is safe here. The function isn't exported and has only the single call site.

### Expected behavior

Without protobuf installed, exceptions from the tokenizer constructor should surface to the caller instead of being replaced by `ImportError: ...requires the protobuf library...`.

--- Comment by paarths-collab at 2026-04-15T16:22:02Z ---
Hi! I’d like to work on this issue. Could you confirm if it’s still open and guide me on the expected approach?

--- Comment by jw9603 at 2026-04-15T23:08:17Z ---
Thanks for the interest!
I'm already working on this. Fix and a regression test are ready; just waiting for @ArthurZucker or @itazap to confirm the direction.

--- Comment by jw9603 at 2026-04-16T01:05:01Z ---
@Jah-yee You opened #45466 after I explicitly said above that I'd handle this. Per AGENTS.md, opening a PR for someone else's issue requires explicit approval from the issue author or a maintainer. Please close it so I can send my own PR once @ArthurZucker or @itazap confirm the direction.

--- Comment by itazap at 2026-04-16T14:32:09Z ---
sounds good @jw9603 , feel free to open a PR 🤗


ISSUE #45458: Add typing support incrementally (meta issue)
State: closed  |  Labels:
Author: tarekziade  |  Created: 2026-04-15T12:36:24Z

We’re progressively adding typing support to the codebase using `ty`. This issue tracks the overall progress as we extend coverage directory by directory.

# Current status

The tooling is already in place. Type checking is enabled for a subset of directories

You can run it locally with:

```
make typing
```

# How to contribute

To add typing support to a new file or directory:

- Register it in [`utils/check_types.py`](https://github.com/huggingface/transformers/blob/main/utils/check_types.py) under `CHECKER_CONFIG`
- Run `make typing`
- Fix any reported issues

# Helper resources

There is an internal skill to assist with adding or fixing typing when using AI agents

[`.ai/skills/add-or-fix-type-checking/SKILL.md`](https://github.com/huggingface/transformers/blob/main/.ai/skills/add-or-fix-type-checking/SKILL.md)

This guide is continuously refined as we make progress, so it’s a good starting point.


ISSUE #45447: granitemoehybrid: HybridMambaAttentionDynamicCache missing from modeling_granitemoehybrid — breaks ibm-granite/granite-4.0-3b-vision remote code
State: closed  |  Labels:
Author: Steve-Allison  |  Created: 2026-04-15T06:13:12Z

## Summary

The `ibm-granite/granite-4.0-3b-vision` model's remote `modeling.py` imports `HybridMambaAttentionDynamicCache` from `transformers.models.granitemoehybrid.modeling_granitemoehybrid`. This class does not exist in transformers 5.5.4 (latest) or on the current `main` branch, causing an `ImportError` whenever any application loads this model with `trust_remote_code=True`.

## Error

```
ImportError: cannot import name 'HybridMambaAttentionDynamicCache' from
'transformers.models.granitemoehybrid.modeling_granitemoehybrid'
```

## Reproduction

```python
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained(
    "ibm-granite/granite-4.0-3b-vision",
    trust_remote_code=True,
)
```

Stack trace excerpt:

```
File "~/.cache/huggingface/modules/transformers_modules/<hash>/modeling.py", line 19
    from transformers.models.granitemoehybrid.modeling_granitemoehybrid import (
        HybridMambaAttentionDynamicCache,
    )
ImportError: cannot import name 'HybridMambaAttentionDynamicCache'
```

## Investigation

- `HybridMambaAttentionDynamicCache` existed in transformers 4.40.x (introduced with the Jamba model, commit `3f20877`).
- The class was removed / not carried forward during the 4.x → 5.x cache refactoring.
- A search across the entire `transformers` 5.5.4 install finds **zero occurrences** of the symbol.
- The same search on the current `main` branch also finds **zero occurrences**.
- The `granitemoehybrid` module in 5.5.4 has no cache class of its own; `olmo_hybrid` introduced `OlmoHybridDynamicCache` as the new pattern for hybrid Mamba/attention cache.
- Draft PR [#44445](https://github.com/huggingface/transformers/pull/44445) ("Adding support for GraniteDoclingHybrid") is open but still a draft and does not yet add this class.

## Environment

| | |
|---|---|
| transformers | 5.5.4 |
| Python | 3.14.4 |
| PyTorch | 2.11.0 |
| Platform | macOS 26.4.1 arm64 |

## Expected behaviour

Either:
1. `HybridMambaAttentionDynamicCache` (or a compatible replacement) is exported from `transformers.models.granitemoehybrid.modeling_granitemoehybrid` so the model's remote code loads without error; or
2. The `granitemoehybrid` module's public API documents the correct replacement so downstream model authors (ibm-granite) can update their `modeling.py`.

A parallel issue has been filed against [docling-project/docling#3303](https://github.com/docling-project/docling/issues/3303) since `ChartExtractionModelGraniteVisionV4` in docling 2.88.0 triggers this crash path.

--- Comment by Rocketknight1 at 2026-04-15T11:08:53Z ---
Hmm, this is strange. That model was only added in March, after the `v5` release. So there shouldn't be a backward compatibility issue there. I'm not sure why the remote code there is referencing missing classes!

--- Comment by zucchini-nlp at 2026-04-15T11:34:41Z ---
Btw, we will support Granite4-Vision soon natively in transformers. IBM team is already working on it :)

--- Comment by Rocketknight1 at 2026-04-15T12:17:46Z ---
Moving to a native implementation seems like the best solution, yes!

--- Comment by github-actions[bot] at 2026-05-15T08:58:11Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by zucchini-nlp at 2026-05-18T01:05:14Z ---
Model shipped in https://github.com/huggingface/transformers/pull/45597, so now you can use Granite 4V natively with transformers :)


ISSUE #45446: Incorrect PyTorch version check for AuxRequest import in flex_attention
State: closed  |  Labels: bug
Author: ZSLsherly  |  Created: 2026-04-15T05:26:22Z

### System Info

In src/transformers/integrations/flex_attention.py, the code currently checks for PyTorch version >= 2.9.0 to import AuxRequest from torch.nn.attention.flex_attention. However, AuxRequest was actually introduced in PyTorch 2.9.1.
According to the official PyTorch documentation, AuxRequest is available starting from version 2.9.1:https://docs.pytorch.org/docs/2.9/nn.attention.flex_attention.html.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch

#This will fail in PyTorch 2.9.0
from torch.nn.attention.flex_attention import AuxRequest
```

### Expected behavior

The version check should be updated to >= 2.9.1 to ensure compatibility with PyTorch 2.9.0.

--- Comment by ZSLsherly at 2026-04-15T11:36:37Z ---
This issue is fine


ISSUE #45440: Native `DeepseekV3MoE` diverges from the remote DeepSeekV3 implementation
State: closed  |  Labels: bug
Author: casinca  |  Created: 2026-04-14T17:50:49Z

### System Info

Hello, the `DeepseekV3MoE` class in transformers (native) differs from the official remote DeepSeekV3
implementation (which was updated for a bug but not in `transformers`, hence the difference).

<details>
<summary> See trf DeepSeekV3 MoE code </summary>

https://github.com/huggingface/transformers/blob/155db7146371335bdfa93f239c3b868b280e30b7/src/transformers/models/deepseek_v3/modular_deepseek_v3.py#L113-L166

</details>

The divergence comes from how experts are masked in the specific case of aux-loss free load balancing and negative
biases being added.
`scores_for_choice` is normally not negative (sigmoid [0,1]) but it can with + negative bias. So 0.0 is not a floor anymore, but just another value in the score distribution hence the need to switch from `masked_fill(~score_mask.bool(), 0.0)` to `masked_fill(~score_mask.bool(), -inf)`.

See the official remote update:
https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/commit/e9b33add76883f293d6bf61f6bd89b497e80e335#d2h-632685
Same fix is applied across the whole V3-* collection: https://huggingface.co/collections/deepseek-ai/deepseek-v3

(btw, I also need this for the MiMo-V2 model PR https://github.com/huggingface/transformers/pull/45144 in order to inherit properly.)

### Who can help?

Was discussed with Vasqu already cc'ed in the PR

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

### See example below (where a masked expert is chosen over a valid expert with a negative score)

`score_mask` = [1,1,0,0] (Only experts 0 and 1 should be eligible)

With `0.0`:

let's say `scores_for_choice = [0.9, -0.1, 0.0, 0.0]`
topk(k=2) → expert 0 (0.9) and expert 3 (0.0) are chosen = bad
Expert 3 is in the non-selected group but beats expert 1 (−0.1).

### Expected behavior

With `float("-inf")`:

`scores_for_choice = [0.9, -0.1, -inf, -inf]`
topk(k=2) → expert 0 (0.9) and expert 1 (-0.1) = good


ISSUE #45431: Wrong checkpoint path in Dinov2 model_docs
State: closed  |  Labels:
Author: ambroiseodt  |  Created: 2026-04-14T13:53:21Z

Wrong checkpoint path in Dinov2 model_docs.

The current checkpoint "google/dinov2-base-patch16-224" does not exist.  The correct one should be "facebook/dinov-base".

This issue is fixed in PR #45430

### Who can help?

@yonigozlan @molbap @stevhliu

### Notes

This is a minor issue, but it might help new users. Thanks for your time and for maintaining such a great library.

### Update

The PR #45430 solving this issue was merged, so I am closing the issue.


ISSUE #45419: Chat template inconsistencies in tool-calling support
State: open  |  Labels:
Author: qgallouedec  |  Created: 2026-04-13T23:27:06Z

Chat templates across model families handle tool-calling messages inconsistently. This creates fragility for any library (like TRL) that needs to construct tool-calling conversations programmatically, since there's no single "safe" way to build an assistant message with `tool_calls`.

I ran a systematic check across all tool-calling-capable templates. Two axes were tested:

1. **Argument format**: `arguments` as a `dict` vs a JSON `str`
2. **Content field**: `content=None`, `content=""`, or omitting the `content` key entirely

## Results

### Argument format (`dict` vs JSON `str`)

| Model | `dict` | `str` |
| --- | --- | --- |
| [DeepSeekV3](https://huggingface.co/deepseek-ai/DeepSeek-R1) | :x: | :white_check_mark: |
| [DeepSeekV3-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) | :x: | :white_check_mark: |
| [DeepSeekV4](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/discussions/146) (when pr merged) |  :white_check_mark: | :warning: silent miscompile |
| [EXAONE-4.5](https://huggingface.co/LGAI-EXAONE/EXAONE-4.5-33B) | :white_check_mark: | :white_check_mark: |
| [Gemma4](https://huggingface.co/google/gemma-4-E2B-it) | :white_check_mark: | :warning: double-escaped |
| [GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) | :white_check_mark: | :x: |
| [GLM4MOE](https://huggingface.co/zai-org/GLM-4.5) | :white_check_mark: | :x: |
| [GptOSS](https://huggingface.co/openai/gpt-oss-20b) | :white_check_mark: | :warning: double-escaped |
| [Holo3](https://huggingface.co/Hcompany/Holo3-35B-A3B) | :white_check_mark: | :x: |
| [LFM2.5-VL](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) | :white_check_mark: | :x: |
| [Llama3.1](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | :white_check_mark: | :warning: double-escaped |
| [Llama3.2](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | :white_check_mark: | :warning: double-escaped |
| [Marco-Mini](https://huggingface.co/AIDC-AI/Marco-Mini-Instruct) | :white_check_mark: | :white_check_mark: |
| [MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) | :white_check_mark: | :x: |
| [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) | :white_check_mark: | :x: |
| [Qwen2.5](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) | :white_check_mark: | :warning: double-escaped |
| [Qwen3](https://huggingface.co/Qwen/Qwen3-8B) | :white_check_mark: | :white_check_mark: |
| [Qwen3-Coder](https://huggingface.co/Qwen/Qwen3-Coder-Next) | :white_check_mark: | :x: |
| [Qwen3MOE](https://huggingface.co/Qwen/Qwen3-30B-A3B) | :white_check_mark: | :white_check_mark: |
| [Qwen3VL](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) | :white_check_mark: | :white_check_mark: |
| [Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-0.8B) | :white_check_mark: | :x: |

:warning: = no error, but produces wrong output (e.g. `"{\\"city\\": \\"Paris\\"}"` instead of `{"city": "Paris"}`). Only Qwen3, EXAONE-4.5, and Marco-Mini normalize both formats to the same output.

### Content field alongside `tool_calls`

| Model | `content=None` | `content=""` | No `content` key |
| --- | --- | --- | --- |
| [DeepSeekV3](https://huggingface.co/deepseek-ai/DeepSeek-R1) | :white_check_mark: | :white_check_mark: | :x: |
| [DeepSeekV3-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) | :x: | :white_check_mark: | :x: |
| [EXAONE-4.5](https://huggingface.co/LGAI-EXAONE/EXAONE-4.5-33B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Gemma4](https://huggingface.co/google/gemma-4-E2B-it) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [GLM4MOE](https://huggingface.co/zai-org/GLM-4.5) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [GptOSS](https://huggingface.co/openai/gpt-oss-20b) | :x: | :white_check_mark: | :white_check_mark: |
| [Holo3](https://huggingface.co/Hcompany/Holo3-35B-A3B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [LFM2.5-VL](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) | :x: | :white_check_mark: | :warning: missing trailing `<\|im_end\|>\n` |
| [Llama3.1](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Llama3.2](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Marco-Mini](https://huggingface.co/AIDC-AI/Marco-Mini-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) | :warning: literal `"None"` | :white_check_mark: | :white_check_mark: |
| [Qwen2.5](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3](https://huggingface.co/Qwen/Qwen3-8B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3-Coder](https://huggingface.co/Qwen/Qwen3-Coder-Next) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3MOE](https://huggingface.co/Qwen/Qwen3-30B-A3B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| [Qwen3VL](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) | :x: | :white_check_mark: | :white_check_mark: |
| [Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-0.8B) | :white_check_mark: | :white_check_mark: | :white_check_mark: |

`content=""` is the only universally safe option today.

### Reproduction

<details>
<summary>Script to reproduce both tables</summary>

```python
from jinja2 import TemplateError
from transformers import AutoTokenizer

models = {
    "DeepSeekV3": "deepseek-ai/DeepSeek-R1",
    "DeepSeekV3-0528": "deepseek-ai/DeepSeek-R1-0528",
    "EXAONE-4.5": "LGAI-EXAONE/EXAONE-4.5-33B",
    "Gemma4": "google/gemma-4-E2B-it",
    "GLM-5.1": "zai-org/GLM-5.1",
    "GLM4MOE": "zai-org/GLM-4.5",
    "GptOSS": "openai/gpt-oss-20b",
    "Holo3": "Hcompany/Holo3-35B-A3B",
    "LFM2.5-VL": "LiquidAI/LFM2.5-VL-450M",
    "Llama3.1": "meta-llama/Llama-3.1-8B-Instruct",
    "Llama3.2": "meta-llama/Llama-3.2-1B-Instruct",
    "Marco-Mini": "AIDC-AI/Marco-Mini-Instruct",
    "MiniMax-M2.5": "MiniMaxAI/MiniMax-M2.5",
    "MiniMax-M2.7": "MiniMaxAI/MiniMax-M2.7",
    "Qwen2.5": "Qwen/Qwen2.5-32B-Instruct",
    "Qwen3": "Qwen/Qwen3-8B",
    "Qwen3-Coder": "Qwen/Qwen3-Coder-Next",
    "Qwen3MOE": "Qwen/Qwen3-30B-A3B",
    "Qwen3VL": "Qwen/Qwen3-VL-2B-Instruct",
    "Qwen3.5": "Qwen/Qwen3.5-0.8B",
}

tc_dict = [{"type": "function", "function": {"name": "f", "arguments": {"a": 1}}}]
tc_str = [{"type": "function", "function": {"name": "f", "arguments": '{"a": 1}'}}]
MISSING = object()

def render(tokenizer, tool_calls, content=MISSING):
    msg = {"role": "assistant", "tool_calls": tool_calls}
    if content is not MISSING:
        msg["content"] = content
    try:
        return tokenizer.apply_chat_template([{"role": "user", "content": "hi"}, msg], tokenize=False)
    except (TemplateError, TypeError, KeyError):
        return None

def cell(result, reference=None):
    if result is None:
        return ":x:"
    if reference is not None and result != reference:
        return ":warning:"
    return ":white_check_mark:"

# ── Table 1: argument format ─────────────────────────────────────────────────
print("| Model | `dict` | `str` |")
print("|---|---|---|")
for name, model_id in models.items():
    tok = AutoTokenizer.from_pretrained(model_id)
    r_dict = render(tok, tc_dict, "")
    r_str = render(tok, tc_str, "")
    # Reference is the dict output when available, else str
    ref = r_dict if r_dict is not None else r_str
    print(f"| {name} | {cell(r_dict)} | {cell(r_str, ref)} |")

print()

# ── Table 2: content field ────────────────────────────────────────────────────
print("| Model | `content=None` | `content=\"\"` | No `content` key |")
print("|---|---|---|---|")
for name, model_id in models.items():
    tok = AutoTokenizer.from_pretrained(model_id)
    # Use each model's working arg format
    tc = tc_dict if render(tok, tc_dict, "") is not None else tc_str
    ref = render(tok, tc, "")
    r_none = render(tok, tc, None)
    r_missing = render(tok, tc, MISSING)
    print(f"| {name} | {cell(r_none, ref)} | {cell(ref)} | {cell(r_missing, ref)} |")
```

</details>

## Discussion

### `arguments` format: template bug

The [transformers docs](https://huggingface.co/docs/transformers/en/chat_extras#tool-calling-example) already specify that `arguments` should be a `dict`. So templates that reject `dict` args (DeepSeekV3) can be considered "broken".

> [!TIP]
> **Recommendation 1**: fix templates that reject `dict` args via PRs on the Hub model repos (e.g. `deepseek-ai/DeepSeek-R1`).

Templates that silently double-escape `str` args (Llama, Gemma4, Qwen2.5, GptOSS) are a softer issue: the caller is passing the wrong type, but the template produces wrong output instead of failing loudly.

> [!TIP]
> **Recommendation 2**: `apply_chat_template` could emit a warning when `arguments` is a string, to catch silent double-escaping early.

### `content` field: no spec exists

There is no documented contract for `content` in assistant messages with `tool_calls`. All three variants are legitimate (`None` from the OpenAI API, `""` from downstream normalization, absent key from the transformers docs examples), yet 7/20 templates break or misbehave on at least one.

> [!TIP]
> **Recommendation 3**: define the contract — in assistant messages with `tool_calls`, the `content` key MAY be absent. Templates must handle this case.

> [!TIP]
> **Recommendation 4**: `apply_chat_template` should drop `content` from assistant messages when it is `None`, so that templates only need to handle two cases (key absent vs string value which could technically mean different things) instead of three. #45422

> [!TIP]
> **Recommendation 5**: build a Space that analyzes any model's chat template and checks compliance with these contracts (dict args support, absent content key handling, etc.). Since templates live in Hub repos and not in transformers, a central linting tool would help model authors catch issues before users hit them.

What do you think?

--- Comment by zeel2104 at 2026-04-15T15:37:11Z ---
If it is open for contribution, I would like to work on it

--- Comment by Rocketknight1 at 2026-04-24T15:08:26Z ---
Sorry for taking a while to get to these! Firstly, I don't think we want to enforce any contract for tools being passed as strings. I think once you're doing that, you're in the realm of undefined behaviour. Our API expectation is tool dicts, which is also what Transformers creates if you just pass in a Python function. If your template wants to support something else too, it can, but I'm not going to worry too much about ensuring consistent behaviour there. However, we **should** try to chase down templates that don't accept dicts, and make sure they do. Emitting a warning when users pass a string is fine too, it might improve API compliance.

I think recommendations 3 and 4 are also good - your PR #45422 already addresses this to some extent. I think we can be clear in the docs that content is an optional field when tool calls are present, and that `content = None` or a missing `content` key are treated as equivalent.


ISSUE #45412: RT-DETR models do not release memory when deleted / garbage-collected
State: closed  |  Labels: bug
Author: dhdaines  |  Created: 2026-04-13T15:46:43Z

### System Info

Transfomers: 5.5.3
PyTorch: 2.8.0+cu126
TorchVision: 0.23.0+cu126
System: Debian 13 (trixie)
Python: 3.13.5

### Who can help?

@yonigozlan @molbap

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using slightly modified example scripts from the documentation (e.g. https://huggingface.co/docs/transformers/model_doc/detr?usage=AutoModel), we see that while Deformable DETR releases (nearly all) memory after deleting the model and running GC, RT-DETRv2 (and also RT-DETR) hold onto a significant amount of GPU memory which can never be released.  Here is the code snippet in question:

```python
import gc
import torch
import requests

from PIL import Image
from transformers import AutoModelForObjectDetection, AutoImageProcessor

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

device = torch.device("cuda")
image_processor = AutoImageProcessor.from_pretrained("PekingU/rtdetr_v2_r18vd",
                                                     backend="pil")
model = AutoModelForObjectDetection.from_pretrained("PekingU/rtdetr_v2_r18vd").to(device)
inputs = image_processor(images=image, return_tensors="pt").to(device)

print("before: %d" % (torch.cuda.memory_allocated() / 1024 / 1024))
with torch.inference_mode():
    outputs = model(**inputs)
print("after: %d" % (torch.cuda.memory_allocated() / 1024 / 1024))

del inputs
del outputs
del model
del image_processor
gc.collect()
torch.cuda.empty_cache()
print("gc: %d" % (torch.cuda.memory_allocated() / 1024 / 1024))
```

### Expected behavior

Expect the final number to be small, reflecting only the unfreeable (thanks a lot, NVidia!) CUDA context, as it is when using DETR or Deformable-DETR models, for example `facebook/detr-resnet-50`:

```console
before: 178
after: 188
gc: 8
```

What actually happens (using the RT-DETR model in the code snippet above):

```console
before: 81
after: 103
gc: 85
```

--- Comment by dhdaines at 2026-04-13T15:49:37Z ---
If I had to guess, I'd say this is most likely also a memory leak on CPU.  I'll take a look with a memory profiler...

--- Comment by dhdaines at 2026-04-13T16:25:47Z ---
Not certain how to best interpret these memory-profiler graphs (and wherever the memory is actually allocated is buried under so many hideous layers of abstraction that I have no hope of understanding anything) but it looks like on CPU we have similar behaviour - I stuck a `time.sleep(2)` after calling `gc.collect()` so that you can see more clearly how much actually gets freed:

Plain old DETR:

RT-DETRv2:

--- Comment by dhdaines at 2026-04-13T16:27:16Z ---
Maybe related to: https://github.com/lyuwenyu/RT-DETR/issues/93

--- Comment by dhdaines at 2026-04-13T17:34:34Z ---
This is almost certainly due to the weird way that `RTDetrV2ForObjectDetection` is implemented, as it seems to both inherit from `RTDetrV2PreTrainedModel` but also contain another instance of `RTDetrV2Model` (which inherits from `RTDetrV2PreTrainedModel`!!!) with which it shares various tensors.  In particular, this code seems like it might be creating a circular reference from `self.model.decoder` back to `self`? https://github.com/huggingface/transformers/blob/c6c8503869367af938666810e01a71866ca4fe93/src/transformers/models/rt_detr_v2/modular_rt_detr_v2.py#L453-L454

If we also delete `model.model.decoder` before deleting `model` in the code snippet, then we already see a slight improvement, **BUT** obviously there is a bunch of other memory leaking somewhere:

```console
before: 81
after: 104
gc: 73
```

--- Comment by yonigozlan at 2026-04-13T21:19:36Z ---
Hey @dhdaines ! Thanks for opening this issue and for the thorough analysis, I'll look into it this week :)

--- Comment by tchiayan at 2026-04-16T01:13:58Z ---
Probably the issue was caused by `lru_cache` https://github.com/huggingface/transformers/blob/main/src/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py#L968

I'm verified using JIT compile to bypass the `lru_cache`
```python
import gc
import torch
import requests

from PIL import Image
from transformers import AutoModelForObjectDetection, AutoImageProcessor

torch.cuda.memory._record_memory_history()

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

device = torch.device("cuda")
image_processor = AutoImageProcessor.from_pretrained("PekingU/rtdetr_v2_r18vd",
                                                     backend="pil")
model = AutoModelForObjectDetection.from_pretrained("PekingU/rtdetr_v2_r18vd").to(device)

# Compile the model
model = torch.compile(model)

inputs = image_processor(images=image, return_tensors="pt").to(device)

print("before: %d" % (torch.cuda.memory_allocated() / 1024 / 1024))
with torch.inference_mode():
    outputs = model(**inputs)
print("after: %d" % (torch.cuda.memory_allocated() / 1024 / 1024))

del inputs
del outputs
del model
del image_processor
gc.collect()
torch.cuda.empty_cache()
torch.cuda.synchronize()
print("gc: %d" % (torch.cuda.memory_allocated() / 1024 / 1024))

torch.cuda.memory._dump_snapshot("snapshot.pickle")
```

Output:
```
before: 81
after: 103
gc: 8
```

--- Comment by dhdaines at 2026-04-16T11:56:08Z ---
> Probably the issue was caused by `lru_cache` https://github.com/huggingface/transformers/blob/main/src/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py#L968
>

Oh yes, absolutely, `lru_cache` will obviously hold on to tensors forever (that's what it does...). Good catch!

--- Comment by github-actions[bot] at 2026-05-14T08:52:32Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #45406: transformers serve crashes with AttributeError: 'Gemma4Processor' object has no attribute '_tokenizer'
State: closed  |  Labels: bug
Author: asdat3  |  Created: 2026-04-13T13:57:05Z

### System Info

transformers version: 5.5.3
Python version: 3.12.3
PyTorch version: 2.11.0+cu130
Platform: Linux (Ubuntu 24.04)

### Who can help?

@ArthurZucker ? idk
I will open an PR once i find time. I "fixed" it locally for now.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

pip install transformers[serving]
transformers serve TrevorJS/gemma-4-31B-it-uncensored
Then send any chat completion request:
bashcurl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "TrevorJS/gemma-4-31B-it-uncensored", "messages": [{"role": "user", "content": "Hello"}]}'

Full Traceback
```
File ".../transformers/cli/serving/utils.py", line 565, in generate_streaming
    streamer = DirectStreamer(processor._tokenizer, loop, queue, skip_special_tokens=True)
                              ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Gemma4Processor' object has no attribute '_tokenizer'. Did you mean: 'tokenizer'?
```
The same error also occurs on line 664 for the continuous batching code path:
streamer = CBStreamer(self._cb, request_id, processor._tokenizer, loop, text_queue)

### Expected behavior

**Expected Behavior**
Chat completions should work without error. DirectStreamer and CBStreamer require the raw Rust tokenizers.Tokenizer object, which is available at processor.tokenizer._tokenizer for Gemma4Processor.

**Actual Behavior**
Every request crashes with AttributeError immediately after the model loads.

**Fix**
Two lines in src/transformers/cli/serving/utils.py need updating:
Line 565 (standard streaming path):
```
# Before
streamer = DirectStreamer(processor._tokenizer, loop, queue, skip_special_tokens=True)

# After
streamer = DirectStreamer(processor.tokenizer._tokenizer, loop, queue, skip_special_tokens=True)
```
Line 664 (continuous batching path):
```
# Before
streamer = CBStreamer(self._cb, request_id, processor._tokenizer, loop, text_queue)

# After
streamer = CBStreamer(self._cb, request_id, processor.tokenizer._tokenizer, loop, text_queue)
```
Alternatively, a more robust fix would be to make both call sites handle both cases:
pythonrust_tokenizer = getattr(processor, "_tokenizer", None) or processor.tokenizer._tokenizer
This would be defensive against other processor types that may also use the public .tokenizer attribute.

**Notes**

Affects all Gemma 4 models (google/gemma-4-*) since Gemma4Processor uses the public .tokenizer attribute rather than ._tokenizer
Gemma 4 support was added in v5.5.0 (#45192) and transformers serve was not tested against it
No existing GitHub issues or public reports found for this specific error as of April 2026 — this appears to be a gap in integration testing between the new serving CLI and Gemma4's processor design

--- Comment by zucchini-nlp at 2026-04-13T14:52:10Z ---
Fixed by https://github.com/huggingface/transformers/pull/45368 I believe

--- Comment by zucchini-nlp at 2026-04-13T15:33:43Z ---
Merged! Please check

--- Comment by asdat3 at 2026-04-13T15:40:32Z ---
oh, I think you are correct. Will check in a second if the merge "fixed it".
Thank you very much for your super fast answer and all your work.

--- Comment by asdat3 at 2026-04-13T15:42:10Z ---
working 🫡


ISSUE #45405: `MIN_PEFT_VERSION` bumped to 0.18.2 which is not yet released on PyPI
State: closed  |  Labels: bug
Author: artem-spector  |  Created: 2026-04-13T13:31:01Z

### System Info

- `transformers` version: 5.6.0.dev0 (post-commit c585eeaa65)
- Platform: Linux-5.14.0-570.12.1.el9_6.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.15
- Huggingface_hub version: 1.9.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.11.0+cu130 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@BenjaminBosworking with integrations/peft.py, relevant imports

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

 Commit c585eeaa65 ("Load adapter with TP") added:

  ```python
  from peft.utils.save_and_load import _maybe_shard_state_dict_for_tp
```

  and bumped MIN_PEFT_VERSION to "0.18.2" in src/transformers/integrations/peft.py.

  However, peft 0.18.2 has not been released on PyPI yet — the latest available version is 0.18.1.

  This means anyone running transformers from main who calls enable_adapters() or any peft integration path hits:

  ImportError: ... requires a minimum version of 0.18.2

  Installing peft from git (pip install git+https://github.com/huggingface/peft.git) gives 0.18.2.dev0, but this also fails the version check because PEP 440 orders 0.18.2.dev0 < 0.18.2.

### Expected behavior

 Either:
  1. Wait for peft 0.18.2 to be released before bumping MIN_PEFT_VERSION, or
  2. Use 0.18.2.dev0 as the minimum version so the git-installed dev version satisfies the check, or
  3. Gate the new _maybe_shard_state_dict_for_tp import behind a version check so the rest of the peft integration still works with 0.18.1

--- Comment by Rocketknight1 at 2026-04-13T13:40:22Z ---
cc @BenjaminBossan

--- Comment by BenjaminBossan at 2026-04-13T13:49:46Z ---
Yes, sorry for the inconvenience. PEFT v0.19.0 will be released this week.

--- Comment by artem-spector at 2026-04-13T13:59:14Z ---
ok, thanks!


ISSUE #45399: Fallback to kernels-community/flash-attn2 is blocked by other checks when fa2 is not installed
State: closed  |  Labels: bug
Author: efsotr  |  Created: 2026-04-13T10:20:19Z

### System Info

- `transformers` version: 5.5.3
- Platform: Linux-5.4.0-216-generic-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 1.10.1
- Safetensors version: 0.5.2
- Accelerate version: 1.6.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: False
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: False
        - tpu_use_cluster: False
        - tpu_use_sudo: False
- DeepSpeed version: 0.16.5
- PyTorch version (accelerator?): 2.7.0+cu126 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA L40

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("models/Llama-3.2-1B", torch_dtype="auto", attn_implementation="flash_attention_2", device_map="cpu")
```

```log
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[4], [line 2](vscode-notebook-cell:?execution_count=4&line=2)
      1 from transformers import AutoModelForCausalLM
----> [2](vscode-notebook-cell:?execution_count=4&line=2) model = AutoModelForCausalLM.from_pretrained("models/Llama-3.2-1B", torch_dtype="auto", attn_implementation="flash_attention_2", device_map="cpu")

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:387, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    385     if model_class.config_class == config.sub_configs.get("text_config", None):
    386         config = config.get_text_config()
--> [387](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:387)     return model_class.from_pretrained(
    388         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    389     )
    390 raise ValueError(
    391     f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    392     f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping)}."
    393 )

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:4092, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, weights_only, *model_args, **kwargs)
   4090 config = copy.deepcopy(config)  # We do not want to modify the config inplace in from_pretrained.
   4091 with ContextManagers(model_init_context):
-> [4092](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:4092)     model = cls(config, *model_args, **model_kwargs)
   4093     patch_output_recorders(model)
   4095     if hf_quantizer is not None:  # replace module with quantized modules (does not touch weights)

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:435, in LlamaForCausalLM.__init__(self, config)
    434 def __init__(self, config):
--> [435](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py:435)     super().__init__(config)
    436     self.model = LlamaModel(config)
    437     self.vocab_size = config.vocab_size

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1255, in PreTrainedModel.__init__(self, config, *inputs, **kwargs)
   1251 self.name_or_path = config.name_or_path
   1253 # Check the attention implementation is supported, or set it if not yet set (on the internal attr, to avoid
   1254 # setting it recursively)
-> [1255](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1255) self.config._attn_implementation_internal = self._check_and_adjust_attn_implementation(
   1256     self.config._attn_implementation,
   1257     is_init_check=True,
   1258     # We need to use this constant that is set through context manager as it cannot be forwarded in the model's __init__
   1259     allow_all_kernels=hub_kernels.ALLOW_ALL_KERNELS,
   1260 )
   1261 # Check the experts implementation is supported, or set it if not yet set (on the internal attr, to avoid
   1262 # setting it recursively)
   1263 self.config._experts_implementation_internal = self._check_and_adjust_experts_implementation(
   1264     self.config._experts_implementation
   1265 )

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1865, in PreTrainedModel._check_and_adjust_attn_implementation(self, attn_implementation, is_init_check, allow_all_kernels)
   1863         raise e
   1864 else:
-> [1865](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1865)     applicable_attn_implementation = self.get_correct_attn_implementation(
   1866         applicable_attn_implementation, is_init_check
   1867     )
   1869     # preload flash attention here to allow compile with fullgraph
   1870     if is_flash_attention_requested(requested_attention_implementation=applicable_attn_implementation):

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1912, in PreTrainedModel.get_correct_attn_implementation(self, requested_attention, is_init_check)
   1908 if is_flash_attention_requested(requested_attention_implementation=applicable_attention) and (
   1909     fa_matched := re.search(r"^flash_attention_(\d)$", applicable_attention)
   1910 ):
   1911     fa_version = int(fa_matched.group(1))  # last digit
-> [1912](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1912)     self._flash_attn_can_dispatch(flash_attn_version=fa_version, is_init_check=is_init_check)
   1913 elif "flex_attention" in applicable_attention:
   1914     self._flex_attn_can_dispatch(is_init_check)

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1647, in PreTrainedModel._flash_attn_can_dispatch(self, flash_attn_version, is_init_check)
   1644     raise ValueError(f"Requested Flash Attention {flash_attn_version} which is not supported.")
   1646 # Check if we can even use the FA version based on the env of the user
-> [1647](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1647) self._flash_attn_import_error(**FLASH_ATTENTION_COMPATIBILITY_MATRIX[flash_attn_version])
   1649 # Check for attention dropout, which is incompatible with newer FA versions
   1650 # (many should not really care about dropout as it is not super effective, hence warning for now)
   1651 if flash_attn_version > 2:

File ~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1602, in PreTrainedModel._flash_attn_import_error(self, flash_attn_version, general_availability_check, pkg_availability_check, supported_devices, custom_supported_devices, cuda_min_major_version)
   1600 # Can the package be seen in the import structure
   1601 if not pkg_availability_check():
-> [1602](https://vscode-remote+ssh-002dremote-002bserver35.vscode-resource.vscode-cdn.net/home/linli/Quant/Ours/~/anaconda3/envs/LLM/lib/python3.10/site-packages/transformers/modeling_utils.py:1602)     raise ImportError(
   1603         f"{preface} the package for FlashAttention{flash_attn_version} doesn't seem to be installed."
   1604     )
   1605 # Minimum version (FA2 only)
   1606 elif flash_attn_version == 2 and not is_flash_attn_greater_or_equal("2.3.3"):

ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package for FlashAttention2 doesn't seem to be installed.
```

### Expected behavior

If fa2 is not installed, the code should fall back to `kernels-community/flash-attn2` successfully.

--- Comment by vasqu at 2026-04-13T19:04:04Z ---
Hmm, checked locally on latest commit and it works with the fallback. Can you
- Update kernels version to 0.12+
- Update torch version to 2.10+
- Can you check whether this works:
```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="kernels-community/flash-attn2", device_map="cpu")
```
These can be checked individually. We do raise a normal error of FA2 not being available, so it might be a kernels issue not properly loading / something going wrong.

(Btw, flash attn on cpu should not work either way.)