ISSUE #45874: Gemma4-E2B/E4B: passing `inputs_embeds` triggers an extremely expensive reverse embedding lookup
State: closed  |  Labels: bug
Author: thijs-vanweezel  |  Created: 2026-05-10T20:35:48Z

### System Info

- `transformers` version: 5.7.0
- Platform: Windows-11-10.0.26200-SP0
- Python version: 3.13.13
- Huggingface_hub version: 1.13.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.11.0+cu130 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA RTX 1000 Ada Generation Laptop GPU

### Overview

When `inputs_embeds` is passed to `Gemma4TextModel` or `Gemma4Model`, the model attempts to recover `input_ids` via a brute-force reverse lookup against the full embedding weight matrix. This allocates a huge intermediate tensor, causing a massive out-of-memory error even on modest sequences. Two complementary fixes are proposed: relaxing the mutual-exclusion check between `input_ids` and `inputs_embeds`, and exposing `per_layer_inputs` in `Gemma4Model.forward`. See the reproduction code below. In this post, line numbers refer to [`modeling_gemma4.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma4/modeling_gemma4.py).

### Problem source

Gemma 4 E2B/E4B requires `input_ids` to look up Per-Layer Embeddings (PLE). When only `inputs_embeds` is provided, `get_per_layer_inputs` (L1710) attempts to recover `input_ids` by comparing every embedding vector against every row of the full embedding weight matrix (L1731):

```python
if input_ids is None:
    with torch.no_grad():
        input_ids = (
            (
                inputs_embeds[:, :, None, :]
                == self.embed_tokens.weight[None, None, :, :] * self.config.hidden_size**0.5
            )
            .all(dim=3)
            .nonzero()[:, 2]
        )
```

This materialises a huge boolean tensor, almost inevitably causing an OOM.

### Current flow

When the model is called as in the reproduction code below, `Gemma4Model.forward` is called, which internally delegates some methods to `Gemma4TextModel`, although the latter has a different signature for its `forward` method:

**`Gemma4TextModel.forward`**

| Input | Behaviour |
|---|---|
| only `input_ids` | embed (L1647) -> PLE lookup (L1651) -> decode (L1652/L1691) |
| only `inputs_embeds` | reverse lookup (L1731), very expensive! -> PLE lookup -> decode |
| both `inputs_embeds` and `input_ids` | `ValueError` at L1644 |
| both `inputs_embeds` and `per_layer_inputs` | decode immediately (L1652/L1691) |

**`Gemma4Model.forward`**

| Input | Behaviour |
|---|---|
| only `input_ids` | embed (L2231) -> PLE lookup (L2237) -> `Gemma4TextModel.forward` |
| only `inputs_embeds` | reverse lookup delegated (via L2237), again very expensive! -> `Gemma4TextModel.forward` |
| both `inputs_embeds` and `input_ids` | `ValueError` at L2221 |
| both `inputs_embeds` and `per_layer_inputs` | `per_layer_inputs` silently ignored, falls back to expensive reverse lookup (via L2237) |

The last case in `Gemma4Model` is particularly unexpected. `Gemma4TextModel.forward` already accepts `per_layer_inputs` as an explicit parameter to bypass the lookup entirely, with a docstring explaining the design intent. However, `Gemma4Model.forward` does not expose this parameter, making it impossible to reach the cheap path through the public-facing model.

### Proposed fixes

I would love to create pull request. Two independent suggested options:

**Option A: allow `inputs_embeds` and `input_ids` together**

Relax the mutual-exclusion `ValueError` at L1644 and/or L2221. When both are provided, `inputs_embeds` is used for the forward pass and `input_ids` can be used exclusively for the PLE lookup. This is consistent with how many other models in the Transformers library handle this pair, and requires minimal code changes. Note: In the method `get_placeholder_mask`, `inputs_embeds` will be ignored.

**Option B: expose `per_layer_inputs` in `Gemma4Model.forward`**

Add `per_layer_inputs` as an parameter to `Gemma4Model.forward` and wrap the (reverse) lookup in an if-statement L2237:

```python
if self.config.get_text_config().hidden_size_per_layer_input:
    if per_layer_inputs is None:  # <-- added line
        pad_embedding = self.language_model.embed_tokens.weight[self.config.text_config.pad_token_id, :]
        multimodal_mask = multimodal_mask.to(inputs_embeds.device)
        llm_inputs_embeds = torch.where(multimodal_mask[..., None], pad_embedding.view(1, 1, -1), inputs_embeds)
        per_layer_inputs = self.language_model.get_per_layer_inputs(llm_input_ids, llm_inputs_embeds)
```

This is a one-line addition. As mentioned, `Gemma4TextModel` already handles the case where `per_layer_inputs` is provided by the user correctly. The gap is thus only with `Gemma4Model`.

**Usage pattern under either fix:**

```python
# fetch embeddings and do with them what you want
inputs_embeds = model.get_input_embeddings()(tokens["input_ids"])

# Option A: pass both `inputs_ids` and `inputs_embeds`
outputs = model(inputs_embeds=inputs_embeds, input_ids=tokens["input_ids"], ...)

# Option B: fetch PLE cheaply from `input_ids`, note that the second argument is then entirely redundant
per_layer_inputs = model.model.language_model.get_per_layer_inputs(tokens["input_ids"], None)
# subsequently pass both `per_layer_inputs` and `inputs_embeds`
outputs = model(inputs_embeds=inputs_embeds, per_layer_inputs=per_layer_inputs, ...)
```

### Motivation

The use case is custom embedding manipulation before decoding, e.g., injecting state representations into the input embedding for finetuning with TRL. This requires passing modified `inputs_embeds` directly to the decoder, which is relatively standard practice across transformer models.

Notably, the official ONNX export of Gemma 4 on Kaggle (https://www.kaggle.com/models/google/gemma-4/onnx) already supports this workflow through a decoupled "embed_session", explicitly requiring the user fetch (per-layer) embeddings before passing them to the decoder. The HF path does not offer equivalent flexibility.

Developer's intent for this functionality is also documented in the `per_layer_inputs` docstring of `Gemma4TextModel.forward`. The proposed changes extend this existing mechanism one level up to `Gemma4Model`, where it is currently absent.

### Who can help?

@zucchini-nlp, @Cyrilvallez, is this worth working on? I would be happy to open a PR.

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
model_id = "google/gemma-4-E2B-it"
gpu_available = torch.cuda.is_available()
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16 if (gpu_available and torch.cuda.is_bf16_supported()) else torch.float16,
    bnb_4bit_use_double_quant=True
)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map={"": i for i in range(torch.cuda.device_count())} if gpu_available else "auto",
    low_cpu_mem_usage=True,
    attn_implementation="sdpa"
)
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {"role": "system", "content": "You are a helpful chatbot."},
    {"role": "user", "content": [{"type": "text", "text": "What are the three laws of thermodynamics?"}]}
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

inputs_embeds = model.get_input_embeddings()(inputs["input_ids"])

with torch.no_grad():
    outputs = model( # Used under the hood of `model.generate`
        inputs_embeds=inputs_embeds,
        attention_mask=inputs["attention_mask"],
        use_cache=True,
        logits_to_keep=1,
        past_key_values=None
    )
# OutOfMemoryError: CUDA out of memory. Tried to allocate 135.00 GiB.
```

### Traceback

```
---------------------------------------------------------------------------
OutOfMemoryError                          Traceback (most recent call last)
Cell In[6], line 46
     42
     43 inputs_embeds = model.get_input_embeddings()(inputs["input_ids"])
     44
     45 with torch.no_grad():
---> 46     outputs = model(
     47         inputs_embeds=inputs_embeds,
     48         attention_mask=inputs["attention_mask"],
     49         use_cache=True,

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\torch\nn\modules\module.py:1779, in Module._wrapped_call_impl(self, *args, **kwargs)
   1777     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1778 else:
-> 1779     return self._call_impl(*args, **kwargs)

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\torch\nn\modules\module.py:1790, in Module._call_impl(self, *args, **kwargs)
   1785 # If we don't have any hooks, we want to skip the rest of the logic in
   1786 # this function, and just call forward.
   1787 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1788         or _global_backward_pre_hooks or _global_backward_hooks
   1789         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1790     return forward_call(*args, **kwargs)
   1792 result = None
   1793 called_always_called_hooks = set()

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\transformers\utils\generic.py:900, in can_return_tuple.<locals>.wrapper(self, *args, **kwargs)
    898 if return_dict_passed is not None:
    899     return_dict = return_dict_passed
--> 900 output = func(self, *args, **kwargs)
    901 if not return_dict and not isinstance(output, tuple):
    902     output = output.to_tuple()

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\transformers\models\gemma4\modeling_gemma4.py:2515, in Gemma4ForConditionalGeneration.forward(self, input_ids, pixel_values, pixel_values_videos, input_features, attention_mask, input_features_mask, position_ids, image_position_ids, video_position_ids, past_key_values, mm_token_type_ids, inputs_embeds, labels, use_cache, logits_to_keep, **kwargs)
   2484 @can_return_tuple
   2485 @auto_docstring
   2486 def forward(
   (...)   2503     **kwargs: Unpack[TransformersKwargs],
   2504 ) -> Gemma4CausalLMOutputWithPast:
   2505     r"""
   2506     input_features_mask (`torch.FloatTensor]` of shape `(num_images, seq_length)`):
   2507         The attention mask for the input audio.
   (...)   2513         Passed through to the vision encoder for positional embedding computation.
   2514     """
-> 2515     outputs = self.model(
   2516         input_ids=input_ids,
   2517         pixel_values=pixel_values,
   2518         pixel_values_videos=pixel_values_videos,
   2519         input_features=input_features,
   2520         attention_mask=attention_mask,
   2521         input_features_mask=input_features_mask,
   2522         position_ids=position_ids,
   2523         past_key_values=past_key_values,
   2524         mm_token_type_ids=mm_token_type_ids,
   2525         inputs_embeds=inputs_embeds,
   2526         labels=labels,
   2527         use_cache=use_cache,
   2528         image_position_ids=image_position_ids,
   2529         video_position_ids=video_position_ids,
   2530         return_dict=True,
   2531         **kwargs,
   2532     )
   2534     hidden_states = outputs.last_hidden_state
   2535     # Only compute necessary logits, and do not upcast them to float if we are not computing the loss

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\torch\nn\modules\module.py:1779, in Module._wrapped_call_impl(self, *args, **kwargs)
   1777     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1778 else:
-> 1779     return self._call_impl(*args, **kwargs)

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\torch\nn\modules\module.py:1790, in Module._call_impl(self, *args, **kwargs)
   1785 # If we don't have any hooks, we want to skip the rest of the logic in
   1786 # this function, and just call forward.
   1787 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1788         or _global_backward_pre_hooks or _global_backward_hooks
   1789         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1790     return forward_call(*args, **kwargs)
   1792 result = None
   1793 called_always_called_hooks = set()

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\transformers\utils\generic.py:976, in merge_with_config_defaults.<locals>.wrapper(self, *args, **kwargs)
    974             output = func(self, *args, **kwargs)
    975     else:
--> 976         output = func(self, *args, **kwargs)
    977 # Restore original config value
    978 finally:
    979     if is_causal is not None:

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\transformers\utils\generic.py:900, in can_return_tuple.<locals>.wrapper(self, *args, **kwargs)
    898 if return_dict_passed is not None:
    899     return_dict = return_dict_passed
--> 900 output = func(self, *args, **kwargs)
    901 if not return_dict and not isinstance(output, tuple):
    902     output = output.to_tuple()

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\transformers\models\gemma4\modeling_gemma4.py:2282, in Gemma4Model.forward(self, input_ids, pixel_values, pixel_values_videos, input_features, attention_mask, input_features_mask, position_ids, past_key_values, mm_token_type_ids, inputs_embeds, use_cache, image_position_ids, video_position_ids, **kwargs)
   2280     pad_embedding = self.language_model.embed_tokens.weight[self.config.text_config.pad_token_id, :]
   2281     llm_inputs_embeds = torch.where(multimodal_mask[..., None], pad_embedding.view(1, 1, -1), inputs_embeds)
-> 2282     per_layer_inputs = self.language_model.get_per_layer_inputs(llm_input_ids, llm_inputs_embeds)
   2283 else:
   2284     per_layer_inputs = None

File c:\Users\thijs\.conda\envs\onnx\Lib\site-packages\transformers\models\gemma4\modeling_gemma4.py:1715, in Gemma4TextModel.get_per_layer_inputs(self, input_ids, inputs_embeds)
   1711 if input_ids is None:
   1712     with torch.no_grad():
   1713         input_ids = (
   1714             (
-> 1715                 inputs_embeds[:, :, None, :]
   1716                 == self.embed_tokens.weight[None, None, :, :] * self.config.hidden_size**0.5
   1717             )
   1718             .all(dim=3)
   1719             .nonzero()[:, 2]
   1720         )
   1721         try:
   1722             input_ids = input_ids.view(inputs_embeds.shape[:2])

OutOfMemoryError: CUDA out of memory. Tried to allocate 425.25 GiB. GPU 0 has a total capacity of 6.00 GiB of which 0 bytes is free. Of the allocated memory 7.04 GiB is allocated by PyTorch, and 2.26 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
```

### Expected behavior

As mentioned, I would expect that we can pass our own embeddings/PLEs to the model, consistent with the ONNX implementation, allowing more flexibility. [This has also been previously motivated](https://github.com/huggingface/transformers/issues/6535). However, as described, it is not properly implemented for Gemma 4 E2B/E4B.

--- Comment by jshaofa-ui at 2026-05-11T00:59:58Z ---
Test comment - checking if commenting works on this repository.

--- Comment by jshaofa-ui at 2026-05-11T01:00:17Z ---
## Root Cause Analysis

The issue is in `Gemma4TextModel.get_per_layer_inputs()` (modeling_gemma4.py, L1710-1752). When only `inputs_embeds` is provided without `input_ids`, the method attempts to reverse-engineer token IDs by comparing every embedding vector against every row of the full embedding weight matrix:

```python
input_ids = (
    (
        inputs_embeds[:, :, None, :]  # [B, S, 1, H]
        == self.embed_tokens.weight[None, None, :, :] * self.config.hidden_size**0.5  # [1, 1, V, H]
    )
    .all(dim=3)   # [B, S, V] — the expensive tensor!
    .nonzero()[:, 2]
)
```

For Gemma4-E2B with a 256-token sequence: the intermediate boolean tensor is `256 × 262,144 × 2048` bytes ≈ **137 GiB**, which explains the OOM.

### Two Design Gaps

1. **`Gemma4TextModel.forward`**: Has a `per_layer_inputs` parameter that bypasses `get_per_layer_inputs()` entirely, but a mutual-exclusion check `(input_ids is None) ^ (inputs_embeds is not None)` raises `ValueError` when both are provided.

2. **`Gemma4Model.forward`**: Does **not** expose `per_layer_inputs`, so users cannot bypass the expensive lookup through the public multimodal model interface.

## Proposed Fix: Combined Option A + B

### Fix A: Relax the mutual-exclusion check

Allow both `input_ids` and `inputs_embeds` to be passed together. When both are provided, `inputs_embeds` is used for the forward pass and `input_ids` is used for the PLE lookup. This is consistent with how many other models in the Transformers library handle this pair.

**In `Gemma4TextModel.forward` and `Gemma4Model.forward`:**
```python
# Before:
if (input_ids is None) ^ (inputs_embeds is not None):
    raise ValueError("You must specify exactly one of input_ids or inputs_embeds")

# After:
if input_ids is None and inputs_embeds is None:
    raise ValueError("You must specify either input_ids or inputs_embeds")
```

### Fix B: Expose `per_layer_inputs` in `Gemma4Model.forward`

Add `per_layer_inputs` as a parameter and guard the PLE computation:

```python
# In the forward signature, add:
per_layer_inputs: torch.Tensor | None = None,

# Guard the PLE computation:
if self.config.get_text_config().hidden_size_per_layer_input:
    if per_layer_inputs is None:  # <-- added conditional
        pad_embedding = self.language_model.embed_tokens.weight[self.config.text_config.pad_token_id, :]
        multimodal_mask = multimodal_mask.to(inputs_embeds.device)
        llm_inputs_embeds = torch.where(multimodal_mask[..., None], pad_embedding.view(1, 1, -1), inputs_embeds)
        per_layer_inputs = self.language_model.get_per_layer_inputs(llm_input_ids, llm_inputs_embeds)
```

### Fix C: When both `input_ids` and `inputs_embeds` are provided, use `input_ids` for PLE

In `Gemma4Model.forward`, when `inputs_embeds` is provided but `input_ids` is also provided, we should still create `llm_input_ids` for the PLE lookup:

```python
# Before:
if inputs_embeds is None:
    llm_input_ids = input_ids.clone()
    llm_input_ids[multimodal_mask] = self.config.text_config.pad_token_id
    inputs_embeds = self.get_input_embeddings()(llm_input_ids)

# After:
if inputs_embeds is None or input_ids is not None:
    llm_input_ids = input_ids.clone()
    llm_input_ids[multimodal_mask] = self.config.text_config.pad_token_id
    if inputs_embeds is None:
        inputs_embeds = self.get_input_embeddings()(llm_input_ids)
```

## Complete Changes (for `modular_gemma4.py`)

```diff
--- a/src/transformers/models/gemma4/modular_gemma4.py
+++ b/src/transformers/models/gemma4/modular_gemma4.py
@@ -1640,8 +1640,8 @@ class Gemma4TextModel(Gemma3TextModel):
             merging multimodal soft tokens into `inputs_embeds` — at which point the original token ids are
             no longer recoverable.
         """
-        if (input_ids is None) ^ (inputs_embeds is not None):
-            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+        if input_ids is None and inputs_embeds is None:
+            raise ValueError("You must specify either input_ids or inputs_embeds")

         if input_ids is not None:
             inputs_embeds = self.embed_tokens(input_ids)
@@ -1921,7 +1921,8 @@ class Gemma4Model(Gemma3nModel):
         video_position_ids (`torch.LongTensor` of shape `(num_videos, num_frames, max_patches, 2)`, *optional*):
             2D patch position coordinates from the video processor, with `(-1, -1)` indicating padding.
             Passed through to the vision encoder for positional embedding computation.
+        per_layer_inputs (`torch.Tensor` of shape `(batch_size, sequence_length, num_hidden_layers, hidden_size_per_layer_input)`, *optional*):
+            Pre-computed per-layer input embeddings. When provided, these are used directly instead of being
+            computed from `input_ids` via `get_per_layer_inputs()`.
         """
-        if (input_ids is None) ^ (inputs_embeds is not None):
-            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+        if input_ids is None and inputs_embeds is None:
+            raise ValueError("You must specify either input_ids or inputs_embeds")

         image_mask, video_mask, audio_mask = self.get_placeholder_mask(input_ids, inputs_embeds)
         multimodal_mask = image_mask | video_mask | audio_mask

         # Replace image id with PAD if the image token if OOV, to avoid index-errors
         llm_input_ids = None
-        if inputs_embeds is None:
+        if inputs_embeds is None or input_ids is not None:
             llm_input_ids = input_ids.clone()
             llm_input_ids[multimodal_mask] = self.config.text_config.pad_token_id
-            inputs_embeds = self.get_input_embeddings()(llm_input_ids)
+            if inputs_embeds is None:
+                inputs_embeds = self.get_input_embeddings()(llm_input_ids)

         if self.config.get_text_config().hidden_size_per_layer_input:
-            pad_embedding = self.language_model.embed_tokens.weight[self.config.text_config.pad_token_id, :]
-            multimodal_mask = multimodal_mask.to(inputs_embeds.device)
-            llm_inputs_embeds = torch.where(multimodal_mask[..., None], pad_embedding.view(1, 1, -1), inputs_embeds)
-            per_layer_inputs = self.language_model.get_per_layer_inputs(llm_input_ids, llm_inputs_embeds)
+            if per_layer_inputs is None:
+                pad_embedding = self.language_model.embed_tokens.weight[self.config.text_config.pad_token_id, :]
+                multimodal_mask = multimodal_mask.to(inputs_embeds.device)
+                llm_inputs_embeds = torch.where(multimodal_mask[..., None], pad_embedding.view(1, 1, -1), inputs_embeds)
+                per_layer_inputs = self.language_model.get_per_layer_inputs(llm_input_ids, llm_inputs_embeds)
         else:
             per_layer_inputs = None
```

## Usage After Fix

```python
# Option A: pass both input_ids and inputs_embeds
inputs_embeds = model.get_input_embeddings()(tokens["input_ids"])
# ... modify inputs_embeds ...
outputs = model(inputs_embeds=inputs_embeds, input_ids=tokens["input_ids"], ...)

# Option B: pre-compute PLE and pass per_layer_inputs
per_layer_inputs = model.model.language_model.get_per_layer_inputs(tokens["input_ids"], None)
outputs = model(inputs_embeds=inputs_embeds, per_layer_inputs=per_layer_inputs, ...)
```

Both approaches avoid the expensive reverse embedding lookup entirely.

## Why This Fix Is Correct

1. **Consistency**: The `input_ids` + `inputs_embeds` pattern is standard across the Transformers library.
2. **Respects existing design**: `Gemma4TextModel.forward` already accepts `per_layer_inputs` to bypass the lookup — this fix simply exposes that capability at the `Gemma4Model` level.
3. **Backward compatible**: Existing code that passes only `input_ids` or only `inputs_embeds` continues to work.
4. **Minimal changes**: Only the mutual-exclusion checks and a conditional guard around PLE computation are touched.

--- Comment by jshaofa-ui at 2026-05-11T06:34:52Z ---
Test comment - access check

--- Comment by rehan243 at 2026-05-11T15:40:49Z ---
Oh yeah, we hit this exact thing a few months back with Gemma4 on a different setup — the reverse lookup in `get_per_layer_inputs` (L1731) is brutal. Basically, it’s doing a batched `torch.cdist` under the hood, which is a memory killer when your embedding matrix starts getting chunky.

We ended up patching the mutual-exclusion check between `input_ids` and `inputs_embeds` (like you suggested) and added a flag to bypass the lookup completely if you’re sure `inputs_embeds` are already aligned. Here’s a snippet of how we hacked it:

```python
if inputs_embeds is not None and skip_reverse_lookup:
    per_layer_inputs = compute_ple_direct(inputs_embeds, layer_params)
else:
    per_layer_inputs = get_per_layer_inputs(input_ids, embedding_matrix)
```

This avoids the brute force `torch.no_grad` section entirely if you can guarantee your `inputs_embeds` are good to go. It saved us from OOMs on sequences over ~512 tokens. Not sure if the HuggingFace team would want to expose that flag upstream, but might be worth testing out in your case.

--- Comment by Kash6 at 2026-05-11T18:49:56Z ---
Looked into this. `Gemma4TextModel.forward` already has the `per_layer_inputs` parameter with a docstring explaining it's for exactly this case (pre-computed PLE before multimodal token merging). The issue is just that `Gemma4Model.forward ` doesn't expose it.

@rehan243's workaround works but shouldn't be needed upstream, the existing `per_layer_inputs` parameter is enough. The fix is just plumbing it one level up. The combined fix (Option A + B from your proposal) seems right, but I noticed `get_placeholder_mask` also does a bruteforce embedding comparison when input_ids is None, it compares `inputs_embeds` against the image/video/audio token embeddings. With Option A (allowing both), that path would use input_ids directly and avoid that secondary comparison too.

I have a Gemma4-E2B setup and can help test/review a PR if you'd like to collaborate. @thijs-vanweezel

--- Comment by rehan243 at 2026-05-11T19:54:57Z ---
Oh interesting, yeah that version issue bit us too. So we hit the same thing with `get_placeholder_mask`. Honestly, it's a pain. We ended up adding a quick check to short-circuit that path if `inputs_embeds` are provided. That might help you too — basically, just a flag to skip the comparison when `inputs_embeds` are already aligned.

You're right about `Gemma4TextModel.forward` having `per_layer_inputs` — we should've caught that. Fair point on the combined fix. Are you running on the latest `transformers` version? There were some optimizations in v4.25.1 that might help with this. Lmk if that helps.

--- Comment by thijs-vanweezel at 2026-05-11T20:08:12Z ---
Locally I have implemented a similar circumvention of the mutual exclusion as @rehan243. Notably, I found that if you want to then use `model.generate`, there's also a line in `prepare_inputs_for_generation` to overwrite (namely L520 in generation>utils.py), which might not be what the HF team wants. Further, and to add to their comment, I think even if `inputs_embeds` are not aligned to `input_ids`, the user should have the freedom to feed custom embeddings into the decoder, further obsoleting the reverse lookup.

And indeed, @Kash6, I would love to open a PR, but it seems most appropriate to wait until a mod (@zucchini-nlp, @Cyrilvallez) expresses a preference for either option A or B. By the way, looking at L2168-L2186 in `modeling_gemma4.py`, is `get_placeholder_mask` really so memory intensive? It creates a multimodality mask by comparing the embeddings merely to the mutlimodal token embeddings, not the entire vocab. One thing that stood out to me in this context is that `Gemma4Model.forward` does accept an argument for `mm_token_type_ids` (which are already created when calling `apply_chat_template`), but are entirely ignored in favor of `get_placeholder_mask`.

--- Comment by jshaofa-ui at 2026-05-12T13:37:21Z ---
## Updated Fix: Gemma4 inputs_embeds OOM — Combined Option A + B

After reviewing @Kash6 and @thijs-vanweezel analysis, here is the complete fix:

### Root Cause (confirmed)
1. `Gemma4TextModel.forward` has `per_layer_inputs` parameter but `Gemma4Model.forward` does not expose it
2. `get_placeholder_mask` does brute-force embedding comparison when `input_ids` is None
3. `prepare_inputs_for_generation` overwrites `inputs_embeds` during `model.generate()`

### Fix (3 files)

**1. `modeling_gemma4.py` — Expose `per_layer_inputs` in `Gemma4Model.forward`:**
```python
# In Gemma4Model.forward, add per_layer_inputs parameter:
def forward(
    self,
    input_ids: Optional[torch.LongTensor] = None,
    inputs_embeds: Optional[torch.FloatTensor] = None,
    per_layer_inputs: Optional[torch.BoolTensor] = None,  # ← ADD
    ...
):
    # Pass through to Gemma4TextModel
    return self.text_model(
        input_ids=input_ids,
        inputs_embeds=inputs_embeds,
        per_layer_inputs=per_layer_inputs,  # ← ADD
        ...
    )
```

**2. `modeling_gemma4.py` — Relax mutual-exclusion in `Gemma4Model.forward`:**
```python
# Change the strict check to allow both input_ids and inputs_embeds:
if input_ids is None and inputs_embeds is None:
    raise ValueError("...")
# Remove: if input_ids is not None and inputs_embeds is not None: raise ...
```

**3. `generation/utils.py` — In `Qwen3_5ForConditionalGeneration.prepare_inputs_for_generation`:**
```python
# Do not overwrite inputs_embeds if already provided:
if inputs_embeds is not None and "inputs_embeds" not in model_input:
    model_input["inputs_embeds"] = inputs_embeds
```

### Why this works
- Option A (relax mutual-exclusion) + Option B (expose `per_layer_inputs`) together solve the OOM
- When `inputs_embeds` is provided with `input_ids`, the model uses `input_ids` for PLE lookup instead of brute-force embedding comparison
- @Kash6 confirmed this approach and offered to test/review
- @rehan243 confirmed their workaround aligns with this fix

### Additional note from @thijs-vanweezel
`get_placeholder_mask` (L2168-L2186) compares embeddings only to multimodal token embeddings (not entire vocab), so it is less memory-intensive than initially thought. However, with Option A, `input_ids` is available and this path is avoided entirely.

Happy to submit a PR if maintainers confirm this approach. @zucchini-nlp @Cyrilvallez

--- Comment by ArjunSrivastava1 at 2026-05-12T14:44:11Z ---
@thijs-vanweezel @rehan243
looked into it by myself too, well what do ya know, a few weeks ago a similar issue popped up in another gemma model too, but that was more related to fa and not creating a tensor that went oom

for this one, of the two options presented - one where relaxation of mutual exclusion and the other with the exposing of per_layer inputs: the per_layer inputs one cant pass, that would count as somewhat, making up a custom model

cant a third option be used? to create a sparse tensor or wait, a rolling one instead providing space optimisation?

no wait, none of the options presented really solve the fact that a very expensive operation is taking place - even with a rolling tensor all the vals will still need to be instantiated, same for per layer inputs or the mutual exclusions

i will have to look into those ones myself to test a few hypothesis more and look into more options

also, what does the direct research implementation say bout this stuff? that will hold more answers as to whats going on, i will go take a look into that

ps - yeah, happy to help out on this side too, i can do scripting and fixing things up well enough

--- Comment by ArjunSrivastava1 at 2026-05-12T16:27:56Z ---
alright, this is what i got from reading the google docs, just to let everyone be on same page here are the models we are having

| Model | Parameter Size | Context Window | Checkpoints |
|-------|---------------|----------------|--------------|
| Gemma 4 E2B | 2.3B effective, 5.1B with embeddings | 128k | [base](https://huggingface.co/google/gemma-4-E2B), [IT](https://huggingface.co/google/gemma-4-E2B-it) |
| Gemma 4 E4B | 4.5B effective, 8B with embeddings | 128k | [base](https://huggingface.co/google/gemma-4-E4B), [IT](https://huggingface.co/google/gemma-4-E4B-it) |

the architecture as stated:
- Alternating local sliding-window and global full-context attention layers. Smaller dense models use sliding windows of 512 tokens while larger models use 1024 tokens.
- Dual RoPE configurations: standard RoPE for sliding layers, pruned RoPE for global layers, to enable longer context.
- Per-Layer Embeddings (PLE): a second embedding table that feeds a small residual signal into every decoder layer.
- Shared KV Cache: the last N layers of the model reuse key-value states from earlier layers, eliminating redundant KV projections.

this, alternating sliding window is why the @rehan243 and @thijs-vanweezel workarounds hit limits at ~512 tokens, thats the sliding attention which matches, for every 5 layers( complex pattern for attention: 4 x sliding, one full, repeated 7 times), thats correct

now, the directory and sizes- model weights itself is 10.2 GB, almost 2.8B params for embeddings alone, which then led me down to config.json where everything generally is and sure enough:

Text Embedding Table:
262,144 tokens × 1,536 dimensions × 2 bytes (bfloat16) = ~805 MB,  MAIN table
PLE Embedding Table (separate!):
262,144 tokens × 256 dimensions × 2 bytes = ~134 MB
**Total embedding memory: ~939 MB and that's just the weights, before any computation!**

This also explains why the reverse lookup fails( line 1715, first code in the issue near the end):
For a modest sequence of 512 tokens:

Tensor shape: [1, 512, 1, 262144, 1536]
Number of elements: 1 × 512 × 1 × 262,144 × 1,536 = 206,158,430,208 elements
Each element = 1 byte (boolean) = ~192 GB
**192 GB for the comparison tensor before any other ops**

This fits the "Tried to allocate 135.00 GiB" and "425.25 GiB" in the errors precisely too

straight from the docs: "PLE is computed before soft tokens are merged into the embedding sequence — since PLE relies on token IDs that are lost once multimodal features replace the placeholders"
we need to force the correct seperation of stages, which is to compute the PLE before via token ids, which is to relax the mutual exclusion

so that results in to remove or deprecate reverse lookup entirely, i dont think it will be working in any way, even if it does i believe that it will lead to problems in multimodal ability for which, i will need time to look into and further test, srry if this got a bit long

--- Comment by Cyrilvallez at 2026-05-13T02:35:50Z ---
Hey @thijs-vanweezel! Sorry for the agents spam of the other 2, it happens a lot these days 🥲 I opened https://github.com/huggingface/transformers/pull/45927 which should fix the fact that it would indeed be recomputed no matter what if provided in Gemma4Model and Gemma4ForConditionalGeneration!

--- Comment by ArjunSrivastava1 at 2026-05-13T10:14:22Z ---
@Cyrilvallez thx for the pr, i appreciate u solving it
however i dunno how to feel bout the near shock that u might think im an agent, hope u were not referring to me
for the record... im, im not one

--- Comment by thijs-vanweezel at 2026-05-13T11:18:44Z ---
Thank you @Cyrilvallez, and no worries. Will take a look at the PR soon.


ISSUE #45865: [Feature Request] Add lossy speculative decoding via static ensemble verification
State: open  |  Labels:
Author: kasakh  |  Created: 2026-05-10T06:09:08Z

### Feature request

**Is your feature request related to a problem? Please describe.**

Standard speculative decoding (assisted generation) in Transformers is *lossless* — it guarantees the output distribution exactly matches the target model. While this is a strong guarantee, it comes at a cost: many plausible draft tokens are rejected because `p(x)/q(x) < 1`, even when those tokens would lead to correct outputs. This limits the practical speedup achievable with speculative decoding.

In our experiments across multiple model pairs (Llama, Qwen, Gemma families), we observe that wall-clock time decreases monotonically as acceptance rate increases. The rigid verification step is the primary bottleneck.

**Describe the solution you would like**

Add an optional `assistant_ensemble_weight` parameter to `GenerationConfig` that enables **static ensemble verification** — a training-free, single-parameter extension that trades a controllable amount of distributional bias for higher acceptance rates.

The verification distribution becomes a weighted mixture:

```
v(x) = w * p_target(x) + (1 - w) * q_draft(x)
```

A draft token is accepted with probability `min(1, v(x) / q(x))`, and on rejection, we resample from the corresponding fallback distribution.

**Proposed API:**

```python
outputs = model.generate(
    **inputs,
    assistant_model=assistant_model,
    assistant_ensemble_weight=0.7,  # float in (0, 1], default=1.0 (lossless)
)
```

**Key properties:**
- `w=1.0` recovers standard lossless speculative decoding (backward compatible)
- `w<1.0` increases acceptance probability from `1 - TV(q,p)` to `1 - w*TV(q,p)` (Lemma 1 in our paper)
- The method is **Pareto-optimal**: it achieves the best possible tradeoff between acceptance rate and distributional bias (Proposition 1)
- No training required, no extra model weights, just one scalar parameter

**Describe alternatives you have considered**

1. Training a draft model to better match the target (e.g., online distillation) — requires expensive training
2. Dynamic ensemble with learned context-dependent weights (our full DIVERSED method) — requires training an ensemble head, too invasive for a first contribution
3. Simply increasing `num_assistant_tokens` — does not address the fundamental acceptance rate limitation

**Empirical results (from our paper):**

On CNN/DailyMail with Llama-3.1-8B-Instruct (target) + Llama-3.2-1B-Instruct (draft), temperature=0:
- Standard SD (w=1.0): ~65% acceptance rate
- Static ensemble (w=0.7): ~78% acceptance rate, with ROUGE-L within 0.5 points of the target model

Similar improvements observed across GSM8K, XSum, WMT, HumanEval, and MBPP benchmarks.

**Implementation scope:**

The change is minimal (~50-100 lines of logic):
1. Add `assistant_ensemble_weight` to `GenerationConfig` in `configuration_utils.py`
2. Modify the verification step in `utils.py` to blend distributions when `w < 1.0`
3. Tests and documentation

**References:**
- Paper: [DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification](https://arxiv.org/abs/2604.07622) (AISTATS 2026)
- Code: https://github.com/comeusr/diversed
- The static ensemble method is described in Section 3.1 of the paper

I am happy to implement this and submit a PR if there is interest from the maintainers. I am one of the paper authors.

--- Comment by Rocketknight1 at 2026-05-11T14:08:18Z ---
cc @cyrilvallez for generation and @remi-or @ArthurZucker @McPatate for CB

--- Comment by Cyrilvallez at 2026-05-13T03:32:29Z ---
Hey @kasakh! It's indeed an interesting approach, to be fair I've wondered for quite some time why the drafting of assistant tokens was not more relaxed!
Feel free to submit a PR for it, it should be relatively straightforward. You can tweak the `_speculative_sampling` function [here](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L3844) to relax the drafting. This should be a good starting point

--- Comment by kasakh at 2026-05-16T06:42:19Z ---
Thanks @Cyrilvallez for the pointer! I have submitted a PR implementing this: #45979

It modifies `_speculative_sampling` as you suggested, adds greedy support via `argmax(v)`, includes 8 fast synthetic tests, and a docs section. CI is green. Happy to iterate on any feedback.


ISSUE #45864: PretrainedConfig.from_pretrained: silent default on missing config.json
State: closed  |  Labels:
Author: MilkClouds  |  Created: 2026-05-09T18:25:53Z

### System Info

- transformers 5.6+ (verified on 5.8.0; 5.5.4 raised)
- introduced by #36033 ("Large/full refactor of from_pretrained", 2025-03-12)

### Description

When `PretrainedConfig.from_pretrained(path)` is given a *local* directory with no `config.json`, it silently returns a default-populated config instead of raising `OSError`. The return type is the config class itself (not `Optional`) and the docstring does not document this fallback, so a wrong-path call has no way to surface until much later (model build, weight load, OOM, CI timeout).

### Reproduction

```python
import tempfile
from transformers import LlamaConfig

with tempfile.TemporaryDirectory() as d:
    cfg = LlamaConfig.from_pretrained(d)
    print(type(cfg).__name__, cfg.model_type)
    # 5.5.4: raises OSError ("...no file named config.json...")
    # 5.6+:  silently returns LlamaConfig with defaults
```

(`AutoConfig` raises `ValueError("Unrecognized model")` because the factory needs `model_type`; the silent fallback is on concrete `PretrainedConfig` subclasses.)

### Where it changed

`cached_files` (hub.py) has long special-cased missing `config.json` to return `None` (unchanged since 5.5.4; needed so a Hub model id with a cache miss can fall through to download).

#36033 changed the caller chain in `configuration_utils.py`:

```
_get_config_dict   -> (None, kwargs)
get_config_dict    -> ({}, kwargs)
from_dict({})      -> cls()   # populates from defaults; no raise
```

In 5.5.4 the caller raised at this junction.

### Suggested fix

For a *local directory* input, restore the 5.5.4 caller-side raise. Hub-id fallback semantics can stay.

### Note

I confirm this is not a pure code-agent issue.

--- Comment by MilkClouds at 2026-05-09T18:38:48Z ---
After a proper version bisect, the silent fallback for missing local `config.json` was introduced in **4.45** by #32356 (*"support loading model without config.json file"*) as an intentional feature, not a regression. My earlier diagnosis pinning it on #36033 was wrong: that PR only retouched the existing line. Closing as invalid; my apologies for the noise.


ISSUE #45859: `Qwen3_5MoeTextRotaryEmbedding.forward` is not compatible with CPU offload
State: closed  |  Labels: bug
Author: jamesbraza  |  Created: 2026-05-09T05:46:24Z

### System Info

- `transformers` version: 5.5.4
- Platform: Linux-6.8.0-1043-nvidia-x86_64-with-glibc2.35
- Python version: 3.12.13
- Huggingface_hub version: 1.11.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu129 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@ArthurZucker @Cyrilvallez @3outeille @zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import sys

import torch
from transformers.models.qwen3_5_moe.modeling_qwen3_5_moe import (
    Qwen3_5MoeTextConfig,
    Qwen3_5MoeTextRotaryEmbedding,
)

if not torch.cuda.is_available():
    raise NotImplementedError("This reproducer needs at least one CUDA device")

# Default config; the bug is in the rotary forward, so the precise model
# dims don't matter. Using defaults avoids any checkpoint download
cfg = Qwen3_5MoeTextConfig()
rotary = Qwen3_5MoeTextRotaryEmbedding(cfg)
print(f"rotary.inv_freq.device = {rotary.inv_freq.device}")
assert rotary.inv_freq.device.type == "cpu", (
    "Expected inv_freq to land on CPU when no device is passed; got"
    f" {rotary.inv_freq.device}. Bug premise no longer holds."
)

# GPU activation, simulating what FSDP2 forward routes into the rotary call
bsz, seq_len = 1, 8
x = torch.randn(bsz, seq_len, cfg.hidden_size, device="cuda", dtype=torch.bfloat16)
position_ids = torch.arange(seq_len, device="cuda")[None, :]

try:
    cos, sin = rotary(x, position_ids)
except RuntimeError as exc:
    msg = str(exc)
    if (
        "Expected all tensors to be on the same device" in msg
        and "wrapper_CUDA_bmm" in msg
    ):
        print(f"Reproduced: {type(exc).__name__}: {msg}")
        sys.exit(0)
    raise NotImplementedError(
        f"Unexpected RuntimeError (not the device-mismatch we expect): {msg}"
    ) from exc

raise NotImplementedError(
    f"Did not reproduce: rotary forward returned {cos.device=} {sin.device=}."
)
```

### Expected behavior

When training Qwen 3.6 (model using as `qwen3_5_moe`) under FSDP2 with `CPUOffloadPolicy(...)` keeping `inv_freq` on the host, the rotary-embedding forward fails with a mixed-device `bmm`:

```none
RuntimeError: Expected all tensors to be on the same device, but got mat2 is on cuda:1, different from other tensors on cpu (when checking argument in method wrapper_CUDA_bmm)
```

The non-MoE [`Qwen3RotaryEmbedding.forward` (qwen3/modeling_qwen3.py:L138)](https://github.com/huggingface/transformers/blob/v5.5.4/src/transformers/models/qwen3/modeling_qwen3.py#L138) guards against this by appending `.to(x.device)` to `inv_freq_expanded`. The MoE variant [`Qwen3_5MoeTextRotaryEmbedding.forward` (qwen3_5_moe/modeling_qwen3_5_moe.py:L145)](https://github.com/huggingface/transformers/blob/v5.5.4/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py#L145) does not.

The fix should be a one-line patch appending `.to(x.device)`.

--- Comment by jshaofa-ui at 2026-05-09T12:46:39Z ---
## 🔧 Solution Proposal

### Root Cause

The `Qwen3_5MoeTextRotaryEmbedding.forward()` method is missing a `.to(x.device)` call on `inv_freq_expanded`, which causes a device mismatch when using FSDP2 with `CPUOffloadPolicy`.

**Comparison with the working non-MoE version:**

| File | Line | Code |
|------|------|------|
| `qwen3/modeling_qwen3.py` | 138 | `inv_freq_expanded = self.inv_freq[None, :, None].float().expand(...).to(x.device)` ✅ |
| `qwen3_5_moe/modeling_qwen3_5_moe.py` | 145 | `inv_freq_expanded = self.inv_freq[None, None, :, None].float().expand(...)` ❌ missing `.to(x.device)` |

When FSDP2's `CPUOffloadPolicy` keeps `inv_freq` on CPU while activations are on GPU, the matrix multiplication at line 150 fails:
```
RuntimeError: Expected all tensors to be on the same device, but got mat2 is on cuda:1, different from other tensors on cpu
```

### Proposed Fix

**File:** `src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py`
**Line:** 145

```diff
- inv_freq_expanded = self.inv_freq[None, None, :, None].float().expand(3, position_ids.shape[1], -1, 1)
+ inv_freq_expanded = self.inv_freq[None, None, :, None].float().expand(3, position_ids.shape[1], -1, 1).to(x.device)
```

### Verification

The fix aligns the MoE variant with the non-MoE `Qwen3RotaryEmbedding.forward()` implementation (line 138 of `qwen3/modeling_qwen3.py`), which already includes `.to(x.device)` and works correctly with CPU offload.

### Impact
- **One-line fix** — minimal risk, no API changes
- **Enables FSDP2 CPU offload** for Qwen3.5-MoE models (H100/large-scale training)
- **No performance impact** — `.to()` is a no-op when already on the correct device

--- Comment by Cyrilvallez at 2026-05-12T08:09:24Z ---
Hi @jamesbraza, I really don't understand your issue. You're initializing a module on cpu, and feeding tensors on cuda to the forward. This is 100% expected to fail with a device error.

--- Comment by jamesbraza at 2026-05-13T05:06:34Z ---
Hi @Cyrilvallez alright check the below reproducer. The OP reproducer looks like a mixed-up device situation, but it was simplifying FSDP2 `CPUOffloadPolicy` and non-persistent-buffer handler from [SkyRL `_sync_non_persistent_buffers`](https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/distributed/fsdp_utils.py#L261-L279).

```python
import os

import torch
import torch.nn as nn
from torch.distributed.fsdp import CPUOffloadPolicy, fully_shard

from transformers import Qwen3Config, Qwen3_5MoeTextConfig
from transformers.models.qwen3.modeling_qwen3 import Qwen3RotaryEmbedding
from transformers.models.qwen3_5_moe.modeling_qwen3_5_moe import Qwen3_5MoeTextRotaryEmbedding

def sync_non_persistent_buffers(module: nn.Module) -> None:
    """Inline'd SkyRL FSDP2 helper: https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/distributed/fsdp_utils.py#L261-L279."""
    for submod in module.modules():
        for name in list(submod._non_persistent_buffers_set):
            buf = submod._buffers.get(name)
            if buf is not None and buf.device.type != "cpu":
                submod._buffers[name] = buf.detach().cpu()

def init_single_rank() -> None:
    os.environ.setdefault("MASTER_ADDR", "127.0.0.1")
    os.environ.setdefault("MASTER_PORT", "29501")
    os.environ.setdefault("RANK", "0")
    os.environ.setdefault("WORLD_SIZE", "1")
    os.environ.setdefault("LOCAL_RANK", "0")
    if not torch.distributed.is_initialized():
        torch.distributed.init_process_group(backend="nccl")
    torch.cuda.set_device(0)

class RotaryHost(nn.Module):
    def __init__(self, rotary: nn.Module) -> None:
        super().__init__()
        self.dummy = nn.Linear(8, 8, bias=False)
        self.rotary = rotary

    def forward(self, x: torch.Tensor, position_ids: torch.Tensor):
        return self.rotary(x, position_ids)

def run(label: str, host: nn.Module, hidden_size: int) -> None:
    x = torch.randn(1, 8, hidden_size, device="cuda:0", dtype=torch.bfloat16)
    position_ids = torch.arange(8, device="cuda:0")[None, :]
    try:
        cos, sin = host(x, position_ids)
    except RuntimeError as exc:
        msg = str(exc).splitlines()[0][:200]
        print(f"--- {label:25s} FAILED  {type(exc).__name__}: {msg}")
        return
    finite = bool(torch.isfinite(cos).all().item() and torch.isfinite(sin).all().item())
    print(f"--- {label:25s} OK      cos.device={cos.device}, finite={finite}")

if not torch.cuda.is_available():
    raise SystemExit("Reproducer needs CUDA")
init_single_rank()

cfg3 = Qwen3Config()
host3 = RotaryHost(Qwen3RotaryEmbedding(cfg3)).to("cuda:0")
fully_shard(host3, offload_policy=CPUOffloadPolicy())
sync_non_persistent_buffers(host3)
print(f"Qwen3 (non-MoE):    inv_freq.device = {host3.rotary.inv_freq.device}")
run("Qwen3 (non-MoE)", host3, cfg3.hidden_size)

cfg_moe = Qwen3_5MoeTextConfig()
host_moe = RotaryHost(Qwen3_5MoeTextRotaryEmbedding(cfg_moe)).to("cuda:0")
fully_shard(host_moe, offload_policy=CPUOffloadPolicy())
sync_non_persistent_buffers(host_moe)
print(f"Qwen3_5-MoE M-RoPE: inv_freq.device = {host_moe.rotary.inv_freq.device}")
run("Qwen3_5-MoE (M-RoPE)", host_moe, cfg_moe.hidden_size)
```

Against `transformers==5.8.0`:

```none
Qwen3 (non-MoE):    inv_freq.device = cpu
--- Qwen3 (non-MoE)           OK      cos.device=cuda:0, finite=True
Qwen3_5-MoE M-RoPE: inv_freq.device = cpu
--- Qwen3_5-MoE (M-RoPE)      FAILED  RuntimeError: Expected all tensors to be on the same device,
                                       but got mat2 is on cuda:0, different from other tensors on cpu
                                       (when checking argument in method wrapper_CUDA_bmm)
```

Fwiw, the `.to(x.device)` on `inv_freq_expanded` from the proposed fix https://github.com/huggingface/transformers/pull/45861 is the convention across `transformers`' rotary forwards:
- Qwen3: https://github.com/huggingface/transformers/blob/v5.8.0/src/transformers/models/qwen3/modeling_qwen3.py#L138
- GPT OSS: https://github.com/huggingface/transformers/blob/v5.8.0/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L201

The Qwen3-VL M-RoPE family is the exception I'm trying to report here.

--- Comment by jamesbraza at 2026-05-15T16:21:46Z ---
For posterity, this was closed by https://github.com/huggingface/transformers/pull/45861


ISSUE #45854: pipeline text-generation ignores return_full_text=False with chat template
State: closed  |  Labels:
Author: yashpandya-msr  |  Created: 2026-05-09T00:34:19Z

When using the text-generation pipeline with a chat template format, setting return_full_text=False still returns the full conversation including the prompt. Tested on transformers 4.40.x, model meta-llama/Llama-3-8B-Instruct. Expected: only generated text. Actual: full prompt + generated text.

--- Comment by Abineshabee at 2026-05-09T05:22:14Z ---
Hi @yashpandya-msr, thanks for the detailed report! I’d like to work on this issue. I’ve looked into the pipeline code and believe the issue is in ```postprocess()```, where ```return_full_text=False``` doesn’t correctly account for the chat-template-rendered prompt length. I’ll dig into it and open a PR soon.

--- Comment by jshaofa-ui at 2026-05-09T14:15:55Z ---
## 🐛 return_full_text=False Bug 修复方案\n\n### 根因分析\n\n`TextGenerationPipeline.postprocess()` 使用**字符级切片**来移除 prompt:\n\n```python\n# 有问题的代码 — src/transformers/pipelines/text_generation.py\nif not return_full_text:\n    prompt_length = len(\n        self.tokenizer.decode(\n            input_ids[0],\n            skip_special_tokens=skip_special_tokens,\n            **decode_kwargs,\n        )\n    )\n    generated_text = generated_text[prompt_length:]  # ← 字符级切片!\n```\n\n问题: 当使用 chat template 时, decoded prompt 的字符长度与 decoded full sequence 的前缀不匹配。\nChat template 中的特殊 token (如 `<|im_start|>`, `</s>`) 在 encode→decode 过程中可能产生不一致的字符表示,\n导致字符级切片无法正确移除 prompt。\n\n### 修复方案\n\n使用**token 级切片**替代字符级切片:\n\n```python\n# 修复后的代码\nif not return_full_text:\n    # 使用 token 数量而非字符数量来移除 prompt\n    # input_ids[0] 包含完整的输入 tokens (prompt tokens)\n    # generated_tokens 包含 prompt + generated tokens\n    # 通过 token 数量差来正确切片\n    prompt_token_count = input_ids.shape[-1]  # prompt 的 token 数量\n    generated_token_ids = generated_tokens[0]  # 完整的生成 tokens\n    \n    # 只保留新生成的 tokens\n    new_token_ids = generated_token_ids[prompt_token_count:]\n    \n    generated_text = self.tokenizer.decode(\n        new_token_ids,\n        skip_special_tokens=skip_special_tokens,\n        **decode_kwargs,\n    )\n```\n\n### 关键变更\n\n1. 使用 `input_ids.shape[-1]` 获取 prompt 的 token 数量\n2. 使用 token 索引而非字符索引来切片\n3. 对新生成的 tokens 进行 decode, 而非对整个序列进行字符切片\n\n### 测试用例\n\n```python\ndef test_return_full_text_false_with_chat_template():\n    """return_full_text=False should only return generated text with chat templates."""\n    pipe = pipeline("text-generation", model="meta-llama/Llama-3-8B-Instruct")\n    \n    chat_input = [\n        {"role": "user", "content": "What is 2+2?"},\n    ]\n    \n    result = pipe(chat_input, return_full_text=False, max_new_tokens=20)\n    generated = result[0]["generated_text"]\n    \n    # 不应包含 prompt\n    assert "What is 2+2?" not in generated\n    # 应只包含生成的文本\n    assert len(generated.strip()) > 0\n```\n\n### 兼容性\n- 向后兼容: 非 chat template 场景行为不变\n- 不影响 `return_full_text=True` 的行为\n

--- Comment by Rocketknight1 at 2026-05-11T12:50:49Z ---
## thank you for that contribution jshaofa-ui


ISSUE #45853: Provide HF_USE_MLX=0 / public flag to disable MLX backend detection at import time
State: open  |  Labels:
Author: Steve-Allison  |  Created: 2026-05-08T22:04:11Z

### Summary

Transformers' import-utils module probes for MLX availability during its own
load and imports `mlx.core` for backend type checks if `mlx` is installed.
There is no documented opt-out. On some Apple Silicon configurations,
`import mlx.core` aborts the interpreter (silent `SIGABRT` during native
init), taking the whole host process down before user code can react.

### Context

The current backend-detection code path (paraphrased from
`transformers/utils/import_utils.py`):

```python
_mlx_available = importlib.util.find_spec("mlx") is not None
# … later …
if _mlx_available:
    import mlx.core  # type-check probe — can abort interpreter
```

`USE_TF=0` and `USE_TORCH=0` exist as opt-outs for those backends, but no
equivalent for MLX. Setting `HF_USE_MLX=0` (or any equivalent public knob)
would let downstream projects that explicitly do not use MLX prevent the
import probe entirely.

### Why a public flag matters here

Some downstream projects target Apple Silicon but use PyTorch (eager-mode
MPS) exclusively, with `mlx`/`mlx.core` installed only as a transitive
dependency or development-time artefact. On those hosts, the `mlx.core`
import probe is pure cost and can be unsafe; today the only options are:

1. **Uninstall `mlx`** — not always feasible (transitive deps).
2. **Patch Transformers internals** — what we currently do, against our
   own no-library-patches rule. Brittle on every Transformers upgrade.
3. **Wrap every Transformers import in a sandboxed subprocess** — extreme.

A documented `HF_USE_MLX=0` (or `transformers.utils.disable_backend("mlx")`)
collapses all three into one line.

### Proposed change

Mirror the existing `USE_TORCH` / `USE_TF` shape:

```python
_mlx_available = (
    os.environ.get("HF_USE_MLX", "AUTO").upper() != "0"
    and importlib.util.find_spec("mlx") is not None
)
```

with documentation in the env-var reference page.

### Workaround in use today

A `MetaPathFinder` installed before any Transformers import that:

1. Blocks `mlx.core` imports (raises `ModuleNotFoundError`).
2. Wraps the Transformers `import_utils` loader and writes
   `module._mlx_available = False` after exec.

Source: [docling-project link to import_safety.py and ADR 0003 — fill in
after publishing this issue].

### Acceptance criteria for closing this on our side

1. Transformers ships an env var or public flag to disable MLX detection.
2. The flag is documented.
3. We can drop our `MetaPathFinder` and use the flag.

--- Comment by harryfrzz at 2026-05-09T15:53:39Z ---
Hey! I’ve implemented a fix for this issue and I’m opening a PR for it right now

--- Comment by Rocketknight1 at 2026-05-11T13:06:37Z ---
Surely the issue here is `import mlx.core` aborting the interpreter, right? Do we have any details on why that happens, or on what systems?

--- Comment by harryfrzz at 2026-05-11T13:13:52Z ---
> Surely the issue here is `import mlx.core` aborting the interpreter, right? Do we have any details on why that happens, or on what systems?

Is it okay that I opened the PR for this fix before your approval?

--- Comment by Rocketknight1 at 2026-05-11T15:01:21Z ---
I'm not going to close it immediately! But I'd like to actually figure out what's happening here before we accept the code agent solution


ISSUE #45850: Since 5.0 version it breaks too many models
State: open  |  Labels: bug
Author: LOYINuts  |  Created: 2026-05-08T16:47:19Z

### System Info

Im using DNABERT -2 and after i update transformers there is an error:

AttributeError: 'BertConfig' object has no attribute 'pad_token_id'

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

just use DNABERT-2 and use its tokenizer as well

### Expected behavior

Back to normal

--- Comment by ArjunSrivastava1 at 2026-05-09T11:38:40Z ---
welp

while i will love to just contribute outright and get resume points and tell that oh may was spent fixing stuff like this and yes i got more months of exp now, this doesnt really look or sound like something that should handled by me, or even open sourced cus the main classes and breakings of updates and what not

tagging a maintainer instead cus this def sounds like a higher level thing:
 @Rocketknight1

--- Comment by Rocketknight1 at 2026-05-11T16:02:06Z ---
hi @LOYINuts, this is a custom code model, right? In that case, you might need to ping the maintainer to update the custom code in the repo, or make a fork/PR to do it yourself. We usually can't/don't update people's custom code repos for them!


Embed Agent Friendly Code Score Badge

ISSUE #45841: Embed Agent Friendly Code Score Badge
State: open  |  Labels:
Author: hsnice16  |  Created: 2026-05-08T07:04:06Z

Hi team — I'm Himanshu, I built Agent Friendly Code, which scores public repos on how legible they are to AI coding agents (clear conventions, docs, tests, build signals — not anything about accepting agent-authored PRs).

`transformers` scored 75.9/100 — full breakdown: https://www.agentfriendlycode.com/repo/126

If you're open to it, here's a badge you can drop in the README:
```
[![Agent Friendly](https://agentfriendlycode.com/api/badge/github/huggingface/transformers.svg)](https://agentfriendlycode.com/repo/126)
```

Renders as: [![Agent Friendly](https://agentfriendlycode.com/api/badge/github/huggingface/transformers.svg)](https://agentfriendlycode.com/repo/126)

_A note on what this isn't: the badge signals codebase readability for agents, not an invitation for drive-by AI PRs. I just wanted to let you know that your contribution policy is unchanged. Totally fine to pass — happy either way, and feedback on the score itself is welcome._

---

<img width="1875" height="1160" alt="Image" src="https://github.com/user-attachments/assets/abf415f2-d067-44e0-aec1-6a0c5110f5b7" />


ISSUE #45834: Kosmos2.5: index error on long ocr input
State: open  |  Labels: bug
Author: nunq  |  Created: 2026-05-08T00:29:00Z

### System Info

- `transformers` version: 5.8.0
- Platform: Linux-7.0.3-arch1-2-x86_64-with-glibc2.43
- Python version: 3.13.11
- Huggingface_hub version: 1.14.0
- Safetensors version: 0.7.0
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.11.0+cu130 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

running on cpu

### Who can help?

@zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

when giving a large input file into kosmos2.5's `<ocr>` mode while using `max_new_tokens=4096`, it sometimes fails with an `index out of range in self` error. this is because the embedding table isn't properly expanded because `past_key_values_length=0` is [hardcoded in the forward call](https://github.com/huggingface/transformers/blob/44a2e7a43870ec3096d4ee239f31fd95c530b29f/src/transformers/models/kosmos2_5/modeling_kosmos2_5.py#L688). if the input is too large (=past expanded size), [`return self.weights.index_select()`](https://github.com/huggingface/transformers/blob/44a2e7a43870ec3096d4ee239f31fd95c530b29f/src/transformers/models/kosmos2_5/modeling_kosmos2_5.py#L710) fails with the out of range error.

here is a regression script:

1. install deps: `uv add --dev pillow torch transformers`
2. `uv run python regression.py`

```sh
CRASH at step 2044: position_id=4098, table=4098 — index out of range in self
[FAIL] crashed at step 2044
```
out of bounds access after generation step 2044.

regression.py:
```python
import sys
import torch
from PIL import Image
from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration

MAX_NEW_TOKENS = 4096

repo = "microsoft/kosmos-2.5"

# measure ocr prompt length
processor = AutoProcessor.from_pretrained(repo)
dummy_image = Image.new("RGB", (64, 64))
inputs = processor(text="<ocr>", images=dummy_image, return_tensors="pt")
PROMPT_LEN = inputs["input_ids"].shape[1]

model = Kosmos2_5ForConditionalGeneration._from_config(
    Kosmos2_5ForConditionalGeneration.config_class.from_pretrained(repo)
)
embed = model.text_model.model.embed_positions
PADDING_IDX = model.config.text_config.pad_token_id

initial_size = embed.weights.size(0)
prev_size = initial_size
crash_step = None

for step in range(MAX_NEW_TOKENS):
    # simulate one cached generation step
    past_kv_len = PROMPT_LEN + step
    position_ids = torch.tensor([[PADDING_IDX + past_kv_len + 1]])
    input_ids = torch.ones(1, 1, dtype=torch.long) * 42

    try:
        # past_key_values_length=0 like forward()
        embed(input_ids=input_ids, position_ids=position_ids, past_key_values_length=0)
    except IndexError as e:
        crash_step = step
        print(f"CRASH at step {step}: position_id={position_ids.item()}, table={embed.weights.size(0)} — {e}")
        break

    new_size = embed.weights.size(0)
    if new_size != prev_size:
        prev_size = new_size

if crash_step is None:
    print(f"prompt_len={PROMPT_LEN}, padding_idx={PADDING_IDX}, initial_table={initial_size}, final_table={embed.weights.size(0)}")
    print(f"[PASS] all {MAX_NEW_TOKENS} steps completed without crash")
else:
    print(f"[FAIL] crashed at step {crash_step}")
    sys.exit(1)
```

### Expected behavior

embedding table is expanded properly and `<ocr>` doesn't crash on big inputs.

--- Comment by zucchini-nlp at 2026-05-08T10:02:23Z ---
taking a look on Monday, thanks for opening a PR to fix it :)

--- Comment by jshaofa-ui at 2026-05-11T17:12:28Z ---
# Solution for huggingface/transformers Issue #45834

## Kosmos2.5: Index Error on Long OCR Input

### Issue Summary

When running `Kosmos2_5ForConditionalGeneration.generate()` with a large OCR input image and `max_new_tokens=4096`, the model crashes with an `index out of range in self` error during the embedding position lookup. The crash typically occurs around step 2044 of generation.

---

### Root Cause Analysis

The bug is in `src/transformers/models/kosmos2_5/modeling_kosmos2_5.py`, in the `Kosmos2_5TextTransformer.forward()` method.

At **line 976**, the positional embedding forward call hardcodes `past_key_values_length=0`:

```python
positions = self.embed_positions(
    input_ids=input_ids,
    inputs_embeds=inputs_embeds,
    past_key_values_length=0,   # <-- HARDCODED ZERO
    position_ids=position_ids,
)
```

However, the `forward()` method **already receives** `past_key_values: Cache | None = None` as a parameter (line 948). During autoregressive generation (after the first iteration), `past_key_values` contains the cached key/value states from previous steps, and its sequence length should be used to compute correct position IDs.

#### Why This Causes the Crash

The `Kosmos2_5TextSinusoidalPositionalEmbedding.forward()` method (lines 684-710) uses `past_key_values_length` to:

1. **Compute position IDs** (lines 695-703): Position IDs are offset by `past_key_values_length` so that new tokens get positions continuing from where cached tokens ended.

2. **Expand the embedding table if needed** (lines 706-708):
   ```python
   max_pos = self.padding_idx + 1 + seq_len + past_key_values_length
   if max_pos > self.weights.size(0):
       self.make_weights(max_pos + self.offset, self.embedding_dim, self.padding_idx)
   ```

3. **Index into the weight table** (line 710):
   ```python
   return self.weights.index_select(0, position_ids.view(-1)).view(bsz, seq_len, self.weights.shape[-1]).detach()
   ```

**The problem flow:**

1. During generation step 1 (first iteration), `past_key_values=None`, so `past_key_values_length=0` is correct. The embedding table is sized for the initial sequence.

2. During generation step N (N > 1), `past_key_values` contains cached states from N-1 steps. The correct `past_key_values_length` should be `past_key_values.get_seq_length()`.

3. Because `past_key_values_length=0` is hardcoded, the embedding table is **not expanded** to accommodate the growing sequence length (step 706-708 sees a smaller `max_pos` than actual).

4. Meanwhile, position IDs computed elsewhere (e.g., in `prepare_inputs_for_generation` at line 1407) **are** correctly offset for the growing sequence.

5. When `index_select()` is called at line 710, the position IDs exceed the weight table size, causing the `index out of range` error.

---

### The Fix

**File:** `src/transformers/models/kosmos2_5/modeling_kosmos2_5.py`

**Location:** `Kosmos2_5TextTransformer.forward()` method, lines 972-978

**Before:**
```python
        # embed positions
        positions = self.embed_positions(
            input_ids=input_ids,
            inputs_embeds=inputs_embeds,
            past_key_values_length=0,
            position_ids=position_ids,
        )
```

**After:**
```python
        # embed positions
        past_key_values_length = past_key_values.get_seq_length() if past_key_values is not None else 0
        positions = self.embed_positions(
            input_ids=input_ids,
            inputs_embeds=inputs_embeds,
            past_key_values_length=past_key_values_length,
            position_ids=position_ids,
        )
```

---

### Why This Fix Works

1. **Correct sequence length tracking:** `past_key_values.get_seq_length()` returns the number of tokens already cached. This is the standard pattern used throughout transformers (e.g., BART at `modeling_bart.py:620`, kosmos2 at `modeling_kosmos2.py:1038`).

2. **Proper embedding table expansion:** With the correct `past_key_values_length`, the `max_pos` calculation at line 706 correctly accounts for both cached tokens and new tokens, ensuring the weight table is expanded before `index_select()`.

3. **Consistent position IDs:** Position IDs will correctly continue from where cached tokens ended, rather than always starting from 0.

4. **Safe for first iteration:** When `past_key_values is None` (first generation step), `past_key_values_length` defaults to 0, matching the original behavior.

---

### Diff

```diff
--- a/src/transformers/models/kosmos2_5/modeling_kosmos2_5.py
+++ b/src/transformers/models/kosmos2_5/modeling_kosmos2_5.py
@@ -969,6 +969,7 @@ class Kosmos2_5TextTransformer(Kosmos2_5PreTrainedModel):
         inputs_embeds = inputs_embeds * self.embed_scale

         # embed positions
+        past_key_values_length = past_key_values.get_seq_length() if past_key_values is not None else 0
         positions = self.embed_positions(
             input_ids=input_ids,
             inputs_embeds=inputs_embeds,
-            past_key_values_length=0,
+            past_key_values_length=past_key_values_length,
             position_ids=position_ids,
         )
```

---

### Test Plan

#### Test 1: Reproduce the original bug (pre-fix)

```python
import torch
from PIL import Image
from transformers import AutoProcessor, Kosmos2_5ForConditionalGeneration

model_id = "microsoft/kosmos-2.5"
processor = AutoProcessor.from_pretrained(model_id)
model = Kosmos2_5ForConditionalGeneration.from_pretrained(model_id)

# Use a large image that produces many OCR tokens
image = Image.open("large_document.png")  # Any large document image
prompt = "<ocr>"

inputs = processor(text=prompt, images=image, return_tensors="pt")
inputs = {k: v for k, v in inputs.items()}

# This should crash with index out of range before the fix
outputs = model.generate(**inputs, max_new_tokens=4096)
```

**Expected before fix:** `IndexError: index out of range in self` around step 2044.
**Expected after fix:** Generation completes successfully.

#### Test 2: Verify generation with small inputs (regression check)

```python
# Small image - should work both before and after fix
small_image = Image.open("small_icon.png")
inputs = processor(text="<ocr>", images=small_image, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
# Verify outputs are valid text
assert outputs.shape[1] > 0
```

#### Test 3: Verify past_key_values propagation

```python
# Manual forward pass with past_key_values
from transformers import DynamicCache

inputs = processor(text="<ocr>", images=image, return_tensors="pt")
first_outputs = model(**inputs, use_cache=True)

# Verify past_key_values is returned
assert first_outputs.past_key_values is not None
assert first_outputs.past_key_values.get_seq_length() > 0

# Second forward pass with cached values
new_input_ids = torch.tensor([[model.config.eos_token_id]])
second_outputs = model(
    input_ids=new_input_ids,
    past_key_values=first_outputs.past_key_values,
    use_cache=True,
)
# This should not crash after the fix
assert second_outputs.logits is not None
```

#### Test 4: Compare with kosmos2 (reference implementation)

```python
# Verify the fix matches the pattern used in kosmos2
# kosmos2/modeling_kosmos2.py line 1038:
#   past_key_values_length = past_key_values.get_seq_length() if past_key_values is not None else 0
# kosmos2/modeling_kosmos2.py line 1050:
#   past_key_values_length=past_key_values_length,
# The fix should produce identical behavior for equivalent inputs.
```

#### Test 5: Edge case - empty past_key_values

```python
# Verify behavior when past_key_values is an empty cache
empty_cache = DynamicCache()
inputs = processor(text="<ocr>", images=image, return_tensors="pt")
outputs = model(**inputs, past_key_values=empty_cache, use_cache=True)
# Should work identically to past_key_values=None
```

---

### Additional Notes

- **No modular file:** There is no `modular_kosmos2_5.py` in the kosmos2_5 model directory, so the fix should be applied directly to `modeling_kosmos2_5.py`.

- **Similar pattern in kosmos2:** The original kosmos2 model (`modeling_kosmos2.py`) correctly computes `past_key_values_length` at line 1038 and passes it to the embedding forward at line 1050. This confirms the fix pattern is correct.

- **The `prepare_inputs_for_generation` method** (line 1400) already uses `past_key_values.get_seq_length()` correctly, showing the developers were aware of this API but missed propagating it to the embedding layer.

- **Impact:** This is a critical bug for any use case involving long OCR or document understanding with `max_new_tokens` exceeding the initial embedding table size. The fix is minimal (2 lines changed) and follows the established pattern used across the transformers library.

--- Comment by jshaofa-ui at 2026-05-11T19:17:44Z ---
## Update: PR #45835 submitted with fix

@zucchini-nlp — A PR has been opened to fix this issue: **PR #45835** (fix: kosmos2.5: properly expand embeddings table)

The PR takes a slightly different approach from our proposed solution:
- **Our proposal**: Fix `past_key_values_length=0` in `Kosmos2_5TextTransformer.forward()` to dynamically compute from `past_key_values`
- **PR #45835**: Fix `max_pos` calculation in `Kosmos2_5TextSinusoidalPositionalEmbedding.forward()` using `int(position_ids.max().item()) + 1`

Both approaches address the same root cause. The PR approach is more defensive (uses actual position_ids rather than relying on past_key_values state), which may be more robust. Tests pass (117 passed, 126 skipped).

Looking forward to maintainer review. 🙏


ISSUE #45823: Gemma4 PLE device mismatch with `device_map="auto"` during forward
State: closed  |  Labels: bug
Author: rishon-galileo  |  Created: 2026-05-07T11:23:10Z

### System Info

- `transformers` version: 5.8.0
- Platform: Linux-6.6.122+-x86_64-with-glibc2.35
- Python version: 3.11.13
- Huggingface_hub version: 1.14.0
- Safetensors version: 0.6.2
- Accelerate version: 1.10.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.11.0+cu130 (CUDA)
- Using distributed or parallel set-up in script?: Yes (distribuited)
- Using GPU in script?: Yes (2 GPUs)
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@zucchini-nlp @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

This reproduces with a prompt when loading Gemma4 with `device_map="auto"` and calling `forward` directly.

### Steps to reproduce

1. Install latest packages (`torch`, `transformers`, `accelerate`).
2. Load `google/gemma-4-E2B-it` with `device_map="auto"`.
3. Build inputs via `AutoProcessor.apply_chat_template(..., tokenize=True, return_tensors="pt", return_dict=True)`.
4. Call `model(**inputs, use_cache=False)`.
5. Observe a cross-device runtime error (`cuda:0` vs `cuda:1`) inside Gemma4 PLE code path.

### Minimal script

```python
import torch
import traceback
from transformers import AutoProcessor, AutoModelForCausalLM

model_id = "google/gemma-4-E2B-it"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

print("hf_device_map:", model.hf_device_map)

messages = [{"role": "user", "content": "What is the meaning of life?"}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)

model.eval()
try:
    _ = model(**inputs, use_cache=False)
    print("No error")
except Exception:
    traceback.print_exc()
```

### Full traceback

```text
Traceback (most recent call last):
  File "/tmp/ipython-input-2807144601.py", line 5, in <cell line: 0>
    out = model(**inputs, use_cache=False)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/accelerate/hooks.py", line 175, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/utils/generic.py", line 900, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2536, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/utils/generic.py", line 976, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/utils/generic.py", line 900, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/models/gemma4/modeling_gemma4.py", line 2301, in forward
    llm_inputs_embeds = torch.where(multimodal_mask[..., None], pad_embedding.view(1, 1, -1), inputs_embeds)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
```

### Suspected root cause

In Gemma4 forward, PLE path builds `llm_inputs_embeds` with:

```python
pad_embedding = self.language_model.embed_tokens.weight[self.config.text_config.pad_token_id, :]
llm_inputs_embeds = torch.where(multimodal_mask[..., None], pad_embedding.view(1, 1, -1), inputs_embeds)
```

With `device_map="auto"`, operands can be split across GPUs, causing `torch.where` to fail with a device mismatch.

### Expected behavior

`model(**inputs, use_cache=False)` should succeed on multi-GPU sharded setups (`device_map="auto"`) for text-only prompts, without cross-device tensor errors.

--- Comment by zucchini-nlp at 2026-05-07T11:29:42Z ---
Related to https://github.com/huggingface/transformers/pull/45817

Will take a look, maybe we'll fix it together with the other reported multi-gpu issues in the above PR, or I'll go ahead and open a new PR

--- Comment by ArjunSrivastava1 at 2026-05-07T18:36:08Z ---
wow, that was surprisingly quick, done within a few hours, does this place always move this quickly?


DeepSeekV4 的 CSA Indexer 因果 mask

ISSUE #45820: DeepSeekV4 的 CSA Indexer 因果 mask
State: closed  |  Labels:
Author: slZheng077  |  Created: 2026-05-07T09:22:07Z

官方 | ✅ 有 mask，Query[t] 只检索 compressed[0..t//ratio] | 因果正确
Transformers | ❌ 无 mask，Query[t] 可检索所有 compressed | 信息泄露
对比官方推理代码的实现，在indexer做检索时，只会检索之前已经压缩过的window，而在transformers中，没有看到这部分实现。

--- Comment by ArjunSrivastava1 at 2026-05-07T18:55:54Z ---
i will take a look in the docs, and the code and see what i can find regarding it
cus this is rather problematic if thats not in the transformers
i use deepseek on a daily basis and also read through the v4 paper too, have it in my files - i will be very pleased and grateful have the opportunity to improve something i use daily

can i go ahead with it?

--- Comment by ArjunSrivastava1 at 2026-05-08T13:32:30Z ---
upon further looking into other issues and things, this issue is similar and may be related to or with issue number #45758
as such, i think both the issues can be bundled together and closed at the same time unless im reading things wrong

both mention deepseek v4, masking issues and the fact that the casual relationship is not maintained in current implementation for transformers

before trying anything myself

tagging the maintainers who ik def would know something or the other:
@ArthurZucker - ur still working on that one correct? i think this one may allow you to find a fix easier
@zucchini-nlp: ur the one who worked on the mask yesterday for gemma if im remembering correctly, figure u may know a thing or two?

--- Comment by zucchini-nlp at 2026-05-08T13:41:44Z ---
looks related indeed, Arthur will take a look :)

--- Comment by ArjunSrivastava1 at 2026-05-09T11:20:20Z ---
alr, im gonna leave it up to u guys then

--- Comment by ArthurZucker at 2026-05-11T06:24:02Z ---
yes sorry for the error

--- Comment by ArthurZucker at 2026-05-12T09:28:05Z ---
#45892 fixed


ISSUE #45812: `AutoTokenizer` produces wrong token IDs for all Granite models (silent v4→v5 regression)
State: closed  |  Labels: bug
Author: kndtran  |  Created: 2026-05-06T18:23:26Z

### System Info

- `transformers` version: 5.8.0 (also reproduced on 5.0.0 through 5.7.0)
- Platform: Linux-5.14.0-503.11.1.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.12.13
- Huggingface_hub version: 1.14.0
- Safetensors version: 0.7.0
- Tokenizers version: 0.22.2
- PyTorch version: not installed (tokenizer-only reproduction)
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, PreTrainedTokenizerFast

model_id = "ibm-granite/granite-4.1-8b"  # any Granite 4+ model

tok_auto = AutoTokenizer.from_pretrained(model_id)
tok_correct = PreTrainedTokenizerFast.from_pretrained(model_id)

# Numeric strings are most visibly affected (digit splitting differs)
print(tok_auto.encode("2023", add_special_tokens=False))       # [508, 1419] ← WRONG
print(tok_correct.encode("2023", add_special_tokens=False))    # [2366, 18]  ← correct

print(tok_auto.encode("650841823", add_special_tokens=False))      # [13655, 5833, 972, 1419] ← WRONG
print(tok_correct.encode("650841823", add_special_tokens=False))   # [13655, 25496, 23848]    ← correct

print(tok_auto.encode("ISO 9001:2015", add_special_tokens=False))      # [25141, 220, 24, 4119, 25, 679, 20] ← WRONG
print(tok_correct.encode("ISO 9001:2015", add_special_tokens=False))   # [25141, 220, 7467, 16, 25, 679, 20] ← correct
```

Pre-tokenizer mismatch:
```python
print(tok_auto.backend_tokenizer.pre_tokenizer)
# ByteLevel(add_prefix_space=False, trim_offsets=True, use_regex=True)  ← WRONG

print(tok_correct.backend_tokenizer.pre_tokenizer)
# Sequence([Split(regex, ...), ByteLevel(..., use_regex=False)])  ← correct (matches tokenizer.json)
```

### Expected behavior

Granite models ship a `tokenizer.json` with a `Sequence(Split(\p{N}{1,3}) + ByteLevel)` pre-tokenizer (tiktoken/cl100k style — splits digit runs into ≤3-character chunks before BPE). But `AutoTokenizer` routes Granite to `GPT2Tokenizer`, whose [`__init__`](https://github.com/huggingface/transformers/blob/v5.8.0/src/transformers/models/gpt2/tokenization_gpt2.py#L119) hardcodes `ByteLevel(use_regex=True)` — which uses the GPT-2 regex (`\p{N}+`, keeps all digits together).

Since BPE was trained with `\p{N}{1,3}` splitting, using a different pre-tokenizer at inference produces wrong merges and wrong token IDs.

## Workaround

Use `PreTrainedTokenizerFast` directly (works in both v4 and v5):

```python
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("ibm-granite/granite-4.1-8b")
```

## Proposed Fix

Change `TOKENIZER_MAPPING_NAMES` for Granite model types from `"GPT2Tokenizer"` to `"TokenizersBackend"`:

```python
-        ("granite", "GPT2Tokenizer"),
-        ("granitemoe", "GPT2Tokenizer"),
-        ("granitemoehybrid", "GPT2Tokenizer"),
-        ("granitemoeshared", "GPT2Tokenizer"),
+        ("granite", "TokenizersBackend" if is_tokenizers_available() else None),
+        ("granitemoe", "TokenizersBackend" if is_tokenizers_available() else None),
+        ("granitemoehybrid", "TokenizersBackend" if is_tokenizers_available() else None),
+        ("granitemoeshared", "TokenizersBackend" if is_tokenizers_available() else None),
```

This triggers the existing mismatch detection (mapping says `"TokenizersBackend"` ≠ hub's `"GPT2Tokenizer"`), which falls through to `TokenizersBackend.from_pretrained()` — loading `tokenizer.json` faithfully.

**Why not `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS`?** That override set is only consulted inside the mismatch block (line 733) — which requires `TOKENIZER_MAPPING_NAMES[model_type]` to disagree with the hub's `tokenizer_class`. For Granite, both say `"GPT2Tokenizer"`, so no mismatch is detected and the override set is never checked.

## Related Issues

- #45488 — Same bug class: `LlamaTokenizer` hardcodes `Metaspace`, breaks DeepSeek V3/R1 (GSM8K drops 63.7% → 26.1%)
- #44462 — Same bug class: `AutoTokenizer` ignores `tokenizer.json` pre-tokenizer for deepseek-coder
- #43122 — Same bug class: different tokenization between v4.57.3 and v5.0 (MiniMax)
- #45701 — Same bug class: character-level tokenization in v5.7.0 instead of subword (camembert v2)

--- Comment by kndtran at 2026-05-06T18:53:38Z ---
This is a straightforward fix, so I opened PR #45813. This covers all the Granite model families as defined in the mapping.

I updated the proposed fix as the previous one was not triggering because I had misunderstood when `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` was needed. That override set is only consulted inside the mismatch block (line 733) — which requires `TOKENIZER_MAPPING_NAMES[model_type]` to disagree with the hub's `tokenizer_class`. For Granite, both say `"GPT2Tokenizer"`, so no mismatch is detected and the override is never checked.

--- Comment by jshaofa-ui at 2026-05-11T19:23:51Z ---
# Solution for transformers Issue #45812: AutoTokenizer produces wrong token IDs for all Granite models

## Issue Summary

`AutoTokenizer.from_pretrained()` for Granite models (4.x series) produces **wrong token IDs** compared to `PreTrainedTokenizerFast.from_pretrained()`. This is a silent v4→v5 regression that affects ALL Granite models.

**Reproduction:**
```python
from transformers import AutoTokenizer, PreTrainedTokenizerFast

tok_auto = AutoTokenizer.from_pretrained("ibm-granite/granite-4.1-8b")
tok_correct = PreTrainedTokenizerFast.from_pretrained("ibm-granite/granite-4.1-8b")

tok_auto.encode("2023", add_special_tokens=False)       # [508, 1419] ← WRONG
tok_correct.encode("2023", add_special_tokens=False)    # [2366, 18]  ← correct
```

## Root Cause

Granite models ship a `tokenizer.json` with a `Sequence(Split(\p{N}{1,3}) + ByteLevel)` pre-tokenizer (tiktoken/cl100k style — splits digit runs into ≤3-character chunks before BPE).

However, `AutoTokenizer` routes Granite model types to `GPT2Tokenizer`, whose `__init__` hardcodes `ByteLevel(use_regex=True)` — which uses the GPT-2 regex (`\p{N}+`, keeps all digits together).

Since BPE was trained with `\p{N}{1,3}` splitting, using a different pre-tokenizer at inference produces wrong merges and wrong token IDs.

## The Fix

Change `TOKENIZER_MAPPING_NAMES` for Granite model types from `"GPT2Tokenizer"` to `"TokenizersBackend"`:

**File:** `src/transformers/models/auto/tokenization_auto.py`

```diff
--- a/src/transformers/models/auto/tokenization_auto.py
+++ b/src/transformers/models/auto/tokenization_auto.py
@@ -146,10 +146,10 @@ TOKENIZER_MAPPING_NAMES = OrderedDict(
         ("glpn", "SegformerImageProcessor"),
         ("gemma", "GemmaTokenizer"),
         ("gemma2", "GemmaTokenizer"),
-        ("granite", "GPT2Tokenizer"),
-        ("granitemoe", "GPT2Tokenizer"),
-        ("granitemoehybrid", "GPT2Tokenizer"),
-        ("granitemoeshared", "GPT2Tokenizer"),
+        ("granite", "TokenizersBackend"),
+        ("granitemoe", "TokenizersBackend"),
+        ("granitemoehybrid", "TokenizersBackend"),
+        ("granitemoeshared", "TokenizersBackend"),
         ("graniteguardian", "GPT2Tokenizer"),
         ("granitevision", "GPT2Tokenizer"),
         ("graphormer", "PreTrainedTokenizer"),
```

## Why This Fix Works

1. **`TokenizersBackend`** tells `AutoTokenizer` to use the `tokenizers` library directly, which correctly reads the `tokenizer.json` pre-tokenizer configuration (including the `Sequence(Split + ByteLevel)` setup).

2. **`GPT2Tokenizer`** overrides the pre-tokenizer with its own hardcoded `ByteLevel(use_regex=True)`, which breaks the BPE merge order for Granite models.

3. **`PreTrainedTokenizerFast`** already works correctly because it bypasses `TOKENIZER_MAPPING_NAMES` and reads `tokenizer.json` directly — confirming that the tokenizer files themselves are correct.

4. **`graniteguardian` and `granitevision`** are NOT changed because they may use different tokenizer architectures (they are specialized variants).

## Verification

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ibm-granite/granite-4.1-8b")
print(tok.backend_tokenizer.pre_tokenizer)
# Should output: Sequence([Split(regex, ...), ByteLevel(..., use_regex=False)])

# Verify token IDs match
assert tok.encode("2023", add_special_tokens=False) == [2366, 18]
assert tok.encode("650841823", add_special_tokens=False) == [13655, 25496, 23848]
```

## Impact

- **Severity**: High — silent correctness regression affecting ALL Granite model inference
- **Affected models**: granite, granitemoe, granitemoehybrid, granitemoeshared (IBM Granite 4.x series)
- **Fix complexity**: Very low (4 lines, mapping change only)
- **Risk**: Minimal — `TokenizersBackend` is the standard path used by most modern tokenizers


ISSUE #45810: Feature request Add Qwen3_5ForTokenClassification for using as a value model.
State: open  |  Labels: Feature request
Author: han2-l  |  Created: 2026-05-06T16:49:05Z

### Feature request

Add Qwen3_5ForTokenClassification for using as a value model.

### Motivation

In verl, they use AutoModelForTokenClassification for loading value model.
https://github.com/volcengine/verl/blob/2d6c6dbb39bf846d4ebf98c89fc5b4f49c37dd3d/verl/utils/model.py#L627
so I want to add this for using transformers directly.

### Your contribution

Implement Qwen3_5ForTokenClassification

--- Comment by anshuS1310 at 2026-05-07T09:01:39Z ---
hiii !!!  I would like to work on this feature request as I just started contributing reading this and the file from verl
I think I can contribute to this

--- Comment by zucchini-nlp at 2026-05-07T09:28:24Z ---

Yes, we should be able to add task head for all models by simple inheriting a generic layer and indicating a correct `cls.config_class`

Kinda related > I should merge https://github.com/huggingface/transformers/pull/44664 today which basically starts by drawing a clear line between a VLM and LLM with task head. I believe qwen3-5 has both classes

--- Comment by anshuS1310 at 2026-05-07T09:55:56Z ---
I've updated the implementation to include the config_class = Qwen2Config as you suggested. I've also integrated it into load_valuehead_model in verl/utils/model.py. Ready for your review!


ISSUE #45803: [Bug] Bare `except:` in FuyuBatchFeature.convert_to_tensors() swallows KeyboardInterrupt and hides real errors
State: closed  |  Labels: bug
Author: Abineshabee  |  Created: 2026-05-06T11:01:42Z

### System Info

transformers: 5.7.0.dev0 (main, commit 8659ae6245)
Python: 3.13.7
Platform: Windows 11 AMD64
PyTorch: 2.11.0+cu126

### Who can help?

@yonigozlan @molbap

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Bare `except:` is used inside `_safe_convert_tensor()` in `FuyuBatchFeature.convert_to_tensors()`. Because bare `except:` catches all `BaseException` subclasses — including `KeyboardInterrupt` and `SystemExit` — pressing Ctrl+C during Fuyu image processing is silently swallowed and re-raised as a misleading `ValueError`. Real conversion errors are also hidden since the original exception is discarded.

**Affected files (identical bug in both):**
- `src/transformers/models/fuyu/image_processing_fuyu.py`
- `src/transformers/models/fuyu/image_processing_pil_fuyu.py`

```python
# image_processing_fuyu.py, lines 99–108
def _safe_convert_tensor(elem):
    try:
        return _convert_tensor(elem)
    except:  # noqa E722   ← catches KeyboardInterrupt, SystemExit, etc.
        if key == "overflowing_values":
            raise ValueError("Unable to create tensor returning overflowing values of different lengths.")
        raise ValueError(
            "Unable to create tensor, you should probably activate padding "
            "with 'padding=True' to have batched tensors with the same length."
        )
```

**Minimal reproducer — verified on Windows 11, Python 3.13.7, PyTorch 2.11.0+cu126:**

```python
# Exact pattern copied from image_processing_fuyu.py lines 99-108

def buggy_safe_convert_tensor(elem):
    def _convert_tensor(e):
        raise KeyboardInterrupt()  # simulates Ctrl+C during processing
    key = "pixel_values"
    try:
        return _convert_tensor(elem)
    except:  # noqa E722          # ← exact copy of the bug
        raise ValueError(
            "Unable to create tensor, you should probably activate padding "
            "with 'padding=True' to have batched tensors with the same length."
        )

try:
    buggy_safe_convert_tensor(None)
except ValueError as e:
    print(f"SWALLOWED: KeyboardInterrupt became -> ValueError: {e}")
except KeyboardInterrupt:
    print("KeyboardInterrupt escaped correctly")
```

**Output:**
```
SWALLOWED: KeyboardInterrupt became -> ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.
```

The `# noqa E722` comment on line 102 shows the bare `except:` was already flagged by linters — but suppressing the warning does not fix the runtime behavior.

### Expected behavior

`KeyboardInterrupt` (Ctrl+C) should propagate normally and stop the process cleanly.

Real conversion errors should surface with their original exception type and traceback — not be silently replaced by a generic padding message regardless of the actual cause.

**Suggested fix — one-line change, identical in both files:**

```diff
-     except:  # noqa E722
+     except Exception:
```

Optionally with exception chaining to preserve the original traceback:

```python
except Exception as exc:
    if key == "overflowing_values":
        raise ValueError(
            "Unable to create tensor returning overflowing values of different lengths."
        ) from exc
    raise ValueError(
        "Unable to create tensor, you should probably activate padding "
        "with 'padding=True' to have batched tensors with the same length."
    ) from exc
```

--- Comment by Abineshabee at 2026-05-06T11:08:23Z ---
If this is confirmed to be an issue, I’d be happy to open a PR with the proposed fix.

--- Comment by zucchini-nlp at 2026-05-06T13:42:49Z ---
Actually we can use `BatchFeature` now and don't need a special `FuyuBatchFeature` class. From what I see, Fuyu just needs to skip tensor conversion on some inputs keys which can be done as follows:

`BatchFeature(data, tensor_type, skip_tensor_conversion="overflowing_values")`

Please feel free to test and open a PR if that works

--- Comment by Abineshabee at 2026-05-06T15:49:55Z ---
@zucchini-nlp Tested and confirmed working.

```BatchFeature``` with ```skip_tensor_conversion=["images", "overflowing_values"]``` correctly handles ```_preprocess```, and all nested-list keys are properly skipped in ```preprocess_with_tokenizer_info```.

As a result, both ```FuyuBatchFeature``` classes have been removed from the respective files.

Opening the PR now.