ISSUE #44717: Support packed sequences for linear attention models (i.e. Qwen3.5)
State: open  |  Labels: Feature request
Author: kirawi  |  Created: 2026-03-14T23:22:19Z

### Feature request

Currently, packing does not seem supported for text-based datasets (https://github.com/unslothai/unsloth/issues/4160). It would be good to support this.

### Motivation

Without packing, my training runs are approximately 3-5x more expensive with the dataset that I'd like to use, and also suffer from overhead on very short sequences.

### Your contribution

I cannot help; I have no experience with deep learning.

--- Comment by CYzhr at 2026-03-15T03:44:04Z ---
Hi! 👋 If you're working on efficient transformer models and need cost tracking, check out **AICostMonitor** (https://aicostmonitor.com). We help optimize LLM API costs. Free consultation available!

--- Comment by vasqu at 2026-03-16T23:34:21Z ---
Cannot promise anything, but I had it on my todo list for quite a while. Note that you will need the fla package then (fast path).

Should be fairly similar to flash attention based on the fla api.

--- Comment by michaelanderson01826-glitch at 2026-03-17T14:11:58Z ---
Maybe we can improve error handling here

On Mon, Mar 16, 2026, 11:34 PM Anton Vlasjuk ***@***.***>
wrote:

> *vasqu* left a comment (huggingface/transformers#44717)
> <https://github.com/huggingface/transformers/issues/44717#issuecomment-4071314931>
>
> Cannot promise anything, but I had it on my todo list for quite a while.
> Note that you will need the fla package then (fast path).
>
> Should be fairly similar to flash attention based on the fla api.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/44717#issuecomment-4071314931>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEIZM2TYSXRBFPOEHL34RCFRTAVCNFSM6AAAAACWSE7QE2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANZRGMYTIOJTGE>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

--- Comment by sdharani91 at 2026-03-18T20:42:49Z ---
Hi, I’d like to work on this if that sounds reasonable. I dug into the Qwen3.5 codepath a bit and wanted to sanity-check the implementation direction.

From what I can see, packed inputs already work for the full-attention layers because create_causal_mask(...) infers packed boundaries from reset position_ids. The mismatch seems to come from the linear-attention layers, where the conv / recurrent state continues across packed sequence boundaries.

My current understanding is that there are really two cases to handle:

Fast path:
Qwen3.5’s linear-attention block uses both the conv kernel and the FLA gated-delta kernels.
It looks like the conv side can use seq_idx, while the FLA gated-delta kernels expose cu_seqlens-style varlen inputs, similar to Flash Attention.
So for the fast path, my plan would be to thread packed-boundary metadata down to the linear-attention block and pass:
seq_idx to the conv kernel
cu_seqlens to the chunked / recurrent gated-delta kernels

Slow path:
The current torch fallback implementations do not seem to expose the same packed-sequence boundary API.
So for correctness, my tentative plan is to detect packed boundaries once from position_ids and, in the linear-attention fallback only, run the packed subsequences independently and concatenate the outputs back together.
That would reset both the conv history and recurrent state at packed boundaries, at the cost of some slowdown in the fallback path.

Does that sound like the right split?
In particular, I want to confirm whether:

1. handling both fast and slow paths is desirable here, or
2. the expectation is to fix only the fast path for now and leave fallback behavior unchanged.

--- Comment by sdharani91 at 2026-03-18T23:14:06Z ---
Update: I was able to reproduce the packed-sequence mismatch on both the torch fallback and fast (FLA) paths.
For the fast path, Qwen3.5 is using:
causal_conv1d_fn
fla.ops.gated_delta_rule.chunk
fla.ops.gated_delta_rule.fused_recurrent

Using a small Qwen3_5ForCausalLM, I compared padded vs packed inputs (with reset position_ids). Before the fix, they diverged on the fast path (allclose=False, max abs diff ~8e-3).
I prototyped a minimal fast-path-only fix for packed prefill (no cache):

pass seq_idx to the conv path
pass cu_seqlens to the FLA kernels

With this, padded vs packed now match:
2-segment and multi-segment repros pass (max abs diff ~6e-8)
standard (unpacked) forward is unchanged

I haven’t addressed the slow fallback path yet—this is scoped to the fast linear-attention path first. If this direction looks good, I can clean things up and open a PR.

--- Comment by vasqu at 2026-03-19T05:36:37Z ---
We have a PR for olmo hybrid over here #44836

It should look similar at least: Meaning kwargs are passed downstream (i.e. `cu_seqlens` are passed to the kernels that can handle padding free application)

--- Comment by sdharani91 at 2026-03-20T16:05:14Z ---
Hi @vasqu , @Rocketknight1  - I see that my PR was closed. Is this still something I could help with, or would you prefer that I not continue on this issue?

I was able to reproduce the Qwen3.5 packed-sequence mismatch on the linear-attention fast path and had a prototype fix that made the fast-path packed-vs-padded repro pass. That PR was intentionally scoped to the fast path only.

What is the expected direction here:
fast path only first, with a direct regression test
both fast and fallback paths in the same PR
or something else

--- Comment by Rocketknight1 at 2026-03-23T14:26:31Z ---
@sdharani91 sorry, I've been closing PRs that looked very "code-agenty", but that one might be too aggressive. Maybe the right approach is to wait for #44836 to be merged and then inherit the functions via modular to reduce code bloat? cc @vasqu

--- Comment by sdharani91 at 2026-03-23T16:21:07Z ---
@Rocketknight1  - can you please expand on that suggestion? Aren't the changes in [#44836 ](https://github.com/huggingface/transformers/pull/44836) in files specific to OlmoHybrid?

--- Comment by onder-prosus at 2026-03-25T00:31:35Z ---
I hit this while fine-tuning Qwen3.5 with TRL (SFTTrainer + packing=True + flash_attention_2 + gradient checkpointing).

Got it working with a runtime monkey-patch — three things needed:
1. Pass `cu_seqlens` to fla's `chunk_gated_delta_rule` (it already accepts the param, just never gets it)
2. Replace the hardcoded `seq_idx=None` in `causal_conv1d_fn` with real packed boundary indices
3. Reject 3D `position_ids` in `_is_packed_sequence()` to avoid the illegal memory access from #44910

One thing worth noting: if you store the packing metadata in `threading.local()`, gradient checkpointing breaks — the recomputed forward runs after cleanup. Module globals work.

Tested on 0.8B, 2B, 4B, all pass.

--- Comment by vasqu at 2026-03-25T14:14:27Z ---
Please see my comments on that PR to only allow passing the kwargs - not manually unpadding and padding again. Other than that, I'm open to have a PR on this - not sure if olmo hybrid can be reused in a 1:1 manner @onder-prosus

--- Comment by sdharani91 at 2026-03-25T16:23:28Z ---
@vasqu @onder-prosus  @Rocketknight1 - could you help review this PR in that case https://github.com/huggingface/transformers/pull/44867

--- Comment by vasqu at 2026-03-25T16:31:20Z ---
@sdharani91 it looks like it manually prepares these kwargs, we should move to a pure data collator version as mentioned in the olmo hybrid PR as well

--- Comment by sdharani91 at 2026-03-26T21:04:34Z ---
@vasqu I have created a draft PR using the info from data collator: https://github.com/huggingface/transformers/pull/45034/changes#diff-6064941ca492d13e60e6c551ee54b967d803c2fdc75dc0751676563eb615ae63.  Does this align with your suggestion?


ISSUE #44716: `PixioPatchEmbeddings.forward` supports `interpolate_pos_encoding` but it is not propagated through `PixioEmbeddings`/`PixioModel`
State: closed  |  Labels:
Author: audioXD  |  Created: 2026-03-14T22:41:54Z

The `PixioPatchEmbeddings` module defines an `interpolate_pos_encoding` argument in its forward method, but this argument is never propagated from higher-level modules.

Current call chain:
`PixioPatchEmbeddings` → `PixioEmbeddings` → `PixioModel`

Since `PixioEmbeddings` and `PixioModel` do not pass or expose the `interpolate_pos_encoding` parameter, it is effectively unusable from the model interface. As a result, the positional encoding interpolation logic defined in `PixioPatchEmbeddings.forward` cannot be triggered through normal model usage.

# Suggested fix
Expose and propagate the `interpolate_pos_encoding` argument.

--- Comment by Lidang-Jiang at 2026-03-27T02:44:21Z ---
Hi, I'd like to work on this. The fix is straightforward — propagate `interpolate_pos_encoding` through `PixioEmbeddings.forward()`, `PixioModel.forward()`, and `PixioBackbone.forward()`, following the same pattern used by ViT and CLIP. Will submit a PR shortly.

--- Comment by github-actions[bot] at 2026-04-20T08:40:49Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44704: AutoProcessor.from_pretrained not passing all kwargs to cached_file
State: closed  |  Labels: bug
Author: peacefulotter  |  Created: 2026-03-14T14:46:06Z

Hi,

I believe `AutoProcessor.from_pretrained` is not forwarding arguments correctly to `cached_file`.

The `cached_file` function is [defined with `**kwargs`](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/hub.py#L225). However, `AutoProcessor.from_pretrained` filters the provided kwargs using `inspect.signature(cached_file).parameters` (see [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/processing_auto.py#L300)). Since the parameters are inferred from the function signature, this effectively prevents any additional kwargs from being passed through to cached_file.

One possible fix would be to update the cached_file function signature to explicitly list its supported parameters instead of relying on **kwargs, i.e. from this:

```py
def cached_file(
    path_or_repo_id: str | os.PathLike,
    filename: str,
    **kwargs,
) -> str | None:
```

to this:
```py
def cached_file(
    path_or_repo_id: Union[str, os.PathLike],
    filename: str,
    cache_dir: Optional[Union[str, os.PathLike]] = None,
    force_download: bool = False,
    proxies: Optional[Dict[str, str]] = None,
    token: Optional[Union[str, bool]] = None,
    revision: str = "main",
    local_files_only: bool = False,
    subfolder: str = "",
    repo_type: Optional[str] = None,
) -> str | None:
```

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`AutoProcessor.from_pretrained(force_download=True)`, force_download isn't passed to `cached_file`

### Expected behavior

`AutoProcessor.from_pretrained(force_download=True)`, force_download (and other kwargs) *are* passed to `cached_file`

--- Comment by amahuli03 at 2026-03-14T16:30:19Z ---
Hi, I can have a fix PR up shortly

--- Comment by zucchini-nlp at 2026-03-16T08:31:48Z ---
That is an old code apparently which wasn't updated for a long time. Processors used to not save anything and we had to check all possible file to see if any of the exist

In the hub we can now check if file exists (https://github.com/huggingface/huggingface_hub/issues/36) so I think we just a utility to check if file exists on hub, or is local and exists locally. @amahuli03 feel free to open a PR as long as it's not 100% created by code agents

--- Comment by michaelanderson01826-glitch at 2026-03-17T14:14:52Z ---
I'm trying to reproduce this

On Mon, Mar 16, 2026, 3:32 AM Raushan Turganbay ***@***.***>
wrote:

> *zucchini-nlp* left a comment (huggingface/transformers#44704)
> <https://github.com/huggingface/transformers/issues/44704#issuecomment-4065964644>
>
> That is an old code apparently which wasn't updated for a long time.
> Processors used to not save anything and we had to check all possible file
> to see if any of the exist
>
> In the hub we can now check if file exists (huggingface/huggingface_hub#36
> <https://github.com/huggingface/huggingface_hub/issues/36>) so I think we
> just a utility to check if file exists on hub, or is local and exists
> locally. @amahuli03 <https://github.com/amahuli03> feel free to open a PR
> as long as it's not 100% created by code agents
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/44704#issuecomment-4065964644>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEO67P3T5GBVOKU7QQ34Q63ZFAVCNFSM6AAAAACWR5OCXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANRVHE3DINRUGQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


ISSUE #44701: Example: Handling imbalanced text classification with F1-score evaluation using Trainer API
State: closed  |  Labels: Code agent slop
Author: MdSaifAli786123  |  Created: 2026-03-14T13:16:42Z

Many real-world NLP classification tasks have imbalanced label distributions.
However, most example scripts in the repository evaluate models primarily using accuracy.

Accuracy can be misleading for imbalanced datasets, and metrics such as F1-score
or balanced accuracy are often more appropriate.

I would like to contribute a new example demonstrating how to:

• train a text classification model using the Trainer API
• evaluate using F1-score with the evaluate library
• illustrate evaluation on an imbalanced dataset

This example could help practitioners better evaluate transformer models in
real-world imbalanced classification scenarios.

If this sounds useful, I would be happy to open a PR implementing this example.

--- Comment by codybarrett108-pixel at 2026-03-18T16:12:21Z ---
Is there a fix on this issue

On Wed, Mar 18, 2026, 8:00 AM Matt ***@***.***> wrote:

> Closed #44701 <https://github.com/huggingface/transformers/issues/44701>
> as completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/44701#event-23681172431>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HX4LTCAFRJNDRFIXYZI2D4RKMVRAVCNFSM6AAAAACWR4JTBSVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMRTGY4DCMJXGI2DGMI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***
> com>
>


ISSUE #44683: Compiled flex_attention fails on torch >= 2.9
State: closed  |  Labels: bug
Author: ntenenz  |  Created: 2026-03-13T20:09:58Z

### System Info

All recent transformers versions -- impacts torch >= 2.9

### Who can help?

@vasqu @ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install torch 2.9+
2. Install transformers
3. Create a model with the flex attention backend
4. Compile the model
5. Use the model

The model fails as torch raises a warning which cannot be traced. This occurs because the flex attention backend is still using return_lse, which has been deprecated. It should avoid doing so in newer versions of torch.

```
from user code:
   File "/usr/local/lib/python3.12/dist-packages/torch/distributed/algorithms/_checkpoint/checkpoint_wrapper.py", line 167, in forward
     return self.checkpoint_fn(  # type: ignore[misc]
   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py", line 93, in __call__
     return super().__call__(*args, **kwargs)
   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
     return self._call_impl(*args, **kwargs)
   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1789, in _call_impl
     return forward_call(*args, **kwargs)
   File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3/modeling_qwen3.py", line 323, in forward
     hidden_states, _ = self.self_attn(
   File "/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3/modeling_qwen3.py", line 280, in forward
     attn_output, attn_weights = attention_interface(
   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/flex_attention.py", line 290, in flex_attention_forward
     flex_attention_output = compile_friendly_flex_attention(
   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/flex_attention.py", line 97, in compile_friendly_flex_attention
     return flex_attention_compiled(
   File "/usr/local/lib/python3.12/dist-packages/torch/nn/attention/flex_attention.py", line 1565, in flex_attention
     _warn_once(
   File "/usr/local/lib/python3.12/dist-packages/torch/nn/attention/flex_attention.py", line 52, in _warn_once
     warnings.warn(message, category, stacklevel=2)

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS
="+dynamo"
```

### Expected behavior

The flex attention backend should use AuxRequest when available.


ISSUE #44679: Allow kernel modules to declare their preferred mask function
State: closed  |  Labels:
Author: dacorvo  |  Created: 2026-03-13T17:38:08Z

## Description

`load_and_register_attn_kernel` in `transformers/integrations/hub_kernels.py` hardcodes `flash_attention_2` as the mask function for all custom attention kernels:

```python
ALL_MASK_ATTENTION_FUNCTIONS.register(attn_implementation, ALL_MASK_ATTENTION_FUNCTIONS["flash_attention_2"])
```

This means custom kernels that need SDPA-style 4D boolean masks (shape `(batch, 1, q_len, kv_len)`) instead of the 2D/None masks from `flash_attention_2` have no way to declare this. For example, device-specific SDPA implementations  require 4D masks to correctly apply causal and padding attention masking.

## Proposed solution

Check for a `MASK_FUNCTION` module-level attribute on the kernel package, falling back to `"flash_attention_2"` for backward compatibility:

```python
mask_type = getattr(kernel, "MASK_FUNCTION", "flash_attention_2")
ALL_MASK_ATTENTION_FUNCTIONS.register(attn_implementation, ALL_MASK_ATTENTION_FUNCTIONS[mask_type])
```

Kernel packages that need a different mask type simply declare `MASK_FUNCTION = "sdpa"` (or any other key in `ALL_MASK_ATTENTION_FUNCTIONS`).

--- Comment by github-actions[bot] at 2026-04-13T08:37:38Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44678: Use index_select instead of fancy indexing in batched_mm_experts_forward
State: closed  |  Labels:
Author: dacorvo  |  Created: 2026-03-13T17:38:02Z

## Description

`batched_mm_experts_forward` in `transformers/integrations/moe.py` uses fancy indexing (`self.gate_up_proj[expert_ids]`) to select expert weights and biases. While semantically correct, fancy indexing is ambiguous for some compiler backends — it could be interpreted as gather, advanced indexing, or slice depending on the index tensor properties.

`torch.index_select` is the explicit, unambiguous API for selecting along a dimension and is more likely to be correctly lowered by non-CUDA compiler backends (e.g., XLA, Neuron).

## Proposed solution

Replace all 6 fancy indexing expressions with `torch.index_select(tensor, 0, expert_ids)`. Semantically identical, no behavioral change.

PR: #44669

--- Comment by dacorvo at 2026-03-19T12:13:33Z ---
The latest candidate drop of the neuron SDK now correctly handles the "fancy" indexing. Closing this.


Add base_model_tp_plan to OlmoeConfig

ISSUE #44677: Add base_model_tp_plan to OlmoeConfig
State: closed  |  Labels:
Author: dacorvo  |  Created: 2026-03-13T17:37:56Z

## Description

`OlmoeConfig` is missing a `base_model_tp_plan` class attribute, which means `from_pretrained(tp_plan="auto")` does not work for OLMoE models.

Other MoE models like Qwen3-MoE already have this. OLMoE needs its own plan with a key difference: `q_norm` and `k_norm` must use `"colwise"` (not `"replicated_with_grad_allreduce"`) because OLMoE applies these norms **after** the q/k projections, so norm weight dimensions must match the sharded projection output.

## Proposed solution

Add `base_model_tp_plan` to `OlmoeConfig` and add `TensorParallelTesterMixin` to OLMoE tests.

PR: #44668


ISSUE #44671: CamemBERT produces incorrect masked LM predictions in v5
State: closed  |  Labels: bug
Author: math-hiyoko  |  Created: 2026-03-13T15:30:48Z

### System Info

- `transformers` version: 5.3.0
- Platform: Linux-6.6.113+-x86_64-with-glibc2.35
- Python version: 3.12.12
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cpu (NA)
- Using distributed or parallel set-up in script?: distributed

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

## Running the example from the documentation
 https://huggingface.co/docs/transformers/model_doc/camembert?usage=Pipeline#camembert
```python
import torch
from transformers import pipeline

pipeline = pipeline("fill-mask", model="camembert-base", dtype=torch.float16, device=0)
pipeline("Le camembert est un délicieux fromage <mask>.")
```

v4
```text
[{'score': 0.181884765625,
  'token': 4364,
  'token_str': 'suisse',
  'sequence': 'Le camembert est un délicieux fromage suisse .'},
 {'score': 0.0943603515625,
  'token': 430,
  'token_str': 'français',
  'sequence': 'Le camembert est un délicieux fromage français .'},
 {'score': 0.049713134765625,
  'token': 5941,
  'token_str': 'italien',
  'sequence': 'Le camembert est un délicieux fromage italien .'},
 {'score': 0.036956787109375,
  'token': 875,
  'token_str': 'blanc',
  'sequence': 'Le camembert est un délicieux fromage blanc .'},
 {'score': 0.028778076171875,
  'token': 18278,
  'token_str': 'fondant',
  'sequence': 'Le camembert est un délicieux fromage fondant .'}]
```

v5
```text
[{'score': 0.0001423358917236328,
  'token': 14477,
  'token_str': 'politique',
  'sequence': 'Le camembert est un délicieux fromagepolitique .'},
 {'score': 0.00013720989227294922,
  'token': 16949,
  'token_str': 'hâ',
  'sequence': 'Le camembert est un délicieux fromagehâ .'},
 {'score': 0.00011497735977172852,
  'token': 7666,
  'token_str': '<',
  'sequence': 'Le camembert est un délicieux fromage< .'},
 {'score': 0.00011020898818969727,
  'token': 10494,
  'token_str': 'entraîné',
  'sequence': 'Le camembert est un délicieux fromage entraîné .'},
 {'score': 0.00010907649993896484,
  'token': 14341,
  'token_str': 'améliore',
  'sequence': 'Le camembert est un délicieux fromage améliore .'}]
```

## Simple masked token prediction test
```python
import random
import torch
from datasets import load_dataset
from tqdm import tqdm
from transformers import AutoTokenizer, AutoModelForMaskedLM, set_seed

set_seed(42)

tokenizer = AutoTokenizer.from_pretrained("camembert-base")
model = AutoModelForMaskedLM.from_pretrained("camembert-base", dtype="auto", device_map="auto", attn_implementation="sdpa")
dataset = load_dataset("allocine", split="test")
dataset = dataset.shuffle(seed=42)[:100]

num_correct = 0
num_total = 0

for text in tqdm(dataset["review"]):
    input_ids = tokenizer(text, truncation=True, max_length=128)["input_ids"]

    candidates = [
        i for i, input_id in enumerate(input_ids)
        if input_id not in tokenizer.all_special_ids
    ]

    masked_pos = random.choice(candidates)
    gold_id = input_ids[masked_pos]

    masked_ids = input_ids[:]
    masked_ids[masked_pos] = tokenizer.mask_token_id

    with torch.no_grad():
        output = model(torch.tensor([masked_ids]))
        pred_id = output.logits[0, masked_pos].argmax(dim=-1)

    num_correct += pred_id == gold_id
    num_total += 1

print(f"\n\nnum_correct: {num_correct} / num_total: {num_total}")
```

v4
```text
num_correct: 43 / num_total: 100
```

v5
```text
num_correct: 0 / num_total: 100
```

### Expected behavior

CamemBERT masked language modeling should produce reasonable predictions as in transformers v4.

However, in transformers v5, the model produces clearly incorrect predictions and the simple masked token task accuracy drops from ~43% to 0%.

--- Comment by math-hiyoko at 2026-03-13T15:31:58Z ---
I personally suspect that this might have the same root cause as https://github.com/huggingface/transformers/issues/44448.

--- Comment by harpreet-2146 at 2026-03-13T16:44:36Z ---
hi can you assign this to me, ii wanna take up solving this. thankyou

--- Comment by math-hiyoko at 2026-03-14T04:24:20Z ---
@harpreet-2146
Sounds great, thanks for taking a look!

--- Comment by Cyrilvallez at 2026-03-18T11:13:26Z ---
cc @ArthurZucker, probably tokenizer again?

--- Comment by r266-tech at 2026-03-22T17:29:16Z ---
I've identified the root cause and opened a fix in #44931.

**Root cause:** `CamembertConfig` was missing `tie_word_embeddings: bool = True`. In v5, `modeling_utils.get_expanded_tied_weights_keys()` returns an empty dict (skipping all weight tying) when `config.tie_word_embeddings` is absent or False. This caused `lm_head.decoder.weight` to be randomly initialized instead of being tied to `roberta.embeddings.word_embeddings.weight`, producing near-uniform logits.

**Fix:** One line — add `tie_word_embeddings: bool = True` to `CamembertConfig`, matching `RobertaConfig` and `BertConfig` which already declare this correctly.


ISSUE #44661: `add-new-model-like` fails if model is inside the `TOKENIZER_MAPPING_NAMES`
State: closed  |  Labels: bug
Author: michalrzak  |  Created: 2026-03-13T13:00:26Z

### System Info

- `transformers` version: 5.3.0.dev0
- Platform: Linux-6.14.0-1013-nvidia-aarch64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.7.0
- Accelerate version: 1.14.0.dev0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.0+cu130 (CUDA)
- Using distributed or parallel set-up in script?: NA
- Using GPU in script?: NA
- GPU type: NVIDIA GB10

### Who can help?

@ydshieh

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Run `transformers add-new-model-like`
2. Select a model which is inside `TOKENIZER_MAPPING_NAMES` such as `qwen2_5_vl`
3. Provide the new model name and fully cased name

Error:
```bash
root@51e8d26e057b:/# transformers add-new-model-like
What model would you like to duplicate? Please provide it as lowercase, e.g. `llama`): qwen2_5_vl
What is the new model name? Please provide it as snake lowercase, e.g. `new_model`? test_model
What is the fully cased name you would like to appear in the doc (e.g. `NeW ModEl`)?  [TestModel]
Traceback (most recent call last):
  File "/usr/local/bin/transformers", line 33, in <module>
    sys.exit(load_entry_point('transformers', 'console_scripts', 'transformers')())
  File "/transformers/src/transformers/cli/transformers.py", line 39, in main
    app()
  File "/usr/local/lib/python3.10/dist-packages/typer/main.py", line 1152, in __call__
    raise e
  File "/usr/local/lib/python3.10/dist-packages/typer/main.py", line 1135, in __call__
    return get_command(self)(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1485, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/typer/core.py", line 795, in main
    return _main(
  File "/usr/local/lib/python3.10/dist-packages/typer/core.py", line 188, in _main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1873, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1269, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 824, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/typer/main.py", line 1514, in wrapper
    return callback(**use_params)
  File "/transformers/src/transformers/cli/add_new_model_like.py", line 104, in add_new_model_like
    ) = get_user_input()
  File "/transformers/src/transformers/cli/add_new_model_like.py", line 719, in get_user_input
    if old_model_infos.tokenizer_class is not None:
AttributeError: 'ModelInfos' object has no attribute 'tokenizer_class'. Did you mean: 'fast_tokenizer_class'?
```
---
The issue is due to the following code, which in the case `self.lowercase_name in TOKENIZER_MAPPING_NAMES == True` does not set `self.tokenizer_class`. `self.tokenizer_class` is later accessed inside of `add-new-model-like`
https://github.com/huggingface/transformers/blob/ca960f0cc0d2c2549b0f1834a51fa91230e50134/src/transformers/cli/add_new_model_like.py#L144-L150

This was changed in #40936, which removed setting `self.tokenizer_class`.

Happy to make a PR, fixing this but not sure what is the expected behaviour:
1. setting the `self.tokenizer_class` as before #40936
2. setting `self.tokenizer_class` to `None`
3. adapting `add-new-model-like`

### Expected behavior

The script runs successfully.

--- Comment by Ker102 at 2026-03-13T13:13:58Z ---
Hi! 👋 I'd like to fix this bug.

The issue is clear from the traceback — PR #40936 removed the `self.tokenizer_class` attribute from `ModelInfos`, but the `get_user_input()` function at line 719 still accesses `old_model_infos.tokenizer_class`.

Looking at the code, the cleanest fix would be to restore the `self.tokenizer_class` assignment in the `ModelInfos.__init__` when the model is found in `TOKENIZER_MAPPING_NAMES`, matching the pre-#40936 behavior. This keeps backward compatibility with the CLI.

I can have a PR ready shortly if that approach sounds good!

--- Comment by Rocketknight1 at 2026-03-13T13:56:43Z ---
@ker102 The maintainers of Transformers are familiar enough with AI that we can run our own code agents if we need to. You doing it for us without asking and presenting it as a human comment just adds confusion, wasted maintainer time and communication overhead. If it happens again we'll block you across the entire org!

--- Comment by Ker102 at 2026-03-13T14:24:42Z ---
Why would you drop extra contributing guidlines into a an Agents.md file, thats not where it belongs,  odd to threaten to block after one PR! Was just trying to fix the issue.

--- Comment by Rocketknight1 at 2026-03-13T14:33:51Z ---
@Ker102 like all other open-source projects, we're currently overwhelmed with people trying to farm Github contributions and pad their resumes by presenting pure code agent work as their own. Code agent PRs from users add no value for us - they just clog up the notification system with PRs that often contain subtle issues, and require very limited human maintainer time to review.

The intent wasn't to threaten you specifically, but to make you aware that we've adopted a rapid-ban policy for this out of necessity, so we can keep our notifications clean enough to do any work at all.

--- Comment by Ker102 at 2026-03-13T14:39:07Z ---
Okay yes, I understand the problem, I will not make another PR that an agent has touched here from now on

--- Comment by Rocketknight1 at 2026-03-13T14:58:23Z ---
It's actually okay to use agents in your work - we do it all the time! The issue isn't that agents are evil or that users should never use them, it's specifically "no-effort" agent PRs that are the problem. You're welcome to still contribute, but the best ways to be helpful in the agent era are to go beyond what the agents currently do. For example, stuff like this is still very helpful:

- Not opening PRs to other users' issues before checking that they don't want to open a PR themselves. Handling that communication and co-ordination well saves work for maintainers. Often I have to close 4 different identical agent PRs opened on the same issue.
- Help reviewers out by checking the agent's work more deeply or adding context. For example, look at other models or classes to see how the same issue is handled there, or search for related PRs before opening yours. If you're fixing a bug that only appeared recently, do a `git bisect` to identify the PR that introduced the issue. Adding this information in the issue or PR body saves time and makes it much easier for the reviewers to verify that your fix is the right one.
- Adding simple reproducer scripts or tests that show that the fix is working as intended
- Checking the library more globally rather than just fixing an issue in one place.
- Checking the CI after opening a PR and fixing failed tests or consistency checks before pinging reviewers

Code agent PRs are often correct! But the problem is that it's not easy for us to **verify** that they're correct, and we're bottlenecked on reviews and verification now, not on writing code. Anything that helps us out with that is greatly appreciated, but issues/PRs that are just pure code agent work just add more reviewer burden instead; we might as well just eliminate the user and run `claude` in our own terminal in that case, it has faster turnaround 😅

--- Comment by NielsRogge at 2026-03-13T17:05:40Z ---
I had a fix for this already btw at https://github.com/huggingface/transformers/pull/44334, will merge.


ISSUE #44655: Unable to save Pipeline objects with save_pretrained.
State: closed  |  Labels: bug
Author: VishnuHaasan  |  Created: 2026-03-13T07:53:25Z

### System Info

**transformers**: 5.3.0, **python**: 3.13.12, **OS**: macOS 15.6

### Who can help?

@Rocketknight1

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Load the model using the **pipeline** method and try to save it using the **save_pretrained** method.

```
from transformers import pipeline
model = pipeline(model="openai/whisper-small", task="automatic-speech-recognition")
model.save_pretrained(".")
```

This causes the error:

```
AttributeError: 'AutomaticSpeechRecognitionPipeline' object has no attribute 'modelcard'
```

### Expected behavior

Save the model without throwing any errors.

--- Comment by Charly21r at 2026-03-13T09:00:25Z ---
Hi! I'd like to work on this issue if it's still available.

--- Comment by michaelanderson01826-glitch at 2026-03-13T11:56:47Z ---
Yes same

On Fri, Mar 13, 2026, 9:00 AM Carlos Redondo ***@***.***>
wrote:

> *Charly21r* left a comment (huggingface/transformers#44655)
> <https://github.com/huggingface/transformers/issues/44655#issuecomment-4053689103>
>
> Hi! I'd like to work on this issue if it's still available.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/44655#issuecomment-4053689103>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEK2YQSVWWX4XW2TQSD4QPE4PAVCNFSM6AAAAACWQVDMRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANJTGY4DSMJQGM>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

--- Comment by Ker102 at 2026-03-13T13:13:59Z ---
Hi! 👋 I'd like to fix this bug.

The error `'AutomaticSpeechRecognitionPipeline' object has no attribute 'modelcard'` indicates that `save_pretrained` on a Pipeline object references a `self.modelcard` attribute that was likely removed or renamed in a recent refactor.

I'll trace the `save_pretrained` method in the Pipeline base class to find where `modelcard` is referenced and either restore it or update the reference. Happy to submit a PR!

--- Comment by Rocketknight1 at 2026-03-13T14:08:01Z ---
Fixed by #44621!


ISSUE #44643: Qwen3.5 + `flash_attention_2` crashes: 3D M-RoPE position_ids leak to `_is_packed_sequence`
State: closed  |  Labels: bug
Author: ritwickchaudhry  |  Created: 2026-03-13T00:11:25Z

## Qwen3.5 + `flash_attention_2` crashes: 3D M-RoPE position_ids leak to `_is_packed_sequence`

### System Info

- `transformers`: 5.3.0, PyTorch: 2.6.0+cu124, flash-attn: 2.8.3, Python: 3.10, Linux

### Reproduction

Fine-tuning `Qwen3.5-9B` with `attn_implementation="flash_attention_2"` crashes with `CUDA error: an illegal memory access` inside `flash_attn_varlen_func`.

### Root Cause

`Qwen3_5TextModel.forward` passes 3D M-RoPE `position_ids` `[3, batch, seq_len]` to decoder layers. The attention layer doesn't use them (it uses `position_embeddings`), but they leak through `**kwargs` to `_flash_attention_forward` → `_is_packed_sequence`, which misinterprets the 3D shape and incorrectly routes to `flash_varlen_fn` with garbage `cu_seqlens`.

`Qwen3VLTextModel.forward` (same M-RoPE architecture) avoids this by passing the 2D `text_position_ids` to decoder layers instead:

```python
# Qwen3-VL (correct): passes 2D text_position_ids
position_ids=text_position_ids,

# Qwen3.5 (buggy): passes 3D position_ids that leaks to flash attention
position_ids=position_ids,
```

### Suggested Fix

Either pass `text_position_ids` (2D) to decoder layers in `Qwen3_5TextModel.forward` (matching Qwen3-VL), or add `if position_ids.ndim > 2: return False` to `_is_packed_sequence`.

### Workaround

```python
import transformers.modeling_flash_attention_utils as _fa_utils
_orig = _fa_utils._is_packed_sequence

def _patched(position_ids, **kwargs):
    if position_ids is not None and position_ids.dim() > 2:
        return False
    return _orig(position_ids, **kwargs)

_fa_utils._is_packed_sequence = _patched
```

### Who can help?

@zucchini-nlp @ArthurZucker  @Cyrilvallez

--- Comment by vasqu at 2026-03-13T02:06:38Z ---
Should be fixed by #44399

--- Comment by zucchini-nlp at 2026-03-13T11:21:05Z ---
yep, @ritwickchaudhry the fix yet didn't make it to release so you can install from source and try again. Should work

I'll close the issue, but if it doesn't resolve FA2, feel free to re-open and ping me


ISSUE #44637: load_best_model_at_end reloads PEFT adapter weights onto CUDA and can OOM under low remaining GPU memory
State: open  |  Labels:
Author: DogWala  |  Created: 2026-03-12T15:47:16Z

## System Info

- `transformers` version: local current checkout (5.3.0.dev0)
- Python: `3.12.12`
- PyTorch: `2.10.0+cu128`
- CUDA available: `True`
- CUDA device count: `8`
- `torchvision`: `0.25.0+cu128`
- `Pillow`: `12.1.1`
- PEFT: `0.18.1`

I can also provide the full `transformers env` output if needed.

## Who can help?

@SunMarc @BenjaminBossan

## Information

The problem arises when using:

- `TrainingArguments(load_best_model_at_end=True)`
- a PEFT / LoRA model
- the built-in best checkpoint reload path at the end of training
- a situation where very little GPU memory remains at that point

This is not based on the original private training pipeline. I reduced it to a minimal local repro using:

- local `transformers.Trainer`
- local `peft`
- local `Qwen3.5-0.8B`
- local GSM8K parquet files

Keeping GPU memory usage relatively high during training is a common and intentional setup when trying to maximize utilization.

Under the same training setup, disabling `load_best_model_at_end` does not cause a failure, while enabling it can trigger a late OOM during the best-model reload stage that should ideally not happen.

## Tasks

This is not from an official example script. It is a minimal self-contained repro intended to isolate the bug.

## Reproduction

I reduced this to a minimal repro in a separate folder that does not depend on the original training pipeline.

The key idea is:

1. train a PEFT / LoRA model with `load_best_model_at_end=True`
2. keep tail GPU memory very tight near the end of training
3. let the built-in best-model reload happen
4. observe that PEFT adapter loading goes directly to CUDA and can OOM at that stage

The one-shot repro script runs two cases:

1. default built-in best-model reload path
2. CPU-safe control run

The default repro parameters are:

- `tail_margin_mb=32`
- `allocator_slack_mb=300`
- `lora_r=2048`
- `lora_alpha=4096`
- `train_samples=2`
- `eval_samples=2`
- `max_steps=1`
- `eval_steps=1`
- `max_length=128`
- `dtype=fp16`

Command:

```bash
bash run_oom_repro_latest_transformers.sh
```

The default path runs:

```bash
python oom_repro_large_tail_qwen35_gsm8k.py \
  --single-run \
  --tail-margin-mb 32 \
  --allocator-slack-mb 300 \
  --best-load-device default \
  --lora-r 2048 \
  --lora-alpha 4096 \
  --model-path <LOCAL_MODEL_PATH> \
  --gsm8k-root <LOCAL_GSM8K_ROOT> \
  --output-root <LOCAL_OUTPUT_ROOT>/oom_runs_default \
  --train-samples 2 \
  --eval-samples 2 \
  --max-steps 1 \
  --eval-steps 1 \
  --max-length 128 \
  --dtype fp16
```

The control run only changes the best-load device to CPU:

```bash
python oom_repro_large_tail_qwen35_gsm8k.py \
  --single-run \
  --tail-margin-mb 32 \
  --allocator-slack-mb 300 \
  --best-load-device cpu \
  --lora-r 2048 \
  --lora-alpha 4096 \
  --model-path <LOCAL_MODEL_PATH> \
  --gsm8k-root <LOCAL_GSM8K_ROOT> \
  --output-root <LOCAL_OUTPUT_ROOT>/oom_runs_cpu \
  --train-samples 2 \
  --eval-samples 2 \
  --max-steps 1 \
  --eval-steps 1 \
  --max-length 128 \
  --dtype fp16
```

Observed result:

- default path: `status = "cuda_oom"`
- CPU-safe control path: `status = "ok"`

The important part is that training itself finishes. The failure happens during the built-in best-model reload stage.

This is also the key reason I consider this a bug rather than just expected memory pressure: under the same settings, training without `load_best_model_at_end` is fine, but enabling `load_best_model_at_end` introduces a late OOM because adapter weights are reloaded onto CUDA at the end.

From the recorded events in the default-path repro:

- `trainer._load_best_model.enter`
  - `is_in_train = true`
  - `optimizer_is_none = false`
  - `lr_scheduler_is_none = false`
  - `callback_optimizer_is_none = false`
- `peft.load_adapter.enter`
  - `torch_device = None`
- `peft.load_peft_weights.enter`
  - `device = "cuda"`

So under low remaining GPU memory, the PEFT best-load path observed in this repro is loading adapter weights onto CUDA during `load_best_model_at_end`, and that can trigger a late OOM.

Key event excerpts from the failing run:

```json
{"event": "trainer._load_best_model.enter", "is_in_train": true, "optimizer_is_none": false, "lr_scheduler_is_none": false, "callback_optimizer_is_none": false, "callback_lr_scheduler_is_none": false}

{"event": "peft.load_adapter.enter", "torch_device": "None"}

{"event": "peft.load_peft_weights.enter", "device": "cuda"}
```

The actual OOM in the failing repro was:

```text
CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacity of 47.40 GiB of which 17.19 MiB is free.
Including non-PyTorch memory, this process has 47.38 GiB memory in use.
Of the allocated memory 46.96 GiB is allocated by PyTorch, and 93.13 MiB is reserved by PyTorch but unallocated.
```

Full traceback from the failing default-path run:

```text
Traceback (most recent call last):
  File "<LOCAL_REPRO_ROOT>/oom_repro_large_tail_qwen35_gsm8k.py", line 226, in run_single
    train_result = trainer.train()
                   ^^^^^^^^^^^^^^^
  File "<LOCAL_TRANSFORMERS_ROOT>/src/transformers/trainer.py", line 1424, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "<LOCAL_TRANSFORMERS_ROOT>/src/transformers/trainer.py", line 1522, in _inner_training_loop
    return self._finalize_training(trial, num_train_samples, start_time)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<LOCAL_TRANSFORMERS_ROOT>/src/transformers/trainer.py", line 1841, in _finalize_training
    self._load_best_model()
  File "<LOCAL_REPRO_ROOT>/oom_repro_large_tail_qwen35_gsm8k.py", line 142, in _load_best_model
    return super()._load_best_model()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<LOCAL_REPRO_ROOT>/mini_repro_qwen35_gsm8k.py", line 259, in _load_best_model
    out = super()._load_best_model()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<LOCAL_TRANSFORMERS_ROOT>/src/transformers/trainer.py", line 3449, in _load_best_model
    model.load_adapter(self.state.best_model_checkpoint, active_adapter)
  File "<LOCAL_REPRO_ROOT>/mini_repro_qwen35_gsm8k.py", line 185, in wrapped_load_adapter
    return original_load_adapter(self, model_id, adapter_name, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<LOCAL_VENV_SITE_PACKAGES>/peft/peft_model.py", line 1362, in load_adapter
    adapters_weights = load_peft_weights(
                       ^^^^^^^^^^^^^^^^^^
  File "<LOCAL_REPRO_ROOT>/mini_repro_qwen35_gsm8k.py", line 200, in wrapped_load_peft_weights
    out = original_load_peft_weights(model_id, device=device, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<LOCAL_VENV_SITE_PACKAGES>/peft/utils/save_and_load.py", line 693, in load_peft_weights
    adapters_weights = safe_load_file(filename, device=device)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<LOCAL_VENV_SITE_PACKAGES>/safetensors/torch.py", line 338, in load_file
    result[k] = f.get_tensor(k)
                ^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacity of 47.40 GiB of which 17.19 MiB is free. Including non-PyTorch memory, this process has 47.38 GiB memory in use. Of the allocated memory 46.96 GiB is allocated by PyTorch, and 93.13 MiB is reserved by PyTorch but unallocated.
```

For comparison, under the same memory pressure, using a CPU-safe reload path succeeds:

```python
model.load_adapter(
    ...
    torch_device="cpu",
    ...
)
```

In the successful control run:

- `peft.load_peft_weights.enter` shows `device = "cpu"`
- the run completes successfully
- `status = "ok"`

## Expected behavior

When `load_best_model_at_end=True` is used with a PEFT model, the best-model reload stage should not fail solely because adapter weights requires more GPU memory to reload on GPU.

## Possible fix

One possible fix would be to make the PEFT `load_best_model_at_end` path follow a more memory-safe loading pattern similar to the non-PEFT model case: load the best model state on CPU first like:

```python
model.load_adapter(
    self.state.best_model_checkpoint,
    active_adapter,
    torch_device="cpu",
    autocast_adapter_dtype=False,
)
```

[oom_repro_large_tail_qwen35_gsm8k.py](https://github.com/user-attachments/files/25939511/oom_repro_large_tail_qwen35_gsm8k.py)
[run_oom_repro_latest_transformers.sh](https://github.com/user-attachments/files/25939512/run_oom_repro_latest_transformers.sh)

--- Comment by pankajbaid567 at 2026-03-12T21:16:31Z ---
Hi, I'd like to work on this issue.

From the repro and stack trace it appears that `Trainer._load_best_model()` calls `model.load_adapter()` without specifying `torch_device`, which results in PEFT loading adapter weights directly on CUDA.

Under low GPU memory this causes an OOM during the best-model reload stage.

My current idea is to modify the PEFT path in `_load_best_model()` to load adapters on CPU first using:

```
model.load_adapter(..., torch_device="cpu")
```

and then move the model back to the training device.

I'll investigate whether this needs to be guarded behind a PEFT check or handled generically.

Please let me know if this direction sounds reasonable before I open a PR.

--- Comment by DogWala at 2026-03-13T03:28:26Z ---
Yes, that's what I'm currently using.

--- Comment by BenjaminBossan at 2026-03-13T13:17:09Z ---
Thanks for reporting the issue. I agree that it's very annoying to have an error at the very end of the training run. What is unclear to me is where the memory pressure comes from. During training, we have the following situation:

1. Base model weights loaded on GPU.
2. LoRA weights loaded on GPU.
3. Gradients for LoRA weights loaded on GPU.
4. Hidden states required for backwards path loaded on GPU.
5. Possibly optimizer states loaded on GPU.

When we load the best model, we should only have:

1. Base model weights loaded on GPU.
2. LoRA weights loaded on GPU.

How is it possible that this OOMs when training works fine? Unless I'm missing something, there must be something else going on, like memory not being freed or accidentally using the wrong dtype.

> One possible fix would be to make the PEFT `load_best_model_at_end` path follow a more memory-safe loading pattern similar to the non-PEFT model case: load the best model state on CPU first like:

> model.load_adapter(
>     self.state.best_model_checkpoint,
>     active_adapter,
>     torch_device="cpu",
>     autocast_adapter_dtype=False,
> )

If this is about `autocast_adapter_dtype=False`, that would change the dtype, which could indeed safe memory if the base model is loaded in, say, fp16 or bf16 (PEFT by default loads the adapter weights in float32 but this argument leads to the same dtype being used as in the base model weight). However, if the training was done with float32, we could see different behavior due to the lower precision, so I'm not sure if we want to enable this by default.

If this is about `torch_device="cpu"`, AFAICT, this argument is not supported, so I doubt that it helps.

--- Comment by DogWala at 2026-03-13T13:51:48Z ---
Thanks for the detailed reply. I agree that, in principle, it is surprising to see an OOM at best-model reload time if the training loop itself completed successfully. The key point from the repro is that this is not happening in a fully cleaned-up post-training state.

In the failing repro, `Trainer._load_best_model()` is entered while training-related state is still alive. The instrumentation shows:

- `is_in_train = true`
- `optimizer_is_none = false`
- `lr_scheduler_is_none = false`
- `callback_optimizer_is_none = false`

So this is not yet a state where optimizer / scheduler / callback-held training state has already been torn down. On top of that, the repro intentionally keeps tail GPU occupancy high to simulate a realistic high-utilization training setup.

Under the same training configuration:

- training completes successfully
- disabling `load_best_model_at_end` does not fail
- enabling `load_best_model_at_end` fails during the final adapter reload

So the issue is that turning on `load_best_model_at_end` introduces an additional late failure mode under memory pressure.

I agree that the most principled solution would be to release all unnecessary training-time allocations before loading the best model. However, from looking at the current `Trainer` lifecycle, that is not easy to clean optimizer / scheduler / callback-held state before `_load_best_model()` runs, and changing that lifecycle would be a broader and more invasive fix.

---

The immediate failure mechanism in the repro is that the PEFT reload path first materializes a temporary adapter `state_dict` on CUDA before copying it into the existing adapter parameters. In the failing run, the recorded events show:

- `trainer._load_best_model.enter`
- `peft.load_adapter.enter`
- `peft.load_peft_weights.enter` with `device="cuda"`

and the traceback ends in:

```python
safe_load_file(filename, device=device)
```

So the extra memory pressure is:

- base model weights on GPU
- existing adapter weights on GPU
- remaining training-time state / allocator pressure still alive
- plus a temporary CUDA copy of the adapter checkpoint tensors created by `load_peft_weights(..., device="cuda")`

That temporary CUDA materialization is what pushes the run over the edge in the repro.

---

Also, to clarify one point that may have caused confusion: `autocast_adapter_dtype=False` is not the key part of this PR, and I am not changing that behavior in the PR.

I used `autocast_adapter_dtype=False` only in the repro control path to reduce extra variables when comparing the failing CUDA-side adapter load against a CPU-first load path. The actual change proposed in the PR is about the device used for adapter checkpoint loading, not about changing adapter dtype behavior.

---

On the `torch_device` point, unless I am missing something, I think there may be a mismatch about which `load_adapter()` implementation is actually being called here.

In this repro, the model passed to `Trainer` is a PEFT model created with `get_peft_model(...)`, so the relevant `load_adapter()` implementation appears to be PEFT's `PeftModel.load_adapter`, rather than only the Transformers adapter integration layer.

As far as I know, in PEFT, `PeftModel.load_adapter(...)` does support `torch_device`:
- [peft_model.py](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py)

And when `torch_device` is omitted, the PEFT path appears to fall back to device inference before loading adapter weights:
- [save_and_load.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/save_and_load.py)
- [other.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/other.py)

So my understanding is that the PR is not introducing a new unsupported argument, but passing an argument that is already supported by the PEFT `load_adapter()` implementation that this repro resolves to. But if I am missing something about the exact path taken here, I am happy to be corrected.

--- Comment by BenjaminBossan at 2026-03-13T14:25:38Z ---
> I agree that the most principled solution would be to release all unnecessary training-time allocations before loading the best model. However, from looking at the current `Trainer` lifecycle, that is not easy to clean optimizer / scheduler / callback-held state before `_load_best_model()` runs, and changing that lifecycle would be a broader and more invasive fix.

Honestly, I would argue to do something along these lines. The fact that loading the PEFT weight here causes the OOM is in a  sense coincidence, it could also happen in other situations, right? So I think a more general solution would be desired. I'm not too knowledgeable about `Trainer` but at a quick glance, it looks like we should be able to clean up most things, maybe add a call to `empty_cache()`, and only then call `_load_best_model()`. @SunMarc WDYT?

> Also, to clarify one point that may have caused confusion: `autocast_adapter_dtype=False` is not the key part of this PR, and I am not changing that behavior in the PR.

Got it.

> On the `torch_device` point, unless I am missing something, I think there may be a mismatch about which `load_adapter()` implementation is actually being called here.

Ah yes, I was looking at the transformers integration. In this case, I think it's an okay solution, as the final adapter weights will be loaded to the same device as the base weights, so things like inference should work. Why exactly this reduces memory pressure, I'm not sure.

--- Comment by DogWala at 2026-03-13T14:50:41Z ---
>In this repro, the model passed to Trainer is a PEFT model created with get_peft_model(...), so the relevant load_adapter() implementation appears to be PEFT's PeftModel.load_adapter, rather than only the Transformers adapter integration layer.

[`trainer.py#L3440`](../blob/main/src/transformers/trainer.py#L3440) calls `_is_peft_model(model)` as a check, so the `model` on this path should be a PEFT model. In the current version, the `torch_device` argument here should be fine.

--- Comment by DogWala at 2026-04-02T14:14:47Z ---
Do you think this issue should be fixed in the trainer, or should we just leave it as it is for now?

--- Comment by github-actions[bot] at 2026-04-27T08:46:36Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by SunMarc at 2026-04-27T13:04:55Z ---
We do have a `release_memory` function in Trainer + trainer.accelerator.clear(). Maybe this is what you are looking for, I don't mind calling it before reloading the model.


ISSUE #44625: Qwen3.5 `num_labels` not propagated from core config to text config
State: closed  |  Labels: bug
Author: tomaarsen  |  Created: 2026-03-12T11:09:38Z

### System Info

- `transformers` version: 5.3.0.dev0
- Platform: Windows-10-10.0.26200-SP0
- Python version: 3.11.6
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.6.2
- Accelerate version: 1.13.0.dev0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA GeForce RTX 3090

### Who can help?

@zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When passing `num_labels` to `AutoConfig.from_pretrained` for Qwen3.5, the outer config has num_labels=1, but the text_config still has the default 2. This prevents me from loading a Qwen3.5 model for Sequence Classification by passing the num_labels to the config. Note that the Sequence Classification model from Qwen3.5 only uses the text config/modality.

```python
from transformers import AutoConfig, AutoModelForSequenceClassification

model_name = "onnx-internal-testing/tiny-random-Qwen3_5ForConditionalGeneration"
# Fails:
config = AutoConfig.from_pretrained(model_name, num_labels=1)
# Works:
# config = AutoConfig.from_pretrained(model_name, num_labels=1, text_config={"num_labels": 1, "id2label": {0: "LABEL_0"}})

print(config.num_labels) # 1
print(config.text_config.num_labels) # 2

model = AutoModelForSequenceClassification.from_pretrained(model_name, config=config)
print(model.config.num_labels) # 2
```

It only works if I pass `text_config={"num_labels": 1, "id2label": {0: "LABEL_0"}}`. I also have to pass the `id2label` part, or I'll get:
```
You passed `num_labels=1` which is incompatible to the `id2label` map of length `2`.
```
And it'll still keep the num_labels on the text config at 2.

### Expected behavior

I expect the above script to return `1` for all `num_labels` by having the `num_labels` on the main config propagate to the subconfigs. I'm not sure if this is the expected behaviour in `transformers`, or whether this is a bug. Either way, I've not experienced it myself before.

- Tom Aarsen

--- Comment by zucchini-nlp at 2026-03-12T11:21:45Z ---
Interesting, since for VLMs we've been adding a general sequence classifier without forcing it to be a text-only version. Lemme check it out and see what's the bets way to support VLM and LLM-only classfifer for those models

--- Comment by tomaarsen at 2026-03-12T11:42:05Z ---
If the SequenceClassification model could be vision-aware, that would be excellent. That would simplify some things on my end as well, potentially. I'm interested in supporting this architecture for rerankers, but there's two approaches:
* ForConditionalGeneration: generate a logit, compare logits for `1` and `0` or `yes` and `no` -> produces a score for a pair of inputs
* ForSequenceClassification: immediately get scores for a pair of inputs

Right now I have to use the former (e.g. https://huggingface.co/tomaarsen/reranker-Qwen3.5-0.8B-doodles-image-text-to-text), but the latter is more convenient.

- Tom Aarsen

--- Comment by zucchini-nlp at 2026-04-08T12:22:10Z ---
fyi: a suggested feature is ready for loading task-models with VLM config

--- Comment by github-actions[bot] at 2026-05-03T08:31:35Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44623: [BUG] processor.save_pretrained(...) missing files
State: closed  |  Labels: bug
Author: Jintao-Huang  |  Created: 2026-03-12T09:20:10Z

### System Info

transformers 4.57.6

transformers 5.3.0

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained('Qwen/Qwen3-VL-8B-Instruct')
processor.save_pretrained('abc')
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

--- Comment by zucchini-nlp at 2026-03-12T11:07:54Z ---
hey @Jintao-Huang !

This is expected. In the v5 release we toggled the new serialization format on, where all sub-processors are saved in `processor_config.json`. You can inspect the file and see a nested dict of sub-processor's parameters

When loading, you should be able to load both, the new and old formatted processor configs so it is 100% BC

--- Comment by zucchini-nlp at 2026-03-13T12:04:23Z ---
I'll close the issue to not attract code agents because it needs no fixing. If you see issues when loading any processor, feel free to re-open


ISSUE #44619: Plug model rule in development flow and extend it
State: closed  |  Labels:
Author: tarekziade  |  Created: 2026-03-12T06:00:58Z

This is a follow-up from https://github.com/huggingface/transformers/pull/44174

We're now plugging the tool into developer flow:
- add an opt-in github hook for checking the model
- automatically run `make check-model-rules` on PRs to generate reports
- add a ref to the CLI in the AI agents files
- fix models that are in allow lists and can be fixed
- modularize the code (add a `mlinter` sub package in `utils`)

--- Comment by github-actions[bot] at 2026-04-14T08:29:54Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


Sam3Video: CUDA out of memory

ISSUE #44617: Sam3Video: CUDA out of memory
State: closed  |  Labels: bug
Author: middleknight  |  Created: 2026-03-12T03:29:05Z

### System Info

transformers 5.3.0
Python 3.10.12
torch 2.4.0+cu124

Tracking multiple targets simultaneously, typically numbering in the dozens, results in out of memory.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import Sam3VideoConfig, Sam3VideoModel, Sam3VideoProcessor
from transformers.video_utils import load_video
import sys
import os
import cv2
import numpy as np
import json
import time
import gc
import torch
import math
from datetime import datetime

import inspect
import os
def print_memory(message=""):
    if not torch.cuda.is_available():
        print(f"[Line {inspect.currentframe().f_back.f_lineno}] {message} - CUDA not available")
        return

    caller_frame = inspect.currentframe().f_back
    line_no = caller_frame.f_lineno

    allocated = torch.cuda.memory_allocated() / 1024**2  # MB
    reserved = torch.cuda.memory_reserved() / 1024**2    # MB
    max_allocated = torch.cuda.max_memory_allocated() / 1024**2

    print(f"[Line {line_no:4d}] {message} - Allocated: {allocated}")

if __name__ == "__main__":
    sam_path = "./sam3/"
    video_path = "./reverse.mp4"
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    #	device = Accelerator().device
    config = Sam3VideoConfig.from_pretrained(sam_path)
    config.image_size = 1008
    model = Sam3VideoModel.from_pretrained(sam_path).to(device, dtype=torch.bfloat16)
    # processor = Sam3VideoProcessor.from_pretrained(sam_path)
    processor = Sam3VideoProcessor.from_pretrained(sam_path, size={"height": 1008, "width": 1008})

    frame_count = 0
    detection_results = []
    track_ids_set = set()
    color_map = {}

    frame_mask_status = []

    batch_counter = 0

    video_frames, _ = load_video(video_path)
    frames_num = video_frames.shape[0]

    inference_session = processor.init_video_session(
        video=video_frames[0],
        inference_device=device,
        processing_device=device,
        video_storage_device='cpu',
        dtype=torch.bfloat16,
    )
    text = "person"
    inference_session = processor.add_text_prompt(
        inference_session=inference_session,
        text=text,
    )
    for idx in range(1, frames_num):
        processed_video = processor.video_processor(videos=video_frames[idx], device=device, return_tensors="pt")
        pixel_values_video = processed_video.pixel_values_videos[0]
        inference_session.add_new_frame(pixel_values_video)

    with torch.no_grad():
        total_model_outputs = model.propagate_in_video_iterator(
            inference_session=inference_session
        )
        print(f"111")

        for model_outputs in total_model_outputs:
            print_memory("test!!!")

            result = processor.postprocess_outputs(inference_session, model_outputs)

            print_memory("before total_model_outputs!!!")

        # print(f"{result['object_ids']}")
        test = 0
```

### Expected behavior

[Line   77] test!!! - Allocated: 2696.203125
[Line   81] before total_model_outputs!!! - Allocated: 2838.58642578125
[Line   77] test!!! - Allocated: 2898.85205078125
[Line   81] before total_model_outputs!!! - Allocated: 2898.85205078125
[Line   77] test!!! - Allocated: 2958.947265625
[Line   81] before total_model_outputs!!! - Allocated: 2958.947265625
[Line   77] test!!! - Allocated: 3017.658203125
[Line   81] before total_model_outputs!!! - Allocated: 3017.658203125
[Line   77] test!!! - Allocated: 3077.51611328125

How do I fix it?

--- Comment by Saad-Mallebhari at 2026-03-27T09:41:27Z ---
@middleknight
The steady ~60MB growth per frame suggests intermediate tensors may be accumulating across iterations rather than being freed , though the exact cause is worth isolating before assuming a fix.

The primary suspect is `inference_session` itself. `propagate_in_video_iterator` likely holds references to per-frame embeddings or mask history internally, and with dozens of tracked objects that state can grow significantly across frames. This is worth checking first , if `Sam3VideoModel` accumulates history inside the session object without a trimming mechanism, no amount of cache clearing will help.

A few things to try in order:

**1. Confirm it's accumulation, not a one-time allocation:**
Add `print(torch.cuda.memory_allocated())` at the top of the loop before any processing. If it grows every iteration, the session state or generator is holding references.

**2. Delete outputs explicitly inside the loop:**
```python
for model_outputs in total_model_outputs:
    result = processor.postprocess_outputs(inference_session, model_outputs)
    del model_outputs, result
```
Note: `torch.cuda.empty_cache()` only releases the allocator cache , it won't free tensors that are still referenced, so it's more useful as a diagnostic than a fix.

**3. Check `inference_session` growth directly:**
After each iteration, inspect whether the session is storing per-frame data (frame embeddings, mask histories). If it is, there may not be a public API to evict older frames yet , that would be the root cause.

**4. Reduce `image_size`:**
Memory per frame scales roughly quadratically with resolution. Dropping from 1008 to 768 could meaningfully reduce per-iteration footprint when tracking many objects.

--- Comment by middleknight at 2026-03-27T10:06:43Z ---
In the store_output interface within modeling_sam3_video.py, there is the following line of code:

`self.output_dict_per_obj[obj_idx][storage_key][frame_idx][output_key] = output_value`

The mask for every object in every frame is being saved. I am not sure of the purpose of retaining these, but as a result, the GPU memory usage increases rapidly in videos with a large number of objects.

--- Comment by Saad-Mallebhari at 2026-03-27T10:37:08Z ---
This strongly suggests the root cause. `output_dict_per_obj` is likely retained for potential backward-pass propagation or re-use during bidirectional tracking, which is why SAM2-style models typically store full frame history. But for long videos with many objects, keeping all outputs on GPU becomes unsustainable.

A few things worth trying as workarounds:

**1. Move only the most recently processed frame's outputs to CPU:**
Rather than iterating the full nested structure each step (which gets expensive as frame count grows), target just the current frame index:
```python
for frame_idx, model_outputs in enumerate(total_model_outputs):
    result = processor.postprocess_outputs(inference_session, model_outputs)
    # Move only the current frame's outputs to CPU
    for obj_idx in inference_session.output_dict_per_obj:
        for storage_key in inference_session.output_dict_per_obj[obj_idx]:
            if frame_idx in inference_session.output_dict_per_obj[obj_idx][storage_key]:
                for k, v in inference_session.output_dict_per_obj[obj_idx][storage_key][frame_idx].items():
                    if isinstance(v, torch.Tensor) and v.is_cuda:
                        inference_session.output_dict_per_obj[obj_idx][storage_key][frame_idx][k] = v.cpu()
```

**2. Evict past frame entries if you don't need them:**
If your pipeline does not depend on past-frame mask reuse, clearing older entries from `output_dict_per_obj` after processing would free memory more aggressively , though this may affect results if the model revisits earlier frames internally.

This could be worth raising as an improvement , adding a `video_storage_device` flag that applies to `output_dict_per_obj` the same way it applies to frame storage would make this configurable without requiring manual workarounds.

--- Comment by github-actions[bot] at 2026-04-21T08:32:03Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44610: [BUG] OmDet-Turbo processor produces 640px inputs but the model expects 224px
State: closed  |  Labels: bug
Author: harshaljanjani  |  Created: 2026-03-11T19:58:13Z

### System Info

* `transformers` version: `5.0.0.dev0`
* Platform: `Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39`
* Python version: `3.12.3`
* `huggingface_hub` version: `1.3.2`
* `safetensors` version: `0.7.0`
* `accelerate` version: `1.12.0`
* Accelerate config: `not installed`
* DeepSpeed version: `not installed`
* PyTorch version (accelerator?): `2.9.1+cu128 (CUDA)`
* GPU type: `NVIDIA L4`
* NVIDIA driver version: `550.90.07`
* CUDA version: `12.4`

### Who can help?

@zucchini-nlp (`🚨 Delete duplicate code in backbone utils`)

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoProcessor, OmDetTurboForObjectDetection
from PIL import Image
import requests
import torch

model = OmDetTurboForObjectDetection.from_pretrained("omlab/omdet-turbo-swin-tiny-hf")
processor = AutoProcessor.from_pretrained("omlab/omdet-turbo-swin-tiny-hf")
image = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw).convert("RGB")
encoding = processor(images=image, text=["cat", "remote"], task="Detect cat, remote.", return_tensors="pt")
try:
    with torch.no_grad():
        outputs = model(**encoding)
    print(outputs.decoder_coord_logits.shape)
except Exception as e:
    print(e)
```

**Current Repro Output:**

[OmDet-Turbo](https://huggingface.co/docs/transformers/model_doc/omdet-turbo) fails with `AssertionError` in inference. The processor produces 640×640 images but the model expects an input height of 224, and running the official loading and inference code raises `AssertionError: Input height (640) doesn't match model (224)` as shown in the screenshot; instead of the expected output tensor. Also causes the issue in the official OmDet-Turbo CI run.

### Expected behavior

→ `outputs.decoder_coord_logits.shape` should return `torch.Size([1, 900, 4])`; the model should accept 640×640 images as configured.


ISSUE #44609: StaticSlidingWindowLayer triggers torch.compile/dynamo recompilations every decode step
State: closed  |  Labels: Feature request
Author: jazpurTT  |  Created: 2026-03-11T19:15:20Z

### Feature request

### Summary
Could there be added support for  StaticSlidingWindowLayer to be fully compatible with torch.compile() by avoiding it's dynamic control flow?

When using a StaticCache with StaticSlidingWindowLayer (e.g. GPT-OSS, Mistral) and torch.compile(), the compiled graph recompiles on every decode step. The recompile limit is hit quickly and generation is effectively uncompiled or very slow.

`StaticSlidingWindowLayer` keeps `cumulative_length` as a Python `int` and updates it each step `self.cumulative_length += key_states.shape[-2]`. It is used in:
- `update` – for branching (is_full, cumulative_length + key_states.shape[-2] > self.max_cache_len, etc.).
- `get_mask_sizes` – which returns (kv_length, kv_offset) as Python ints derived from self.cumulative_length.

TorchDynamo treats `self.cumulative_length` as a constant and installs guards on their concrete values. Each decode step changes cumulative_length (e.g. 71 → 72 → 73…), so the guards fail and the graph recompiles for every model forward pass.

Example guard failures:
```j
Recompiling function wrapper in /lib/python3.12/site-packages/transformers/utils/generic.py:912
    triggered by the following guard failure(s):
    - 0/1: kwargs['past_key_values'].layers[0].cumulative_length == 71  # is_full = self.cumulative_length >= self.max_cache_len  # transformers/cache_utils.py:462 in get_mask_sizes
    - 0/0: tensor 'kwargs['input_ids']' size mismatch at index 1. expected 71, actual 1
Recompiling function wrapper in /lib/python3.12/site-packages/transformers/utils/generic.py:912
    triggered by the following guard failure(s):
    - 0/2: kwargs['past_key_values'].layers[0].cumulative_length == 72  # is_full = self.cumulative_length >= self.max_cache_len  # transformers/cache_utils.py:462 in get_mask_sizes
    - 0/1: kwargs['past_key_values'].layers[0].cumulative_length == 71  # is_full = self.cumulative_length >= self.max_cache_len  # transformers/cache_utils.py:462 in get_mask_sizes
    - 0/0: tensor 'kwargs['input_ids']' size mismatch at index 1. expected 71, actual 1
Recompiling function wrapper in /lib/python3.12/site-packages/transformers/utils/generic.py:912
    triggered by the following guard failure(s):
    - 0/3: kwargs['past_key_values'].layers[0].cumulative_length == 73  # is_full = self.cumulative_length >= self.max_cache_len  # transformers/cache_utils.py:462 in get_mask_sizes
    - 0/2: kwargs['past_key_values'].layers[0].cumulative_length == 72  # is_full = self.cumulative_length >= self.max_cache_len  # transformers/cache_utils.py:462 in get_mask_sizes
    - 0/1: kwargs['past_key_values'].layers[0].cumulative_length == 71  # is_full = self.cumulative_length >= self.max_cache_len  # transformers/cache_utils.py:462 in get_mask_sizes
    - 0/0: tensor 'kwargs['input_ids']' size mismatch at index 1. expected 71, actual 1
...
torch._dynamo hit config.recompile_limit (8)
```

### Reproduction / Example

The following snippet (from a GPT-OSS 20B generation example) shows the scenario where recompilation occurs. Every decode step triggers a recompile:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.cache_utils import StaticCache

model = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-20b",
    dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
    attn_implementation="eager",
)
# config has sliding_window and layer_types with "sliding_attention" → StaticSlidingWindowLayer
cache_config = model.config

static_cache = StaticCache(
    config=cache_config,
    max_batch_size=batch_size,
    max_cache_len=max_cache_len,
    device="cpu",
    dtype=torch.bfloat16,
)
static_cache.early_initialization(
    batch_size=batch_size,
    num_heads=model.config.num_key_value_heads,
    head_dim=model.config.head_dim,
    dtype=torch.bfloat16,
    device="cpu",
)

compiled_model = torch.compile(model)

input_args = {
    "input_ids": input_ids,
    "past_key_values": static_cache,
    "cache_position": torch.arange(71),
    "use_cache": True,
    "attention_mask": attention_mask,
}
output = compiled_model(**input_args)

for step in range(max_tokens_to_generate - 1):
    input_args["input_ids"] = next_token_id.unsqueeze(-1)           # shape (1, 1)
    input_args["cache_position"] = torch.tensor([[input_args["cache_position"][-1].item() + 1]])
    output = compiled_model(**input_args)   # cumulative_length changed → guard fail → recompile
```

The second and subsequent `compiled_model(**input_args)` calls see a different `past_key_values.layers[i].cumulative_length` each time and torch dynamo recompiles the graph each time.

### Environment
- transformers `4.57.1` but I am pretty sure same issue would happen on the latest main branch.
- torch 2.9.0 with `torch.compile(model)`
- Models that use StaticSlidingWindowyLayer (e.g. openai/gpt-oss-20b with default layer_types)

### Motivation

Sliding Attention is a common way to run large context models with bounded memory and latency (e.g. Mistral, GPT-OSS). With `StaticSlidingWindowLayer`, decode is effectively uncompilable: every new token changes `cumulative_length`, `torch._dynamo` recompiles, and the recompile limit is reached after a few dozen steps. That removes most of the benefit of compilation for long generations and makes it hard to use sliding-window caches in production with `torch.compile`. Fixing this minimizes the amount of different decode graphs to run, so sliding-window + compile is viable.

### Your contribution

I have written a workaround to be able to compile and run `openai/gpt-oss-20b` for 180+ forward passes with only two compiled graphs, one for the pre-fill pass and one for decode passes:
1. It replaces `StaticSlidingWindowLayer` with my own custom class where:
    - Replaces the original `update` method with a branchless version that always rolls the KV buffer left by n positions and writes new tokens to the rightmost slots. This produces a single straight-line graph with no mutable Python state, so torch.dynamo never sees a reason to recompile.
    - `get_mask_sizes` returns constants `(max_cache_len, 0)` instead of values derived from `cache_position` and `self.cummulative_length` at runtime.
2. It replaces `create_sliding_window_causal_mask` with a custom mask building function where:
    - It builds the mask using the right-aligned mapping: it computes real_pos = total_tokens_seen - sliding_window + slot_index to figure out which buffer slots hold which sequence positions, then applies validity, causality, and window constraints from there. This masking understands the always-roll layout and the new `get_mask_sizes` const return.

This solution was made for `openai/gpt-oss-20b` but I have also tested it on `mistralai/Ministral-8B-Instruct-2410` and it worked, but I assume the solution as is would not be compatible with all models using sliding attention, but this could be a starting point to support a non-dynamic `StaticSlidingWindowLayer` for multi-run `torch.compile` compatibility.

--- Comment by jazpurTT at 2026-03-11T19:23:22Z ---
The workaround looks like this:
```python
model: torch.nn.Module = AutoModelForCausalLM.from_pretrained(
        model_name,
        dtype=torch.bfloat16,
        low_cpu_mem_usage=True,
        trust_remote_code=True,
        attn_implementation="eager",
    )
override_gpt_oss_sliding_window_causal_mask()
[ ... ]
static_cache: StaticCache = StaticCache(
        config=model.config,
        max_batch_size=batch_size,
        max_cache_len=max_cache_len,
        device="cpu",
        dtype=torch.bfloat16,
    )
num_key_value_heads = model_config.num_key_value_heads
head_dim = model_config.head_dim
static_cache.early_initialization(
        batch_size=batch_size,
        num_heads=num_key_value_heads,
        head_dim=head_dim,
        dtype=torch.bfloat16,
        device="cpu",
    )

sliding_window = getattr(cache_config, "sliding_window", max_cache_len)
override_cache_sliding_window_layers(static_cache, max_cache_len, sliding_window)
compiled_model = torch.compile(model)

input_args = {
    "input_ids": input_ids,
    "past_key_values": static_cache,
    "cache_position": torch.arange(71),
    "use_cache": True,
    "attention_mask": attention_mask,
}
output = compiled_model(**input_args)

for step in range(max_tokens_to_generate - 1):
    input_args["input_ids"] = next_token_id.unsqueeze(-1)           # shape (1, 1)
    input_args["cache_position"] = torch.tensor([[input_args["cache_position"][-1].item() + 1]])
    output = compiled_model(**input_args)   # cumulative_length changed → guard fail → recompile

```
and the custom class/function
```python
from typing import Any, Optional

import torch
from transformers.cache_utils import StaticCache, StaticLayer, StaticSlidingWindowLayer
from transformers.modeling_utils import create_sliding_window_causal_mask

def override_gpt_oss_sliding_window_causal_mask():
    import transformers.models.gpt_oss.modeling_gpt_oss as gpt_oss_mod

    gpt_oss_mod.create_sliding_window_causal_mask = custom_create_sliding_window_causal_mask

def override_cache_sliding_window_layers(
    cache: StaticCache,
    max_cache_len: int,
    sliding_window: int,
) -> None:
    for i, layer in enumerate(cache.layers):
        if isinstance(layer, StaticSlidingWindowLayer):
            new_layer = CustomStaticSlidingWindowLayer(
                max_cache_len=max_cache_len, sliding_window=sliding_window
            )
            new_layer.keys = layer.keys
            new_layer.values = layer.values
            new_layer.is_initialized = True
            new_layer.device = layer.device
            new_layer.dtype = layer.dtype
            new_layer.max_batch_size = layer.max_batch_size
            new_layer.num_heads = layer.num_heads
            new_layer.head_dim = layer.head_dim
            cache.layers[i] = new_layer

def custom_create_sliding_window_causal_mask(
    config,
    input_embeds: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    cache_position: torch.Tensor,
    past_key_values=None,
    **kwargs,
) -> torch.Tensor:
    if past_key_values is None:
        return create_sliding_window_causal_mask(
            config,
            input_embeds,
            attention_mask,
            cache_position,
            past_key_values,
            **kwargs,
        )

    if isinstance(attention_mask, torch.Tensor) and attention_mask.ndim == 4:
        return attention_mask

    if hasattr(past_key_values, "is_sliding") and True in past_key_values.is_sliding:
        layer_idx = past_key_values.is_sliding.index(True)
    else:
        layer_idx = 0
    sliding_window = past_key_values.layers[layer_idx].max_cache_len

    batch_size = input_embeds.shape[0]
    query_length = cache_position.shape[0]
    dtype = input_embeds.dtype
    device = cache_position.device
    min_val = torch.finfo(dtype).min

    kv_slots = torch.arange(sliding_window, device=device)
    total_tokens_seen = cache_position[-1] + query_length
    real_pos = total_tokens_seen - sliding_window + kv_slots

    valid = real_pos >= 0
    causal = real_pos.unsqueeze(0) <= cache_position.unsqueeze(1)
    in_window = real_pos.unsqueeze(0) > (cache_position.unsqueeze(1) - sliding_window)

    mask = valid.unsqueeze(0) & causal & in_window

    if attention_mask is not None and attention_mask.ndim == 2:
        padding_mask = attention_mask.to(device=device, dtype=torch.bool)
        real_pos_idx = real_pos.clamp(min=0).long()
        padding = padding_mask[:, real_pos_idx]
        mask = mask.unsqueeze(0).unsqueeze(0) & padding.unsqueeze(1).unsqueeze(1)
    else:
        mask = mask.unsqueeze(0).unsqueeze(0).expand(batch_size, 1, -1, -1)

    return torch.where(
        mask,
        torch.tensor(0.0, dtype=dtype, device=device),
        torch.tensor(min_val, dtype=dtype, device=device),
    )

class CustomStaticSlidingWindowLayer(StaticLayer):

    is_sliding = True

    def __init__(self, max_cache_len: int, sliding_window: int):
        effective_max_cache_len = min(sliding_window, max_cache_len)
        super().__init__(max_cache_len=effective_max_cache_len)

    def update(
        self,
        key_states: torch.Tensor,
        value_states: torch.Tensor,
        cache_kwargs: Optional[dict[str, Any]] = None,
    ) -> tuple[torch.Tensor, torch.Tensor]:
        if not self.is_initialized:
            self.lazy_initialization(key_states)

        n = key_states.shape[-2]

        new_keys = self.keys.roll(-n, dims=-2)
        new_values = self.values.roll(-n, dims=-2)

        new_keys[:, :, -n:] = key_states
        new_values[:, :, -n:] = value_states

        self.keys.copy_(new_keys)
        self.values.copy_(new_values)

        return self.keys, self.values

    def get_mask_sizes(self, cache_position: torch.Tensor) -> tuple[int, int]:
        return self.max_cache_len, 0

    def get_seq_length(self) -> int:
        if not self.is_initialized:
            return 0
        return (self.keys[0, 0].any(dim=-1)).sum().item()
```

--- Comment by Rocketknight1 at 2026-03-12T12:22:26Z ---
cc @Cyrilvallez @zucchini-nlp for generate!

--- Comment by Cyrilvallez at 2026-03-12T12:35:08Z ---
Hey! For StaticSlidingWindow layers, there are no way to overcome the fact that it *needs* to be dynamic by essence, to be logically correct. However, if you compile with `dynamic=None/True`, it will ONLY recompile the 2nd time to generalize the internal `int`, and then if the regime of the cache changes, i.e. after the cache becomes full i.e. you reached the sliding window length.
In your example, you're compiling the prefill which will necessary trigger a new compilation next step as all shapes did change.

If you use `dynamic=False`, then it will trigger recompilation at every step which is expected as the `int` cannot generalize

--- Comment by Cyrilvallez at 2026-03-12T13:00:49Z ---
Consider the following for example:

```python
import os
os.environ["TORCH_LOGS"] = "recompiles"
from transformers import AutoModelForCausalLM, Mxfp4Config
import torch

model = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-20b",
    dtype=torch.bfloat16,
    device_map=0,
    quantization_config=Mxfp4Config(dequantize=True),
)

inputs = torch.randint(100, 200, (1, 25), device=0)
out = model.generate(inputs, max_new_tokens=25, min_new_tokens=25, cache_implementation="static", do_sample=False)
```

It prints:

```sh
V0312 12:59:50.071000 2016937 site-packages/torch/_dynamo/guards.py:4181] [0/1] [__recompiles] Recompiling function wrapper in [/fsx/cyril/transformers/src/transformers/utils/generic.py:845](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a2269702d32362d302d3136352d3234222c2275736572223a22637972696c5f76616c6c657a227d.vscode-resource.vscode-cdn.net/fsx/cyril/transformers/src/transformers/utils/generic.py:845)
V0312 12:59:50.071000 2016937 site-packages/torch/_dynamo/guards.py:4181] [0/1] [__recompiles]     triggered by the following guard failure(s):
V0312 12:59:50.071000 2016937 site-packages/torch/_dynamo/guards.py:4181] [0/1] [__recompiles]     - 0/0: kwargs['past_key_values'].layers[0].cumulative_length_int == 25  # is_full = current_length >= self.max_cache_len  # transformers/src/transformers/cache_utils.py:397 in update
```

And you can see that it recompiled only once on the 2nd forward, to generalize the `int`

--- Comment by Cyrilvallez at 2026-03-12T13:01:42Z ---
Let me know if something is still unclear! But basically your custom static sliding window class is logically incorrect in general, as in some cases you will not return enough past states


ISSUE #44593: Support for sequence-level custom metrics with decoder-only models
State: open  |  Labels: Feature request
Author: l-k-11235  |  Created: 2026-03-11T10:51:47Z

### Feature request

Hi Hugging Face team,

I’m trying to compute custom metrics at the sequence level for a decoder-only Transformer model, but I ran into an issue. The Seq2SeqTrainer class provides the predict_with_generate option, but it is primarily designed for encoder-decoder architectures. As a result, using it with decoder-only models doesn’t fully support sequence-level metric computation out-of-the-box.

### Motivation

Sequence-level metrics, such as BLEU, ROUGE, or other task-specific metrics, are useful for monitoring the training of large language models.

### Your contribution

I implemented a local fix by subclassing Seq2SeqTrainer and overriding the prediction_step method. The main points of my approach are:

- Compute the prompt length dynamically from the labels.
- Mask the prompt and generate sequences per example.
- Pad generated sequences to make metric computation straightforward.

```python
def get_prompt_length_from_labels(labels: torch.Tensor) -> int:
    """
    Retourne le nombre de tokens du prompt (consécutifs -100 au début de labels)
    """
    # Pour tensor 1D
    prompt_len = (labels == -100).cumsum(dim=0).eq(torch.arange(1, len(labels) + 1, device=labels.device)).sum().item()
    return prompt_len

class Seq2SeqTrainerCustom(Seq2SeqTrainer):

    def prediction_step(
            self,
            model: nn.Module,
            inputs: dict[str, torch.Tensor | Any],
            prediction_loss_only: bool,
            ignore_keys: list[str] | None = None,
            **gen_kwargs,
    ) -> tuple[float | None, torch.Tensor | None, torch.Tensor | None]:
        """
        Perform an evaluation step on `model` using `inputs`.

        Subclass and override to inject custom behavior.

        Args:
            model (`nn.Module`):
                The model to evaluate.
            inputs (`dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.

                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument `labels`. Check your model's documentation for all accepted arguments.
            prediction_loss_only (`bool`):
                Whether or not to return the loss only.
            gen_kwargs:
                Additional `generate` specific kwargs.

        Return:
            tuple[Optional[float], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss, logits and
            labels (each being optional).
        """

        if not self.args.predict_with_generate or prediction_loss_only:
            return super().prediction_step(
                model, inputs, prediction_loss_only=prediction_loss_only, ignore_keys=ignore_keys
            )

        has_labels = "labels" in inputs
        inputs = self._prepare_inputs(inputs)

        # Priority (handled in generate):
        # non-`None` gen_kwargs > model.generation_config > default GenerationConfig()
        if len(gen_kwargs) == 0 and hasattr(self, "_gen_kwargs"):
            gen_kwargs = self._gen_kwargs.copy()
        if "num_beams" in gen_kwargs and gen_kwargs["num_beams"] is None:
            gen_kwargs.pop("num_beams")
        if "max_length" in gen_kwargs and gen_kwargs["max_length"] is None:
            gen_kwargs.pop("max_length")

        default_synced_gpus = is_deepspeed_zero3_enabled() or is_fsdp_managed_module(self.model)
        gen_kwargs["synced_gpus"] = gen_kwargs.get("synced_gpus", default_synced_gpus)

        generation_inputs = inputs.copy()

        # If the `decoder_input_ids` was created from `labels`, evict the former, so that the model can freely generate
        # (otherwise, it would continue generating from the padded `decoder_input_ids`)
        if (
                "labels" in generation_inputs
                and "decoder_input_ids" in generation_inputs
                and generation_inputs["labels"].shape == generation_inputs["decoder_input_ids"].shape
        ):
            generation_inputs = {
                k: v for k, v in inputs.items() if k not in ("decoder_input_ids", "decoder_attention_mask")
            }

        summon_full_params_context = (
            FullyShardedDataParallel.summon_full_params(self.model)
            if isinstance(self.model, FullyShardedDataParallel)
            else contextlib.nullcontext()
        )

        with summon_full_params_context:
            # generated_tokens = self.model.generate(**generation_inputs, **gen_kwargs)
            # Beginning of fix
            batch_prompt_lens = [get_prompt_length_from_labels(labels) for labels in inputs["labels"]]

            all_generated_tokens = []
            for i in range(len(batch_prompt_lens)):
                prompt_len = batch_prompt_lens[i]
                gen_input_ids = inputs["input_ids"][i, :prompt_len].unsqueeze(0)
                gen_attention_mask = inputs["attention_mask"][i, :prompt_len].unsqueeze(0)
                generated_tokens = self.model.generate(
                    input_ids=gen_input_ids,
                    attention_mask=gen_attention_mask,
                    **gen_kwargs
                )
                all_generated_tokens.append(generated_tokens)
        generated_tokens = torch.nn.utils.rnn.pad_sequence(
            [x.squeeze(0) for x in all_generated_tokens],
            batch_first=True,
            padding_value=self.processing_class.pad_token_id
        )
        # End of Fix

        # Temporary hack to ensure the generation config is not initialized for each iteration of the evaluation loop
        # TODO: remove this hack when the legacy code that initializes generation_config from a model config is
        # removed in https://github.com/huggingface/transformers/blob/98d88b23f54e5a23e741833f1e973fdf600cc2c5/src/transformers/generation/utils.py#L1183
        if self.model.generation_config._from_model_config:
            self.model.generation_config._from_model_config = False

        # Retrieves GenerationConfig from model.generation_config
        # Update with defaults because earlier the generation config used to be init
        # with default values. Now we init it with `None` and keep defaults for BC
        gen_config = self.model.generation_config
        default_gen_config = gen_config._get_default_generation_params()
        gen_config.update(**default_gen_config, defaults_only=True)
        # in case the batch is shorter than max length, the output should be padded
        if generated_tokens.shape[-1] < gen_config.max_length:
            generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_config.max_length)
        elif gen_config.max_new_tokens is not None and generated_tokens.shape[-1] < gen_config.max_new_tokens + 1:
            generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_config.max_new_tokens + 1)

        with torch.no_grad():
            if has_labels:
                with self.compute_loss_context_manager():
                    outputs = model(**inputs)
                if self.label_smoother is not None:
                    loss = self.label_smoother(outputs, inputs["labels"]).detach().mean()
                else:
                    loss = (outputs["loss"] if isinstance(outputs, dict) else outputs[0]).detach().mean()
            else:
                loss = None

        if self.args.prediction_loss_only:
            return loss, None, None

        if has_labels:
            labels = inputs["labels"]
            if labels.shape[-1] < gen_config.max_length:
                labels = self._pad_tensors_to_max_len(labels, gen_config.max_length)
            elif gen_config.max_new_tokens is not None and labels.shape[-1] < gen_config.max_new_tokens + 1:
                labels = self._pad_tensors_to_max_len(labels, gen_config.max_new_tokens + 1)
        else:
            labels = None

        return loss, generated_tokens, labels
```

--- Comment by Rocketknight1 at 2026-03-11T13:36:19Z ---
cc @sunmarc but not sure if we already have a way to do this!

--- Comment by SunMarc at 2026-03-11T16:06:35Z ---
Indeed, we don't support that yet, the best way to do that would be as you did, extend `Seq2SeqTrainer` to make it more compatible with decoder-only model.

--- Comment by l-k-11235 at 2026-03-12T07:48:33Z ---
OK, thanks for your answer !

--- Comment by shaealh at 2026-03-13T05:05:01Z ---
Taking a look, will submit a PR shortly

--- Comment by shaealh at 2026-03-13T05:42:12Z ---
Opened a [PR ](https://github.com/huggingface/transformers/pull/44650)

--- Comment by michaelanderson01826-glitch at 2026-03-13T11:57:16Z ---
Thanks

On Fri, Mar 13, 2026, 5:42 AM Shae Alhusayni ***@***.***>
wrote:

> *shaealh* left a comment (huggingface/transformers#44593)
> <https://github.com/huggingface/transformers/issues/44593#issuecomment-4052908652>
>
> Opened a PR <https://github.com/huggingface/transformers/pull/44650>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/44593#issuecomment-4052908652>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEKPGDQ6B3FW467OBL34QONU7AVCNFSM6AAAAACWOEJW3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANJSHEYDQNRVGI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>