PULL REQUEST #45776: Gate enable_gqa=True on actual flash-attention eligibility
State: open  |  Merged: False
Author: dvdimitrov13  |  Base: main
Labels:
Created: 2026-05-04T16:35:28Z

## Summary

Follow-up to discussion in #45636 — narrowing to the GQA gating that @vasqu identified as the part worth tightening.

`sdpa_attention_forward` decides between materializing `repeat_kv` and passing `enable_gqa=True` to torch's SDPA via `use_gqa_in_sdpa()`. The previous condition only checked torch version + `attention_mask is None`, so `enable_gqa=True` could be set in cases where the active SDPA backend cannot honor it. `enable_gqa=True` is FA-only today — any other backend silently misroutes the dispatcher.

Concretely, the existing path misfires whenever:
- The user wraps the call in `sdpa_kernel([SDPBackend.EFFICIENT_ATTENTION])` (or any non-FA pin).
- FA is globally disabled (`torch.backends.cuda.enable_flash_sdp(False)`).
- Inputs violate FA's library caps: `head_dim > 256` (e.g. Gemma 4's `head_dim=320`), fp32, sm < 80, etc.

## Fix

Tighten `use_gqa_in_sdpa()` to also check:
1. `torch.backends.cuda.flash_sdp_enabled()` — captures `sdpa_kernel(...)` context and global toggles.
2. `torch.backends.cuda.can_use_flash_attention(SDPAParams(...))` — pytorch's canonical FA-eligibility predicate, evaluated on the actual inputs.

Reusing pytorch's predicate (rather than rolling our own head_dim/dtype/arch table) means transformers won't drift as FA's constraint set evolves.

The signature of `use_gqa_in_sdpa` grows from `(attention_mask, key)` to `(attention_mask, query, key, value, is_causal, dropout)`. It has exactly one in-tree caller (`sdpa_attention_forward` itself), where all the new args are already in scope. The XPU path is unchanged.

In `sdpa_attention_forward` the `is_causal` resolution moves up by ~12 lines so the probe receives the resolved value; the later `is_causal = query.shape[2] > 1 and attention_mask is None and is_causal` line still applies the same filtering before the SDPA call — functionally identical.

## Tests (`tests/utils/test_sdpa_attention.py`)

Two test classes:

**`UseGqaInSdpaLogicTest`** — 7 mocked tests, run on any backend (CPU CI inclusive):
- `attention_mask` present → False
- old torch → False
- `flash_sdp_enabled() == False` → False
- FA enabled but inputs ineligible (mocked) → False
- happy path (FA enabled + eligible) → True
- XPU branch unchanged
- `SDPAParams` raises (defensive fallback) → False

**`UseGqaInSdpaCudaTest`** (`@require_torch_gpu`) — 5 end-to-end probes on real CUDA inputs:
- default context, FA-friendly shape (head_dim=128, bf16) → True (regression guard)
- inside `sdpa_kernel([EFFICIENT_ATTENTION])` → False
- inside `sdpa_kernel([MATH])` → False
- `head_dim=320` (Gemma 4 case) → False
- fp32 inputs → False

### Verified locally

CUDA tests run on RTX 3090 (sm_86), torch 2.7.1+cu126:

```
test_default_context_fa_friendly_shape_returns_true PASSED
test_efficient_only_context_returns_false           PASSED
test_fp32_returns_false                              PASSED
test_head_dim_too_large_returns_false                PASSED
test_math_only_context_returns_false                 PASSED
============================== 5 passed in 3.07s ==============================
```

Logic tests:
```
============================== 7 passed in 2.68s ==============================
```

`ruff check` and `ruff format --check` both pass.

## Open question

Per-call cost of constructing `SDPAParams` + `can_use_flash_attention`. Microbench welcome — happy to defer the probe (e.g. cache the answer per shape signature) if it shows up on the trace. My instinct is it's cheap relative to the SDPA call itself, but I haven't measured.

cc @vasqu @Cyrilvallez per the discussion in #45636.

--- Comment by vasqu at 2026-05-04T17:18:27Z ---
Yea, honestly I don't think it takes too much time but we never know. I guess it defers to the sdp utils within torch itself

On another note, we should probably make more diffs between cpu and cuda backend then

--- Comment by dvdimitrov13 at 2026-05-04T18:42:22Z ---
Thanks for the review @vasqu. Two updates:

### 1. Bench results

Ran microbench + small-model end-to-end on RTX 4090 (sm_89), torch 2.5.1+cu124.

**Microbench — `use_gqa_in_sdpa` per call:**

| Shape | Context | Old (ns) | New (ns) | Δ |
|---|---|---:|---:|---:|
| training (B=4, T=2048, Hq=16, Hkv=4, D=128) | default (FA on)        | 47  | 1860 | +1812 |
| training | `sdpa_kernel([EFFICIENT])`    | 46  | 144  | +98   |
| prefill  (B=1, T=512, …)                    | default (FA on)        | 45  | 1743 | +1698 |
| prefill  | `sdpa_kernel([EFFICIENT])`    | 45  | 142  | +97   |
| decode   (B=1, T=1, …)                      | default (FA on)        | 45  | 1751 | +1705 |
| decode   | `sdpa_kernel([EFFICIENT])`    | 45  | 142  | +97   |

So worst-case is **~1.8 µs/call** when probing (FA-eligible inputs); when the user has pinned a non-FA backend (the original bug) it short-circuits at `flash_sdp_enabled()` for ~100 ns extra.

**End-to-end on `HuggingFaceTB/SmolLM2-135M`** (134.5M, GQA 9q/3kv, head_dim=64, FA-eligible — so it always hits the full probe path):

| seq_len | stock median | patched median | Δ      | overhead |
|---:|---:|---:|---:|---:|
| 1024 | 22.943 ms | 23.535 ms | +0.59 ms | +2.6 % |
| 4096 | 23.495 ms | 24.084 ms | +0.59 ms | +2.5 % |

The absolute delta is constant (~0.59 ms / forward) and dominated by the per-layer probe summed across 30 layers. The relative cost shrinks as seq grows / model gets bigger / attention dominates more — at 135M with the small head_dim, the attention work itself is a small fraction of forward time, so this is a near-worst-case for relative impact. As you suspected, "deferring to the sdp utils within torch itself" doesn't make the probe free — pytorch effectively pays this cost twice now (once in transformers, once again in the dispatcher when the SDPA call actually fires).

If you'd want the probe cached by shape signature, I can add that — but it'd need invalidation on `sdpa_kernel(...)` context entries (since `flash_sdp_enabled()` is what changes). Open to your preference.

### 2. CPU/CUDA differentiation

Added an explicit `query.device.type != "cuda"` short-circuit before the FA probe in [820f237](https://github.com/huggingface/transformers/pull/45776/commits/820f237) — `torch.backends.cuda.*` is meaningless on CPU/other accelerators, so we now fall back to `repeat_kv` directly. Added a logic test (`test_cpu_inputs_reject_without_probing`) that asserts the probe APIs are *not* called when `query` is a CPU tensor.

XPU branch unchanged (per existing scope).

--- Comment by dvdimitrov13 at 2026-05-05T18:31:30Z ---
Thanks for the close read @vasqu. Three replies in [d6b8dca](https://github.com/huggingface/transformers/pull/45776/commits/d6b8dca):

**Re L54 (CPU FA backend):** You're right, I checked. CPU SDPA honors `enable_gqa=True` natively (verified on torch 2.8.0, bf16, GQA shape — returns shape-correct output matching manually-repeated K/V to within 1e-2). My gate broke BC.

**Re L51 (BC for non-CUDA / non-XPU):** Restructured the gate per your suggestion. New shape:
- xpu  → existing path (`>=2.8`)
- cuda → new FA-eligibility probe (`flash_sdp_enabled()` + `can_use_flash_attention`)
- everything else (cpu, npu, mps, …) → existing BC path (`return True` when torch>=2.5 and no mask), no probe

So the new behavior is strictly opt-in for CUDA — other devices get the pre-PR behavior unchanged.

**Re L82 (comment):** Restored the original `is_causal` comment.

Logic verified locally; CI re-running.

--- Comment by vasqu at 2026-05-06T14:33:27Z ---
Ok based on the failures, `SDPAParams` is not (fullgraph) compile friendly which really means we cannot just dispatch to torch for this instance and need manual checks...

This is super annoying

--- Comment by dvdimitrov13 at 2026-05-06T18:53:20Z ---
On the SDPAParams fullgraph thing — happy to try and give you a hand.

One option that might dodge the manual-checks regression: the input-shape constraints (head_dim, dtype, arch) are constant once the layer's built — only the kernel context varies at runtime. So we could keep the canonical pytorch probe but run it once at init and cache a bool on the module. Forward reads the bool + ANDs `attention_mask is None` and (optionally) `flash_sdp_enabled()`. Dynamo treats cached Python bools as constants, fullgraph stays clean, no manual constraint table to maintain.

Two things I'd defer to you on:
- Where the init-time probe should live. I'd guess there's an attention-interface hook for this kind of thing — happy to follow whatever convention you point at.
- Whether to keep `flash_sdp_enabled()` in forward at all. Dropping it loses the `sdpa_kernel([EFFICIENT])` post-init pin coverage, but removes the last `torch.backends.cuda.*` call from the hot path.

On the other comments: happy to move tests to `test_modeling_utils` and trim to the real edge cases (FA-eligible → enable_gqa, head_dim>256 → repeat_kv, post-init pin → repeat_kv, CPU BC → enable_gqa).

Let me know if this is helpful or you would rather have me out of the way.

--- Comment by github-actions[bot] at 2026-05-06T18:58:09Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45776&sha=be3ff7

--- Comment by vasqu at 2026-05-06T19:02:29Z ---
I've submitted an issue to the torch team https://github.com/pytorch/pytorch/issues/182684 - let's first see what they have to say tbh. It depends on whether we have a wrong assumption here or whether they fix it on their side

I dont think we have an easy path on init time because these conditionals will change dynamically based on the input (for example the seq len between q and kv is also deciding the backend), not sure I can follow on that point tbh. My idea was to use the backends check only but if that does not consider the enabled backends for SDPA, we might have to include that one as well then.

First, we do need to figure the fullgraph compatibility imo

--- Comment by vasqu at 2026-05-06T19:04:02Z ---
I'd really like to use the torch native check if possible, manually keeping track of those conditions will be unmaintainable especially across different torch versions

--- Comment by vasqu at 2026-05-04T17:16:59Z ---
Yea would be nice if we could bench how much time this takes, best on smaller models (<200M parameters)

--- Comment by vasqu at 2026-05-05T13:49:21Z ---
I do think there is a FA cpu backend, no? Could you check

--- Comment by vasqu at 2026-05-05T13:50:12Z ---
I would check for the device type here and make this the default fallback for anything other than xpu and cuda - just to keep BC behavior for now until we check other devices

--- Comment by vasqu at 2026-05-05T13:50:47Z ---
I would just keep the original comment, the new comment is not super useful imo

--- Comment by vasqu at 2026-05-06T13:58:47Z ---
I've changed the order a bit to be more in sync with the preparations e.g. re `is_causal` and the attention mask, + slight style changes for BC

--- Comment by vasqu at 2026-05-06T14:00:45Z ---
Imo, should be moved to test_modeling_utils where we already test parts of attentions

--- Comment by vasqu at 2026-05-06T14:02:16Z ---
Would be nice to cut this a bit down tho for the real edge cases, i.e. what failed prev etc + what should have the same result


make sure we call check_auto in CI

PULL REQUEST #45775: make sure we call check_auto in CI
State: closed  |  Merged: True
Author: tarekziade  |  Base: main
Labels:
Created: 2026-05-04T07:27:15Z

# What does this PR do?

per https://github.com/huggingface/transformers/pull/45774 we did not run check_auto in the CI

Commit 0e15778202 ("Dynamic auto mapping" [PR #45018](https://github.com/huggingface/transformers/pull/45018), Apr 16 2026) is the one that created `utils/check_auto.py` in the first place. In that same commit, the Makefile was changed so that:

- The CI-run targets (style, check-code-quality) had their old `auto_mappings` entry renamed to `sort_auto_mappings`  i.e., they kept running only the simple alphabetical sorter
  (`utils/sort_auto_mappings.py`), which doesn't have the tempfile bug.
- The new `auto_mappings` entry (the buggy `check_auto.py` regenerator) was added only to `check-repo` and `fix-repo`; both local-developer-only targets.

So the buggy regenerator has never had CI coverage since it was introduced. Before PR #45018, the auto_mappings name in CI referred to the old simple sort script. It wasn't dropped from CI; it was effectively replaced under a different name when the new dynamic mechanism was bolted on.

The current PR fixes it by:

- making sure we run it now in CI
- share the same list of check across all make targets, so we don't regress again there
- display a diff in check_auto so it's easier to debug
- fixed a big in glob.blog ordering (system dependent)

--- Comment by tarekziade at 2026-05-04T07:28:43Z ---
Part of the problem is that the Makefile has too many targets that do the same things. I will make sure we de-dupe. WIll add more commits here

--- Comment by tarekziade at 2026-05-04T07:29:43Z ---
proof of wiring : https://app.circleci.com/pipelines/github/huggingface/transformers/173694/workflows/d70c1229-cb57-4444-9331-88c78581d942/jobs/2296112

--- Comment by HuggingFaceDocBuilderDev at 2026-05-04T07:39:29Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45775). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-04T08:12:07Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

--- Comment by tarekziade at 2026-05-04T08:12:52Z ---
run-slow: auto

--- Comment by github-actions[bot] at 2026-05-04T08:14:11Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25308310605)

This comment contains `run-slow`, running the specified jobs:

models: ["models/auto"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-04T08:21:52Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25308310605)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [afa24183](https://github.com/huggingface/transformers/commit/afa24183cf4224de36b6e3969986a584e4c9f6f4) | workflow commit (merge commit) |
| PR | [e90065ec](https://github.com/huggingface/transformers/commit/e90065ec2600e2b07f7ef9c38d13fb7b44b6ecc9) | branch commit (from PR) |
| main | [54e90e08](https://github.com/huggingface/transformers/commit/54e90e08334e63b6b10b50641c6d09d9383f9a90) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by tarekziade at 2026-05-04T08:41:50Z ---
> Thanks! `make fix-repo` used to do less though no?

Fix thanks for spotting. recap:

- check-code-quality: 6
- check-repository-consistency: 18
- check-repo: 24 (sum of both for local runs)
- fix-repo: 15 (with fixes)

--- Comment by tarekziade at 2026-05-04T09:37:28Z ---
run-slow: maskformer

--- Comment by github-actions[bot] at 2026-05-04T09:39:07Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25311862962)

This comment contains `run-slow`, running the specified jobs:

models: ["models/maskformer"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-04T09:48:01Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25311862962)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [b736538f](https://github.com/huggingface/transformers/commit/b736538f850625d8a4f005381d8ee36b6524251e) | workflow commit (merge commit) |
| PR | [b20ced55](https://github.com/huggingface/transformers/commit/b20ced55f5afc4a26970c22db9f01f8dc9f9d162) | branch commit (from PR) |
| main | [54e90e08](https://github.com/huggingface/transformers/commit/54e90e08334e63b6b10b50641c6d09d9383f9a90) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by ydshieh at 2026-05-04T11:06:51Z ---
> great to check-in with YihDar i believe

＠tarekziade, since you already got 2 ✅  + you are the author of these new "make / checker" stuffs, I believe in you too.

But if you prefer me to take a look on specific points, let me know.

--- Comment by tarekziade at 2026-05-04T11:56:13Z ---
Last run in CI, see `python utils/check_auto.py`

```
Generate auto mappings
$ python utils/check_auto.py
OK (15.04s)

Public imports
$ python -c "from transformers import *"
OK (30.07s)

Import complexity
$ python utils/check_import_complexity.py
Import complexity OK: 579 modules (max 1000)
OK (2.51s)

Copied code consistency
$ python utils/check_copies.py
OK (9.68s)

Modular file conversions
$ python utils/check_modular_conversion.py
OK (9.40s)

Documentation table of contents
$ python utils/check_doc_toc.py
OK (0.36s)

Modeling rules documentation
$ python utils/check_modeling_rules_doc.py --rules-toml utils/rules.toml
OK (0.56s)

Docstring formatting
$ python utils/check_docstrings.py
OK (29.44s)

Dummy objects
$ python utils/check_dummies.py
OK (0.17s)

Repository structure
$ python utils/check_repo.py
Repository-wide checks:
    - checking all models are included.
    - checking all models are public.
    - checking all models have tests.
    - checking all objects have documentation.
    - checking all models are in at least one auto class.
    - checking all names in auto name mappings are defined.
    - checking all keys in auto name mappings are defined in `CONFIG_MAPPING_NAMES`.
    - checking all auto mappings could be imported.
    - checking the DEPRECATED_MODELS constant is up to date.
    - checking all models accept **kwargs in their call.
OK (40.40s)

Init files
$ python utils/check_inits.py
OK (0.12s)

Pipeline type hints
$ python utils/check_pipeline_typing.py
OK (12.66s)

Config docstrings
$ python utils/check_config_docstrings.py
OK (15.93s)

Config attributes
$ python utils/check_config_attributes.py
OK (17.00s)

Doctest list
$ python utils/check_doctest_list.py
OK (0.14s)

Model metadata
$ python utils/update_metadata.py --check-only
OK (13.75s)

Model dates
$ python utils/add_dates.py --check-only
OK (2.56s)

Dependency versions table
$ python setup.py deps_table_update
running deps_table_update
OK (0.78s)

All 18 checks passed in 3m27.24s.
```

--- Comment by Cyrilvallez at 2026-05-04T08:12:51Z ---
Looks like we had less than those before for the `fix-repo` no?

--- Comment by Cyrilvallez at 2026-05-04T08:15:07Z ---
Why not add the `,` litterally in the string before?

--- Comment by tarekziade at 2026-05-04T08:18:19Z ---
to minimize typo accidents, but yeah I can simplify for sure

--- Comment by zucchini-nlp at 2026-05-04T08:19:43Z ---
nice catch!

--- Comment by tarekziade at 2026-05-04T08:28:20Z ---
good catch

--- Comment by Cyrilvallez at 2026-05-04T09:50:21Z ---
Trusting you that this work (not 100% sure how args are forwarded here), but could you just drop the `print` please? It's expected to skip them, so not warranting any print

--- Comment by ydshieh at 2026-05-04T11:04:46Z ---
It's under `args.fix`, so keep a message here makes sense IMO.

Also it's a check utility (unlike modeling or loading code), so should not cause "a lot of warning" as we saw before.

--- Comment by tarekziade at 2026-05-04T11:54:23Z ---
this will generate a single line at the top, not adding all the checks in details, see an example:

``` bash
(venv) m5 ➜transformers git:(tarek-fix-check-aut) ✗ make fix-repo
Skipping 9 check-only checker(s) in fix mode: types, modeling_structure, imports, import_complexity, repo, inits, config_docstrings, config_attributes, update_metadata

✓ Ruff linting (0.09s)
$ ruff check examples tests src utils scripts .circleci/create_circleci_config.py benchmark benchmark_v2 setup.py conftest.py --fix --exclude
All checks passed!

✓ Ruff formatting (0.03s)
$ ruff format examples tests src utils scripts .circleci/create_circleci_config.py benchmark benchmark_v2 setup.py conftest.py --exclude
4293 files left unchanged

✓ Import ordering (0.04s)
$ python utils/custom_init_isort.py

✓ Sort auto mappings (0.03s)
$ python utils/sort_auto_mappings.py

✓ Generate auto mappings (4.00s)
$ python utils/check_auto.py --fix_and_overwrite

✓ Copied code consistency (5.24s)
$ python utils/check_copies.py --fix_and_overwrite

✓ Modular file conversions (2.44s)
$ python utils/check_modular_conversion.py --fix_and_overwrite

✓ Documentation table of contents (0.08s)
$ python utils/check_doc_toc.py --fix_and_overwrite

✓ Modeling rules documentation (0.05s)
$ python utils/check_modeling_rules_doc.py --rules-toml utils/rules.toml --fix_and_overwrite

✓ Docstring formatting (7.30s)
$ python utils/check_docstrings.py --fix_and_overwrite

✓ Dummy objects (0.02s)
$ python utils/check_dummies.py --fix_and_overwrite

✓ Pipeline type hints (2.59s)
$ python utils/check_pipeline_typing.py --fix_and_overwrite

✓ Doctest list (0.02s)
$ python utils/check_doctest_list.py --fix_and_overwrite

✓ Model dates (0.51s)
$ python utils/add_dates.py

✓ Dependency versions table (0.12s)
$ python setup.py deps_table_update
running deps_table_update

All 15 checks passed in 22.56s.
```


Fix auto mapping script

PULL REQUEST #45774: Fix auto mapping script
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-05-04T07:02:40Z

# What does this PR do?

As per the title. The current script fully truncates the file instead, so running `make fix-repo` results in a lot of errors and corrupted library.
It's not really possible to both write and read the same file with the same `open`, without messing with `.seek()`.

I have no idea how it was not caught on the CI. cc @tarekziade @ydshieh @zucchini-nlp can you please make sure the script is run/checked on the CI?? The fact that it was not caught before makes me think we are not checking that the mappings are up-to-date and not corrupted!

--- Comment by HuggingFaceDocBuilderDev at 2026-05-04T07:13:57Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45774). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45773: Fix UnboundLocalError for `is_updated` in encoder-decoder cross-attention
State: open  |  Merged: False
Author: SeaL773  |  Base: main
Labels:
Created: 2026-05-04T06:26:53Z

## What does this PR do?

Fixes an `UnboundLocalError` in the cross-attention forward pass of several encoder-decoder models. The variable `is_updated` is only assigned inside a conditional block that checks for `EncoderDecoderCache`, but is referenced unconditionally afterward. When `past_key_values` is `None` or a non-`EncoderDecoderCache` type (e.g., `DynamicCache` during speculative decoding), the variable is never bound.

**Error:**
```
File ".../transformers/models/whisper/modeling_whisper.py", line 323, in forward
    if is_cross_attention and past_key_values and is_updated:
                                                  ^^^^^^^^^^
UnboundLocalError: cannot access local variable 'is_updated' where it is not associated with a value
```

## Affected models

- `whisper` (`modeling_whisper.py`)
- `moonshine` (`modeling_moonshine.py`, `modular_moonshine.py`)
- `moonshine_streaming` (`modeling_moonshine_streaming.py`)
- `pix2struct` (`modeling_pix2struct.py`)
- `audioflamingo3` (`modeling_audioflamingo3.py`)

## Root cause

The buggy pattern:
```python
# is_updated only assigned inside this block
if past_key_values is not None and isinstance(past_key_values, EncoderDecoderCache):
    is_updated = past_key_values.is_updated.get(self.layer_idx)
    ...

# Unconditional reference — UnboundLocalError if the block above was skipped
if is_cross_attention and past_key_values and is_updated:
    ...
```

## Fix

Add `is_updated = False` initialization before the conditional block. This is consistent with the pattern already used in T5, BART, xglm, BioGPT, Informer, and other encoder-decoder models:

```python
is_updated = False
if past_key_values is not None and isinstance(past_key_values, EncoderDecoderCache):
    is_updated = past_key_values.is_updated.get(self.layer_idx)
    ...
```

## How to reproduce

```python
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq, AutoModelForCausalLM, AutoTokenizer
import torch

main_model = AutoModelForSpeechSeq2Seq.from_pretrained("openai/whisper-small", torch_dtype=torch.float16).to("cuda")
assistant = AutoModelForCausalLM.from_pretrained("distil-whisper/distil-small.en", torch_dtype=torch.float16).to("cuda")
processor = AutoProcessor.from_pretrained("openai/whisper-small")
tokenizer = AutoTokenizer.from_pretrained("openai/whisper-small")
asst_tokenizer = AutoTokenizer.from_pretrained("distil-whisper/distil-small.en")

# Any audio input
import numpy as np
audio = np.zeros(16000, dtype=np.float32)  # 1 second silence
inputs = processor(audio, return_tensors="pt", sampling_rate=16000)
features = inputs.input_features.to("cuda", dtype=torch.float16)

# This raises UnboundLocalError
main_model.generate(
    input_features=features,
    assistant_model=assistant,
    tokenizer=tokenizer,
    assistant_tokenizer=asst_tokenizer,
)
```

## Reference

Correct implementations for comparison:
- [`models/t5/modeling_t5.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py) — initializes `is_updated = False` before conditional
- [`models/xglm/modeling_xglm.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/xglm/modeling_xglm.py) — same pattern
- [`models/biogpt/modeling_biogpt.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/biogpt/modeling_biogpt.py) — same pattern

--- Comment by SeaL773 at 2026-05-04T07:22:36Z ---
> Looks sane to me, can you add a regression test for e.g. whisper 👀

Of course, WIP

--- Comment by vasqu at 2026-05-04T15:19:26Z ---
run-slow: audioflamingo3, moonshine, moonshine_streaming, pix2struct, whisper

--- Comment by vasqu at 2026-05-04T15:19:54Z ---
I'm checking with our slow ci @SeaL773 but should be good to merge. More of a sanity check :hugs: then merging

--- Comment by github-actions[bot] at 2026-05-04T15:24:24Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25327376682)

This comment contains `run-slow`, running the specified jobs:

models: ["models/audioflamingo3", "models/moonshine", "models/moonshine_streaming", "models/pix2struct", "models/whisper"]
quantizations: []

--- Comment by HuggingFaceDocBuilderDev at 2026-05-04T16:37:28Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45773). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-05-04T16:54:48Z ---
It looks like whisper got completely wrecked by this PR - see https://github.com/huggingface/transformers/actions/runs/25327376682/job/74253441213

Not sure whether to just skip for now but it seems like a speculative decoding test `test_speculative_decoding_distil` now causes a cuda error --> corrupted cuda state --> subsequent tests all fail

--- Comment by github-actions[bot] at 2026-05-04T17:08:34Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25327376682)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [d91d7f49](https://github.com/huggingface/transformers/commit/d91d7f49ab6fd751d417cef5095af7c6b2e75509) | workflow commit (merge commit) |
| PR | [11a88d86](https://github.com/SeaL773/transformers/commit/11a88d8616611bde505c044a58ce62ba55cd2965) | branch commit (from PR) |
| main | [d63bb4ac](https://github.com/huggingface/transformers/commit/d63bb4ac4a24b2f75244cf586919b18223506a4e) | base commit (on `main`) |

### Model CI Report

❌ **[1 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/9308d1f97d6ffd5c9741944f1dab3d2ac6aae1ea/2026-05-04/runs/34134-25327376682/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [whisper](https://github.com/huggingface/transformers/actions/runs/25327376682/job/74253441213):
    tests/models/whisper/test_modeling_whisper.py::WhisperModelIntegrationTests::test_speculative_decoding_distil (❌ ⟹ ❌)

--- Comment by SeaL773 at 2026-05-04T17:44:12Z ---
Thanks for running the slow CI! I took a closer look at the failures — I don't think any of them are related to the `is_updated` fix:

- **5× dtype mismatch** (`test_large_*`, `test_distil_token_timestamp_generation`): `RuntimeError: Input type (float) and bias type (c10::Half)` — these tests load `whisper-large-v3` / `distil-large-v3` (Hub config `torch_dtype: float16`) without passing `torch_dtype`, so the model ends up in fp16 while `input_features` from the processor stays fp32. The error is in the encoder `Conv1d`, unrelated to the attention cache path.
- **`test_small_longform_timestamps_generation`**: `KeyError: 0`
- **`test_small_token_timestamp_generation`**: `AssertionError: Tensor-likes are not close!`
- **`test_speculative_decoding_distil`**: CUDA device-side assert (async), which then corrupts CUDA state and cascades to all subsequent GPU tests.

The fix only adds `is_updated = False` before the `isinstance(past_key_values, EncoderDecoderCache)` check. For `EncoderDecoderCache` — which the generation framework always creates for encoder-decoder models — line 314 immediately overwrites it, so there's zero behavioral change on that path. I traced the full speculative decoding call chain and confirmed the crash happens inside the assistant model's `_sample()` → `WhisperForCausalLM.forward()`, where the decoder also creates `EncoderDecoderCache` (because `encoder_hidden_states is not None`).

These all look like pre-existing issues on main. Let me know if there's anything else I can help with!

--- Comment by vasqu at 2026-05-04T18:00:07Z ---
Isn't the issue that `test_speculative_decoding_distil` now leads to a corrupted cuda state? It seems to make all other tests fail after that. Or were they all failing before as well because then we/I should investigate asap where the initial failures started

Tl;dr: All the tests that now fail after `test_speculative_decoding_distil` are majority wise new failures

--- Comment by vasqu at 2026-05-04T18:12:16Z ---
I don't doubt that the fix is wrong per se but it revealed a different issue in that speculative decoding test which now instead of falling silently (unbound error) leads to a corrupted cuda state.

That is really bad because we now have "new" failures after that due to that corrupted state. We either have to check if we can fix the speculative decoding test here or whether we skip it for now.

Having 200+ failing tests is bad no matter whether you know they would pass normally. It will also make it hard to track when real new failures would appear because the corrupted state would persist and hide that info

--- Comment by SeaL773 at 2026-05-04T18:25:19Z ---
> I don't doubt that the fix is wrong per se but it revealed a different issue in that speculative decoding test which now instead of falling silently (unbound error) leads to a corrupted cuda state.
>
> That is really bad because we now have "new" failures after that due to that corrupted state. We either have to check if we can fix the speculative decoding test here or whether we skip it for now.
>
> Having 200+ failing tests is bad no matter whether you know they would pass normally. It will also make it hard to track when real new failures would appear because the corrupted state would persist and hide that info

Sorry about the previous reply, LLM generate it and I didn't review it carefully before posting.

I looked into the cuda issue. The root cause is that `WhisperForCausalLM` sets `config.is_encoder_decoder = False`, so `_prepare_cache_for_generation()` creates a plain `DynamicCache` instead of wrapping it in `EncoderDecoderCache`.

When this cache reaches `WhisperDecoder.forward()`, the `past_key_values is None` check is false (it's `a DynamicCache`), so the `EncoderDecoderCache` initialization is skipped. The decoder then passes a `DynamicCache` into cross-attention, where self-attention and cross-attention kv get mixed into the same cache, therefore causing the CUDA device-side assert.

--- Comment by SeaL773 at 2026-05-04T18:40:52Z ---
> I don't doubt that the fix is wrong per se but it revealed a different issue in that speculative decoding test which now instead of falling silently (unbound error) leads to a corrupted cuda state.
>
> That is really bad because we now have "new" failures after that due to that corrupted state. We either have to check if we can fix the speculative decoding test here or whether we skip it for now.
>
> Having 200+ failing tests is bad no matter whether you know they would pass normally. It will also make it hard to track when real new failures would appear because the corrupted state would persist and hide that info

CI is all green now. The decoder cache fix in `ec4148d` should address the speculative decoding CUDA issue.

--- Comment by github-actions[bot] at 2026-05-05T17:08:52Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, moonshine, moonshine_streaming, pix2struct, whisper

--- Comment by vasqu at 2026-05-04T14:54:54Z ---
1. Can we add this PR as reference to the comment, e.g. "Regression test based on `#45773`: `UnboundLocalError` on non EncoderDecoder caches. This can happen e.g. when using assistant models." <- much shorter and contains what we need
2. Sync all the comments of the other model's tests? There are now multiple different formulations for essentially the same thing
3. A few models are probably inheriting from others here? E.g. Audio flamingos attention is inherited from whipser. Let's identify only those core models where we add a test

--- Comment by SeaL773 at 2026-05-04T15:16:48Z ---
> 1. Can we add this PR as reference to the comment, e.g. "Regression test based on `#45773`: `UnboundLocalError` on non EncoderDecoder caches. This can happen e.g. when using assistant models." <- much shorter and contains what we need
> 2. Sync all the comments of the other model's tests? There are now multiple different formulations for essentially the same thing
> 3. A few models are probably inheriting from others here? E.g. Audio flamingos attention is inherited from whipser. Let's identify only those core models where we add a test

Thanks for the review! Addressed all three points: unified the docstring, kept tests only for core models (Whisper + Pix2Struct), and removed the inherited ones.

--- Comment by vasqu at 2026-05-05T13:48:56Z ---
I'm sorry but this shows some fundamental issue, not feeling to good about fixing it this way. It would be nice to investigate when things started to initially fail and verify that we indeed need this

--- Comment by SeaL773 at 2026-05-05T17:21:20Z ---
So I dug into the history to understand when this started breaking.

`test_speculative_decoding_distil` was passing before **#38235**. The old attention code just checked `if past_key_value is not None:` and called `.is_updated.get()` directly, no isinstance gate, so it always went through the EncoderDecoderCache path as long as the cache existed.

**#38235** refactored the attention and changed that to `isinstance(past_key_values, EncoderDecoderCache)`, but didn't add a fallback `is_updated = False` for the non-EncoderDecoderCache case. That's the latent bug.

Then **#41505** changed `_prepare_cache_for_generation()` so that it pre-creates a `DynamicCache` for the assistant model. Since `WhisperForCausalLM` has `is_encoder_decoder = False`, the framework doesn't wrap it in `EncoderDecoderCache`. The decoder sees a non-None `past_key_values` at line 740, skips its own cache init, and the attention layer ends up with a plain `DynamicCache` during cross-attention.

Without this PR that was a silent `UnboundLocalError`. With `is_updated = False` the code gets further and mixes self/cross-attention kv into the same cache, which corrupts CUDA state.

This isn't Whisper-specific. `BartForCausalLM`, `MBartForCausalLM` etc. all do the same `is_encoder_decoder = False` with cross-attention pattern.

I've reverted the decoder cache wrap for now.

For the cache mismatch, I think fixing it in `_prepare_cache_for_generation()` might be the cleanest path since it would cover all the ForCausalLM models at once instead of patching each decoder individually. Maybe checking for cross-attention capability (e.g. `encoder_hidden_states` in the forward signature) in addition to `is_encoder_decoder`. But you know the generation framework better, so curious what you think.

--- Comment by SeaL773 at 2026-05-05T17:35:27Z ---
I tested this locally. Adding a second condition in '_prepare_cache_for_generation()' that checks for 'encoder_outputs' in 'model_kwargs' (instead of relying on 'is_encoder_decoder' alone) fixes the speculative decoding issue and passes all three cases: assisted generation, normal generation, and standalone decoder-only. Happy to push it here if you want, or leave it for a separate PR.

--- Comment by vasqu at 2026-05-06T13:17:15Z ---
So in essence, we still had encoder hidden states during speculative decoding and that caused issues as the cache wasn't suitable enough anymore. I would not expect encoder hidden states in speculative decoding tbh but maybe that is indeed expected for those certain models that are built from encoder-decoder models.

cc @zucchini-nlp @Cyrilvallez maybe you have a better intuition here and it looks valid to me to fix the cache preparation in this instance

--- Comment by zucchini-nlp at 2026-05-06T13:29:45Z ---
not really following

If we have a Bart with decoder-only part (BartForCausalLM) that tries to use assistant model (another BartForCausalLM), then encoder hidden states can't be computed. If both are `BartForConditionalGeneration`, then `config.is_encoder_decoder=True`

I can't think of a case where one is enc-dec and the other is dec-only, as it won't just work

--- Comment by vasqu at 2026-05-06T15:12:51Z ---
It's a bit weird because currently, speculative decoding tests with whisper fail because of an unbounded var error --> that has hidden quite a bit on what is going on

When you properly define the variable (similar to Bart), then we now get cuda assertion errors (likely IMAs). The likely root cause seems that `encoder hidden states` seem to be passed when using speculative decoding; the question now is
1. Was that intended, i.e. having encoder hidden states, and we should fix the cache style as we force a simple dynamic cache, not encoder-decoder, which leads to the failures
2. Was not intended, i.e. no encoder hidden states, and we should fix our input preparation somewhere

The case stands that something goes wrong re speculative decoding and whisper

--- Comment by zucchini-nlp at 2026-05-06T15:27:52Z ---
I can take a look, which slow test is that? Smth tells me that input args are all messed up from the beginning

--- Comment by vasqu at 2026-05-06T15:37:36Z ---
Whisper re https://github.com/huggingface/transformers/pull/45773#issuecomment-4373267850

--- Comment by zucchini-nlp at 2026-05-06T16:09:43Z ---
very weird, I see that we have a enc-dec target and causal assistant. Also, in `WhisperDecoder` we still have `encoder_hidden_states` in signature because it is used in two places: in causal and conditional LMs

IIUC the idea was to allow spec dec to work on encoder-decoder models by running an encoder only once. Thus we were just supposed to pass over already encoded features. Then the assistant doesn't need to load `WhisperEncoder` to vram (why waste memory) so we are added a CausalLM counterpart specifically for this case

To sum up, if i am correct, then we could check in generation if target is enc-dec and adjust the assistant cache accordingly. IMO that should be fine and not-breaking

--- Comment by vasqu at 2026-05-06T16:23:04Z ---
Yea, I think this was the idea of @SeaL773 as well to adjust the cache respectively so that we properly encounter the proper cache in this case. Ig we can see if the outputs are sane whether this is truly BC :eyes:


test pull request

PULL REQUEST #45772: test pull request
State: closed  |  Merged: False
Author: henloworldowo  |  Base: main
Labels:
Created: 2026-05-04T05:53:33Z

test-pr

# What does this PR do?

my first ever pull request, please ignore I am new to github :D

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


PULL REQUEST #45771: perf(qwen3_vl): replace Conv3d with F.linear in patch embed forward
State: closed  |  Merged: False
Author: jashshah999  |  Base: main
Labels:
Created: 2026-05-04T00:51:53Z

## What does this PR do?

Replaces the Conv3d forward pass in `Qwen3VLVisionPatchEmbed` with `F.linear` on the reshaped weight. When `stride == kernel_size`, Conv3d is mathematically equivalent to extracting non-overlapping patches and applying a linear projection. The Conv3d codepath triggers an extremely slow cuDNN kernel on some GPU/dtype combinations (~50,000x slower on Blackwell + bf16, ~62x on other configs per the issue benchmarks).

The fix reshapes the input to `(batch, in_channels * t * h * w)` and uses `F.linear(input, weight.view(embed_dim, -1), bias)`. Same weight tensor, just reshaped at forward time, so existing checkpoints load without changes.

## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md)?
- [x] This PR fixes a bug (issue #45750)
- [x] Backward compatible (same weights, same outputs, just faster)

Fixes #45750.

--- Comment by github-actions[bot] at 2026-05-04T00:53:00Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_vl

--- Comment by vasqu at 2026-05-04T07:34:47Z ---
It has already been handled as an extra feature, see my comment here please: https://github.com/huggingface/transformers/issues/45750#issuecomment-4369077946

--- Comment by jashshah999 at 2026-05-04T17:50:00Z ---
Closing, this is already handled in #45041. Thanks for pointing it out.


PULL REQUEST #45770: Unwrap `text_config` in `AutoModelFor*.from_config`
State: closed  |  Merged: True
Author: jamesbraza  |  Base: main
Labels:
Created: 2026-05-04T00:39:39Z

Fixes https://github.com/huggingface/transformers/issues/45759

@ArthurZucker @Cyrilvallez @zucchini-nlp

--- Comment by Qodo-Free-For-OSS at 2026-05-04T09:00:32Z ---
Hi, `_BaseAutoModelClass.from_config` and `.from_pretrained` now copy `quantization_config` onto the unwrapped text config whenever the parent has the attribute, even if the value is `None`; downstream quantizer detection treats this as pre-quantized and will crash on `None.get(...)`.

**Severity:** action required | **Category:** correctness

**How to fix:** Guard quantization_config propagation

**Agent prompt to fix** - you can give this to your LLM of choice:

> ### Issue description
> `AutoModelFor*.from_config`/`from_pretrained` now propagates `quantization_config` from a composite parent config to the unwrapped `text_config`. If the parent has `quantization_config=None`, the propagated attribute presence causes `get_hf_quantizer()` to treat the model as pre-quantized and crash when it calls `.get()` on `None`.
>
> ### Issue Context
> `get_hf_quantizer` uses `hasattr(config, "quantization_config")` (not value checks) to decide `pre_quantized`, and `AutoHfQuantizer.supports_quant_method()` assumes a dict-like object and calls `.get()`.
>
> ### Fix Focus Areas
> - src/transformers/models/auto/auto_factory.py[241-249]
> - src/transformers/models/auto/auto_factory.py[394-402]
>
> ### What to change
> - Only set `config.quantization_config` on the text sub-config when the parent value is not `None`.
> - Prefer `getattr(parent_config, "quantization_config", None)` over `hasattr`.
> - Consider also preserving an existing non-None `quantization_config` already present on the text sub-config (avoid overwriting).

---
*[Qodo](https://github.com/marketplace/qodo-merge-pro-for-open-source) code review - free for open-source.*

--- Comment by github-actions[bot] at 2026-05-04T17:55:11Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, qwen3_5, qwen3_5_moe

--- Comment by jamesbraza at 2026-05-04T17:56:36Z ---
> Hi, `_BaseAutoModelClass.from_config` and `.from_pretrained` now copy `quantization_config` onto the unwrapped text config whenever the parent has the attribute, even if the value is `None`; downstream quantizer detection treats this as pre-quantized and will crash on `None.get(...)`.
>
> **Severity:** action required | **Category:** correctness
>
> **How to fix:** Guard quantization_config propagation
>
> ...

Sure @Qodo-Free-For-OSS , just pushed a commit adding this to both call sites in `auto_factory.py` (`from_config` and `from_pretrained`):

```python
# Check both `quantization_config` being present and also not null,
# as a `config.json` can have `"quantization_config": null` in it
parent_quant = getattr(parent_config, "quantization_config", None)
if parent_quant is not None:
    config.quantization_config = parent_quant
```

--- Comment by HuggingFaceDocBuilderDev at 2026-05-05T10:51:41Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45770). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by jamesbraza at 2026-05-04T00:41:06Z ---
Matching https://github.com/huggingface/transformers/pull/45494 here, fyi

--- Comment by zucchini-nlp at 2026-05-04T08:39:24Z ---
very weird test haha, since the resulting model class is a VLM class anyway. I don't remember the history, so ig it was causalLM and at some point we stopped supporting it 🤔

--- Comment by zucchini-nlp at 2026-05-04T08:46:36Z ---
tbh i am not even sure we are actually using it, except for qwen3.5. From what I see in mapping, gemma models will load a VLM class. And this makes sense if a complete config is passed/downloaded

Not sure what was the idea with qwen3.5 as I wasn't there when shipping the model, but a VLM class should be completely capable of running text-only samples

--- Comment by jamesbraza at 2026-05-04T16:53:24Z ---
Lol yeah sorry, I reverted this `test_modeling_gemma3.py` change (it was unrelated to the Qwen3 issue I'm trying to fix, I was just maintaining symmetry across tests). You're right in your intuition, these Gemma 3 test changes were actually dead code.

--- Comment by jamesbraza at 2026-05-04T17:08:29Z ---
Fwiw I just re-ran the minimal reproducer from https://github.com/huggingface/transformers/issues/45759 for `Qwen/Qwen3.6-35B-A3B`, the crash I reported does happen. This makes sense because `qwen3_5_moe` backs `Qwen/Qwen3.6-…` too.

> a VLM class should be completely capable of running text-only samples

My motivation mainly is for [SkyRL's FSDP setup](https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/workers/model_wrapper.py#L137-L139) it wants `AutoModelForCausalLM` to give back a text-only class. Mainly this PR just brings `from_config` in line with `from_pretrained`'s current behavior.

Are you thinking something else? I think I may not be fully appreciating your comment here.


PULL REQUEST #45769: feat: add bf16_loss training argument for VRAM-efficient QLoRA by keeping loss in BF16 during training to save ~1.4 GB VRAM
State: closed  |  Merged: False
Author: butterwecksolutions  |  Base: main
Labels:
Created: 2026-05-03T23:41:12Z

  ## Problem

  During QLoRA training with `--bf16`, logits are upcast from BF16 to
  FP32 during loss computation, allocating 600 MB–1.4 GB per logit
  tensor at model forward time. This is disproportionately expensive
  when the model weights are already quantized to 4-bit precision.

  ## Fix

  New `--bf16_loss` flag (default: `False`, opt-in). When set:
  - Requires `--bf16` (validated at startup)
  - Wraps `model.forward()` in `torch.amp.autocast(device_type="cuda", dtype=torch.bfloat16)`
  - Logits stay in BF16 → ~600 MB–1.4 GB saved per forward pass

  ## Trade-off

  Negligible precision loss for most workloads. QLoRA already
  quantizes weights to 4-bit — BF16 logits preserve more signal
  than the quantization boundaries discard.

  ## Verification

  Tested on Qwen3-VL-9B QLoRA training (RTX 3090 24 GB):

  | Metric | fp32 loss | bf16 loss |
  |--------|----------|-----------|
  | Logit tensor dtype | FP32 (4 bytes/elt) | BF16 (2 bytes/elt) |
  | Peak VRAM | 22.5 GB (before other optimizations) | ~21 GB (estimated) |
  | Loss convergence | identical | identical |
  | Gradients | baseline | indistinguishable |

  Combined with [bitsandbytes#1935](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1935) and [TRL#5694](https://github.com/huggingface/trl/pull/5694), peak VRAM drops
  from ~22.5 GB to ~12.5 GB in full QLoRA training.

--- Comment by Rocketknight1 at 2026-05-05T13:29:03Z ---
cc @sunmarc @benjaminbossan

--- Comment by BenjaminBossan at 2026-05-05T14:03:04Z ---
Thanks for the PR @butterwecksolutions. Do you have a small script that can be used to measure this difference?

--- Comment by butterwecksolutions at 2026-05-05T21:26:59Z ---
> Thanks for the PR @butterwecksolutions. Do you have a small script that can be used to measure this difference?

OK, this request is a result of a debug marathon of my training pipeline. i currently build a reproducer script with all my fixes and update this within the next few days. Please wait for my results until further investigation.

--- Comment by BenjaminBossan at 2026-05-06T11:50:02Z ---
Thanks for working on the reproducer. Something very simple, even with dummy data, would be absolutely fine.

--- Comment by butterwecksolutions at 2026-05-07T05:07:07Z ---
  ## Systematic Isolation Complete — No Measurable Effect at Current Scale

  Same 14-test reproducer (7 patches × 2 modes, fresh GPU per test)
  used to isolate the VRAM leak root cause.

  **bf16_loss training arg (#45769): 0.0 GB effect.**

  | Mode | Baseline | P2 (#45769) | Δ |
  |------|----------|-------------|---|
  | VL (seq_len=512) | 20.3 GB | 20.3 GB | **0.0 GB** |
  | TEXT (seq_len=4096) | 8.5 GB | 8.5 GB | **0.0 GB** |

  The concept is sound — keeping loss in BF16 avoids an fp32 upcast that
  saves VRAM in principle. But at this model scale (9B, QLoRA, RTX 3090),
  the measurable impact is 0.0 GB — the loss tensor is not the bottleneck.

  The dominant VRAM leak (7.0 GB) traces entirely to async CUDA streams in
  OffloadActivations (TRL #5700). That fix alone drops VL training from
  20.3→13.3 GB. All other candidate fixes show zero additional effect.

  This PR isn't wrong — it just doesn't move the needle for the specific
  leak we were chasing. The feature flag itself may still be useful for
  other configurations.

  **Full reproducer, HTML report, raw training logs:**
  - https://github.com/butterwecksolutions/Testrepo/tree/main/vram-fix-reproducer
  - https://butterwecksolutions.github.io/Testrepo/vram-fix-reproducer/vram_test_results.html

  Closing this PR. Thanks for the review.

--- Comment by BenjaminBossan at 2026-05-07T10:00:06Z ---
@butterwecksolutions Thanks for double-checking. If you find a situation where this leads to a significant reduction in memory usage, feel free to re-open the PR.


PULL REQUEST #45760: End-to-end test of Gemma 3 + FA2 construction
State: closed  |  Merged: False
Author: jamesbraza  |  Base: main
Labels:
Created: 2026-05-03T19:25:43Z

This small PR adds an end-to-end regression test for the fix in https://github.com/huggingface/transformers/pull/45589 for https://github.com/huggingface/transformers/issues/45588:
1. Builds a sinkless tiny Gemma3 with FA2
2. Runs a forward pass.

Cherry-picked from @truffle-dev's fork; verified on compute cluster with H100.

This PR adds test coverage for the fix in https://github.com/huggingface/transformers/pull/45589 for https://github.com/huggingface/transformers/issues/45588.

@ArthurZucker @Cyrilvallez

--- Comment by github-actions[bot] at 2026-05-03T19:26:50Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3

--- Comment by Cyrilvallez at 2026-05-11T07:00:17Z ---
Hey! This test is just running on forward on fa2, which is already tested in other integration tests! So there is no need to add it again!


PULL REQUEST #45757: fix: align attention_mask padding with appended eos token in clvp
State: closed  |  Merged: False
Author: CharlieKerfoot  |  Base: main
Labels:
Created: 2026-05-03T18:36:20Z

## Summary
In `_pad_extra_bos_eos_tokens` at `src/transformers/models/clvp/modeling_clvp.py:144`, when `add_eos_token=True` the eos token is appended to the right of `input_ids` but `attention_mask` was padded on the left. The mask no longer lines up with the tokens, so the appended eos position gets the wrong mask value and a real token at position 0 is masked out.

## Fix
Pad `attention_mask` on the right `(0, 1)` instead of the left `(1, 0)` so the new mask entry corresponds to the appended eos token.

--- Comment by github-actions[bot] at 2026-05-03T18:37:30Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: clvp

--- Comment by Rocketknight1 at 2026-05-05T13:22:03Z ---
I think this is incorrect - when padding is present, the EOS token is in the middle of the array, not at the right end, and appending a 1 at the end will just enable attention to some random padding token. Appending on the left is correct in this case, even though this is not where the EOS token is, because left-padding with 1 basically extends the 1s one index further into the array, covering the new EOS token location.


PULL REQUEST #45756: fix attribute access in PermuteForRope._apply
State: open  |  Merged: False
Author: CharlieKerfoot  |  Base: main
Labels:
Created: 2026-05-03T18:34:30Z

## Summary
`PermuteForRope._apply` in src/transformers/core_model_loading.py:387 calls `self.config.getattr("num_attention_heads", 1)`, but config objects have no `getattr` method, so this raises `AttributeError` every time the op runs.

## Fix
Use the builtin `getattr(self.config, "num_attention_heads", 1)` to read the attribute with a default fallback, matching the original intent.

## Tests
Loading a model that exercises `PermuteForRope` no longer raises `AttributeError` on the config lookup.

--- Comment by Rocketknight1 at 2026-05-05T13:16:05Z ---
cc @arthurzucker since it was added in #41580 - bug looks real but I think this is dead code that never gets called. Can we just remove the class?


PULL REQUEST #45755: fix: return correct forward output in AriaTextForCausalLM
State: closed  |  Merged: False
Author: CharlieKerfoot  |  Base: main
Labels: Code agent slop
Created: 2026-05-03T18:22:43Z

## Summary
`AriaTextForCausalLM.forward` in `src/transformers/models/aria/modular_aria.py:873` called `super().forward(self, **super_kwargs)` and discarded the result, so the model passed self twice and returned `None` instead of the LM output.

## Fix
Drop the spurious self argument and return the parent forward result.

## Tests
Verified locally that `AriaTextForCausalLM` now returns a `CausalLMOutputWithPast` instead of `None` on a forward pass. The relevant Aria tests selected via `utils/tests_fetcher.py` pass.

--- Comment by github-actions[bot] at 2026-05-03T18:23:53Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria

--- Comment by Rocketknight1 at 2026-05-05T13:09:44Z ---
Your code agent does not understand how the modular system works


PULL REQUEST #45754: Fix mps device check for moe histogram routing
State: closed  |  Merged: False
Author: belamaran96-coder  |  Base: main
Labels:
Created: 2026-05-03T17:26:18Z

# What does this PR do?

Fixes #45685

This PR updates the device check during the histogram calculation in the MoE routing logic (`moe.py` and `finegrained_fp8.py`).

Apple Silicon (`mps`) does not support `histc` for integer types, so it must be cast to a float before the operation, aligning it with the CPU behavior.

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the contributor guideline, Pull Request section?
- [x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes?
- [ ] Did you write any new necessary tests?

## Who can review?

cc @amyeroberts (as discussed in the issue)

--- Comment by github-actions[bot] at 2026-05-03T17:39:37Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45754&sha=bf5603

--- Comment by molbap at 2026-05-04T13:23:55Z ---
I think this is a duplicate of https://github.com/huggingface/transformers/pull/45687 . Feel free to reopen if not


PULL REQUEST #45752: Fix unhandled exception noise from background safetensors conversion thread
State: open  |  Merged: False
Author: dhruv7477  |  Base: main
Labels:
Created: 2026-05-03T12:43:34Z

The background `Thread-auto_conversion` in `modeling_utils.py` was spawned with `ignore_errors_during_conversion=False`. When `get_repo_discussions()` raises `HfHubHTTPError` 403 (discussions disabled on the repo), the exception propagated uncaught inside the thread, and Python printed the full traceback to stderr — the noise reported in #44403.

Since this thread is explicitly fire-and-forget (the comment on line 720 reads "try to launch safetensors conversion for next time"), errors from it should never surface to the user. Changing the flag to `True` causes `auto_conversion` to catch and suppress the exception cleanly.

A prior attempt (#44440) was closed because it was opened two days after the issue while discussion was still ongoing, and made additional changes to `safetensors_conversion.py`. This PR is a single-line fix to the actual root cause.

Fixes #44403

---

Tests run: `python -m pytest tests/utils/test_modeling_utils.py -x -v`
Result: 94 passed, 38 skipped, 1 xfailed

I used an AI assistant to help trace the root cause and identify the relevant code path, but I reviewed the change, ran the tests, and verified the fix against the stack trace in the issue.

--- Comment by github-actions[bot] at 2026-05-03T12:56:34Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45752&sha=ac7fda

--- Comment by dhruv7477 at 2026-05-03T13:55:42Z ---
Re: Failing CI checks

The two failing tests (test_tp_generation for exaone4 and test_ep_forward for gpt_oss) are pre-existing failures unrelated to this PR.

GptOssModelTest::test_ep_forward — tracked by open issue #45161 ("Only TP not working with GPT-OSS MoE model", filed April 1, 2026). The ProcessRaisedException in EP forward is a known infrastructure issue.

Exaone4ModelTest::test_tp_generation — SIGABRT (process crash) in the distributed subprocess. This is a CUDA/NCCL-level crash that cannot be caused by changing the ignore_errors_during_conversion flag on a background Python thread in modeling_utils.py.

This PR touches a single line in the background safetensors conversion thread. It has no interaction with tensor-parallel or expert-parallel computation paths. Could a maintainer re-run the required run_tests check?


Add Conformer model

PULL REQUEST #45751: Add Conformer model
State: open  |  Merged: False
Author: jonghwanhyeon  |  Base: main
Labels:
Created: 2026-05-03T09:00:02Z

# What does this PR do?

[`Parakeet`](https://huggingface.co/docs/transformers/model_doc/parakeet) (based on [`FastConformer`](https://huggingface.co/papers/2305.05084)) is already available in `transformers`, but it does not cover the original [`Conformer`](https://huggingface.co/papers/2005.08100) architecture. `FastConformer` is an efficient variant of `Conformer`, but many existing ASR checkpoints on the model hub were trained with the original `Conformer` architecture and are not compatible with the current `Parakeet`/`FastConformer` implementation.

This PR enables those checkpoints to be converted and used through `transformers` APIs.

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@eustlb @ebezzam @vasqu

--- Comment by github-actions[bot] at 2026-05-03T09:01:12Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, conformer

--- Comment by ebezzam at 2026-05-04T14:02:07Z ---
Hi @jonghwanhyeon thank you for your contribution! Can you take a look at the Wav2VecConformer model? I think it already implements the original conformer, at least according to the [docstring](https://github.com/huggingface/transformers/blob/a5b83a724661471fc55aa9d204be61a3488a2847/src/transformers/models/wav2vec2_conformer/modular_wav2vec2_conformer.py#L381).

--- Comment by jonghwanhyeon at 2026-05-05T00:48:26Z ---
Thanks for your response, @ebezzam

I am also aware that the current `transformers` repository contains code related to `Conformer`.

However, the implementations of those models seem closer to borrowing only parts of the architecture proposed in the [Conformer paper](https://huggingface.co/papers/2005.08100), especially the Conformer encoder layer.

For example, `Wav2Vec2-Conformer` uses Conformer-style attention, but its input is processed in the same way as Wav2Vec2: `raw waveform` -> `code vector`.

By contrast, in `Conformer`, the input is processed in the following order: `raw waveform` -> `mel spectrogram` -> `convolutional subsampling`.

Therefore, I think it would still be useful to add a standalone Conformer implementation that follows the original paper.

--- Comment by jonghwanhyeon at 2026-05-11T19:43:29Z ---
Hi, @ebezzam gentle ping on this.

--- Comment by jonghwanhyeon at 2026-05-12T14:46:16Z ---
Thanks for the feedback, @eustlb .

I agree that `conformer` is a common building block across several audio models, and that it would be ideal to eventually standardize and reuse it via the `modular` approach. For this PR, I intentionally kept the implementation standalone to avoid introducing regressions in existing models that already work.

The main motivation for this PR is to make existing [`conformer` ASR checkpoints](https://huggingface.co/models?sort=downloads&search=conformer+ctc) usable through the standard `transformers` APIs, especially for non-English and on-device ASR use cases.

While `transformers` already supports strong alternatives such as `whisper`, as well as efficient variants like `fastconformer/parakeet`, I think supporting the original `conformer` still has practical value: it offers a reasonable accuracy/compute trade-off, making it useful for local and/or on-device use cases.

I understand the concern about maintenance burden. My hope is that the added burden remains relatively limited, since `conformer` is a stable and widely used architecture.


PULL REQUEST #45749: fix: correct spelling in continuous_api docstring
State: closed  |  Merged: True
Author: Dhruv908615  |  Base: main
Labels:
Created: 2026-05-03T06:21:02Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by Dhruv908615 at 2026-05-03T08:47:02Z ---
Hi, I have fixed the issue as described. Please review when possible. Thank you!

--- Comment by HuggingFaceDocBuilderDev at 2026-05-05T16:13:04Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45749). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45748: Optimize Qwen3 RoPE: precompute cos/sin cache for static rope_type
State: closed  |  Merged: False
Author: RobTand  |  Base: main
Labels: Code agent slop
Created: 2026-05-02T19:12:41Z

## Summary

Precompute the RoPE cos/sin tables for `rope_type == "default"` at module init and `index_select` in `forward`, replacing the per-call `inv_freq @ position_ids` BMM. Dynamic rope types (NTK-aware scaling, etc.) continue to use the existing per-call path via `@dynamic_rope_update`.

## Motivation

1. **Performance**: eliminates a small per-call cuBLAS BMM and an FP32 autocast block on every forward. Most useful at decode where each step pays the latency.
2. **Robustness**: the BMM hits a cuBLAS deterministic-mode edge case with bf16 + `:4096:8` workspace config that produces NaN. Precompute side-steps it. (Underlying issue is arguably PyTorch/cuBLAS, but precompute is a useful workaround that also helps perf.)
3. **Standard pattern**: older RoPE implementations always cached; the dynamic recompute was added specifically to support changing `inv_freq` for long-context scaling.

## What changed

- `src/transformers/models/qwen3/modeling_qwen3.py`:
  - `Qwen3RotaryEmbedding.__init__` now precomputes `cos_cached`/`sin_cached` for `max_position_embeddings` x `head_dim` when `rope_type == "default"`, using `torch.outer` instead of BMM.
  - `Qwen3RotaryEmbedding.forward` indexes into the cache when available and the requested positions fit the static cache; it falls back to the existing BMM path for dynamic rope types and long-position extrapolation.
  - `@dynamic_rope_update` decorator preserved so NTK paths get their inv_freq updates.
  - The cache buffers are non-persistent derived buffers, matching existing rotary buffer practice and avoiding checkpoint bloat. They are rebuilt in `_init_weights` so meta/from_pretrained loading initializes them after `inv_freq` is restored.

## Math equivalence

Verified bit-identical to the existing forward via `torch.testing.assert_close(rtol=0, atol=0)` on representative position vectors. See new test `test_rope_precomputed_cache_matches_legacy_path`.

## Memory

Adds two FP32 buffers of shape `(max_position_embeddings, head_dim)` per Qwen3RotaryEmbedding instance. For Qwen3 with max_position=128k and head_dim=128, that's about 128 MB per RoPE module. There's typically one shared rotary instance per model, so the overhead is modest relative to model weights.

## Status

DRAFT - opened for early feedback on the approach before broader test/benchmark work and before considering whether to mirror this change to other model families (Llama, Mistral, DeepSeek-V3, Gemma 3, etc., which all use a similar dynamic-RoPE forward pattern).

## Test

- New unit test verifies cached and legacy paths produce bit-identical cos/sin.
- Existing Qwen3 tests pass: `pytest tests/models/qwen3/`.

--- Comment by github-actions[bot] at 2026-05-02T19:13:54Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3


Fix split batch size

PULL REQUEST #45747: Fix split batch size
State: closed  |  Merged: False
Author: Prachi-kushwaha  |  Base: main
Labels:
Created: 2026-05-02T19:06:12Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


PULL REQUEST #45746: Fix link to modular transformers documentation
State: closed  |  Merged: True
Author: SangbumChoi  |  Base: main
Labels:
Created: 2026-05-02T13:47:03Z

Updated link to the modular transformers documentation for clarity.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-05-05T13:00:47Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45746). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45745: feat: add crop() to StaticCache layers for assisted generation
State: open  |  Merged: False
Author: ArkaD171717  |  Base: main
Labels:
Created: 2026-05-02T09:31:22Z

Assisted generation needs to roll back the KV cache when candidate tokens get
rejected
`DynamicLayer.crop()` does this by slicing, but `StaticLayer` can't slice
because its tensors have static memory addresses pinned by
`mark_static_address()` for torch.compile/cudagraphs

`StaticLayer.crop()` zeros out the evicted slots with `fill_()`/indexing and
updates `cumulative_length` in-place
Tensor identity is preserved so torch.compile doesn't break

`StaticSlidingWindowLayer.crop()` delegates to the above but raises ValueError
if the window has already been exceeded (same semantics as the dynamic version,
evicted tokens are gone)

Removes the hard ValueError in `_assisted_decoding` that was blocking
StaticCache entirely
Also propagates the main model's `cache_implementation` to the assistant model
in `AssistedCandidateGenerator` instead of always falling back to
`"dynamic_full"`, per the reviewer request on #34797

11 unit tests for crop() (basic, negative, overflow clamp, tensor identity,
crop-then-update, sliding window)
8 integration tests for assisted generation with static cache (including
output-matching against dynamic cache)

Closes #32946

--- Comment by ArkaD171717 at 2026-05-02T18:57:21Z ---
I think @LagunaModelTest test_tp_generation failure is unrelated to this PR its a gloo transport layer mismatch in the tensor parallel test harness. All the other tests passed. Could a maintainer re-run the failed `tests_tensor_parallel_ci` job?

--- Comment by Qodo-Free-For-OSS at 2026-05-03T08:56:06Z ---
Hi, Assisted decoding always calls `outputs.past_key_values.crop(...)`, but for encoder-decoder models `past_key_values` is an `EncoderDecoderCache` whose `crop()` rejects non-dynamic caches, so static cache assisted generation will raise `TypeError` at runtime.

**Severity:** action required | **Category:** correctness

**How to fix:** Support static in EncoderDecoderCache.crop

**Agent prompt to fix** - you can give this to your LLM of choice:

> ### Issue description
> Assisted decoding unconditionally calls `outputs.past_key_values.crop(...)`. For encoder-decoder models, `past_key_values` is an `EncoderDecoderCache`, whose `crop()` currently enforces dynamic-only caches via `check_dynamic_cache()`. After this PR, assisted generation may select static caches, causing a runtime error.
>
> ### Issue Context
> - `_prepare_cache_for_generation` now permits static cache implementations for assisted generation.
> - `_assisted_decoding` still calls `.crop()` on `outputs.past_key_values`.
> - `EncoderDecoderCache.crop()` must either support static `self_attention_cache` or assisted generation must continue to force dynamic caches for encoder-decoder models.
>
> ### Fix Focus Areas
> - src/transformers/cache_utils.py[1620-1627]
> - src/transformers/generation/utils.py[1848-1859]
> - src/transformers/generation/utils.py[3614-3616]
>
> ### Implementation guidance
> Choose one:
> 1) Update `EncoderDecoderCache.crop()` to allow static `self_attention_cache` (e.g., call `self.self_attention_cache.crop(...)` regardless of cache class, and only enforce dynamic for methods that truly require it).
> 2) Gate static-cache allowance in `_prepare_cache_for_generation` for encoder-decoder assisted generation (keep forcing `dynamic_full` there).
> Also add an encoder-decoder assisted-generation test with `cache_implementation="static"`.

We noticed a couple of other issues in this PR as well - happy to share if helpful.

---
*Found by [Qodo](https://qodo.ai) code review*

--- Comment by ArkaD171717 at 2026-05-03T11:02:51Z ---
Thank you for catching that, @Qodo-Free-For-OSS. It's now fixed and pushed. Please do share those other issues! I don't need agent prompts.

--- Comment by ArkaD171717 at 2026-05-07T15:15:26Z ---
Fixed Merge conflict

--- Comment by Rocketknight1 at 2026-05-08T11:18:47Z ---
cc @cyrilvallez for generation

--- Comment by ArkaD171717 at 2026-05-13T17:21:01Z ---
Hello @Cyrilvallez thanks for the reply,

The motivation is torch.compile / CUDA graphs; DynamicCache allocates new tensors on every grow/slice which forces graph recompilation and StaticCache pins tensor addresses via mark_static_address() so the compiled graph stays valid across calls. Assisted gen needs crop() to roll back rejected candidates but because of all the above slicing would create new tensors and break the compiled graph
Thats why StaticLayer.crop() zeroes out evicted slots in place (fill_/indexing) and updates cumulative_length with fill_(). B/c id(tensor) is preserved torch.compile doesnt see new obj and doesnt retrace
So advantage of StaticCache isnt lost but the compiled forward pass stays intact (the rollback itself is O(evicted_tokens) zeroing). The rollback cost is just a memset on evicted region and not a recompilation

Sorry about the test files issue, I moved all of them into existing files and will push that shortly

--- Comment by ArkaD171717 at 2026-05-15T19:57:27Z ---
accidentally closed wrong PR 😭😭Just to be clear this one is still open


PULL REQUEST #45744: [MINISTRAL3] Fix conversion script yarn's apply_scale support.
State: closed  |  Merged: True
Author: juliendenize  |  Base: main
Labels:
Created: 2026-05-02T09:03:59Z

# What does this PR do?

This PR fixes the conversion script for Ministral3 architecture. Models such as https://huggingface.co/mistralai/Mistral-Medium-3.5-128B that can be converted through this script don't have a properly parsed `apply_scale` for yarn leading to poor performance. Indeed this model has vision but still should apply yarn scaling which is prevented by the current main state as the rule to disable yarn scaling was whether the model had vision or not. This was only true for the Devstral release.

See here for the original config format: https://huggingface.co/mistralai/Mistral-Medium-3.5-128B/blob/main/params.json#L26

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@ArthurZucker @Cyrilvallez

--- Comment by github-actions[bot] at 2026-05-02T09:05:38Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ministral3

--- Comment by vasqu at 2026-05-03T12:26:45Z ---
Does the flag always exist? Should we be extra careful about using `.get` with a default value instead?

--- Comment by juliendenize at 2026-05-03T17:40:06Z ---
Yes this flag always exists. Keeping a direct access should always work and if it doesn't it means the config is wrongly made.

--- Comment by vasqu at 2026-05-03T17:50:07Z ---
Perfect thanks for clarifying, merging :)


PULL REQUEST #45743: fix(bitsandbytes): implement reverse_op for Bnb4bitDeserialize and Bnb8bitDeserialize
State: closed  |  Merged: False
Author: Kaisan10  |  Base: main
Labels:
Created: 2026-05-02T07:35:16Z

# What does this PR do?

## Problem

Calling `save_pretrained()` on a model that was loaded with `load_in_4bit=True`
(or `load_in_8bit=True`) and then had LoRA adapters merged raises:

```
File "transformers/core_model_loading.py", in reverse_transform
    kwargs["operations"] = [op.reverse_op for op in self.operations[::-1]]
File "transformers/core_model_loading.py", in reverse_op
    raise NotImplementedError
NotImplementedError
```

`revert_weight_conversion()` is called during `save_pretrained()` and
iterates over `model._weight_conversions`. When the model was loaded with
bitsandbytes 4-bit or 8-bit quantization, a `WeightConverter` containing
`Bnb4bitDeserialize` (or `Bnb8bitDeserialize`) is registered. These
operations load the model by collecting multiple checkpoint sub-keys
(`weight.absmax`, `weight.quant_state.*`, etc.) into a single `Params4bit`
object, but their `reverse_op` was never implemented, so saving always fails.

This affects **any** model + LoRA workflow that uses bitsandbytes 4-bit or
8-bit loading — not just specific architectures.

## Root cause

`Bnb4bitDeserialize` and `Bnb8bitDeserialize` inherit `reverse_op` from
`ConversionOps`, which raises `NotImplementedError` by default. The
transformers documentation states that "Operations are fully reversible",
but these two classes were never given a concrete reverse.

## Fix

Add `Bnb4bitSerialize` and `Bnb8bitSerialize` as the respective reverse
operations, and wire them up via `reverse_op`.

- If the in-memory weight is still a `Params4bit` / `Int8Params` (normal
  re-save of a quantized model), the serializer reconstructs the original
  multi-key checkpoint format via `quant_state.as_dict()`.
- If the weight has already been dequantized (e.g. after LoRA merge into the
  base model), it is returned as a plain tensor under the `weight` key —
  matching the format that a non-quantized `save_pretrained()` would produce.

## How to reproduce

```python
from unsloth import FastLanguageModel  # or standard PEFT
model, tokenizer = FastLanguageModel.from_pretrained(
    "any-lora-adapter-trained-on-4bit-model",
    load_in_4bit=True,
)
model.save_pretrained_merged("output", tokenizer, save_method="merged_4bit_forced")
# → NotImplementedError
```

## Testing

- [x] Unit test: load a 4-bit model, call `save_pretrained`, reload, verify weights match. (Confirmed with facebook/opt-125m)
- [x] Unit test: load a 4-bit model + LoRA, merge, call `save_pretrained`, verify no error and output starts writing. (Confirmed with Gemma4 + Unsloth)
- [ ] Same two tests for 8-bit. (Logic implemented, manual verification pending)

--- Comment by Rocketknight1 at 2026-05-05T12:51:02Z ---
cc @sunmarc maybe

--- Comment by Kaisan10 at 2026-05-13T13:23:52Z ---
My PR wasn't very good.
I made many mistakes.
I'm giving up on this PR.
Thank you.

--- Comment by Copilot at 2026-05-02T07:45:20Z ---
`reverse_op` was added on `Bnb8bitQuantize`, but the `WeightConverter` used for pre-quantized int8 loading registers `Bnb8bitDeserialize` as its operation (see `quantizer_bnb_8bit.py`). As a result, `revert_weight_conversion()` will still hit `Bnb8bitDeserialize.reverse_op` (currently inherited and raising `NotImplementedError`), so the original `save_pretrained()` failure remains. Move/duplicate this `reverse_op` implementation onto `Bnb8bitDeserialize` (and consider whether `Bnb8bitQuantize` needs a reverse op at all).

--- Comment by Copilot at 2026-05-02T07:45:20Z ---
`Bnb8bitSerialize` currently emits `weight.SCB`, but the corresponding int8 `WeightConverter` expects an `SCB` entry (and uses `input_dict["SCB"]` in `Bnb8bitDeserialize`). This mismatch will produce an incorrectly-named key on save and can also prevent the prefix/suffix renaming logic in `WeightConverter.convert()` from working during `revert_weight_conversion()`. The serializer should output keys consistent with the converter patterns (e.g. `SCB` rather than `weight.SCB`, and likely `weight_format` if it is required for round-tripping).

--- Comment by Copilot at 2026-05-02T07:45:21Z ---
For the dequantized case, `Bnb4bitSerialize` returns only `{ "weight": weight }`. During `revert_weight_conversion()`, the reverse converter is anchored on the *first* original source pattern (currently `weight.nested_absmax` in `quantizer_bnb_4bit.py`), so `WeightConverter.convert()` will rename the returned `weight` key into `...weight.nested_absmax` rather than `...weight`, producing an incorrect checkpoint layout. To make the "already dequantized" path produce a plain `...weight` entry, the reverse mapping needs to anchor on `weight` (e.g. by reordering the original `source_patterns` to start with `weight`, or by adjusting serialization/renaming so the anchor key aligns with the desired output name).

--- Comment by Copilot at 2026-05-02T07:45:21Z ---
The newly introduced reverse operations for bitsandbytes (4-bit/8-bit) should be covered by unit tests that exercise `save_pretrained()` after loading a *pre-quantized* checkpoint, including the "dequantized after LoRA merge" scenario described in the PR. There are existing quantization tests under `tests/quantization/bnb/` (including 4-bit serialization tests that are currently skipped on `NotImplementedError`), so adding/adjusting tests there would prevent regressions and validate the key names produced by the serializers.