PULL REQUEST #45555: perf: avoid recomputing rotary_emb for each layer in some Google and ModernBERT models
State: closed  |  Merged: True
Author: casinca  |  Base: main
Labels:
Created: 2026-04-21T18:56:51Z

# What does this PR do?

Following https://github.com/huggingface/transformers/pull/45144#issuecomment-4290633520

> tl;dr: `self.rotary_emb` was called N_layers times but only 2 calls are needed (full attn and swa) other calls  were overwritting the same dict keys, which was inefficient.
> This now gives a little boost depending on `config.layer_types` size, with the 4B model inference can be ~10% faster (see img below for a benchmark with 300 toks)

See picture below (of the comment https://github.com/huggingface/transformers/pull/45144#discussion_r3106982629) for details *(comment link will not be directly available after resolve and will just point to the linked PR)*

  Click to expand image

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

It was discussed with @vasqu and also @zucchini-nlp.
I kept Raushan suggestion with just set() (but can also do `self.rotary_emb.layer_types` it's a minor detail either way.)

--- Comment by casinca at 2026-04-21T18:59:40Z ---
I checked gemma4, they are correctly doing:
https://github.com/huggingface/transformers/blob/85099df959f924e7f67ccc94c290af30d2dda6c0/src/transformers/models/gemma4/modular_gemma4.py#L1236

But I'll check if some other Google models are also doing it the unoptimized way

--- Comment by vasqu at 2026-04-21T20:03:40Z ---
run-slow: gemma3

--- Comment by github-actions[bot] at 2026-04-21T20:05:08Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24743673653)

This comment contains `run-slow`, running the specified jobs:

models: ["models/gemma3"]
quantizations: []

--- Comment by casinca at 2026-04-21T20:21:31Z ---
> Careful approval, just checking with run-slow + update us if you find other models :D

Yep, 4 other models and upon looking, it seems safe to extend

Since it's 1 LoC change, I believe it's not worth extra PR noise, I can broaden the changes here + update title? Let me know what you'd prefer, thanks.

--- Comment by vasqu at 2026-04-21T20:23:52Z ---
Yes sounds good! Can you wait for the slow tests to return results before pushing? Other than that, feel free to add all of them here + lets adjust the PR title

--- Comment by github-actions[bot] at 2026-04-21T20:36:30Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24743673653)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [6adc6aa0](https://github.com/huggingface/transformers/commit/6adc6aa0654effe884a780b2e9a98c773e273dd8) | workflow commit (merge commit) |
| PR | [67620c54](https://github.com/casinca/transformers/commit/67620c545e53e87af304c77eea84650eb8b556b4) | branch commit (from PR) |
| main | [85099df9](https://github.com/huggingface/transformers/commit/85099df959f924e7f67ccc94c290af30d2dda6c0) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by github-actions[bot] at 2026-04-21T20:49:37Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3, gemma3n, modernbert, modernbert_decoder, t5gemma2

--- Comment by zucchini-nlp at 2026-04-22T09:28:53Z ---
run-slow: gemma3, gemma3n, modernbert, modernbert_decoder, t5gemma2

--- Comment by github-actions[bot] at 2026-04-22T09:30:13Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24770917370)

This comment contains `run-slow`, running the specified jobs:

models: ["models/gemma3", "models/gemma3n", "models/modernbert", "models/modernbert_decoder", "models/t5gemma2"]
quantizations: []

--- Comment by HuggingFaceDocBuilderDev at 2026-04-22T09:38:20Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45555). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-22T10:05:32Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24770917370)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [e428da9d](https://github.com/huggingface/transformers/commit/e428da9d62ba75de70ddf883b7aa11ea0ab2bb42) | workflow commit (merge commit) |
| PR | [31bb4ddc](https://github.com/casinca/transformers/commit/31bb4ddc41048b2722660cd9eab6eb891758c598) | branch commit (from PR) |
| main | [77c0e6e7](https://github.com/huggingface/transformers/commit/77c0e6e7ce5537558f07bb9147d0df4a0132e6f3) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by vasqu at 2026-04-22T10:52:32Z ---
Awesome, merging this :)


[docs] multi-turn tool calling

PULL REQUEST #45554: [docs] multi-turn tool calling
State: closed  |  Merged: True
Author: stevhliu  |  Base: main
Labels:
Created: 2026-04-21T17:27:02Z

adds docs for #45485:

- example of multi-turn tool calling and update its no longer limited to the Qwen model family
- light polish (e.g. fix anchor links that have the same titles)

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T17:36:27Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45554). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by SunMarc at 2026-04-22T12:18:01Z ---
maybe we can ask the user to open an issue if they want to have specific model added ?

--- Comment by SunMarc at 2026-04-22T12:20:01Z ---
let's remove instructions field here as it is irrelevant.
```suggestion
```

--- Comment by SunMarc at 2026-04-22T12:20:50Z ---
```suggestion
```

--- Comment by SunMarc at 2026-04-22T12:26:28Z ---
basically, we respect the OpenAI Standard, so it means the following file: https://developers.openai.com/api/docs/guides/function-calling?api-mode=chat
Maybe we can either redirect to this doc or we can try to replicate the same things but with simpler examples ? i have a `test_responses_multi_turn_non_streaming` and `test_chat_multi_turn_non_streaming` if you want some example. Showing the non streaming example will be shorter and better as you did here but I feel like the examples are a bit incomplete as you don't define tools for example.

--- Comment by stevhliu at 2026-04-22T16:29:45Z ---
sounds good, i added a link to the openai doc and a short sentence that the tool used in this example is from the tool defined in the `## Tool calling` section above


[docs] per-request sampling params

PULL REQUEST #45553: [docs] per-request sampling params
State: closed  |  Merged: True
Author: stevhliu  |  Base: main
Labels:
Created: 2026-04-21T16:54:24Z

adds docs for #45026

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T17:04:46Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45553). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by remi-or at 2026-04-22T01:33:29Z ---
```suggestion
Enable `per_request_processors` to apply `temperature`, `top_k`, and `top_p` independently per request within the same forward pass to allow different sampling parameters for different requests (creative, high-temperature outputs versus precise, low-temperature ones for example).
Each parameter must have a non-default value in the `GenerationConfig` to guarantee the associated logits processor is created at runtime. For instance, if per-request temperatures are needed, then the `GenerationConfig` temperature should not be `None` or `1`. Requests with temperatures of 1 can still be created afterwards.


Remove warnings for modernbert

PULL REQUEST #45552: Remove warnings for modernbert
State: open  |  Merged: False
Author: jjjamie  |  Base: main
Labels:
Created: 2026-04-21T15:57:53Z

Gets rid of annoying logging when importing modernbert

```
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 No checkpoint found for ModernBertForMaskedLM.forward. Please add a `checkpoint` arg to `auto_docstring` or add one in ModelConfig's docstring
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 No checkpoint found for ModernBertForSequenceClassification.forward. Please add a `checkpoint` arg to `auto_docstring` or add one in ModelConfig's docstring
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 No checkpoint found for ModernBertForTokenClassification.forward. Please add a `checkpoint` arg to `auto_docstring` or add one in ModelConfig's docstring
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 No checkpoint found for ModernBertForQuestionAnswering.forward. Please add a `checkpoint` arg to `auto_docstring` or add one in ModelConfig's docstring
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 No checkpoint found for ModernBertForMultipleChoice.forward. Please add a `checkpoint` arg to `auto_docstring` or add one in ModelConfig's docstring
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
[run] 🚨 Config not found for model. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py
[run] 🚨 Something went wrong trying to find the model name in the path: /usr/local/lib/python3.12/dist-packages/transformers/models/modernbert/modular_modernbert.py
```

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [X ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

(maybe for later)

--- Comment by Rocketknight1 at 2026-04-22T13:18:53Z ---
Are these warnings still firing on `main`..? Can you give me a reproducer that shows them?

--- Comment by jjjamie at 2026-04-22T13:41:20Z ---
Thanks for the comment. It's possible they're not - I opened this as a placeholder to fill in properly later. Will check.


PULL REQUEST #45551: Add ForSequenceClassification heads for the OLMo family
State: closed  |  Merged: True
Author: earino  |  Base: main
Labels:
Created: 2026-04-21T14:06:50Z

# What does this PR do?

Adds `ForSequenceClassification` for Olmo, Olmo2, and Olmo3 so `AutoModelForSequenceClassification.from_pretrained("allenai/OLMo-2-0425-1B")` (and the Olmo/Olmo3 equivalents) work.

Structure follows the same pattern as Gemma2, Qwen3, and Glm4: one `OlmoForSequenceClassification(LlamaForSequenceClassification): pass` in `modular_olmo.py`, then Olmo2 and Olmo3 each subclass the previous one. After `make fix-repo`, the generated `modeling_*.py` files use the `GenericForSequenceClassification` mixin, same as Jamba, JetMoe, Ministral3, and Gemma3Text.

Scope is the dense chain only. OlmoHybrid and the MoE branch (Olmoe, FlexOlmo) can come as follow-up PRs.

Fixes #45529

## Code Agent Policy

This was drafted with Claude Code. @Rocketknight1 explicitly opted in for this specific change on the coordination issue:

> This is welcome! Sequence classification heads are often not included in the initial PR adding a new causal LM, but we're happy to add them. Your reason for needing it is good, and PRs like this are usually very easy to automate, so I'm happy for it to be mostly AI-written.

https://github.com/huggingface/transformers/issues/45529#issuecomment-4288374995

I read every changed line before each commit, ran the local test suite on my machine, and did GPU validation on real checkpoints before opening this. I can defend each change.

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? #45529
- [x] Did you make sure to update the documentation with your changes? Added autodoc sections to `docs/source/en/model_doc/olmo{,2,3}.md`.
- [x] Did you write any new necessary tests? See test plan below.

## Changes

- `src/transformers/models/olmo/modular_olmo.py`: `class OlmoForSequenceClassification(LlamaForSequenceClassification): pass`
- `src/transformers/models/olmo2/modular_olmo2.py`: subclass of `OlmoForSequenceClassification`
- `src/transformers/models/olmo3/modular_olmo3.py`: subclass of `Olmo2ForSequenceClassification`
- Regenerated `modeling_olmo.py`, `modeling_olmo2.py`, `modeling_olmo3.py` via `make fix-repo`
- Registered all three in `MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING_NAMES` in `models/auto/modeling_auto.py`
- Added autodoc sections to `docs/source/en/model_doc/olmo{,2,3}.md`
- Tests:
  - Olmo and Olmo2 use the older `ModelTesterMixin` pattern. Added the new class to `all_model_classes` and `text-classification`/`zero-shot` to `pipeline_model_mapping`.
  - Olmo3 uses the newer `CausalLMModelTester`. Set `sequence_classification_class = Olmo3ForSequenceClassification`, which auto-enables the three `test_sequence_classification_model*` tests from the base class.

## Test plan

Local, on MPS:

- `make style`, `make typing`, `make check-repo` all pass
- `pytest tests/models/olmo tests/models/olmo2 tests/models/olmo3`: 413 passed, 0 failing. 4 `test_tp_*` tests deselected because they require CUDA multi-GPU.
- Olmo3's three `test_sequence_classification_model*` tests pass

GPU validation notebook with outputs: https://gist.github.com/earino/2bc6f246eef21a36c3c64d64150b9510

Ran on an NVIDIA RTX PRO 6000 Blackwell. For each of `allenai/OLMo-1B-hf`, `allenai/OLMo-2-0425-1B`, and `allenai/Olmo-3-7B-Instruct`:

- `AutoModelForSequenceClassification.from_pretrained(...)` dispatches to the right class
- Forward returns logits of shape `(batch, num_labels)`
- Loss is finite at random init, roughly `ln(num_labels)` as expected
- Backward produces finite gradients for every trainable parameter
- The library's `LOAD REPORT` correctly shows `score.weight | MISSING` (new head) and `lm_head.weight | UNEXPECTED` (causal-LM head unused)

A LoRA fine-tune on IMDB with `allenai/OLMo-2-0425-1B` (4.2M trainable params, 250 steps) brings loss from 1.48 over the first 20 steps down to 0.0005 over the last 20. I only ran the full training loop on one of the three because the classification head implementation is identical across them (same `GenericForSequenceClassification` mixin, different backbones and `*PreTrainedModel` bases), so the training-loop plumbing is shared; smoke-test forward/backward covers the other two. Happy to extend the LoRA run to Olmo and Olmo3 if you'd prefer.

## Who can review?

cc @Rocketknight1 as requested on #45529.

--- Comment by github-actions[bot] at 2026-04-21T14:08:00Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, olmo, olmo2, olmo3

--- Comment by Rocketknight1 at 2026-04-22T13:46:34Z ---
run-slow: auto, olmo, olmo2, olmo3

--- Comment by github-actions[bot] at 2026-04-22T13:48:07Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24781888832)

This comment contains `run-slow`, running the specified jobs:

models: ["models/auto", "models/olmo", "models/olmo2", "models/olmo3"]
quantizations: []

--- Comment by HuggingFaceDocBuilderDev at 2026-04-22T13:57:23Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45551). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-22T14:19:45Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24781888832)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [22f14e8d](https://github.com/huggingface/transformers/commit/22f14e8d4f100c78685f7faa8329947e15760ba5) | workflow commit (merge commit) |
| PR | [812cba00](https://github.com/earino/transformers/commit/812cba005964768e456af5ab5c09d59584e61d3f) | branch commit (from PR) |
| main | [8fb7c7e5](https://github.com/huggingface/transformers/commit/8fb7c7e57ab7658ca930ede58118a9c0f9f5f206) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


Add runner selection for mi325 GPU type

PULL REQUEST #45550: Add runner selection for mi325 GPU type
State: open  |  Merged: False
Author: glegendre01  |  Base: main
Labels:
Created: 2026-04-21T12:37:31Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T12:47:50Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45550). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Rocketknight1 at 2026-04-22T12:54:28Z ---
cc @ydshieh @tarekziade for the SSH runner maybe?


PULL REQUEST #45549: fix: apply channel averaging correctly in audio feature extractors
State: open  |  Merged: False
Author: jonghwanhyeon  |  Base: main
Labels:
Created: 2026-04-21T11:42:03Z

# What does this PR do?

This PR fixes incorrect channel averaging in the following audio feature extractors:

- `CohereAsrFeatureExtractor`
- `ParakeetFeatureExtractor`
- `Phi4MultimodalFeatureExtractor`
- `LasrFeatureExtractor`
- `VoxtralRealtimeFeatureExtractor`

The bug had two parts:

- batched tensor inputs shaped `(batch, channels, length)` were averaging the last dimension instead of the channel dimension.
- batched sequence inputs were just computing `speech.mean(...)` without writing the result back into `raw_speech`.

This change averages batched tensor inputs across channels with `mean(1)` and writes the mono-converted sample back to `raw_speech` for sequence inputs.

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@eustlb @ebezzam @vasqu

--- Comment by github-actions[bot] at 2026-04-21T11:43:12Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cohere_asr, lasr, parakeet, phi4_multimodal, voxtral_realtime


PULL REQUEST #45548: Fix EP + DeepSpeed ZeRO-3 loading via accelerate launch
State: open  |  Merged: False
Author: AmineDiro  |  Base: main
Labels:
Created: 2026-04-21T11:08:38Z

## Issue
Expert Parallelism (`DistributedConfig(enable_expert_parallel=True)`) hangs during model loading when launched through `accelerate launch` with a DeepSpeed ZeRO-3 config. EP works on its own (via `torchrun`) and ZeRO-3 works on its own — but the two conflict inside `from_pretrained` because every ZeRO-3 code path is gated on a single env-driven flag (`is_deepspeed_zero3_enabled()`), and EP needs the *non*-ZeRO-3 path at every one of those gates.

When you run EP through `accelerate launch` with DeepSpeed ZeRO-3, the environment variable makes `is_deepspeed_zero3_enabled()` return `True` everywhere. Every gate takes the ZeRO-3 path. But EP is fundamentally incompatible with ZeRO-3's initialization, as they shard weights in completely different ways:

- **ZeRO-3**: creates lazy partitioned params via `deepspeed.zero.Init()`, then loads weights through `GatheredParameters` (all-gather before writing, re-partition after).
- **EP**: creates a model on the meta device, registers sharding hooks via `distribute_model()`, then loads weights through the standard path where `shard_and_distribute_module` slices each expert tensor by EP rank.

These two can't coexist in the same loading flow. ZeRO-3's lazy params break EP's sharding hooks. EP's meta tensors break ZeRO-3's `GatheredParameters` (which expects `ds_id`, `ds_shape` attributes).

## Fix
This PR routes EP through the standard (non-zero3) path inside `from_pretrained`, lets `distribute_model()` shard experts as usual, and then lets `deepspeed.initialize()` wrap the already-loaded, already-sharded model afterward.

1. **`get_init_context`** accepts `distributed_config`; when EP+DS, use **meta device** (not `zero.Init`, not real tensors). Meta allocation is free, and `init_weights()` is skipped; checkpoint weights overwrite everything anyway.
2. **`from_pretrained`** clears `device_map` set by `initialize_tensor_parallelism` when EP+DS. EP needs all ranks to read all shard files for the hooks, so we skip the `accelerate` dispatch split.
3. **`_load_pretrained_model`** when EP+DS, skips the zero3 loading branch and uses the standard `convert_and_load_state_dict_in_model` path, passing `model.tp_plan` (the property, which returns the EP plan when EP is on) instead of `model._tp_plan`.
4. **`_move_missing_keys_from_meta_to_device`** when EP+DS, does **not** early-return; runs the standard path to move meta buffers (`inv_freq`, etc.) to CPU.
5. **`_initialize_missing_keys`** when EP+DS, uses standard `initialize_weights()` (no `GatheredParameters`, since params are real/empty, not ZeRO-3-partitioned).

## Test

Minimal EP + DeepSpeed ZeRO-3 verification. Smulates `accelerate launch` by setting HfDeepSpeedConfig so
`is_deepspeed_zero3_enabled()` returns True and  the signal that made `from_pretrained` hang before the fix.

Run on 4xH100:

```python
import os
import torch
import torch.distributed as dist
import deepspeed
from deepspeed import comm as ds_comm

from transformers import AutoModelForCausalLM
from transformers.integrations.deepspeed import HfDeepSpeedConfig
from transformers.distributed.configuration_utils import DistributedConfig

local_rank = int(os.environ["LOCAL_RANK"])
world_size = int(os.environ["WORLD_SIZE"])

dist.init_process_group("nccl")
torch.cuda.set_device(local_rank)

ds_config = {
    "train_batch_size": world_size,
    "train_micro_batch_size_per_gpu": 1,
    "bf16": {"enabled": True},
    "zero_optimization": {"stage": 3, "overlap_comm": True, "contiguous_gradients": True},
}
_dschf = HfDeepSpeedConfig(ds_config)  # strong ref; the global weakref dies if GC'd

mesh = dist.init_device_mesh("cuda", (world_size,))
model = AutoModelForCausalLM.from_pretrained(
    "openai/gpt-oss-20b",
    dtype=torch.bfloat16,
    distributed_config=DistributedConfig(enable_expert_parallel=True),
    device_mesh=mesh,
    attn_implementation="eager",
)
model = model.to(f"cuda:{local_rank}")

if dist.get_rank() == 0:
    w = model.model.layers[0].mlp.experts.gate_up_proj
    print(f"expert shape per rank (post-EP, pre-DS): {tuple(w.shape)}  (32 experts / EP=4 = 8)")

ds_comm.init_distributed("nccl")
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)
engine, optimizer, _, _ = deepspeed.initialize(model=model, optimizer=optimizer, config=ds_config)

x = torch.randint(0, 1000, (1, 8), device=f"cuda:{local_rank}")
out = engine(input_ids=x, labels=x.clone(), use_cache=False)
engine.backward(out.loss)
engine.step()

if dist.get_rank() == 0:
    print(f"loss={out.loss.item():.4f}")

dist.destroy_process_group()

```

## Before submitting
- [x] I confirm that this is not a pure code agent PR.
- [x] Did you read the [contributor
guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?

## Who can review?

@3outeille  @ArthurZucker (distributed / TP / EP  implementation)

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T11:19:09Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45548). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by AmineDiro at 2026-04-22T09:56:44Z ---
> I am not seeing where
>
> > lets distribute_model() shard experts as usual, and then lets deepspeed.initialize() wrap the already-loaded, already-sharded model afterward.
>
> happens? What I mean by this is before did you already have to call deepspeed.initialize() or not and why don't we return `deepspeed.initialize(self)`? ( I could be lacking context so cc @SunMarc as well)

The `distribute_model()` and `deepspeed.initialize()` sequence happens across two files and it's not all in this PR. Inside from_pretrained we have :
```python
# modeling_utils.py ~L4164
if _torch_distributed_available and device_mesh is not None:
   model = distribute_model(model, tp_plan, distributed_config, device_mesh, tp_size)
```

`distribute_model()` runs on the meta-device model **before** weight loading. It only registers hooks on the EP-relevant modules and nothing else. Then `_load_pretrained_model` runs, and when it hits an expert key that matches the EP plan, `shard_and_distribute_module ` calls  `GroupedGemmParallel.shard_tensor()` slices the expert dim by EP rank during the write. At the end, we get back a regular PreTrainedModel whose expert params are already the per-rank shards ( for example [8, 2880, 5760] instead of [32, 2880, 5760] for gpt-oss-20b on EP=4).

Outside `from_pretrained`: `accelerator.prepare(model)`  calls `deepspeed.initialize(model, optimizer, config)` afterwards, which ZeRO-3-wraps the already-sharded model. This part is entirely the caller's responsibility from what I can understand 🤔

> why not return deepspeed.initialize(self)?
Do you mean to return it from `from_pretrained` directly?  If that's the case, I think the API would need to change. The `deepspeed.initialize` needs an optimizer and a full DS config. Also, in `accelerate` we call the `from_pretrained`, construct an optimizerthen calls `accelerator.prepare(model, optimizer)` which internally does
deepspeed initialize. If from_pretrained returned a `DeepSpeedEngine`, `accelerator.prepare` would re-wrap it and probably fail ??

--- Comment by AmineDiro at 2026-04-22T10:09:55Z ---
> I do want and did not know we could bypass zero's "sharding" of the weights. This would be quite ideal if we do the sharding / device placement.

Yeah, I was surprised it worked too 😅 . But  `deepspeed.initialize` itself will ZeRO-3-partition whatever it's handed. but `zero.Init` is probabl required to avoid holding the full model on each rank during creation, that's why think it's a clearner pattern.  Skipping `zero.Init` would work in models where we have TP/EP but for densemodels where we don't have TP sharding, it would probably OOM :/

--- Comment by ArthurZucker at 2026-04-22T10:21:42Z ---
For Dense models the whole purpose of our weight loader is to avoid holding all tensor, but instead of reading on 1 rank / sharding (super slow) we leverage shard on read. But yeah otherwise fine to keep it scope but weight converter is important!

--- Comment by AmineDiro at 2026-05-05T13:15:42Z ---
Friendly ping @3outeille @ArthurZucker 🤗

With #45621 now in main, the EP infra this PR builds on is fully good to go.

Scope is still narrow loading-side only: from_pretrained returns a correctly EP-sharded model that deepspeed  then wraps.  Repro in the description still passes against the current main; happy to rebase if it helps.

--- Comment by AmineDiro at 2026-05-13T07:58:53Z ---
@ArthurZucker
The CI is failing The tests never actually ran (the "Run tests" step is unfinished in all 5 jobs; they stopped at "Download and extract HuggingFace hub cache") ??

I don't think this is related to the code in this PR ?

--- Comment by ArthurZucker at 2026-04-22T05:40:22Z ---
btw the fix can probably be applied to all DS3 no? not just this?
DS3 does not work with moe, nor with any "conversion"no ?

--- Comment by AmineDiro at 2026-04-22T09:37:39Z ---
 Good point, but I'd keep it narrow to EP+DS3 for now. Two reasons:

 1. I think dense + DS3 still benefits from zero.Init. From what I understand the whole purpose of zero.Init is to avoid a peak-memory spike by partitioning params across ranks during creation. If we switched dense DS3 to meta + standard loader, every rank would temporarily hold a full-precision copy of the weights before deepspeed.initialize partitions them 🤔

 >  nor with any "conversion"no ?
 2. Yes you're right, I just realized that. The `_load_state_dict_into_zero3_model` doesn't run `WeightConverter`, so any model that depends on a conversion mapping  will silently load the wrong keys under DS3.  I think, though, this might need a separate issue. Fixing it probably means routing the DS3 through `convert_and_load_state_dict_in_model` with a gather around the per-param ?? I'm happy to open a follow-up issue.

wdyt ?

--- Comment by ArthurZucker at 2026-04-22T10:20:19Z ---
> partitioning params across ranks during creation.

we do shard on read, so in the same way I think our peak memory is equivalent (tho I might be wrong)

for 2. yeah that can be nice

--- Comment by AmineDiro at 2026-04-22T14:20:51Z ---
Oh ok I missed that part , I'll read more carefully! So I can open up an issue with suggested fix 👍🏼


PULL REQUEST #45547: Add disable_mmap kwarg to from_pretrained with hf-mount auto-detection
State: closed  |  Merged: True
Author: rtrompier  |  Base: main
Labels:
Created: 2026-04-21T09:01:12Z

## What

Adds a new `disable_mmap` kwarg to `PreTrainedModel.from_pretrained` (and plumbs it through `load_state_dict` / `LoadStateDictConfig` / `_load_pretrained_model`). When enabled, safetensors checkpoints are read fully into memory and parsed via `safetensors.torch.load(bytes)` instead of being memory-mapped with `safe_open`.

## Why

- Loading a safetensors model from a FUSE filesystem can deadlock the Python process when mmap + kernel readahead + the parallel state-dict loader trigger many concurrent page-faults. We observed this in production on HF Spaces/Endpoints, which expose Hub repos via a FUSE mount named `hf-mount`: captured kernel stacks showed Python hanging on page-faults with a saturated FUSE queue.
- Bypassing mmap (reading the file straight into RAM) side-steps the deadlock with a modest RAM cost.
- `diffusers` already ships the exact same kwarg (`disable_mmap` in `diffusers/models/model_loading_utils.py`); adding it to `transformers` makes the two libraries consistent and unblocks users who currently hit `TypeError: ...__init__() got an unexpected keyword argument 'disable_mmap'` when they try to pass it through.

## How

- New `_is_on_hf_mount(path)` helper parses `/proc/mounts` on Linux and returns `True` if `path` resolves under a mountpoint whose device string is `hf-mount` (returns `False` on non-Linux).
- `load_state_dict(...)` gains a `disable_mmap: bool | None = None` kwarg. When `None`, it auto-detects via `_is_on_hf_mount`. When truthy (and `map_location != "meta"`), the safetensors file is loaded with `safetensors.torch.load(open(path, "rb").read())`.
- `LoadStateDictConfig` gets a matching `disable_mmap` field; `from_pretrained` forwards its new kwarg into it; the multi-shard safetensors branch in `_load_pretrained_model` mirrors the same no-mmap path per file.
- Auto-detection is a no-op on every platform except Linux with an `hf-mount` FUSE mount, so there's no behavior change for existing users.

## Tests

Adds `DisableMmapLoadingTest` in `tests/utils/test_modeling_utils.py` covering:
- `_is_on_hf_mount` matches when `/proc/mounts` lists `hf-mount` for the path.
- `_is_on_hf_mount` does not match for a regular filesystem.
- `_is_on_hf_mount` short-circuits to `False` on non-Linux platforms.
- `load_state_dict(..., disable_mmap=True)` returns tensors equal to the mmap path.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T09:11:50Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45547). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ArthurZucker at 2026-04-21T09:13:10Z ---
@bot /style

--- Comment by github-actions[bot] at 2026-04-21T09:13:50Z ---
Style fix bot fixed some files and pushed the changes.

--- Comment by ArthurZucker at 2026-04-21T09:10:51Z ---
```suggestion
                    if load_config.disable_mmap or _is_on_hf_mount(file)
                        with open(file, "rb") as _fh:
                            merged_state_dict.update(_safe_load_bytes(_fh.read()))
                        continue
```

--- Comment by ArthurZucker at 2026-04-21T09:12:21Z ---
torch.testing.assert_allclose but a nit


PULL REQUEST #45546: feat: Add GGUF loading support for Llama 4 (text)
State: open  |  Merged: False
Author: garybadwal  |  Base: main
Labels:
Created: 2026-04-21T08:58:51Z

# What does this PR do?

Fixes #33260

Adds GGUF loading support for the **Llama 4** text backbone (`llama4_text`), the last major Llama-family architecture that still raised `GGUF model with architecture llama4 is not supported yet.` Users can now do:

```python
AutoModelForCausalLM.from_pretrained(
    "unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF",
    gguf_file="Llama-4-Scout-17B-16E-Instruct-Q2_K_L.gguf",
)
```
Scope is text-only, matches what llama.cpp's convert_hf_to_gguf.py emits for Llama 4 (vision tensors are skipped there), and is consistent with the existing gemma3_text GGUF path.

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@SunMarc  — you own #33260 and the GGUF loader; would appreciate your review.
cc @MekkCyber @Isotr0py — you weighed in on #36144 (DeepSeek GGUF) and have context on the GGUF MoE pipeline.
cc @ArthurZucker — as maintainer of the Llama 4 text model in case anything about the MoE / tokenizer path looks off.

--- Comment by garybadwal at 2026-04-21T14:38:26Z ---
Thanks @SunMarc, I'll check these and add more strong test case in this.

Again thank you for your precision time. 🫡

--- Comment by garybadwal at 2026-04-25T06:47:49Z ---
Hey @SunMarc, thanks for the feedback! I've addressed all three comments:

- Removed `test_llama4_config_mapping` and `test_llama4_architecture_mapping` as suggested.
- Added a proper `test_llama4_q2_k_l_tokenizer` test that saves/reloads the tokenizer and validates round-trip encoding.
- Updated `test_llama4_q2_k_l` to assert against an exact `EXPECTED_TEXT` string instead of the weak length check.

Regarding the CI failure, it's in `GlmImageModelTest::test_sample_generate_dict_output` which is a pre-existing tensor size mismatch in `modeling_glm_image.py`.

My PR only touches ggml.py, `modeling_gguf_pytorch_utils.py`, and the GGUF test file, so this failure is unrelated to my changes. But if there is anything that I can do about this please let me know.

--- Comment by garybadwal at 2026-05-04T16:30:08Z ---
Hi @SunMarc can you review the PR once again ? I made the changes you requested.

--- Comment by github-actions[bot] at 2026-05-10T10:21:38Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ggml

--- Comment by garybadwal at 2026-05-11T07:33:12Z ---
Hi @SunMarc, just wanted to check with you if you got time to look into this PR, I made the changes you requested.

--- Comment by SunMarc at 2026-05-19T05:35:20Z ---
@ArthurZucker is currently working on a refactor of gguf integration. So i would like to wait a bit before proceeding with this PR ! https://github.com/huggingface/transformers/pull/44794

--- Comment by garybadwal at 2026-05-20T06:47:10Z ---
> @ArthurZucker is currently working on a refactor of gguf integration. So i would like to wait a bit before proceeding with this PR ! #44794

No problem @SunMarc, just let me know whenever you want me to make changes of required in this PR.

Thank you for your reply 🫡

--- Comment by SunMarc at 2026-04-21T12:25:45Z ---
those tests are a bit weak, not sure that passing the expected_mappings is a good enough test. we need to test the model and the tokenizer.

--- Comment by SunMarc at 2026-04-21T12:25:56Z ---
not needed

--- Comment by SunMarc at 2026-04-21T12:26:10Z ---
this is not enough for a test

--- Comment by garybadwal at 2026-04-25T06:53:47Z ---
addressed 👍🏻

--- Comment by garybadwal at 2026-04-25T06:54:03Z ---
addressed 👍🏻

--- Comment by garybadwal at 2026-04-25T06:54:09Z ---
addressed 👍🏻


fix table update versions

PULL REQUEST #45544: fix table update versions
State: closed  |  Merged: True
Author: tarekziade  |  Base: main
Labels:
Created: 2026-04-21T07:51:02Z

# What does this PR do?

When running locally with `3.10.19` I noticed `make fix-repo` skipped the table update,

```
Table updated only when running 3.10.x, detected version is 3.10.19 (main, Oct  9 2025, 15:25:03) [Clang 17.0.0 (clang-1700.6.3.2)].
```
Which led me to push the `mlinter` split without proper update of that dep.

This patch fixes the behavior and updates the table.

--- Comment by tarekziade at 2026-04-21T07:53:12Z ---
Introduced in https://github.com/huggingface/transformers/commit/e958b5647768949c4880d5b2570a6e73396b47ac

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T08:00:59Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45544). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-21T09:35:12Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45544&sha=4cf538

--- Comment by ArthurZucker at 2026-04-21T08:15:53Z ---
this will break pypi upload as we cannot have pinned git deps!
```bash
HTTP Error 400: Invalid value for requires_dist. Error: Can't have direct dependency: 'example @ git+https://github.com/example/example.git@v0.1.5'
```


ci: OTEL support

PULL REQUEST #45543: ci: OTEL support
State: open  |  Merged: False
Author: tarekziade  |  Base: main
Labels:
Created: 2026-04-21T07:23:23Z

# What does this PR do?

Adds OTEL support for pytest runs

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T07:34:21Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45543). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45541: Fix local_files_only tokenizer fallback when tokenizer files are missing (Issue 45538)
State: open  |  Merged: False
Author: Brianzhengca  |  Base: main
Labels:
Created: 2026-04-21T01:29:43Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

Fixes https://github.com/huggingface/transformers/issues/45538

## Root Cause

`from_pretrained(..., local_files_only=True)` let missing tokenizer files resolve to None and still proceeded into _from_pretrained(...). That allowed a stub tokenizer to initialize instead of raising, which led to the bogus huge `model_max_length`.

## Describe the Fix
The fix adds a fail-closed check in `PreTrainedTokenizerBase.from_pretrained`.

After resolving files, it now checks whether all real tokenizer assets for that class resolved to None.
  If they did, it raises the existing `Can't load tokenizer... OSError` instead of continuing into
  `_from_pretrained(...)` and constructing a stub tokenizer

## Local Tests
- [x] Check with `BertTokenizer` and `CLIPTokenizer` using a nonexistent model id and
     `local_files_only=True`.
     Before the fix, they loaded unexpectedly with the huge fallback max length.
     After the fix, they raised the expected `OSError`.

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. (https://github.com/huggingface/transformers/issues/45538)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @Cyrilvallez

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-04-22T04:58:35Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45541). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Brianzhengca at 2026-04-22T08:16:38Z ---
@ArthurZucker, just fixed my code for the tests. Please take a look when you have time. Thank you!


PULL REQUEST #45540: Fix cross-attention cache layer type for T5Gemma2 long inputs
State: closed  |  Merged: True
Author: Beichen-Ma  |  Base: main
Labels:
Created: 2026-04-21T01:17:27Z

Fixes #45521. Cross-attention in `T5Gemma2ForConditionalGeneration` is supposed to attend to all encoder tokens, but for inputs whose encoder length is >= sliding_window (default 4096) generation crashes with:
```
RuntimeError: The size of tensor a (4097) must match the size of tensor b (5018) at non-singleton dimension 3
```

The root cause was in `T5Gemma2ForConditionalGeneration._prepare_cache_for_generation`, the cross-attention config was being stripped of its sliding-window settings via `del`:
```
cross_attn_config = copy.deepcopy(self.config.get_text_config(decoder=True))
del cross_attn_config.sliding_window
del cross_attn_config.layer_types
```

`T5Gemma2DecoderConfig` with defaults `sliding_window: int | None = 4096` and `layer_types: list[str] | None = None`. Removing the instance attributes therefore makes attribute lookup fall back to those class defaults, so `cross_attn_config` once again is `sliding_window=4096`.

`DynamicCache.__init__` sees `sliding_window=4096` with `layer_types=None` will auto-derives `layer_types = ["sliding_attention"] * num_hidden_layers`, and instantiates `DynamicSlidingWindowLayer` for every cross-attention layer. On update, those layers truncate the encoder K/V states to the last `sliding_window-1` tokens:
```
self.keys   = full_key_states[:,   :, -self.sliding_window + 1 :, :]
self.values = full_value_states[:, :, -self.sliding_window + 1 :, :]
```
So when `enc_len == 4096`, the cached cross-attention keys end up with shape [..., 4095, head_dim], which (after concatenation with the decoder self-attention key in `T5Gemma2MergedAttention.forward`) yields an `attn_weights` last-dim of 4097. Hence the mismatch.

## Fix
Explicitly set `sliding_window` to null and `layer_types` to full attention for all layers, instead of deleting the instance attributes.

## Tests
- Added test `T5Gemma2ModelTest::test_cross_attention_cache_is_not_sliding`, which asserts that after `generate()` every layer of `output.past_key_values.cross_attention_cache` is `DynamicLayer`. Confirmed test fails on main branch and passes on this branch.
- `tests/models/t5gemma2/test_modeling_t5gemma2.py` passes.
- Verified provided end-to-end [reproducer](https://github.com/junos-ai-org/jiutian-treb/blob/experiment-setup/experiments/t5gemma_vs_qwen_treb/insights/transformers_bug_repro.py) passed after the fix.
```
python /tmp/transformers_bug_repro.py
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████| 1327/1327 [00:04<00:00, 323.79it/s]

--- target=2500 ---
[transformers] The following generation flags are not valid and may be ignored: ['top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
OK  (input=2500, output=17)

--- target=3500 ---
OK  (input=3500, output=17)

--- target=4000 ---
OK  (input=4000, output=17)

--- target=4090 ---
OK  (input=4090, output=17)

--- target=4100 ---
OK  (input=4100, output=17)

--- target=4500 ---
OK  (input=4500, output=17)

--- target=5000 ---
OK  (input=5000, output=17)

--- target=8000 ---
OK  (input=8000, output=17)
```

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by Rocketknight1 at 2026-04-21T12:20:49Z ---
cc @vasqu since you were active in the original issue!

--- Comment by vasqu at 2026-04-27T14:03:17Z ---
run-slow: t5gemma2

--- Comment by github-actions[bot] at 2026-04-27T14:04:42Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24999654601)

This comment contains `run-slow`, running the specified jobs:

models: ["models/t5gemma2"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-27T14:14:43Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24999654601)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [51d3ecdb](https://github.com/huggingface/transformers/commit/51d3ecdb93ba7255a3ad670f8b904ad6ed95fbf9) | workflow commit (merge commit) |
| PR | [b8c3dffd](https://github.com/Beichen-Ma/transformers/commit/b8c3dffd2c27fe11ba804ec35e0c584cc87b9b6b) | branch commit (from PR) |
| main | [ded2b747](https://github.com/huggingface/transformers/commit/ded2b747bde5e9933c140c29ca3615d759f5744d) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by github-actions[bot] at 2026-04-27T15:33:50Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: t5gemma2

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T15:44:10Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45540). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-04-23T15:54:56Z ---
Imo, we can be a bit harder on the test and also create a minimal sliding window where we would expect it to crash otherwise

--- Comment by Beichen-Ma at 2026-04-25T07:31:45Z ---
Thank you for your suggestion. I agree with it and added a minimal sliding window to the test.


PULL REQUEST #45539: [modular] Fix modular logic broken in #45045
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-04-21T00:24:28Z

# What does this PR do?

Revert all changes to modular that were added in https://github.com/huggingface/transformers/pull/45045. The changes break modular's purpose, as it takes the nodes from modular verbatim, meaning they cannot use inheritance which is the desired usage of modular. See e.g. [here](https://github.com/huggingface/transformers/pull/45045/changes#diff-25076f03f50607a3b76ad0a3166d6ac411da62f48bd8e37f6529ea6015842dd6L168-R174) where the nice inheritance had to be removed and use full definition instead, which is really against modular's purpose... There are many more similar instances in the same PR.

This PR does the same, but without breaking modular's logic!

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T00:34:53Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45539). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-21T09:35:58Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: conditional_detr, deepseek_vl, deformable_detr, efficientloftr, ernie4_5_vl_moe, glm_image, grounding_dino, lightglue, llava_onevision, mask2former, paddleocr_vl, rt_detr, segformer, smolvlm, video_llama_3, yolos

--- Comment by Cyrilvallez at 2026-04-22T01:15:55Z ---
Got offline approval from the lm_head @ArthurZucker, merging!


PULL REQUEST #45537: NVFP4 quantization: streaming loader, fused MoE experts (Qwen + Llama…
State: closed  |  Merged: False
Author: ddreeselogs  |  Base: main
Labels: Code agent slop
Created: 2026-04-20T23:59:40Z

(no description)


PULL REQUEST #45535: [Sam3LiteText] Remove unnecessary modules/configs
State: closed  |  Merged: True
Author: yonigozlan  |  Base: main
Labels:
Created: 2026-04-20T23:17:21Z

I missed that some last minute modifications to modular in #44320 added unnecessary configs and modules in the modeling code

Cc @vasqu, @NielsRogge

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T23:27:54Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-04-21T12:08:15Z ---
run-slow: sam3_lite_text

--- Comment by github-actions[bot] at 2026-04-21T12:09:12Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: sam3_lite_text

--- Comment by github-actions[bot] at 2026-04-21T12:09:31Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24721495772)

This comment contains `run-slow`, running the specified jobs:

models: ["models/sam3_lite_text"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-21T12:18:48Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24721495772)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [a45f471e](https://github.com/huggingface/transformers/commit/a45f471ee71a99358bd6e254da0f311c055d3eb7) | workflow commit (merge commit) |
| PR | [79142e24](https://github.com/yonigozlan/transformers/commit/79142e24bf1d8d2a690bb8f190a51daf46e50f13) | branch commit (from PR) |
| main | [67fb8bb3](https://github.com/huggingface/transformers/commit/67fb8bb36987c8171b96e15acd49ea0a837693c7) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


🚨 [ALM] Add base model without head

PULL REQUEST #45534: 🚨 [ALM] Add base model without head
State: open  |  Merged: False
Author: eustlb  |  Base: main
Labels:
Created: 2026-04-20T15:09:17Z

# What does this PR do?

This is motivated to simplify changes required for vLLM compatibility as seen in [vLLM PR #39330](https://github.com/vllm-project/vllm/pull/39330)

Fix a discrepancy in ALMs design compared to VLMs that for most of them don't have a base model class because the use the causal model for the text model directly, while Llava style (which is indeed more aligned with the lib philosophy) is rather:
XxModel: encoder + projector + language **base model** backbone
XxForConditionalGeneration: XxModel + lm_head

## why do we actually have this discrepancy?

qwen2_audio was the first ALM in transformers, modelling reflects the status of llava when it was added
all the ALMs merged since then replicated this.

llava and VLMs evolved thanks to
- #37033
- #41589
- #42156

## 🚨 Breaking changes
We don't fully ensure BC, **this is breaking:**
- `AutoModel.from_pretrained("Qwen/Qwen2-Audio-7B")` now returns `Qwen2AudioModel`, not `Qwen2AudioForConditionalGeneration`, some for other ALMs 🚨
- Auto mappings: AutoModel now returns base model (`Qwen2AudioModel`)
- likewise model.language_model is still returned but it now returns base model and not the causal one as before 🚨
- `model.language_model` on `ForContionalGeneation` class is not accessible anymore. Use model.get_decoder()

## how do we ensure BC (the more possible)

- [x] weight loading: thanks to the dynamic weight loader and conversion mapping
- [x] ensure models cards for the references hub checkpoints in doc and test don't use `AutoModel` (since this will be broken)

## models to which this applies
- [x] audioflamingo3
- [x] glmasr
- [x] granite_speech
- [x] granite_speech_plus
- [x] musicflamingo
- [x] qwen2_audio
- [x] vibevoice_asr
- [x] voxtral
- [x] voxtral_realtime

TODO
- [x] make sure to remove the get/set_input_embeddings when #46029 is merged

--- Comment by github-actions[bot] at 2026-05-13T15:03:07Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25807273349)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [0efc926d](https://github.com/huggingface/transformers/commit/0efc926dcc2dc8990e4535c88e0d258418d7408b) | workflow commit (merge commit) |
| PR | [69964108](https://github.com/huggingface/transformers/commit/69964108a28499d62caa04ee8dfa026d8047b706) | branch commit (from PR) |
| main | [98a25186](https://github.com/huggingface/transformers/commit/98a2518663d5824ee0d2b01359d2083abe88a2b3) | base commit (on `main`) |

⚠ No test being reported (jobs are skipped or cancelled)!

--- Comment by github-actions[bot] at 2026-05-13T15:04:47Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25807479027)

This comment contains `run-slow`, running the specified jobs:

models: ["models/audioflamingo3", "models/auto", "models/glmasr", "models/granite_speech", "models/granite_speech_plus", "models/musicflamingo", "models/qwen2_audio", "models/vibevoice_asr"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-13T15:23:34Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25807479027)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [5f1de948](https://github.com/huggingface/transformers/commit/5f1de9480e1ecf3e755ad4359bb3c7ce01542e54) | workflow commit (merge commit) |
| PR | [5184f47c](https://github.com/huggingface/transformers/commit/5184f47cf716a4e4b2197d0262af0e9bb57eddfd) | branch commit (from PR) |
| main | [98a25186](https://github.com/huggingface/transformers/commit/98a2518663d5824ee0d2b01359d2083abe88a2b3) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-05-13T16:47:19Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45534). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by eustlb at 2026-05-18T17:46:21Z ---
@ArthurZucker thanks for the review, addressed your comments 🤗 important note:
following #42156, I simply removed the properties used to access model.language_model, etc. My initial preference would have been to go through a deprecation cycle first and raise a warning when users access model.language_model, prompting them to use model.get_decoder() instead. However, since we did not do that for VLMs, I kept the behavior consistent here as well. WDYT?

--- Comment by github-actions[bot] at 2026-05-21T11:49:29Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: audioflamingo3, auto, glmasr, granite_speech, granite_speech_plus, musicflamingo, parakeet, qwen2_audio, vibevoice_asr, voxtral, voxtral_realtime

--- Comment by github-actions[bot] at 2026-05-21T12:02:58Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45534&sha=157f7a

--- Comment by eustlb at 2026-05-14T09:23:46Z ---
That's the main diff with how the breaking change was introduced for VLMs. To address the fact that `model.language_model` would now be failing , a property
```python
@property
def language_model(self):
    return self.model.language_model
```
was added. As explained in the PR comment, I prefer using a decorator for that. Curious about what you think here @ArthurZucker

--- Comment by ArthurZucker at 2026-05-14T09:38:24Z ---
not sure you need these with the auto features of PreTrainedModel anymore, I might be wrong tho

--- Comment by ArthurZucker at 2026-05-14T09:41:00Z ---
same

--- Comment by ArthurZucker at 2026-05-18T06:38:55Z ---
we usually put this under the PretrainedModel

--- Comment by ArthurZucker at 2026-05-18T06:53:48Z ---
should be transparent with `EmbeddingAccessMixin`

--- Comment by ArthurZucker at 2026-05-18T06:54:45Z ---
that's a bit weird! There should be a way around this no?

--- Comment by ArthurZucker at 2026-05-18T06:55:17Z ---
same comment about `EmbeddingAccessMixin`

--- Comment by ArthurZucker at 2026-05-18T06:55:33Z ---
would be nice to avoi

--- Comment by ArthurZucker at 2026-05-18T07:05:41Z ---
more fan of the property TBF! More reliable and less checks

--- Comment by ArthurZucker at 2026-05-18T07:05:54Z ---
Cool!

--- Comment by eustlb at 2026-05-18T10:08:11Z ---
missed this one, my bad... thanks for flagging

--- Comment by eustlb at 2026-05-18T10:09:11Z ---
Ok thanks! removed it

--- Comment by eustlb at 2026-05-18T12:06:50Z ---
you're right! thanks. Added it to match llava that also use it in a redundant way, so opened #46029 to fix it as well as going a bit further

--- Comment by ArthurZucker at 2026-05-19T04:33:02Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-05-19T04:33:13Z ---
```suggestion
```
if None we should not ship it

--- Comment by ArthurZucker at 2026-05-19T04:33:36Z ---
```suggestion
```
same

--- Comment by ArthurZucker at 2026-05-19T04:34:26Z ---
```suggestion
    _tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}
```

--- Comment by ArthurZucker at 2026-05-19T04:44:23Z ---
```suggestion
    _tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}
```


PULL REQUEST #45533: Fix AMD CI: rebuild torchvision with libjpeg + refresh expectations
State: closed  |  Merged: True
Author: Abdennacer-Badaoui  |  Base: main
Labels:
Created: 2026-04-20T14:39:53Z

## Summary
- Rebuild `torchvision` from source in the AMD CI image so `torchvision.io.decode_image` (used by `load_image_as_tensor`) has libjpeg/libpng support. This unblocks the wave of `decode_jpeg: torchvision not compiled with libjpeg support` failures introduced when image loading switched from PIL to `torchvision.io` in this [PR](https://github.com/huggingface/transformers/pull/45195). The build is working, see [here](https://github.com/huggingface/transformers/actions/runs/24663986002).
- Keep ROCm image ops (nms, roi_align, deform_conv) on GPU for CUDA parity by passing `FORCE_CUDA=1` and filtering hipify-emitted `*_hip.cpp` breadcrumbs out of the sources globs.
- Refresh expectations for `qwen2_5_vl` and `modernbert`. These drifted because the AMD image pins flash-attention to an older commit (`6387433…`) to keep the Docker build under the 6h GitHub Actions limit (~4h with the pin, >6h on newer heads), and that version produces slightly different numerics on the affected tests.

--- Comment by github-actions[bot] at 2026-04-20T14:41:08Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: modernbert, qwen2_5_vl

--- Comment by tarekziade at 2026-04-21T09:04:21Z ---
run-slow: modernbert, qwen2_5_vl

--- Comment by github-actions[bot] at 2026-04-21T09:05:37Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24713779543)

This comment contains `run-slow`, running the specified jobs:

models: ["models/modernbert", "models/qwen2_5_vl"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-21T09:22:40Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24713779543)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [d3ff7892](https://github.com/huggingface/transformers/commit/d3ff7892ea7829b5e3bd94bcb38b6500c5b1f2ac) | workflow commit (merge commit) |
| PR | [5a4d5390](https://github.com/Abdennacer-Badaoui/transformers/commit/5a4d539067a29891f12771f65a2bb6b71cb358fc) | branch commit (from PR) |
| main | [9dff7ca5](https://github.com/huggingface/transformers/commit/9dff7ca5c9693f4c02cdd2a9c2abc4772fcea5da) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by tarekziade at 2026-04-21T09:07:22Z ---
just curious, what did install torchvision prior to this line so we have to remove it? could we skip that initial install?


[Model] Add SLANet Model Support

PULL REQUEST #45532: [Model] Add SLANet Model Support
State: closed  |  Merged: True
Author: zhang-prog  |  Base: main
Labels: New model
Created: 2026-04-20T12:41:39Z

(no description)

--- Comment by github-actions[bot] at 2026-04-22T03:16:26Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, slanet

--- Comment by zhang-prog at 2026-04-22T03:59:56Z ---
> Re: the image caching, yes would be nice if you could another PR for that for other models 🤗

https://github.com/huggingface/transformers/pull/45562 PTAL.🤗

--- Comment by vasqu at 2026-04-22T10:55:05Z ---
run-slow: slanet

--- Comment by github-actions[bot] at 2026-04-22T10:56:24Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24774470557)

This comment contains `run-slow`, running the specified jobs:

models: ["models/slanet"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-22T11:04:46Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24774470557)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [a90c5a81](https://github.com/huggingface/transformers/commit/a90c5a8177583a9e0c063a3221500e5e3ca30894) | workflow commit (merge commit) |
| PR | [682a5b16](https://github.com/zhang-prog/transformers/commit/682a5b1640031b5b1a75f8b064c539dab4670950) | branch commit (from PR) |
| main | [f048e845](https://github.com/huggingface/transformers/commit/f048e845684894fe60440bb8506f26ffaf7b69ac) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-04-22T11:16:42Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45532). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-04-20T20:01:54Z ---
Might have missed these the last models, but would be nice if we could use httpx instead - we cleaned our code base to use httpx internally so would be nice to have it in examples as well

--- Comment by vasqu at 2026-04-20T20:02:45Z ---
Neat, already used the auto mappings :D

--- Comment by vasqu at 2026-04-20T20:05:08Z ---
```suggestion
    hidden_size: int = 256
```
should be inherited instead, they have the same values if I see it correctly

--- Comment by vasqu at 2026-04-20T20:06:35Z ---
```suggestion
    csp_num_blocks: int = 1
```
nit: naming to be consistent with the layer kwarg

--- Comment by vasqu at 2026-04-20T20:06:58Z ---
I think in the docs we can at least also have the full name behind what CSP means

--- Comment by vasqu at 2026-04-20T20:10:02Z ---
Looks like we mostly inherit from slanext so the prefix should be the same as well

--- Comment by vasqu at 2026-04-20T20:12:04Z ---
Would be nice if we could support this - probably some missing flags and adding gradient ckpt layer where suitable

--- Comment by vasqu at 2026-04-20T20:13:57Z ---
Imo we can use https://github.com/huggingface/transformers/blob/ef97a7525dc6e4665db4ba5e1ec40d79e43cc74e/src/transformers/models/pp_lcnet/modeling_pp_lcnet.py#L68

The squeeze module can be identity, we just need to correctly set it https://github.com/huggingface/transformers/blob/ef97a7525dc6e4665db4ba5e1ec40d79e43cc74e/src/transformers/models/pp_lcnet/modeling_pp_lcnet.py#L93-L95

--- Comment by vasqu at 2026-04-20T20:16:02Z ---
typing

--- Comment by vasqu at 2026-04-20T20:21:15Z ---
```suggestion
        self.conv1 = SLANetConvLayer(in_channels, mid_channels, 1, activation=activation)
        self.conv2 = SLANetConvLayer(in_channels, mid_channels, 1, activation=activation)
        self.conv3 = SLANetConvLayer(2 * mid_channels, out_channels, 1, activation=activation)

        self.bottlenecks = nn.ModuleList(
            [SLANetBottleneck(...) for _ in range(num_blocks)]
        )
```
I think we can try to imitate `DFineCSPRepLayer` a bit and reference it in the docstrings

--- Comment by vasqu at 2026-04-20T20:22:36Z ---
```suggestion
    def forward(self, hidden_states: torch.Tensor):
        residual = self.conv1(hidden_states)

        hidden_states = self.conv2(hidden_states)
        for bottleneck in self.bottlenecks:
            hidden_states = bottleneck(hidden_states)

        hidden_states = torch.cat((hidden_states, residual), dim=1)
        hidden_states = self.conv3(hidden_states)

        return hidden_states
```
renames then

--- Comment by vasqu at 2026-04-20T20:24:58Z ---
This module seems a bit useless - I think we can just move it to the parent directly tbh

--- Comment by vasqu at 2026-04-20T20:27:36Z ---
Nit: We can also just use the list comprehensions for creation at least (more modular friendly)

--- Comment by vasqu at 2026-04-20T20:27:55Z ---
This will then also become a simple module list per list comprehension

--- Comment by vasqu at 2026-04-20T20:28:16Z ---
typing

--- Comment by vasqu at 2026-04-20T20:30:35Z ---
Or will the other models rely on similar structures 🤔

--- Comment by vasqu at 2026-04-20T20:43:04Z ---
Imo, this should be `SLANetBackbone` instead, similar to `SLANeXtBackbone`

Not that it needs the same naming but from the structure it should be the same
```
class SLANetBackbone(SLANetPreTrainedModel):
    def __init__(self, config: SLANetConfig):
        super().__init__(config)
        self.vision_backbone = load_backbone(config)
        self.post_cs_pan = SLANetCSPPAN(self.backbone.num_features[2:], config)

        self.post_init()

    @can_return_tuple
    @auto_docstring
    def forward(
        self, hidden_states: torch.FloatTensor, **kwargs: Unpack[TransformersKwargs]
    ) -> tuple[torch.FloatTensor] | SLANetForTableRecognitionOutput:
        outputs = self.vision_backbone(hidden_states, **kwargs)
        hidden_states = self.post_cs_pan(outputs.feature_maps)
        return BaseModelOutputWithNoAttention(
            last_hidden_state=hidden_states,
            hidden_states=outputs.hidden_states,
        )
```
Just as an example

--- Comment by vasqu at 2026-04-20T20:44:31Z ---
```suggestion
class SLANetForTableRecognition(SLANeXtForTableRecognition):
    _keys_to_ignore_on_load_missing = ["num_batches_tracked"]

    @can_return_tuple
    @auto_docstring
    def forward(
        self, pixel_values: torch.FloatTensor, **kwargs: Unpack[TransformersKwargs]
    ) -> tuple[torch.FloatTensor] | SLANetForTableRecognitionOutput:
        outputs = self.model(pixel_values, **kwargs)
        head_outputs = self.head(outputs.last_hidden_state, **kwargs)
        # Key difference: no attentions in its vision model
        return SLANetForTableRecognitionOutput(
            last_hidden_state=head_outputs.last_hidden_state,
            hidden_states=outputs.hidden_states,
            head_hidden_states=head_outputs.hidden_states,
            head_attentions=head_outputs.attentions,
        )
```

--- Comment by vasqu at 2026-04-20T20:45:16Z ---
Then we also don't override `base_model_prefix` (i.e. `base_model_prefix = "backbone"` )and use the same namings

--- Comment by vasqu at 2026-04-20T20:47:31Z ---
lets add a small docstring for why we override

--- Comment by vasqu at 2026-04-20T20:50:53Z ---
We changed the logic a bit to cache more, can you try to refactor
1. Add to the list of images https://github.com/huggingface/transformers/blob/ef97a7525dc6e4665db4ba5e1ec40d79e43cc74e/utils/fetch_hub_objects_for_ci.py#L40
2. Then use `url_to_local_path` to get the appropriate (cached) image path https://github.com/huggingface/transformers/blob/ef97a7525dc6e4665db4ba5e1ec40d79e43cc74e/tests/models/gemma4/test_modeling_gemma4.py#L431-L433

Tbh, might be better to do for all PP models that have been added (in a separate PR)

--- Comment by zhang-prog at 2026-04-21T06:28:00Z ---
Done. If this change looks good to you, I’ll open a new PR to update the previous models as well.

--- Comment by zhang-prog at 2026-04-21T06:28:01Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:02Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:05Z ---
oh, I see

--- Comment by zhang-prog at 2026-04-21T06:28:06Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:08Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:09Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:11Z ---
done, move it to the parent

--- Comment by zhang-prog at 2026-04-21T06:28:13Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:15Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:17Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:19Z ---
cool! done

--- Comment by zhang-prog at 2026-04-21T06:28:25Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:27Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:28Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:29Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:31Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:33Z ---
Done

--- Comment by zhang-prog at 2026-04-21T06:28:34Z ---
Done

--- Comment by vasqu at 2026-04-21T12:28:01Z ---
I think we need to add the slanext image processor to the auto mappings in `image_processing_auto` (under `MISSING_IMAGE_PROCESSOR_MAPPING_NAMES`)

Seems like the auto mappings didnt pick it up

--- Comment by vasqu at 2026-04-21T12:29:44Z ---
```suggestion
        Number of blocks within the Cross Stage Partial (CSP) layer.
```

--- Comment by vasqu at 2026-04-21T12:32:11Z ---
Super nit: the order is a bit weird, would change it to have the config as 1st arg

--- Comment by vasqu at 2026-04-21T12:33:14Z ---
```suggestion
    @can_return_tuple
    @auto_docstring
    def forward(
        self, hidden_states: torch.FloatTensor, **kwargs: Unpack[TransformersKwargs]
    ) -> tuple[torch.FloatTensor] | SLANetForTableRecognitionOutput:
        outputs = self.vision_backbone(hidden_states, **kwargs)
        hidden_states = self.post_csp_pan(outputs.feature_maps)
        return BaseModelOutputWithNoAttention(
            last_hidden_state=hidden_states,
            hidden_states=outputs.hidden_states,
        )
```
I think we don't need those decorators, we rely on the backbone to collect the hidden states and it does so by itself

--- Comment by vasqu at 2026-04-21T12:34:57Z ---
I think we should not inherit here and inherit from `BaseModelOutputWithNoAttention` instead but with the same additions as `SLANeXt`; just to avoid confusion about attentions being in the output

--- Comment by zhang-prog at 2026-04-22T03:15:27Z ---
u r right, done

--- Comment by zhang-prog at 2026-04-22T03:15:29Z ---
Done

--- Comment by zhang-prog at 2026-04-22T03:15:31Z ---
Done

--- Comment by zhang-prog at 2026-04-22T03:15:32Z ---
Done


PULL REQUEST #45531: Revert "Fix: modular image processors (#45492)"
State: closed  |  Merged: True
Author: tarekziade  |  Base: main
Labels:
Created: 2026-04-20T12:21:09Z

This reverts commit cba1e406d27fdaf96a30c7f9562ccf85fd41946a.

--- Comment by github-actions[bot] at 2026-04-20T12:22:19Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: llava_onevision

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T12:31:18Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45531). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


[CB] Changes for long generation

PULL REQUEST #45530: [CB] Changes for long generation
State: closed  |  Merged: True
Author: remi-or  |  Base: main
Labels:
Created: 2026-04-20T10:33:54Z

## Summary

This PR fixes some issues related to memory to make long generation (16K+) easier.

  - Fix KV dedup for decode batches (scheduler.py): Decode-only batches don't consume the read_indices cache budget, so don't reject them on that basis. Also gate the decode fast path on max_blocks_per_request > 0 instead of unconditionally enabling it.
  - Fix memory estimation (requests.py): Use torch.cuda.mem_get_info instead of device_properties().total_memory, which ignored CUDA context/driver overhead (~0.5 GiB) and caused overcommit/OOM.
  - Raise max_memory_percent default 0.8 → 0.9 (configuration_utils.py): Now safe with the corrected memory accounting above.
  - Write-only fast path (cache.py, input_outputs.py, scheduler.py): When a batch has no past-cache reads (pure prefills), skip the index_select read-back, avoid allocating/transferring read_index, and return the input KV states directly. Also adjusts the CUDA-graph key to depend on the block-table path rather than max_kv_read > 0.
  - Two-peak memory model (cache.py): Replace the single peak_activation_per_token with two activation peaks — LM head (hidden + logits, N-independent) and attention (hidden + Q + new K/V + cache K/V reads, grows with N). Solve the memory polynomial for each peak independently and take the most restrictive (num_blocks, max_batch_tokens). Bumps _upper_bound_max_batch_tokens 256 → 1024.

# Performances

Pretty good, lot of workloads benefit from 80% to 90% raise in cache space

| Arguments                                                       | Main (tok/s) | Current (tok/s) | Diff (%) |
|-----------------------------------------------------------------|--------------|-----------------|----------|
| --samples 10                                                    | 869.0        | 890.37          | +2.5%    |
| --samples 20 --num-blocks 20                                    | 517.95       | 520.16          | +0.4%    |
| --samples 50                                                    | 3629.88      | 3638.6          | +0.2%    |
| --samples 100                                                   | 5375.41      | 5522.83         | +2.7%    |
| --samples 100 --attn flash_attention_2                          | 3666.82      | 3743.47         | +2.1%    |
| --samples 100 --attn sdpa                                       | 1030.21      | 1053.57         | +2.3%    |
| --samples 500 --no-use-async                                    | 6621.78      | 8020.64         | +21.1%   |
| --samples 500 --use-async                                       | 7963.66      | 9332.71         | +17.2%   |
| --samples 32 --max-new-tokens 2048 --use-async                  | 2033.87      | 2064.29         | +1.5%    |
| --samples 32 --max-new-tokens 2048 --use-async --block-table 32 | 2716.64      | 2734.81         | +0.7%    |
| --samples 500 --add-prefix --compile                            | 7649.48      | 8882.62         | +16.1%   |
| --samples 50 --num-return-sequences 8 --do-sample               | 869.94       | 980.13          | +12.7%   |
| --samples 100 --num-return-sequences 4 --do-sample              | 1708.88      | 1925.25         | +12.7%   |

# Tests

- [x] `make style` and `make_typing` pass
- [x] RUN_SLOW=1 pytest tests/generation/test_continuous_batching.py
- [x] RUN_SLOW=1 pytest tests/cli/test_serve.py
- [x] RUN_SLOW=1 pytest tests/generation/test_paged_attention.py

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T10:44:04Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45530). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ArthurZucker at 2026-04-23T07:55:26Z ---
nice!