PULL REQUEST #45528: qa: re-run modular converter when the script itself is modified
State: closed  |  Merged: True
Author: tarekziade  |  Base: main
Labels:
Created: 2026-04-20T10:08:06Z

# What does this PR do?

- When changing the checker itself, we want to ignore the `guaranteed_no_diff` shortcut since changing the checker might impact models not in the PR

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T10:18:57Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45528). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-20T12:04:01Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: conditional_detr, deepseek_vl, deepseek_vl_hybrid, deformable_detr, ernie4_5_vl_moe, grounding_dino, mask2former, paddleocr_vl, rt_detr, segformer, smolvlm, video_llama_3

--- Comment by zucchini-nlp at 2026-04-20T10:13:32Z ---
actually very much looks like caused by https://github.com/huggingface/transformers/pull/45492, could you verify it is really unrelated?

--- Comment by tarekziade at 2026-04-20T12:05:59Z ---
What happened is that the PR change the modular conversion process and that was impacting other models as well. They were not checked in the PR because it's ignored for models not changed in the PR.

--- Comment by zucchini-nlp at 2026-04-20T12:25:21Z ---
great!

--- Comment by zucchini-nlp at 2026-04-20T12:32:59Z ---
hmm not sure if we need unittests tho, I don't think anyone will delete `modular_model_converter.py` from the checker

--- Comment by tarekziade at 2026-04-20T13:37:42Z ---
this is not deleting the file, it's just simulating a change in `modular_model_converter.py` in the diff to make sure that we detect it, so we run the modular converter on all model like in main and not partially skip. It's what happened in #45492

--- Comment by zucchini-nlp at 2026-04-20T13:57:40Z ---
yeah, I got it, I meant delete the you added for "diff on modular_model_converter.py" with a commont why it is there


Reapply modular to examples

PULL REQUEST #45527: Reapply modular to examples
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-04-20T10:05:43Z

# What does this PR do?

As per the title.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T10:23:09Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45527). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


xpu output align with cuda in test case

PULL REQUEST #45526: xpu output align with cuda in test case
State: closed  |  Merged: True
Author: sywangyi  |  Base: main
Labels:
Created: 2026-04-20T08:00:28Z

InternVLQwen2IntegrationTest::test_qwen2_medium_model_integration_video

- Intel XPU: @IlyasMoutawwakil

--- Comment by github-actions[bot] at 2026-04-20T08:01:33Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: internvl

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T08:59:24Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45526). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45525: Fix CSM `TextToAudioPipeline` missing `<bos>` token
State: closed  |  Merged: True
Author: jiqing-feng  |  Base: main
Labels:
Created: 2026-04-20T06:05:57Z

## What does this PR do?

`CsmProcessor` defaults `add_special_tokens=False` (designed for `apply_chat_template`, which includes `<bos>` in the Jinja template). When the pipeline calls `preprocessor(text)` directly for raw text input, `<bos>` (128000) and `<eos>` (128001) are missing from the tokenized sequence. Without these tokens the model receives malformed input it was never trained on, making generation unstable — certain seed/sampling parameter combinations cause the model to emit all-zero codebook frames, which are treated as EOS (`codebook_eos_token_id=0`), resulting in an empty audio tensor that crashes Mimi's Conv1d decoder.

Fix: set `add_special_tokens=True` for CSM in pipeline `preprocess`.

## Reproduction

```python
from transformers import pipeline, set_seed

pipe = pipeline("text-to-speech", model="sesame/csm-1b")
set_seed(777)
output = pipe("Hello, my dog is cooler than you!", forward_params={"do_sample": True, "temperature": 0.7, "top_k": 50, "top_p": 0.95})
# RuntimeError: Calculated padded input size per channel: (0). Kernel size: (1).
# Kernel size can't be greater than actual input size
```

error:
```
......
  File "/home/jiqing/transformers/src/transformers/models/csm/generation_csm.py", line 478, in generate
    codec_decode_output = self.codec_model.decode(audio_codes_batch.transpose(0, 1).unsqueeze(0))                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                      File "/home/jiqing/transformers/src/transformers/models/mimi/modeling_mimi.py", line 1666, in decode
    audio_values, decoder_past_key_values = self._decode_frame(                                                                                ^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/models/mimi/modeling_mimi.py", line 1619, in _decode_frame
    embeddings = self.quantizer.decode(codes)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/models/mimi/modeling_mimi.py", line 1344, in decode
    quantized_out = self.semantic_residual_vector_quantizer.decode(codes[:, : self.num_semantic_quantizers])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/models/mimi/modeling_mimi.py", line 1292, in decode
    quantized_out = self.output_proj(quantized_out)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 385, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 380, in _conv_forward
    return F.conv1d(
           ^^^^^^^^^
RuntimeError: Calculated padded input size per channel: (0). Kernel size: (1). Kernel size can't be greater than actual input size
```

Hi @Rocketknight1 . Would you please review this PR? Thanks!

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T15:40:05Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45525). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45524: utils: handle flash_attn missing from importlib packages_distributions without crashing
State: open  |  Merged: False
Author: SAY-5  |  Base: main
Labels:
Created: 2026-04-20T05:44:53Z

Fixes #45520.

`is_flash_attn_2_available`, `is_flash_attn_3_available`, `is_flash_attn_4_available`, and `is_flash_attn_greater_or_equal` all do two checks:

```python
is_available, _ = _is_package_available("flash_attn", return_version=True)
is_available = is_available and "flash-attn" in [
    pkg.replace("_", "-") for pkg in PACKAGE_DISTRIBUTION_MAPPING["flash_attn"]
]
```

Step 1 uses `importlib.util.find_spec`, which returns a spec if any `flash_attn` import is findable, an editable install, a namespace package, a bundled shim, or a stub module sitting under another project. Step 2 then assumes that every findable import name also has an entry in `importlib.metadata.packages_distributions()`.

That assumption does not hold in practice. On Python 3.13 with ComfyUI setups (as reported in #45520), and more generally in any environment where the `flash_attn` import is resolvable via a non-pip source, `packages_distributions()` has no `"flash_attn"` key. Because the list comprehension is evaluated before the `in` operator, short-circuit evaluation of the outer `and` does not protect us, the `KeyError` fires during `transformers` import and takes down the whole process before any model is loaded.

Swap the four raising subscripts for `.get(name, [])`. If the name is missing from the distribution map we simply conclude that the requested flash-attention flavour is not properly installed, which is the answer `is_flash_attn_*_available()` would have returned anyway, instead of raising. The inner helper `_is_package_available` already wraps the same subscript in a `try/except`, so we are only making the outer call sites match that contract.

--- Comment by Rocketknight1 at 2026-04-21T12:16:47Z ---
cc @vasqu


PULL REQUEST #45523: Fix Seq2SeqLM ExecuTorch export: add encoder_attention_mask to decoder and use static encoder shapes
State: open  |  Merged: False
Author: duyhv-qualgo  |  Base: main
Labels:
Created: 2026-04-20T05:20:19Z

## Problem

Two related bugs in `src/transformers/integrations/executorch.py` that break seq2seq (T5) ExecuTorch export:

### Bug 1 — `encoder_attention_mask` not forwarded to decoder

`Seq2SeqLMDecoderExportableModuleWithStaticCache.forward` calls `self.decoder(...)` without passing `encoder_attention_mask`. For T5, the cross-attention module computes a relative position bias scaled by `key_length`. When no mask is provided, `key_length` equals the full padded sequence length (e.g. 512) instead of the real encoder output length. This causes a ~20× logit scale error and completely wrong greedy-decoding outputs.

**Verified**: exporting and running T5 without this fix produces semantically wrong translations. With the fix, ExecuTorch output matches HuggingFace `model.generate()` exactly (5/5 test cases, exact string match).

### Bug 2 — Dynamic encoder dim conflicts with static KV cache

`Seq2SeqLMExportableModule._export_decoder` marks `encoder_hidden_states` dim-1 as a dynamic symbol (`encoder_hidden_seq_length`). With transformers 5.0, T5's cross-attention causal mask slices against the static KV-cache size:

```python
causal_mask = mask[:, :, :, : key_states.shape[-2]]  # static int
position_bias = position_bias + causal_mask           # symbolic dim → conflict
```

This raises `RuntimeError: tensor a (1024) must match tensor b (s96)` during `torch.export`.

**Fix**: remove the dynamic encoder dim; callers pad encoder inputs to `max_cache_len` (the static export shape), which is the correct assumption for static-shape ExecuTorch deployment.

## Changes

- `Seq2SeqLMDecoderExportableModuleWithStaticCache.forward`: add `encoder_attention_mask: Tensor | None = None` parameter and pass it to `self.decoder(...)`.
- `Seq2SeqLMExportableModule._export_decoder`: accept and pass `encoder_attention_mask`; remove dynamic encoder sequence length shape (use `dynamic_shapes=None`).
- `Seq2SeqLMExportableModule.export()`: pass `encoder_attention_mask` (default: all-ones of shape `[batch, max_cache_len]`).
- `Seq2SeqLMExportableModule.generate()`: build `encoder_attention_mask` from `prompt_token_ids != 0` and pass it to each decoder step.

## Test plan

- [x] Verified on a custom Helsinki-NLP–style T5 seq2seq checkpoint: ExecuTorch fp32 export produces exact token-level match with `model.generate()` on 5 translation test cases after this fix
- [x] Existing `tests/models/test_modeling_t5.py` `@slow` tests (if CI can run them)
- [x] `tests/integrations/test_executorch.py` — no seq2seq-specific tests exist yet; a follow-up can add one

## Who can review?
cc @vasqu @Cyrilvallez

--- Comment by vasqu at 2026-04-23T15:58:11Z ---
@Cyrilvallez could you check it out

--- Comment by Cyrilvallez at 2026-05-12T07:44:24Z ---
Hey! Thanks for the PR @duyhv-qualgo! Could you please provide a min repro of the issue?


[Trainer] Add ddp_static_graph option

PULL REQUEST #45519: [Trainer] Add ddp_static_graph option
State: closed  |  Merged: True
Author: KeitaW  |  Base: main
Labels:
Created: 2026-04-20T02:08:37Z

# What does this PR do?

Exposes PyTorch DDP's `static_graph` flag via a new `ddp_static_graph: Optional[bool]` field on `TrainingArguments`, forwarded through `Trainer._build_accelerator_args` into Accelerate's [`DistributedDataParallelKwargs`](https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/dataclasses.py) (which already supports it; only the Transformers-side plumbing was missing).

This completes the set of DDP flags already partially exposed on `TrainingArguments` (`ddp_find_unused_parameters`, `ddp_bucket_cap_mb`, `ddp_broadcast_buffers`). Today a user can configure nearly everything about DDP except `static_graph`, and today's only workarounds are monkey-patching `DistributedDataParallel.__init__` via a `sitecustomize.py` shim or subclassing `Trainer` to override `_wrap_model` — neither portable.

Fixes #45518

## Why

Per [PyTorch's DDP docs](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html), `static_graph=True` relaxes several DDP reducer constraints for users who can guarantee a stable graph across iterations: *"Reentrant backwards"*, *"Activation checkpointing when model has unused parameters"*, *"There are model parameters that are outside of forward function"*, and *"Potentially improve performance when there are unused parameters."*

A common HF Trainer scenario where this matters: a model with trainable parameters that don't contribute to loss on every iteration (e.g. frozen submodules, or multi-head models where only one head is trained). Under DDP with `ddp_find_unused_parameters=False` (the Trainer default), such a model fails at iter 1 with:

> `RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss.`

The common workaround is `ddp_find_unused_parameters=True`, but that forces DDP to traverse the autograd graph on every iteration to find unused params — a measurable per-step cost. `static_graph=True` is the performance-optimal alternative (PyTorch's own note: *"potentially improve performance when there are unused parameters"*): DDP records the participating-parameter set on iter 1 and assumes it's stable thereafter.

The earlier blocker that once made `static_graph=True` unsafe with HF models (`ModelOutput` subclasses not registered as pytree nodes) was fixed in #25358 (closed #25357, merged 2023-08) and is still live in `src/transformers/utils/generic.py` — where the repo itself [documents `static_graph=True` safety with `ModelOutput`](https://github.com/huggingface/transformers/blob/main/src/transformers/utils/generic.py#L383-L384) in a docstring.

## Changes

- **`src/transformers/training_args.py`** — new `ddp_static_graph: bool | None = field(default=None, …)`, mirroring the `ddp_broadcast_buffers` pattern. Docstring entry explains the supported-use-cases surface and caveats (DDP-only; incompatible with re-entrant activation checkpointing; requires stable graph).
- **`src/transformers/trainer.py`** (`_build_accelerator_args`) — one new conditional following the existing `if self.args.ddp_* is not None:` pattern for `bucket_cap_mb` and `broadcast_buffers`. When the flag is `None` (default), the kwarg is never added to `ddp_kwargs`, so `DistributedDataParallelKwargs`' own default (`False`) applies. Strictly additive; no existing behavior changes.
- **`tests/trainer/test_trainer.py`** — new `TrainerDDPKwargsTest` class with three tests:
  - `ddp_static_graph=True` → handler has `static_graph=True` (positive).
  - `ddp_static_graph=False` → handler has `static_graph=False` (positive).
  - `ddp_static_graph=None` (default) → handler preserves Accelerate's default `False` (regression guard — the conditional in `_build_accelerator_args` must NOT leak the kwarg when unset).

All three pass locally.

## Interaction caveats (documented in help text)

- **DDP-only.** The field has no effect under FSDP or DeepSpeed (the conditional only fires on the DDP path, matching the other `ddp_*` fields).
- **Gradient checkpointing.** Using `static_graph=True` alongside re-entrant activation checkpointing (`use_reentrant=True`) is unsafe per PyTorch; the docstring warns. Non-reentrant checkpointing (`use_reentrant=False`) is fine.
- **Requires stable graph.** Modules with data-dependent control flow that changes which parameters are touched per iteration are incompatible with `static_graph=True` by PyTorch's own contract.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. — #45518
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@SunMarc — Trainer / Accelerate integration.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T12:33:33Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45519). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by SunMarc at 2026-04-20T15:34:01Z ---
keep it short like the other args, users interested will check ddp docs for that

--- Comment by SunMarc at 2026-04-20T15:34:17Z ---
```suggestion
                "`DistributedDataParallel`. "
```

--- Comment by KeitaW at 2026-04-21T12:16:36Z ---
Done — trimmed to two lines matching the `ddp_broadcast_buffers` / `ddp_bucket_cap_mb` style. Thanks!

--- Comment by KeitaW at 2026-04-21T12:17:43Z ---
Applied — dropped the FSDP/DeepSpeed sentence from the `help` string. Thanks!


PULL REQUEST #45516: T5Gemma2: fix `prepare_decoder_input_ids_from_labels`
State: closed  |  Merged: True
Author: Tokarak  |  Base: main
Labels:
Created: 2026-04-19T17:22:36Z

# What does this PR do?

Rename arg from `input_ids` to `labels` (using IDE rename), to be
consistent with other models' `prepare_decoder_input_ids_from_labels`.

Fix broken interaction when DataCollatorForSeq2Seq passes argument
as a hard-coded "labels" kwarg.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Tagging @CyrilVallez.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models:  @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by Tokarak at 2026-04-19T17:39:27Z ---
I just verified that this patch applied to my local transformers install does in fact fix the bug with DataCollatorForSeq2Seq

--- Comment by Tokarak at 2026-04-19T21:04:21Z ---
The issue mentioned above contains a script that can reproduce the bug this PR is fixing

--- Comment by Rocketknight1 at 2026-04-20T14:57:44Z ---
@Tokarak fix looks good, can you run `make fix-repo` to propagate the change from the `modular` file?

--- Comment by Tokarak at 2026-04-20T19:52:02Z ---
done

--- Comment by github-actions[bot] at 2026-04-20T19:53:00Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: t5gemma2

--- Comment by HuggingFaceDocBuilderDev at 2026-04-21T12:42:39Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45516). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45515: Fix CUDA availability check in get_device_properties()
State: closed  |  Merged: False
Author: Jah-yee  |  Base: main
Labels: Code agent slop
Created: 2026-04-19T11:59:23Z

Good day

## Summary

This PR fixes a bug in `src/transformers/testing_utils.py` where `get_device_properties()` calls `torch.cuda.get_device_capability()` without first checking if a GPU is actually available.

## Bug Description

When CUDA is installed on a system (e.g., `torch.version.cuda` is not `None`) but no GPU is present, calling `torch.cuda.get_device_capability()` raises a `RuntimeError`.

## Fix

Changed the condition from:

```python
if IS_CUDA_SYSTEM or IS_ROCM_SYSTEM:
    import torch
    major, minor = torch.cuda.get_device_capability()
```

to:

```python
if (IS_CUDA_SYSTEM or IS_ROCM_SYSTEM) and torch.cuda.is_available():
    major, minor = torch.cuda.get_device_capability()
```

This ensures the GPU capability call only happens when a GPU is actually present, and also removes the redundant `import torch` statement that was inside the conditional block (the import is already at the top of the file).

## Reference

Fixes #45341

---

Thank you for your attention. If there are any issues or suggestions, please leave a comment and I will address them promptly.

Warmly,
RoomWithOutRoof

--- Comment by github-actions[bot] at 2026-04-19T12:16:25Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45515&sha=b5c6f9


PULL REQUEST #45514: Fix GraniteMoeHybrid _update_mamba_mask crash on attention-only models
State: closed  |  Merged: True
Author: tianhaocui  |  Base: main
Labels:
Created: 2026-04-19T10:27:36Z

Fixes #45507

## Summary

`GraniteMoeHybridModel._update_mamba_mask` calls `past_key_values.has_previous_state()` without checking whether the model actually has mamba layers. When all layers are attention-only (no mamba layers in `config.layers_block_type`), `has_previous_state()` fails to find a `LinearAttentionCacheLayerMixin` layer and raises `ValueError`.

## Fix

Check `config.layers_block_type` for mamba layers before calling `has_previous_state()`. If no mamba layers exist, return the attention mask as-is since the mamba mask optimization is irrelevant.

Applied to both `modeling_granitemoehybrid.py` and `modular_granitemoehybrid.py`.

--- Comment by Rocketknight1 at 2026-04-21T12:16:38Z ---
cc @vasqu maybe since you volunteered!

--- Comment by tianhaocui at 2026-04-23T16:04:09Z ---
Thanks for the suggestion @vasqu — you're right that the guard belongs at the call site rather than inside `_update_mamba_mask`.

Updated in ec7cd01: the check is now in `forward()`, skipping `_update_mamba_mask` entirely when no mamba layers exist. I kept the two-variable structure (`causal_mask` + `mamba_mask`) instead of a dict since there are only two mask types and it stays consistent with the existing loop at L253.

`_update_mamba_mask` is reverted to its original form — it no longer needs to know about layer types.

--- Comment by vasqu at 2026-04-23T16:06:06Z ---
Let's add a fast test as well please; should be easy by forcing the layer types on construction of the dummy model

--- Comment by tianhaocui at 2026-04-23T16:28:50Z ---
Added a fast test in 89ef90c — `test_attention_only_forward` constructs a model with `layers_block_type` set to all `"attention"` (no mamba layers) and runs a forward pass to verify it doesn't crash. This covers the regression from #45507.

--- Comment by github-actions[bot] at 2026-04-27T12:11:41Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: granitemoehybrid

--- Comment by vasqu at 2026-04-27T12:12:27Z ---
run-slow: granitemoehybrid

--- Comment by github-actions[bot] at 2026-04-27T12:13:44Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24994286805)

This comment contains `run-slow`, running the specified jobs:

models: ["models/granitemoehybrid"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-27T12:25:44Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24994286805)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [1000bf90](https://github.com/huggingface/transformers/commit/1000bf902246c7cbdeead6e910d6ba4d77c06195) | workflow commit (merge commit) |
| PR | [9d046317](https://github.com/tianhaocui/transformers/commit/9d046317f5f8f64abaf275856685f00963883729) | branch commit (from PR) |
| main | [bbb51c83](https://github.com/huggingface/transformers/commit/bbb51c83c151c3285d7f0e63ea6a68165c61cc58) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T12:38:47Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45514). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-04-23T15:50:26Z ---
Imo, this is the wrong point to intervene; we should follow the layer types logic of hybrid attentions

I.e. https://github.com/huggingface/transformers/blob/57f9936a2619d2f2d4af89bde34d5eb611c2b728/src/transformers/models/granitemoehybrid/modular_granitemoehybrid.py#L237-L243
needs to change to something like
```python
        causal_masks = {}
        for layer_type in set(self.config.layer_types):
            if "mamba" in layer_type:
                causal_mask[layer_type] = self._update_mamba_mask(attention_mask, past_key_values)
            else:
                causal_mask[layer_type] = create_causal_mask(
                    config=self.config,
                    inputs_embeds=inputs_embeds,
                    attention_mask=attention_mask,
                    past_key_values=past_key_values,
                )
        ...
        for i, decoder_layer in enumerate(self.layers):
            hidden_states = decoder_layer(
                hidden_states,
                attention_mask=causal_masks[self.config.layer_types[i]],
                past_key_values=past_key_values,
                use_cache=use_cache,
                position_embeddings=position_embeddings,
                **kwargs,
            )
```


PULL REQUEST #45513: [Qwen3.5] Fix GDN linear attention multi-token cached forward
State: closed  |  Merged: True
Author: kashif  |  Base: main
Labels:
Created: 2026-04-19T09:40:59Z

# What does this PR do?

The gated-delta-net (GDN) forward only used the cached recurrent state when `seq_len == 1`. For any multi-token forward with a populated cache (e.g. chunked prefill continuation or speculative-decoding verification), it fell through to `chunk_gated_delta_rule(initial_state=None)`, silently restarting the linear layers from zero and ignoring the prefill state.

This breaks the causal-LM invariant that the logits at position `i` must not depend on whether later tokens are batched into the same call — position 0 of a 16-token verify forward ended up differing from the corresponding single-token cached decode, collapsing to high-frequency context tokens and destroying speculative-decoding correctness.

Add a `use_cached_chunk` path that, when `has_previous_state` is true and `seq_len > 1`:
- reads the cached `conv_state` / `recurrent_state`,
- prepends the conv context onto the chunk input so the causal conv sees the correct left-context,
- drops the prepended context from the output,
- passes the cached `recurrent_state` as `initial_state` to `chunk_gated_delta_rule`.

The same fix propagates to `qwen3_5_moe` via the modular system and is applied to `qwen3_next` and `olmo_hybrid`.

Add a unit test that compares the first-position output of a multi-token cached forward against the single-token cached forward on the same token and cache. Without this fix the mismatch is 100%.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-04-19T09:50:33Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45513). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Cyrilvallez at 2026-04-23T05:42:48Z ---
cc @vasqu here to double-check as well: I had raised similar concerns when I was refactoring the mamba caches, but I'm less familiar than you with all the mamba kernels and how the updates should exactly look like. But some mambas use the `seq_len == 1` part and other don't, and as I said before idk if it was really  intended or not.

Fix looks legit though, but would appreciate second opiniuon!

--- Comment by kashif at 2026-04-23T06:39:23Z ---
Thanks for the thorough check! The `seq_len == 1` pattern isn't intentional — I believe it was copied forward from mamba → qwen3_next → qwen3_5 without anyone  exercising the "populated cache + seq_len > 1" path (plain `.generate()` always has `seq_len == 1` in decode, and training doesn't use the cache). I grepped  transformers and the same `has_previous_state(...) and seq_len == 1` guard appears in 9 mamba-family files: `bamba`, `falcon_h1`, `granitemoehybrid`, `jamba`,  `olmo_hybrid`, `qwen3_next`, `qwen3_5`, `qwen3_5_moe`, `zamba`. Every one of them drops the cached recurrent state in the chunk path (passes  `initial_state=None`) and zero-pads the conv state instead of prepending the cached context.

The canonical reference is `flash-linear-attention`. I grepped `fla/layers/` and every linear-attention implementation (`gated_deltanet`, `delta_net`, `gla`,  `simple_gla`, `hgrn`, `hgrn2`, `rwkv6`, `rwkv7`, `lightnet`, `gated_deltaproduct` — 10 I checked) follows the same canonical pattern:

  ```python
  recurrent_state = last_state['recurrent_state'] if last_state is not None else None
  mode = 'fused_recurrent' if q_len <= 64 else self.mode   # perf heuristic, not correctness
  o, recurrent_state = chunk_or_fused_recurrent_kernel(..., initial_state=recurrent_state, ...)
```
  No seq_len gating on correctness. The `q_len <= 64` cutoff in FLA is a kernel-selection heuristic (fused-recurrent is faster for short seqs), not a correctness   gate — both kernels accept and propagate `initial_state`.

  This fix brings qwen3_5 in line with FLA. I've constrained the change to qwen3_5 + qwen3_5_moe (via modular) so it's reviewable; happy to open follow-up PRs  for the other mamba-family files if you'd like them audited together.

  The new unit test (`test_linear_attention_multi_token_cached_forward_matches_single_token`) is the minimal invariant the old code violated — a causal LM's logits   at position `i` must not depend on whether later tokens are batched into the same call, across any combination of `seq_len` and cache state. 100% element mismatch  on main, passes after the fix.

--- Comment by kashif at 2026-04-23T12:58:29Z ---
thanks @vasqu let me simplify and update

--- Comment by kashif at 2026-04-23T13:33:42Z ---
should i do the qwen3_next and olmo_hybrid fixes in a followup PR or this one @vasqu ?

--- Comment by vasqu at 2026-04-23T15:13:09Z ---
@kashif would be nice if we could do those models here as well - they are all based on GDN

--- Comment by kashif at 2026-04-23T15:22:10Z ---
yup! fixing all GDN ones

--- Comment by kashif at 2026-04-27T09:09:53Z ---
@vasqu if you can have another look whenever you are free

--- Comment by github-actions[bot] at 2026-04-27T13:06:05Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: olmo_hybrid, qwen3_5, qwen3_5_moe, qwen3_next

--- Comment by vasqu at 2026-04-27T13:11:25Z ---
run-slow: olmo_hybrid, qwen3_5, qwen3_5_moe, qwen3_next

--- Comment by github-actions[bot] at 2026-04-27T13:12:36Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24997034619)

This comment contains `run-slow`, running the specified jobs:

models: ["models/olmo_hybrid", "models/qwen3_5", "models/qwen3_5_moe", "models/qwen3_next"]
quantizations: []

--- Comment by vasqu at 2026-04-27T13:15:07Z ---
Just sanity checking with run-slow, but should be good will merge afterwards

--- Comment by github-actions[bot] at 2026-04-27T13:26:29Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24997034619)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [37b7ffb3](https://github.com/huggingface/transformers/commit/37b7ffb31e69a7dad52d765d9855740681f81af4) | workflow commit (merge commit) |
| PR | [40f471b9](https://github.com/kashif/transformers/commit/40f471b930b1dd40a78a59a855231663b351f951) | branch commit (from PR) |
| main | [e651c68e](https://github.com/huggingface/transformers/commit/e651c68e69f01745433b2326684cbc50f57c301e) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by vasqu at 2026-04-27T13:31:46Z ---
@kashif thanks for the PR :hugs: merging now!

--- Comment by vasqu at 2026-04-23T12:43:09Z ---
Why not remove the `seq_len > 1` then? I agree that this shouldn't be a restriction as it will be used as initial state and the forward essentially stays the same. Same/Similar for qwen3 next then

We have something similar to Mamba2, unsure about Mamba1. Would be a nice to have there as well but not directly needed in this PR. Reminds me of these lines that were removed https://github.com/huggingface/transformers/blob/83a6c5b577cafa607d59e78af4de86592b9903ee/src/transformers/models/mamba2/modeling_mamba2.py#L642-L645

--- Comment by vasqu at 2026-04-23T12:50:22Z ---
Ok I see now, we have to accomodate for the seq_len > 1 case. I'd still keep the terms `precomputed_state` and something along `single_token vs chunk_tokens` to have clearer boundaries

--- Comment by vasqu at 2026-04-23T15:12:30Z ---
```suggestion
```
Ah sorry I meant it a bit differently, imo both are precomputed states that are differing in their amount of tokens so I'd like to split the logic explicitly with focusing on seq_len where needed. So
1. Only use precomputed states (and remove the seq len condition)
2. Where needed check for the initial seq len to determine if we have a single decode step or chunks

--- Comment by vasqu at 2026-04-27T12:41:49Z ---
I think we can keep the comment and adjust `prefill mode/multi-token decode`

--- Comment by vasqu at 2026-04-27T12:43:02Z ---
Let's add this comment to the first if block related to this - similar to olmo hybrid --> prepend context and drop it afterwards


PULL REQUEST #45512: [OutputRecorder] re.search on layer_name
State: open  |  Merged: False
Author: eustlb  |  Base: main
Labels:
Created: 2026-04-19T08:38:10Z

# What does this PR do?

As discussed in #44408, this adds the possibility to do regex matching for `layer_name` in the `OutputRecorder`.

```python
OutputRecorder(GraniteSpeechConformerBlock, layer_name=r"layers\.(6|12|18)$")
```

**Backward compatibillity:**

`re.search` has the same behavior as `substring in string` except for patterns like `layer_name=".attention"`  with the dot now matching `Xattention`. None of the layer_name in the lib trigger this edge case though, but I still updated them all to foster good practices

--- Comment by HuggingFaceDocBuilderDev at 2026-04-19T08:47:26Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45512). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-19T08:55:10Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blip, blip_2, decision_transformer, gpt2, instructblip, instructblipvideo, minimax, qwen3_omni_moe, qwen3_vl_moe, switch_transformers, vjepa2

--- Comment by zvik at 2026-04-19T10:17:05Z ---
Maybe it will be better to add another option for regex instead of breaking other models:
See: https://github.com/huggingface/transformers/compare/main...zvik:transformers:output-recorder-regex


PULL REQUEST #45511: Fix NaN in Gemma3/EmbeddingGemma when batching mixed-length sequences…
State: closed  |  Merged: False
Author: Kabir08  |  Base: main
Labels:
Created: 2026-04-19T08:27:15Z

This fixes the NaN issue when batching mixed-length sequences with sliding window attention (e.g., with `EmbeddingGemma` / Gemma3 models).

The patch ensures that when a query position's entire sliding window falls within the padding region (all keys masked with `-inf`), the attention bias/mask is adjusted to allow all keys (bias=0). This produces a valid uniform softmax distribution instead of `softmax([-inf, ...]) = NaN`.

This matches the intended behavior for padding positions, which are always excluded from downstream pooling/embedding computation and thus do not affect the final output. The change prevents silent numerical instability on certain GPU SDPA backends and in eager-mode attention, while keeping CPU behavior and single-sequence (`batch_size=1`) results unchanged.

Fixes #45491.

### What does this PR do?

When a short sequence is padded to match a much longer one in a batch (common in batched inference with variable-length inputs), some query positions in the short sequence may have their entire sliding window (e.g., size 512 for Gemma3 local attention layers) fall completely within the padding tokens of the longer sequences.

This results in an attention score row of all `-inf`, causing `softmax` to produce `NaN` (0/0) on affected GPU kernels and in some eager implementations. The NaNs then propagate to the final embeddings (e.g., all-NaN vectors for the shortest items in a batch when using `SentenceTransformer.encode_query()` or similar).

The fix detects these all-masked rows in the sliding window mask construction and sets the bias to 0 for those positions, yielding a uniform attention distribution. Since these positions correspond exclusively to padding tokens, they are masked out or ignored in subsequent mean-pooling / embedding extraction steps, so the output remains correct and identical to per-example encoding.

This resolves the reported bug for `google/EmbeddingGemma-300M` (and other Gemma3 variants using sliding window attention) on GPU with mixed-length batches, without changing behavior for valid (non-padding) positions or non-sliding-window models.

### Related issues / context
- Root cause: Sliding window attention + dynamic padding in batched processing → all-padding windows → numerical instability in softmax.
- Affects: Models with `sliding_window` in their config (Gemma3 local layers, window size typically 512 or similar).
- Previously worked around by forcing `batch_size=1` or re-encoding NaN results individually.

No new dependencies. No breaking changes.

## Code Agent Policy
The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by code agents. ...

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
  → Discussed in [#45491](https://github.com/huggingface/transformers/issues/45491)
- [ ] Did you make sure to update the documentation with your changes? ...
- [ ] Did you write any new necessary tests?

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

**Recommended tags (attention + Gemma-related maintainers — keep under 3 where possible):**
- @ArthurZucker (text models, attention, Gemma expertise)
- @Cyrilvallez (text models, model loading, attention)
- @vasqu (attention)

--- Comment by vasqu at 2026-04-20T13:17:43Z ---
See my comment here: https://github.com/huggingface/transformers/issues/45491#issuecomment-4281075391

This is likely a padding side issue, not really a transformers issue

--- Comment by Kabir08 at 2026-04-20T15:18:14Z ---
Understood, i'm closing the pr


PULL REQUEST #45510: cache_utils: fix QuantizedLayer to correctly propagate reorder_cache, crop, and batch ops to quantized buffers
State: closed  |  Merged: False
Author: GitGlimpse895  |  Base: main
Labels:
Created: 2026-04-19T07:34:56Z

# What does this PR do?

`QuantizedLayer` maintains two separate storage regions: a full-precision
residual buffer (`self.keys` / `self.values`) and a quantized buffer
(`self._quantized_keys` / `self._quantized_values`). However, the four
mutation methods inherited from `DynamicLayer` — `reorder_cache`,
`crop`, `batch_repeat_interleave`, and `batch_select_indices` — only
operated on the residual buffer, silently leaving the quantized buffer
untouched.

**Concrete failure modes:**
- **Beam search** (`reorder_cache`): the quantized buffer stays in
  original beam order while the residual reorders, causing crossed
  attention across beams with no error raised.
- **Constrained generation rollback** (`crop`): `cumulative_length`
  diverges from the actual stored state, corrupting subsequent
  `get_seq_length` calls.
- **Group beam search / contrastive decoding** (`batch_select_indices`,
  `batch_repeat_interleave`): batch dimension of the quantized buffer
  is never updated, producing mismatched batch sizes between the two
  storage regions.

This PR overrides all four methods in `QuantizedLayer`. Since
`_quantized_keys`/`_quantized_values` are opaque backend objects for
both `QuantoQuantizedLayer` and `HQQQuantizedLayer`, the fix uses a
dequantize → operate → re-quantize pattern, which is backend-agnostic
and does not compound quantization error meaningfully beyond what is
already introduced at the original quantization step.

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@gante @SunMarc

- [x] I confirm that this is not a pure code agent PR.

--- Comment by GitGlimpse895 at 2026-04-21T16:07:53Z ---
Hi @zucchini-nlp, thanks for the feedback!

You’re right—this was primarily an improvement PR and isn't a strict requirement for my current workflow. Since it falls into the "nice to have" category and I definitely understand the goal of keeping maintenance costs down, I’ll go ahead and close this one out.

Thanks again for your time and the review!

--- Comment by github-actions[bot] at 2026-04-21T16:21:58Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45510&sha=3a04d2

--- Comment by zucchini-nlp at 2026-04-21T12:55:52Z ---
tbh haven't seen any requets to add beam search support with quantized cache prev, do you have a need for that or is this an improvement PR?

--- Comment by zucchini-nlp at 2026-04-21T12:56:45Z ---
If you need this feature, I don't mind adding it. But we need a test in `tests/generation/test_utils.py`. Otherwise, I'd prefer to not increase maintenance cost for things that are "nice to have but not used"


PULL REQUEST #45509: Fix get_device_properties crash when CUDA is installed but no GPU
State: closed  |  Merged: False
Author: Jah-yee  |  Base: main
Labels: Code agent slop
Created: 2026-04-18T23:41:33Z

Good day

## Problem
In `src/transformers/testing_utils.py`, the `get_device_properties()` function checks `IS_CUDA_SYSTEM` to determine whether to call `torch.cuda.get_device_capability()`. However, `IS_CUDA_SYSTEM` is set to `True` when `torch.version.cuda` is not `None`, regardless of whether a GPU is actually available.

When CUDA is installed (e.g., CUDA toolkit on a headless cloud instance with no GPU), but no GPU device is present, `torch.cuda.get_device_capability()` raises a `RuntimeError`, causing the function to crash.

## Solution
Added a guard check for `torch.cuda.is_available()` before calling `torch.cuda.get_device_capability()`:

```python
# Before
if IS_CUDA_SYSTEM or IS_ROCM_SYSTEM:

# After
if (IS_CUDA_SYSTEM or IS_ROCM_SYSTEM) and torch.cuda.is_available():
```

When CUDA is installed but no GPU is available, the function will now fall through to the `else` branch and return `(torch_device, None, None)` instead of crashing.

## Testing
- The fix is a minimal, targeted change — only one line modified.
- The reproduction is straightforward: run `get_device_properties()` on a CUDA-installed but GPU-less environment (e.g., cloud CPU-only instance with CUDA toolkit).
- The existing test `test_get_device_properties` should continue to pass on GPU-enabled machines.

Fixes https://github.com/huggingface/transformers/issues/45341

---

Thank you for your attention. If there are any issues or suggestions, please leave a comment and I will address them promptly.

Warmly,
RoomWithOutRoof

--- Comment by github-actions[bot] at 2026-04-18T23:55:36Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45509&sha=92806c

--- Comment by Rocketknight1 at 2026-04-20T12:12:49Z ---
Blocked because your code agent is spamming us with multiple identical PRs https://github.com/huggingface/transformers/pull/45515


PULL REQUEST #45508: [Doc] Fix 'tokenized' -> 'tokenizer' typo in streamer docstrings
State: closed  |  Merged: True
Author: avasis-ai  |  Base: main
Labels:
Created: 2026-04-18T21:48:22Z

## Summary

Fixes a systematic docstring typo in all three streamer classes in `src/transformers/generation/streamers.py`:

- `TextStreamer`
- `TextIteratorStreamer`
- `AsyncTextIteratorStreamer`

Each had `"The tokenized used to decode the tokens."` in the `tokenizer` parameter description. This should be `"The tokenizer used to decode the tokens."` — the parameter accepts a tokenizer object, not a tokenized output.

## Why this is not a duplicate

No existing PR addresses this specific documentation error across all three streamer classes.

## AI assistance disclosure

This PR was prepared with AI assistance. The submitting human has reviewed every changed line and confirms the fix is correct.

Co-authored-by: opencode

--- Comment by HuggingFaceDocBuilderDev at 2026-04-20T05:08:09Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45508). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45506: Add full GGUF loading support for GPT‑OSS (fixes #43366, supersedes #43757) latest
State: closed  |  Merged: True
Author: sirzechs66  |  Base: main
Labels:
Created: 2026-04-18T08:43:19Z

# What does this PR do?

This PR adds full GGUF loading support for GPT‑OSS models (20B/120B). It allows Transformers (and consequently vLLM) to directly load GPT‑OSS GGUF files without falling back to a wrong architecture. The changes include:
- Architecture registration in GGUF mappings.
- A custom `GptOssTensorProcessor` to handle MoE expert splitting and gate/up interleaving.
- Reconstruction of nested `rope_scaling` (YaRN) from flat GGUF metadata.
- Tests: fast registration test + slow integration test using a real 20B GGUF file.

Fixes #43366, supersedes #43757.
Related vLLM issue: https://github.com/vllm-project/vllm/issues/22353

## Code Agent Policy
- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). – *Not applicable, it adds a feature.*
- [x] Did you read the contributor guideline, Pull Request section? – Yes.
- [x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case. – Issue #43366, discussion in comments.
- [x] Did you make sure to update the documentation with your changes? – yes!
- [x] Did you write any new necessary tests? – Yes, in test/quantization/test_ggml.py

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Tagging:
- @SunMarc (original issue tagger, quantization)
- @CyrilVallez (model loading)
- @ArthurZucker (tokenizers, model structure)

--- Comment by sirzechs66 at 2026-04-18T09:01:44Z ---
@SunMarc please review this

--- Comment by sirzechs66 at 2026-04-19T06:16:12Z ---
@ArthurZucker , @Rocketknight1 please review and merge these changes all things are tested and ready

--- Comment by sirzechs66 at 2026-04-20T08:12:23Z ---
@SunMarc please review it is ready to be merged

--- Comment by github-actions[bot] at 2026-04-20T16:07:57Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ggml

--- Comment by HuggingFaceDocBuilderDev at 2026-04-22T06:38:44Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45506). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by SunMarc at 2026-04-20T15:52:48Z ---
let's not modify the docs
```suggestion

```

```


PULL REQUEST #45504: .4021378118068288:1e40fd96a800b4038c914120b0aa85c2_69e2faf342c82e665fe04170.69e2faf642c82e665fe04174.69e2faf6d18af355a624eac3:Trae CN.T(2026/4/18 11:31:02)
State: closed  |  Merged: False
Author: kchpp940  |  Base: main
Labels:
Created: 2026-04-18T03:53:58Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by github-actions[bot] at 2026-04-18T03:55:08Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: clap, deit, depth_anything

--- Comment by github-actions[bot] at 2026-04-18T04:11:03Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45504&sha=58e4a6


PULL REQUEST #45503: .4021378118068288:aafb9167aaa6b321205f754209b0cbcb_69e2f25642c82e665fe0407f.69e2f43542c82e665fe04083.69e2f435d18af355a624eac1:Trae CN.T(2026/4/18 11:02:13)
State: closed  |  Merged: False
Author: kchpp940  |  Base: main
Labels:
Created: 2026-04-18T03:28:18Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by github-actions[bot] at 2026-04-18T03:29:34Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: clap, deit, depth_anything


PULL REQUEST #45502: .4021378118068288:4da7ed27ccaa5f974fe4a552e2b67bb6_69e2eea842c82e665fe04002.69e2eec142c82e665fe04006.69e2eec18b8bd167e2faa241:Trae CN.T(2026/4/18 10:38:57)
State: closed  |  Merged: False
Author: kchpp940  |  Base: main
Labels:
Created: 2026-04-18T02:59:11Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


PULL REQUEST #45501: .4021378118068288:e3506c3c5a98ec3a50332c6102362804_69e2e58842c82e665fe03f65.69e2e58c42c82e665fe03f69.69e2e58b8b8bd167e2faa23e:Trae CN.T(2026/4/18 09:59:40)
State: closed  |  Merged: False
Author: kchpp940  |  Base: main
Labels:
Created: 2026-04-18T02:26:56Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


PULL REQUEST #45500: Add full GGUF loading support for GPT‑OSS (fixes #43366, supersedes #43757) latest
State: closed  |  Merged: False
Author: sirzechs66  |  Base: main
Labels:
Created: 2026-04-18T02:16:41Z

# What does this PR do?

This PR adds full GGUF loading support for GPT‑OSS models (20B/120B). It allows Transformers (and consequently vLLM) to directly load GPT‑OSS GGUF files without falling back to a wrong architecture. The changes include:
- Architecture registration in GGUF mappings.
- A custom `GptOssTensorProcessor` to handle MoE expert splitting and gate/up interleaving.
- Reconstruction of nested `rope_scaling` (YaRN) from flat GGUF metadata.
- Tests: fast registration test + slow integration test using a real 20B GGUF file.

Fixes #43366, supersedes #43757.
Related vLLM issue: https://github.com/vllm-project/vllm/issues/22353

## Code Agent Policy
- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). – *Not applicable, it adds a feature.*
- [x] Did you read the contributor guideline, Pull Request section? – Yes.
- [x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case. – Issue #43366, discussion in comments.
- [x] Did you make sure to update the documentation with your changes? – yes!
- [x] Did you write any new necessary tests? – Yes, in test/quantization/test_ggml.py

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Tagging:
- @SunMarc (original issue tagger, quantization)
- @CyrilVallez (model loading)
- @ArthurZucker (tokenizers, model structure)

--- Comment by github-actions[bot] at 2026-04-18T02:17:45Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ggml

--- Comment by github-actions[bot] at 2026-04-18T02:36:10Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45500&sha=483e8f


PULL REQUEST #45499: .4021378118068288:94a295563cf6b5aa7d67bd0f2c0cd7a7_69e2df7342c82e665fe03edd.69e2df7542c82e665fe03ee1.69e2df758b8bd167e2faa23c:Trae CN.T(2026/4/18 09:33:41)
State: closed  |  Merged: False
Author: kchpp940  |  Base: main
Labels:
Created: 2026-04-18T01:54:36Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


PULL REQUEST #45498: .4021378118068288:98296403e0cd6dedb7b420b80d0fe80b_69e2ce7c42c82e665fe03e0c.69e2ce9842c82e665fe03e10.69e2ce988b8bd167e2faa239:Trae CN.T(2026/4/18 08:21:44)
State: closed  |  Merged: False
Author: kchpp940  |  Base: main
Labels:
Created: 2026-04-18T01:12:45Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->