PULL REQUEST #46140: Fix LlamaConfig rejecting explicit head_dim when hidden_size is not divisible by num_attention_heads
State: closed  |  Merged: False
Author: adityasingh2400  |  Base: main
Labels: Code agent slop
Created: 2026-05-21T11:23:35Z

# What does this PR do?

Fixes #46082.

`LlamaAttention` already sizes its projections from `num_attention_heads * head_dim` rather than `hidden_size`, so a config where `hidden_size % num_attention_heads != 0` is well-defined as long as `head_dim` is explicitly provided. The divisibility check in `LlamaConfig.validate_architecture` fires unconditionally though (it runs after `__post_init__` has filled in the fallback `head_dim`, so checking `head_dim is not None` in the validator doesn't work).

This PR follows the approach @matdou outlined in the issue:

- Capture `self._head_dim_was_explicit = self.head_dim is not None` in `__post_init__` before falling back to the derived value.
- Gate the divisibility error in `validate_architecture` on `not self._head_dim_was_explicit`.

`_head_dim_was_explicit` is recomputed in `__post_init__`, so save/reload via `save_pretrained` / `from_pretrained` works without persisting the flag (the saved `head_dim` is the explicit value, so the flag is set correctly on reload).

The original validation error is preserved when `head_dim` is *not* explicitly provided.

## Reproduction (from the issue)

```python
from transformers import LlamaConfig, LlamaForCausalLM

config = LlamaConfig(
    vocab_size=99,
    hidden_size=512,
    intermediate_size=1024,
    num_hidden_layers=1,
    num_attention_heads=9,
    num_key_value_heads=1,
    head_dim=56,
)
model = LlamaForCausalLM(config)
```

Passes after this change, raises before.

## Tests

Added two cases in `tests/models/llama/test_modeling_llama.py`:

- `head_dim` explicit + non-divisible dims, config accepted, model instantiates.
- `head_dim` omitted + non-divisible dims, original `ValueError` still raised.

## Who can review?

@ArthurZucker @Cyrilvallez

Credit to @matdou for the diagnosis in the issue comments.

--- Comment by github-actions[bot] at 2026-05-21T11:31:52Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: arcee, aria, cwm, deepseek_v2, eurobert, higgs_audio_v2, hrm_text, jais2, llama

--- Comment by github-actions[bot] at 2026-05-21T11:50:16Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46140&sha=70eb70

--- Comment by adityasingh2400 at 2026-05-21T12:08:04Z ---
CI note: the 4 failing tests on `tests_torch` / `tests_training_ci` / `tests_tensor_parallel_ci` are all in `tests/models/cohere2_moe/test_modeling_cohere2_moe.py` and reproduce identically on an unrelated PR opened a few minutes after this one (see #46136). The failing assertions are:

- `Cohere2MoeModelTest::test_training_overfit`, `AssertionError: 0.27068585289520714 not greater than 0.9` (the exact same float value reproduces across runs, so it is deterministic, not flaky)
- `Cohere2MoeModelTest::test_tp_forward` / `test_tp_backward` / `test_tp_generation`, `KeyError: 'rowwise'` raised by the TP partition spec on a Cohere2 MoE layer

Both classes of failure were introduced when `cohere2_moe` landed yesterday (#46115 on 2026-05-20). My change is scoped to `LlamaConfig` and the modular-converted descendants (arcee, aria, cwm, deepseek_v2, eurobert, higgs_audio_v2, hrm_text, jais2). `cohere2_moe` does not derive from Llama and is not touched by this PR.

Happy to file a separate PR for the cohere2_moe breakage if no one is on it already, but flagging it here so this PR is not held on CI red that is upstream of it.


chore: update self-comment-ci.yml

PULL REQUEST #46138: chore: update self-comment-ci.yml
State: open  |  Merged: False
Author: hf-security-analysis[bot]  |  Base: main
Labels:
Created: 2026-05-21T09:57:53Z

Update `.github/workflows/self-comment-ci.yml` workflow configuration.

cc @guarin @molbap

Closes huggingface/tracking-issues#487
<!--slack ts:1779357475.432589 channel:C0AJSP0D53L-->

--- Comment by HuggingFaceDocBuilderDev at 2026-05-21T10:10:08Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46138). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Update self-comment-ci

PULL REQUEST #46137: Update self-comment-ci
State: closed  |  Merged: True
Author: guarin  |  Base: main
Labels:
Created: 2026-05-21T09:41:02Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-05-21T09:52:56Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46137). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #46136: Fix is_last off-by-one in MaskGenerationPipeline for partial batches
State: open  |  Merged: False
Author: J3r3myPerera  |  Base: main
Labels:
Created: 2026-05-21T07:50:15Z

Fixes #46123

MaskGenerationPipeline.preprocess used i == n_points - points_per_batch to spot the last batch. When n_points isn't a multiple of points_per_batch, that's never true — PipelinePackIterator hits StopIteration and quietly drops the last batch's results.

Fix: i + points_per_batch >= n_points.
Two fast unit tests in test_pipelines_mask_generation.py: one for the partial-batch case (100 points, batch 64), one for an exact multiple (128 points, batch 64).

`python -m pytest tests/pipelines/test_pipelines_mask_generation.py::MaskGenerationPipelineTests::test_preprocess_is_last_partial_batch tests/pipelines/test_pipelines_mask_generation.py::MaskGenerationPipelineTests::test_preprocess_is_last_exact_multiple -v
`
#2 passed

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
Discussed in https://github.com/huggingface/transformers/issues/46123
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?
Added test_preprocess_is_last_partial_batch and test_preprocess_is_last_exact_multiple to tests/pipelines/test_pipelines_mask_generation.py.

## Who can review?

cc @Rocketknight1 @yonigozlan @qubvel

--- Comment by Shashank-Tripathi-07 at 2026-05-21T11:42:39Z ---
Hey bro, the code looks good but you said you didn't use AI agents to make this PR but there are em dashes very visible on the comment you made and also in the original issue. This can be a problem as the repo doesn't like Agent Slop even 1%. Take a look again for safety on this.

--- Comment by J3r3myPerera at 2026-05-21T11:49:22Z ---
The CI failures here are pre-existing on main — not caused by this change.
ci/circleci: tests_tensor_parallel_ci — all 3 failures are in Cohere2MoeModelTest (test_tp_forward, test_tp_backward, test_tp_generation), crashing with KeyError: 'rowwise' in distributed/tensor_parallel.py. This PR doesn't touch any of that.

ci/circleci: tests_training_overfit_ci — 1 failure, also Cohere2MoeModelTest::test_training_overfit, loss only drops 27% vs a 90% threshold. Unrelated.

Only two files changed:

src/transformers/pipelines/mask_generation.py (1 line)
tests/pipelines/test_pipelines_mask_generation.py (2 tests)

Neither touches Cohere2MoeModel or anything in the distributed training path.

--- Comment by J3r3myPerera at 2026-05-21T11:56:15Z ---
> Hey bro, the code looks good but you said you didn't use AI agents to make this PR but there are em dashes very visible on the comment you made and also in the original issue. This can be a problem as the repo doesn't like Agent Slop even 1%. Take a look again for safety on this.

Fair point, and I'll own it. I did use AI to help word the PR description and the issue comment. The fix itself I worked out on my own: i == n_points - points_per_batch only hits when n_points is an exact multiple, so any partial tail batch never gets flagged as last, PipelinePackIterator raises StopIteration and the results are quietly dropped. Replacing it with i + points_per_batch >= n_points handles both cases. I understand what the code does and why the old condition was wrong.

That said, em dashes in prose aren't really a reliable signal for agent-generated code. Plenty of people type them on purpose. The actual thing to check is whether the logic holds up. Which I'd rather be judged on.

--- Comment by Rocketknight1 at 2026-05-21T12:46:45Z ---
You can ignore those comments, he's just annoyed I wouldn't listen when he claimed his Claude PR was human-written. In this case the actual fix is one line and seems correct, so I don't really care too much whether an agent wrote it or not. You do not actually need to go around hiding all the em-dashes :sweat_smile:

--- Comment by Rocketknight1 at 2026-05-21T13:21:34Z ---
@J3r3myPerera looks like there might be some CI instability at the moment. Can you wait a bit and then try rebasing or rerunning tests? Once the CI is green ping me and I'll merge it.

--- Comment by github-actions[bot] at 2026-05-21T13:30:20Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46136&sha=ba5335

--- Comment by HuggingFaceDocBuilderDev at 2026-05-21T13:31:34Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46136). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by J3r3myPerera at 2026-05-21T18:21:07Z ---
> @J3r3myPerera looks like there might be some CI instability at the moment. Can you wait a bit and then try rebasing or rerunning tests? Once the CI is green ping me and I'll merge it.

Sure mate no worries.


PULL REQUEST #46135: [Test][Kosmos2.5] Add XPU expectations for integration tests
State: open  |  Merged: False
Author: YangKai0616  |  Base: main
Labels:
Created: 2026-05-21T06:24:52Z

# What does this PR do?

This PR:

1. Updates the Kosmos2.5 integration tests to use the shared `Expectations` helper for device-specific generated outputs, matching the style used in other `transformers` tests.
2. Adds XPU ground truth for the eager and sdpa generation paths.

@ydshieh , pls help review, thx!

--- Comment by github-actions[bot] at 2026-05-21T06:26:02Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: kosmos2_5


PULL REQUEST #46134: Validate shard filenames in get_checkpoint_shard_files
State: open  |  Merged: False
Author: karnakarreddi  |  Base: main
Labels:
Created: 2026-05-21T04:55:27Z

weight_map values from the checkpoint index file are joined to the model directory path without validation. A crafted index containing entries like '../../etc/passwd' causes the loader to resolve paths outside the model directory when loading from a local folder.

Add a containment check using os.path.realpath so each resolved shard path must stay under the base model directory. Legitimate shard filenames are flat names (e.g. model-00001-of-00003.safetensors) and never need to escape the snapshot directory, so this is purely defensive with no expected regressions.

Fixes #46097

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@Rocketknight1

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by github-actions[bot] at 2026-05-21T05:12:11Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46134&sha=239470


PULL REQUEST #46130: Fix image-segmentation pipeline support for RF-DETR
State: closed  |  Merged: True
Author: yonigozlan  |  Base: main
Labels:
Created: 2026-05-20T21:47:18Z

# What does this PR do?
The pipeline was passing a non supporter arg to the post_processing method, which was causing a crash. This PR fixes it and add a snippet in the docs

--- Comment by github-actions[bot] at 2026-05-20T21:48:28Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: rf_detr

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T22:01:01Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46130). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Add SAM 3.1 to Transformers

PULL REQUEST #46128: Add SAM 3.1 to Transformers
State: open  |  Merged: False
Author: yonigozlan  |  Base: main
Labels: New model
Created: 2026-05-20T21:00:58Z

# What does this PR do?

First draft for SAM3.1 model.

Current state:
- Sam31TrackerVideoModel seems to be fully working prediction quality wise, but could use some polishing in the code, although modular is already doing some heavy lifting.
- Sam31VideoModel is mostly working, but there are still tiny differences with the original implementation that compound over time. Not ready quality wise and will need more investigating to understand what's going wrong compared to the original implementation

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T21:13:05Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46128). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-21T22:57:44Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, sam3, sam3_tracker_video, sam3_video

--- Comment by github-actions[bot] at 2026-05-21T23:06:00Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46128&sha=10412e


PULL REQUEST #46127: [deepseek_v4] Add DeepseekV4NextNPredictor class for MTP draft-head support
State: open  |  Merged: False
Author: pasta-paul  |  Base: main
Labels:
Created: 2026-05-20T20:41:45Z

## Summary

DeepSeek-V4-Flash ships with a Multi-Token-Prediction (MTP) draft block (`num_nextn_predict_layers = 1`) whose weights are stored under an `mtp.0.*` prefix in the safetensors index. The current implementation drops these on load via:

```python
_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]
```

…which makes the MTP head unusable for vLLM's speculative decoding (`--speculative-config '{"method":"mtp","num_speculative_tokens":N}'`).

## Changes

1. Adds `DeepseekV4NextNPredictor` as a `DeepseekV4DecoderLayer` subclass with MTP-specific projections (`e_proj`, `h_proj`, `enorm`, `hnorm`, `norm`) and head parameters (`hc_head_fn`, `hc_head_base`, `hc_head_scale`). Mirrors the upstream reference at `DeepSeek-V4-Flash/inference/model.py:MTPBlock`.

2. Wires `DeepseekV4Model.__init__` to instantiate `self.mtp` as a `nn.ModuleList` of `num_nextn_predict_layers` blocks. Empty when the config has `num_nextn_predict_layers = 0` (no behavior change for non-MTP variants).

3. Drops the silent-drop regex from `_keys_to_ignore_on_load_unexpected` (now `[]`). With the MTP module class instantiated, `mtp.*` keys load into real submodules instead of being filtered out.

4. Auto-extends `config.layer_types` and `config.mlp_layer_types` from inside `DeepseekV4NextNPredictor.__init__` to cover the MTP layer's index. `DeepseekV4Attention` and `DeepseekV4SparseMoeBlock` index these lists by `layer_idx`; without the extension, instantiating at `layer_idx = num_hidden_layers` raises `IndexError`.

## Verification

Tested locally with the DSv4-Flash config (`num_hidden_layers=43`, `num_nextn_predict_layers=1`, `hc_mult=8`):

```
OK: MTP block instantiated
params: 6.629190907 B
missing expected children: none
hc_head_* params: ['hc_head_fn', 'hc_head_base', 'hc_head_scale']
```

Class instantiates, `named_modules()` enumerates the full sub-tree (including `mtp.0.mlp.experts.0..255.{gate_proj,down_proj,up_proj}` and `mtp.0.self_attn.{q_a_proj,q_b_proj,kv_proj,o_a_proj,o_b_proj}`), `from_pretrained` loads `mtp.0.*` keys into the new submodules.

## Open items (for follow-up commits or follow-on PRs)

- `DeepseekV4NextNPredictor.forward()` implementation for inference-time draft. This PR adds the structural shell; the weights load and submodules are walkable for downstream calibration / quantization tooling, but invoking the draft head at inference time requires the embed + e_proj/h_proj composition and the `super().forward` + `norm` + head wiring.
- Integration with `DeepseekV4Model.forward` (gated kwarg to invoke the MTP draft after the main forward).
- Tests under `tests/models/deepseek_v4/`:
  - `from_pretrained` round-trip preserves `mtp.0.*` weights
  - `model.mtp[0]` reachable via `named_modules()`
  - `num_nextn_predict_layers = 0` path unchanged

## Note on modular vs generated file

I edited both `modular_deepseek_v4.py` (source of truth) and `modeling_deepseek_v4.py` manually because my local environment doesn't have `libcst` installed to run `utils/modular_model_converter.py`. Please re-run the converter if the manual edit to the generated file drifts from what the converter would produce.

## Related

Discussed at vllm-project/llm-compressor#2735 (the calibration side of the MTP-retention story).

cc @kylesayrs — companion to your active DSv4 iteration on `kylesayrs/transformers-v5` in the llm-compressor repo.

--- Comment by pasta-paul at 2026-05-20T21:36:58Z ---
## Correction to design — `layer_type="sliding_attention"` for MTP

While verifying this PR against an actual DSv4-Flash MTP checkpoint, I found that the layer_type extension logic needs to be more specific than "repeat the last main-layer's type" (which was the initial approach in this PR's first push).

### Evidence

The DSv4-Flash config has three attention `layer_types`:

```
Counter({'compressed_sparse_attention': 21,    # full attn + Compressor + Indexer
         'heavily_compressed_attention': 20,   # full attn + Compressor
         'sliding_attention': 2})              # full attn, compressor = None
```

`DeepseekV4Attention.__init__` instantiates the compressor conditionally:

```python
self.compressor = (
    COMPRESSOR_CLASSES[self.layer_type](config)
    if self.layer_type != "sliding_attention" else None
)
```

An upstream-format DSv4-Flash MTP checkpoint has **only the standard attn projections** under `mtp.0.attn.*` (`wq_a`, `wq_b`, `wkv`, `wo_a`, `wo_b` + `q_norm`, `kv_norm`, `attn_sink`) — **no `compressor.*` or `indexer.*` keys**. The matching layer_type is therefore `sliding_attention` (the only one with `compressor = None`), not a copy of the last main-layer's type.

### Symptom of the wrong choice

With `layer_type = "compressed_sparse_attention"` (last main layer's type), the MTP `DeepseekV4Attention` instantiates empty `compressor` + `indexer` submodules. The checkpoint has no `mtp.0.attn.compressor.*` keys to load → those submodules stay uninitialized → `_initialize_weights` falls through to `_init_weights` → `init.normal_` random-initializes the MTP block. Silent corruption of MTP weights.

I'll push the corrected logic (`layer_types[mtp_idx] = "sliding_attention"`, `mlp_layer_types[mtp_idx] = "moe"`) plus the updated PR body shortly. Wanted to leave this comment now so anyone reading the PR mid-iteration sees the right design.

A test that catches this regression class:

```python
def test_mtp_from_pretrained_no_silent_random_init():
    model = AutoModelForCausalLM.from_pretrained("...DSv4-Flash BF16 with mtp...")
    # Pick any deterministic-loaded MTP projection
    loaded = model.model.mtp[0].self_attn.q_a_proj.weight.detach().cpu()
    # Compare against the raw safetensors value
    import safetensors.torch as st
    with st.safe_open("...shard with mtp.0.attn.wq_a.weight...", framework="pt") as f:
        source = f.get_tensor("mtp.0.attn.wq_a.weight").cpu()
    assert (loaded.float() - source.float()).abs().max() < 1e-4
```

--- Comment by github-actions[bot] at 2026-05-20T21:38:33Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_v4

--- Comment by pasta-paul at 2026-05-21T03:08:44Z ---
## Design note — `DeepseekV4NextNPredictor` is quantization-format-agnostic

While running our smoke (the W4A16-experts + FP8_BLOCK-attention + BF16-MTP recipe described at vllm-project/llm-compressor#2735), we landed on a sharper design point worth noting for this PR:

**The class shim enables MTP retention regardless of what precision the rest of the model uses.** Users can:

- Quantize the MTP block alongside the main MoE (W4A16 / NVFP4 / FP8 / etc.) — `targets=` regex would naturally match `mtp.*.mlp.experts.*` if not explicitly excluded
- Preserve MTP at higher precision than the main model (e.g., BF16 MTP + W4A16 experts) — `ignore=["lm_head", r"re:.*mtp\..*"]`

The latter is what DeepSeek's own MXFP4 release does, and what we're shipping. Speculative-decoding throughput depends on draft-head acceptance rate, which is quantization-noise-sensitive; preserving MTP at higher precision than the main MoE is the conservative recipe.

This PR's responsibility is just **making the class exist** so `mtp.*` keys have a submodule to land in. The quantization decision is downstream and recipe-level. Mentioning here because the PR description shouldn't constrain users to a particular precision choice for the MTP block.

Tests under `tests/models/deepseek_v4/` should also cover:

```python
def test_mtp_block_can_remain_bf16_when_main_model_is_quantized():
    """User opts out of MTP quantization via ignore=. MTP weights stay BF16."""
    # ...
```

In addition to the round-trip + presence tests previously listed.

--- Comment by Rocketknight1 at 2026-05-21T11:48:12Z ---
Hey, sorry, we probably won't review this right now! I don't think we want to merge a "structural shell" without a `forward()` or anything. You're correct that the draft head is currently not being loaded, but if we're going to add it we'll probably want to be able to test that it's working correctly, which we can't do when it's just a naked weight-container.

cc @arthurzucker

--- Comment by pasta-paul at 2026-05-21T13:52:43Z ---
Thanks Matt — your point stands, a structural shell without `forward()` isn't useful. I'll expand the PR with a functional `forward()` + tests. To save a round-trip I researched the two design choices myself rather than asking; calling out my proposed approach so you and Arthur can correct me before I sink time in the wrong direction.

## Q1: standalone module vs deeper integration

**Proposed: keep `DeepseekV4NextNPredictor` standalone, add an explicit invocation path.**

Specifically:

- `DeepseekV4Model.__init__` instantiates `self.mtp` as `nn.ModuleList` (already done in this PR).
- `DeepseekV4Model.forward()` does **not** invoke MTP by default — zero-cost path for non-MTP DSv4 variants (`num_nextn_predict_layers = 0`).
- Add a convenience method `DeepseekV4Model.compute_mtp_logits(input_ids, previous_hidden_states, position_ids, ...)` that runs a single MTP step. Returns the draft logits and the post-MTP residual (so callers can re-feed it for `num_speculative_tokens > 1`).
- Speculative-decoding orchestration (sampling, acceptance, draft chaining) is downstream tooling's responsibility (vLLM, sglang) — transformers exposes the primitive, doesn't bake in a speculation protocol.

Why standalone:

1. Both reference implementations (DeepSeek's release `inference/model.py:MTPBlock`, vLLM's `vllm/models/deepseek_v4/nvidia/mtp.py:DeepSeekV4MultiTokenPredictorLayer`) treat MTP as a separate forward path called *after* the main model, not interleaved into its forward.
2. Keeps the main `forward()` unchanged — no perf regression for the non-MTP path.
3. More unit-testable in isolation: synthetic hidden states + synthetic input_ids → MTP forward, without running the full 43-layer decoder.

## Q2: which reference implementation

**Proposed: port the math from DeepSeek's release inference code, cross-check class structure against vLLM.**

The decisive factor: vLLM's `HCHeadOp.forward_native` is `raise NotImplementedError` (`vllm/model_executor/layers/mhc.py:266`). vLLM's HC ops are CUDA/Triton kernels with no pure-PyTorch reference. We can't directly port `vllm/.../mtp.py` because the HC math (the load-bearing MTP-specific kernel) is only available there as `_hc_head_cuda_impl`.

DeepSeek's release `inference/model.py` *does* have the pure-PyTorch HC math:

```python
def hc_pre(self, x, hc_fn, hc_scale, hc_base):
    # x: [b,s,hc,d], hc_fn: [mix_hc,hc*d], hc_base: [mix_hc]
    shape, dtype = x.size(), x.dtype
    x = x.flatten(2).float()
    rsqrt = ...  # rsqrt of mean of squares (RMS) over the flattened HC dim
    mixes = F.linear(x, hc_fn) * rsqrt
    pre, post, comb = hc_split_sinkhorn(mixes, hc_scale, hc_base, ...)
    y = torch.sum(pre.unsqueeze(-1) * x.view(shape), dim=2)
    return y.to(dtype), post, comb
```

And the MTP block forward is concretely:

```python
def forward(self, x, start_pos, input_ids):
    e = self.embed(input_ids)
    e = self.enorm(e)
    x = self.hnorm(x)
    x = self.e_proj(e).unsqueeze(2) + self.h_proj(x)
    x = super().forward(x, start_pos, input_ids)  # the inherited Block's forward
    logits = self.head(x, self.hc_head_fn, self.hc_head_scale, self.hc_head_base, self.norm)
    return logits
```

That's the math I'll port. vLLM's `mtp.py` is then a cross-check for module hierarchy decisions (e.g., should the MTP block subclass `DeepseekV4DecoderLayer` like DeepSeek release does, or compose one as a member? — DeepSeek release uses inheritance; vLLM uses composition; our current PR uses inheritance to match DeepSeek release; I'll keep that).

I'll cite both references in the file header + commit message.

## Tests

I'll add three under `tests/models/deepseek_v4/`:

1. **Synthetic forward shape/no-NaN** — tiny config (small vocab, small hidden, 2 decoder layers, 1 MTP layer), random init, run `compute_mtp_logits` once, assert output shape and finiteness.
2. **Weight round-trip preserves `mtp.*` keys** — `from_pretrained`/`save_pretrained` cycle leaves the `mtp.0.*` parameters bit-identical (catches the regression where the silent-drop regex was reintroduced).
3. **HC math equivalence with the DeepSeek release reference** — implement the same `hc_pre`/`hc_post` computation against a fixed seed and compare to our ported version with `torch.allclose(..., atol=1e-5)`. No weight download required; deterministic on synthetic inputs.

## Cross-library coordination

This PR is the upstream-modeling half of an MTP-preserving DSv4 quant story. The downstream serve-path detection lives in vLLM PR #43319 (the sibling B300 chat's work) — once both land, an MTP-preserving DSv4-Flash artifact ships clean across the whole stack: transformers loads + recognizes MTP, llm-compressor calibrates it (PR #2739), vLLM serves it with speculative decoding (#43319). Happy to coordinate test runs across libraries if useful.

I'll keep this PR open and push the revised commit with forward + tests. Aiming for ~24h. Let me know if you'd prefer a different invocation surface than `compute_mtp_logits(...)` — that's the one externally-visible API choice I can't fully decide without your input.

cc @arthurzucker

--- Comment by pasta-paul at 2026-05-21T20:59:57Z ---
**Update — the load-path side is now end-to-end validated** (responding to @Rocketknight1's "wait until it works" point from earlier in the thread).

We've shipped an NVFP4-FP8 quantization of DSV4-Flash that retains the MTP block in the saved safetensors and runs through vLLM's MTP draft path with `--speculative-config method=mtp`:

- Artifact: [`canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP`](https://huggingface.co/canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP) (172 GB, 35 shards, 799 MTP tensors at BF16, public on HF).
- **MTP draft acceptance, AIME 2024 (thinking=high, 30 problems, c=8): 81.60%** — Prometheus counters confirm draft tokens emitted + accepted. Per-position acceptance 96.6% / 79.4% (positions 0 / 1).
- **Quality parity vs BF16+MTP reference**: 83.33% raw pass@1 (both), 96.0% vs 96.15% non-truncated at the same 65K max_tokens cap.
- **Decode speedup vs no-MTP** (same NVFP4 quant): 1.84× per-request median tok/s on AIME reasoning, 2.13× at c=1 chat coding. Across the chat-template c=1→c=16 sweep, acceptance held flat at 87.92–88.27% — no batch-load degradation.

The validation went through vLLM's own MTP load path (PRs [#43319](https://github.com/vllm-project/vllm/pull/43319) etc. for the BF16-on-disk detection that lets MTP coexist with quantized main weights). What this demonstrates for #46127: the saved \`mtp.*\` keys are functionally useful — when retained, they drive a measurable spec-decode speedup on real benchmarks. The transformers-side \`DeepseekV4NextNPredictor\` class would let other inference engines (not just vLLM) consume these weights through stock transformers' load path.

Maintainer ask was to add \`forward()\` + tests before merge — totally fair. The above is offered as supporting evidence that the load side has downstream value, in case it helps prioritize.


PULL REQUEST #46126: EP + Trainer integration on top of DistributedConfig (#45028)
State: open  |  Merged: False
Author: AmineDiro  |  Base: main
Labels:
Created: 2026-05-20T20:29:05Z

Builds on [#45028](https://github.com/huggingface/transformers/pull/45028) (`refactor-tp-dtensor`) so that **Expert Parallelism actually runs** on any MoE model that has a `base_ep_plan`:
  ```python
  DistributedConfig(enable_expert_parallel=True, tp_size=N, fsdp_size=M)
  ```

I was validated end-to-end with  `trl.SFTTrainer` on `Qwen3-30B-A3B` at multiple shapes (16k / 32k / 64k, 2n / 4n / 8n).  Healthy training (loss bit-exact match against our prod fork at LR=0) and very competitive MFU 👏🏼 👏🏼

> **Best run: 64k ctx length,  2 nodes, EP=16 + sonicmoe + Liger + compile = 62.8 % window MFU / 35 % causal-adjusted, peak GPU mem 45 GB**.

# What does this PR do?
- **Register `ep_router` + `moe_experts_ep_allreduce` ParallelStyles** (`distributed/tensor_parallel.py`). `RouterParallel` masks non-local expert scores.  `MoEExpertsParallel` shards on the experts dim.
- `model._tp_plan` → `model.tp_plan` so `enable_expert_parallel=True` actually selects `_ep_plan` (the property override at `modeling_utils.py:1459`).
- **`Trainer._clip_grad_norm` routes to the mesh-aware streaming clip** (`trainer.py`) when on the native path. Stock `accelerator.clip_grad_norm_` across cross-mesh DTensor grads crashes on the first `optimizer.step`.
- **`Trainer.create_optimizer` calls `maybe_disable_foreach_and_fused_for_mixed_dtensor_groups`** helper; Adam fused/foreach rejects mixed Tensor/DTensor groups under `fsdp_size=1`.
- **`Trainer.compute_loss` divides `loss_scale` by `tp_size` for the `DistributedConfig` path** (`trainer.py`). Companion to #45994 (which only handles `parallelism_config.tp_size`).
- **Biggest change is that I introduce `Trainer.is_native_distributed` flag**, exclusive with `is_fsdp_enabled` / `is_deepspeed_enabled` @3outeille  tell me what you think.   This is made to make clear when we are running in native vs accelerate accross branches.
- **Qwen3MoE `base_model_ep_plan` simplified** to use `moe_experts_ep_allreduce`: @ArthurZucker  I wonder if we need to have this ep plan dynamically set by the user and not part of the model definition 🤔

# Note:
This PR depends on :
- ⚠ **[#46113 (clip_grad_norm streaming)](https://github.com/huggingface/transformers/pull/46113)** — required. This PR's `_clip_grad_norm` route depends on the streaming O(1)-per-param clip from #46113. Without it, every 30B+ EP+FSDP `optimizer.step` allocates a ~60 GB transient (current `clip_grad_norm` replicates every DTensor grad). So we should land #46113 first.
- **[#45994 (loss-scale fix under TP)](https://github.com/huggingface/transformers/pull/45994)**  PR. This PR extends #45994 to the native FSDP path.

## Who can review?

@3outeille @ArthurZucker @SunMarc

--- Comment by github-actions[bot] at 2026-05-20T20:30:13Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_moe

--- Comment by github-actions[bot] at 2026-05-20T20:41:57Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46126&sha=e8de65

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T20:42:59Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46126). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #46125: Fix _is_package_available reporting available without a version
State: open  |  Merged: False
Author: blipbyte  |  Base: main
Labels:
Created: 2026-05-20T19:46:05Z

  `_is_package_available` confuses a directory on `sys.path` with
  an installed PyPI package, returning `(True, "N/A")`. Callers
  that compare versions (`is_kernels_available` for example) then
  crash on `version.parse("N/A")`.

  Fix: if no real version can be read, mark the package as
  unavailable.

  Found while running tests in PyCharm — its pytest runner adds
  `tests/` to `sys.path`, where `tests/kernels/` shadows the real
  PyPI `kernels` package.

  **Tests**
  - Crash before; clean `False` after.
  - `make check-code-quality && make typing && make check-repo` pass.

  **AI**
  Drafted with Claude. I reviewed every line and ran the repro
  locally. No existing issue or open PR.

--- Comment by Rocketknight1 at 2026-05-21T11:41:05Z ---
Yeah, I think the bug is real, but I'm not sure about this fix. I'm fine with a small agent PR to fix this, but is there a more robust approach, maybe using `PACKAGE_DISTRIBUTION_MAPPING`? We have a lot of code that does something like `_is_package_available("optimum")[0]` and I think that would still fail with this fix if there's a name conflict in `sys.path`.

--- Comment by blipbyte at 2026-05-21T13:45:00Z ---
Update: I tried the `PACKAGE_DISTRIBUTION_MAPPING` approach, but it caused a regression — `packages_distributions()` does not index several installed packages in my local env
  (`tokenizers`, `safetensors`, `numpy`, `packaging`, `filelock`), so the gate rejected them and CI failed (e.g. `TokenizersBackend` could no longer be imported because
  `is_tokenizers_available()` returned False).

  Reverted to the original narrow fix in the editable-install fallback. It addresses the reported `is_kernels_available()` crash but doesn't cover
  `_is_package_available("foo")[0]`-style callers against sys.path shadows, as you noted. Open to suggestions for a more robust check that doesn't depend on `packages_distributions()` coverage.

--- Comment by github-actions[bot] at 2026-05-21T14:29:47Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46125&sha=ea7e3c


PULL REQUEST #46124: Fix crash and tokenization mismatch in speculative decoding with different tokenizers
State: open  |  Merged: False
Author: rohanbalsaraf  |  Base: main
Labels:
Created: 2026-05-20T18:41:24Z

# What does this PR do?

This PR fixes two critical issues in `AssistedCandidateGeneratorDifferentTokenizers` and `UniversalSpeculativeDecodingGenerator` used for speculative decoding when the assistant and target models have different tokenizers/vocabularies.

### The Problems
1. **Runtime Crash:** The assistant model would often crash with a `RuntimeError` (size mismatch in RoPE/positional embeddings). This happened because it was incorrectly inheriting stale `position_ids` and `token_type_ids` from the main model'\''s context, rather than generating its own based on its specific tokenizer'\''s sequence length.
2. **Tokenization Mismatch:** The code was unintentionally adding a BOS (Beginning of Sentence) token during every iterative re-tokenization step. This led to "stuttering" in the generated sequence (e.g., `<s> Hello <s> world`). Additionally, aggressive space cleaning was causing BPE-based tokenizers to get out of sync.

### The Fix
- **Kwarg Management:** Explicitly `pop` stale `position_ids`, `attention_mask`, and `token_type_ids` before the assistant model'\''s forward pass, allowing it to correctly recalculate them for its own sequence length.
- **Controlled Tokenization:** Updated `convert_source_tokens_to_target_tokens` to include an `add_special_tokens` flag. It now uses `False` during iterative generation to prevent BOS duplication and sets `clean_up_tokenization_spaces=False` to preserve BPE consistency.
- **Unified Logic:** Aligned the sampling path (`UniversalSpeculativeDecodingGenerator`) with the greedy path for consistency.
- **Test Improvement:** Updated `test_speculative_decoding_equals_regular_decoding` to use a string-based prompt. Since different-tokenizer speculation relies on a text bottleneck, random integer IDs are not a valid test case for this specific architecture.

These issues were discovered through a proactive analysis of the `tests/generation/` suite where `test_speculative_decoding_equals_regular_decoding` was failing.

## Code Agent Policy
- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that'\''s the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)?
- [ ] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests? (Updated existing failing test to be robust).

## Who can review?
@zucchini-nlp @gante

--- Comment by github-actions[bot] at 2026-05-20T18:58:30Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46124&sha=f42500

--- Comment by Rocketknight1 at 2026-05-21T11:23:22Z ---
Not sure who's maintaining the universal speculative decoder rn, maybe @cyrilvallez?


PULL REQUEST #46122:  Add cache-aware streaming and RNN-T head for Parakeet
State: open  |  Merged: False
Author: jmayank1511  |  Base: main
Labels: Audio
Created: 2026-05-20T18:13:18Z

# What does this PR do?

  Extends Parakeet TDT (#44171) with cache-aware streaming + an RNN-T
   head. The cache-aware variant is shipped as **Nemotron Speech
  Streaming** (`nvidia/nemotron-speech-streaming-en-0.6b`).

  ### Adds
  - `ParakeetForRNNT` — RNN-T head (LSTM prediction net + joint
  network) on the shared encoder. Supports regular and cache-aware
  checkpoints. Offline via `generate(...)`.
  - `ParakeetCacheAwareStreamingBuffer` + `streaming_step()` +
  `get_initial_streaming_state()` — chunk-by-chunk greedy RNN-T
  decoding with encoder KV cache, decoder LSTM state, and last-token
  threaded across chunks. Mirrors NeMo's
  `_greedy_decode_blank_as_pad_loop_frames`.
  - Prompt-conditioned multilingual — one-hot language MLP after
  encoder, resolved via `target_lang`.
  - NeMo `CausalConv2D` padding alignment in subsampling.
  - Unigram SentencePiece support in `ParakeetConverter`.
  - Doc + unit tests for `ParakeetForRNNT`.

## Code Agent Policy
- [x]  I confirm that this is not a pure code agent PR.

 ## Before submitting
  - [ ] This PR fixes a typo or improves the docs.
  - [x] Did you read the [contributor
  guideline](https://github.com/huggingface/transformers/blob/main/CON
  TRIBUTING.md#create-a-pull-request)?
  - [ ] Was this discussed/approved via a Github issue or the forum?
  - [x] Did you make sure to update the documentation with your
  changes?
  - [x] Did you write any new necessary tests?

## Who can review?

 @eustlb @ebezzam @vasqu

--- Comment by github-actions[bot] at 2026-05-20T18:14:32Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, parakeet


PULL REQUEST #46120: Add video_processor support to VideoClassificationPipeline
State: closed  |  Merged: False
Author: Shashank-Tripathi-07  |  Base: main
Labels: Code agent slop
Created: 2026-05-20T15:18:30Z

## What does this PR do?

Fixes #41950.

`VideoClassificationPipeline` previously hardcoded `_load_image_processor = True`, which crashes for models that ship a `VideoProcessor` instead of an `ImageProcessor` (e.g. vjepa2). This PR extends the pipeline to support both, following the established `_load_*` flag pattern used throughout the pipeline base class.

## Changes

- **`pipelines/base.py`**: Add `_load_video_processor = None` class flag and `video_processor` parameter/attribute to `Pipeline.__init__`, matching the existing pattern for `image_processor`, `feature_extractor`, etc.
- **`pipelines/__init__.py`**: Import `AutoVideoProcessor`, add `_resolve_video_processor()` function, and wire it into the `pipeline()` factory.
- **`pipelines/video_classification.py`**: Change `_load_image_processor` from `True` to `None` (optional), add `_load_video_processor = None`, update `preprocess()` to use whichever preprocessor is present (`video_processor` takes priority over `image_processor` over `feature_extractor`), and add a validation error if none are found.
- **`tests/`**: Add `test_video_processor_path` to verify the new code path.

## Before / After

Before: crashes for vjepa2-style models with `AttributeError` -- pipeline always requires an image_processor.

After: works with either processor type; existing image_processor/feature_extractor models are unaffected.

## Tests

```
RUN_PIPELINE_TESTS=1 pytest tests/pipelines/test_pipelines_video_classification.py -k "test_small_model_pt or test_video_processor_path" -v
# 2 passed
```

--- Comment by Shashank-Tripathi-07 at 2026-05-20T16:02:47Z ---
The 8 CI failures are all in 	ests/models/cohere2_moe/ (tensor parallelism and training overfit tests), completely unrelated to this PR. Our changes only touch pipelines/base.py, pipelines/__init__.py, pipelines/video_classification.py, and the video classification test file. The cohere2_moe failures appear to be pre-existing flaky tests failing independently of this PR.

--- Comment by Shashank-Tripathi-07 at 2026-05-20T18:11:45Z ---
@J3r3myPerera I went through your review of the code and did the changes. Nice attempt reviewing my PR !! Currently waiting for a maintainer to review the PR. 🚀

--- Comment by github-actions[bot] at 2026-05-20T18:24:27Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46120&sha=661597

--- Comment by Shashank-Tripathi-07 at 2026-05-21T10:58:23Z ---
@Rocketknight1 brother, I didn't use any AI Agent to code this, I only used AI to find an issue to work on. The code was written by me. This is bad.

--- Comment by Rocketknight1 at 2026-05-21T11:01:44Z ---
```python
def _resolve_video_processor(
    video_processor, load_video_processor, model_name, config, task, hub_kwargs, model_kwargs
):
    """Resolve and optionally load the video processor for video-capable pipelines."""

    def load(video_processor):
        video_processor = _infer_pipeline_component(
            video_processor,
            model_name,
            config,
            "Impossible to guess which video processor to use. "
            "Please provide a BaseVideoProcessor class or a path/identifier to a pretrained video processor.",
        )

        if not isinstance(video_processor, (str, tuple)):
            return video_processor

        return AutoVideoProcessor.from_pretrained(video_processor, _from_pipeline=task, **hub_kwargs, **model_kwargs)

    return _load_pipeline_component(load_video_processor, video_processor, load)
```

There is no human in the world who writes code like this with perfectly-formatted docstrings for what should be a tiny fix. This patch is LLM written.

--- Comment by Shashank-Tripathi-07 at 2026-05-21T11:03:43Z ---
No, In my classes of best practices. I was taught to do docstrings even for the smallest things to allow developers to understand the code later on. I just followed it taking care that people who read the code later understand it. Should I leave docstrings just cause my patch is small (to look human) ?

--- Comment by Shashank-Tripathi-07 at 2026-05-21T11:06:22Z ---
Maybe, next time I'll read the repo practices more than this time to make sure my code matches what the other members have been doing. But this time writing my code as slop is just injustice.

--- Comment by Shashank-Tripathi-07 at 2026-05-21T12:40:56Z ---
@Rocketknight1 Fair point on the docstring, I removed it and collapsed the function signature to match how the rest of the file is written. The logic itself I did write by reading through how _resolve_image_processor works and following the same pattern.

--- Comment by J3r3myPerera at 2026-05-20T17:38:29Z ---
VIDEO_PROCESSOR_MAPPING is being imported here but not used anywhere. Suggest removing it.

--- Comment by J3r3myPerera at 2026-05-20T17:40:41Z ---
Missing annotation here. Since all other parameters here has a type annotation.
Suggest:
`video_processor: "BaseVideoProcessor | None" = None,`


PULL REQUEST #46118: Defensive: validate tar members in parakeet NeMo conversion (CWE-22 / Zip Slip)
State: closed  |  Merged: False
Author: JAE0Y2N  |  Base: main
Labels: Code agent slop
Created: 2026-05-20T11:02:07Z

While checking `src/transformers/models/parakeet/convert_nemo_to_hf.py` I noticed `unzip(zip_path, dest_dir)` at line 692 passes the .nemo archive through `tarfile.extractall(dest_dir)` without any per-entry path validation. Python's `tarfile.extractall` doesn't reject entries with `..` segments or absolute paths, so a malicious `.nemo` archive could write outside `dest_dir` — classic Zip Slip / CWE-22, but tar-flavored.

Threat model is defense-in-depth: the conversion script is meant to be run by users importing NeMo models, and the realistic-exploit precondition is a malicious `.nemo` distributed through whatever channel the user trusts. That's an upstream-trust event larger than this script. But the fix is one helper and matches Python's own [`PEP 706` data-filter](https://peps.python.org/pep-0706/) direction, so it's cheap defense-in-depth at the helper boundary.

## What the patch does

`_safe_extractall(tar, dest_dir)` runs before `extractall` and rejects any member whose normalized path would land outside `dest_dir`:

* Members with absolute paths or drive-prefixed paths are rejected outright.
* For every other member, `os.path.realpath(os.path.join(dest_real, member.name))` is compared against `os.path.realpath(dest_dir)` via `os.path.commonpath`. If the resolved entry is not under the destination, raise `ValueError`.
* Behavior for well-formed archives: unchanged.

This is the same hardening I added recently to `langgraph` zip-handling (#7873) — same pattern, tar instead of zip.

## Test plan

* Manual reproducer: build a 2-entry `.tar` with one valid file and one entry named `../escape.txt`, point `_safe_extractall` at a temp dir, confirm it raises `ValueError`. Done locally before sending this PR.
* `convert_nemo_to_hf.py` is unchanged for the well-formed NeMo archives that hf-hub-distributed models ship.

Happy to add a unit test under the right test module if you point me at one — I don't see existing unit tests for this conversion script.

--- Comment by github-actions[bot] at 2026-05-20T11:03:29Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: parakeet


PULL REQUEST #46117: [Generation] Fix continuous batching crash when out of memory during …
State: closed  |  Merged: False
Author: rohanbalsaraf  |  Base: main
Labels:
Created: 2026-05-20T11:01:54Z

…request fork

If a request is being forked but there are not enough cache blocks to hold the forked cache, the continuous batching server would crash. This makes `fork_blocks` atomic, introduces `can_fork_request` to check if enough memory is available, and safely falls back to creating a new pending request that pre-fills from scratch if memory is limited.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by Rocketknight1 at 2026-05-20T11:26:34Z ---
cc @remi-or @mcpatate

--- Comment by github-actions[bot] at 2026-05-20T11:49:44Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46117&sha=47081a

--- Comment by rohanbalsaraf at 2026-05-20T12:18:58Z ---
> A few comments to address and I will need to run the test, but rn this looks like a nice contribution inside a tricky part of the code. Impressive, thank you!

thank you but I will keep contributing to this project. rn I'm just getting started contributing to open source

--- Comment by remi-or at 2026-05-20T11:59:09Z ---
I don't think this line is necessary, you can just replace `forked_by_reference` with `parent_blocks` in the return statement

--- Comment by remi-or at 2026-05-20T12:03:50Z ---
"First" instead of "First phase" since we no longer have a second phase

--- Comment by remi-or at 2026-05-20T12:06:10Z ---
Still, better leave this for type checking purposes. Agreed it should not trigger.

--- Comment by remi-or at 2026-05-20T12:06:20Z ---
Maybe make it an error


[Fix] Stranded params

PULL REQUEST #46116: [Fix] Stranded params
State: open  |  Merged: False
Author: remi-or  |  Base: main
Labels:
Created: 2026-05-20T10:34:14Z

With the new distributed loading scheme introduced in #45028 some parameters like RoPE's inv_freq may stay on CPU after the model is loaded. This is because the device map is None so `_move_missing_keys_from_meta_to_device` moves those keys to the CPU as a default.
This is not great because 1. it means we have host-to-device transfers in `forward` and 2. this crashes CUDA graphs captures.
To fix this, in this context only (there is a device mesh but no device map) we use the current device as a fallback for those params.

This fixes TP2 in continuous batching.
Not sure I understand our model loading thoroughly so asking for a careful review here!

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T10:47:45Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46116). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-20T12:29:11Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46116&sha=9d3f64


Add new cohere2_moe model

PULL REQUEST #46115: Add new cohere2_moe model
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels: New model
Created: 2026-05-20T09:37:46Z

# What does this PR do?

As per the title!

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T09:51:26Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46115). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-20T10:15:32Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, cohere2, cohere2_moe, cohere2_vision

--- Comment by github-actions[bot] at 2026-05-20T10:46:22Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46115&sha=250645


PULL REQUEST #46114: [DON"T MERGE] Forked pr to test migration - check OTEL
State: open  |  Merged: False
Author: ydshieh  |  Base: main
Labels:
Created: 2026-05-20T09:31:17Z

# What does this PR do?

[DON"T MERGE] Forked pr to test migration - check OTEL

--- Comment by github-actions[bot] at 2026-05-20T09:32:25Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: vit

--- Comment by github-actions[bot] at 2026-05-20T09:36:14Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46114&sha=bca925

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T09:44:46Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46114). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #46113: clip_grad_norm avoid O(global_params) replication
State: open  |  Merged: False
Author: AmineDiro  |  Base: main
Labels:
Created: 2026-05-20T09:27:15Z

Followup to the https://github.com/huggingface/transformers/pull/45028 . I've been integrating the new native FSDP+TP  to my benchmark.

Current `clip_grad_norm` currently has two issues that compose:
-  it materializes a full replicated copy of every DTensor gradient via `_replicate_dtensor(g).to_local()` into a Python list, holding all of them live for the duration of the function. Per-rank scratch is **O(global_params) bytes** — independent of how sharded the model is. On Qwen3-30B-A3B at `tp_size=8, fsdp_size=2` this is a **~60 GB transient peak per rank**, OOMing 80 GB H100s regardless of context length, batch size, or optimizer.
- (BUG) The function passes `grads` (raw gradient tensors) to `torch.nn.utils.clip_grads_with_norm_`, but that API expects `parameters` and re-extracts `.grad` internally. Raw tensors have `p.grad = None`, so the inner list is empty and the function returns silently. **The reported `total_norm` is correct, but no gradient is ever actually scaled**: `max_norm` is probably ignored.

##  Mem Profile
`torch.cuda.memory._record_memory_history(enabled='all', context='all', stacks='all')` at job start (rank 0 only) — captures every allocation/free with full call stack (C++ + Python).

`torch.cuda.memory.memory_snapshot()` + `torch.cuda.memory_stats()` at 11 named lifecycle stages — pickled to disk per stage so the OOM can't lose the data:

```
S0_after_pg_init                # baseline
S1_after_from_pretrained        # post weight load (shard-on-read)
S2_after_batch_ready            # post tokenization
S3_step{0,1}_after_forward      # post forward pass
S4_step{0,1}_after_backward     # post backward
S5_step0_after_clip             # post clip_grad_norm (step 0)
S6_step0_after_optim_step       # post optimizer.step (Adam state alloc)
S7_step0_after_zero_grad        # post zero_grad
S99_OOM_at_step{N}              # captured inside the try/except on OOM
```
| stage                       | allocated | reserved                      | active blocks                   |
| --------------------------- | --------- | ----------------------------- | ------------------------------- |
| `S0_after_pg_init`          | 0.00 GB   | 0.00 GB                       | 0                               |
| `S1_after_from_pretrained`  | 5.17 GB   | 5.82 GB                       | 531                             |
| `S2_after_batch_ready`      | 5.17 GB   | 5.82 GB                       | 533                             |
| `S3_step0_after_forward`    | 8.13 GB   | 10.37 GB                      | 596                             |
| `S4_step0_after_backward`   | 18.27 GB  | 21.55 GB                      | 683                             |
| `S5_step0_after_clip`       | 18.27 GB  | **81.57 GB** ← reserved spike | 684                             |
| `S6_step0_after_optim_step` | 28.60 GB  | 81.77 GB                      | 2277                            |
| `S7_step0_after_zero_grad`  | 23.43 GB  | 81.77 GB                      | 2227                            |
| `S3_step1_after_forward`    | 18.49 GB  | 81.78 GB                      | 2191                            |
| `S4_step1_after_backward`   | 28.60 GB  | 81.78 GB                      | 2277                            |
| **`S99_OOM_at_step1`**      | 29.00 GB  | 82.07 GB                      | **peak_since_reset = 82.37 GB** |

`reserved` jumps from 21.55 GB (post-backward step 0) to 81.57 GB (post-clip step 0) — a 60 GB transient that the caching allocator never gives back to the driver. This is **exactly** the gradient-replicate scratch buffer `clip_grad_norm` builds, plus its caching residue. The OOM at step 1 occurs because the second invocation tries to allocate the same 60 GB of scratch on top of 28.6 GB of live tensors plus the ~5 GB FSDP staging + 7 GB autograd retained.
## Validation

- ✅ **Memory profile diff**: reserved at S5_step0 drops from 81.57 GB → 21.55 GB (–60 GB) after skipping the call. Stays stable through 5 steps.

## Who can review?
- distributed: @3outeille @ArthurZucker

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T09:38:50Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46113). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #46112: feat: Add support for JetBrains' `Mellum` v2 code generation model
State: open  |  Merged: False
Author: shadeMe  |  Base: main
Labels:
Created: 2026-05-20T09:24:48Z

# What does this PR do?

Adds support for a new model architecture - [Mellum](https://www.jetbrains.com/mellum/) v2 is an update to JetBrains' open-weights code generation model that is built on a Mixture-of-Experts architecture (Mellum v1 was based on Llama2).

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@ArthurZucker @Cyrilvallez

--- Comment by github-actions[bot] at 2026-05-20T09:25:59Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, mellum

--- Comment by github-actions[bot] at 2026-05-20T09:37:56Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46112&sha=d7bd64


PULL REQUEST #46111: Fix a regression in encoder-decoder generation cache initialization
State: open  |  Merged: False
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-20T09:13:10Z

For encoder-decoder models, `generate()` was passing the decoder config to both the self-attention cache and the cross-attention cache. This is incorrect for models like T5Gemma with decoder sliding-window layers: the cross-attention cache could inherit the decoder sliding-window structure and truncate encoder key/value states, causing FlashAttention generation to crash.

This PR keeps the decoder config for the self-attention cache, but initializes the cross-attention cache without the decoder config so cross-attention remains full-length.

Tested with:

```bash
pytest -q -rA tests/models/t5gemma/test_modeling_t5gemma.py::T5GemmaModelTest::test_generate_beyond_sliding_window_with_flash_attn
```
@vasqu @ArthurZucker @CyrilVallez, pls help review, thx!


PULL REQUEST #46110: [loading] Free up tensors faster inside ConversionOps
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-05-20T09:05:04Z

# What does this PR do?

As per the title. Mostly taken from https://github.com/huggingface/transformers/pull/44794, but I need it rn

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T09:17:30Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46110). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #46109: add more generic support for distributed trainer tests
State: open  |  Merged: False
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-20T08:29:30Z

@ydshieh pls help review, thx!


Fix deepspeed docker

PULL REQUEST #46108: Fix deepspeed docker
State: open  |  Merged: False
Author: SunMarc  |  Base: main
Labels:
Created: 2026-05-20T07:38:46Z

# What does this PR do?

This PR updates the deepspeed docker as it was installing an old version of nvtx which was part of the base image. SInce deepseed relies on a newer version if it is installed, this caused issues and the CI was failing.

`(line 239)  AttributeError: module 'nvtx' has no attribute 'get_domain'` cc @qgallouedec for viz

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T07:50:23Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46108). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


[fix] toctree

PULL REQUEST #46106: [fix] toctree
State: open  |  Merged: False
Author: stevhliu  |  Base: main
Labels:
Created: 2026-05-20T07:36:08Z

fixes misaligned toctree in **Models > Contribute > Multimodal models**. i don't think the doc-builder supports this many nested levels which is why it breaks

<img width="264" height="358" alt="Screenshot 2026-05-20 at 4 35 48 PM" src="https://github.com/user-attachments/assets/73377df9-a636-4b00-b94a-5497f6a54e97" />

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T07:50:20Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46106). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #46105: Allow `ydshieh2` for now for testing migration
State: closed  |  Merged: True
Author: ydshieh  |  Base: main
Labels:
Created: 2026-05-20T07:16:30Z

# What does this PR do?

Well well for testing purpose.

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T07:27:18Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46105). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


add inference mode for TP

PULL REQUEST #46104: add inference mode for TP
State: open  |  Merged: False
Author: 3outeille  |  Base: main
Labels:
Created: 2026-05-20T07:14:15Z

remove dtensor for TP inference. Rely on setting `model.eval` to trigger automatically the inference mode of TP

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T07:34:23Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46104). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-20T08:25:02Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46104&sha=76b591

--- Comment by 3outeille at 2026-05-20T08:00:55Z ---
to review

--- Comment by 3outeille at 2026-05-20T08:01:02Z ---
to review

--- Comment by 3outeille at 2026-05-20T08:01:10Z ---
to review


PULL REQUEST #46103: [Fix] Remove calls to unsupported functions in TP
State: closed  |  Merged: True
Author: remi-or  |  Base: main
Labels:
Created: 2026-05-20T07:13:39Z

In #45028 some code was introduced that is not compatible with torch < 2.9.1 . Since we support up to torch 2.4 this PR fixes two instances of this and also adds a TODO for check when instantiating a distributed model.

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T07:25:49Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46103). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


clean + fix fsdp tied weights

PULL REQUEST #46102: clean + fix fsdp tied weights
State: open  |  Merged: False
Author: 3outeille  |  Base: main
Labels:
Created: 2026-05-20T07:12:11Z

clean + fix fsdp tied weights

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T07:26:55Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46102). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Qwen2 audio parallel

PULL REQUEST #46101: Qwen2 audio parallel
State: open  |  Merged: False
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-20T07:07:35Z

This PR fixes the failed test case: `tests/models/qwen2_audio/test_modeling_qwen2_audio.py::Qwen2AudioForConditionalGenerationModelTest::test_model_parallel_beam_search`
@vasqu pls help review, thx!

--- Comment by github-actions[bot] at 2026-05-20T07:08:50Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen2_audio


[DO NOT merge!] test migration

PULL REQUEST #46100: [DO NOT merge!] test migration
State: open  |  Merged: False
Author: ydshieh2  |  Base: main
Labels:
Created: 2026-05-20T06:27:36Z

# What does this PR do?

[DO NOT merge!] test migration

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T06:43:26Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46100). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-20T09:20:46Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: vit

--- Comment by github-actions[bot] at 2026-05-20T09:25:46Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46100&sha=bca925


Sequence Parallel injection as a class

PULL REQUEST #46099: Sequence Parallel injection as a class
State: open  |  Merged: False
Author: 3outeille  |  Base: main
Labels:
Created: 2026-05-20T05:57:58Z

(no description)

--- Comment by github-actions[bot] at 2026-05-20T05:59:15Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: apertus, arcee, aria, cohere, cohere2, cwm, deepseek_v2, ernie4_5, eurobert, exaone4, gemma, gemma2, gemma3, glm, glm4, gpt_neox

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T06:10:29Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46099). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-20T06:12:23Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46099&sha=eb018b


PULL REQUEST #46098: Fix Mamba2Mixer: use_cache with seq_len > 1 silently produces incorrect results (both CPU and GPU paths) (#46032)
State: open  |  Merged: False
Author: sirzechs66  |  Base: main
Labels:
Created: 2026-05-20T05:22:33Z

## What does this PR do?

Fixes a silent-corruption bug when using `use_cache=True` with `seq_len > 1` in Mamba2 (chunked prefill). Previously the model wrongly assumed that a non‑null cache always implied single‑token decoding (`seq_len=1`). This PR implements proper state carry‑over for both convolution and SSM states, enabling correct chunked processing without crashes or numerical errors.

Key changes:
- **`cuda_kernels_forward`** – added a branch for `seq_len > 1` that uses `initial_states` and `return_final_states` in the fast CUDA kernels (`causal_conv1d_fn`, `mamba_chunk_scan_combined`).
- **`torch_forward`** – added a token‑by‑token fallback when `is_decoding` and `seq_len > 1` (CPU path).
- **Warning** – informs users that the fast path requires `causal-conv1d>=1.2.0` and `mamba-ssm>=2.0.0`.
- **Tests** – added `test_chunked_prefill_with_cache` to `tests/models/mamba2/test_modeling_mamba2.py` (covers GPU and CPU).
- **Documentation** – updated `docs/source/en/model_doc/mamba2.md` with a “Chunked generation for long sequences” section and added the three inference modes to the Notes.

Fixes #46032.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case). – Not applicable.
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case. – Issue #46032.
- [x] Did you make sure to update the documentation with your changes? – Yes, docstring and model doc.
- [x] Did you write any new necessary tests? – Yes, new test method in test_modeling_mamba2.py.

## Who can review?

@zucchini-nlp @Cyrilvallez (Mamba2 maintainers and model loading). Also tagging @ArthurZucker for general model architecture.

--- Comment by github-actions[bot] at 2026-05-20T05:23:45Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mamba2

--- Comment by sirzechs66 at 2026-05-20T05:25:04Z ---
@vasqu Please review these changes..
thank you

--- Comment by Rocketknight1 at 2026-05-20T11:13:36Z ---
There was an existing PR open for this at #46084

--- Comment by sirzechs66 at 2026-05-20T11:48:44Z ---
> There was an existing PR open for this at #46084

Yes I did not know when I started this that someone was actively working on it,

If you want you can approve this or we can close this.

All the tests and checks have been passed

--- Comment by sirzechs66 at 2026-05-20T18:30:42Z ---
@clara2911 please take a look at this as well


Fix torchao behavior for xpu

PULL REQUEST #46096: Fix torchao behavior for xpu
State: closed  |  Merged: True
Author: jiqing-feng  |  Base: main
Labels:
Created: 2026-05-20T04:48:23Z

This PR fixes torchao behavior on XPU.

It updates the torchao integration to use the current accelerator instead of assuming CUDA during offload, which prevents XPU runs from incorrectly moving modules to CUDA. It also aligns the torchao int4 XPU tests with the expected XPU configuration and output behavior, including B60-specific ground truth updates where needed.

Overall, this makes torchao XPU support more robust and keeps the test suite consistent with actual XPU execution results.

--- Comment by github-actions[bot] at 2026-05-20T06:05:20Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: torchao_integration

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T06:19:04Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46096). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by SunMarc at 2026-05-20T05:51:17Z ---
we can't do that as it is not supported by all version of torch that we support. guard it with is_torch_accelerator_available()

--- Comment by jiqing-feng at 2026-05-20T06:07:01Z ---
Fixed.


PULL REQUEST #46094: [docs] reasoning support for transformers serve
State: closed  |  Merged: True
Author: stevhliu  |  Base: main
Labels:
Created: 2026-05-20T04:32:18Z

adds docs for #45690

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T04:44:17Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46094). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by SunMarc at 2026-05-20T05:49:14Z ---
maybe we can add a warning to say that not all models supports reasoning and setting the flag will do nothing in that case


Clean up new models after release

PULL REQUEST #46092: Clean up new models after release
State: open  |  Merged: False
Author: zucchini-nlp  |  Base: main
Labels:
Created: 2026-05-20T03:00:54Z

# What does this PR do?

As per title, delete unused/dead code from modular. The models were shipped in a rush to add them day zero and we didn't have time to properly clean it up

--- Comment by github-actions[bot] at 2026-05-20T03:07:28Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: exaone4_5, minicpmv4_6

--- Comment by github-actions[bot] at 2026-05-20T03:14:12Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=46092&sha=5656eb

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T03:20:36Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46092). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by zucchini-nlp at 2026-05-20T03:54:59Z ---
run-slow: minicpmv4_6

--- Comment by github-actions[bot] at 2026-05-20T03:56:11Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26140261300)

This comment contains `run-slow`, running the specified jobs:

models: ["models/minicpmv4_6"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-20T04:20:41Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26140261300)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [90f48533](https://github.com/huggingface/transformers/commit/90f485334a6da0490ac6162191a7b90a96949b28) | workflow commit (merge commit) |
| PR | [5656ebfe](https://github.com/zucchini-nlp/transformers/commit/5656ebfe76173e5f97b807c606ba59e734656e60) | branch commit (from PR) |
| main | [ba06e3fb](https://github.com/huggingface/transformers/commit/ba06e3fbdf355c363ac067ebcda210017e90a852) | base commit (on `main`) |

### Model CI Report

❌ **[14 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/99a272bf70e94a524ffd42dab718f498a85c043d/2026-05-20/runs/34916-26140261300/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [minicpmv4_6](https://github.com/huggingface/transformers/actions/runs/26140261300/job/76884308520):
    tests/models/minicpmv4_6/test_modeling_minicpmv4_6.py::MiniCPMV4_6IntegrationTest::test_small_model_vision_generation_batch (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_modeling_minicpmv4_6.py::MiniCPMV4_6IntegrationTest::test_small_model_vision_generation_batch_mixed (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_apply_chat_template_audio_0 (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_apply_chat_template_audio_1 (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_apply_chat_template_audio_2 (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_apply_chat_template_audio_3 (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_apply_chat_template_decoded_video_0 (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_apply_chat_template_tool_calls_no_content (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_apply_chat_template_video_frame_sampling (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_model_input_names (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_processor_text_has_no_visual (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_processor_with_multiple_inputs (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_video_processing (✅ ⟹ ❌)
    tests/models/minicpmv4_6/test_processing_minicpmv4_6.py::MiniCPMV4_6ProcessorTest::test_video_processor_defaults (✅ ⟹ ❌)


PULL REQUEST #46091: Update tokenizer mappings to use TokenizersBackend for additional models
State: open  |  Merged: False
Author: itazap  |  Base: main
Labels:
Created: 2026-05-20T02:57:32Z

see the description / report here: https://github.com/huggingface/transformers/pull/45936

this PR is just the auto changes from the PR above. it's for models that don't have their own Tokenizer class so we don't have `test_tokenization_*.py` for these models.

fixes https://github.com/huggingface/transformers/issues/45920

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T03:09:41Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46091). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-20T07:32:36Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, auto

--- Comment by itazap at 2026-05-20T07:08:04Z ---
this test was changed a few months ago and im changing it back


skip deepgemm test except cuda

PULL REQUEST #46090: skip deepgemm test except cuda
State: open  |  Merged: False
Author: jiqing-feng  |  Base: main
Labels:
Created: 2026-05-20T02:30:23Z

Currently deepgemm only support cuda, so let's skip the tests except cuda.

--- Comment by github-actions[bot] at 2026-05-20T02:31:38Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: finegrained_fp8

--- Comment by jiqing-feng at 2026-05-21T01:59:57Z ---
Hi @ydshieh . Would you please review this PR? Thanks!


Test

PULL REQUEST #46089: Test
State: closed  |  Merged: False
Author: jiqing-feng  |  Base: test
Labels:
Created: 2026-05-20T02:29:41Z

Currently deepgemm only support cuda, so let's skip the tests except cuda.


Remove accidentally added `tmp.py`

PULL REQUEST #46087: Remove accidentally added `tmp.py`
State: closed  |  Merged: True
Author: hmellor  |  Base: main
Labels:
Created: 2026-05-20T01:16:38Z

Slipped through in https://github.com/huggingface/transformers/pull/45028

--- Comment by HuggingFaceDocBuilderDev at 2026-05-20T01:27:11Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46087). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


update xpu expectation for falcon mamba

PULL REQUEST #46086: update xpu expectation for falcon mamba
State: open  |  Merged: False
Author: sywangyi  |  Base: main
Labels:
Created: 2026-05-20T01:16:02Z

- Intel XPU: @IlyasMoutawwakil

--- Comment by github-actions[bot] at 2026-05-20T01:17:16Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: falcon_mamba


PULL REQUEST #46084: Fix use_cache with seq_len > 1 ( #46032)
State: open  |  Merged: False
Author: Ramshankar07  |  Base: main
Labels:
Created: 2026-05-19T20:01:34Z

# What does this PR do?

Add a third dispatch path to Mamba2Mixer for the case where a prior cached state exists and seq_len > 1 (chunked prefill / speculative decode verification).

Previously:
   - cuda_kernels_forward: .squeeze(1) crashed or corrupted tensors for seq_len > 1 when has_previous_state was True.
   - torch_forward: dt[:, 0, :] silently dropped all tokens after index 0, and previous_states was always zero-initialised ignoring the cache.

Both paths now:
   - Gate the single-step kernel on seq_len == 1.
   - For seq_len > 1, prepend the cached conv buffer (last K-1 entries) to the chunk input, run a full causal conv, drop the prepended prefix from the output, and pass the cached recurrent_state as initial_states  to mamba_chunk_scan_combined / as previous_states in the naive SSD.

This PR follows the same pattern applied to GDN-based models in #45513,  as discussed in the issues I added test_mamba2_chunked_prefill (CPU/torch path, always runs) and test_mamba2_chunked_prefill_cuda_path (skipped without mamba-ssm).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #46032

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #46032
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@zucchini-nlp @Cyrilvallez 🤗@vasqu @Clara2911
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by Ramshankar07 at 2026-05-19T20:02:38Z ---
I didn't run the cuda tests, can anybody test it?

--- Comment by github-actions[bot] at 2026-05-20T20:31:44Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mamba2

--- Comment by clara2911 at 2026-05-20T10:17:59Z ---
Thank you for this PR and your super quick work, it looks like a solid solution to the issue I described. I am going to leave the full review to the mamba experts e.g. @vasqu. However, one possible suggestion: was it a specific choice to manually prepend the history here instead of passing `initial_states`? I believe we could perhaps significantly simplify this manual `torch.cat([history, new_tokens])` + slicing the output `[:, -seq_len:, :]` by passing the history to `causal_conv1d_fn` directly through `initial_states` and `return_final_states` parameters. Similarly to how `mamba_chunk_scan_combined` handles it below

--- Comment by Ramshankar07 at 2026-05-20T16:54:00Z ---
I wrote the cpu path first since it always works, no special packages needed. The cpu path uses plain pytorch’s  nn.conv1d which has no shortcut, so  I have to manually glue the history onto the front then worked on the GPU  path, I just copied the same idea since it was already working. Thanks for pointing out, I'll re implement this

--- Comment by Ramshankar07 at 2026-05-20T18:56:46Z ---
Now I removed it and directly used causal_conv1d_fn. lmk if I did any redundant changes.


test

PULL REQUEST #46083: test
State: closed  |  Merged: False
Author: ZappD0S  |  Base: main
Labels: Code agent slop
Created: 2026-05-19T19:36:52Z

(no description)

--- Comment by Rocketknight1 at 2026-05-20T11:46:11Z ---
The fix and the PR description were both clearly written by an LLM. There's like 100 LOC changes for what feels like it should be a one-line fix, and the original issue was already closed, and there was discussion with another user there, which this PR just ignored. We aren't really able to review submissions like that in detail!