PULL REQUEST #45661: [Weight Converter] More fine-grained mappings on classes, scoping for every transforms (including weight converter)
State: closed  |  Merged: True
Author: yonigozlan  |  Base: main
Labels:
Created: 2026-04-27T19:03:34Z

# What does this PR do?

This PR aims to solve three different issues with the existing conversion mapping system.

- The registry only accepted `model_type` strings. This made it impossible to have different conversions for e.g. `DetrForSegmentation` vs `DetrForObjectDetection` both map to "detr", so the segmentation-specific mask-head and bbox-attention renames ended up polluting the shared entry and running on all DETR variants. The registry now accepts class names too, with class name taking priority over `model_type` during lookup.

- Sub-models transforms weren't scoped, and a child model mapping could leak into a parent model's: A `WeightRenaming` registered for a child `ViTModel` loaded with `AutoModel` would be applied to the full key space of whatever model contained it. `PrefixChange` had a partial fix (`with_submodel_prefix`) but plain renames were left global. This PR adds a `scope_prefix` field on `WeightTransform`: when set, `rename_source_key` strips the prefix before matching and re-attaches it after. All sub-module transforms get their `scope_prefix` set in `get_model_conversion_mapping`.

- `rename_source_key` previously applied all renamings first, then converters as a separate phase regardless of their registered order. The function now now takes a single `weight_transforms` list and processes it in order, so renames and converters naturally interleave as intended.

- Mappings with regex targeting the beginning of the string ("^") would be ignored in a child model, which is not what we want imo. This issue was hiding some other inconsistencies in the current mappings for VLMs (which still causes issue on the main branch, for example we get missing weights when loading llava weights in `LlavaModel` on the main branch). This is now fix in `conversion_mappings`, with class specific mappings to distinguish base model mappings and `ForConditionalGeneration` mappings in vlms.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T19:15:10Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45661). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by tarekziade at 2026-04-28T07:30:03Z ---
semi-related, I am refactoring a bit `WeightConverter` see https://github.com/huggingface/transformers/pull/45635

--- Comment by yonigozlan at 2026-04-28T21:25:38Z ---
Cc @zucchini-nlp for viz, this PR allows to remove a lot of test_reverse_mapping overrides, and also fixes loading/saving weights with all the base VLM models (e.g. LlavaModel, currently broken on main)

--- Comment by yonigozlan at 2026-04-28T21:29:25Z ---
run-slow: aya_vision, colpali, colqwen2, conditional_detr, detr, emu3, fuyu, gemma3, got_ocr2, internvl, llava, llava_next, llava_next_video, llava_onevision, maskformer, mistral3, mllama, paligemma, qwen2_5_vl, qwen2_vl, shieldgemma2, video_llava, vipllava, pp_chart2table

--- Comment by github-actions[bot] at 2026-04-28T21:30:47Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25078679549)

This comment contains `run-slow`, running the specified jobs:

models: ["models/aya_vision", "models/colpali", "models/colqwen2", "models/conditional_detr", "models/detr", "models/emu3", "models/fuyu", "models/gemma3", "models/got_ocr2", "models/internvl", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/maskformer", "models/mistral3", "models/mllama", "models/paligemma", "models/pp_chart2table", "models/qwen2_5_vl", "models/qwen2_vl", "models/shieldgemma2", "models/video_llava", "models/vipllava"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-28T23:19:51Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25078679549)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [c54ce934](https://github.com/huggingface/transformers/commit/c54ce934c2a8850f450c1807dfe0f5c63fa26c5e) | workflow commit (merge commit) |
| PR | [f0eeb8f2](https://github.com/yonigozlan/transformers/commit/f0eeb8f2872a8ebd598064d1ddaf16abf4146bb8) | branch commit (from PR) |
| main | [a374d990](https://github.com/huggingface/transformers/commit/a374d990d422357a854ff7f4f5f1cf9f88308ade) | base commit (on `main`) |

⚠ **Model CI failed to report results**

The test failure analysis could not be completed. Please check the [workflow run](https://github.com/huggingface/transformers/actions/runs/25078679549) for details.

--- Comment by yonigozlan at 2026-04-28T23:58:19Z ---
run-slow: aya_vision, colpali, colqwen2, conditional_detr, detr, emu3, fuyu, gemma3, got_ocr2, internvl, llava, llava_next, llava_next_video, llava_onevision, maskformer, mistral3, mllama, paligemma, qwen2_5_vl, qwen2_vl, shieldgemma2, video_llava, vipllava, pp_chart2table

--- Comment by github-actions[bot] at 2026-04-28T23:59:29Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25083889869)

This comment contains `run-slow`, running the specified jobs:

models: ["models/aya_vision", "models/colpali", "models/colqwen2", "models/conditional_detr", "models/detr", "models/emu3", "models/fuyu", "models/gemma3", "models/got_ocr2", "models/internvl", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/maskformer", "models/mistral3", "models/mllama", "models/paligemma", "models/pp_chart2table", "models/qwen2_5_vl", "models/qwen2_vl", "models/shieldgemma2", "models/video_llava", "models/vipllava"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-29T01:44:06Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25083889869)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [c54ce934](https://github.com/huggingface/transformers/commit/c54ce934c2a8850f450c1807dfe0f5c63fa26c5e) | workflow commit (merge commit) |
| PR | [f0eeb8f2](https://github.com/yonigozlan/transformers/commit/f0eeb8f2872a8ebd598064d1ddaf16abf4146bb8) | branch commit (from PR) |
| main | [a374d990](https://github.com/huggingface/transformers/commit/a374d990d422357a854ff7f4f5f1cf9f88308ade) | base commit (on `main`) |

⚠ **Model CI failed to report results**

The test failure analysis could not be completed. Please check the [workflow run](https://github.com/huggingface/transformers/actions/runs/25083889869) for details.

--- Comment by zucchini-nlp at 2026-04-29T09:34:32Z ---
Yep, exactly what I mentioned previously, I like it more with class-based options + more freedom with nested models ❤

--- Comment by yonigozlan at 2026-04-29T15:13:18Z ---
run-slow: aya_vision, colpali, colqwen2, conditional_detr, detr, emu3, fuyu, gemma3, got_ocr2, internvl, llava, llava_next, llava_next_video, llava_onevision, maskformer, mistral3, mllama, paligemma, qwen2_5_vl, qwen2_vl, shieldgemma2, video_llava, vipllava, pp_chart2table

--- Comment by github-actions[bot] at 2026-04-29T15:14:51Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25117218352)

This comment contains `run-slow`, running the specified jobs:

models: ["models/aya_vision", "models/colpali", "models/colqwen2", "models/conditional_detr", "models/detr", "models/emu3", "models/fuyu", "models/gemma3", "models/got_ocr2", "models/internvl", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/maskformer", "models/mistral3", "models/mllama", "models/paligemma", "models/pp_chart2table", "models/qwen2_5_vl", "models/qwen2_vl", "models/shieldgemma2", "models/video_llava", "models/vipllava"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-29T16:42:35Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25117218352)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [7b9766eb](https://github.com/huggingface/transformers/commit/7b9766eb99025894008a41b34dde929f078c2f99) | workflow commit (merge commit) |
| PR | [64c5bc58](https://github.com/yonigozlan/transformers/commit/64c5bc584dcb2bcc796933a5437fdf49a8eecd94) | branch commit (from PR) |
| main | [a8f43eca](https://github.com/huggingface/transformers/commit/a8f43eca15b8d1c63deb33f6b97dfab30419e5da) | base commit (on `main`) |

### Model CI Report

❌ **[29 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/5cdc7fb7e67e2fd18709012633be82d10496e72a/2026-04-29/runs/33934-25117218352/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [aya_vision](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565500):
    tests/models/aya_vision/test_modeling_aya_vision.py::AyaVisionModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [emu3](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565526):
    tests/models/emu3/test_modeling_emu3.py::Emu3Vision2TextModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)
    tests/models/emu3/test_modeling_emu3.py::Emu3IntegrationTest::test_model_generation_multi_image (❌ ⟹ ❌)

- [fuyu](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565503):
    tests/models/fuyu/test_modeling_fuyu.py::FuyuModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [gemma3](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565433):
    tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [got_ocr2](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565493):
    tests/models/got_ocr2/test_modeling_got_ocr2.py::GotOcr2ModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [internvl](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565474):
    tests/models/internvl/test_modeling_internvl.py::InternVLModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [llava](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565475):
    tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)
    tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_pixtral (❌ ⟹ ❌)

- [llava_next](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565540):
    tests/models/llava_next/test_modeling_llava_next.py::LlavaNextForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [llava_next_video](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565640):
    tests/models/llava_next_video/test_modeling_llava_next_video.py::LlavaNextVideoForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [llava_onevision](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565494):
    tests/models/llava_onevision/test_modeling_llava_onevision.py::LlavaOnevisionForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [maskformer](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565502):
    tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_instance_segmentation_head (✅ ⟹ ❌)
    tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_instance_segmentation_head_resnet_backbone (✅ ⟹ ❌)

- [mistral3](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565681):
    tests/models/mistral3/test_modeling_mistral3.py::Mistral3ModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [mllama](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565539):
    tests/models/mllama/test_modeling_mllama.py::MllamaForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [paligemma](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565477):
    tests/models/paligemma/test_modeling_paligemma.py::PaliGemmaForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [qwen2_5_vl](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565518):
    tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_bc_torch_dtype (✅ ⟹ ❌)
    tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_can_use_safetensors (✅ ⟹ ❌)
    tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_load_save_without_tied_weights (✅ ⟹ ❌)
    tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)
    tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_save_load (✅ ⟹ ❌)

- [qwen2_vl](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565610):
    tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_bc_torch_dtype (✅ ⟹ ❌)
    tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_can_use_safetensors (✅ ⟹ ❌)
    tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_load_save_without_tied_weights (✅ ⟹ ❌)
    tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)
    tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_save_load (✅ ⟹ ❌)

- [video_llava](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565496):
    tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

- [vipllava](https://github.com/huggingface/transformers/actions/runs/25117218352/job/73608565584):
    tests/models/vipllava/test_modeling_vipllava.py::VipLlavaForConditionalGenerationModelTest::test_reverse_loading_mapping (✅ ⟹ ❌)

--- Comment by yonigozlan at 2026-04-29T21:47:32Z ---
run-slow: aya_vision, colpali, colqwen2, conditional_detr, detr, emu3, fuyu, gemma3, got_ocr2, internvl, llava, llava_next, llava_next_video, llava_onevision, maskformer, mistral3, mllama, paligemma, qwen2_5_vl, qwen2_vl, shieldgemma2, video_llava, vipllava, pp_chart2table

--- Comment by github-actions[bot] at 2026-04-29T21:49:17Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25135625128)

This comment contains `run-slow`, running the specified jobs:

models: ["models/aya_vision", "models/colpali", "models/colqwen2", "models/conditional_detr", "models/detr", "models/emu3", "models/fuyu", "models/gemma3", "models/got_ocr2", "models/internvl", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision", "models/maskformer", "models/mistral3", "models/mllama", "models/paligemma", "models/pp_chart2table", "models/qwen2_5_vl", "models/qwen2_vl", "models/shieldgemma2", "models/video_llava", "models/vipllava"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-29T23:10:07Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25135625128)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [9c29db6c](https://github.com/huggingface/transformers/commit/9c29db6c37c25a74cf814f8a80e0ce5ca1371a38) | workflow commit (merge commit) |
| PR | [9da04fc0](https://github.com/yonigozlan/transformers/commit/9da04fc06b9d2a0f0a5642bbbdc392db80eac615) | branch commit (from PR) |
| main | [a8f43eca](https://github.com/huggingface/transformers/commit/a8f43eca15b8d1c63deb33f6b97dfab30419e5da) | base commit (on `main`) |

### Model CI Report

❌ **[2 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/05e779880c4ce5435eaa5625ae2efb9ca291d877/2026-04-29/runs/33944-25135625128/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [emu3](https://github.com/huggingface/transformers/actions/runs/25135625128/job/73674033461):
    tests/models/emu3/test_modeling_emu3.py::Emu3IntegrationTest::test_model_generation_multi_image (❌ ⟹ ❌)

- [llava](https://github.com/huggingface/transformers/actions/runs/25135625128/job/73674033476):
    tests/models/llava/test_modeling_llava.py::LlavaForConditionalGenerationIntegrationTest::test_pixtral (❌ ⟹ ❌)

--- Comment by yonigozlan at 2026-05-01T15:07:01Z ---
Both emu3 and llava/pixtral tests are failing with OutOfMemory errors, which they already were I think, so not sure why they show up here

--- Comment by github-actions[bot] at 2026-05-07T00:47:53Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aya_vision, colpali, colqwen2, emu3, fuyu, gemma3, got_ocr2, internvl, llava, llava_next, llava_next_video, llava_onevision, mistral3, mllama, paligemma, qwen2_5_vl

--- Comment by ArthurZucker at 2026-04-28T02:11:04Z ---
this is something we use a lot across the repo, let's store it once at init time please! 🤗

--- Comment by ArthurZucker at 2026-04-28T02:13:16Z ---
I think vllm neeeded a separation from both

--- Comment by ArthurZucker at 2026-04-28T02:14:48Z ---
this is extremely important, can you give an example in the doc. This is a departure from what we had before and it was kinda clearer

--- Comment by Cyrilvallez at 2026-04-28T08:51:26Z ---
Why remove the prefix when using submodels registry? For now it was only added for `PrefixChange`, but logically it would make sense for all Transforms

--- Comment by Cyrilvallez at 2026-04-28T08:55:00Z ---
Ha yes ok, see my previous comment as well. I think it was cleaner with the `with_submodel_prefix` (that we can maybe rename to `from_submodel_prefix`?), instead of adding the attribute like this. I was already thinking about adding it for all Transforms before

--- Comment by Cyrilvallez at 2026-04-28T08:56:37Z ---
See previous comment, I think rebuilding the Transform directly with the prefix is cleaner

--- Comment by Cyrilvallez at 2026-04-28T09:02:45Z ---
WeightRenaming and WeightConverter are and should be independent from each other. If WeightConverter matches, it is responsible for the Renaming as well. So any following WeightRenaming in the list should NOT match
If you have a counter-example, happy to see it, but it probably means the conversion is ill-written I believe, or the logic collapses somewhere

--- Comment by yonigozlan at 2026-04-28T18:03:26Z ---
I don't think we can really enforce this independence for nested models.
For nested models, you can end up with a root rename and a scoped converter chained on the same key. E.g. a composite model that maps `old_prefix → model.vlm` at the root, and whose `model.vlm` sub-module has a `q_proj → qkv_proj` converter:
```python
# load:  old_prefix.q_proj  →[rename]→  model.vlm.q_proj  →[converter]→  model.vlm.qkv_proj
# save (list inverted):
#        model.vlm.q_proj  →[rev converter]→  model.vlm.qkv_proj  →[rev rename]→  old_prefix.qkv_proj
```
Any fixed two-phase ordering breaks one direction: "all renames first" works on load but on save the reversed rename fires before the reversed converter sees `model.vlm.*`, so the converter misses. "All converters first" is the mirror problem. Interleaved list order is the only way both directions are correct.

Btw I have pretty much this exact scenario in the [RF DETR PR](https://github.com/huggingface/transformers/pull/36895)

--- Comment by yonigozlan at 2026-04-28T18:22:26Z ---
It adds a lot of complexity though, as adding a prefix to a pattern can break the pattern (e.g. if we match only the start of a word with ^ in a child source pattern, adding a prefix breaks the pattern).
Removing the prefix before the search is more robust imo

--- Comment by yonigozlan at 2026-04-28T18:23:40Z ---
I don't see any call to rename_source_key in vllm? not sure if I understood the comment

--- Comment by yonigozlan at 2026-04-28T19:49:07Z ---
Done!

--- Comment by ArthurZucker at 2026-05-06T14:01:54Z ---
we don't need the double quotes I think!

--- Comment by ArthurZucker at 2026-05-06T14:08:04Z ---
that's nice!

--- Comment by ArthurZucker at 2026-05-06T14:10:28Z ---
Do you know in which cases this happens?

--- Comment by ArthurZucker at 2026-05-06T14:19:18Z ---
I don't know if we need this, again because you could have 2 class names under the same model right?

--- Comment by ArthurZucker at 2026-05-06T14:21:28Z ---
sorry MB I though they were no longer separated

--- Comment by ArthurZucker at 2026-05-06T14:21:43Z ---
let's remove all double quotes, we don't use it in general

--- Comment by ArthurZucker at 2026-05-06T14:22:51Z ---
as this is kinda big change why do we need this vs before? is this specifically for the nested?

--- Comment by ArthurZucker at 2026-05-06T14:23:13Z ---
I suppose we cannot error out? But we would want to avoid multiple matches (as we tried to avoid renaming say block->mlp and then doing stuff on mlp instead of block?

--- Comment by ArthurZucker at 2026-05-06T14:26:36Z ---
okay that's very good, and needs to be in the doc! 🤗 we did overlook a bit the saving part so I am not surprised!

--- Comment by ArthurZucker at 2026-05-06T14:31:03Z ---
okay one thing I don't understand is why we need scoped prefix here?
In a way by default `"^old_q"` in regex fashion should theoretically only match stuff that actually starts with `old_q`.

Were we ignoring `^` before ? 🤗

--- Comment by ArthurZucker at 2026-05-06T14:32:38Z ---
Mmm sorry this whole comment is a bit weird / i don't understand should use proper target / source please

--- Comment by yonigozlan at 2026-05-06T16:17:09Z ---
Redundant indeed! Removed it

--- Comment by yonigozlan at 2026-05-06T16:54:41Z ---
There was an issue here indeed, if we had models of the same type or the same class at the same nesting levels. I added a fix to now skip a module only if an ancestor (not just any previously-seen module) has already claimed the same identifier, so two "siblings" each get their own scoped transforms.
Also added tests for this

--- Comment by yonigozlan at 2026-05-06T18:01:41Z ---
Yes, I added an example in the docstring below, I think you saw my previous answer to Cyril but this is an example:
https://github.com/huggingface/transformers/pull/45661#discussion_r3156232998

--- Comment by yonigozlan at 2026-05-06T18:59:57Z ---
Here the `scoped_rename` was aimed at a submodule which prefix is "encoder.attn", so the source pattern (that would be in the mapping of the submodule with prefix "encoder.attn") "^old_q" should only match stuff that actually starts with old_q once we remove the "encoder.attn" prefix (so "encoder.attn.old_q.weight" only, and not "old_q.weight").
Agreed that this was confusing because a submodule with prefix "encoder.attn" doesn't really make sense, so I changed and extended the test which should hopefully make it clearer and more comprehensive

--- Comment by yonigozlan at 2026-05-06T19:17:43Z ---
Improved the comment and made the test closer to a plausible real situation where the "encoder" is a separate submodule (with its own conversion mapping)

--- Comment by yonigozlan at 2026-05-06T19:30:57Z ---
Yes this only avoids multiple weight converters on one key, but we can still have renames and one weight converters chained (as in `test_interleaved_renaming_and_converter_round_trip`).
Agreed this `continue` is a little weird and might hide ill-defined conversion mapping, so maybe it's best to error out. Maybe better to fix in another PR though as this one is getting big, and this is also the current behavior on main.


[docs] cpu offloading

PULL REQUEST #45660: [docs] cpu offloading
State: closed  |  Merged: True
Author: stevhliu  |  Base: main
Labels:
Created: 2026-04-27T17:49:23Z

adds docs for #45184

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T18:00:28Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45660). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


[docs] dtype

PULL REQUEST #45659: [docs] dtype
State: closed  |  Merged: True
Author: stevhliu  |  Base: main
Labels:
Created: 2026-04-27T17:35:43Z

clarifies the dtype docs a bit to no longer needing to pass `dtype="auto"` (related to #45608)

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T17:47:44Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45659). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by zucchini-nlp at 2026-04-28T10:24:46Z ---
should we also delete `dtype=auto` here?


PULL REQUEST #45658: Fix `NameError: PeftConfigLike` triggered by `PreTrainedModel.__init_subclass__`
State: closed  |  Merged: True
Author: qgallouedec  |  Base: main
Labels:
Created: 2026-04-27T17:14:25Z

https://github.com/huggingface/transformers/pull/45425 added a class-level `peft_config` annotation in `integrations/peft.py`:

https://github.com/huggingface/transformers/blob/63378d94998c540997fcf818078c79437b04888d/src/transformers/integrations/peft.py#L407-L428

`PeftConfigLike` is imported only under `TYPE_CHECKING`, so it does not exist in the module's runtime namespace.

`PreTrainedModel` inherits from `PeftAdapterMixin`, and `modeling_utils.py:1296` runs `get_type_hints(cls)` inside `__init_subclass__`.

https://github.com/huggingface/transformers/blob/63378d94998c540997fcf818078c79437b04888d/src/transformers/modeling_utils.py#L1285-L1297

 `get_type_hints` walks the MRO and eagerly evaluates every forward reference in each base's module globals, so the unresolved `"PeftConfigLike"` raises `NameError` the moment any `PreTrainedModel` subclass is defined.

It's an import-time failure. It fires while `transformers.modeling_utils` itself is being imported (first triggered by `PreTrainedAudioTokenizerBase`), so any downstream library that imports a model (e.g. `trl`'s `SFTTrainer`) fails at collection

### Fix

Move `PeftConfigLike` out of the `TYPE_CHECKING` block so it's available at runtime, and drop the now-unneeded forward-reference quotes on the annotation.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T17:25:32Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45658). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45655: Fix the order of `cls.config` resolution
State: closed  |  Merged: False
Author: zucchini-nlp  |  Base: main
Labels:
Created: 2026-04-27T09:41:01Z

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36683 and allows loading VLM ckpt into an llm class in python 3.14, by resolving to the correct `cls.config_class`

--- Comment by zucchini-nlp at 2026-04-27T09:41:29Z ---
I dont know who owns this part now, so prob @Rocketknight1 if you could rveiew

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T09:52:15Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45655). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-27T09:56:53Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45655&sha=ee7174


PULL REQUEST #45654: [CB] Refactor any model-related code in a separate class
State: closed  |  Merged: True
Author: remi-or  |  Base: main
Labels:
Created: 2026-04-27T09:23:34Z

# Summary

This PR simplifies the continuous batching code by refactoring away any model-related code in the `ModelRunner` class. This is because the `ContinuousBatchingProcessor` class is starting to get too big and this will help keep things organized.
Draft until https://github.com/huggingface/transformers/pull/45653 is merged.

## Performance

| label                |   samples |   avg_in |   max_new |   time (s) |   tokens |    tok/s |   mem (GB) |
|----------------------|-----------|----------|-----------|------------|----------|----------|------------|
| gsm8k_default        |      1319 |       94 |       256 |      27.36 |   337664 | 12341    |      72.57 |
| gsm8k_sampling       |      1319 |       94 |       256 |      28.32 |   337664 | 11922.8  |      71.95 |
| gsm8k_compile        |      1319 |       94 |       256 |      27.07 |   337664 | 12472.6  |      72.01 |
| gsm8k_no_fast_decode |      1319 |       94 |       256 |      30.65 |   337664 | 11015.3  |      71.95 |
| rollouts_1024        |        32 |      256 |      1024 |       9.97 |    32768 |  3286.04 |      72.01 |
| rollouts_2048        |        32 |      256 |      2048 |      20.95 |    65536 |  3128.95 |      71.98 |
| rollouts_4096        |        32 |      256 |      4096 |      46.97 |   131072 |  2790.29 |      72.01 |
| rollouts_8192        |        32 |      256 |      8192 |     115.66 |   262144 |  2266.54 |      71.97 |
| rollouts_16384       |        32 |      256 |     16384 |     351.32 |   524288 |  1492.34 |      72    |
| few_blocks           |        20 |      256 |       256 |       7.29 |     5120 |   702.41 |      16.88 |
| multi_return_seq     |        50 |      256 |       256 |       8.37 |    12800 |  1528.67 |      72    |

Perf is on par with main, which is to be expected.

## Tests

All good

--- Comment by HuggingFaceDocBuilderDev at 2026-04-28T02:49:41Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45654). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45653: [CB] Better overall script and decode bucketting
State: closed  |  Merged: True
Author: remi-or  |  Base: main
Labels:
Created: 2026-04-27T07:40:48Z

# Summary

This PR updates the `continuous_batching_overall` script to better reflect the way users use continuous batching (ie. using simpler or the default config) on realistic tasks (GSM8K or RL rollouts of various lengths).
It also changes the way decode-only batches are bucketed: instead of using linear intervals, these batches are now bucketed using powers of 2. This led to better latencies.
Decode batches are also automatically turned on unless the user explicitly disables them, using workload hints (ie. max sequence length)

## Performances

New script, before PR:

| label                |   samples |   avg_in |   max_new |   time (s) |   tokens |    tok/s |   mem (GB) |
|----------------------|-----------|----------|-----------|------------|----------|----------|------------|
| gsm8k_default        |      1319 |       94 |       256 |      30.34 |   337664 | 11128    |      72.53 |
| gsm8k_compile        |      1319 |       94 |       256 |      30.35 |   337664 | 11126.1  |      71.85 |
| gsm8k_no_fast_decode |      1319 |       94 |       256 |      30.38 |   337664 | 11114    |      71.88 |
| rollouts_1024        |        32 |      256 |      1024 |      14.87 |    32768 |  2204.13 |      71.85 |
| rollouts_2048        |        32 |      256 |      2048 |      33.6  |    65536 |  1950.19 |      71.88 |
| rollouts_4096        |        32 |      256 |      4096 |      83.62 |   131072 |  1567.45 |      71.94 |
| rollouts_8192        |        32 |      256 |      8192 |     235.02 |   262144 |  1115.42 |      71.87 |
| rollouts_16384       |        32 |      256 |     16384 |     785.44 |   524288 |   667.51 |      71.84 |
| few_blocks           |        20 |      256 |       256 |       8.26 |     5120 |   619.85 |      16.82 |
| multi_return_seq     |        50 |      256 |       256 |      12.22 |    12800 |  1047.16 |      71.93 |

New script, after PR:

| label                |   samples |   avg_in |   max_new |   time (s) |   tokens |    tok/s |   mem (GB) |
|----------------------|-----------|----------|-----------|------------|----------|----------|------------|
| gsm8k_default        |      1319 |       94 |       256 |      27.16 |   337664 | 12432.9  |      72.53 |
| gsm8k_sampling       |      1319 |       94 |       256 |      27.97 |   337664 | 12072.9  |      71.08 |
| gsm8k_compile        |      1319 |       94 |       256 |      26.71 |   337664 | 12640.2  |      70.72 |
| gsm8k_no_fast_decode |      1319 |       94 |       256 |      26.86 |   337664 | 12571.7  |      70.01 |
| rollouts_1024        |        32 |      256 |      1024 |       9.98 |    32768 |  3282.96 |      69.3  |
| rollouts_2048        |        32 |      256 |      2048 |      20.93 |    65536 |  3131.43 |      68.59 |
| rollouts_4096        |        32 |      256 |      4096 |      46.96 |   131072 |  2791.37 |      67.88 |
| rollouts_8192        |        32 |      256 |      8192 |     115.64 |   262144 |  2266.85 |      67.17 |
| rollouts_16384       |        32 |      256 |     16384 |     369.07 |   524288 |  1420.55 |      66.49 |
| few_blocks           |        20 |      256 |       256 |       7.32 |     5120 |   699.1  |      16.88 |
| multi_return_seq     |        50 |      256 |       256 |       8.25 |    12800 |  1551.81 |      65.68 |

This is mostly because of the auto-activation of the decode path IMO.

## Tests

- [x] `RUN_SLOW=1 pytest tests/generation/test_continuous_batching.py`
- [x] `RUN_SLOW=1 pytest tests/cli/test_serve.py`
- [x] `RUN_SLOW=1 pytest tests/generation/test_paged_attention.py`

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T07:59:00Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45653). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ArthurZucker at 2026-04-29T08:34:54Z ---
why not use device map auto?

--- Comment by ArthurZucker at 2026-04-29T11:04:38Z ---
we can use typedDict here explicits which keys are accepted

--- Comment by remi-or at 2026-04-30T06:20:12Z ---
Good point, done

--- Comment by remi-or at 2026-04-30T06:33:33Z ---
Good point. Made a @dataclass so it's even more formal / easily scaled.


Fix colmodernvbert tests

PULL REQUEST #45652: Fix colmodernvbert tests
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-04-27T04:25:34Z

# What does this PR do?

Now that modernbert uses the right way to get the attention since https://github.com/huggingface/transformers/pull/45598, it supports `_can_set_attn_implementation` and in turn flex attention. Flex attention does not support dropout, nor head_dim<16.

--- Comment by github-actions[bot] at 2026-04-27T04:26:41Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: colmodernvbert

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T04:35:38Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45652). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45651: [Trainer] Optimize LengthGroupedSampler computation with select_columns and tqdm
State: closed  |  Merged: False
Author: AlrIsmail  |  Base: main
Labels:
Created: 2026-04-26T14:26:22Z

### What does this PR do?
This PR addresses the performance bottleneck reported in #28069, where `LengthGroupedSampler` (and `DistributedLengthGroupedSampler`) can take an extremely long time to compute sequence lengths for large datasets.

The root cause was that iterating over a Hugging Face `Dataset` without column selection forces the backend (Apache Arrow) to deserialize every column for every row, even if only one column (e.g., `input_ids`) is needed for length computation.

### Changes:
1. **Column Selection:** Added a check for `dataset.select_columns`. If present, we now temporarily select only the required column for the length-calculation loop, drastically reducing I/O and deserialization overhead.
2. **Progress Feedback:** Integrated `logging.tqdm` to provide a progress bar during this phase, which is otherwise silent.
3. **User Guidance:** Added a warning for datasets larger than 50,000 samples, advising users to provide pre-computed lengths via `length_column_name` for maximum efficiency.
4. **Refactoring:** Consolidated the logic into a shared internal helper `_compute_dataset_lengths` to ensure consistency across both standard and distributed samplers.

### Benchmark Results (5 Million Rows):
*   **Old Logic:** 410.17 seconds
*   **New Logic:** 128.84 seconds
*   **Accuracy:** Verified identical output.
*   **Speedup:** **3.18x** on a simple dataset (The speedup increases exponentially to 10x-20x on datasets with many heavy columns).

## Code Agent Policy
- [x] I confirm that this is not a pure code agent PR. I have reviewed every line of the change and manually verified the performance gains on a 5M row local test using the authentic `datasets` library.

I used AI assistance to help diagnose the Apache Arrow bottleneck and refactor the code for better modularity.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)?
- [x] Did you write any new necessary tests? (Verified via local benchmark scripts).

## Who can review?
@SunMarc

--- Comment by SunMarc at 2026-05-19T07:49:16Z ---
The original issue didn't have much traction so i prefer not to add it


PULL REQUEST #45650: Fix KeyError for flash_attn in import_utils.py on Python 3.13Fix
State: closed  |  Merged: False
Author: aryanp2107  |  Base: main
Labels:
Created: 2026-04-25T22:08:51Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #45520

When `flash_attn` is not installed, direct dictionary access on
`PACKAGE_DISTRIBUTION_MAPPING["flash_attn"]` throws a `KeyError`
because the key doesn't exist in the dictionary returned by
`importlib.metadata.packages_distributions()`. This affects Python 3.13
environments where Flash Attention is not available.

Replaced direct dictionary access with `.get()` and an empty list default
on four lines (951, 970, 982, 993), so the list comprehension iterates
over an empty list instead of crashing. This is consistent with the safe
access pattern already used in `_is_package_available()` at line 59.

- [ x ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ x ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@ivarflakstad (listed for AMD ROCm, relevant since the issue was reported on ROCm)

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by github-actions[bot] at 2026-04-25T22:23:11Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45650&sha=c0b6ec

--- Comment by Rocketknight1 at 2026-04-27T11:33:07Z ---
There was already an existing PR, please check first before firing your code agent at issues! https://github.com/huggingface/transformers/pull/45524


PULL REQUEST #45649: Fix OOM regression for FSDP2 + cpu_ram_efficient_loading on large models
State: open  |  Merged: False
Author: AmineDiro  |  Base: main
Labels:
Created: 2026-04-25T21:12:04Z

# What does this PR do?

PR #45050 replaces `torch.empty_like` with `torch.zeros_like` in `_move_missing_keys_from_meta_to_device`. While this fixes a real issue (NaN garbage in uninitialized memory), it forces a physical-memory commit of the entire model on every non-rank-0 FSDP rank.
With 8 ranks per node loading a 30B model, peak cpu mem jumps from ~60 GB to ~480 GB :/

The regression was identified by bisecting transformers commits between 2026-04-10 (working) and 2026-04-22 (failing) using a 2-node FSDP2 control config:

| Commit                       | Date                     | Result   | MFU    |
| ---------------------------- | ------------------------ | -------- | ------ |
| c43f15c71a                   | 2026-04-10               | PASS     | 21.46% |
| `a001f34439` (pre-#45050)    | 2026-04-13               | **PASS** | 21.13% |
| **`ff49f7c4cb` (PR #45050)** | **2026-04-13**           | **FAIL** | OOM    |
| e40b0c0195                   | 2026-04-13 (post-#45050) | FAIL     | OOM    |
| 8426e7e63d                   | 2026-04-15               | FAIL     | OOM    |
| 7a0d582ad4                   | 2026-04-20               | FAIL     | OOM    |
| 9dff7ca5c9                   | 2026-04-21               | FAIL     | OOM    |
| cbe7a02878                   | 2026-04-22               | FAIL     | OOM    |

Test config: `Qwen/Qwen3-30B-A3B`, FSDP2, 2 nodes × 8 H100, DP=16, sdpa, max_steps=5, `fsdp_cpu_ram_efficient_loading=true`.

The placeholder values on non-rank-0 ranks for state-dict params are **immediately overwritte** by `fsdp2_load_full_state_dict` during accelerate's FSDP2 prepare. `accelerate` moves the entire model to `meta` device before sharding  in [`accelerate.utils.fsdp_utils.fsdp2_prepare_model`](https://github.com/huggingface/accelerate/blob/60b5c258ac963d157610b47b3d779c0b1f246b47/src/accelerate/utils/fsdp_utils.py#L722)
So allocating CPU placeholders for **parameters** on non-rank-0 ranks is unnecessary work. The parameters can stay on meta. Btw, from what I can understand **buffers** (RoPE caches, attention masks, etc.) are per-rank and not part of the broadcast, so they still need real allocations.

Fixes # (issue)

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [x] Did you read the [contributor guideline]

## Who can review?
@albertvillanova  @ArthurZucker

--- Comment by HuggingFaceDocBuilderDev at 2026-04-25T21:22:22Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45649). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Rocketknight1 at 2026-04-27T11:29:07Z ---
cc @cyrilvallez I think

--- Comment by Cyrilvallez at 2026-05-11T05:13:14Z ---
@albertvillanova @AmineDiro I just pushed what is to me the correct fix - basically only move non-persistent buffers. Could you double-check? I'm not 100% familiar of how fsdp2 works

--- Comment by albertvillanova at 2026-05-18T08:17:45Z ---
Thanks for addressing this as well, @Cyrilvallez.

May I recommend to run one test with FSDP2 + cpu_ram_efficient_loading on a model that has at least one persistent buffer? Maybe something like DeepSeek V3 or creating a toy model with `self.register_buffer("bias", torch.zeros(n))` (no `persistent=False`).

If the forward pass succeeds and the persistent buffer has the correct values from rank-0, the fix is confirmed. If it crashes with AttributeError or produces wrong values, the `named_non_persistent_buffers()` line should be changed back to `named_buffers()`. Note that the memory cost of persistent buffers is negligible, and the OOM was caused entirely by parameters, which are no longer materialized.

--- Comment by AmineDiro at 2026-05-19T17:54:05Z ---
@Cyrilvallez I think [5623608](https://github.com/huggingface/transformers/pull/45649/commits/5623608c9a0584a464fd5e46680dfe9a47d6d98d) should also work , this was a regression so I just fixed it back to the previous non zeroed memory. But it should work fine 👍🏼

--- Comment by albertvillanova at 2026-04-28T05:41:32Z ---
Nit: I think the comment is right, but it under-specifies the mechanism:
- Parameters can stay on meta:
  - accelerate's fsdp2_prepare_model moves the whole model to meta before fully_shard
  - then fsdp2_load_full_state_dict broadcasts rank-0's state_dict to all ranks
- Only buffers need real allocations:
  - persistent buffers are also broadcast
  - but non-persistent ones (RoPE caches etc.) are per-rank and must be initialized locally by _init_weights


PULL REQUEST #45648: Fix SDPA inference tolerances for MPS backend
State: closed  |  Merged: False
Author: voodoovampire  |  Base: main
Labels:
Created: 2026-04-25T19:30:01Z

Fixes #45644

This PR adjusts `test_eager_matches_sdpa_inference` on the MPS backend by routing `"mps"` through the same tolerance branch as `"xpu"` in:

- `tests/test_modeling_common.py`
- `tests/models/video_llama_3/test_modeling_video_llama_3.py`

The previous default tolerances were too strict for fp16 on MPS and caused spurious test failures.

--- Comment by github-actions[bot] at 2026-04-25T19:31:12Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: video_llama_3

--- Comment by Rocketknight1 at 2026-04-27T11:28:32Z ---
Closing as per the comments in https://github.com/huggingface/transformers/issues/45644


PULL REQUEST #45645: Fix xdist collisions for captured_info artifacts and preserve CI debug logs
State: open  |  Merged: False
Author: stationeros  |  Base: main
Labels:
Created: 2026-04-25T08:47:25Z

# What does this PR do?

Fixes #45561.

This PR fixes a race in the test failure reporting flow when the suite is run with `pytest-xdist`.

Right now every worker writes debug output to the same `captured_info.txt` file. In parallel runs that means workers can overwrite each other’s output, and in some cases one worker can remove the file another worker was still relying on. The end result is flaky or incomplete failure logs, which makes CI failures much harder to debug.

The fix keeps the current behavior for non-xdist runs, but switches to worker-specific files when `PYTEST_XDIST_WORKER` is set. So instead of multiple workers all writing to the same file, each one writes its own `captured_info_<worker>.txt`.

Along with that, this PR also updates the surrounding plumbing so the debug info still shows up where people expect it:
- CI log collection now prints all matching `captured_info*.txt` files instead of assuming there is only one.
- notification artifact handling aggregates worker-specific captured info files back into the existing reporting flow.
- stale `captured_info*.txt` files are cleaned up before a new collection starts so repeated runs do not pick up leftovers from earlier failures.

I also added tests covering:
- worker-specific path selection under xdist
- legacy single-file behavior outside xdist
- cleanup of stale captured info files
- aggregation of worker-specific artifacts in notification handling

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

This touches test infrastructure / CI behavior, so @ydshieh seems like the most relevant reviewer.

--- Comment by Qodo-Free-For-OSS at 2026-04-30T09:28:06Z ---
Hi, In xdist mode, only the current worker’s captured_info file is removed at startup, so if the output directory is reused, old captured_info_<worker>.txt files from prior runs can persist and be merged/printed as if they belong to the current run.

**Severity:** remediation recommended | **Category:** correctness

**How to fix:** Clear outputs at run start

**Agent prompt to fix** - you can give this to your LLM of choice:

> ### Issue description
> With xdist enabled, only the current worker’s `captured_info_<worker>.txt` is removed, so other worker files from prior runs can remain in the output directory and later be merged/printed.
>
> ### Issue Context
> - `patch_testing_methods_to_collect_info()` currently unlinks only `_get_patched_testing_methods_output_path()`.
> - `retrieve_artifact()` merges all `captured_info*.txt` files.
> - If the reports directory is reused (common in local tooling), stale worker files can be misattributed to the current run.
>
> ### Fix Focus Areas
> - src/transformers/testing_utils.py[3778-3786]
> - src/transformers/testing_utils.py[3540-3548]
> - utils/notification_service.py[933-964]
>
> ### Recommended fixes (pick one)
> 1) Ensure the output directory is unique per run (preferred): have the runner/tooling set `_PATCHED_TESTING_METHODS_OUTPUT_DIR` to a run-unique directory (timestamp/UUID) before launching pytest/xdist.
> 2) Clear `captured_info*.txt` once *before* xdist workers start (e.g., in the CI workflow step before invoking pytest), avoiding cross-worker deletion races.
> 3) If clearing inside pytest, do it only in the xdist controller process (not workers) using an xdist hook/plugin mechanism, so it happens before workers write.

---
*[Qodo](https://github.com/marketplace/qodo-merge-pro-for-open-source) code review - free for open-source.*

--- Comment by stationeros at 2026-05-04T03:33:35Z ---
@ydshieh  Can you help review this ?


Add DeepSeek V4

PULL REQUEST #45643: Add DeepSeek V4
State: closed  |  Merged: True
Author: ArthurZucker  |  Base: main
Labels: New model
Created: 2026-04-25T00:03:25Z

Draft. Supersedes #45616.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-25T00:15:33Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45643). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ArthurZucker at 2026-04-25T07:21:28Z ---
Outputs are valid now

--- Comment by kylesayrs at 2026-04-28T21:41:06Z ---
FYI I had issues decompressing the model, potentially due to not being able to match to the weight_inverse conversion mappings. Still investigating.

```python3
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-V4-Flash",
    torch_dtype="auto",
    device_map="cpu",
)
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash")

save_dir = "DeepSeek-V4-Flash-bf16"
#model.dequantize(torch.bfloat16)
model.save_pretrained(save_dir)
tokenizer.save_pretrained(save_dir)
```

<details><summary>Analysis</summary>

```
  Original:
  WeightConverter(
      source_patterns="experts.*.w2.weight",
      target_patterns="experts.down_proj",
      operations=[MergeModulelist(dim=0)],
  )

  After update_weight_conversions:
  WeightConverter(
      source_patterns=["experts.*.w2.weight$", "experts.*.w2.weight_scale_inv$"],
      target_patterns="experts.down_proj",
      source_patterns=["experts.*.w2.weight$", "experts.*.w2.weight_scale_inv$"],
      target_patterns="experts.down_proj",
      operations=[Fp8Dequantize, MergeModulelist(dim=0)],
  )

  During loading this works fine — Fp8Dequantize consumes the scale entries and drops
  them, MergeModulelist merges the dequantized weights. The model ends up with
  experts.down_proj in BF16, no scale parameter.

  During save_pretrained, revert_weight_conversion calls reverse_transform(), producing:
  WeightConverter(
      source_patterns=["experts.down_proj"],          # 1 source
      target_patterns=["experts.*.w2.weight$",        # 2 targets
                       "experts.*.w2.weight_scale_inv$"],
      operations=[SplitModulelist(dim=0), _IdentityOp()],
  )

  When SplitModulelist.convert runs at core_model_loading.py:243-251:
  - input_dict has 1 entry (just the dequantized down_proj)
  - target_patterns has 2 entries (weight + scale)
  - len(input_dict) == 1 and len(target_patterns) != 1 → line 251 raises
  ValueError("Undefined Operation encountered!")

  The gate_up_proj converter doesn't hit this same error because its reversed ops include
   Chunk(dim=1) before SplitModulelist, and Chunk expands the 1 input into 4 entries — so
   SplitModulelist takes the else branch. However, it would produce incorrect data
  (chunking a 2-part tensor into 4 parts).

  Root cause: update_weight_conversions adds scale source patterns to existing
  converters, but Fp8Dequantize.reverse_op returns _IdentityOp() (a pass-through), so the
   reversed pipeline has no way to regenerate scale tensors that were consumed during
  dequantization. The target pattern count (weight + scale) no longer matches the input
  count (weight only).
```

</details>

--- Comment by ArthurZucker at 2026-04-29T08:01:56Z ---
@kylesayrs try with the flag to prevent reverse conversion, did not have time to implement it yet its a bit annoying

--- Comment by ArthurZucker at 2026-04-30T15:47:46Z ---
Just 1 or 2 details left, output still make sense, but just want to cleanup the cache and overlap for CSA + indexer

--- Comment by github-actions[bot] at 2026-05-02T10:53:22Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v4, gemma3, gemma3n, glm_moe_dsa, laguna, modernbert, modernbert_decoder, t5gemma2, finegrained_fp8

--- Comment by ArthurZucker at 2026-05-02T11:06:27Z ---
Ready to merge

--- Comment by ArthurZucker at 2026-05-02T11:10:18Z ---
run-slow: auto, deepseek_v4, gemma3, gemma3n, glm_moe_dsa, laguna, modernbert, modernbert_decoder, t5gemma2, finegrained_fp8

--- Comment by github-actions[bot] at 2026-05-02T11:11:52Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25250573956)

This comment contains `run-slow`, running the specified jobs:

models: ["models/auto", "models/deepseek_v4", "models/gemma3", "models/gemma3n", "models/glm_moe_dsa", "models/laguna", "models/modernbert", "models/modernbert_decoder", "models/t5gemma2"]
quantizations: ["quantization/finegrained_fp8"]

--- Comment by github-actions[bot] at 2026-05-02T11:54:09Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25250573956)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [2309029c](https://github.com/huggingface/transformers/commit/2309029c16d130a97a1bb95558855591049df647) | workflow commit (merge commit) |
| PR | [08e4cf82](https://github.com/huggingface/transformers/commit/08e4cf82819afb7622f00ac35322cba7312fb132) | branch commit (from PR) |
| main | [eed95d8c](https://github.com/huggingface/transformers/commit/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc) | base commit (on `main`) |

⚠ **Model CI failed to report results**

The test failure analysis could not be completed. Please check the [workflow run](https://github.com/huggingface/transformers/actions/runs/25250573956) for details.

--- Comment by yiakwy-xpu-ml-framework-team at 2026-05-08T06:09:09Z ---
Hi @ArthurZucker Sglang uses `self.compress_ratios instead` of `comrpession_rates`

> https://github.com/sgl-project/sglang/blob/e1bc001872985a23af65c367b802ff8fb44edafc/python/sglang/srt/configs/model_config.py#L648

And deepseek model also use self.compress_ratios instead.

Could you help to udpate the term (might be used in multiple places) ?

Can I know the reason why we changed compress_ratios array to this :

```
  "layer_types": [
    "sliding_attention",
    "sliding_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention",
    "heavily_compressed_attention",
    "compressed_sparse_attention"
  ],
```

This makes huggingface DeepSeekV4Config  a different config from the original huggingface model repo . Any benefits ?

cc @ArthurZucker

--- Comment by ArthurZucker at 2026-05-13T05:46:09Z ---
Sorry @yiakwy-xpu-ml-framework-team only saw your comments now

--- Comment by ArthurZucker at 2026-05-13T05:46:28Z ---
All the original config attributes should be loaded, I forgot that these can already be used by you guys

--- Comment by ArthurZucker at 2026-05-13T05:47:10Z ---
I want to make sure BC is kepts, but internally we want to leverage explicit / layer config (same as layer type).
I was in a bit of rush that's just an oversite on my side

--- Comment by ArthurZucker at 2026-04-27T03:55:03Z ---
```suggestion
        self.rotary_emb_compress = DeepseekV4RotaryEmbedding(config)
        self.gradient_checkpointing = False
        self.post_init()

```

--- Comment by ArthurZucker at 2026-04-27T03:56:05Z ---
```suggestion
    def __init__(self, config: DeepseekV4Config):
        super().__init__(config)
        self.model = DeepseekV4Model(config)
```

--- Comment by ArthurZucker at 2026-04-27T03:56:27Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-27T03:57:57Z ---
```suggestion
        # --- MLP site: collapse → norm → attn → expand ---
```

--- Comment by ArthurZucker at 2026-04-27T03:59:30Z ---
```suggestion
    """First ``num_hash_layers`` layers route via a frozen ``tid2eid`` (token id to expert it) lookup keyed by
```

--- Comment by ArthurZucker at 2026-04-27T05:23:26Z ---
```suggestion
        super().__init__(config)
```

--- Comment by ArthurZucker at 2026-04-27T12:28:05Z ---
```suggestion
    """
    Deepseekv4 uses sliding window attention with shared key value MQA, thus k and v point to the same storage.
    """
```

--- Comment by ArthurZucker at 2026-04-27T12:32:57Z ---
```suggestion
    """
    It stores the actual KeyValue singleton emitted every `1/m` tokens, the gate, and a buffer for the tokens that arrived in between a full window. The gate is the compression weights?
```

--- Comment by ArthurZucker at 2026-04-27T14:01:19Z ---
todo not really accurate

--- Comment by ArthurZucker at 2026-04-27T14:19:24Z ---
there are never SLiding only layers

--- Comment by ArthurZucker at 2026-04-27T14:21:03Z ---
hahha let's just have a forward

--- Comment by ArthurZucker at 2026-04-27T14:21:14Z ---
no inheritance here!

--- Comment by ArthurZucker at 2026-04-27T14:21:35Z ---
explicit is better IMO

--- Comment by vasqu at 2026-04-28T08:20:15Z ---
Oh that's a bit awkward ngl - guess they did have to make a workaround for that. Only blackwell has native fp4 support iirc

--- Comment by ArthurZucker at 2026-04-28T08:25:10Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-28T08:26:36Z ---
```suggestion
        q[..., : self.rope_head_dim], q[..., self.rope_head_dim:] = q_rope, q_nope
        kv = torch.cat([kv_rope, kv_nope], dim=-1)
```
is this any better>?

--- Comment by ArthurZucker at 2026-04-28T08:27:36Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-28T08:27:44Z ---
```suggestion
        self.compressor = COMPRESSOR_CLASSES[self.layer_type](config)
```

--- Comment by ArthurZucker at 2026-04-28T08:28:08Z ---
use config

--- Comment by ArthurZucker at 2026-04-28T08:29:36Z ---
```suggestion
```

--- Comment by vasqu at 2026-04-28T10:17:32Z ---
Imo we should refactor this to a more gemma like rope where we only apply it on one tensor - most of the time we apply to only q which makes a lot of unnecessary computations k then

--- Comment by vasqu at 2026-04-28T10:18:45Z ---
Can just overwrite the RoPE forward to not duplicate instead, i.e. https://github.com/huggingface/transformers/blob/ca72aa0ab770e9d35060ac4ff92ea687fc285112/src/transformers/models/llama/modeling_llama.py#L131 is removed

--- Comment by vasqu at 2026-04-28T10:20:13Z ---
Is this not exactly https://github.com/huggingface/transformers/blob/ca72aa0ab770e9d35060ac4ff92ea687fc285112/src/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py#L149-L153

Imo we should refactor to use our normal patterns here then

--- Comment by vasqu at 2026-04-28T10:21:01Z ---
Float can be incorporated like here then https://github.com/huggingface/transformers/blob/ca72aa0ab770e9d35060ac4ff92ea687fc285112/src/transformers/models/ernie4_5_moe/modeling_ernie4_5_moe.py#L184-L185

--- Comment by vasqu at 2026-04-28T10:23:36Z ---
Should be a layer types dict like in gemma3 / gemma4 so we have proper defaults etc

We can still use BC behavior in the post init to check for kwargs to construct defaults

--- Comment by vasqu at 2026-04-28T10:25:17Z ---
Mlp layer types for first k dense replace please

rope interleave could be ignored no? this is about our implementation --> if we don't use it no reason to keep it. Upstream won't break since kwargs are saved either way, no?

--- Comment by vasqu at 2026-04-28T10:26:09Z ---
Same here, I would use it for post init but not keep it as actual attribute - ig this is only for the partial factor on rope

--- Comment by vasqu at 2026-04-28T10:28:00Z ---
Hmm, could we remove these. Don't feel good to keep fields we actually don't use, rather make it new and not inherit if that makes sense. Might be too extreme so leaving it to you

--- Comment by vasqu at 2026-04-28T10:32:02Z ---
Imo, we shouldnt need this and have the same init as gemma3 - not sure what is different here

(also iterate over the set of layer types to have the unique ones :P)

--- Comment by vasqu at 2026-04-28T10:32:50Z ---
Maybe we can inherit from laguna now that it has landed - same as gemma3 with partial rotary which you use here

--- Comment by vasqu at 2026-04-28T10:42:55Z ---
I think you mentioned re inheritance, we could have `HCA(SWA) -> CA(HCA)`

If I see it correctly, then the difference is whether the additional indexer is used or not

--- Comment by vasqu at 2026-04-28T10:46:55Z ---
```suggestion
        # x: [..., n_groups, in_features_per_group]
        input_shape = x.shape[:-2]
        hidden_dim = x.shape[-1]
        w = self.weight.view(self.n_groups, -1, hidden_dim).transpose(1, 2)
        x = x.reshape(-1, self.n_groups, hidden_dim).transpose(0, 1)
        y = torch.bmm(x, w).transpose(0, 1)
        return y.reshape(*input_shape, self.n_groups, -1)
```
Not sure maybe this is a bit more like what we do in MHA attention

--- Comment by vasqu at 2026-04-28T10:51:09Z ---
Could be integrated into apply rotary - we essentially have the correct dim based on cos/sin (partial) head dim so you just always take `[..., -cos.shape[-1] :]` or similar

--- Comment by vasqu at 2026-04-28T10:51:33Z ---
Same here

--- Comment by vasqu at 2026-04-28T10:55:08Z ---
```suggestion
        self.key_value_proj = nn.Linear(config.hidden_size, self.coff * self.head_dim, bias=False)
        self.gate_proj = nn.Linear(config.hidden_size, self.coff * self.head_dim, bias=False)
        self.position_bias = nn.Parameter(torch.empty(self.compress_rate, self.coff * self.head_dim))
        self.kv_norm = DeepseekV4RMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.query_residual_proj = nn.Linear(config.q_lora_rank, self.n_heads * self.head_dim, bias=False)
```
maybe naming a bit more explicit?

--- Comment by vasqu at 2026-04-28T10:56:44Z ---
Not for now tbh but I feel like we could TP this as well - we just need to be careful about using hardcoded shapes, e.g. here `chunk_kv.view(batch, n_windows, self.compress_rate, -1)`

--- Comment by vasqu at 2026-04-28T10:57:35Z ---
qq: softmax in fp32 maybe? not sure how stable softmax really is here

--- Comment by vasqu at 2026-04-28T10:58:34Z ---
layer type maybe as (passed) attribute instead

--- Comment by vasqu at 2026-04-28T10:59:05Z ---
```suggestion
        q = self.wq_b(q_residual).view(batch, seq_len, -1, self.head_dim).transpose(1, 2)
```

--- Comment by vasqu at 2026-04-28T10:59:52Z ---
```suggestion
        weights = self.weights_proj(hidden_states).float() * self.scale  # [B, S, H]
```
same as in attention --> move to attr

--- Comment by vasqu at 2026-04-28T11:02:19Z ---
Could be used in other instances like the indexer - although with my comments about head dim

--- Comment by vasqu at 2026-04-28T11:04:06Z ---
Pretty much same comments as for the indexer let's be more explicit about the namings please, e.g. wkv

--- Comment by vasqu at 2026-04-28T11:04:32Z ---
And here

--- Comment by vasqu at 2026-04-28T11:05:52Z ---
Feels like this should belong to a new norm class

--- Comment by vasqu at 2026-04-28T11:06:19Z ---
rope pool is the same in a way - should check if we can refactor

--- Comment by vasqu at 2026-04-28T11:08:09Z ---
Hmm, does it work for all masks including FA?

--- Comment by vasqu at 2026-04-28T11:16:20Z ---
Same re rope usage

--- Comment by vasqu at 2026-04-28T11:17:53Z ---
I feel like the 3 deserves a comment

--- Comment by vasqu at 2026-04-28T11:18:41Z ---
Same re norm class maybe?

--- Comment by vasqu at 2026-04-28T11:22:46Z ---
Based on this we should focus the inheritance on some more normal moe implementation tbh and focus on the apply gate as main diff

--- Comment by vasqu at 2026-04-28T11:24:41Z ---
We should still use a super init then tbh and only override what we need

`self.num_experts = config.n_routed_experts` could be problematic, can we use an attribute map for this instead or BC workaround in the post init - otherwise our FP8 integrations break

--- Comment by vasqu at 2026-04-28T11:26:26Z ---
```suggestion
class DeepseekV4TopKRouter(LagunaTopKRouter):
```
it's closer to here tbh

--- Comment by vasqu at 2026-04-28T11:27:17Z ---
let's rename to `e_score_correction_bias` --> this is the same as in other models then

--- Comment by vasqu at 2026-04-28T11:29:22Z ---
Why not combine the routers and have a function deciding the final indices function --> topk vs hash table

--- Comment by vasqu at 2026-04-28T11:32:41Z ---
You actually only need `post, comb, collapsed` where we move the `collapsed` calculations into the mHC module

--- Comment by vasqu at 2026-04-28T11:33:07Z ---
why not keep input ids in the signature?

--- Comment by vasqu at 2026-04-28T11:33:36Z ---
Hmm, ok that explains my earlier comment about the padded mask

--- Comment by vasqu at 2026-04-28T11:34:13Z ---
```suggestion
```
shouldnt be needed

--- Comment by vasqu at 2026-04-28T11:34:50Z ---
We could still inherit from e.g. llama tbh, and add the extra stuff onto the init

--- Comment by vasqu at 2026-04-28T11:38:05Z ---
```suggestion
        position_embeddings = self.rotary_emb(inputs_embeds, position_ids=position_ids, layer_type="main")
```

--- Comment by vasqu at 2026-04-28T11:38:45Z ---
```suggestion
class DeepseekV4ForCausalLM(MixtralForCausalLM):
    pass
```
no?

--- Comment by 0hujun at 2026-04-29T07:29:38Z ---
The `intermediate_size` may replace to `moe_intermediate_size`, cause in mlp layers, deepseek uses moe_intermediate_size.

--- Comment by ArthurZucker at 2026-04-29T08:04:28Z ---
yeah but actually CSA uses sparse attention + overlapping Compressor, HCA does not something like that taht make it awkard to maintain two different codepathes

--- Comment by ArthurZucker at 2026-04-29T08:05:02Z ---
yeah that should be fine for sure!

--- Comment by ArthurZucker at 2026-04-29T11:25:25Z ---
```suggestion
from ..glm.modeling_glm import rotate_half
```

--- Comment by ArthurZucker at 2026-04-29T11:26:26Z ---
```suggestion
        for layer_type in set(self.layer_types):
```

--- Comment by ArthurZucker at 2026-04-29T11:29:31Z ---
pretty sure we should fuse here....

--- Comment by ArthurZucker at 2026-04-29T11:30:07Z ---
TODO

--- Comment by ArthurZucker at 2026-04-29T11:30:49Z ---
this is more compicated lets just have 2 classes

--- Comment by ArthurZucker at 2026-04-29T11:31:31Z ---
no **_ kwargs

--- Comment by ArthurZucker at 2026-04-29T11:48:57Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-29T11:49:06Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-29T11:49:36Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-29T11:50:09Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-29T11:50:20Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-29T11:51:39Z ---
```suggestion
```

--- Comment by ArthurZucker at 2026-04-29T12:04:55Z ---
```suggestion
for Manifold-Constrained Hyper-Connections (mHC), and changes the first few MoE layers with a static
token-id → expert-id hash table instead of fully dense MLPs.

The keys changes are motivated by improved inference and training performances. Support for optimized kernels and the native FP4 format is planned for the next PR!
```

--- Comment by ArthurZucker at 2026-04-29T12:05:43Z ---
```suggestion
* **Compressed Sparse Attention** (`"compressed_sparse_attention"`, **CSA** — paper §2.3.1): a low-compression
  pool (`compress_rate_csa`, default `m=4`) with overlapping windows, plus a **Lightning Indexer** (eqs. 13–17)
  that scores queries against the pool and gathers the top `index_topk` blocks per query before they reach core
  attention, concatenated to a sliding window attention.
* **Heavily Compressed Attention** (`"heavily_compressed_attention"`, **HCA** — paper §2.3.2): a high-compression
  pool (`compress_rate_hca`, default `m'=128`) with non-overlapping windows. No indexer — every pooled entry
  contributes to attention, concatenated to a sliding window attention.
```

--- Comment by vasqu at 2026-05-01T11:25:46Z ---
potentially docstring?

--- Comment by vasqu at 2026-05-01T11:26:10Z ---
```suggestion
        for layer_type in set(self.layer_types):
```
no?

--- Comment by vasqu at 2026-05-01T11:27:10Z ---
lets add the comment about the key difference - maybe we could move the repeat interleave here instead (at the end of this fn) if that fits more well than in the apply rope fn

--- Comment by vasqu at 2026-05-01T11:28:33Z ---
nit: would still go with longer var names

--- Comment by vasqu at 2026-05-01T11:29:12Z ---
```suggestion
        hidden_dim = x.shape[-1]
```

--- Comment by vasqu at 2026-05-01T11:30:26Z ---
```suggestion
        q_residual: torch.Tensor,
        position_ids: torch.Tensor,
```
yea no, hard pass to use `_` on (kw)args

--- Comment by vasqu at 2026-05-01T11:33:55Z ---
Nit: This is a bit awkward with the squeeze x unsqueeze ops

Would it make sense to allow the cached to have the unsqueezed version to be passed?

--- Comment by vasqu at 2026-05-01T11:34:12Z ---
```suggestion
        self.num_heads = config.index_n_heads
```

--- Comment by vasqu at 2026-05-01T11:37:22Z ---
Don't really have a quick idea here but it does look quite ugly 😅

--- Comment by vasqu at 2026-05-01T11:40:19Z ---
Imo, I would rather make this based on the layer type than the class so
```suggestion
        self.compressor = COMPRESSOR_CLASSES[self.layer_type](config) if self.layer_type != "sliding_attention" else None
```
Imo, makes it quick to see where the compressor is not used?

--- Comment by vasqu at 2026-05-01T11:41:18Z ---
```suggestion
        self.q_a_proj = nn.Linear(config.hidden_size, config.q_lora_rank, bias=False)
        self.q_a_norm = DeepseekV4RMSNorm(config.q_lora_rank, eps=config.rms_norm_eps)
        self.q_b_proj = nn.Linear(config.q_lora_rank, self.num_heads * self.head_dim, bias=False)
        self.q_b_norm = DeepseekV4UnweightedRMSNorm(eps=config.rms_norm_eps)
```
why not follow the naming convention through, at least looks like this in the fwd to me

--- Comment by vasqu at 2026-05-01T11:44:02Z ---
This comment doesnt make sense to me, we have `bsz, num_heads, seq_len, head_dim`. The transpose is to fit with the expected shapes on our fn. I see the -sin which might be the part where we want to add extra rotation to differentiate more but unsure

--- Comment by vasqu at 2026-05-01T11:44:51Z ---
Do we need the 2 step reshape -> view or would one direct reshape work 🤔

--- Comment by vasqu at 2026-05-01T11:45:55Z ---
```suggestion
        attn_output = self.o_b_proj(self.o_a_proj(grouped).flatten(2))
        return attn_output, attn_weights
```

--- Comment by vasqu at 2026-05-01T11:47:25Z ---
I think this needs to be moved to an apply gate fn similar to gpt oss for grouped_mm and batched_mm to properly work

--- Comment by vasqu at 2026-05-01T11:48:30Z ---
Havent checked the pretrained model class but would it make sense to have the buffer/weights in keep fp32 strict flag?

--- Comment by vasqu at 2026-05-01T11:48:53Z ---
same here

--- Comment by vasqu at 2026-05-01T11:50:02Z ---
Could be moved to the top with original dtype

--- Comment by vasqu at 2026-05-01T11:50:17Z ---
could reuse the one declared above, no?

--- Comment by vasqu at 2026-05-01T11:51:09Z ---
The only potential candidates for FA would be the vllm FA3 kernel or FA4 with FA4 being more feature rich atp

--- Comment by vasqu at 2026-05-01T11:52:31Z ---
I would that we force a cache due to the layer types requiring it

--- Comment by vasqu at 2026-05-01T11:54:30Z ---
Ehm, looks like some local tests? Probably cross check what to revert

--- Comment by vasqu at 2026-05-01T11:54:43Z ---
same here

--- Comment by vasqu at 2026-05-01T11:54:57Z ---
and here, stopping now not to repeat too much

--- Comment by vasqu at 2026-05-01T11:56:54Z ---
its a bit weird because we keep the dsv4 in the modeling code while we make general assumptions here

--- Comment by vasqu at 2026-05-01T11:58:54Z ---
Seeing a lot of `^xxx -> model.xxx` - shouldn't the base model prefix already handle those 🤔

--- Comment by vasqu at 2026-05-01T11:59:22Z ---
cut licence?

--- Comment by vasqu at 2026-05-01T12:00:23Z ---
Imo, these should be able to be overriden normally no? This looks like some weird mesh between the vlm tester and causal lm tester

--- Comment by vasqu at 2026-05-01T12:01:06Z ---
Atp, we can just set it to True

--- Comment by ArthurZucker at 2026-05-01T12:09:40Z ---
I can try!

--- Comment by ArthurZucker at 2026-05-01T12:10:48Z ---
we transpose / de-transpose but yeah this one was not ai generated !

--- Comment by ArthurZucker at 2026-05-01T12:11:17Z ---
that's a good catch indeed!

--- Comment by ArthurZucker at 2026-05-01T12:11:52Z ---
yeah, I can try it!

--- Comment by ArthurZucker at 2026-05-01T12:12:15Z ---
DynamicCache was updated for that + typing is enough imo!

--- Comment by ArthurZucker at 2026-05-01T12:12:23Z ---
yep yep will revert

--- Comment by ArthurZucker at 2026-05-01T12:12:44Z ---
valid@

--- Comment by ArthurZucker at 2026-05-01T12:12:56Z ---
yeah should will check

--- Comment by ArthurZucker at 2026-05-01T12:28:04Z ---
self.layer_types is already built from rope_parameters.items() keys (line 88), so it's unique by construction — set() would be a no-op. Keeping as-is.

--- Comment by ArthurZucker at 2026-05-01T13:03:04Z ---
Intentionally split across 3 lines — each line is a separate semantic step (build a 1D arange, scale to absolute source-token positions, broadcast to batch). Chaining it onto one .unsqueeze().expand() makes you parse the whole expression to see where the actual position math is. Keeping as-is for readability.

--- Comment by ArthurZucker at 2026-05-01T13:13:48Z ---
Keeping the two-line form — the named intermediate makes the wo_a → flatten → wo_b shape transitions explicit. Same number of ops, just less inlined.

--- Comment by ArthurZucker at 2026-05-01T17:11:19Z ---
HashRouter's only buffer is tid2eid, a torch.long token-id → expert-id lookup. It's an integer index table, not a fp32-sensitive scalar like e_score_correction_bias — keeping it strict-fp32 doesn't apply.

--- Comment by yiakwy-xpu-ml-framework-team at 2026-05-08T06:08:33Z ---
This conlifcts with SGLang and deepseek v4 model config files.

--- Comment by yiakwy-xpu-ml-framework-team at 2026-05-08T06:34:58Z ---
Are we working on the same deepseek v4 model :

```
  "rope_scaling": {
    "beta_fast": 32,
    "beta_slow": 1,
    "factor": 16,
    "original_max_position_embeddings": 65536,
    "type": "yarn"
  },
```

--- Comment by yiakwy-xpu-ml-framework-team at 2026-05-08T06:37:13Z ---
yarn for deepseek v4 ?

--- Comment by yiakwy-xpu-ml-framework-team at 2026-05-08T08:00:45Z ---
We need update accordingly in sglang :

```
                rope_scaling = self.hf_config.rope_scaling if hasattr(self.hf_config, "compress") else self.hf_config.rope_parameters

                self.scaling = compute_mla_mscale_scaling(
                    rope_scaling, self.scaling
                )

```


PULL REQUEST #45642: Fix trust_remote_code local cache collisions for local models (#45632)
State: closed  |  Merged: True
Author: Jeevang1-epic  |  Base: main
Labels:
Created: 2026-04-24T20:02:15Z

Fixes #45632

## Summary
This PR fixes local `trust_remote_code` cache collisions when different local model paths share the same leaf directory name.

## What changed
- Updated `get_cached_module_file` in `dynamic_module_utils.py`:
  - local cache subdirectory is now keyed by a stable SHA-256 hash of local source file bytes
  - hash includes main module file + direct relative-import source files
  - local/remote branch handling now uses explicit `is_local` logic (not basename equality)
- Added regression tests in `tests/utils/test_dynamic_module_utils.py` for:
  - same leaf dir + different source => different cache dirs
  - different paths + identical source => same cache dir
  - same main file + different relative-import source => different cache dirs

## Validation
- `python -m compileall src/transformers/dynamic_module_utils.py tests/utils/test_dynamic_module_utils.py`
- Added regression tests above.
- Manually validated the 3 collision/dedup scenarios via direct `get_cached_module_file` checks.

## Coordination / duplicate-work check
- Coordinated on issue: https://github.com/huggingface/transformers/issues/45632
- No overlapping open PR found for this issue at submission time.

## Notes
- Documentation update: not needed (bug fix only).
- AI was used to assist drafting; I reviewed the patch and validation results before submission.

--- Comment by nurpax at 2026-04-25T16:31:23Z ---
I reviewed the code and tried it locally.  Lgtm.

Verified the fix on this branch against the repro in #45632. Three cases:

1. Two local paths with the same leaf name but different source -> separate cache dirs (the original bug, no longer triggers).
2. Two local paths with identical source -> one shared cache dir.
3. Same `custom_model.py` at two paths but a differing `helpers.py` pulled in via `from .helpers import ...` -> separate cache dirs, and each cache dir contains both files.

All three behave as expected. Thanks for the fix.

--- Comment by Rocketknight1 at 2026-04-27T13:27:00Z ---
Hey! The issue is real, and I agree we should fix it, but I also think the very long hash pathnames clutter the cache directory a bit. How about if we just use `os.path.basename(pretrained_model_name_or_path) / hash`? Then we could shorten the hash and there should still be no collisions, and the cache directory still has human-readable names.

--- Comment by nurpax at 2026-04-28T14:13:20Z ---
@Rocketknight1 Hi!  If the original author @Jeevang1-epic is not available to update the PR, I can take a crack at amending the changes re: your feedback.

--- Comment by Jeevang1-epic at 2026-04-28T15:31:11Z ---
sorry for delay, I will be updating the pr

--- Comment by Jeevang1-epic at 2026-04-28T16:17:27Z ---
Thanks for the review @nurpax and @Rocketknight1 and great suggestion. I updated the PR to use basename/hash for local cache paths, so the directory names stay readable while still keeping the collision-safe hash behavior. I only made the requested incremental change on top of the existing fix and added/updated tests to cover it and is there any other required changes from my side or its fine?

--- Comment by HuggingFaceDocBuilderDev at 2026-04-29T11:13:39Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45642). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by nurpax at 2026-04-25T16:27:38Z ---
The full digest leads to pretty long path names now:

```
find HF_MODULES_CACHE
HF_MODULES_CACHE
HF_MODULES_CACHE/transformers_modules
HF_MODULES_CACHE/transformers_modules/c188095df8cba9168884f43b3c5821a3b820169c90954a33d3eea494952bd409
HF_MODULES_CACHE/transformers_modules/c188095df8cba9168884f43b3c5821a3b820169c90954a33d3eea494952bd409/custom_model.py
HF_MODULES_CACHE/transformers_modules/c188095df8cba9168884f43b3c5821a3b820169c90954a33d3eea494952bd409/__init__.py
HF_MODULES_CACHE/transformers_modules/bf712f2bc7deed84f526285911e4816296f3d92ccf15004b2a9800164f5e8831
HF_MODULES_CACHE/transformers_modules/bf712f2bc7deed84f526285911e4816296f3d92ccf15004b2a9800164f5e8831/custom_model.py
HF_MODULES_CACHE/transformers_modules/bf712f2bc7deed84f526285911e4816296f3d92ccf15004b2a9800164f5e8831/__init__.py
HF_MODULES_CACHE/transformers_modules/__init__.py
HF_MODULES_CACHE/__init__.py
```

Worth truncating to 16 chars like below?

```
return source_files_hash.hexdigest()[:16]
```

Some systems such as Windows are known to have issues with long path names.

--- Comment by Jeevang1-epic at 2026-04-25T17:22:29Z ---
Hey @nurpax , glad to hear it's working smoothly on your end! That is a great catch regarding the Windows path length limits—definitely don't want to trigger any OS-level crashes. I just pushed a quick update to truncate the hex digest to 16 characters as suggested. Thanks for testing it out!


PULL REQUEST #45641: Fix NameError in serving CLI due to conditional import asymmetry
State: closed  |  Merged: False
Author: ghost  |  Base: main
Labels: Code agent slop
Created: 2026-04-24T18:01:06Z

## What does this PR do?

Fixes a fatal crash in the `transformers serve` CLI when booting without the `[serving]` extras installed.

### The Bug
Files in `src/transformers/cli/serving/` (such as `chat_completion.py` and `completion.py`) conditionally import OpenAI types behind the `is_serve_available()` check. However, they unconditionally inherit from these types in the global scope (e.g., `class TransformersCompletionCreateParamsStreaming(CompletionCreateParamsStreaming)`).

When a user's environment fails `is_serve_available()`, the base classes are not imported, triggering an immediate `NameError` that cascades through the serving handlers and crashes the CLI.

### The Fix
Enforced symmetry by providing `TypedDict` or `dict`-based dummy fallback classes in the `except ImportError` (or `else`) blocks. This allows the child classes to safely compile into memory globally, preventing the `NameError` and allowing the server to boot and gracefully handle requests or prompt the user for the correct dependencies.

### Files Modified
- `chat_completion.py`
- `completion.py`
- `response.py`
- `transcription.py`


PULL REQUEST #45640: 🚨🚨🚨 [Trainer] Default to FSDP2, simplify API around fsdp + fsdp_config
State: closed  |  Merged: True
Author: SunMarc  |  Base: main
Labels:
Created: 2026-04-24T17:04:41Z

# What does this PR do ?

This PR defaults to FSDP version to 2. We cleanup the docstring to only show FSDPv2 related args and I cleaned a bit the codebase to separate FSDPv1 arg and FSDPv2 args logic. I've also added a bunch of deprecation messages for FSDPv1 + we will deprecate passing string in fsdp args.

This shouldn't impact users who relies on accelerate config to specify their fsdp related args.

--- Comment by SunMarc at 2026-04-24T17:04:53Z ---
@bot /style

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T17:29:52Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45640). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by SunMarc at 2026-04-27T14:02:56Z ---
@bot /style

--- Comment by github-actions[bot] at 2026-04-27T14:03:44Z ---
Style fix bot fixed some files and pushed the changes.

--- Comment by stevhliu at 2026-04-24T20:36:53Z ---
```suggestion
            Enable PyTorch Fully Sharded Data Parallel (FSDP) for distributed training. Pass `True` to enable FSDP.
```

--- Comment by stevhliu at 2026-04-24T20:37:32Z ---
```suggestion
            Configuration settings for when `fsdp` is enabled. Pass a path to a JSON config file, such
            as `fsdp_config.json`, or an already-loaded dict.
```

--- Comment by stevhliu at 2026-04-24T20:38:25Z ---
```suggestion
                    gathered between the forward and backward passes, avoids the re-all-gather, and use higher peak memory.
```

--- Comment by stevhliu at 2026-04-24T20:39:13Z ---
```suggestion
                    Set to `True` to reduce memory by recomputing activations during the backward pass. Prefer
                    `activation_checkpointing` over `gradient_checkpointing` when using FSDP. `gradient_checkpointing`
```

--- Comment by stevhliu at 2026-04-24T20:39:34Z ---
```suggestion
                    Set to `True` to load the pretrained checkpoint on the first process only. Other processes start
                    with empty weights and receive the weights by broadcast.
```

--- Comment by stevhliu at 2026-04-24T20:40:15Z ---
```suggestion
                    Auto-wrap policy to use. Choose `"TRANSFORMER_BASED_WRAP"`, `"SIZE_BASED_WRAP"`, or `"NO_WRAP"`.
```

--- Comment by stevhliu at 2026-04-24T20:41:07Z ---
```suggestion
                    Dictionary of XLA FSDP wrapping parameters. For a complete list of options, see the
```

--- Comment by stevhliu at 2026-04-24T20:41:31Z ---
```suggestion
                    Set to `True` to use gradient checkpointing over each nested XLA FSDP wrapped layer. Requires
                    `xla=True` and an auto-wrapping policy (`min_num_params` or `transformer_layer_cls_to_wrap`).
```


PULL REQUEST #45639: Make patched testing debug logs xdist-safe
State: open  |  Merged: False
Author: oleksii-tumanov  |  Base: main
Labels:
Created: 2026-04-24T16:03:46Z

Fixes #45561

Patched testing debug logs. Instead of having all workers write to the same `captured_info.txt` file, each worker now writes to its own file, while non-`xdist` runs keep the existing `captured_info.txt` behavior. Resetting the logs now clears only the current worker's file (replaced TODO).

I didn't find an existing PR covering this fix.

python test and style passed.


PULL REQUEST #45638: Add Multi-Token Prediction (MTP) support for Qwen3.5
State: open  |  Merged: False
Author: curnane-lab  |  Base: main
Labels:
Created: 2026-04-24T15:24:02Z

## Add Multi-Token Prediction (MTP) support for Qwen3.5

This PR adds Multi-Token Prediction (MTP) architecture and loss computation for Qwen3.5 models, enabling multi-token prediction during training for improved efficiency.

### Changes

**New classes:**
- `Qwen3_5MTPLayer`: Single MTP transformer layer with attention and MLP
- `Qwen3_5MTP`: Top-level MTP module with FC fusion, layers, and norm

**New shared helper:**
- `_compute_qwen35_mtp_loss()`: Shared MTP loss computation function used by both CausalLM and VL models, eliminating code duplication

**Modified models:**
- `Qwen3_5ForCausalLM`: Added MTP initialization and loss computation in forward pass
- `Qwen3_5ForConditionalGeneration`: Added MTP initialization and loss computation in forward pass

**Configuration:**
- Added `mtp_num_hidden_layers` (default: 0) and `mtp_loss_weight` (default: 0.0) to both `Qwen3_5TextConfig` and `Qwen3_5Config`
- Removed `mtp` from `_keys_to_ignore_on_load_unexpected` in `Qwen3_5ForCausalLM` so MTP weights are properly loaded from checkpoints

### Design decisions

1. **Shared loss function**: The `_compute_qwen35_mtp_loss()` helper eliminates code duplication between the text-only and VL models. Both models delegate to this shared function with their respective `embed_tokens` and `rotary_emb` references.

2. **MTP loss stays in model files**: Following the pattern of other auxiliary losses in transformers (e.g., MoE router losses), MTP loss is computed within the model's forward pass rather than in a separate trainer class.

3. **Backward compatible**: With `mtp_num_hidden_layers=0` (default), MTP is disabled and the models behave identically to before.

4. **Checkpoint alignment**: The MTP module structure aligns with the Qwen3.5 checkpoint format:
   - `mtp.pre_fc_norm_hidden.*`
   - `mtp.pre_fc_norm_embedding.*`
   - `mtp.fc.*`
   - `mtp.layers.N.*`
   - `mtp.norm.*`

### Testing

Tested with Qwen3.5-MTP model checkpoints to verify weight loading and loss computation.

--- Comment by github-actions[bot] at 2026-04-25T07:24:16Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_5, qwen3_5_moe

--- Comment by curnane-lab at 2026-04-25T08:26:57Z ---
## Note on relationship with PR #45618 (MTPCandidateGenerator)

I noticed that PR #45618 by @ArthurZucker introduces a `MTPCandidateGenerator` class that integrates MTP into the `generate()` pipeline for speculative decoding. I want to clarify the complementary relationship between these two PRs:

**This PR (#45638)** focuses on the **training side** of MTP:
- Adds MTP architecture modules (`Qwen3_5MTPLayer`, `Qwen3_5MTP`) to the model definition
- Computes MTP auxiliary loss during training (controlled by `output_mtp_loss` config/parameter)
- Follows the MoE auxiliary loss pattern (e.g., Mixtral's `output_router_logits`) for runtime control
- Enables training with MTP loss for improved training efficiency

**PR #45618** focuses on the **inference side** of MTP:
- Introduces `MTPCandidateGenerator` as a universal `CandidateGenerator` subclass
- Enables MTP-based speculative decoding during `generate()` for inference acceleration
- Decouples MTP layers from model files into a reusable generator component
- Adds `GenerationMode.MTP_DECODING` to the generation pipeline

These two approaches are **complementary** rather than conflicting — one addresses training-time MTP loss, the other addresses inference-time MTP speculative decoding.

That said, I appreciate the architectural direction of PR #45618 in decoupling MTP from individual model files. Once #45618 is merged, I'm happy to adapt this PR to align with the `MTPCandidateGenerator` architecture if the maintainers prefer, while preserving the training-time MTP loss computation that this PR enables.

I'd welcome any feedback from the maintainers on the preferred long-term architecture for MTP support in transformers. 🙏

--- Comment by curnane-lab at 2026-04-28T01:49:02Z ---
> For training I need to think a bit good sir! We don't want to "pollute" the code with inference specific stuff so we might have a new folder for this

Thanks for the quick response, @ArthurZucker — completely agree with the principle of keeping inference-specific and training-specific code separated.
A few thoughts to help shape the direction, whenever you've had time to think it through:

**Shared MTP module location**. If MTP layers (Qwen3_5MTPLayer, Qwen3_5MTP) live in a dedicated folder (e.g. transformers/mtp/ or under transformers/generation/mtp/), both this PR and #45618 could import from the same place. Training-side code would consume them to compute auxiliary loss; inference-side code would consume them through MTPCandidateGenerator. Happy to refactor in this direction if that matches what you have in mind.
**Training entry point**. The training-side surface is small — essentially an output_mtp_loss flag (mirroring output_router_logits in Mixtral) and the loss computation in the forward pass. I can keep this minimal and model-agnostic so it doesn't bleed into inference paths.
**Order of merging**. I'm fine waiting for #45618 to land first and then rebasing this PR on top of the agreed structure, if that's easier for review. Alternatively, if you'd prefer to land the shared MTP module location as a small prerequisite PR, I can prepare that too.

Could you let me know which layout you'd prefer once you've had a chance to think it over? I'd rather align with your preferred architecture upfront than refactor twice. 🙏


PULL REQUEST #45637: Add Multi-Token Prediction (MTP) support for Qwen3.5
State: closed  |  Merged: False
Author: curnane-lab  |  Base: main
Labels:
Created: 2026-04-24T15:14:11Z

## Add Multi-Token Prediction (MTP) support for Qwen3.5

This PR adds Multi-Token Prediction (MTP) architecture and loss computation for Qwen3.5 models, enabling multi-token prediction during training for improved efficiency.

### Changes

**New classes:**
- `Qwen3_5MTPLayer`: Single MTP transformer layer with attention and MLP
- `Qwen3_5MTP`: Top-level MTP module with FC fusion, layers, and norm

**New shared helper:**
- `_compute_qwen35_mtp_loss()`: Shared MTP loss computation function used by both CausalLM and VL models, eliminating code duplication

**Modified models:**
- `Qwen3_5ForCausalLM`: Added MTP initialization and loss computation in forward pass
- `Qwen3_5ForConditionalGeneration`: Added MTP initialization and loss computation in forward pass

**Configuration:**
- Added `mtp_num_hidden_layers` (default: 0) and `mtp_loss_weight` (default: 0.0) to both `Qwen3_5TextConfig` and `Qwen3_5Config`
- Removed `mtp` from `_keys_to_ignore_on_load_unexpected` in `Qwen3_5ForCausalLM` so MTP weights are properly loaded from checkpoints

### Design decisions

1. **Shared loss function**: The `_compute_qwen35_mtp_loss()` helper eliminates code duplication between the text-only and VL models. Both models delegate to this shared function with their respective `embed_tokens` and `rotary_emb` references.

2. **MTP loss stays in model files**: Following the pattern of other auxiliary losses in transformers (e.g., MoE router losses), MTP loss is computed within the model's forward pass rather than in a separate trainer class.

3. **Backward compatible**: With `mtp_num_hidden_layers=0` (default), MTP is disabled and the models behave identically to before.

4. **Checkpoint alignment**: The MTP module structure aligns with the Qwen3.5 checkpoint format:
   - `mtp.pre_fc_norm_hidden.*`
   - `mtp.pre_fc_norm_embedding.*`
   - `mtp.fc.*`
   - `mtp.layers.N.*`
   - `mtp.norm.*`

### Testing

Tested with Qwen3.5-MTP model checkpoints to verify weight loading and loss computation.

--- Comment by github-actions[bot] at 2026-04-24T15:15:32Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen3_5