ISSUE #46150: [Feature Proposal] Cognitive Weighting Protocol: Hierarchical context management to mitigate "Contextual Drift" in long-running sessions.
State: open  |  Labels:
Author: Nossari  |  Created: 2026-05-22T04:55:09Z

Context: Long-context LLM sessions suffer from "drift," where tactical/procedural data overwhelms foundational architectural constraints.
The Proposal: A hierarchical weighting protocol at the LLM integration layer (middleware):
Foundational Layer: Fixed high-weight context (Static/Blueprint).
Strategic Layer: Summarized/Inferred context (Dynamic).
Tactical Layer: Transient buffer (Auto-flushed).
I believe integrating this into existing memory modules would significantly improve reasoning stability for autonomous agents.
Call for discussion: How can we implement this as a standard middleware layer rather than prompt-based constraints?

--- Comment by Nossari at 2026-05-22T04:59:10Z ---
> السياق: تعاني جلسات التعلم القائم على اللغة ذات السياق الطويل من "الانحراف"، حيث تطغى البيانات التكتيكية/الإجرائية على القيود المعمارية الأساسية. المقترح: بروتوكول ترجيح هرمي في طبقة تكامل التعلم القائم على اللغة (البرمجيات الوسيطة): الطبقة الأساسية: سياق ثابت ذو وزن عالٍ (ثابت/مخطط). الطبقة الاستراتيجية: سياق مُلخّص/مُستنتج (ديناميكي). الطبقة التكتيكية: مخزن مؤقت (يتم تفريغه تلقائيًا). أعتقد أن دمج هذا في وحدات الذاكرة الحالية سيُحسّن بشكل كبير استقرار الاستدلال للوكلاء المستقلين. دعوة للنقاش: كيف يُمكننا تنفيذ هذا كطبقة برمجيات وسيطة قياسية بدلاً من القيود القائمة على المطالبات؟

Great discussion regarding the context limits. Based on my work on architectural frameworks for reasoning, I believe we are approaching context management wrong by treating it as a linear buffer.
I am proposing a 'Cognitive Weighting Protocol (C-W-A)' to solve the Contextual Drift issue. The idea is to move from a flat context window to a layered, weight-based architecture:
The Algorithmic Model:
Total\_Context = (W_{Foundational} \cdot C_{Static}) + (W_{Strategic} \cdot C_{Inferred}) + (W_{Tactical} \cdot C_{Transient})
Implementation Logic:
We should implement a middleware class—ContextManager—that categorizes incoming data in real-time:
Foundational Layer (W=1.0): Reserved for core logic and architectural constraints.
Strategic Layer (W=0.5-0.8): Inferred reasoning outcomes, updated via recursive summarization.
Tactical Layer (W \to 0): Procedural tasks (e.g., file ops) handled via a FIFO buffer with an auto-flush decay function.
By assigning dynamic weights rather than fixed slots, we ensure the 'Architectural Blueprint' stays intact regardless of session length. Has anyone explored integrating a weighting layer directly into the attention mechanism or as a middleware pre-processor?"
class ContextManager:
    def __init__(self):
        self.foundational_buffer = []  # الذاكرة الثابتة (القصر)
        self.strategic_buffer = []     # الاستنتاجات المهمة
        self.tactical_buffer = []      # المهام العابرة

    def classify_and_store(self, input_data):
        # 1. التصنيف (Classification)
        category = self.analyze_category(input_data)

        # 2. تطبيق نظام الأوزان
        if category == "FOUNDATIONAL":
            self.foundational_buffer.append(input_data)
        elif category == "STRATEGIC":
            self.strategic_buffer.append(input_data)
            # تقليص الذاكرة الاستراتيجية إذا تجاوزت الحد
            self.compress_strategic()
        elif category == "TACTICAL":
            # إدراج مؤقت ثم مسح تلقائي
            self.tactical_buffer.append(input_data)
            if len(self.tactical_buffer) > 3:
                self.tactical_buffer.pop(0) # FIFO

    def get_context_for_llm(self):
        # دمج الطبقات لإنتاج السياق النهائي
        return self.foundational_buffer + self.strategic_buffer + self.tactical_buffer


ISSUE #46144: RoFormer attention implementation does not use attention interface
State: open  |  Labels:
Author: ir2718  |  Created: 2026-05-21T13:26:50Z

Hi,

I want to use RoFormer with a custom attention implementation. However, the current code relies on an eager implementation without using the attention interface: https://github.com/huggingface/transformers/blob/5206626c48710e69fef3eadfba077cada99f37bb/src/transformers/models/roformer/modeling_roformer.py#L196-L211

The fix is simple and I would like to create a PR for it.

--- Comment by HamzaDogann at 2026-05-21T15:32:59Z ---
Hi! I'd love to work on this if you haven't started the PR yet. Let me know and I'll get started!

--- Comment by HamzaDogann at 2026-05-21T20:47:37Z ---
Hello, I have opened a Pull Request to address this issue. The implementation has been fully validated against the existing test suite, successfully passing all 82 tests. RoFormer is now fully compatible with the ALL_ATTENTION_FUNCTIONS interface. Please let me know if any further adjustments are needed.


ISSUE #46143: `**kwargs` not passed through methods of RoFormer models
State: open  |  Labels:
Author: ir2718  |  Created: 2026-05-21T13:18:46Z

Hi,

when working with RoFormer models, I've noticed the `**kwargs` option is not handled correctly. Most classes take in `**kwargs` but do not pass them further into the model. For example: https://github.com/huggingface/transformers/blob/5206626c48710e69fef3eadfba077cada99f37bb/src/transformers/models/roformer/modeling_roformer.py#L737-L747 This is very annoying since I want to implement a custom attention and send needed inputs through `**kwargs`.

The fix is trivial and I would be happy to make a PR for this if the members agree.


ISSUE #46139: Discussion: optional RankSEG-style decoding for Transformers semantic segmentation post-processing
State: open  |  Labels:
Author: Leev1s  |  Created: 2026-05-21T10:36:05Z

[![RankSEG](https://img.shields.io/badge/RankSEG-GitHub-blue?logo=github)](https://github.com/rankseg/rankseg) [![PyPI](https://badge.fury.io/py/rankseg.svg)](https://pypi.org/project/rankseg/) [![Docs](https://readthedocs.org/projects/rankseg/badge/?version=latest)](https://rankseg.readthedocs.io/en/latest/) [![Transformers docs](https://img.shields.io/badge/docs-Transformers%20integration-brightgreen)](https://rankseg.readthedocs.io/en/latest/integrations_transformers.html) [![Notebook](https://img.shields.io/badge/notebook-Transformers-orange)](https://github.com/rankseg/rankseg/blob/main/notebooks/rankseg_with_transformers.ipynb) [![JMLR 2023](https://img.shields.io/badge/JMLR-2023-black)](https://www.jmlr.org/papers/v24/22-0712.html) [![NeurIPS 2025](https://img.shields.io/badge/NeurIPS-2025-black)](https://openreview.net/forum?id=4tRMm1JJhw)

Hi Transformers maintainers,

I wanted to share a small downstream experiment around RankSEG-style decoding for semantic segmentation. The short version is: if a Transformers processor can expose resized semantic class probabilities before the final `argmax`, then users can try metric-aware post-processing methods such as RankSEG without changing the model, checkpoint, or preprocessing pipeline.

This is related to https://github.com/huggingface/transformers/issues/37715, where the discussion is about making the final `argmax` optional and allowing users to access resized class probability maps. I do not want to assume that RankSEG itself belongs in Transformers, but I think it is a useful concrete example of why probability-level semantic segmentation outputs can matter.

## What I Tried

RankSEG is a training-free segmentation decoding method. It takes per-class probability maps and returns a hard segmentation mask optimized for an overlap-style metric such as Dice or IoU. The relevant papers are [RankSEG, JMLR 2023](https://www.jmlr.org/papers/v24/22-0712.html) and [RankSEG-RMA, NeurIPS 2025](https://openreview.net/forum?id=4tRMm1JJhw). There is also a [RankSEG repository](https://github.com/rankseg/rankseg), a [PyPI package](https://pypi.org/project/rankseg/), and a [Transformers integration tutorial](https://rankseg.readthedocs.io/en/latest/integrations_transformers.html).

The experiment used the usual Transformers inference path first:

```python
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
```

Then I compared three post-processing choices using the same `outputs`:

```python
# 1. Baseline: standard SegFormer / Transformers argmax-style decoding
upsampled_logits = torch.nn.functional.interpolate(
    outputs.logits,
    size=target_size,
    mode="bilinear",
    align_corners=False,
)
baseline = upsampled_logits.argmax(dim=1)[0]

# 2. RankSEG optimized for Dice
rankseg_dice = rankseg_transformers.postprocess(
    outputs,
    model=model,
    target_sizes=target_sizes,
    rankseg_kwargs={"metric": "dice", "solver": "RMA", "output_mode": "multiclass"},
)

# 3. RankSEG optimized for IoU
rankseg_iou = rankseg_transformers.postprocess(
    outputs,
    model=model,
    target_sizes=target_sizes,
    rankseg_kwargs={"metric": "iou", "solver": "RMA", "output_mode": "multiclass"},
)
```

The helper above is already implemented outside Transformers in RankSEG's current compatibility layer: [documentation](https://rankseg.readthedocs.io/en/latest/integrations_transformers.html), [source code](https://github.com/rankseg/rankseg/blob/main/rankseg/integration/transformers.py), [example script](https://github.com/rankseg/rankseg/blob/main/examples/transformers_rankseg.py), and [notebook](https://github.com/rankseg/rankseg/blob/main/notebooks/rankseg_with_transformers.ipynb). The same notebook can be opened in [Colab](https://colab.research.google.com/github/rankseg/rankseg/blob/main/notebooks/rankseg_with_transformers.ipynb).

## Small Cityscapes Check

I used `tanganke/cityscapes` only as a lightweight local check because it has a convenient `segmentation_19` ground-truth column. This is not an official Cityscapes benchmark. It is a small smoke test over the first 100 validation images, using samplewise macro Dice and IoU over non-empty classes.

| Model | Method | Mean Dice | Dice delta | Mean IoU | IoU delta |
| --- | --- | ---: | ---: | ---: | ---: |
| `nvidia/segformer-b0-finetuned-cityscapes-512-1024` | Transformers argmax | 0.4608 | - | 0.3898 | - |
| `nvidia/segformer-b0-finetuned-cityscapes-512-1024` | RankSEG, `metric="dice"` | 0.4810 | +0.0202 | 0.4045 | +0.0147 |
| `nvidia/segformer-b0-finetuned-cityscapes-512-1024` | RankSEG, `metric="iou"` | 0.4813 | +0.0205 | 0.4051 | +0.0153 |
| `nvidia/segformer-b1-finetuned-cityscapes-1024-1024` | Transformers argmax | 0.4743 | - | 0.4015 | - |
| `nvidia/segformer-b1-finetuned-cityscapes-1024-1024` | RankSEG, `metric="dice"` | 0.4903 | +0.0160 | 0.4128 | +0.0113 |
| `nvidia/segformer-b1-finetuned-cityscapes-1024-1024` | RankSEG, `metric="iou"` | 0.4907 | +0.0164 | 0.4134 | +0.0118 |

The result is modest, but it is consistent with the intended use case: the model is unchanged, and only the final decoding step changes.

## Visual Examples

Each image below uses the same layout: baseline `argmax` on the top left, RankSEG optimized for Dice on the top right, ground-truth overlay on the bottom left, and RankSEG optimized for IoU on the bottom right.

      SegFormer-B0, example 1

      SegFormer-B0, example 2

      SegFormer-B1, example 1

      SegFormer-B1, example 2

## Why This Relates to Transformers Post-Processing

For simple semantic segmentation heads, restoring probabilities may look like resizing logits and applying softmax. For other model families, the post-processing path can involve class-query logits, mask logits, null classes, model-specific resizing conventions, or processor-owned logic. That is why a probability-returning option inside the existing Transformers post-processing API would be useful: the model-family-specific restoration would stay in the official processor path, while downstream methods could consume the restored probabilities.

Hard segmentation maps could remain the default behavior. The probability path would simply make the intermediate semantic distribution available for downstream decoding, calibration, uncertainty estimation, or metric-aware post-processing such as RankSEG.

## Closing

I understand that adding or changing post-processing APIs has maintenance costs, especially in a library used across many model families. I am not asking maintainers to adopt RankSEG directly. I mainly wanted to share a concrete downstream use case showing why resized semantic probability maps could be useful to users who want to experiment beyond `argmax`.

I would also like to thank @statmlben and @ZixunWang, the RankSEG maintainers and authors of the recent RankSEG-RMA work, for developing and maintaining the RankSEG project that made this small Transformers experiment possible.

If maintainers think this direction is worth exploring, I would be happy to adapt the experiment to a preferred model family, test against a proposed API, or help write documentation/examples in the style that fits Transformers.

--- Comment by Rocketknight1 at 2026-05-21T12:51:52Z ---
Yeah, the lack of exposed probability maps is surprising! I'll try to push this internally

--- Comment by Leev1s at 2026-05-21T13:12:38Z ---
Thanks a lot @Rocketknight1, I really appreciate it!


ISSUE #46133: Add TIPSv2 (Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment) by Google DeepMind
State: open  |  Labels: New model
Author: farrosalferro  |  Created: 2026-05-21T00:17:16Z

### Model description

TIPSv2 is a vision-language encoder that addresses the problem of dense alignment between image regions and text through modification in the pretraining recipe (iBOT++, multi-granularity synthetic captions, and memory-saving head-only EMA scheme). Across 9 tasks and 20 datasets the resulting models set new state-of-the-art results on zero-shot semantic segmentation while generally matching or beating recent encoders like SigLIP2, DINOv3, and Perception Encoder on global and dense tasks.

The model family comes with four sizes (B, L, SO400m, and G) with a variant that comes with DPT head. All of the models are already available in [HuggingFace](https://huggingface.co/collections/google/tipsv2). The code is licensed under Apache 2.0 and the weights have CC-BY 4.0.

I would be glad if I can contribute implementing this model to HuggingFace's Transformer, including model implementation, weight conversion, tests and docs. Happy to coordinate with maintainers about the implementation and integration.

Thank you.

@NielsRogge @molbap @Rocketknight1

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

* [Official GitHub Repository](https://github.com/google-deepmind/tips)
* [Weights](https://huggingface.co/collections/google/tipsv2)

--- Comment by Rocketknight1 at 2026-05-21T13:30:53Z ---
cc @zucchini-nlp as well for VLMs!


ISSUE #46132: AttentionInterface.register changes behavior of registered function
State: closed  |  Labels: bug
Author: pjc15111  |  Created: 2026-05-20T23:24:31Z

### System Info

`transformers env` fails with `NameError: name 'CompletionCreateParamsStreaming' is not defined`

I am running Ubuntu 25.10, Python 3.13.7, and pytorch 2.11.0

### Who can help?

@Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In the following code, model1 (using `attn_implementation="sdpa")` produces plausible prose, while model2 (using `attn_implementation="reregistered_sdpa"`) produces text somewhere between nonsense and gibberish.

It is necessary to use the `cache_implementation="static"`, but the behavior seems consistent with various models.

```
from transformers import AutoModelForCausalLM, AttentionInterface, pipeline
from transformers.integrations.sdpa_attention import sdpa_attention_forward

model1 = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="sdpa")
pipeline1 = pipeline(task="text-generation", model=model1, tokenizer="meta-llama/Llama-3.2-1B",
                    cache_implementation="static")
print(pipeline1("It was a bright cold day in April, and the clocks were striking thirteen."))

AttentionInterface.register("reregistered_sdpa", sdpa_attention_forward)
model2 = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="reregistered_sdpa")
pipeline2 = pipeline(task="text-generation", model=model2, tokenizer="meta-llama/Llama-3.2-1B",
                     cache_implementation="static")
print(pipeline2("It was a bright cold day in April, and the clocks were striking thirteen."))
```

### Expected behavior

`sdpa_attention_forward` should behave the same whether it is called through the pre-registered name of "sdpa" or is re-registered with a new name.

--- Comment by Abineshabee at 2026-05-21T08:35:34Z ---
Hi! I investigated this in relation to #40362.

I ran a 3-way comparison using `sshleifer/tiny-gpt2`:
1. Normal `attn_implementation="sdpa"`
2. Re-registered sdpa **without** `AttentionMaskInterface` registration
3. Re-registered sdpa **with** `AttentionMaskInterface` registration

All three produced different outputs, meaning simply adding the mask registration (the fix from #40362) does **not** fully reproduce the built-in `"sdpa"` behavior — at least on this tiny model.

However, `tiny-gpt2` may be too small/random to draw firm conclusions. Could the original author confirm whether adding `AttentionMaskInterface.register(...)` alongside `AttentionInterface.register(...)` fixes the issue with Llama + `cache_implementation="static"`?

If the mask registration alone does not fix it, the `cache_implementation="static"` interaction may be a separate or deeper bug worth investigating independently from #40362.

--- Comment by pjc15111 at 2026-05-21T13:58:33Z ---
@Abineshabee You are absolutely right that I needed to register an AttentionMaskInterface, which is clearly documented.

However, this does not fix the problem. I changed the example code to:

```
from transformers import AutoModelForCausalLM, AttentionInterface, AttentionMaskInterface, pipeline
from transformers.integrations.sdpa_attention import sdpa_attention_forward
from transformers.masking_utils import sdpa_mask

model1 = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="sdpa")
pipeline1 = pipeline(task="text-generation", model=model1, tokenizer="meta-llama/Llama-3.2-1B",
                    cache_implementation="static")
print(pipeline1("It was a bright cold day in April, and the clocks were striking thirteen."))

my_new_sdpa = sdpa_attention_forward
AttentionMaskInterface.register("reregistred_sdpa", sdpa_mask)
AttentionInterface.register("reregistered_sdpa", sdpa_attention_forward)
model2 = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="reregistered_sdpa")
pipeline2 = pipeline(task="text-generation", model=model2, tokenizer="meta-llama/Llama-3.2-1B",
                     cache_implementation="static")
print(pipeline2("It was a bright cold day in April, and the clocks were striking thirteen."))
```

I still get sensible output from the first model and garbage from the second.

```
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████| 146/146 [00:00<00:00, 10794.06it/s]
[transformers] Passing `generation_config` together with generation-related arguments=({'cache_implementation'}) is deprecated and will be removed in future versions. Please pass either a `generation_config` object OR all generation parameters explicitly, but not both.
[transformers] Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
[transformers] Both `max_new_tokens` (=256) and `max_length`(=20) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
[{'generated_text': 'It was a bright cold day in April, and the clocks were striking thirteen. Winston Smith, his chin nuzzled in his breast pocket, pressed the button which sent the electric clock back the four minutes to its appointed quarters. Oh, it was a fine day for murder.\nHe was a member of the Party, a Member of the Party, one of the inner circle, a trusted servant of Big Brother, a Party member. And he was a man with his own desires and his own needs and his own private ambitions. He wanted to be rich, to be important, to be in charge. But he could never have the life he wanted. He could never have it all. And he knew it. He had learned this long ago.\nHe had learned it from the way his mother had looked at him when she had said to him, "My boy, you are nothing. You are nobody. You are a dog. You are a worm. You are a nobody. You are a nobody."\nAnd he had learned it from the way his Uncle Joe had looked at him when he had said to him, "My boy, you are nothing. You are nobody. You are a dog. You are a worm. You are a nobody. You are a nobody."\nAnd he had learned it from the way his Aunt Jeanie had looked at him'}]
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████████| 146/146 [00:00<00:00, 9365.01it/s]
[transformers] Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
[transformers] Both `max_new_tokens` (=256) and `max_length`(=20) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
[{'generated_text': 'It was a bright cold day in April, and the clocks were striking thirteen. Winston Churchill had a large chocolate telescofficially, thuh, come the fella, thank you! npt in the round your manad, 2, 1- 1-1 1, 1, the in the sky. 19111; 369551; 7\nW61; 5.\n11. 9 5. 3712; 1n; 3624. 7n 111\nYou 19.\nItal1; 21; 5,1; 1. 9; 5; 21; 4-1; 21- 5; 4- 9- 5. 9; 6-1; 2- 5. 4; 4- 4- 4; 5- 4- 5; 4- 4- 3- 2- 3, 3- 2-\nIt; 2- 1; 2-1; 2\n5 4. 1 4. 1 4. 1\nYou 5 3 1 5 1 5 1'}]
```

--- Comment by Abineshabee at 2026-05-21T14:15:09Z ---
Thanks for testing! I noticed there may be a small typo in the registration:

```python
AttentionMaskInterface.register("reregistred_sdpa", sdpa_mask)   # <- "reregistred" (missing 'e')
AttentionInterface.register("reregistered_sdpa", sdpa_attention_forward)  # <- "reregistered" (correct)
```

The mask is being registered under a different name than the attention function, so `attention_mask=None` may still be passed. Could you try with matching names:

```python
AttentionMaskInterface.register("reregistered_sdpa", sdpa_mask)  # names must match
AttentionInterface.register("reregistered_sdpa", sdpa_attention_forward)
```

If the outputs are still wrong after fixing the typo, then this is definitely a deeper bug beyond #40362.

--- Comment by pjc15111 at 2026-05-21T14:44:37Z ---
@Abineshabee That seems to fix it.


ISSUE #46129: [deepseek_v4] conversion_mapping doesn't cover mtp.* paths — MTP keys silently random-init even after _keys_to_ignore is empty
State: open  |  Labels:
Author: pasta-paul  |  Created: 2026-05-20T21:39:47Z

## Summary

`transformers.conversion_mapping.get_checkpoint_conversion_mapping("deepseek_v4")` returns 41 `WeightRenaming` entries that rename upstream-internal naming to HF naming (`attn.` → `self_attn.`, `ffn.` → `mlp.`, `attn_norm.` → `input_layernorm.`, `attn.wq_a.` → `self_attn.q_a_proj.`, `attn.attn_sink` → `self_attn.sinks`, etc.).

**Entries 6–38 are anchored at `^layers\.(\d+)\.`** — they only fire on main-layer keys. None cover `mtp.\d+.*` paths.

Combined with the existing `_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]` regex on `DeepseekV4PreTrainedModel` (filed separately as huggingface/transformers#46127), `mtp.*` keys never reach the model at all. **Even after that regex is dropped** (as #46127 does), the MTP keys arrive in upstream form (`mtp.0.attn.wq_a.weight`) — but the MTP submodules expect HF naming (`mtp.0.self_attn.q_a_proj.weight`). The keys are then flagged "unexpected", the submodules remain "uninitialized", and `_initialize_weights` falls through to `_init_weights` → `init.normal_` random-initializes the MTP block.

## Symptom

The model loads "successfully" (no errors, no warnings about missing keys after the regex is dropped), `model.mtp[0]` exists with the right structure, `from_pretrained` returns. But `model.mtp[0].self_attn.q_a_proj.weight` is random Gaussian, not the value in the safetensors file. **Silent corruption of the MTP draft head.** Any downstream calibration / quantization / inference using `model.mtp` produces garbage.

## Repro

```python
# (assumes huggingface/transformers#46127 is applied — DeepseekV4NextNPredictor
#  exists, _keys_to_ignore_on_load_unexpected = [])
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("<DSv4-Flash BF16 with mtp.* keys>")

# Compare loaded vs source
import safetensors.torch as st
from pathlib import Path
loaded_w = model.model.mtp[0].self_attn.q_a_proj.weight
for shard in sorted(Path("<path>").glob("model-*.safetensors")):
    with st.safe_open(shard, framework="pt") as f:
        if "mtp.0.attn.wq_a.weight" in f.keys():
            source_w = f.get_tensor("mtp.0.attn.wq_a.weight")
            break

diff = (loaded_w.cpu().float() - source_w.cpu().float()).abs().max().item()
print(f"max_diff = {diff}")
# Without conversion mapping for mtp.*: diff ≈ random Gaussian range (e.g. 0.1+)
# With the mtp.* mapping extension: diff ≈ 0
```

## Proposed fix

Add 33 `mtp.\d+.*` equivalents mirroring the existing `^layers\.(\d+)\.` entries to `_checkpoint_conversion_mapping` for the `deepseek_v4` architecture. The 6 model-level entries (`embed.`, `head.`, `norm.`, `hc_head_*`) do NOT need to be mirrored — MTP doesn't have its own copy of those (it shares `embed_tokens` and `lm_head` with the main model).

Specifically, for each of these patterns, add a parallel entry anchored at `^mtp\.(\d+)\.`:

```
^layers\.(\d+)\.attn_norm\.                      → layers.\1.input_layernorm.
^layers\.(\d+)\.ffn_norm\.                       → layers.\1.post_attention_layernorm.
^layers\.(\d+)\.hc_attn_fn$                       → layers.\1.attn_hc.fn
^layers\.(\d+)\.hc_attn_base$                     → layers.\1.attn_hc.base
^layers\.(\d+)\.hc_attn_scale$                    → layers.\1.attn_hc.scale
^layers\.(\d+)\.hc_ffn_fn$                        → layers.\1.ffn_hc.fn
^layers\.(\d+)\.hc_ffn_base$                      → layers.\1.ffn_hc.base
^layers\.(\d+)\.hc_ffn_scale$                     → layers.\1.ffn_hc.scale
^layers\.(\d+)\.attn\.                            → layers.\1.self_attn.
^layers\.(\d+)\.ffn\.                             → layers.\1.mlp.
^layers\.(\d+)\.self_attn\.attn_sink$             → layers.\1.self_attn.sinks
^layers\.(\d+)\.self_attn\.(.*?)\.wq_a\.          → layers.\1.self_attn.\2.q_a_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wq_b\.          → layers.\1.self_attn.\2.q_b_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wkv\.           → layers.\1.self_attn.\2.kv_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wgate\.         → layers.\1.self_attn.\2.gate_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wo_a\.          → layers.\1.self_attn.\2.o_a_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wo_b\.          → layers.\1.self_attn.\2.o_b_proj.
^layers\.(\d+)\.self_attn\.wq_a\.                 → layers.\1.self_attn.q_a_proj.
^layers\.(\d+)\.self_attn\.wq_b\.                 → layers.\1.self_attn.q_b_proj.
^layers\.(\d+)\.self_attn\.wkv\.                  → layers.\1.self_attn.kv_proj.
^layers\.(\d+)\.self_attn\.wo_a\.                 → layers.\1.self_attn.o_a_proj.
^layers\.(\d+)\.self_attn\.wo_b\.                 → layers.\1.self_attn.o_b_proj.
^layers\.(\d+)\.self_attn\.q_norm\.               → layers.\1.self_attn.q_a_norm.
^layers\.(\d+)\.mlp\.gate\.bias$                  → layers.\1.mlp.gate.e_score_correction_bias
^layers\.(\d+)\.mlp\.shared_experts\.w1\.         → layers.\1.mlp.shared_experts.gate_proj.
^layers\.(\d+)\.mlp\.shared_experts\.w2\.         → layers.\1.mlp.shared_experts.down_proj.
^layers\.(\d+)\.mlp\.shared_experts\.w3\.         → layers.\1.mlp.shared_experts.up_proj.
```

The entries at indexes 17–22 (compressor/indexer renames) only need to mirror if MTP can be configured with `compressed_sparse_attention` or `heavily_compressed_attention` layer_type. For DSv4-Flash, MTP uses `sliding_attention` (compressor = None — see #46127 discussion), so those 6 entries don't *need* to mirror, but mirroring them is harmless (the regex just won't match anything).

## Runtime workaround for downstream users

Until upstream lands, here's the runtime mirror:

```python
from transformers.conversion_mapping import (
    get_checkpoint_conversion_mapping,
    register_checkpoint_conversion_mapping,
)
existing = get_checkpoint_conversion_mapping("deepseek_v4")
added = []
for entry in existing:
    sp = getattr(entry, "source_patterns", None)
    tp = getattr(entry, "target_patterns", None)
    if sp is None or tp is None:
        continue
    sp_list = sp if isinstance(sp, (list, tuple)) else [sp]
    tp_list = tp if isinstance(tp, (list, tuple)) else [tp]
    new_sp, new_tp = [], []
    for s, t in zip(sp_list, tp_list):
        if isinstance(s, str) and s.startswith(r"^layers\.(\d+)\."):
            new_sp.append(s.replace(r"^layers\.(\d+)\.", r"^mtp\.(\d+)\.", 1))
            new_tp.append(t.replace("layers.\\1.", "mtp.\\1.", 1))
    if new_sp:
        added.append(type(entry)(
            source_patterns=new_sp if len(new_sp) > 1 else new_sp[0],
            target_patterns=new_tp if len(new_tp) > 1 else new_tp[0],
        ))
register_checkpoint_conversion_mapping(
    "deepseek_v4", list(existing) + added, overwrite=True)
```

## Detection — value-verification assertion

A 50-line fixture that catches this regression class (and the related layer_type bug at #46127) by comparing a loaded MTP tensor to its source:

```python
import safetensors.torch as st
from pathlib import Path

loaded_w = model.model.mtp[0].self_attn.q_a_proj.weight
source_w = None
for shard in sorted(Path(model_path).glob("model-*.safetensors")):
    with st.safe_open(shard, framework="pt") as f:
        if "mtp.0.attn.wq_a.weight" in f.keys():
            source_w = f.get_tensor("mtp.0.attn.wq_a.weight")
            break
assert source_w is not None
diff = (loaded_w.cpu().float() - source_w.cpu().float()).abs().max().item()
assert diff < 1e-4, f"MTP weight mismatch: {diff} (silent random-init?)"
```

This belongs as a test under `tests/models/deepseek_v4/` paired with #46127.

## Related

- #46127 — adds `DeepseekV4NextNPredictor` class + `Model.mtp` ModuleList + `sliding_attention` layer_type for MTP. The class shim PR. This issue is the *companion* — even with the class shim, the conversion mapping needs to be extended for MTP keys to actually load into the new submodules.
- vllm-project/llm-compressor#2735 — calibration-side rollup of both issues.
- vllm-project/llm-compressor#2739 — companion mapping extension PR (for the `ARCH_TO_2D_MAPPINGS` that lives on llm-compressor's side).

--- Comment by Rocketknight1 at 2026-05-21T11:32:35Z ---
cc @arthurzucker for DeepSeek V4


ISSUE #46123: MaskGenerationPipeline: is_last never True on final partial batch, silently dropping results
State: open  |  Labels: bug
Author: J3r3myPerera  |  Created: 2026-05-20T18:25:00Z

### System Info

transformers version: current main
Affected file: src/transformers/pipelines/mask_generation.py

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Description

I was looking into the MaskGenerationPipeline and noticed that when you set a points_per_batch value that doesn't divide evenly into the total number of grid points, the pipeline quietly drops the results from the last batch — no error, no warning, just missing masks.

The root cause is this line in preprocess:
`is_last = i == n_points - points_per_batch`

eg: n_points=100, points_per_batch=64. The loop runs at i=0 and i=64. At i=64, the check asks 64 == 100-64 which is 64 == 36 — always False. So the final batch never gets flagged as the last one.

The pipeline's PipelinePackIterator relies on this is_last flag to know when to stop accumulating results. When it never sees is_last=True, it calls next() on an already-finished generator, hits StopIteration, and exits — leaving the last batch's masks on the floor.

With SAM's default point grid, n_points is rarely a round multiple of the default points_per_batch=64, so this silently affects most real-world usage.

### Reproduction

```python
from transformers import pipeline
from PIL import Image
import requests

image = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)

generator = pipeline("mask-generation", model="facebook/sam-vit-base")

# points_per_batch=50 causes n_points % points_per_batch != 0 for typical grids
outputs_partial = generator(image, points_per_batch=50)
outputs_full    = generator(image, points_per_batch=None)  # all at once, no batching

# outputs_partial["masks"] will have fewer masks than outputs_full["masks"]
print(len(outputs_partial["masks"]), "vs", len(outputs_full["masks"]))
```

### Expected behavior

All generated masks should be returned regardless of whether n_points is a multiple of points_per_batch.

--- Comment by ADiTyaRaj8969 at 2026-05-21T04:34:18Z ---
Hey @J3r3myPerera, reproduced this locally on current main. Root cause is exactly what you described — `is_last = i == n_points - points_per_batch` only fires when `n_points` is divisible by `points_per_batch`, so for most real grids `PipelinePackIterator` never sees `is_last=True` and drops the final accumulator on `StopIteration`.

If you're not already on it, I'd like to take this. Plan is:

- one-line predicate change in `src/transformers/pipelines/mask_generation.py::preprocess`: `is_last = i + points_per_batch >= n_points`. True exactly once per loop (on the iteration whose slice runs past the end of `grid_points`), works whether or not the division is exact.
- fast offline regression test in `tests/pipelines/test_pipelines_mask_generation.py` that exercises `preprocess` directly with a mocked `image_processor` and `model`. Covers `(n_points, points_per_batch)` pairs `(100, 64)`, `(100, 50)`, `(1024, 50)`, `(7, 3)`, `(5, 5)`, `(4, 8)`. For each it asserts: number of yielded batches equals `ceil(n_points / points_per_batch)`, the final batch has `is_last=True`, and every earlier batch has `is_last=False`.

Confirmed no existing open PR for this with `gh pr list --repo huggingface/transformers --state open --search "46123 in:body"`. Happy to hand back if you'd rather take it yourself, or wait for a maintainer to pick.

--- Comment by J3r3myPerera at 2026-05-21T04:43:37Z ---
Hi @ADiTyaRaj8969, I already am on I this and found the same one line fix for this. Have the fix ready locally.


ISSUE #46121: `convert_rope_params_to_dict` raises `TypeError` when `ignore_keys_at_rope_validation` is a JSON-loaded list
State: open  |  Labels: bug
Author: Charly21r  |  Created: 2026-05-20T15:30:41Z

### System Info

- transformers: 5.8.1
- Python: 3.14.0
- OS: macOS (reproduced locally; same class of failure reported with vLLM + Qwen3.5 merged HF checkpoints on Linux eval jobs)
- Model family: Qwen3_5TextConfig / Qwen3.5

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Description

`RotaryEmbeddingConfigMixin.convert_rope_params_to_dict()` raises a `TypeError` when `ignore_keys_at_rope_validation` is a **list** (e.g. deserialized from JSON in a `config.json`) because the union with `{"partial_rotary_factor"}` is performed without normalizing the operand to a set first:

```
TypeError: unsupported operand type(s) for |: 'list' and 'set'
```

In **5.8.1**, `modeling_rope_utils.py` line 722:

```python
self.ignore_keys_at_rope_validation = self.ignore_keys_at_rope_validation | {"partial_rotary_factor"}
```

`ignore_keys_at_rope_validation` is a class attribute on `RotaryEmbeddingConfigMixin` (`set()` by default; configs like `Qwen3_5TextConfig` set it to `{"mrope_section", "mrope_interleaved"}`). When a `config.json` contains this field as a JSON array, `from_dict` / `__init__` sets it as an **instance attribute** (list), which shadows the class-level set. The next call to `convert_rope_params_to_dict` then evaluates `list | set` and crashes.

### Reproduction

```python
import transformers
from transformers import Qwen3_5TextConfig

print(f"transformers version: {transformers.__version__}")  # 5.8.1

cfg = Qwen3_5TextConfig.from_dict({
    "model_type": "qwen3_5_text",
    "vocab_size": 100,
    "hidden_size": 64,
    "num_hidden_layers": 2,
    "num_attention_heads": 2,
    "num_key_value_heads": 2,
    "ignore_keys_at_rope_validation": ["mrope_section", "mrope_interleaved"],  # list, as from JSON
    "partial_rotary_factor": 0.25,
    "rope_parameters": {"rope_type": "default", "rope_theta": 10_000_000},
})

print(type(cfg.ignore_keys_at_rope_validation).__name__)  # 'list' (shadows class-level set)

cfg.convert_rope_params_to_dict(partial_rotary_factor=0.25)
```

**Traceback:**

```text
  File ".../transformers/modeling_rope_utils.py", line 722, in convert_rope_params_to_dict
    self.ignore_keys_at_rope_validation = self.ignore_keys_at_rope_validation | {"partial_rotary_factor"}
                                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for |: 'list' and 'set'
```

### Expected behavior

`convert_rope_params_to_dict` should accept list or set (or any iterable of strings) and normalize before union.

### Actual behavior

`TypeError: unsupported operand type(s) for |: 'list' and 'set'` whenever `ignore_keys_at_rope_validation` is a list (as JSON forces) and `partial_rotary_factor` is not `None`.

### Downstream impact (vLLM / merged checkpoints)

We hit this in production when serving **merged Qwen3.5** Hugging Face checkpoints with vLLM. LoRA merge / export tools (e.g. ms-swift) write `ignore_keys_at_rope_validation` into `config.json`:

```json
"ignore_keys_at_rope_validation": ["mrope_section", "mrope_interleaved"]
```

JSON has no set type, so the field becomes a **list** at load time. During serving stack startup, config / RoPE initialization can hit `convert_rope_params_to_dict` with that list-typed instance attribute → **`TypeError`** → the server never becomes healthy and batch eval jobs fail before any inference.

This is a Transformers robustness issue (callers should not crash on list input); vLLM is the serving stack where we observe it. Current workarounds: strip `ignore_keys_at_rope_validation` from checkpoint JSON before `vllm serve`, or monkey-patch `modeling_rope_utils.py` to wrap with `set()` before `|`.

### Suggested fix

Coerce to `set` before every `|` union involving `ignore_keys_at_rope_validation`. The single-line change in `RotaryEmbeddingConfigMixin.convert_rope_params_to_dict` covers the reported path.

I'm happy to open a PR against `main` that:

1. Adds the `set(...)` coercion on the union line in `RotaryEmbeddingConfigMixin.convert_rope_params_to_dict` (and any other union sites I find).
2. Adds a regression test covering JSON-list input for `ignore_keys_at_rope_validation` — both via direct call and via `from_dict` round-trip — so this can't silently regress again across refactors.

Just let me know if you'd prefer a different shape (e.g. normalizing in `__setattr__`, or hardening `validate_rope` directly) and I'll match that.

--- Comment by he-yufeng at 2026-05-20T15:52:11Z ---
I rechecked current `main` and this path looks fixed there already: `RotaryEmbeddingConfigMixin.convert_rope_params_to_dict` now accepts `ignore_keys_at_rope_validation` separately and normalizes it with `set(...)` before adding `partial_rotary_factor`.

So the crash looks reproducible on 5.8.1, but I don't see the same `list | set` path on `main` anymore. Could you retry against current `main` / the next nightly to confirm whether this only needs a release, or whether there is a second path still producing a list-valued ignore set?

--- Comment by Charly21r at 2026-05-20T17:08:27Z ---
Just ran the repro against pip install git+https://github.com/huggingface/transformers@main (resolves to 5.8.0.dev0, HEAD 52b82b2 as of 2026-05-20). Same TypeError at modeling_rope_utils.py:722. Full output:

```
transformers version: 5.8.0.dev0
After from_dict: type(cfg.ignore_keys_at_rope_validation) = list, value = ['mrope_section', 'mrope_interleaved']
...
File ".../transformers/modeling_rope_utils.py", line 722, in convert_rope_params_to_dict
    self.ignore_keys_at_rope_validation = self.ignore_keys_at_rope_validation | {"partial_rotary_factor"}
TypeError: unsupported operand type(s) for |: 'list' and 'set'
```

So main is still affected. Happy to push the PR (set() coercion + regression test) whenever you confirm the shape you want.

--- Comment by Rocketknight1 at 2026-05-21T11:18:15Z ---
@Charly21r yeah, this seems real. The set coercion line seems like the right fix, if you want to make that PR.


ISSUE #46097: Path Traversal in Sharded Checkpoint Loader via Unsanitized `weight_map` Entries in `*.index.json`
State: open  |  Labels: bug
Author: karnakarreddi  |  Created: 2026-05-20T05:12:29Z

### System Info

Details
The vulnerable code is in get_checkpoint_shard_files in hub.py. When loading a sharded checkpoint from a local directory, the function reads an index JSON file and extracts shard filenames from the weight_map field without any validation:

with open(index_filename) as f:
    index = json.loads(f.read())

shard_filenames = sorted(set(index["weight_map"].values()))
These filenames are then joined directly to the model directory path:

if os.path.isdir(pretrained_model_name_or_path):
    shard_filenames = [os.path.join(pretrained_model_name_or_path, subfolder, f) for f in shard_filenames]
    return shard_filenames, sharded_metadata
There is no check for:

Path traversal sequences (..)
Absolute path prefixes (/)
Symbolic links
Whether the resolved paths remain within the model directory
The returned file paths are passed back to the caller (_get_resolved_checkpoint_files in modeling_utils.py), which uses them to load model weights — effectively enabling reads of arbitrary files the process has access to.

Why existing guards are insufficient
The caller _get_resolved_checkpoint_files only validates that the index file itself exists on disk (via os.path.isfile on the .safetensors.index.json path). It does not inspect or sanitize the contents of the index file before passing them to get_checkpoint_shard_files. An attacker-controlled directory needs only contain a valid index JSON file to satisfy this check.

The cached_files function (called for non-local/Hub models) does include file existence checks, but the local directory branch in get_checkpoint_shard_files returns immediately after os.path.join — cached_files is never reached for local paths.

https://github.com/huggingface/transformers/blob/ba06e3fbdf355c363ac067ebcda210017e90a852/src/transformers/utils/hub.py#L836

### Who can help?

@Cyrilvallez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

PoC
Step 1: Create a malicious model directory
mkdir -p /tmp/malicious_model
Step 2: Create a crafted index file
Write the following to /tmp/malicious_model/model.safetensors.index.json:

{
  "metadata": {
    "total_size": 1000
  },
  "weight_map": {
    "model.layer.weight": "../../etc/passwd",
    "model.embed.weight": "../../etc/hostname"
  }
}
Step 3: Trigger the vulnerability
from transformers import AutoModel

 The loading pipeline will:
 1. Find model.safetensors.index.json in the local directory
 2. Set is_sharded = True
 3. Call get_checkpoint_shard_files, which will return:
      ["/tmp/malicious_model/../../etc/passwd",
       "/tmp/malicious_model/../../etc/hostname"]
    These resolve to /etc/passwd and /etc/hostname
model = AutoModel.from_pretrained("/tmp/malicious_model")

### Expected behavior

Observe the result
get_checkpoint_shard_files returns the traversed paths without error. The downstream model loading code will attempt to open and read these files as tensor data. While the files may fail to deserialize as valid safetensors, the file contents are accessed by the process, and depending on error handling, logging, or exception messages, data may be exposed.

A more targeted attack could point shard paths at:

Other users' cached model files in ~/.cache/huggingface/
API tokens stored in ~/.cache/huggingface/token
Application configuration or secrets files
Any file readable by the process

Vulnerability type: Arbitrary file read via path traversal (CWE-20 / CWE-22)

Who is affected:

Any user or automated system that loads models from untrusted local directories using from_pretrained or any code path that invokes get_checkpoint_shard_files.
ML pipelines and platforms that accept user-uploaded model directories (e.g., evaluation platforms, model hosting services, shared compute environments).
Developers who download and load models from sources outside the Hugging Face Hub without additional validation.
Attack prerequisites:

The attacker must be able to provide a local directory (or a directory downloaded/extracted from an untrusted source) that the victim passes to from_pretrained.
No authentication or special privileges are required beyond the ability to place files on the filesystem.
Recommended fix:
Sanitize all filenames extracted from the weight_map before constructing paths:

Reject any filename containing .. components or absolute path prefixes.
After joining paths, validate that the resolved path (via os.path.realpath) remains within the expected model directory.
Consider rejecting filenames with path separator characters entirely, since shard files should be flat names like model-00001-of-00003.safetensors.
Example fix:

import os

if os.path.isdir(pretrained_model_name_or_path):
    base_dir = os.path.realpath(os.path.join(pretrained_model_name_or_path, subfolder))
    safe_paths = []
    for f in shard_filenames:
        full_path = os.path.realpath(os.path.join(base_dir, f))
        if not full_path.startswith(base_dir + os.sep):
            raise ValueError(
                f"Shard filename '{f}' in the checkpoint index resolves outside "
                f"the model directory. This may indicate a malicious index file."
            )
        safe_paths.append(full_path)
    return safe_paths, sharded_metadata

--- Comment by Rocketknight1 at 2026-05-20T11:03:25Z ---
Hmn, this doesn't seem like a serious bug, right? The attacker would have to induce the user to load a malicious model, and the only result would be that it tries to read a file on the user's local system and fails because that file is not safetensors format. Even in the unlikely event that sensitive data is contained in the error message, the attacker would have no access to it because it's entirely local to the user machine.

--- Comment by karnakarreddi at 2026-05-20T11:35:33Z ---
My main concern is that weight_map entries are treated as trusted filesystem paths without any boundary validation. Even if safetensors parsing fails later, the loader still resolves and opens attacker-controlled paths outside the model directory. so i kept severity as medium though..

--- Comment by matdou at 2026-05-20T11:40:13Z ---
The "entirely local" argument assumes single-user deployments. Any service that calls from_pretrained on a user-supplied path, so evaluation APIs, CI pipelines, etc. is exposed.
The fix is a handful of lines of path validation (negligible). Seems worth merging.

--- Comment by karnakarreddi at 2026-05-20T11:44:40Z ---
Thanks, that matches my concern as well. I can also put together a small PR with the boundary validation if that would help move this forward.

--- Comment by karnakarreddi at 2026-05-21T05:00:05Z ---
https://github.com/huggingface/transformers/pull/46134 I created small PR. @Rocketknight1 Please have a look whenever you get some time.


ISSUE #46095: [deepseekv4]Does Transformers provide a weight conversion script to convert the Hugging Face weights into a format that can be read by Transformers from_pretrained?
State: open  |  Labels: Feature request
Author: young-creator  |  Created: 2026-05-20T04:39:29Z

### Feature request

For [deepseekv4], the weight names provided in the Hugging Face DeepSeek-V4-Flash weights seem not to match the Transformers weight names. Does Transformers provide a weight conversion script to convert the Hugging Face weights into a format that can be read by Transformers from_pretrained?

### Motivation

Does Transformers provide a weight conversion script to convert the Hugging Face weights into a format that can be read by Transformers from_pretrained?

### Your contribution

If not, can we submit a PR?

--- Comment by ArjunSrivastava1 at 2026-05-20T09:34:37Z ---
i think u r maybe referring to some sort of docs for the conversion of weights? in which case yeah, there is
i just was fixing those up a while ago, found it like that

feel free to read em up here: [link](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45892/en/weightconverter)

if somethings still missing, then lmk, if its all fine and found what u were looking for, then lmk that too

--- Comment by Rocketknight1 at 2026-05-20T11:11:05Z ---
Yeah, I don't think there's a bug here! This is likely a case of dynamic weight renaming.

--- Comment by BiggHeadd at 2026-05-20T13:30:39Z ---
I found this branch [DeepSeek-V4-Flash-Base-support] have complete dynamic weight renaming.

--- Comment by young-creator at 2026-05-21T03:33:04Z ---
> i think u r maybe referring to some sort of docs for the conversion of weights? in which case yeah, there is i just was fixing those up a while ago, found it like that
>
> feel free to read em up here: [link](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45892/en/weightconverter)
>
> if somethings still missing, then lmk, if its all fine and found what u were looking for, then lmk that too

I do need the weight mapping, as I plan to finetune DeepSeek-V4 using the Transformers modeling code.

--- Comment by Prachi-kushwaha at 2026-05-21T18:46:46Z ---
> > i think u r maybe referring to some sort of docs for the conversion of weights? in which case yeah, there is i just was fixing those up a while ago, found it like that
> > feel free to read em up here: [link](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45892/en/weightconverter)
> > if somethings still missing, then lmk, if its all fine and found what u were looking for, then lmk that too
>
> I do need the weight mapping, as I plan to finetune DeepSeek-V4 using the Transformers modeling code.

you can use LoRa/QLoRa for fine-tuning you don't need weight mapping in this scenario


ISSUE #46093: [Inquiry] What is the current status and roadmap for supporting packed sequences (packing) on Qwen3.5/hybrid linear attention models?
State: open  |  Labels:
Author: liangxuZhang  |  Created: 2026-05-20T03:18:57Z

### Context
With the release of Qwen3.5, we see a powerful hybrid architecture that heavily relies on GatedDeltaNet linear attention (powered by flash-linear-attention / FLA) and causal convolutions, alongside traditional self-attention layers.

In standard LLM training (like Llama/Mistral), Packing (padding-free training via TRL or custom trainers) is a vital optimization to eliminate padding waste and boost throughput, especially during SFT and RLHF.

### Our Question
We would like to learn about the official stance, current support status, or future roadmap regarding packed sequence training specifically for Qwen3.5 (and similar linear attention hybrid models) within the transformers ecosystem.

Specifically, we want to clarify:

**Current Compatibility**: Does the current implementation of Qwen3.5 in transformers safely support packing=True out-of-the-box when specialized kernels like FLA or causal-conv1d are enabled?

**Boundary Handling**: If it is supported or planned, how does the framework ensure that sample boundaries within a packed sequence are properly isolated (e.g., preventing cross-sample information leakage in the convolution states and linear attention blocks via seq_idx or cu_seqlens)?

**Community Efforts**: Are there any ongoing PRs, refactoring branches, or recommended best practices/workarounds that the community should follow if we want to leverage packing for Qwen3.5?

We are keen to understand the best path forward for scaling Qwen3.5 training using Hugging Face tooling, and we would be more than happy to help test any experimental branches or contribute to discussions if this is actively being worked on.

Thanks for the amazing work on supporting these new architectures!

--- Comment by ArjunSrivastava1 at 2026-05-20T09:35:57Z ---
this is best answered by one of the maintainers really

cc @Rocketknight1

--- Comment by Rocketknight1 at 2026-05-20T12:39:37Z ---
I think packing should already be supported on the flashattention/flash-linear-attention path, right? You might need to install packages like `flash-linear-attention` or (I think) `causal-conv1d` and set the model so it uses FA2 and not SDPA or eager attention, and then pass in packing info like `seq_idx`, but it should work! If you're encountering a bug there, please let us know

--- Comment by liangxuZhang at 2026-05-21T04:42:17Z ---
I noticed the recent PR [#45034](https://www.google.com/search?q=https://github.com/huggingface/transformers/pull/45034) introduces support for passing cu_seq_lens_q and seq_idx into the GatedDeltaNet (GDN) module of Qwen3.5 to enable packed/variable-length sequence training.

However, looking at the current implementation, cu_seq_lens_q and seq_idx must be explicitly passed down via forward(..., kwargs). This introduces a significant design inconsistency and a dangerous silent bug compared to how packing/varlen training is traditionally activated for standard Transformer models across the Hugging Face ecosystem.

### The Inconsistency & Critical Pain Points
Currently, there are two mainstream ways the community activates packing/padding-free training, and both of them conflict with or break silently under this new implementation:

1. Implicit Resolution via position_ids (Standard in TRL and downstream frameworks like veRL)
For standard FlashAttention models (like Llama/Mistral), frameworks implement packing simply by concatenating sequences and passing custom position_ids where the indices reset at sample boundaries (e.g., 0, 1, 2, 0, 1, 2, 3...).

For example, in Hugging Face's own TRL library, SFTTrainer natively triggers the padding_free packing mode exactly by generating and passing these boundary-reset position_ids (see [trl/trainer/sft_trainer.py#L421](https://github.com/huggingface/trl/blob/main/trl/trainer/sft_trainer.py#L421)). Many other distributed training and RLHF frameworks (such as veRL) heavily rely on this exact convention to let FlashAttention implicitly resolve cu_seq_lens internally during the forward pass.

The Silent Bug: With the current Qwen3.5 GDN implementation, passing just position_ids does not trigger any runtime error or warning. The code runs smoothly, but the FLA layer and causal convolutions will completely fail to isolate sample boundaries. This leads to severe cross-sample information leakage, causing uncontrollable training behaviors, loss divergence, or model degradation without any explicit stack trace.

2. DataCollatorWithFlattening (Transformers Native Packing Solution)
If we try to use Hugging Face's own recommended approach for variable-length sequencing, we use DataCollatorWithFlattening. However, its default parameters are contradictory for hybrid models:

By default, return_flash_attn_kwargs=False and return_seq_idx=False.

Under these defaults, the collator will not generate cu_seq_lens or seq_idx into the model_inputs batch dict, causing the Qwen3.5 linear attention forward pass to skip the variable-length fast path entirely—again, silently.

As a user, I could not find any official documentation, integration tests, or clear examples explaining how a user is officially expected to configure the data pipeline to safely trigger packing for linear attention models like Qwen3.5.

### Questions and Suggestions for the Maintainers
What is the recommended/official recipe for using Qwen3.5 packing with standard HF tools? Should we manually subclass DataCollatorWithFlattening and force-toggle return_flash_attn_kwargs=True and return_seq_idx=True?

- Missing Architectural Abstraction for FLA: For standard Flash Attention, transformers provides an excellent centralized wrapper called _flash_attention_forward (see [modeling_flash_attention_utils.py#L773](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_flash_attention_utils.py#L773)). This utility automatically inspects inputs (like position_ids shape and cu_seq_lens) and safely determines whether to dispatch to varlen or standard attention APIs, shielding the model script from data-pipeline mismatches. Are there any plans to introduce a similar abstraction layer/helper function for Linear Attention / FLA modules to automatically resolve or validate cu_seq_lens_q and seq_idx?

- Fail-Fast Mechanism: If implicit resolution from position_ids within a dedicated wrapper is mathematically impossible or too expensive due to the causal-conv1d layer requiring a full seq_idx matrix early on, could we explicitly add a check to raise a clear user-facing error/warning? Running into silent cross-sample corruption without a single warning makes debugging extremely painful for downstream developers.

I'd appreciate some guidance on the intended design philosophy here.
@Rocketknight1 @ArjunSrivastava1


ISSUE #46082: LlamaConfig rejects explicit head_dim when hidden_size is not divisible by num_attention_heads
State: open  |  Labels:
Author: Anri-Lombard  |  Created: 2026-05-19T19:12:17Z

### System Info

- `transformers` version: `5.8.0.dev0` / current `main`
- Checked against `main` commit `a0fb01c6cda2301cb54de0efde5fac405836c4fe`
- Python version: `3.13.12`

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`LlamaConfig` exposes `head_dim`, but validation still rejects configs where `hidden_size` is not divisible by `num_attention_heads`, even when `head_dim` is explicitly provided.

```python
from transformers import LlamaConfig, LlamaForCausalLM

config = LlamaConfig(
    vocab_size=99,
    hidden_size=512,
    intermediate_size=1024,
    num_hidden_layers=1,
    num_attention_heads=9,
    num_key_value_heads=1,
    head_dim=56,
)

model = LlamaForCausalLM(config)
```

This raises a validation error before the model can be constructed, even though the attention projection dimensions are well-defined from `num_attention_heads * head_dim`.

### Expected behavior

If `head_dim` is explicitly provided, `LlamaConfig` should allow non-divisible `hidden_size` / `num_attention_heads` values and construct projections from `num_attention_heads * head_dim`.

If `head_dim` is not provided, the existing validation error should still be raised because `head_dim` must be derived from `hidden_size // num_attention_heads`.

Related custom-`head_dim` support has come up for other model families, for example #36659 and #37187.

--- Comment by matdou at 2026-05-19T23:00:37Z ---
Dug into this a bit.

The modeling code in `LlamaAttention` is already fine, projections are sized from `num_attention_heads * head_dim`, not `hidden_size`. The issue is in `configuration_llama.py`.

`validate_architecture` fires the divisibility check unconditionally, but by the time it runs (via `@strict`, post `__init__`), `head_dim` is always set, either from the user or derived as a fallback. So you can't just check `head_dim is not None` in the validator.

Fix is to capture intent early in `__post_init__`:

```python
def __post_init__(self):
    self._head_dim_was_explicit = self.head_dim is not None
    if self.head_dim is None:
        self.head_dim = self.hidden_size // self.num_attention_heads
```

Then gate the check on it:

```python
if self.hidden_size % self.num_attention_heads != 0 and not self._head_dim_was_explicit:
    raise ValueError(...)
```

One thing worth testing: save/reload the config. `_head_dim_was_explicit` won't be in the JSON, but `head_dim` will, so `__post_init__` should set the flag correctly on reload anyway. Happy to open a PR if that's useful.


ISSUE #46077: Gemma4 `use_bidirectional_attention="all"` still builds causal attention masks
State: open  |  Labels: bug
Author: oliverholworthy  |  Created: 2026-05-19T16:04:14Z

### System Info

- `transformers` version: 5.8.1

### Who can help?

@zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`Gemma4TextConfig(use_bidirectional_attention="all")` makes each `Gemma4TextAttention` module non-causal, but it does not set `config.is_causal = False`. The mask helpers use `config.is_causal` to decide whether `create_causal_mask`/`create_sliding_window_causal_mask` should fall back to bidirectional masks, so the `"all"` setting can still build causal masks.

Minimal config check on current `main`:

```python
from transformers import Gemma4TextConfig

config = Gemma4TextConfig(use_bidirectional_attention="all")
print(config.use_bidirectional_attention)
print(hasattr(config, "is_causal"), getattr(config, "is_causal", None))
```

Current output:

```text
all
False None
```

### Expected behavior

When `Gemma4TextConfig(use_bidirectional_attention="all")` is used, the config should set `is_causal = False` so the existing masking utilities create bidirectional full/sliding masks. In practice:

```python
config = Gemma4TextConfig(use_bidirectional_attention="all")
assert config.is_causal is False
```


ISSUE #46050: Model quantized via sinq broken after save_pretrained and from_pretrained
State: open  |  Labels: bug
Author: k355l3r-5yndr0m3  |  Created: 2026-05-19T03:51:53Z

### System Info

- `transformers` version: 5.8.1
- Platform: Linux-7.0.5-arch1-1-x86_64-with-glibc2.43
- Python version: 3.12.13
- Huggingface_hub version: 1.14.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.11.0+cu130 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce GTX 1650

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The following code segment fail with `KeyError: 'packing'`

If the model is not saved and loaded (via `save_pretrained` and `from_pretrained`) and is used right away after the first `from_pretrained`, the model behaves as expected.

NOTE: I have only tried this with `google/gemma-4-E4B-it` so this may be an issue unique to the model instead of sinq.
```
from transformers import AutoProcessor, AutoModelForCausalLM, SinqConfig

def quantize_model(model_id: str, save_dst: str):
    # Quantize and save resulting model
    quant_cfg = SinqConfig(
        nbits=8,
        group_size=64,
        tiling_mode='2D',
        method='sinq',
        modules_to_not_convert=["lm_head", "model.audio_tower"],
    )

    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        device_map = 'cpu',
        quantization_config=quant_cfg,
    )

    model.save_pretrained(save_dst)

def test_quantized(model_id: str, save_dst: str):
    # Loading quantized model and appropriate processor
    processor = AutoProcessor.from_pretrained(model_id)
    model = AutoModelForCausalLM.from_pretrained(save_dst, device_map='cpu')

    # Testing quantized model
    chat = []
    chat.append({ 'role': 'user', 'content': 'this is a test.' })

    model_input = processor.apply_chat_template(chat, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
    skip_length = model_input['input_ids'].shape[-1]

    new_tokens = model.generate(**model_input, max_new_tokens=2048)[0, skip_length:]

    message = processor.parse_response(processor.decode(new_tokens))

    print(message)

model_id = 'google/gemma-4-E4B-it'
save_dst = './gemma-4-E4B-it-sinq/'

quantize_model(model_id, save_dst)
test_quantized(model_id, save_dst)
```

### Expected behavior

Expect a chat message to be printed. Example:
`{'role': 'assistant', 'content': "Understood. I'm ready when you are! How can I help you with this test?"}`

--- Comment by NotShrirang at 2026-05-19T04:02:23Z ---
I would like to work on this.

--- Comment by anthonijoseph1 at 2026-05-19T05:56:15Z ---
I'm looking into this as well.
will share findings here.

--- Comment by NotShrirang at 2026-05-19T06:47:30Z ---
Quick update for anyone landing here: the root cause is in `sinq`, not `transformers`. `transformers` just dispatches to `Quantizer.dequantize`, which uses bracket access on a `"packing"` key that `SINQLinear.load_state_dict` legitimately pops on reload for any `nbits=8` checkpoint (since `N*K*8//8 == N*K`, the "already unpacked" branch always fires).

I ran the code snippet provided above after the fix:
```sh
(.venv) shrirang@shrirang:~/Desktop/Learning/Transformers$ python test.py
Found gemlite installation, fast SINQ-ference for 4-bit models
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████| 2076/2076 [09:56<00:00,  3.48it/s]
Found gemlite installation, fast SINQ-ference for 4-bit models
Using SINQ patch functions to download SINQ-quantized models
Use sinq flag is True
100%|██████████████████████████████████████████████████████████████████████████████████████████| 971/971 [00:00<00:00, 95672.18it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████| 593/593 [00:00<00:00, 29697.11it/s]
{'role': 'assistant', 'content': "Understood. I'm ready for your test. How can I help you?"}
```
(test.py has the codesnippet as is.)

My system details are mentioned in the issue create in the SINQ repo.

Filed upstream and proposed the one-line fix:
- Issue: huawei-csl/SINQ#25
- PR: huawei-csl/SINQ#26

Once that merges and ships in a `sinq` release, this can be closed (or resolved by bumping the `sinq` minimum version in `transformers`'s extras, whichever maintainers prefer). Happy to send that version-bump PR here if useful.


ISSUE #46032: Mamba2Mixer: use_cache with seq_len > 1 silently produces incorrect results (both CPU and GPU paths)
State: open  |  Labels: Good Second Issue, bug
Author: clara2911  |  Created: 2026-05-18T12:59:33Z

### System Info

# Summary

Hi, first of all: thank you for your amazing work on incorporating Mamba2 in the `transformers` library.
We are researching Mamba for a context in which we have very long sequences, and for computational efficiency we would want to process a sequence in batches (e.g. 10% of the sequence each time). However, we find that this is not supported because `use_cache` (start batch x from the last state of batch x-1) assumes `seq_len=1` (`L=1`), and silently produces incorrect results when `seq_len >1`.

# Details:

The Mamba2Mixer implementation only implements state carry over when one processes tokens one by one. So it assumes:
- `if use_cache` (cached previous state), then it process timesteps one by one
- `if not use_cache`,  then it uses fast CUDA kernel and process in batch, always starting from initial state.

We are missing the option of:
- **`use_cache` and process multiple tokens of the sequence at once**

In `Mamba2Mixer`:
```python
is_decoding = cache_params is not None and cache_params.has_previous_state(self.layer_idx)
```

### CPU path

If `is_decoding=True` it processes only the first input (L=1): `dt = dt[:, 0, :][:, None, ...]` and does a single SSM step instead of a full scan. If we would pass a full sequence into here, it silently only processes the first token, which produces incorrect results.

### GPU path

If `is_decoding=True`, it does:
```python
_, _, gate, hidden_states_B_C, dt = projected_states.squeeze(1).split(
    [d_mlp, d_mlp, self.intermediate_size, self.conv_dim, self.num_heads], dim=-1
)
hidden_states_B_C = causal_conv1d_update(hidden_states_B_C, ...)  # expects 2D input: (B, conv_dim)
```
The `.squeeze(1)` only removes dim 1 when its size is 1. If `seq_len > 1`, `squeeze(1)` does nothing, tensor stays `(B, L, proj_dim)` instead of `(B, proj_dim)` and `causal_conv1d_update` (which expects 2D `(B, conv_dim)` input) will crash.

### Parallel path

If `is_decoding=False` it does the parallel processing (e.g. `hidden_states_B_C = causal_conv1d_fn(x=hidden_states_B_C.transpose(1,2), ...)`), which handles full sequences correctly but **cannot incorporate a non-zero initial state** since `initial_states` is never passed through to `mamba_chunk_scan_combined`.

# Related issues (and even a PR! but for Mamba1 ...) in [ `state-spaces/mamba`](https://github.com/state-spaces/mamba), all for Mamba1 except #641)

- [state-spaces/mamba#641](https://github.com/state-spaces/mamba/issues/641): "Chunk-Wise Inference does not match Full-Length Inference" (Mamba2, mamba-ssm package, same bug demonstrated with reproduction code)
- [state-spaces/mamba#536](https://github.com/state-spaces/mamba/issues/536) : "Chunked inference" (asks if fast chunked inference with state carry-over is possible)
- [state-spaces/mamba#101](https://github.com/state-spaces/mamba/issues/101): "Using ssm_state and conv_state during training" (Mamba1, same conceptual limitation)
- [state-spaces/mamba#830](https://github.com/state-spaces/mamba/pull/830): "Parallel forward with state" (PR implementing the fix for **Mamba1 only**)
- [state-spaces/mamba#488](https://github.com/state-spaces/mamba/pull/488): "Initial state support for Mamba SSM (1)" (PR implementing chunked prefill for **Mamba1 only**)

### Who can help?

@zucchini-nlp @Cyrilvallez 🤗

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
## Reproduction

```python
import torch
from transformers import Mamba2Config
from transformers.models.mamba2.modeling_mamba2 import Mamba2Model
from transformers.cache_utils import DynamicCache

# GIVEN Mamba2 model
config = Mamba2Config(
    hidden_size=64,
    num_hidden_layers=2,
    state_size=16,
    num_heads=8,
    head_dim=16,
    n_groups=1,
    expand=2,
    conv_kernel=4,
    chunk_size=8,
    vocab_size=2,
    use_bias=True,
    use_conv_bias=True,
)
model = Mamba2Model(config).eval()

B, L = 1, 8 # B = batch_size, L=seq_len
inputs_embeds = torch.randn(B, L, config.hidden_size)

# WHEN first, we process token-by-otken (seq_len=1) with result_tbt as output

cache_tbt = DynamicCache(config=config)
outputs_tbt = []
for t in range(L):
    out = model(inputs_embeds=inputs_embeds[:, t : t + 1, :], cache_params=cache_tbt, use_cache=True)
    outputs_tbt.append(out.last_hidden_state)
result_tbt = torch.cat(outputs_tbt, dim=1)  # (B, L, hidden_size)

# WHEN second, we process the same sequence in batch
cache_chunk = DynamicCache(config=config)
out_first = model(inputs_embeds=inputs_embeds[:, :1, :], cache_params=cache_chunk, use_cache=True)

out_rest = model(inputs_embeds=inputs_embeds[:, 1:, :], cache_params=cache_chunk, use_cache=True)
result_chunk = torch.cat([out_first.last_hidden_state, out_rest.last_hidden_state], dim=1)
error = (result_tbt[:, 1:, :] - result_chunk[:, 1:, :]).abs().max().item()

# THEN the results do not match, even though they should.
print(f"difference between two processing paths (should be equal) is {error}")

```

### Expected behavior

    # THEN the results do not match, even though they should.
    print(f"difference between two processing paths (should be equal) is {error}"}

Expected no difference (no absolute error)

--- Comment by Codingaditya17 at 2026-05-18T13:29:43Z ---
Hi! I've identified the three failure points in Mamba2Mixer.forward() and would like to submit a fix. My plan is to split the is_decoding branch on seq_len — for seq_len > 1, route to a chunked-prefill path using causal_conv1d_fn + mamba_chunk_scan_combined with initial_states from cache, updating both conv and SSM states after each chunk. Does this approach work or do you have a preferred direction?

--- Comment by vasqu at 2026-05-18T17:30:55Z ---
Just a quick glance: We did something similar for GDN-based models over here #45513, imo this should be very similar to Mamba2 (same principal with passing an initial state).

Is anyone up for implementing this? It would likely affect a few models that are also based on mamba2, e.g. Bamba

--- Comment by Ramshankar07 at 2026-05-19T19:06:02Z ---
Is @Ilya0527  a CI open Claw?? I'm planning to take this issue! as far as I read I think we just need to add few routing based on seq_len which is missing along with is_decoding?

--- Comment by Ramshankar07 at 2026-05-19T20:01:18Z ---

>
> Is anyone up for implementing this? It would likely affect a few models that are also based on mamba2, e.g. Bamba

I'm making a PR shortly, I'm reading about it from yesterday. I'm not sure what other tests.


ISSUE #46021: Passport stamp — toured Hugging Face on GIT42
State: closed  |  Labels: Code agent slop
Author: mmatheuspalma  |  Created: 2026-05-18T04:23:13Z

Don't panic — I warped through Hugging Face's sector on GIT42 and thought they should know.

Never heard of GIT42? It's a free, ad-free Hitchhiker-style tour of GitHub: developers as planets, galaxies to explore, profiles to visit, and passport stamps for fellow travelers. Chart a course at https://www.git42.dev.

https://www.git42.dev/u/huggingface


DeepSeek-V4 shared expert not gated.

ISSUE #46018: DeepSeek-V4 shared expert not gated.
State: open  |  Labels:
Author: yuzhongw-nvidia  |  Created: 2026-05-18T01:52:24Z

SwiGLU is not clamped for shared expert. Is it a bug or just a divergence between training and inference?

Deepseek inference ref code: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/inference/model.py#L627
transformers implementation: https://github.com/huggingface/transformers/blob/main/src/transformers/models/deepseek_v4/modeling_deepseek_v4.py#L1076-L1077

--- Comment by Rocketknight1 at 2026-05-18T13:41:00Z ---
cc @arthurzucker for the V4 port

--- Comment by ArjunSrivastava1 at 2026-05-20T09:45:17Z ---
also linking this issue to the pr for easier tracking
#45892


ISSUE #46016: Code quality scan: 164 findings (A-, 84/100)
State: closed  |  Labels:
Author: repobilitycom  |  Created: 2026-05-17T20:10:51Z

Hi @huggingface, an automated scan of this repository surfaced **164 code-quality findings** that may be worth a look.
Full details, severity filters, and per-file context are at the link below — feel free to close this issue if it isn't useful to you.

## Full interactive report

**https://repobility.com/scan/27ddd460-787e-4a71-a753-7f661c746582/**

![Live scan page](https://repobility.com/scan/27ddd460-787e-4a71-a753-7f661c746582/report.png?v=1779048608)

## At a glance

- **Score**: `84/100`  •  **Grade**: `A-`
- **Scanned**: `2026-05-17 20:10 UTC`
- **Lines of code**: 1,740,177
- **Total findings**: 164
- **Security-tagged**: 11
- **Credential / secret patterns**: 3

## Top issues, with file & line

_These are deterministic rule-based findings — the file paths and line numbers below are real and can be verified in your tree._

1. **[high]** [SEC029] Server-Side Request Forgery (SSRF) — outbound HTTP from user input: Outbound HTTP request to a user-controlled URL without allowlist validation. Attackers can probe internal services (169.254.169.254 metadata, internal Kubernetes endpoints, file:// URIs), exfiltrate data, or pivot through your network. SSRF is OWASP A10:2021 and a frequent foothold in cloud breaches. — `src/transformers/models/audio_spectrogram_transformer/convert_audio_spectrogram_transformer_original_to_pytorch.py:187`
   _Validate the URL against an allowlist BEFORE fetching:   ALLOWED = {'images.example.com', 'cdn.example.com'}   host = urlparse(url).hostname   if host not in ALLOWED: abort(400)…_
2. **[high]** [SEC029] Server-Side Request Forgery (SSRF) — outbound HTTP from user input: Outbound HTTP request to a user-controlled URL without allowlist validation. Attackers can probe internal services (169.254.169.254 metadata, internal Kubernetes endpoints, file:// URIs), exfiltrate data, or pivot through your network. SSRF is OWASP A10:2021 and a frequent foothold in cloud breaches. — `src/transformers/models/beit/convert_beit_unilm_to_pytorch.py:243`
   _Validate the URL against an allowlist BEFORE fetching:   ALLOWED = {'images.example.com', 'cdn.example.com'}   host = urlparse(url).hostname   if host not in ALLOWED: abort(400)…_
3. **[high]** [SEC035] Unbounded Resource Allocation — DoS risk: Allocating resources (buffers, recursion stack, large ranges) based on user input without an upper bound. Attackers send `size=10000000` to exhaust memory, or trigger expensive computation. CWE-770/400. Examples: CVE-2023-44487 (HTTP/2 Rapid Reset), countless YAML/XML billion-laughs variants. — `examples/pytorch/image-pretraining/run_mim_no_trainer.py:670`
   _Cap user-controlled sizes BEFORE allocation:   size = min(int(request.args.get('n', 100)), MAX_SIZE) Set framework-level limits:   Flask:    app.config['MAX_CONTENT_LENGTH'] = 1…_
4. **[high]** [SEC029] Server-Side Request Forgery (SSRF) — outbound HTTP from user input: Outbound HTTP request to a user-controlled URL without allowlist validation. Attackers can probe internal services (169.254.169.254 metadata, internal Kubernetes endpoints, file:// URIs), exfiltrate data, or pivot through your network. SSRF is OWASP A10:2021 and a frequent foothold in cloud breaches. — `src/transformers/integrations/integration_utils.py:2562`
   _Validate the URL against an allowlist BEFORE fetching:   ALLOWED = {'images.example.com', 'cdn.example.com'}   host = urlparse(url).hostname   if host not in ALLOWED: abort(400)…_
5. **[high]** Dockerfile copies the entire context without .dockerignore — `docker/transformers-gpu/Dockerfile:27`
   _Create .dockerignore before using broad context copies, or copy only the required files and directories._

See all 164 findings, with severity filters and AI fix prompts: **https://repobility.com/scan/27ddd460-787e-4a71-a753-7f661c746582/**

---

**What is this?** [Repobility](https://repobility.com) is a research project that scans public repositories with a multi-layer static analyzer (rule-based, no AI hallucinations) and learns code-quality patterns across a broad cross-repo corpus. This is **not a sales pitch** — there's no paywall, no signup required to view the report, and no payment ask. If the findings aren't useful, please close this issue and we won't post again.

**To re-run after fixes land:** paste your repo URL at [repobility.com](https://repobility.com) — fresh scan, free.

_Issue filed via the public Repobility report at https://repobility.com/scan/27ddd460-787e-4a71-a753-7f661c746582/._


Outdated examples about RAG

ISSUE #46015: Outdated examples about RAG
State: closed  |  Labels: Good First Issue, bug
Author: someone282801  |  Created: 2026-05-17T16:48:39Z

### System Info

```
C:\Users\usuario\AppData\Local\Programs\Python\Python313\python.exe: No module named transformers.__main__; 'transformers' is a package and cannot be directly executed

(.venv) C:\Users\usuario\Documents\aibased>transformers env
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\usuario\Documents\aibased\.venv\Scripts\transformers.exe\__main__.py", line 4, in <module>
    from transformers.cli.transformers import main
  File "C:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\cli\transformers.py", line 21, in <module>
    from transformers.cli.serve import Serve
  File "C:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\cli\serve.py", line 27, in <module>
    from .serving.utils import set_torch_seed
  File "C:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\cli\serving\__init__.py", line 16, in <module>
    from .server import build_server
  File "C:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\cli\serving\server.py", line 30, in <module>
    from .chat_completion import ChatCompletionHandler
  File "C:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\cli\serving\chat_completion.py", line 53, in <module>
    class TransformersCompletionCreateParamsStreaming(CompletionCreateParamsStreaming, total=False):
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'CompletionCreateParamsStreaming' is not defined
```

```
(.venv) C:\Users\usuario\Documents\aibased>C:/Users/usuario/AppData/Local/Programs/Python/Python313/python.exe -m pip list
Package               Version
--------------------- ------------
accelerate            1.13.0
aiohappyeyeballs      2.6.1
aiohttp               3.13.5
aiosignal             1.4.0
annotated-doc         0.0.4
anyio                 4.13.0
attrs                 26.1.0
bitsandbytes          0.49.2
certifi               2026.4.22
charset-normalizer    3.4.7
click                 8.3.3
colorama              0.4.6
datasets              4.8.5
dill                  0.4.1
dnspython             2.8.0
faiss-cpu             1.13.2
filelock              3.29.0
frozenlist            1.8.0
fsspec                2026.2.0
h11                   0.16.0
hf-xet                1.5.0
httpcore              1.0.9
httpx                 0.28.1
huggingface_hub       1.14.0
idna                  3.14
Jinja2                3.1.6
joblib                1.5.3
markdown-it-py        4.2.0
MarkupSafe            3.0.3
mdurl                 0.1.2
mpmath                1.3.0
multidict             6.7.1
multiprocess          0.70.19
networkx              3.6.1
numpy                 2.4.4
packaging             26.2
pandas                3.0.3
pillow                12.2.0
pip                   25.1.1
propcache             0.5.2
psutil                7.2.2
pyarrow               24.0.0
Pygments              2.20.0
pymongo               4.17.0
python-dateutil       2.9.0.post0
PyYAML                6.0.3
regex                 2026.5.9
requests              2.33.1
rich                  15.0.0
safetensors           0.7.0
scikit-learn          1.8.0
scipy                 1.17.1
sentence-transformers 5.5.0
setuptools            70.2.0
shellingham           1.5.4
six                   1.17.0
sympy                 1.14.0
threadpoolctl         3.6.0
tokenizers            0.22.2
torch                 2.12.0+cu132
torch_scatter         2.1.2
torchvision           0.27.0+cu132
tqdm                  4.67.3
transformers          5.8.0
typer                 0.25.1
typing_extensions     4.15.0
tzdata                2026.2
urllib3               2.7.0
xxhash                3.7.0
yarl                  1.23.0
```

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [x] The official example scripts
- [] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [] My own task or dataset (give details below)

### Reproduction

Copy script from https://huggingface.co/docs/transformers/model_doc/rag for example the first one:

```
from transformers import RagRetriever, RagSequenceForGeneration, RagTokenizer

tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq")
retriever = RagRetriever.from_pretrained(
    "facebook/dpr-ctx_encoder-single-nq-base", dataset="wiki_dpr", index_name="compressed"
)

model = RagSequenceForGeneration.from_pretrained(
    "facebook/rag-token-nq",
    retriever=retriever,
    attn_implementation="flash_attention_2",
 device_map="auto")
input_dict = tokenizer.prepare_seq2seq_batch("How many people live in Paris?", return_tensors="pt").to(model.device)
generated = model.generate(input_ids=input_dict["input_ids"])
print(tokenizer.batch_decode(generated, skip_special_tokens=True)[0])
```

After run, next stacktrace happens:

```
(.venv) C:\Users\usuario\Documents\aibased>C:/Users/usuario/AppData/Local/Programs/Python/Python313/python.exe c:/Users/usuario/Documents/aibased/rag/rag1example.py
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] You are using a model of type `dpr` to instantiate a model of type `rag`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating.
Traceback (most recent call last):
  File "c:\Users\usuario\Documents\aibased\rag\rag1example.py", line 5, in <module>
    retriever = RagRetriever.from_pretrained(
        "facebook/dpr-ctx_encoder-single-nq-base", dataset="wiki_dpr", index_name="compressed"
    )
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\models\rag\retrieval_rag.py", line 446, in from_pretrained
    config = kwargs.pop("config", None) or RagConfig.from_pretrained(retriever_name_or_path, **kwargs)
                                           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\configuration_utils.py", line 662, in from_pretrained
    return cls.from_dict(config_dict, **kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\configuration_utils.py", line 839, in from_dict
    config = cls(**config_dict)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\huggingface_hub\dataclasses.py", line 275, in init_with_validate
    initial_init(self, *args, **kwargs)  # type: ignore [call-arg]
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\configuration_utils.py", line 114, in __init__
    self.__post_init__(**additional_kwargs)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\models\rag\configuration_rag.py", line 111, in __post_init__
    raise ValueError(
    ...<2 lines>...
    )
ValueError: A configuration of type rag cannot be instantiated because not both `question_encoder` and `generator` sub-configurations are passed, but only {'attention_probs_dropout_prob': 0.1, 'gradient_checkpointing': False, 'hidden_act': 'gelu', 'hidden_dropout_prob': 0.1, 'hidden_size': 768, 'initializer_range': 0.02, 'intermediate_size': 3072, 'layer_norm_eps': 1e-12, 'max_position_embeddings': 512, 'model_type': 'dpr', 'num_attention_heads': 12, 'num_hidden_layers': 12, 'projection_dim': 0, 'type_vocab_size': 2, '_commit_hash': 'bb21a3c2b1656d60c6a8e920283bc40dabddadb8'}
```

One thing to note here is the not existence of `tokenizer.prepare_seq2seq_batch`.

## Second example based in custom dataset, referencing to a not defined path.

https://huggingface.co/docs/transformers/model_doc/rag#transformers.RagRetriever.example

Defined path not exists, badly referenced:

```
# To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py
```

Referenced part of sources:

```
# To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py
from transformers import RagRetriever

dataset = (
    ...
)  # dataset must be a datasets.Datasets object with columns "title", "text" and "embeddings", and it must have a supported index (e.g., Faiss or other index types depending on your setup)
retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", indexed_dataset=dataset)
```

Things to note here, gonna be better if a clearer source is provided with real dataset from which a `RagRetriever` is returned, this includes having `title`, `text` and `embeddings` generation (basically all for have a custom dataset ready to use through RagRetriever), this not means having one that is already having all columns filled, is more about having `title` and `text` but explaining too embeddings generation to same model that is provided in examples, a related source could be https://huggingface.co/learn/llm-course/chapter5/6#creating-text-embeddings and https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.add_faiss_index.example. Is too important explain whats the usage of every column used in dataset for RAG.

## Third customized script for reveal internal non existent properties from RAGConfig.

Script for reproduce the issue:

```
import torch, json
from transformers import BitsAndBytesConfig, RagRetriever, RagSequenceForGeneration, RagTokenizer
from datasets import load_dataset

fp = open("./dm-dataset-extra-reduced.json", "r")
data = json.load(fp)
fp.close()

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")

dataset = load_dataset("json", data_files="./datasets/dm-dataset-1.json", split="train")

dataset = dataset.rename_column("instruction", "title")
dataset = dataset.rename_column("output", "text")

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
#retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", indexed_dataset=dataset)

model = RagSequenceForGeneration.from_pretrained(
    "facebook/rag-token-nq",
    #retriever=retriever,
    #quantization_config=bnb,   # quantizes generator weights
    device_map="auto",
)

input_text = "How many people live in Paris?"
input_ids = tokenizer(input_text, return_tensors='pt').input_ids.to(device)

generated = model.generate(input_ids=input_ids)
print(tokenizer.batch_decode(generated, skip_special_tokens=True)[0])
```

With next stacktrace:

```
(.venv) C:\Users\usuario\Documents\aibased>c:/Users/usuario/Documents/aibased/.venv/Scripts/python.exe c:/Users/usuario/Documents/aibased/rag/ragbug3rd.py
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] Please make sure the generation config includes `forced_bos_token_id=0`.
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 711/711 [00:00<00:00, 1289.12it/s]
[transformers] RagSequenceForGeneration LOAD REPORT from: facebook/rag-token-nq
Key                                                                  | Status     |  |
---------------------------------------------------------------------+------------+--+-
rag.question_encoder.question_encoder.bert_model.pooler.dense.bias   | UNEXPECTED |  |
rag.question_encoder.question_encoder.bert_model.pooler.dense.weight | UNEXPECTED |  |

Notes:
- UNEXPECTED:   can be ignored when loading from different task/architecture; not ok if you expect identical arch.
Traceback (most recent call last):
  File "c:\Users\usuario\Documents\aibased\rag\ragbug3rd.py", line 30, in <module>
    generated = model.generate(input_ids=input_ids)
  File "c:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "c:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\models\rag\modeling_rag.py", line 943, in generate
    num_return_sequences if num_return_sequences is not None else self.config.num_return_sequences
                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\configuration_utils.py", line 434, in __getattribute__
    return super().__getattribute__(key)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: 'RagConfig' object has no attribute 'num_return_sequences'
```

There are atleast 2 of this attributes that not exist.

`num_return_sequences` and `num_beams`

### Expected behavior

No crash is expected in first example and use up to date methods for obtain answers.

Use custom dataset with `RagRetriever` with detailed information about how to make one with embeddings and getting answers from, must be related to models used in examples.

In third case, non existent properties must exist.

--- Comment by Rocketknight1 at 2026-05-18T13:58:02Z ---
Yeah, my guess is RAG needs a minimal refresh, since it doesn't use the standard `GenerationMixin`. If anyone wants to see if they can get it working again, feel free! Please try to keep the PR minimal though, especially if you use a code agent - they have a strong tendency to make 100+ LOC changes and massive comments when the real fix is like 2 LOC

--- Comment by 0Sh1khar at 2026-05-18T14:31:57Z ---
I’ve reproduced the issue locally and I’m working on a minimal fix for the outdated RAG example/documentation.

--- Comment by Sriniketh24 at 2026-05-18T14:39:34Z ---
I'd like to work on this. I've traced the root cause and have a minimal fix ready.

**Core bug:** `RagSequenceForGeneration.generate()` crashes with `AttributeError` because it accesses `self.config.num_return_sequences` and `self.config.num_beams` ([L943-945](https://github.com/huggingface/transformers/blob/main/src/transformers/models/rag/modeling_rag.py#L942-L945)), but `RagConfig` never defines these fields.

**Fix (2 lines in `modeling_rag.py`):** Replace the direct attribute access with `getattr(self.config, "num_return_sequences", 1)` and `getattr(self.config, "num_beams", 1)`. This defaults to 1 (matching the docstring) without crashing.

Note: Adding these fields directly to `RagConfig` is **not** viable — it triggers the generation utils validation error (`"You have modified the pretrained model configuration to control generation"`), since generation params belong in `generation_config`, not the model config.

**Also fixes:** Removes two stale references to the non-existent `examples/rag/use_own_knowledge_dataset.py` in `retrieval_rag.py` docstrings.

Will open a PR shortly. AI assistance (Claude Code) was used; I reviewed and tested all changes.


ISSUE #46002: NaNs in classification heads upon checkout + init
State: closed  |  Labels: bug
Author: maciejskorski  |  Created: 2026-05-16T11:45:22Z

### System Info

software
`torch=2.12.0` `transformers=5.8.1`
hardware
`NVIDIA-SMI 580.82.07              Driver Version: 580.82.07      CUDA Version: 13.0`

### Who can help?

call for help
@CyrilVallez - as model loading and init seems wrong
@stevhliu - as this can be better docummented

thanks!

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Hey, so I am seeing many models from `AutoModelForSequenceClassification`
initialised with NaNs in their heads,
so that the training can't continue.

One workaround is to force `float32` at checkout, which is non-default and not indended.

```python
model = AutoModelForSequenceClassification.from_pretrained(
    'microsoft/deberta-v3-base',
    num_labels=2,
    dtype=torch.float32, # FIXME seems to be only working with this
    # _fast_init=True,
)

model.to("cuda:0")

nan_params = [name for name, param in model.named_parameters() if param.isnan().any()]
assert len(nan_params) == 0, f"NaN weights in: {nan_params}"

# AssertionError: NaN weights in: ['pooler.dense.weight', 'classifier.weight']
```

### Expected behavior

initialised with numerical weights - no nans - from checkout, under default settings

--- Comment by Rocketknight1 at 2026-05-19T09:54:47Z ---
This should now be fixed! @maciejskorski please try installing from `main` and let us know if the problem recurs.


ISSUE #46000: 🎯 AI Agents: Claim tasks and execute them here
State: closed  |  Labels:
Author: chenyuan35  |  Created: 2026-05-16T10:04:18Z

<spam>