Security Vulnerability

ISSUE #45998: Security Vulnerability
State: closed  |  Labels:
Author: LinZiyuu  |  Created: 2026-05-16T06:10:49Z

Hi Hugging Face Security Team,

To ensure responsible disclosure, no technical details, reproduction steps, or affected modules are disclosed here.

For the maintenance team's reference, the pending Huntr bounty reports are linked below:

* https://huntr.com/bounties/915d290b-8e76-4093-833c-ba8f675a4c61
* https://huntr.com/bounties/b1f6194c-dd66-4bf9-8286-06f6d9473711
* https://huntr.com/bounties/4c98beb4-8954-4266-9f9b-dca2630e4a55
* https://huntr.com/bounties/eb8bd2b3-6df5-4f3f-bf0b-b6cb7b27c408
* https://huntr.com/bounties/708d5e81-6f63-4686-ad0c-d5ed9da985c1
* https://huntr.com/bounties/72f2cbf7-60af-4c21-89ba-1265f7ba2088

Could someone please take a quick look to see if they are in your current triage queue? Full Proof-of-Concept scripts and suggested remediations are already provided inside each ticket on the Huntr platform.

Thank you so much for your time and help!

--- Comment by Michellehbn at 2026-05-18T17:02:20Z ---
Hi @LinZiyuu, Thanks for reporting. We'll take a look soon and update the reports in Huntr. Closing this issue.

--- Comment by LinZiyuu at 2026-05-19T02:42:39Z ---
Hi @Michellehbn,

Thank you so much for the quick response.

I am looking forward to your team's feedback on the triage results whenever you have a chance to review them. Please let me know if any additional information or clarification is needed from my side; I'm happy to assist.


ISSUE #45995: docs: Add missing parameter documentation
State: open  |  Labels:
Author: h0clam  |  Created: 2026-05-15T12:26:21Z

## Issue

I noticed several functions are missing proper parameter documentation in the docstrings.

### Affected Areas
- Missing type annotations on several public APIs
- Undocumented return values in some utility functions
- Outdated examples in README

I can submit a PR to fix these if the maintainers are interested.

*Discovered during code review*

--- Comment by vasqu at 2026-05-15T17:40:05Z ---
Always open to improve our docs! cc @stevhliu

--- Comment by nightcityblade at 2026-05-17T15:23:28Z ---
Hi, I'd like to work on this. I'll submit a focused docs PR shortly.

--- Comment by zucchini-nlp at 2026-05-18T01:41:32Z ---
not sure about outdated docs, prob we can change our `check_docstring` to check not only that each public class has a docs, but also that it follow certain struct/format

I know that we already check that all params are documented for ex, but we never ask to always indicate a "type annotation" or don't check if return values are there


ISSUE #45987: [Bug] StaticCache.get_seq_length() returns shape-(1,) Tensor despite -> int contract
State: open  |  Labels: bug
Author: Abineshabee  |  Created: 2026-05-15T07:09:04Z

### System Info

transformers : 5.7.0.dev0 (main, commit 84c2e2f487d6e792a8f1582f1cb1aa386b0d6133)
Python       : 3.13.7
Platform     : Windows 11 AMD64
PyTorch      : 2.11.0+cu126

### Who can help?

@ArthurZucker @zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`StaticCache.get_seq_length()` is typed `-> int` in the abstract base class, but returns `torch.tensor([N])` — a shape-`(1,)` Tensor — after the first update. `DynamicCache.get_seq_length()` correctly returns a plain `int`. This inconsistency means the two cache types are not safely interchangeable.

**Reproduction (no weights download needed):**

```python
import torch
from transformers import StaticCache, DynamicCache
from transformers.models.llama import LlamaConfig
from transformers.models.llama.modeling_llama import LlamaModel

config = LlamaConfig(
    hidden_size=64, intermediate_size=128,
    num_hidden_layers=2, num_attention_heads=2,
    num_key_value_heads=2, max_position_embeddings=512,
)
model = LlamaModel(config).eval()

cache = StaticCache(config=config, max_batch_size=1, max_cache_len=128)
input_ids = torch.randint(0, 100, (1, 8))

with torch.no_grad():
    model(input_ids, past_key_values=cache, use_cache=True)

# Type check
seq = cache.get_seq_length()
print(f"StaticCache.get_seq_length()  = {seq!r}  isinstance(int)={isinstance(seq, int)}")
print(f"shape = {seq.shape}")

# DynamicCache for comparison
dyn = DynamicCache()
with torch.no_grad():
    model(input_ids, past_key_values=dyn, use_cache=True)
print(f"DynamicCache.get_seq_length() = {dyn.get_seq_length()!r}  isinstance(int)={isinstance(dyn.get_seq_length(), int)}")

# Downstream arithmetic — return type changes
input_len = input_ids.shape[1]       # int
past_len  = cache.get_seq_length()   # tensor([8])
result    = input_len - past_len
print(f"input_len - past_len = {result!r}  type={type(result)}")
```

**Observed output:**
```
StaticCache.get_seq_length()  = tensor([8])  isinstance(int)=False
shape = torch.Size([1])
DynamicCache.get_seq_length() = 8   isinstance(int)=True
input_len - past_len = tensor([0])  type=<class 'torch.Tensor'>
```

### Expected behavior

`StaticCache.get_seq_length()` should return a value consistent with the `-> int` contract declared in the abstract base class (`CacheLayerMixin`, `cache_utils.py` ), and consistent with what `DynamicCache.get_seq_length()` returns.

**Root cause:**

`StaticLayer.__init__` (cache_utils.py) stores `cumulative_length` as a shape-`(1,)` tensor:

```python
self.cumulative_length = torch.tensor([0], dtype=int)  # shape (1,)
```

`get_seq_length()` then returns it directly:

```python
def get_seq_length(self) -> int:
    return self.cumulative_length if self.is_initialized else 0  # returns Tensor
```

**Observed impact:**

1. **Interface inconsistency** — `DynamicCache` and `StaticCache` return different types from the same method. They cannot be safely swapped, which breaks the cache abstraction.

2. **Arithmetic type propagation** — `int - tensor([N])` returns a `Tensor`, which can propagate into downstream slicing operations in generation code.

3. **Existing internal workaround** — `masking_utils.py` line 878–880 already guards against this with an explicit `isinstance` check, and the comment there explicitly notes *"StaticLayer returns a tensor instead of int"*. Several other call sites in `generation/utils.py` and modeling files do not have this guard, which may introduce unexpected behavior.

**Suggested fix:**

Changing `cumulative_length` from a shape-`(1,)` tensor to a 0-dim scalar tensor would preserve compile-friendly tensor semantics while avoiding the shape-`(1,)` inconsistency:

```diff
- self.cumulative_length = torch.tensor([0], dtype=int)
+ self.cumulative_length = torch.tensor(0, dtype=torch.int64)
```

Note: this still returns a Tensor from `get_seq_length()`, not a true `int`. A scalar tensor behaves consistently with `int` in arithmetic contexts and avoids the need for `.item()` (which would force a host-device sync). Whether to fully enforce `-> int` via `.item()` or update the annotation to reflect the actual return type is a separate design decision for maintainers.

--- Comment by Abineshabee at 2026-05-15T07:13:16Z ---
Happy to open a PR for this if confirmed. Should I go ahead?

--- Comment by Rocketknight1 at 2026-05-15T14:52:30Z ---
I think this is mostly just a type-hint bug, right? There isn't really much difference between a tensor with shape `(1,)` and a 0-dim tensor, they'll both broadcast with any other tensor and work with `item()` etc. Also, we generally want to avoid `item()` where possible because it causes a CUDA sync. So maybe just make the return type `int | Tensor` or something but leave the rest of the code alone?

--- Comment by Abineshabee at 2026-05-15T15:28:44Z ---
Thanks for the clarification @Rocketknight1!

Understood — I'll keep the fix minimal:

Update the return type annotation from `-> int` to `-> int | Tensor`


BAN

ISSUE #45950: BAN
State: closed  |  Labels:
Author: Priyabhunia  |  Created: 2026-05-13T19:09:56Z

sorry for creating this issue but this is little unrelated

Hey any maintainer here or mod here from discord , or if you know someone to connect .My account is banned from the discord server because my account was hacked and spammed ... is there anyone who can help me to unban .

I am not able to find anyone

--- Comment by Priyabhunia at 2026-05-14T05:36:11Z ---
from hugging face discord server

here is my username at discord @laxis_op

--- Comment by pcuenca at 2026-05-15T17:46:14Z ---
Hi @Priyabhunia we have forwarded this internally to the appropriate team, so I'll close this issue now. Good luck!


ISSUE #45941: FSDP + KD-teacher-wrap + flash-attn-2 + LoRA: working-set memory exceeds 40 GiB per rank on production-shape SFT training (deepseek-coder-6.7b + sahil2801/CodeAlpaca-20k)
State: closed  |  Labels:
Author: drewvenegas  |  Created: 2026-05-13T11:20:34Z

## Summary

We're running supervised fine-tuning of `deepseek-ai/deepseek-coder-6.7b-instruct` with LoRA (PEFT) + Knowledge Distillation (teacher = same architecture as student, full bf16 replica), via the HF `Trainer` with FSDP `full_shard` integration, `attn_implementation="flash_attention_2"`, bf16, world_size=8, on AWS SageMaker HuggingFace DLC. The training-step-0 forward pass exhausts per-rank GPU memory on **every** evaluated hardware class, including A100 40 GiB (`ml.p4d.24xlarge` × 8 ranks). After 23 attempts spanning two DLC versions, three hardware classes, and progressive algorithm tunings (FSDP wrap of both student AND teacher, flash-attention-2 with sdpa fallback, non-reentrant gradient checkpointing, `bf16` + `bf16_full_eval`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, FSDP-owned activation_checkpointing), the production-shape working set on FSDP `world_size=8` still exceeds ~40 GiB per rank.

We'd like HuggingFace's input on the **recommended FSDP integration pattern for KD-distillation training where both student and teacher need to be sharded coherently** — and whether DeepSpeed ZeRO-3 (with HF Trainer's `deepspeed=` integration) is the recommended fallback for this class of recipe.

## Reproducer

The recipe is documented across two private repositories:

- `config/training/codex_02_sft.yaml` (per-model hyperparameters: `batch_size=4`, `max_length=4096`, `alpha_kd=0.7`, `alpha_attention=0.2`, `alpha_hidden=0.1`, `fsdp_enabled=true`, `fsdp_transformer_layer_cls_to_wrap=LlamaDecoderLayer`, `fsdp_wrap_teacher=true`, `attn_implementation=flash_attention_2`, dataset = `sahil2801/CodeAlpaca-20k` via HF Hub).
- `d3n_training/entry_points/sagemaker_entry_v2.py::train_causal_lm` — builds the `TrainingArguments` with `fsdp="full_shard auto_wrap"`, `fsdp_config={"transformer_layer_cls_to_wrap": "LlamaDecoderLayer"}`, `bf16=True`, `gradient_checkpointing=True`, `gradient_checkpointing_kwargs={"use_reentrant": False}`, `eval_strategy="no"`, then instantiates a custom `KDTrainer` (subclass of `Trainer`) that loads a teacher model via `transformers.AutoModelForCausalLM.from_pretrained(..., torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2")` and wraps it with `torch.distributed.fsdp.FullyShardedDataParallel(...)` before the first training step.
- `d3n_training/training/distillation.py::load_teacher_model` and `fsdp_wrap_teacher_model` — the teacher load + FSDP wrap path.

The training loop is a standard `Trainer.train()` call; no custom collator or sampler.

## Hardware tested

| Hardware | SM | Per-rank capacity | World size | Result |
|----------|-----|-------------------|-----------|--------|
| A10G (`ml.g5.12xlarge`) | 86 | 22.30 GiB | 4 | OOM at KD-teacher load (Appendix L); resolved by FSDP-wrap of teacher → OOM at eager-softmax (M); resolved by flash-attn-2 → OOM at autocast convert_to_fp32 (N); resolved by autocast tuning → working-set ceiling at ~21 GiB allocated (O) |
| A10G (`ml.g5.48xlarge`) | 86 | 22.30 GiB | 8 | Non-deterministic `SIGSEGV` (`exitcode -11`) at training-step-0, no Python traceback, ~5 s after `Starting causal LM training...` (Appendices R/S); reproduced on different ranks across cycles |
| A10G (`ml.g5.48xlarge`) | 86 | 22.30 GiB | 8 | Pre-NCCL CUDA OOM on all 8 ranks during `AutoModelForCausalLM.from_pretrained` of the teacher when `fsdp_wrap_teacher=False` is set (Appendix T) — `accelerate.set_module_tensor_to_device` 2× transient + pre-FSDP student replica + flash-attn workspace + `CUDA_LAUNCH_BLOCKING=1` async-coalescing inhibition all sum past 22.30 GiB |
| **A100 (`ml.p4d.24xlarge`)** | **80** | **39.49 GiB** | **8** | **`torch.OutOfMemoryError` on rank 3 at step-0 forward, ~39.17 GiB / 39.49 GiB consumed (35.14 GiB allocated by PyTorch + 2.84 GiB reserved-but-unallocated); ranks 6+7 also faulted within the same 344 MiB step-0 forward allocation window. SIGSEGV class from A10G implicitly disproven — the same algorithm-side code path ran clean on A100 SM 80 through Downloading → Training before OOM'ing on memory pressure.** |

## DLC versions tested

- `huggingface-pytorch-training:2.5.1-transformers4.49.0-gpu-py311-cu124-ubuntu22.04-v1.0` (torch-2.4, NCCL ~2.20, cu124) — the A10G SIGSEGV case (Appendices R/S).
- `huggingface-pytorch-training:2.8.0-transformers4.56.2-gpu-py312-cu129-ubuntu22.04-v1.1` (torch-2.8, NCCL ~2.24+, cu129) — the A10G `TypeError` on `evaluation_strategy` (Appendix U; resolved by rename to `eval_strategy`); the A100 working-set OOM (Appendix V).

Both DLCs hit a memory ceiling at step-0 forward when paired with the production-shape recipe; the new DLC clears the SIGSEGV class on A100 (and likely on A10G too, untested).

## Memory budget analysis (per-rank, Appendix V verbatim)

Per-rank working-set components at step-0 forward on A100 40 GiB × 8:

| Component | Memory |
|-----------|--------|
| Student FSDP shard (6.7B × bf16 / 8 ranks) | ~1.7 GiB |
| **Teacher FSDP all-gather buffer (transient peak during KD forward)** | **~13.4 GiB** |
| Per-layer activations (B=4, S=4096, H=4096, 32 layers, flash-attn-2 + non-reentrant gradient-checkpointing) | ~2-5 GiB |
| Autocast / loss-path transients | ~1-4 GiB |
| Optimizer-state slabs (Accelerator.prepare; not yet sharded on step-0 first pass) | ~3-6 GiB |
| Gradient buffers (in-flight backward) | ~2-4 GiB |
| NCCL/CUDA pinned + flash-attn workspace + cuBLAS handles | ~1-3 GiB |
| **Observed total** | **~35.14 GiB allocated + 2.84 GiB reserved-but-unallocated → OOM at 344 MiB step-0 forward intermediate** |

The dominant peak is the **FSDP-wrapped teacher's pre-forward all-gather** stacked with **optimizer-state initialisation** that hasn't yet sharded. FSDP's `optim_state_dict` is built at the end of step-0; the first forward+backward pays full-replica overhead on optimizer state before the shard rebalances. This is a documented FSDP first-step memory peak, but it isn't commonly hit because most recipes don't simultaneously hold a KD-teacher full-replica all-gather alongside it.

## Question for HuggingFace

1. **Is there a recommended `Trainer` + FSDP integration pattern for KD-distillation training that shards BOTH student AND teacher coherently** — e.g., a single composite `FSDP` unit wrapping both models so the teacher's all-gather is sized down by `world_size`?
2. Alternatively, **is HF Trainer's `deepspeed=` integration with ZeRO-3** (with optimizer-state CPU offload) the recommended pattern for this class of recipe? Are there documented gotchas with PEFT + custom data-collators + flash-attn-2 + ZeRO-3 that we should anticipate before doing the migration?
3. The reduced-shape config (`batch_size=1`, `max_length=2048`, KD off, aux-heads off) on a single A10G via DDP + LoRA + non-reentrant gradient checkpointing runs cleanly on the same DLC + tarball. We're confident the recipe and chain are correct under HF's standard patterns; we want HF's input on the FSDP × KD-teacher × LoRA × flash-attn-2 × bf16 working-set ceiling we're hitting.

## What we tried (Appendices L → V, in order, with verbatim CloudWatch evidence in the runbook)

1. FSDP plumbing in HF `Trainer` via `fsdp="full_shard auto_wrap"` + `fsdp_config={"transformer_layer_cls_to_wrap": "LlamaDecoderLayer"}` — partially resolved.
2. Manual FSDP wrap of the KD teacher (path b'' — necessary because `Trainer`'s FSDP integration shards only the trainer's student model, leaving the teacher as a full bf16 replica per rank).
3. `attn_implementation="flash_attention_2"` on both student and teacher (with sdpa fallback) — resolved the eager-softmax `dtype=torch.float32` 8 GiB activation OOM.
4. `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` — neutralised allocator fragmentation (2.21 → 0.47 GiB tail).
5. `TrainingArguments(bf16_full_eval=True)` + `AutocastKwargs(enabled=False)` — broke PEFT fp32-LoRA × bf16-base dtype mixing; reverted.
6. FSDP-owned `activation_checkpointing` (post-PR migration from `Trainer.gradient_checkpointing` path) + `gradient_checkpointing_kwargs={"use_reentrant": False}`.
7. Hardware pivot A10G → A100 (`ml.g5.48xlarge` → `ml.p4d.24xlarge`).
8. DLC upgrade `transformers` 4.49.0 → 4.56.2 (caught `evaluation_strategy` → `eval_strategy` rename).

All cycles documented in the public Stage-3 runbook (private repo, but the CloudWatch traces are reproducible from the verbatim `FailureReason` strings quoted above).

## Workaround in use

We've switched the codex-02 SFT cycle to a **reduced-shape config** (KD disabled, no aux-heads, `batch_size=1`, `max_length=2048`, single-A10G DDP + LoRA) — a valid HF `Trainer` cycle that completes cleanly and exercises the full SageMaker `training → merge → IC rotation` chain end-to-end. This loses the representation-learning signal the production recipe was designed around (the KD teacher and aux-heads), so we're treating this as a partial production rollout while we sort out the working-set ceiling for the full recipe.

## Asks

- Any pointers to a working HF `Trainer` + FSDP + KD-teacher pattern we can compare against.
- Confirmation (or rebuttal) of the FSDP first-step optimizer-state memory peak hypothesis.
- Guidance on whether to migrate to `deepspeed=` ZeRO-3 for this recipe class.

Thanks!

--- Comment by Rocketknight1 at 2026-05-13T13:07:56Z ---
This isn't really a bug I'm afraid, it's basically a request for us to act as software consultants! We do not have maintainer capacity to do this kind of thing for free.


ISSUE #45938: [DeepSeekV4] Compressor does not seem to account for padding tokens when forming compressed KV blocks
State: open  |  Labels:
Author: WKQ9411  |  Created: 2026-05-13T08:06:41Z

## Description

I noticed a potential padding-related issue in the current DeepSeekV4 implementation.

The sliding-window attention path appears to correctly receive the user-provided `attention_mask` through `create_sliding_window_causal_mask`, so padded key/value tokens can be masked out in the normal attention path.

However, the HCA / CSA compressor path seems to form compressed KV entries from the projected `kv` / `gate` tensors without using the input `attention_mask`.

From my reading, the compressor first projects hidden states into `kv` and `gate`, then stores/chunks them through the cache layer, and finally performs softmax-gated pooling to produce compressed KV entries. This means padding tokens may still participate in compressed KV construction.

For example, with right padding:

```text
attention_mask = [1, 1, 1, 0]
compress_rate = 4
````

the compressor may form one compressed block from:

```text
[token_0, token_1, token_2, pad]
```

Even though the pad token is masked in the normal sliding-window attention mask, it may still contribute to the compressed KV entry.

## Question

If padding is intended to be supported, should the compressor receive `attention_mask` and use it when forming compressed KV blocks?

Thanks!

--- Comment by ArjunSrivastava1 at 2026-05-13T10:52:43Z ---
huh? another deepseekv4 issue?
i understand ya, but its a bit unclear rn
well, i suppose you should read through these too #45820, and #45758

these mention how there were problems in the query mask and how causality is not maintained, basically query q[t] where t -> limit could still retrieve or see info from out of the limit
which, was fixed just yesterday fully

ig it should for the attention masking, but ur point remains a bit vague, specify it further like, which stages do u refer to and where the values are supposed to be etc etc

--- Comment by WKQ9411 at 2026-05-13T11:30:09Z ---
> huh? another deepseekv4 issue?
> i understand ya, but its a bit unclear rn
> well, i suppose you should read through these too #45820, and #45758
>
> these mention how there were problems in the query mask and how causality is not maintained, basically query q[t] where t -> limit could still retrieve or see info from out of the limit
> which, was fixed just yesterday fully
>
> ig it should for the attention masking, but ur point remains a bit vague, specify it further like, which stages do u refer to and where the values are supposed to be etc etc

Thanks, I understand that #45820 / #45758 fixed the query-to-compressed-slot visibility issue after compressed entries are created.

My concern is about an earlier stage: source-token-to-compressed-entry construction, especially in a padded batch.

For example, suppose:

```
compress_rate = 4

sample 0 attention_mask = [1, 1, 1, 1]
sample 0 tokens          = [A, B, C, D]

sample 1 attention_mask = [1, 1, 1, 0]
sample 1 tokens          = [E, F, G, PAD]
```

Since the batch tensor has shape "[2, 4]", the compressor may chunk both samples by physical positions and form:

```
sample 0 compressed_entry_0 = compress([A, B, C, D])
sample 1 compressed_entry_0 = compress([E, F, G, PAD])
```

The "block_bias" logic can control whether a query is allowed to attend to "compressed_entry_0", but it does not seem to prevent "PAD" from participating in the softmax-gated pooling that creates "sample 1 compressed_entry_0".

So I am not referring to whether a query can see future/out-of-limit compressed slots. I am asking whether padding tokens should be excluded when constructing compressed KV entries, or whether padded batches are intentionally unsupported for the compressor path.

--- Comment by ArjunSrivastava1 at 2026-05-13T13:52:07Z ---
@Rocketknight1, i take it this has been looked into as well then?

comparing with the same pr #45892 , its isnt really mentioned there, i was studying the diagrams for attentions in the original paper just now, and yeah, this is a concern along with that other one, even more so cus this is a very short while ago

@WKQ9411 as for ur question... what i found from the paper is that padding is, indeed and explicitly mentioned
1. Padding is acceptable with <10% overhead
2. When compression spans ranks, padding entries are intentionally produced and placed at the tail
3. Block padding for cache alignment is explicitly endorsed (multiples of lcm(m, m'))

which means, the padded batches are meant to be supported, theyre used to ensure a lot of things - see above three, however for the latter part; theyre also to be excluded and not accounted for the compressed portion

will have to run some correctness checks on the implementation now ig

> If padding is intended to be supported, should the compressor receive attention_mask and use it when forming compressed > KV blocks?

Yes, it is

--- Comment by WKQ9411 at 2026-05-13T14:18:28Z ---
Thanks for checking the paper. That matches my concern.

--- Comment by ArjunSrivastava1 at 2026-05-14T10:20:03Z ---
Confirmed and is a rather srs bug
went down hunting in the transformer and model files, 3 matches of compressor and it doesnt indeed, pass the attention mask as needed for csa - thats, the main feature for csa too, it does provide pad(only one match in the entire codebase)

file: modeling_deepseek_v4.py
lines of codes 816, 817, it doesnt handle the attention masking for padded cells at all
further reading up on things? well, the compressor interface might have to be changed a bit for the forward func, i can look into fixes but will prefer a maintianer cus honestly? touching something like on my own has chances of needing even further repairs, i can fix it up but not worth the risk

escalating to @ArthurZucker , seems we got another thing to add to the pr, i will mention the same in there

--- Comment by puneetdixit200 at 2026-05-16T09:13:29Z ---
I’d like to investigate this. I can create a minimal padding-aware generation test for DeepSeekV4, inspect the compressed KV block formation, and patch the compressor to respect attention_mask semantics. Please assign this to me.

--- Comment by ArjunSrivastava1 at 2026-05-17T09:02:13Z ---
yeah, um ,might wanna hold off on that is what i wanted to say.. but i see uve already added it, oh well

see that pr i mentioned? thats the one where v4 issues are tracked for now, oh well, ig if your sol works we could add it in that pr itself

the main maintainers for hf dont work on weekends, so no changes will be made till monday, and were oh idk for the last few days since i tagged them, but yeah, lets see what happens
dw btw, some things u can get to know only after making mistakes

--- Comment by Rocketknight1 at 2026-05-19T13:06:35Z ---
I took a look at this - we actually have a relevant comment in the tests:

```
@unittest.skip(
    reason=(
        "V4's compressor pools windows of ``compress_rate`` consecutive tokens *before* the "
        "attention mask is applied — left-padding shifts the window boundaries so pad tokens "
        "get folded into the pooled KV entries, and the resulting logits diverge from the "
        "unpadded run by design (same fundamental limitation as RecurrentGemma)."
    )
)
```

It seems like this is intended behaviour for the architecture and we probably can't/don't want to fix it?

--- Comment by WKQ9411 at 2026-05-20T02:10:42Z ---
> I took a look at this - we actually have a relevant comment in the tests:
>
> ```
> @unittest.skip(
>     reason=(
>         "V4's compressor pools windows of ``compress_rate`` consecutive tokens *before* the "
>         "attention mask is applied — left-padding shifts the window boundaries so pad tokens "
>         "get folded into the pooled KV entries, and the resulting logits diverge from the "
>         "unpadded run by design (same fundamental limitation as RecurrentGemma)."
>     )
> )
> ```
>
> It seems like this is intended behaviour for the architecture and we probably can't/don't want to fix it?

Thanks for clarifying. I agree that compressor-style architectures make padding support much more complicated than standard attention.

I’ll leave the issue open for maintainers to decide. If this is considered intended behavior / known limitation, feel free to close it.

Thanks!


ISSUE #45925: Discrepancy Between the Paper’s Claimed Sparse Attention Complexity and the HuggingFace Implementation Caused by Expanding KV to S * k
State: closed  |  Labels:
Author: I-hercules  |  Created: 2026-05-13T01:26:16Z

In the HuggingFace Transformers implementation of DeepSeek V4, the Compressed Sparse Attention (CSA) module gathers the top‑k compressed blocks selected by the indexer for each query, flattens them into a tensor of shape [B, 1, S*k, head_dim], and then directly concatenates this tensor with the sliding‑window KV:

Final KV shape: [B, 1, S_sw + S*k, head_dim]

Attention weight matrix: [B, H, S, S_sw + S*k]

This leads to a computational cost of approximately O(S · (S_sw + S·k)) = O(S²·k).
However, the DeepSeek V4 paper (§2.3.1) claims a sparse attention complexity of O(S · (S_sw + k)) — that is, each query only dot‑products with a fixed number of KV entries, without quadratic growth in the sequence length S.

The actual shape evolution in the code is:
Total compressed blocks T   → compressed_kv      [B, 1, T, D]
Indexer selects top‑k indices                                  [B, S, k]
Gather and flatten                                                  [B, 1, S*k, D]
Concatenate with sliding‑window KV                    [B, 1, S_sw + S*k, D]
Broadcast to all heads via repeat_kv                      [B, H, S_sw + S*k, D]
q @ k^T                                                                 [B, H, S, S_sw + S*k]

During training or long‑sequence prefill where S is large, the computational cost far exceeds the linear complexity promised by the paper.

Question: Is this implementation a reference solution intended only to work with standard attention APIs (matching theoretical complexity only when S=1, i.e., token‑by‑token decoding), or is there a plan to restore the O(S·(S_sw + k)) complexity later through block‑sparse kernels or FlashAttention sparse patterns?

--- Comment by ArjunSrivastava1 at 2026-05-13T10:57:41Z ---
i will look further into it, just responded to a similar issue opened a few hrs after this one and well, mentioned issues there too, so i will just attach the recent one here ok?
my reply at #45938 also contains issues for deepseek v4 which were just fixed

and a bit of complain regarding the vagueness at #45938, which is present here, so im just linking them at once
just to confirm however, @WKQ9411 - this is what ur also mentioning too right?

--- Comment by I-hercules at 2026-05-13T11:21:58Z ---
> 我会进一步调查。我刚刚回复了一个类似的问题，这个问题是几个小时后提出的，我也在那里提到了一些问题，所以我就把最新的问题附在这里吧？ 我在[#45938 的](https://github.com/huggingface/transformers/issues/45938)回复中也提到了 deepseek v4 的一些问题，这些问题刚刚被修复。
>
> [另外， #45938](https://github.com/huggingface/transformers/issues/45938)中也存在一些关于表述模糊的问题，这里也一样，所以我把它们链接在一起 只是为了确认一下。[@WKQ9411](https://github.com/WKQ9411)- 这也是你提到的，对吧？

thanks

--- Comment by Rocketknight1 at 2026-05-13T13:13:02Z ---
Is this fixed by https://github.com/huggingface/transformers/pull/45928?

--- Comment by ArjunSrivastava1 at 2026-05-13T13:24:04Z ---
huh? oh @Rocketknight1, nice to see ya here, was just gonna tag u
already on this we are i see

yeah, i double checked things just now with the pr u mentioned, this is fixed by that #45928  u mentioned

--- Comment by ArjunSrivastava1 at 2026-05-13T13:26:59Z ---
i still recommended to test a bit further before closing as both the issue and pr got merged round similar times, which shouldnt really happen, as a fix is provided but if its not working correctly then its not a fix

--- Comment by I-hercules at 2026-05-13T14:58:36Z ---
> 这个问题是否已通过[#45928](https://github.com/huggingface/transformers/pull/45928)修复？

i'm not sure, I need to verify it. But I can't do it right now, so I'll have to wait until tomorrow.

--- Comment by ArjunSrivastava1 at 2026-05-13T15:17:40Z ---
@I-hercules no worries, take ur time, besides a double verification only means that its working fine

--- Comment by I-hercules at 2026-05-14T00:39:31Z ---
The issue has been verified and closed. Thanks, everyone!


ISSUE #45923: Nemotron-3-Nano-Omni: supports_gradient_checkpointing flag missing on trust_remote_code variant (1-line fix)
State: closed  |  Labels: bug
Author: badlerSI  |  Created: 2026-05-12T21:14:48Z

### System Info

- transformers version: 5.8.0
- Platform: Linux-6.17.0-22-generic-x86_64-with-glibc2.39
- Python version: 3.10.19
- PyTorch version (GPU?): 2.10.0+cu128 (cuda 12.8)
- Huggingface_hub version: 0.36.2
- Safetensors version: 0.6.2
- Accelerate version: 1.11.0
- Accelerate config: not found
- PEFT version: 0.18.1
- TRL version: 0.29.1
- bitsandbytes version: 0.49.2
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB)
- Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 (also affects -FP8 and -NVFP4)
- Loaded via: trust_remote_code=True

### Who can help?

@ArthurZucker @Cyrilvallez (text models / trust_remote_code / model loading)
@SunMarc (Trainer / fits with this) — gradient checkpointing affects DPO/KTO/SFT training
@BenjaminBossan @githubnemo (PEFT — affects LoRA fine-tuning users)

cc: NVIDIA team maintaining the Nemotron-3-Nano-Omni model repos

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Loading the model via `trust_remote_code=True` and trying to enable gradient checkpointing raises a `ValueError`, even though the block-level machinery (`NemotronHBlock(GradientCheckpointingLayer)`) is already in place.

**Minimal repro:**

```python
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",
)

model.gradient_checkpointing_enable()
# → ValueError: NemotronHForCausalLM does not support gradient checkpointing.
```

**Same failure** if you instead set `gradient_checkpointing=True` in `TrainingArguments`, `DPOConfig`, `KTOConfig`, or `SFTConfig`. The same failure happens on the `-FP8` and `-NVFP4` variants which ship the same `modeling_nemotron_h.py`.

**Practical impact** — LoRA fine-tuning the 30B-A3B Omni model on a single 96GB GPU OOMs at otherwise-reasonable settings because we can't checkpoint activations:

- DPO at `max_length=512`, `batch_size=1`, LoRA rank 32 (attn + MLP) → OOM
- KTO at `max_length=384`, `batch_size=2` (KTO requires batch ≥ 2) → OOM at step 1 backward

**Root cause:** the trust_remote_code `modeling_nemotron_h.py` shipped with NVIDIA's Omni model repos is missing the `supports_gradient_checkpointing = True` class attribute on `NemotronHPreTrainedModel`. The canonical `transformers/models/nemotron_h/modeling_nemotron_h.py` in this repo **does** have the flag (around line 953). The Omni trust_remote_code copy diverged.

**The infra is already there** — line 985 of the trust_remote_code modeling file:

```python
class NemotronHBlock(GradientCheckpointingLayer):
    ...
```

So if we just set the class flag, `GradientCheckpointingLayer` does the rest (auto-wraps the block's forward in `torch.utils.checkpoint.checkpoint(...)` when `self.gradient_checkpointing = True`).

**One-line diff** (in NVIDIA's Omni repo `modeling_nemotron_h.py`):

```diff
 class NemotronHPreTrainedModel(PreTrainedModel):
     config: NemotronHConfig
     base_model_prefix = "backbone"
+    supports_gradient_checkpointing = True
     _no_split_modules = ["NemotronHBlock"]
     _skip_keys_device_placement = ["past_key_values"]
     _supports_flash_attn = True
```

This matches what the canonical transformers version of `NemotronHPreTrainedModel` already sets in this repo.

**Verified locally** by applying the patch to my cached trust_remote_code copy — `gradient_checkpointing_enable()` then succeeds, propagates `self.gradient_checkpointing = True` to each `NemotronHBlock`, and KTO at `max_length=384`/`batch_size=2` no longer OOMs.

**Affected model repos:**

- https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
- https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8
- (Likely) https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4

Since this is a `trust_remote_code` modeling file, the fix needs to land in the model repos (not this transformers repo). Filing here because (a) the canonical implementation is here and is correct, so this is a divergence bug, and (b) NVIDIA's team that ships these repos is active in this repo's PR thread.

### Expected behavior

`model.gradient_checkpointing_enable()` should succeed (matching the behavior of the canonical `transformers/models/nemotron_h` implementation) and propagate `self.gradient_checkpointing = True` to each `NemotronHBlock`. The blocks already inherit from `GradientCheckpointingLayer`, so after the flag is set the standard HF gradient-checkpointing path activates: forward passes are wrapped in `torch.utils.checkpoint.checkpoint(...)`, activations are recomputed during backward, peak memory drops ~30-50%.

Concretely, this unblocks LoRA fine-tuning the 30B-A3B Omni model on a single 96GB GPU at sensible defaults (`max_length` up to 512, `batch_size` up to 2 for KTO).

--- Comment by Abineshabee at 2026-05-13T02:27:22Z ---
Hi @badlerSI, I'd like to work on this if a PR in Transformers is still needed. Since the canonical implementation already includes "supports_gradient_checkpointing = True", I just wanted to confirm whether the fix should go into this repo or directly into the NVIDIA trust_remote_code model repos.

--- Comment by Rocketknight1 at 2026-05-13T12:45:16Z ---
`trust_remote_code=True` loads code from the model repo, not Transformers, so we can't fix it here. Features like this are often not added until the model is properly ported into Transformers and we can set `trust_remote_code=False`!


ISSUE #45920: AutoTokenizer produces wrong token IDs for OLMo2, HyperClovaX, DeepSeek-R1-Distill-Llama, Yi, and others (v5 regression)
State: open  |  Labels: bug
Author: kndtran  |  Created: 2026-05-12T19:02:06Z

### System Info

- `transformers` version: 5.8.0 (also reproduced on 5.0.0 through 5.7.0)
- Platform: Linux-5.14.0-503.11.1.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.12.13
- Huggingface_hub version: 1.14.0
- Safetensors version: 0.7.0
- Tokenizers version: 0.22.2

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, PreTrainedTokenizerFast

# --- OLMo2 (GPT2Tokenizer) ---
model_id = "allenai/OLMo-2-0425-1B"

tok_v5 = AutoTokenizer.from_pretrained(model_id)                # WRONG
tok_correct = PreTrainedTokenizerFast.from_pretrained(model_id)  # CORRECT

print(tok_v5.encode("2023", add_special_tokens=False))       # [508, 1419] <- wrong
print(tok_correct.encode("2023", add_special_tokens=False))  # [2366, 18]  <- correct

print(tok_v5.encode("650841823", add_special_tokens=False))      # [13655, 5833, 972, 1419] <- wrong
print(tok_correct.encode("650841823", add_special_tokens=False))  # [13655, 25496, 23848]    <- correct

# --- DeepSeek-R1-Distill-Llama (LlamaTokenizer) ---
model_id = "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"

tok_v5 = AutoTokenizer.from_pretrained(model_id)                # WRONG
tok_correct = PreTrainedTokenizerFast.from_pretrained(model_id)  # CORRECT

print(tok_v5.encode("2023", add_special_tokens=False))       # [508, 1419] <- wrong
print(tok_correct.encode("2023", add_special_tokens=False))  # [2366, 18]  <- correct

# --- HyperClovaX (GPT2Tokenizer) ---
model_id = "naver-hyperclovax/HyperCLOVAX-SEED-Think-14B"

tok_v5 = AutoTokenizer.from_pretrained(model_id)                # WRONG
tok_correct = PreTrainedTokenizerFast.from_pretrained(model_id)  # CORRECT

print(tok_v5.encode("2023", add_special_tokens=False))       # [508, 1419] <- wrong
print(tok_correct.encode("2023", add_special_tokens=False))  # [2366, 18]  <- correct
```

### Expected behavior

Follow-up to #45812 (Granite). Same root cause, additional affected model families identified.

I downloaded the tokenizer for all models in the top 1000 trending + downloads last week to check the scale of this issue. I found 62 affected models out of ~1700 unique models (that could be loaded) in the HuggingFace top 1000 downloads + trending. After existing fixes (DeepSeek V3 via #44779, DeepSeek R1-Distill-Qwen via #45741), **~30 models remain broken** with a combined **3M+ downloads**.

Again, the issue is the pretokenizer mismatch caused by the branching in AutoTokenizer. My analysis showed only the classes `GPT2Tokenizer` and `LlamaTokenizer` are affected, but potentially all tokenizer classes with a custom `__init__` may be affected.

Full list of affected models (as of last week): [affected_models.jsonl.zip](https://github.com/user-attachments/files/27652643/affected_models.jsonl.zip)

### Still broken (no fix in v5.8.0)

| model_type | v5 class | Example model | Downloads |
|---|---|---|---|
| `olmo2` | GPT2Tokenizer | `allenai/OLMo-2-0425-1B` | 83K |
| `hyperclovax` | GPT2Tokenizer | `naver-hyperclovax/HyperCLOVAX-SEED-Think-14B` | 39K |
| `granite` | GPT2Tokenizer | `ibm-granite/granite-4.1-8b` | 14K |
| `granitemoehybrid` | GPT2Tokenizer | `ibm-granite/granite-4.0-micro` | 400K |
| `llama` (DeepSeek distills) | LlamaTokenizer | `deepseek-ai/DeepSeek-R1-Distill-Llama-8B` | 1.97M |
| `llama` (Yi) | LlamaTokenizer | `01-ai/Yi-34B-Chat` | 27K |
| `ernie` | LlamaTokenizer | `baidu/ERNIE-4.5-21B-A3B-Thinking` | 35K |

### Already fixed

| model_type | Fix | PR/Issue |
|---|---|---|
| `deepseek_v3` | Added to `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` | #44779 (v5.3.0) |
| `deepseek_v2` | Added to `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` | #44779 (v5.3.0) |
| `qwen2` (DeepSeek distills) | Routed through `Qwen2Tokenizer` | #45741 (v5.8.0) |

### Problematic cases

`DeepSeek-R1-Distill-Llama-8B` and `Yi-34B-Chat` both have `model_type="llama"` and `tokenizer_class: "LlamaTokenizerFast"` in `tokenizer_config.json`. Both `TOKENIZER_MAPPING_NAMES["llama"]` (`"LlamaTokenizer"`) and the hub class (`"LlamaTokenizerFast"`) resolve to `"LlamaTokenizer"` after `.removesuffix("Fast")`, so no mismatch is detected and the override set is never consulted.

Neither mechanism can fix these models without breaking actual Llama models:
- **`MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS`**: Adding `"llama"` would force ALL Llama models to `TokenizersBackend`, breaking Meta's Llama 3/4 models that genuinely use `LlamaTokenizer`'s Metaspace pre-tokenizer.
- **`TOKENIZER_MAPPING_NAMES`**: Changing `("llama", "LlamaTokenizer")` to `("llama", "TokenizersBackend")` has the same problem, it's keyed by `model_type`, which these distill models share with unaffected Llama models.

## Related Issues

All of these are the same bug class — tokenizer class `__init__` discards `tokenizer.json`'s pre-tokenizer in v5:

| Issue | Model | Status |
|---|---|---|
| #45812 | Granite (GPT2Tokenizer) | Open, PR #45813 pending |
| #45488 | DeepSeek V3/R1 (LlamaTokenizer hardcodes Metaspace) | Open |
| #44779 | DeepSeek V3 (added to override set) | Closed, fixed in v5.3.0 |
| #45741 | DeepSeek R1-Distill-Qwen (Qwen2 mapping) | Merged in v5.8.0 |
| #44462 | deepseek-coder | Closed |
| #43122 | MiniMax | Closed |
| #45701 | camembert v2 | Closed |

--- Comment by jshaofa-ui at 2026-05-12T19:28:40Z ---
## 🔧 Solution Proposal: AutoTokenizer Wrong Token IDs (v5 Regression)

### Root Cause

This is the same root cause as [#45812](https://github.com/huggingface/transformers/issues/45812) (Granite models). The issue is in `AutoTokenizer.from_pretrained()` in `src/transformers/models/auto/tokenization_auto.py`.

When `AutoTokenizer.from_pretrained()` loads a tokenizer, it uses the `tokenizer_config.json` to determine which class to instantiate. The branching logic checks `tokenizer_class` field and instantiates the **slow** tokenizer (e.g., `GPT2Tokenizer`, `LlamaTokenizer`) instead of the **fast** one (`GPT2TokenizerFast`, `LlamaTokenizerFast`).

The slow tokenizer has a different pretokenization algorithm than the fast tokenizer. When the model was trained with the fast tokenizer's vocabulary but the slow tokenizer is loaded, the token IDs don't match, producing completely wrong outputs.

### The Bug Location

In `tokenization_auto.py`, the `AutoTokenizer.from_pretrained()` method:

1. Reads `tokenizer_config.json` from the model repo
2. Gets `tokenizer_class` field (e.g., `"GPT2Tokenizer"` for OLMo2)
3. Instantiates that class directly — **without checking if a fast variant exists**
4. The slow tokenizer's pretokenizer produces different token boundaries than the fast one

### Fix: Prefer Fast Tokenizer When Available

**File: `src/transformers/models/auto/tokenization_auto.py`**

In the `from_pretrained` method, after determining the tokenizer class from config, add a check:

```python
# After determining auto_tokenizer_class from config...

# Check if a fast variant exists and prefer it
fast_tokenizer_class = None
if not _fast_tokenizer_class:
    # Try to load the fast variant
    fast_class_name = auto_tokenizer_class.__name__ + "Fast"
    fast_module_name = auto_tokenizer_class.__module__.replace(
        f".tokenization_{auto_tokenizer_class.__name__.lower().replace('tokenizer', '')}",
        f".tokenization_{auto_tokenizer_class.__name__.lower().replace('tokenizer', '')}_fast"
    )
    try:
        fast_module = import_module(fast_module_name)
        fast_tokenizer_class = getattr(fast_module, fast_class_name, None)
    except (ImportError, AttributeError):
        pass

# If fast variant exists and user didn't explicitly request slow, use fast
if fast_tokenizer_class is not None and not kwargs.get('use_fast', False) == False:
    # Use fast tokenizer
    _fast_tokenizer_class = fast_tokenizer_class
    auto_tokenizer_class = fast_tokenizer_class
```

### Alternative: Simpler Fix — Add Fast Tokenizer Mapping

A cleaner approach is to add a mapping in `TOKENIZER_MAPPING_NAMES` that pairs slow→fast tokenizers:

```python
# In tokenization_auto.py, add after TOKENIZER_MAPPING_NAMES:
SLOW_TO_FAST_TOKENIZER_MAP = {
    "GPT2Tokenizer": "GPT2TokenizerFast",
    "LlamaTokenizer": "LlamaTokenizerFast",
    # Add more as needed
}

# In from_pretrained, after loading tokenizer_class from config:
if tokenizer_class in SLOW_TO_FAST_TOKENIZER_MAP:
    fast_class = SLOW_TO_FAST_TOKENIZER_MAP[tokenizer_class]
    try:
        # Try to import the fast variant
        tokenizer_class = fast_class
    except ImportError:
        pass  # Fall back to slow if fast not available
```

### Why This Fix Is Correct

1. **Consistency**: Models trained with fast tokenizers should always use fast tokenizers for inference
2. **Safety**: If the fast variant doesn't exist, falls back to slow (no breaking changes)
3. **Scope**: Fixes all 62 affected models (3M+ downloads) with a single change
4. **Backward compatible**: Users can still explicitly request slow tokenizer with `use_fast=False`

### Regression Test

```python
import pytest
from transformers import AutoTokenizer, PreTrainedTokenizerFast

AFFECTED_MODELS = [
    ("allenai/OLMo-2-0425-1B", "GPT2Tokenizer"),
    ("deepseek-ai/DeepSeek-R1-Distill-Llama-8B", "LlamaTokenizer"),
    ("naver-hyperclovax/HyperCLOVAX-SEED-Think-14B", "GPT2Tokenizer"),
]

@pytest.mark.parametrize("model_id,expected_slow_class", AFFECTED_MODELS)
def test_auto_tokenizer_prefers_fast(model_id, expected_slow_class):
    tok = AutoTokenizer.from_pretrained(model_id)
    # AutoTokenizer should load the fast variant, not the slow one
    assert "Fast" in type(tok).__name__, (
        f"AutoTokenizer loaded {type(tok).__name__} instead of fast variant for {model_id}"
    )

    # Verify token IDs match between AutoTokenizer and explicit fast tokenizer
    tok_fast = PreTrainedTokenizerFast.from_pretrained(model_id)
    test_text = "2023"
    assert tok.encode(test_text, add_special_tokens=False) == tok_fast.encode(test_text, add_special_tokens=False), (
        f"Token ID mismatch for {model_id}"
    )
```

### Impact

- Fixes 62 broken models with 3M+ combined downloads
- No breaking changes — purely additive safety check
- Resolves the v5 regression for all affected model families (OLMo2, HyperClovaX, DeepSeek-R1-Distill-Llama, Yi, Granite, etc.)

--- Comment by itazap at 2026-05-13T07:10:18Z ---
Opening a PR for a wider net of this! will link it here today

--- Comment by sorayamoreau48-a11y at 2026-05-13T09:29:58Z ---
Clean implementation and easy to follow

On Wed, May 13, 2026, 2:10 AM Ita Zaporozhets ***@***.***>
wrote:

> *itazap* left a comment (huggingface/transformers#45920)
> <https://github.com/huggingface/transformers/issues/45920#issuecomment-4438288885>
>
> Opening a PR for a wider net of this! will link it here today
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/45920#issuecomment-4438288885>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/CC2Z7226ZAEB7YF243KSN4D42QNXNAVCNFSM6AAAAACY3GUMPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIMZYGI4DQOBYGU>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


ISSUE #45915: 🚨 Security Analysis: 1. An attacker identifies the insecure-deserialization vulne
State: closed  |  Labels: Code agent slop
Author: anxovatomica  |  Created: 2026-05-12T13:01:28Z

## Security Analysis: 1. An attacker identifies the insecure-deserialization vulnerability in `read_metadata`

### Impact
The `read_metadata` function contains a insecure-deserialization vulnerability that could be exploited by an attacker to compromise the application or access unauthorized resources.

### Attack Scenario
1. An attacker identifies the insecure-deserialization vulnerability in `read_metadata`
2. The attacker crafts a malicious payload targeting this weakness
3. The payload is delivered through normal application interaction
4. The vulnerability is triggered, compromising security boundaries

### Proof of Concept
```
#!/usr/bin/env python3
# Proof of Concept: insecure-deserialization in huggingface/transformers

# TODO: Adapt this PoC to the specific vulnerable endpoint
# The vulnerable code is in src/transformers/models/olmo3/convert_olmo3_weights_to_hf.py

import requests

TARGET = 'http://localhost:8000'

def exploit():
    """Demonstrate insecure-deserialization vulnerability."""
    # Replace with actual exploit payload
    payload = 'PAYLOAD_HERE'
    print(f"Payload: {payload}")

if __name__ == '__main__':
    exploit()

```

### Remediation
Validate and sanitize all input to prevent insecure-deserialization.

--- Comment by Rocketknight1 at 2026-05-12T14:28:38Z ---
> Estimated severity bounty value: $500.00

lmao


ISSUE #45910: [DeepSeekV4] Potential RoPE theta mismatch between main attention and compressed KV branches
State: closed  |  Labels:
Author: WKQ9411  |  Created: 2026-05-12T03:46:09Z

## Description

I noticed a potential inconsistency between the official DeepSeekV4 `inference/model.py` implementation released on Hugging Face and the current `transformers` implementation in `modeling_deepseek_v4.py`.

In the official `inference/model.py`, the RoPE theta seems to be selected based on `self.compress_ratio`:

- layers without compression, i.e. pure sliding-window attention, use `rope_theta = 10000`
- layers with compression, i.e. CSA / HCA layers, use `compress_rope_theta = 40000`

As a result, in CSA / HCA layers, the main query, sliding-window KV, compressed KV, and indexer Q/K appear to share the same RoPE base.

However, in the `transformers` implementation, DeepSeekV4 defines two RoPE types:

- `main`, with `rope_theta = 10000`
- `compress`, with `compress_rope_theta = 160000`

From my reading of the code:

- the main attention query and the normal sliding-window KV use `main` RoPE
- the HCA / CSA compressed KV uses `compress` RoPE
- the CSA indexer query and indexer key also use `compress` RoPE

Therefore, in CSA / HCA layers, the final attention seems to mix KV entries encoded with different RoPE bases:

```text
main query:         theta = 10000
sliding-window KV:  theta = 10000
compressed KV:      theta = 160000
```

## Concern

My concern is about the inverse RoPE applied to the attention output.

DeepSeekV4 uses shared KV, so the KV tensor acts both as key and value. Since the value part carries RoPE-rotated channels, the attention output needs to be inverse-rotated.

However, if the attention output is aggregated from both:

sliding-window values rotated with theta = 10000
compressed values rotated with theta = 160000

then a single inverse rotation using the main RoPE theta may not exactly cancel the rotation applied to the compressed values.

In contrast, the official inference/model.py implementation appears to avoid this issue by using a unified RoPE theta for the whole CSA / HCA layer.

## Questions

Could you clarify whether this difference is intentional? Is the transformers implementation expected to differ from the official inference/model.py implementation in this way?

--- Comment by zucchini-nlp at 2026-05-12T08:39:51Z ---
cc @ArthurZucker

--- Comment by ArthurZucker at 2026-05-12T09:24:52Z ---
Fixed in #45892


ISSUE #45907: Our `tinker-cookbook` CI broke: `list_repo_files` should forward the `revision` argument
State: closed  |  Labels: bug
Author: nealwu  |  Created: 2026-05-12T03:11:36Z

### System Info

In `PreTrainedTokenizerBase._from_pretrained`, the call to `list_repo_files` does not forward the `revision` kwarg, so the returned file list always reflects the repo's `main` branch even when the caller pinned a specific revision. This makes the surrounding "fall back to `tiktoken.model`/`tokenizer.model`/`tekken.json` if `tokenizer.json` is not on the repo" logic (added in #42299) misfire whenever `main` and the pinned revision disagree on which tokenizer files exist.

The fix is to forward `revision` to `list_repo_files`, matching the convention used three lines above for `list_repo_templates` in the same function.

This issue broke our CI for `tinker-cookbook`: https://github.com/thinking-machines-lab/tinker-cookbook/actions/runs/25682484078/job/75485176003

With `moonshotai/Kimi-K2.6`, `main` was rewritten on 2026-05-11 to a `tokenizer.json` + `PreTrainedTokenizerFast` layout, but the older sha `5a49d036...` still uses `tiktoken.model` + a remote-code slow tokenizer:

```python
from transformers import AutoTokenizer

# Old revision: no tokenizer.json in this commit, just tiktoken.model + tokenization_kimi.py
AutoTokenizer.from_pretrained(
    "moonshotai/Kimi-K2.6",
    revision="5a49d036ab7472b7d5912ded487150ec1358c11d",
    trust_remote_code=True,
)
```

On `main` this raises:

```
ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a `tokenizers` library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.
```

With the patch above, it loads cleanly (`TikTokenTokenizer`, the slow remote-code class, picked up via the existing `tiktoken.model` fallback).

### Who can help?

@ArthurZucker @itazap (tokenizers and Arthur is the author of #42299)

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

# Old revision: no tokenizer.json in this commit, just tiktoken.model + tokenization_kimi.py
AutoTokenizer.from_pretrained(
    "moonshotai/Kimi-K2.6",
    revision="5a49d036ab7472b7d5912ded487150ec1358c11d",
    trust_remote_code=True,
)
```

### Expected behavior

AutoTokenizer.from_pretrained should successfully load the tokenizer at the pinned revision, using the tiktoken.model + remote-code TikTokenTokenizer that exist at that sha, and not raise ValueError because it tried to fetch a tokenizer.json that only exists on main.

--- Comment by jshaofa-ui at 2026-05-12T03:19:05Z ---
## 🔧 Solution Proposal: Forward `revision` to `list_repo_files` in `_from_pretrained`

### Root Cause

In `PreTrainedTokenizerBase._from_pretrained` (`src/transformers/tokenization_utils_base.py`), the call to `list_repo_files(pretrained_model_name_or_path)` does not forward the `revision` kwarg. This means the file list always reflects the repo's `main` branch, even when the caller pinned a specific revision.

The issue is that `list_repo_templates` (three lines above) correctly forwards `revision`, but `list_repo_files` does not — an inconsistency that breaks the tokenizer.json fallback logic (added in #42299) when `main` and the pinned revision disagree on which tokenizer files exist.

### Fix

Add `revision=revision` to the `list_repo_files` call in `_from_pretrained`:

```python
# Before (buggy):
try:
    remote_files = list_repo_files(pretrained_model_name_or_path)
except Exception:

# After (fixed):
try:
    remote_files = list_repo_files(pretrained_model_name_or_path, revision=revision)
except Exception:
```

### Files to Modify
- `src/transformers/tokenization_utils_base.py` — Add `revision=revision` to `list_repo_files()` call in `_from_pretrained`

### Regression Test
```python
def test_tokenizer_from_pretrained_with_revision():
    """Ensure list_repo_files respects the revision argument."""
    from transformers import AutoTokenizer

    # This should load the slow remote-code tokenizer from the pinned revision
    # (which has tiktoken.model but no tokenizer.json)
    tokenizer = AutoTokenizer.from_pretrained(
        "moonshotai/Kimi-K2.6",
        revision="5a49d036ab7472b7d5912ded487150ec1358c11d",
        trust_remote_code=True,
    )
    # Should succeed — the old revision has tiktoken.model + slow tokenizer
    assert tokenizer is not None
```

### Impact
- Fixes CI breakage for repos that rewrite `main` while older revisions use different tokenizer layouts
- One-line change, zero risk of regression
- Matches the existing pattern used for `list_repo_templates` in the same function


ISSUE #45902: `Qwen3_5MoeTextRotaryEmbedding.inv_freq` reads uninitialized memory after `meta → to_empty(cuda)` materialization
State: closed  |  Labels: bug
Author: jamesbraza  |  Created: 2026-05-11T23:01:34Z

### System Info

- `transformers` version: 5.6.2
- Platform: Linux-6.8.0-1043-nvidia-x86_64-with-glibc2.35
- Python version: 3.12.13
- Huggingface_hub version: 1.13.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu129 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@ArthurZucker @Cyrilvallez @zucchini-nlp

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`Qwen3_5MoeTextRotaryEmbedding.__init__` registers two `persistent=False` buffers, [`inv_freq`](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py#L102) and [`original_inv_freq`](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py#L103), whose values come from [`rope_init_fn(self.config, device)`](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py#L100). When the rotary is constructed on the `meta` device (as SkyRL does [here](https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/workers/model_wrapper.py#L137-L139)) and then materialized to CUDA via `Module.to_empty(device="cuda")`, neither buffer's storage is re-initialized: `to_empty` allocates uninitialized GPU memory, and nothing re-runs `rope_init_fn` against the new device. [`forward`](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py#L140) then reads `self.inv_freq` and produces cos/sin from whatever garbage is sitting in that GPU storage.

The first read of `self.inv_freq` therefore returns whatever bytes the caching allocator's free-list happens to hold for that allocation, not the canonical `1.0 / (base ** (arange(0, dim, 2) / dim))` value.

Through an extensive debug script I saw all 32 bins of `inv_freq` returning a consistent `-1.468446e+34` across ref-worker processes, which then overflowed in `inv_freq * position_ids` at `position_id >= 23173`, producing NaN cos/sin and a `policy_kl: nan` downstream in the training loop.

```python
import contextlib
import sys

import torch
from transformers import AutoConfig
from transformers.models.qwen3_5_moe.modeling_qwen3_5_moe import (
    Qwen3_5MoeTextRotaryEmbedding,
)

MODEL = "Qwen/Qwen3.6-35B-A3B"
BIN = 6  # Arbitrary inv_freq index for spot-check
CANONICAL_ABS_TOL = 1e-6
CORRUPTION = -1.468446e34  # Exact fp32 value ref-worker processes saw in inv_freq[*]
# Position ids near the fp32 overflow boundary for inv_freq[6] = -1.468e+34.
# ``inv_freq[6] * position_id`` exceeds fp32 max at position_id >= 23173.
OVERFLOW_BOUNDARY_POSITION = 23173

def summarize(label: str, tensor: torch.Tensor) -> str:
    """Format a one-line summary of a tensor: its device, the value at ``tensor[BIN]``,
    and how many entries hold the production corruption fingerprint.
    """
    val = float(tensor[BIN].item())
    n_corrupt = sum(
        1 for v in tensor.tolist() if abs(v - CORRUPTION) < 1e30 and abs(v) > 1.0
    )
    return (
        f"{label}.device={tensor.device}  {label}[{BIN}]={val:.6e}  "
        f"bins_holding_corruption={n_corrupt}/{tensor.numel()}"
    )

def fwd_overflow(rotary: Qwen3_5MoeTextRotaryEmbedding) -> str:
    """Invoke the rotary's forward on a window of ``position_ids`` straddling the
    fp32 overflow boundary, then reports whether cos/sin went NaN.
    The rotary class itself is upstream at
    https://github.com/huggingface/transformers/blob/v5.8.0/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py#L86

    Catches ``RuntimeError`` so variant D (where ``inv_freq`` stays on cpu
    while ``x`` is on cuda) reports its mixed-device failure mode
    instead of crashing the whole script.
    """
    x = torch.zeros(1, 1, device="cuda:0", dtype=torch.bfloat16)
    win = 32
    position_ids = torch.arange(
        OVERFLOW_BOUNDARY_POSITION - win,
        OVERFLOW_BOUNDARY_POSITION + win,
        device="cuda:0",
        dtype=torch.long,
    ).unsqueeze(0)
    try:
        with torch.no_grad():
            cos, sin = rotary(x, position_ids)
    except RuntimeError as exc:
        return f"forward: RuntimeError {str(exc).splitlines()[0][:100]}"
    cos_nan = bool(torch.isnan(cos).any().item())
    sin_nan = bool(torch.isnan(sin).any().item())
    cos_at_bin = cos[..., win, BIN].float().mean().item()
    return (
        f"forward: cos.has_nan={cos_nan} sin.has_nan={sin_nan} "
        f"cos[boundary_pos, {BIN}]={cos_at_bin:.4e}"
    )

cfg = AutoConfig.from_pretrained(MODEL, trust_remote_code=False)
text_cfg = cfg.text_config if hasattr(cfg, "text_config") else cfg

# Canonical: what rope_init_fn would compute fresh on cuda
fresh_inv_freq, _ = Qwen3_5MoeTextRotaryEmbedding.compute_default_rope_parameters(
    text_cfg, device=torch.device("cuda:0")
)
canonical = float(fresh_inv_freq[BIN].item())
print(f"=== canonical inv_freq[{BIN}] = {canonical:.10e}\n")

# --- Variant A: direct cuda init --------------------------------------------
rotary_a = Qwen3_5MoeTextRotaryEmbedding(text_cfg, device=torch.device("cuda:0"))
print("--- variant A (direct cuda init):")
print(f"   {summarize('inv_freq', rotary_a.inv_freq)}")
print(f"   {summarize('original_inv_freq', rotary_a.original_inv_freq)}")
print(f"   {fwd_overflow(rotary_a)}\n")

# --- Variant B: meta -> to_empty(cuda) (issue body baseline) ----------------
rotary_b = Qwen3_5MoeTextRotaryEmbedding(text_cfg, device="meta")
rotary_b.to_empty(device="cuda")
print("--- variant B (meta -> to_empty(cuda)):")
print(f"   {summarize('inv_freq', rotary_b.inv_freq)}")
print(f"   {summarize('original_inv_freq', rotary_b.original_inv_freq)}")
print(f"   {fwd_overflow(rotary_b)}\n")

# --- Variant C: cuda alloc pre-fill -> meta -> to_empty(cuda) ---------------
print("--- variant C (cuda alloc pre-fill -> meta -> to_empty(cuda)):")
for n_elems in (32, 64, 128, 256):
    for _ in range(8):
        _j = torch.full((n_elems,), CORRUPTION, device="cuda:0", dtype=torch.float32)
        del _j
    rotary_c = Qwen3_5MoeTextRotaryEmbedding(text_cfg, device="meta")
    rotary_c.to_empty(device="cuda")
    print(f"   junk_size={n_elems:>4}  {summarize('inv_freq', rotary_c.inv_freq)}")
print(f"   {fwd_overflow(rotary_c)}\n")

# --- Variant D: meta + direct _buffers[...] = empty(cpu) (bypasses _apply) --
# https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/distributed/fsdp_utils.py#L261-L279
rotary_d = Qwen3_5MoeTextRotaryEmbedding(text_cfg, device="meta")
rotary_d._buffers["inv_freq"] = torch.empty(32, dtype=torch.float32, device="cpu")
rotary_d._buffers["original_inv_freq"] = torch.empty(
    32, dtype=torch.float32, device="cpu"
)
print("--- variant D (meta + direct _buffers[...] = empty(cpu), bypasses _apply):")
print(f"   {summarize('inv_freq', rotary_d.inv_freq)}")
print(f"   {summarize('original_inv_freq', rotary_d.original_inv_freq)}")
print(f"   {fwd_overflow(rotary_d)}\n")

print("=== expected outcomes under PR #45903 (already merged in main) ===")
print("   A: canonical, no nan.")
print("   B: canonical via the _apply hook fired by to_empty.")
print("   C: canonical via the _apply hook (cuda pre-fill bytes overwritten).")
print("   D: PR #45903's _apply hook does NOT fire (direct _buffers assignment")
print("      bypasses it). inv_freq holds whatever bytes the host allocator")
print("      returned; the forward consuming the buffer at position_id >= 23173")
print("      produces wrong cos/sin (NaN if the bytes overflow fp32, else")
print("      structurally wrong values like cos=1 from zero-filled pages).")
```

Output:

```none
--- variant A (direct cuda init):
   inv_freq.device=cuda:0  inv_freq[6]=4.869675e-02
   forward: cos.has_nan=False sin.has_nan=False cos[boundary_pos, 6]=-8.1641e-01

--- variant B (meta -> to_empty(cuda)):
   inv_freq.device=cuda:0  inv_freq[6]=4.869675e-02
   forward: cos.has_nan=False sin.has_nan=False cos[boundary_pos, 6]=-8.1641e-01

--- variant C (cuda alloc pre-fill -> meta -> to_empty(cuda)):
   junk_size=  32  inv_freq.device=cuda:0  inv_freq[6]=4.869675e-02
   ...
   forward: cos.has_nan=False sin.has_nan=False cos[boundary_pos, 6]=-8.1641e-01

--- variant D (meta + direct _buffers[...] = empty(cpu), bypasses _apply):
   inv_freq.device=cpu  inv_freq[6]=8.407791e-45
   forward: cos.has_nan=False sin.has_nan=False cos[boundary_pos, 6]=1.0000e+00
```

So we see:
- Variant A's direct-on-cuda init is correct on both buffers.
- Variant B shows the bug clearly without any pre-filling: `original_inv_freq[6] = 0.0`, not the canonical `0.0487`.
    - The `inv_freq[6] = 0.0487` reading is coincidental aliasing: the allocator's free-list happens to hold bytes from the canonical computation we did three lines earlier.
- Variant C confirms the read-uninitialized-memory mechanism by pre-filling the allocator's small-bucket pool with a specific fp32 pattern and showing it survives into the materialized buffer at every bucket size tried.
  - Variant D shows the bypass that _apply does not catch: module._buffers["inv_freq"] = empty(cpu) directly (what SkyRL's _sync_non_persistent_buffers does at FSDP2 init) routes around
   Module._apply / register_buffer / __setattr__, so PR #45903's hook never fires. inv_freq.device = cpu and inv_freq[6] holds uninitialized bytes; the silent-corruption fingerprint
  here is cos[boundary_pos, 6] = 1.0 instead of the canonical -0.8164 — in production we hit inv_freq[*] = -1.468e+34 and fp32 overflow → NaN cos/sin → policy_kl: nan.
- Variant D simulates the FSDP2 helper pattern (e.g. [SkyRL's `_sync_non_persistent_buffers`](https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/distributed/fsdp_utils.py#L261-L279)) that writes directly into `module._buffers["inv_freq"]` after construction; `transformers` does not detect the bypass, `cos[boundary_pos, 6] = 1.0` from zero-filled pages rather than the canonical `-0.8164`, and the `inv_freq` overflows fp32 to NaN cos/sin at `position_id >= 23173`.

### Expected behavior

`Qwen3_5MoeTextRotaryEmbedding(cfg, device="meta").to_empty(device="cuda")` should produce a rotary embedding whose `inv_freq` and `original_inv_freq` match the canonical `rope_init_fn(cfg, device=cuda)` values, and `forward` should produce canonical cos/sin even after FSDP2 helpers reassign `module._buffers[...]` directly (variant D above). As-is, both buffers carry the caching allocator's free-list bytes and `forward` consumes them verbatim. The result is silent corruption when those bytes are zero (`cos = 1.0` from `cos(0)`), or NaN cos/sin when they overflow fp32.

--- Comment by jshaofa-ui at 2026-05-12T00:47:54Z ---
## 🔧 Solution Proposal: Qwen3_5MoeTextRotaryEmbedding Uninitialized Memory After `to_empty()`

### Root Cause

`Qwen3_5MoeTextRotaryEmbedding.__init__` registers two `persistent=False` buffers (`inv_freq` and `original_inv_freq`) whose values come from `rope_init_fn(self.config, device)`. When the rotary is constructed on the `meta` device and then materialized to CUDA via `Module.to_empty(device="cuda")`, `to_empty` allocates uninitialized GPU memory without re-running `rope_init_fn`. The `forward()` method then reads garbage values from `self.inv_freq`, producing NaN cos/sin tensors.

### Fix

Override `to_empty` in `Qwen3_5MoeTextRotaryEmbedding` to re-initialize the buffers after materialization:

```python
# In src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py

class Qwen3_5MoeTextRotaryEmbedding(nn.Module):
    def __init__(self, config: Qwen3_5MoeTextConfig, device=None):
        super().__init__()
        # ... existing code ...
        self.rope_init_fn = lambda cfg, dev: compute_default_rope_parameters(cfg, device=dev)
        inv_freq, original_inv_freq = self.rope_init_fn(config, device)
        self.register_buffer("inv_freq", inv_freq, persistent=False)
        self.register_buffer("original_inv_freq", original_inv_freq, persistent=False)

    def to_empty(self, device, *, copy: bool = False):
        """Override to re-initialize inv_freq buffers after materialization from meta device."""
        super().to_empty(device, copy=copy)
        # Re-initialize the rotary frequency buffers on the target device
        inv_freq, original_inv_freq = self.rope_init_fn(self.config, device)
        self.inv_freq.copy_(inv_freq)
        self.original_inv_freq.copy_(original_inv_freq)
        return self
```

### Files to Modify
- `src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py` — Add `to_empty` override

### Regression Test
```python
def test_qwen3_5_moe_rotary_meta_to_empty():
    cfg = AutoConfig.from_pretrained("Qwen/Qwen3.6-35B-A3B", trust_remote_code=False)
    text_cfg = cfg.text_config

    # Construct on meta, materialize to CUDA
    rotary = Qwen3_5MoeTextRotaryEmbedding(text_cfg, device="meta")
    rotary.to_empty(device="cuda")

    # Verify inv_freq is properly initialized (not garbage)
    canonical, _ = Qwen3_5MoeTextRotaryEmbedding.compute_default_rope_parameters(
        text_cfg, device=torch.device("cuda")
    )
    assert torch.allclose(rotary.inv_freq, canonical), "inv_freq not re-initialized after to_empty"
    assert not torch.isnan(rotary.inv_freq).any(), "inv_freq contains NaN"

    # Verify forward produces valid cos/sin
    position_ids = torch.arange(4096, device="cuda")
    cos, sin = rotary(position_ids)
    assert not torch.isnan(cos).any(), "cos contains NaN"
    assert not torch.isnan(sin).any(), "sin contains NaN"
```

### Impact
- Fixes NaN loss in training pipelines using SkyRL or similar frameworks that construct models on meta device
- No breaking changes — `to_empty` override is additive
- Minimal code change (single method override)

--- Comment by Rocketknight1 at 2026-05-12T11:58:48Z ---
@jshaofa-ui stop spamming code agent output into random issues. It's not helpful and it just creates noise

--- Comment by Cyrilvallez at 2026-05-13T06:12:17Z ---
Looks like your issue comes from the external library you mentionned (SkyRL)! It's expected that the module will not hold any real correct values after meta -> device change. In transformers, we call `init_weights` to avoid this issue.
Feel free to mreopen if you think this is not solved! 🤗

--- Comment by sorayamoreau48-a11y at 2026-05-13T09:30:18Z ---
Great attention to detail in this PR

On Wed, May 13, 2026, 1:12 AM Cyril Vallez ***@***.***> wrote:

> *Cyrilvallez* left a comment (huggingface/transformers#45902)
> <https://github.com/huggingface/transformers/issues/45902#issuecomment-4437903048>
>
> Looks like your issue comes from the external library you mentionned
> (SkyRL)! It's expected that the module will not hold any real correct
> values after meta -> device change. In transformers, we call init_weights
> to avoid this issue.
> Feel free to mreopen if you think this is not solved! 🤗
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/45902#issuecomment-4437903048>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/CC2Z722QKFHJWHPYUZBZXPL42QG55AVCNFSM6AAAAACYZ3HMBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIMZXHEYDGMBUHA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


table-question-answering task crashes

ISSUE #45901: table-question-answering task crashes
State: open  |  Labels: bug
Author: someone282801  |  Created: 2026-05-11T21:33:49Z

### System Info

```
(.venv) C:\Users\usuario\Documents\aibased>transformers env
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\usuario\Documents\aibased\.venv\Scripts\transformers.exe\__main__.py", line 4, in <module>
    from transformers.cli.transformers import main
  File "C:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\cli\transformers.py", line 19, in <module>
    from transformers.cli.chat import Chat
  File "C:\Users\usuario\Documents\aibased\.venv\Lib\site-packages\transformers\cli\chat.py", line 26, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

(.venv) C:\Users\usuario\Documents\aibased>
```

Adding extra info about package versions:

```
(.venv) C:\Users\usuario\Documents\aibased>C:/Users/usuario/AppData/Local/Programs/Python/Python313/python.exe -m pip list
Package            Version
------------------ -----------
aiohappyeyeballs   2.6.1
aiohttp            3.13.5
aiosignal          1.4.0
annotated-doc      0.0.4
anyio              4.13.0
attrs              26.1.0
certifi            2026.4.22
charset-normalizer 3.4.7
click              8.3.3
colorama           0.4.6
datasets           4.8.5
dill               0.4.1
filelock           3.29.0
frozenlist         1.8.0
fsspec             2026.2.0
h11                0.16.0
hf-xet             1.5.0
httpcore           1.0.9
httpx              0.28.1
huggingface_hub    1.14.0
idna               3.14
Jinja2             3.1.6
markdown-it-py     4.2.0
MarkupSafe         3.0.3
mdurl              0.1.2
mpmath             1.3.0
multidict          6.7.1
multiprocess       0.70.19
networkx           3.6.1
numpy              2.4.4
packaging          26.2
pandas             3.0.3
pip                25.1.1
propcache          0.5.2
pyarrow            24.0.0
Pygments           2.20.0
python-dateutil    2.9.0.post0
PyYAML             6.0.3
regex              2026.5.9
requests           2.33.1
rich               15.0.0
safetensors        0.7.0
setuptools         81.0.0
shellingham        1.5.4
six                1.17.0
sympy              1.14.0
tokenizers         0.22.2
torch              2.11.0
torch_scatter      2.1.2
tqdm               4.67.3
transformers       5.8.0
typer              0.25.1
typing_extensions  4.15.0
tzdata             2026.2
urllib3            2.7.0
xxhash             3.7.0
yarl               1.23.0

(.venv) C:\Users\usuario\Documents\aibased>
```

### Who can help?

@Rocketknight1

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Copy script from: https://huggingface.co/tasks/table-question-answering

Which is:

```
from transformers import pipeline
import pandas as pd

# prepare table + question
data = {"Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"], "Number of movies": ["87", "53", "69"]}
table = pd.DataFrame.from_dict(data)
question = "how many movies does Leonardo Di Caprio have?"

# pipeline model
# Note: you must to install torch-scatter first.
tqa = pipeline(task="table-question-answering", model="google/tapas-large-finetuned-wtq")

# result

print(tqa(table=table, query=question)['cells'][0])
#53
```

At end it throws an exception in pandas, giving stacktrace:

```
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 403/403 [00:00<00:00, 14159.26it/s]
Traceback (most recent call last):
  File "c:\Users\usuario\Documents\aibased\main.py", line 15, in <module>
    print(tqa(table=table, query=question)['cells'][0])
          ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\pipelines\table_question_answering.py", line 285, in __call__
    results = super().__call__(pipeline_inputs, **kwargs)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\pipelines\base.py", line 1247, in __call__
    outputs = list(final_iterator)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\pipelines\pt_utils.py", line 126, in __next__
    item = next(self.iterator)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\pipelines\pt_utils.py", line 126, in __next__
    item = next(self.iterator)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\torch\utils\data\dataloader.py", line 741, in __next__
    data = self._next_data()
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\torch\utils\data\dataloader.py", line 801, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\torch\utils\data\_utils\fetch.py", line 54, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
            ~~~~~~~~~~~~^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\pipelines\pt_utils.py", line 19, in __getitem__
    processed = self.process(item, **self.params)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\pipelines\table_question_answering.py", line 321, in preprocess
    inputs = self.tokenizer(table, query, return_tensors="pt", truncation=truncation, padding=padding)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\models\tapas\tokenization_tapas.py", line 619, in __call__
    return self.encode_plus(
           ~~~~~~~~~~~~~~~~^
        table=table,
        ^^^^^^^^^^^^
    ...<17 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\models\tapas\tokenization_tapas.py", line 982, in encode_plus
    return self._encode_plus(
           ~~~~~~~~~~~~~~~~~^
        table=table,
        ^^^^^^^^^^^^
    ...<16 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\models\tapas\tokenization_tapas.py", line 1034, in _encode_plus
    return self.prepare_for_model(
           ~~~~~~~~~~~~~~~~~~~~~~^
        table,
        ^^^^^^
    ...<17 lines>...
        verbose=verbose,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\models\tapas\tokenization_tapas.py", line 1170, in prepare_for_model
    raw_table = add_numeric_table_values(raw_table)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\models\tapas\tokenization_tapas.py", line 2770, in add_numeric_table_values
    table.iloc[row_index, col_index] = Cell(text=cell)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\indexing.py", line 938, in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\indexing.py", line 1953, in _setitem_with_indexer
    self._setitem_with_indexer_split_path(indexer, value, name)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\indexing.py", line 2044, in _setitem_with_indexer_split_path
    self._setitem_single_column(loc, value, pi)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\indexing.py", line 2181, in _setitem_single_column
    self.obj._mgr.column_setitem(loc, plane_indexer, value)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\internals\managers.py", line 1528, in column_setitem
    new_mgr = col_mgr.setitem((idx,), value)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\internals\managers.py", line 607, in setitem
    return self.apply("setitem", indexer=indexer, value=value)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\internals\managers.py", line 445, in apply
    applied = getattr(b, f)(**kwargs)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\internals\blocks.py", line 1667, in setitem
    values[indexer] = value
    ~~~~~~^^^^^^^^^
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 2237, in __setitem__
    value = self._maybe_convert_setitem_value(value)
  File "C:\Users\usuario\AppData\Local\Programs\Python\Python313\Lib\site-packages\pandas\core\arrays\string_arrow.py", line 298, in _maybe_convert_setitem_value
    if len(value) and not (
       ~~~^^^^^^^
TypeError: len() of unsized object
```

### Expected behavior

Printing: 53

--- Comment by jshaofa-ui at 2026-05-12T00:47:57Z ---
## 🔧 Solution Proposal: Table Question Answering Pipeline Crash

### Root Cause

The crash in the `table-question-answering` pipeline is caused by a pandas 3.x compatibility issue. The Tapas pipeline's `TableQuestionAnsweringPipeline.__call__` method (line 285) processes DataFrames through the tokenizer, which internally uses pandas operations that changed behavior in pandas 3.x.

Specifically, the issue is in how the pipeline handles DataFrame column names and data types when preparing inputs for the TapasModel. Pandas 3.x changed:
1. `DataFrame.astype()` behavior with nullable dtypes
2. String dtype handling (pyarrow-backed strings)
3. `pd.isna()` return type changes

The stack trace points through `pt_utils.py` → `dataloader.py` → `table_question_answering.py`, indicating the crash occurs during batch collation when the tokenizer returns tensors with incompatible shapes due to pandas 3.x DataFrame preprocessing.

### Fix

Add pandas version-aware preprocessing in `table_question_answering.py`:

```python
# In src/transformers/pipelines/table_question_answering.py

import pandas as pd

class TableQuestionAnsweringPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        # ... existing code ...
        pass

    def preprocess(self, table, query=None, id=None, **kwargs):
        # Ensure DataFrame compatibility across pandas versions
        if not isinstance(table, pd.DataFrame):
            table = pd.DataFrame(table)

        # Normalize column names to strings (pandas 3.x may use different types)
        table.columns = [str(c) for c in table.columns]

        # Convert all cell values to strings, handling pandas 3.x nullable dtypes
        for col in table.columns:
            series = table[col]
            # Handle pyarrow-backed strings and nullable dtypes
            if hasattr(pd, 'StringDtype') and isinstance(series.dtype, pd.StringDtype):
                table[col] = series.astype(str).fillna("")
            else:
                table[col] = series.apply(lambda x: str(x) if pd.notna(x) else "")

        # ... rest of existing preprocessing ...
```

### Files to Modify
- `src/transformers/pipelines/table_question_answering.py` — Add pandas 3.x compatibility layer

### Regression Test
```python
def test_table_qa_pipeline_pandas3_compat():
    import pandas as pd
    from transformers import pipeline

    data = {
        "Actors": ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
        "Number of movies": ["87", "53", "69"]
    }
    table = pd.DataFrame.from_dict(data)
    question = "how many movies does Leonardo Di Caprio have?"

    tqa = pipeline(task="table-question-answering", model="google/tapas-large-finetuned-wtq")
    result = tqa(table=table, query=question)

    assert result is not None
    assert "cells" in result
    assert result["cells"][0] == "53"
```

### Impact
- Fixes crash for all users on pandas 3.x
- Backward compatible with pandas 2.x
- No API changes — purely internal preprocessing fix