PULL REQUEST #45852: Fix slow Trainer path with 4D attention mask
State: closed  |  Merged: True
Author: qflen  |  Base: main
Labels:
Created: 2026-05-08T19:21:10Z

Fixes #32101

`Trainer` with a pretrained tokenizer and a 4D `attention_mask` is dramatically slower than the 2D-mask path because `PreTrainedTokenizerBase.pad` routes the multi-dim per-sample mask through `to_py_obj` and then rebuilds it with `torch.tensor(deeply_nested_list)` in `BatchEncoding.convert_to_tensors`. That round trip is the dominant cost on the 4D path, and `_pad` itself cannot extend rank > 1 per-sample masks, so the conversion is wasted work.

This PR pops a multi-dim per-sample `attention_mask` out before the per-sample padding loop and stacks it back with `torch.stack` / `np.stack` at the end.

End-to-end on the OP's repro (CPU, `batch=32`, `grad_accum=32`, `seq=1024`, 1 optimizer step) the 4D `Trainer` step drops from 225.59 s to 114.62 s, within 13% of the 2D path's 101.52 s; the collator alone goes from ~3.65 s / batch to ~11.8 ms / batch.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
      Issue: https://github.com/huggingface/transformers/issues/32101.
      Maintainer invitation to open a PR: https://github.com/huggingface/transformers/issues/32101#issuecomment-4406013253.
- [ ] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

## Who can review?

@SunMarc (trainer) and @ArthurZucker (tokenizers, text models).

---

## Numbers

End-to-end Trainer benchmark (CPU, single optimizer step, `batch=32`, `grad_accum=32`, `seq=1024`) matching the OP's repro (tiny `LlamaForCausalLM`, 6 layers, hidden=384, vocab=10):

| `attention_mask` | before | after |
| --- | ---: | ---: |
| 2D `(seq,)` per sample | 99.53 s | 101.52 s |
| 4D `(1, q, q)` per sample | 225.59 s | 114.62 s |

Before the fix the 4D step was 2.27x slower than the 2D step. After the fix it is within 13%, since the remaining cost is dominated by the model forward / backward (~100 s on this CPU). On hardware where the model is faster (e.g. the OP's, where the 2D step took 16 s and the 4D step took 340 s), the fix recovers a much larger relative speedup at the same parity point, for the same reason: the collator stops being the bottleneck.

### Collator-only decomposition

`DataCollatorWithPadding` timed in isolation across three shapes; mean ms / batch over `n_repeat` calls after a warm-up:

| (batch, seq) | n_repeat | mask | before (ms / batch) | after (ms / batch) | speedup |
| --- | ---: | --- | ---: | ---: | ---: |
| (8, 128) | 200 | 2D | 0.32 | 0.33 | ~1.0x |
| (8, 128) | 200 | 4D `(1, q, q)` | 13.16 | 0.26 | **~50x** |
| (16, 256) | 50 | 2D | 1.20 | 1.16 | ~1.0x |
| (16, 256) | 50 | 4D `(1, q, q)` | 110.13 | 0.92 | **~120x** |
| (32, 1024) | 20 | 2D | 8.96 | 8.89 | ~1.0x |
| (32, 1024) | 20 | 4D `(1, q, q)` (OP shape) | 3650.73 | 11.80 | **~310x** |

The 2D path is unchanged within noise. The 4D speedup grows with shape because the dominant pre-fix cost is `torch.tensor` on a deeply nested Python list of size O(batch * q * kv); after the fix, a single `torch.stack` scales linearly with elements rather than super-linearly with Python overhead.

Numerical equivalence is verified by the new `test_4d_attention_mask_preserved` regression test: per-sample shape, dtype, and contents match the stacked batch slice exactly, for both torch and numpy inputs.

## AI assistance disclosure

I've used Opus 4.7 Max Effort for debugging and reviewed / double-checked every line.

- Claimed issue in:
  https://github.com/huggingface/transformers/issues/32101#issuecomment-4396866223

- Why this is not a duplicate:
  No open PR references #32101 in its body and no open PR addresses the same `tokenizer.pad` 4D-mask slowdown (verified via `gh pr list --state open --search`). A commenter from 2024 expressed interest but never opened a PR.

- Tests run:
  - `make style`, `make typing`, `make fix-repo`, all green, no extra diff
  - `pytest tests/trainer/test_data_collator.py` (77 passed, includes 2 new regression tests)
  - `pytest tests/trainer/ --ignore=tests/trainer/distributed -k "attention or 4d or pad or collator"` (80 passed)
  - `pytest tests/models/bert/test_tokenization_bert.py` (51 passed, 2 skipped)
  - `pytest tests/test_tokenization_common.py` (18 passed, 3 skipped)
  - The two new regression tests were verified to fail on main (without the fix) and pass with the fix.
  - Tested on CPU only (no CUDA available locally); the diff has no device-specific code paths so behavior on GPU is the same by construction.

--- Comment by HuggingFaceDocBuilderDev at 2026-05-11T12:52:03Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45852). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Rocketknight1 at 2026-05-12T12:10:27Z ---
Thanks for catching my style issue, merging!

--- Comment by Rocketknight1 at 2026-05-11T12:38:13Z ---
```suggestion
        # Pop 4D nested-list attention masks and stack
        # them at the end to avoid slow `to_py_obj`
```


[docs] chat template

PULL REQUEST #45851: [docs] chat template
State: open  |  Merged: False
Author: stevhliu  |  Base: main
Labels:
Created: 2026-05-08T17:49:15Z

fixes #45205

adds docs for loading and saving modern/legacy chat template formats

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T17:59:33Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by zucchini-nlp at 2026-05-12T11:07:53Z ---
i think we should say explicitly that the other two options are not supported anymore when calling `save_pretrained` and we only support loading

So in older repos, the saved template filenames can be different but users should NEVER try to save a templae in any of older filenames

--- Comment by zucchini-nlp at 2026-05-12T11:09:12Z ---
wait, iirc we were removing it after v5, @Rocketknight1 ?

--- Comment by zucchini-nlp at 2026-05-12T11:09:58Z ---
at least in processing I removed it, from what I see in codebase


PULL REQUEST #45849: Enhance apply_chat_template to support custom field prefilling (reasoning_content, thinking, etc.)
State: closed  |  Merged: False
Author: Mamiglia  |  Base: main
Labels:
Created: 2026-05-08T16:06:43Z

## What does this PR do?

This PR enhances `apply_chat_template` and the underlying `render_jinja_template` utility to support prefilling of custom message fields. This is particularly useful for models that feature internal reasoning traces or multiple output channels (e.g., Qwen, Gemma 4, and OpenAI's Harmony-based models).

### Motivation
Previously, the `continue_final_message` parameter only supported continuing the default `content` field. Users wanting to prefill a reasoning field (like `reasoning_content` or `thinking`) had no clean way to do so via the standard API without manual string manipulation.

### Example: Old vs. New Behavior

Consider a conversation where we want to prefill the reasoning (thought) of the assistant:
```python
messages = [
    {"role": "user", "content": "Explain 1+1"},
    {"role": "assistant", "reasoning_content": "The user wants a simple addition. ", "content": ""}
]
```

#### Old Behavior
Setting `continue_final_message=True` would only target the `content` field.
```python
# Resulting prompt (Qwen-style):
# <|im_start|>user\nExplain 1+1<|im_end|>\n<|im_start|>assistant\n<think>\nThe user wants a simple addition. \n</think>\n
```
The model would start generating a new response or continue the empty `content`, but the `reasoning_content` is already closed with `</think>`. There was no way to stop *inside* the `<think>` tags.

#### New Behavior

```python
tokenizer.apply_chat_template(messages, continue_final_message="reasoning_content", tokenize=False)

# Resulting prompt:
# <|im_start|>user\nExplain 1+1<|im_end|>\n<|im_start|>assistant\n<think>\nThe user wants a simple addition.
```
The prompt stops exactly after the prefilled reasoning, allowing the model to continue its chain of thought naturally.

### Changes
- **Utility Update**: Modified `render_jinja_template` in `src/transformers/utils/chat_template_utils.py` to accept `bool | str` for `continue_final_message`. If a string is provided, it is used as the key to identify which field in the final message should be continued (this is done because there is no standard for which is the name of the "reasoning" field).
- **API Consistency**: Updated the `apply_chat_template` signature and docstrings in both `PreTrainedTokenizerBase` (`src/transformers/tokenization_utils_base.py`) and `ProcessorMixin` (`src/transformers/processing_utils.py`) to expose this new capability.
- **Comprehensive Testing**: Added 4 new test cases to `tests/utils/test_chat_template_utils.py` covering:
    - Standard prefilling of the `content` field.
    - ChatML with `<think>` tags for `reasoning_content` prefilling.
    - Harmony format (gpt-oss) for prefilling specific analysis channels.
    - Gemma 4 turn/channel structure for `thought` prefilling.
    - I have "hard-coded" these because they are just examples of how the chat templates **could** be, from what I see every new model has a new chat template and it's impossible to test every and each chat template on the HF hub.
- **Error Handling**: Added validation to ensure that the field being continued exists in both the message object and the Jinja template, providing clear error messages when they don't.

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

## Who can review?

Tokenizers: @ArthurZucker and @itazap

--- Comment by Rocketknight1 at 2026-05-11T12:30:42Z ---
Hey! This is a cool idea and I think we can support it. However, the PR description says you added test cases, but I don't see them in the diff. We definitely need at least one, since this is adding new functionality to a core method. Also, make sure to `pip install -e .[quality]` and `make fix-repo` to get the code style tests to pass

--- Comment by github-actions[bot] at 2026-05-11T14:02:03Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45849&sha=2358dd

--- Comment by Mamiglia at 2026-05-11T15:09:00Z ---
Oh my bad I pushed the wrong branch! I'm going to close this PR and open a new one with the right branch


Add Maximal Update Parametrization (μP)

PULL REQUEST #45848: Add Maximal Update Parametrization (μP)
State: open  |  Merged: False
Author: qgallouedec  |  Base: main
Labels:
Created: 2026-05-08T15:40:14Z

Adds opt-in μP via two fields on `PretrainedConfig` (`mup: bool`, `mup_base_width: int`) plus a single base-class hook in `PreTrainedModel`. `Trainer.create_optimizer` picks up `config.mup` automatically and applies the AdamW μP learning-rate split (matrix-like → `lr/m`, vector-like → `lr`) following `mup.MuAdam`; users building their own optimizer can call `transformers.integrations.build_mup_param_groups`.

closes https://github.com/huggingface/transformers/issues/16157

## What is it?

[Tensor Programs V](https://hf.co/papers/2203.03466) shows that under a specific (init, forward, optimizer) triple, per-coordinate activation, logit, and weight-update magnitudes stay O(1) as the hidden size `n` grows. The practical consequence is *μTransfer*: the optimal learning rate at a small "base" width is also the optimal LR at any wider model. Tune once on a small proxy, reuse the LR on the full-size model.

## Design

Three rewrites, all generic, all driven by `config.mup`:

- **`PreTrainedModel.post_init`**: walk modules. Replace `get_output_embeddings()` with a `MuReadout` of the same shape; on every attention module that exposes `scaling` and `head_dim`, divide `scaling` by `√head_dim` (turns `1/√d_head` into `1/d_head`, preserves any layer-idx factor).
- **`PreTrainedModel._initialize_weights`**: after the model's own init, divide `Linear`/`Conv1D` weights by `√(d/d0)`. Guarded by `weight._is_hf_initialized` so weights loaded from a state dict are not re-rescaled.
- **`MuReadout.forward`**: `super().forward(x / m)` pre-multiplies the input, matching the Microsoft `mup` library so the bias is correctly unscaled.

The whole feature is ~40 LoC in `modeling_utils.py` plus a small `integrations/mup.py`. It depends only on conventions every modern HF model already follows (`module.scaling`/`module.head_dim`, `get/set_output_embeddings`), which is why it works on Llama and GPT-2 with zero model-file changes (and should work on any similar model).

Because the marker lives in the config, save/load round-trips: a checkpoint trained with `mup=True` rebuilds the μP architecture on `from_pretrained`, so generation doesn't go to garbage. There's a regression test for this (`test_save_load_round_trip`).

## Why this exact recipe

Table 8 of the paper admits several SGD-equivalent formulations that diverge under Adam. This PR matches the canonical `mup.MuAdam` split:

| Parameter category                       | Forward     | Init std       | AdamW LR  |
|------------------------------------------|-------------|----------------|-----------|
| matrix-like (hidden, both dims width)    | x1          | `std / √m`     | `lr / m`  |
| vector-like (readout, embed, bias, LN)   | x1          | unchanged      | `lr`      |
| readout (`MuReadout`)                    | `x / m`     | unchanged      | `lr`      |

Logit boundedness comes from the readout's input rescale, not from a readout LR adjustment. (I had the LR direction inverted in an earlier draft; the coord check below caught it immediately. That's the case for keeping these empirical tests.)

## Empirical validation

```$ python examples/pytorch/mup_demo.py --mode coord
========================================================================
COORDINATE CHECK — per-module mean(|activation|) after a few steps
========================================================================
What to look for:
  • Under μP each row should be roughly constant across w=64…256 (spread ≈ 1).
  • Under SP (standard parametrization, the default) values fan out with width.

=== μP (base_width=64) ===
layer                                          w=64      w=128     w=256       spread
-------------------------------------------------------------------------------------
lm_head                                          0.1484    0.1215    0.0975     1.52x
model.embed_tokens                               0.0159    0.0162    0.0162     1.02x
model.layers.0.mlp.down_proj                     0.0042    0.0042    0.0040     1.05x
model.layers.0.mlp.gate_proj                     0.1310    0.1316    0.1293     1.02x
model.layers.0.mlp.up_proj                       0.1317    0.1312    0.1289     1.02x
model.layers.0.self_attn.k_proj                  0.1322    0.1295    0.1286     1.03x
model.layers.0.self_attn.o_proj                  0.0084    0.0085    0.0084     1.02x
model.layers.0.self_attn.q_proj                  0.1295    0.1263    0.1303     1.03x
model.layers.0.self_attn.v_proj                  0.1291    0.1310    0.1292     1.01x
model.layers.1.mlp.down_proj                     0.0041    0.0041    0.0041     1.01x
model.layers.1.mlp.gate_proj                     0.1330    0.1316    0.1314     1.01x
model.layers.1.mlp.up_proj                       0.1302    0.1320    0.1313     1.01x
model.layers.1.self_attn.k_proj                  0.1326    0.1276    0.1310     1.04x
model.layers.1.self_attn.o_proj                  0.0142    0.0152    0.0155     1.09x
model.layers.1.self_attn.q_proj                  0.1324    0.1269    0.1306     1.04x
model.layers.1.self_attn.v_proj                  0.1291    0.1321    0.1313     1.02x

  → mean spread: 1.06x

=== SP ===
layer                                          w=64      w=128     w=256       spread
-------------------------------------------------------------------------------------
lm_head                                          0.1483    0.2377    0.3851     2.60x
model.embed_tokens                               0.0159    0.0163    0.0163     1.02x
model.layers.0.mlp.down_proj                     0.0042    0.0136    0.0451    10.66x
model.layers.0.mlp.gate_proj                     0.1310    0.1908    0.2769     2.11x
model.layers.0.mlp.up_proj                       0.1317    0.1895    0.2761     2.10x
model.layers.0.self_attn.k_proj                  0.1323    0.1842    0.2632     1.99x
model.layers.0.self_attn.o_proj                  0.0084    0.0158    0.0363     4.34x
model.layers.0.self_attn.q_proj                  0.1296    0.1804    0.2670     2.06x
model.layers.0.self_attn.v_proj                  0.1291    0.1851    0.2618     2.03x
model.layers.1.mlp.down_proj                     0.0041    0.0121    0.0403     9.87x
model.layers.1.mlp.gate_proj                     0.1329    0.1862    0.2718     2.05x
model.layers.1.mlp.up_proj                       0.1302    0.1883    0.2737     2.10x
model.layers.1.self_attn.k_proj                  0.1326    0.1816    0.2720     2.05x
model.layers.1.self_attn.o_proj                  0.0142    0.0306    0.0729     5.13x
model.layers.1.self_attn.q_proj                  0.1326    0.1782    0.2653     2.00x
model.layers.1.self_attn.v_proj                  0.1291    0.1875    0.2715     2.10x

  → mean spread: 3.39x

========================================================================
SUMMARY  μP mean spread = 1.06x   |   SP mean spread = 3.39x
μP keeps activations roughly width-invariant; SP fans out (≈ width range = 4x).
========================================================================
```

Hidden-layer activations are flat across the 4× width range under μP (mean spread 1.06×) and fan out under SP (3.39×). This is the desideratum the paper proves μP enforces.

```
$ python examples/pytorch/mup_demo.py --mode lr-transfer
========================================================================
LEARNING-RATE TRANSFER — final loss after a few steps
========================================================================
What to look for:
  • Under μP the optimal LR (← marker) should be the SAME row at every width.
  • Under SP (standard parametrization, the default) the optimal LR shifts as width grows.

=== μP (base_width=64) ===
        lr  w=64        w=128       w=256
----------------------------------------------
   1.0e-04   4.782     4.793     4.783
   3.0e-04   4.676     4.688     4.677
   1.0e-03   4.437     4.443     4.435
   3.0e-03   4.058     4.052     4.040
   1.0e-02   3.865 ←   3.954 ←   3.661 ←
   1.0e-01   4.525     5.407     5.221

=== SP ===
        lr  w=64        w=128       w=256
----------------------------------------------
   1.0e-04   4.782     4.653     4.217
   3.0e-04   4.675     4.361     3.623
   1.0e-03   4.436     3.864     2.940 ←
   3.0e-03   4.056     3.503 ←   3.001
   1.0e-02   3.846 ←   3.755     4.122
   1.0e-01   4.690     6.336    14.042

========================================================================
SUMMARY  optimal LR per width:
  μP : 1e-02, 1e-02, 1e-02   ✓ width-invariant
  SP : 1e-02, 3e-03, 1e-03   ✗ shifts
========================================================================
```

`←` marks the column-wise minimum. Under μP it stays in one row; under SP it walks, and `lr=1e-2` even diverges by `w=256`. That's μTransfer end-to-end.

## Coverage

Works on any causal LM exposing `module.scaling`/`module.head_dim` and `get/set_output_embeddings` — ~250 of the model files in `src/transformers/models/`. Tested on Llama (CI) and GPT-2 (smoke). Not covered:

- **Inlined-scale legacy LMs** (Falcon, GPT-J, MPT, BLOOM, CodeGen): the attention rewrite no-ops because there's no `module.scaling` to patch: silently leaves `1/√d_head` while the rest applies. One-line per-model fix; not done here.
- **State-space / RNN** (Mamba, RWKV, Falcon-Mamba, Recurrent-Gemma, xLSTM): μP recipe differs.
- **Non-standard attention** (T5/MT5/UMT5, Reformer, Perceiver, BigBird, LongT5, Transfo-XL): would need per-model adapters.
- **Encoder-only / vision / audio / multimodal wrappers**: `mup` exists on their config but is a no-op without a causal-LM head, by design.
- `models/afmoe`'s existing `mup_enabled` is unrelated and left alone.

Out of scope: SGD recipe, automatic base-width detection.

## Tests

`tests/integrations/test_mup.py` — 6 cases on Llama: readout swap, attention scale, hidden init rescale, save/load round-trip, MuAdam-style param groups, and the empirical coord check (μP spread strictly less than SP spread).

## AI assistance

Assisted by Claude Code (Opus 4.7).

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T15:52:49Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45848). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-13T18:41:07Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45848&sha=3289ae


Streamable chat parsing

PULL REQUEST #45847: Streamable chat parsing
State: open  |  Merged: False
Author: Rocketknight1  |  Base: main
Labels:
Created: 2026-05-08T15:19:40Z

Work in progress PR for streamable chat parsing. This introduces a new improved kind of response schema, which I'm calling `response_template`. Response templates are a lot easier to write, since they avoid most of the messy regexes of the older system. More importantly, though, using a region-based system means that model authors can write templates that enable streaming parsing, emitting fields as they're ready, rather than needing the whole message to be ready at once.

TODO:

- [x] Merge/cleanup content types
- [x] Can we merge `coerce` into content types?
- [x] Correct handling of tokens/strings inputs, batched lists, etc.
- [x] Final docs cleanup, ensure examples match API
- [x] Lots and lots of general testing
- [x] Handling of multiple tool calls - do we need to emit arrays differently?
- [x] Explicit option for "multiple literals" as a match without needing a regex, avoids regex buffer

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T15:34:25Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45847). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Rocketknight1 at 2026-05-15T16:24:43Z ---
This should now be ready for review! It's a lot, but thankfully I think you can skim a lot of it. It's also a little ugly right now, because we need to maintain legacy support for the older `parse_response()` method, but once this is in we can begin deprecating that very quickly. That cleanup, and support for batched inputs, will come in a follow-up PR (batched handling will also be needlessly messy because of the legacy code right now, but it'll be much cleaner to add once it's removed)

The main things to review are:
- The doc
- The public-facing changes in `response_parser.py`, `processing_utils.py` and `tokenizer_utils.py`
- The test coverage

Most other files are agent-written boilerplate changes to support the new API (the pipeline + Gemma conversion script changes), or agent-written internal mechanics that are mostly dense and should be well-covered by tests (`content_parsers.py`, `response_templates.py`  and the second half of `response_parser.py`)

--- Comment by github-actions[bot] at 2026-05-20T14:03:44Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4

--- Comment by github-actions[bot] at 2026-05-20T14:03:44Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4

--- Comment by pcuenca at 2026-05-18T11:19:01Z ---
```suggestion
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")["input_ids"].to(model.device)
```

Also, reminder to use a model or revision with a valid `response_template`

--- Comment by pcuenca at 2026-05-18T11:30:36Z ---
```suggestion
The parser will emit **events** as text from the generation process is fed in. This indicates which region is currently being generated. When
```

Trying to clarify the (usual) relationship with generation; perhaps we could provide a combined example that shows streaming generation with response parsing.

--- Comment by pcuenca at 2026-05-18T11:40:58Z ---
In that case we should update the prompt and pass the tools so it triggers the greet_user call, just as it's done later in the advanced usage section.

--- Comment by pcuenca at 2026-05-18T11:46:05Z ---
```suggestion
Finally, you can specify `optional: false` for fields that must be present. If parsing finishes and a non optional field
```

--- Comment by pcuenca at 2026-05-18T11:47:36Z ---
😂

--- Comment by pcuenca at 2026-05-18T11:55:11Z ---
Maybe mention the `transformers["chat_template"]` extra that includes this?

--- Comment by pcuenca at 2026-05-18T12:01:49Z ---
```suggestion
            "transform": "{type: 'function', function: {name: name, arguments: content}}",
},
```

I think this became a jmespatch `transform` string

--- Comment by Rocketknight1 at 2026-05-18T14:13:14Z ---
Ah, you're totally right! I missed that one after deprecating `assemble`

--- Comment by Rocketknight1 at 2026-05-18T14:19:38Z ---
Done!

--- Comment by Rocketknight1 at 2026-05-18T14:27:18Z ---
Fixed the text! I don't want to make the example too complex by adding tool definitions, so I just changed the text so I don't say the event stream matches the example above.

--- Comment by pcuenca at 2026-05-18T15:20:19Z ---
Sounds good!

--- Comment by ariG23498 at 2026-05-19T12:28:01Z ---
This comment should never be resolved. LOVE IT. 😂

--- Comment by pcuenca at 2026-05-19T14:10:44Z ---
Note this is Python-only syntax for named groups, so we'd need a translation layer to ICU somewhere for downstream implementations, cc @xenova. Perhaps we should accept both in transformers and document ICU as the standard?

--- Comment by Rocketknight1 at 2026-05-19T14:26:43Z ---
Ah, I wasn't aware! Is there a universal way to do it, or is there no universal way to make named groups work? I might even prefer unnamed groups and putting the names in a separate field rather than causing cross-language problems

--- Comment by pcuenca at 2026-05-19T14:38:05Z ---
I didn't find a universal way to do it, unfortunately. [This is Python](https://docs.python.org/3/library/re.html#regular-expression-syntax) and this is ICU (screenshots with the relevant fragments below).

Yeah, perhaps unnamed groups is the best option if it doesn't make things too confusing.

--- Comment by Rocketknight1 at 2026-05-19T15:10:26Z ---
Hmmn, it might be possible to handle this universally by just having other language implementations strip the `<>` angle brackets if their regex engine treats them as part of the name!

--- Comment by pcuenca at 2026-05-19T15:54:04Z ---
The main problem is the `P`, the regex has to be rewritten without.

--- Comment by pcuenca at 2026-05-19T16:55:52Z ---
It's more about the `P` in `?P<` vs `?<`, [this is what I'm currently doing](https://github.com/huggingface/swift-transformers/blob/48b3171cf28fbf025ee77709d4f18319ec994316/Sources/Tokenizers/ResponseTemplate.swift#L268-L273).

--- Comment by Rocketknight1 at 2026-05-20T13:13:52Z ---
Ah, I'm sorry, I misread that bit! I'll update the docs

--- Comment by pcuenca at 2026-05-20T14:44:43Z ---
Perhaps we could show a fragment of a rendered Cohere tool call so readers understand it contains `tool_name` and `parameters`.

--- Comment by Rocketknight1 at 2026-05-21T13:23:34Z ---
Should be resolved in the docs now!


PULL REQUEST #45846: [Cache] Add `Cache.snapshot()` / `Cache.restore(snapshot)` for tentative-forward rollback
State: closed  |  Merged: False
Author: kashif  |  Base: main
Labels:
Created: 2026-05-08T11:56:08Z

Speculative decoding (and any "forward then maybe undo" pattern) needs to roll a cache back to a previous state. For full-attention layers, the existing `crop()` API works, but for linear-attention layers whose `crop()` is a documented no-op because the recurrent state has already absorbed every observed token, there has been no supported way to undo a forward pass. Block-diffusion speculative decoders silently corrupt their target cache on hybrid-attention models (e.g. Qwen3.5) as a result.

This change adds:

  * `CacheLayerMixin.snapshot()` / `restore(snapshot)` covering full-attn layers (`DynamicLayer`, `DynamicSlidingWindowLayer`).
  * `LinearAttentionCacheLayerMixin.snapshot()` / `restore(snapshot)` for linear-attn layers. `restore()` uses `.copy_()` into existing tensors when shapes match, so cudagraph static-address assumptions survive.
  * `LinearAttentionAndFullAttentionLayer` overrides that merges both parents.
  * `Cache.snapshot()` / `Cache.restore(snapshot)` that delegate per-layer and reject layout mismatches.

The shape of a snapshot is opaque to callers; the return value is meant to be passed back to `restore()` on the same cache. The intended pattern is

    snap = cache.snapshot()
    run_target_on_speculative_block(...)
    if partial_accept:
        cache.restore(snap)
        run_target_on_accepted_prefix(...)

Tests in `tests/utils/test_cache_utils.py::CacheSnapshotRestoreTest` cover `DynamicLayer` round-trip, `LinearAttentionLayer` round-trip (including static-address preservation), `DynamicCache` aggregate round-trip, and a layout-mismatch rejection.

Verified end-to-end with diffusers' `DFlashPipeline` on `z-lab/Qwen3.5-4B-DFlash`
+ `Qwen/Qwen3.5-4B` (a hybrid-attention target whose target cache contains linear-attention layers): the pipeline drives generation purely through the new `cache.snapshot()` / `cache.restore()` calls and produces `"2 + 2 equals 4."`

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T12:08:11Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45846). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Cyrilvallez at 2026-05-14T07:46:40Z ---
Hey @kashif! Thanks for opening the PR. This is a well known issue - speculative decoding is only supported until the sliding window length, and not supported at all for Mamba currently.
Your snapshot/restore approach will not work in general, as then we would need to recompute the states for the accepted tokens. Imagine the following situations: we draft 5 tokens with the assistant, then accept 3 of those 5 tokens. We cannot restore to the original cache, as we'll be missing the 3 accepted tokens. It would be very inefficient to recompute the states for those 3 tokens, compared to just rolling back 2 tokens out of the 5.
There is no clear efficient solution for sliding and mamba caches unfortunately...

--- Comment by kashif at 2026-05-14T10:05:09Z ---
closing

--- Comment by Copilot at 2026-05-08T12:01:42Z ---
`CacheLayerMixin.snapshot()/restore()` reassign `keys`/`values` and don't capture/restore `cumulative_length`. This breaks `StaticLayer`/`StaticSlidingWindowLayer`, where `update()` mutates `keys`/`values` and `cumulative_length` in-place and relies on `mark_static_address`. A restore after a tentative forward would leave `cumulative_length` advanced (wrong `get_seq_length()`), and reassigning tensors also defeats static-address assumptions. Consider overriding snapshot/restore in static cache layers (store cumulative length(s) and restore via `.copy_()` into existing tensors when possible) or making the base implementation static-layer-safe.

--- Comment by Copilot at 2026-05-08T12:01:42Z ---
`LinearAttentionCacheLayerMixin.restore()` uses `.copy_()` when shapes match, but doesn't check device/dtype. If a layer is offloaded/prefetched (or otherwise moved across devices/dtypes) between `snapshot()` and `restore()`, a shape-only match can still cause `.copy_()` to raise. Consider also checking `device`/`dtype` before copying (and falling back to reassign/clone or `.to(self.device)` as appropriate).

--- Comment by Copilot at 2026-05-08T12:01:42Z ---
`Cache.restore()` only validates snapshot length, but the docstring/error message frames this as a “same layer layout” requirement. A same-length snapshot from a different cache with different per-layer classes can currently fail with a `KeyError` inside a layer `restore()` (or partially restore) rather than a clear `ValueError`. Consider validating per-layer type/layout (e.g., compare `type(layer)` or expected snapshot keys) before applying restores to provide a deterministic, user-friendly error for layout mismatches.

--- Comment by Copilot at 2026-05-08T12:01:43Z ---
New snapshot/restore coverage exercises `DynamicLayer`, `LinearAttentionLayer`, and `DynamicCache`, but the PR also introduces `LinearAttentionAndFullAttentionLayer.snapshot()/restore()` and (via the mixins) impacts static cache layers. Adding unit tests for hybrid layers and at least one static-cache round-trip (including length bookkeeping and static-address preservation expectations) would help prevent regressions in the main target use-cases (hybrid-attention models and compiled/static caches).

--- Comment by Copilot at 2026-05-11T19:09:05Z ---
`restore()` assigns `self.keys`/`self.values` directly from the snapshot tensors, which aliases the snapshot object into the live cache. Any later in-place mutation of the cache tensors (e.g. `reset()` calling `.zero_()`) will also mutate the saved snapshot, making it unsafe to reuse the same snapshot for multiple rollbacks. Consider restoring via `.copy_()` into existing tensors when shapes match (or `clone()` on assignment) to keep snapshots immutable and consistent with the Static/LinearAttention restore implementations.

--- Comment by Copilot at 2026-05-11T19:09:05Z ---
`Cache.restore()` only validates the number of layers in the snapshot. A snapshot taken from a different cache with the same layer count (different layer classes / shapes / key sets) can currently fail with a `KeyError` inside a layer restore, or worse, partially restore without an explicit error. To match the docstring/PR intent of rejecting layout mismatches, consider recording per-layer identity in the snapshot (e.g. `layer_type`/class name and maybe key-set/signature) and raising a `ValueError` when it doesn't match the current cache.

--- Comment by Copilot at 2026-05-11T19:09:06Z ---
PR description mentions a "layout-mismatch rejection" test for `Cache.restore()`, but this new test class doesn't currently include a case that asserts an explicit error on mismatched cache layouts (same layer count, different layer types/shapes). Adding that test here would help prevent regressions and would exercise the new validation logic suggested for `Cache.restore()`.


fix(rf_detr): fix failing tests

PULL REQUEST #45845: fix(rf_detr): fix failing tests
State: closed  |  Merged: True
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-08T08:29:23Z

(no description)

--- Comment by github-actions[bot] at 2026-05-08T09:40:42Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: rf_detr

--- Comment by kaixuanliu at 2026-05-08T09:42:51Z ---
@ydshieh pls help review, thx!

--- Comment by vasqu at 2026-05-08T14:42:57Z ---
run-slow: rf_detr

--- Comment by github-actions[bot] at 2026-05-08T14:44:46Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25561877242)

This comment contains `run-slow`, running the specified jobs:

models: ["models/rf_detr"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-08T14:53:59Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25561877242)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [0494c002](https://github.com/huggingface/transformers/commit/0494c0024d504c6480eb743d8a54dfd1379e3794) | workflow commit (merge commit) |
| PR | [68e9183f](https://github.com/kaixuanliu/transformers/commit/68e9183f62bf9d70e1b8a654df1325fa9a06f5ff) | branch commit (from PR) |
| main | [9ea2afbc](https://github.com/huggingface/transformers/commit/9ea2afbcdad51987b1d2f20380edcce5ecdd3d85) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T14:56:07Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45845). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45844: fix(granite4_vision): auto-fix failing tests
State: closed  |  Merged: True
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-08T08:28:56Z

Add `Granite4VisionWindowQFormerDownsampler` to `no_split_modules` to fix failed model parallel case:
`tests/models/granite4_vision/test_modeling_granite4_vision.py::Granite4VisionModelTest::test_model_parallel_beam_search`

--- Comment by github-actions[bot] at 2026-05-08T09:07:45Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: granite4_vision

--- Comment by kaixuanliu at 2026-05-08T09:16:48Z ---
@ydshieh pls help review, thx!

--- Comment by vasqu at 2026-05-08T14:39:06Z ---
run-slow: granite4_vision

--- Comment by github-actions[bot] at 2026-05-08T14:40:43Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25561684449)

This comment contains `run-slow`, running the specified jobs:

models: ["models/granite4_vision"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-08T14:53:23Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25561684449)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [3daea8f3](https://github.com/huggingface/transformers/commit/3daea8f3887e124603d1e8e1b534918d14149969) | workflow commit (merge commit) |
| PR | [99e31fc0](https://github.com/kaixuanliu/transformers/commit/99e31fc067df94a41d7b1f99c5e8db107d5ac4f7) | branch commit (from PR) |
| main | [95933eb6](https://github.com/huggingface/transformers/commit/95933eb6f4274f044988ff6d5122638a326a0ce8) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T14:57:04Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45844). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


fix model parallel issues for deimv2

PULL REQUEST #45843: fix model parallel issues for deimv2
State: closed  |  Merged: True
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-08T08:13:52Z

This PR tries to fix 2 failed test cases:
```
FAILED tests/models/deimv2/test_modeling_deimv2.py::Deimv2ModelTest::test_model_parallelism
FAILED tests/models/deimv2/test_modeling_deimv2.py::Deimv2DINOv3ModelTest::test_model_parallelism
```
for deimv2 model. @zucchini-nlp pls help review, thx!

--- Comment by github-actions[bot] at 2026-05-18T02:41:02Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deimv2

--- Comment by vasqu at 2026-05-18T07:03:09Z ---
run-slow: deimv2

--- Comment by github-actions[bot] at 2026-05-18T07:04:33Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26018626432)

This comment contains `run-slow`, running the specified jobs:

models: ["models/deimv2"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-18T07:12:40Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26018626432)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [e8269f9c](https://github.com/huggingface/transformers/commit/e8269f9c1204159b5a0199e2a9bf4083293589ef) | workflow commit (merge commit) |
| PR | [33a36c46](https://github.com/kaixuanliu/transformers/commit/33a36c468bce33178cea00a05a7a0e23029fff4a) | branch commit (from PR) |
| main | [1e0594f6](https://github.com/huggingface/transformers/commit/1e0594f6979488a949fee3c2c5aae65100084a6d) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-05-18T07:14:58Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45843). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-05-08T14:40:16Z ---
This is a bit weird, ig this is some underlying model we use via auto model? Usually it should be adjusted there and not here 🤔

--- Comment by kaixuanliu at 2026-05-09T03:06:37Z ---
Hi, thx for the comment. I had the same concern here. But `HGNetV2BasicLayer` is indeed defined by the underlying HGNetV2 backbone, and HGNetV2 already declares it in its own `_no_split_modules`.

The issue here is that Deimv2 loads the backbone through `load_backbone(config)`, but the model-parallel `device_map="auto"` path only uses the top-level `Deimv2Model._no_split_modules` when calling `infer_auto_device_map`. It doesn't automatically merge `_no_split_modules` from the nested backbone model. As a result, the HGNetV2 block can still be split when it is used inside Deimv2, even though HGNetV2 itself marks it as no-split.

So this line is not trying to redefine HGNetV2's rule, but to propagate that rule to the composite Deimv2 model.

--- Comment by kaixuanliu at 2026-05-09T03:25:25Z ---
I updated the Deimv2 backbone wrappers to expose the loaded backbone's `_no_split_modules` instead of hard-coding `HGNetV2BasicLayer` in Deimv2. It looks much better this time, WDYT?

--- Comment by zucchini-nlp at 2026-05-09T19:33:22Z ---
>  path only uses the top-level Deimv2Model._no_split_modules when calling infer_auto_device_map

this is weird, we recurse over all pretrained models to infer `_no_split_modules` already, and I see HGNet has it defined clearly. Can you check what happens here, it should merge for all children right after model is init? Prob the backbone is being init too later or manually swapped over by another at runtime, from top of my head

https://github.com/huggingface/transformers/blob/c9de1097eed992fe205f876415e7a7cfaa06be50/src/transformers/modeling_utils.py#L1390-L1392

--- Comment by kaixuanliu at 2026-05-11T01:27:36Z ---
Sorry, I did not express clearly here. You're right that `post_init()` merges `_no_split_modules` from child modules recursively, and HGNetV2 does define `HGNetV2BasicLayer`.

The real problem here is that in Deimv2 the HGNetV2 backbone is not a direct child of `Deimv2Model`. It is stored under `Deimv2Model.conv_encoder.model`, while `conv_encoder` itself is a plain `nn.Module`, not a `PreTrainedModel`. `PreTrainedModel.post_init()` only checks direct children and relies on each intermediate child to expose the aggregated metadata. Therefore the merge stops at `conv_encoder` unless the wrapper forwards the loaded backbone's `_no_split_modules`.

--- Comment by zucchini-nlp at 2026-05-11T10:18:38Z ---
Hmm, makes me wondwe if we should just iter over all children in `post-init`. @vasqu wdyt, there shouldn't be weird side effects imo?

--- Comment by vasqu at 2026-05-11T12:02:17Z ---
Imo, there shouldn't be any weird side effects but we should be careful how we do it before the iterations explode cc @Cyrilvallez if you have a good idea here

The current state is a bit weird tho as I definitely wouldn't expect this to be limited to "depth 1"

--- Comment by kaixuanliu at 2026-05-13T02:19:04Z ---
@Cyrilvallez Hi, any comments?

--- Comment by Cyrilvallez at 2026-05-13T06:22:31Z ---
Hey! I am very strongly against iterating over all children. This means that we will traverse the WHOLE graph for every PreTrainedModel, i.e. for a usual composite model, we could easily traverse the whole graph at least 5-6 times. This is just an error in model design, we really should not touch general logic and add a lot of overhead/complexity for this. We generally should never have a PreTrainedModel inside a nn.Module inside another PreTrainedModel. This is one bad exception.
What we should do here @kaixuanliu is to grab the attr of the `conv_encoder.model` directly inside the main model, or refactor to make `conv_encoder` a PreTrainedModel as well

--- Comment by kaixuanliu at 2026-05-13T12:42:54Z ---
Thx for the feedback! Then can we merge this PR now?  Or should we make `Deimv2ConvEncoder` inherit `PreTrainedModel` instead of `nn.module` here(TBH I am not quite sure about this way)? @vasqu @zucchini-nlp

--- Comment by vasqu at 2026-05-13T12:45:26Z ---
Looks to me that the refactor to pretrained model is the better way in this instance then.

> but we should be careful how we do it before the iterations explode

My previous comments align with @Cyrilvallez's in a way that it's just not managable in a feasible way sadly. And adding this to the no split module should not happen. So it's a structural issue if that makes sense :hugs:

--- Comment by kaixuanliu at 2026-05-14T01:58:55Z ---
I updated the code to make `Deimv2ConvEncoder` inherit `PreTrainedModel` instead of `nn.module`, and unit tests all get passed on my side, pls help review again. @vasqu

--- Comment by vasqu at 2026-05-15T15:25:30Z ---
Let's add a small comment that this is essentially the dfine conv encoder with (optional) additional projections at the end

--- Comment by vasqu at 2026-05-15T15:26:45Z ---
So we can ignore this as we do have the same restriction in the end?
```suggestion
```

--- Comment by kaixuanliu at 2026-05-18T01:09:49Z ---
Yes, thx for the advice!

--- Comment by zucchini-nlp at 2026-05-18T01:23:59Z ---
iirc many backbone based models have their "backbone" embedded in >1 depth, so if we start by refacto this, would be great to refactor everything

Also to mention that some VLMs also have their mm-projector embedded at depth-2, since adding it high level would mean much more code. For ex: granite vision has qformer pretrained model embedded inside a pooling module

--- Comment by kaixuanliu at 2026-05-18T01:39:30Z ---
Done

--- Comment by kaixuanliu at 2026-05-18T01:40:43Z ---
I made minor adjustment here(updated the comment) to make modular_check pass.

--- Comment by kaixuanliu at 2026-05-18T01:43:51Z ---
@zucchini-nlp Yes, I thought of this as well. Could you help create a new PR to fix this if possible? As I am not model owner and I am not quite confident to make so many adjustments...


fix(laguna): fix failing tests

PULL REQUEST #45842: fix(laguna): fix failing tests
State: closed  |  Merged: True
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-08T07:12:31Z

This PR tries to fix failed test case:
`tests/models/laguna/test_modeling_laguna.py::LagunaModelTest::test_model_parallelism`, before this fix all model weight will be allocated to 1 single device, and it will return error "AttributeError: 'LagunaModel' object has no attribute 'hf_device_map'". W/ this PR it will be split into 2 different cards, and the case will pass. @ydshieh pls help review

--- Comment by github-actions[bot] at 2026-05-08T07:13:49Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: laguna

--- Comment by vasqu at 2026-05-08T14:40:53Z ---
run-slow: laguna

--- Comment by github-actions[bot] at 2026-05-08T14:42:28Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25561772870)

This comment contains `run-slow`, running the specified jobs:

models: ["models/laguna"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-08T14:53:43Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25561772870)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [0f954d70](https://github.com/huggingface/transformers/commit/0f954d7037d0191e74361feb07feb4c122580225) | workflow commit (merge commit) |
| PR | [4053f499](https://github.com/kaixuanliu/transformers/commit/4053f499ff24f5c32a1bf7b3bf23a9c005130f86) | branch commit (from PR) |
| main | [95933eb6](https://github.com/huggingface/transformers/commit/95933eb6f4274f044988ff6d5122638a326a0ce8) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T14:56:15Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45842). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


[Model] Add PP-OCRv6 Models Support

PULL REQUEST #45838: [Model] Add PP-OCRv6 Models Support
State: closed  |  Merged: True
Author: zhang-prog  |  Base: main
Labels: New model
Created: 2026-05-08T03:42:49Z

(no description)

--- Comment by vasqu at 2026-05-12T12:00:58Z ---
@zhang-prog Thank you for all the work, ig we only need the docs then we can merge. I'm sanity checking the slow CI :hugs:

--- Comment by vasqu at 2026-05-12T12:01:12Z ---
run-slow: pp_lcnet_v4, pp_ocrv6_small_rec, pp_ocrv6_tiny_rec

--- Comment by github-actions[bot] at 2026-05-12T12:03:54Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25733075169)

This comment contains `run-slow`, running the specified jobs:

models: ["models/pp_lcnet_v4", "models/pp_ocrv6_small_rec", "models/pp_ocrv6_tiny_rec"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-12T12:24:50Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25733075169)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [c3b892ef](https://github.com/huggingface/transformers/commit/c3b892ef88741b99da91820d431003d76491594f) | workflow commit (merge commit) |
| PR | [dc0d7ada](https://github.com/zhang-prog/transformers/commit/dc0d7adaf96db7b5b3aa97c76840fdd3b359269a) | branch commit (from PR) |
| main | [a25b8efa](https://github.com/huggingface/transformers/commit/a25b8efa0a3220da89493a72f57081aa5720291f) | base commit (on `main`) |

### Model CI Report

❌ **[2 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/407228e0700c9d98333c6c1a6601409998bb8c75/2026-05-12/runs/34497-25733075169/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [pp_ocrv6_small_rec](https://github.com/huggingface/transformers/actions/runs/25733075169/job/75564275047):
    tests/models/pp_ocrv6_small_rec/test_modeling_pp_ocrv6_small_rec.py::PPOCRV6SmallRecModelIntegrationTest::test_inference_text_recognition_head (✅ ⟹ ❌)

- [pp_ocrv6_tiny_rec](https://github.com/huggingface/transformers/actions/runs/25733075169/job/75564274903):
    tests/models/pp_ocrv6_tiny_rec/test_modeling_pp_ocrv6_tiny_rec.py::PPOCRV6TinyRecModelIntegrationTest::test_inference_text_recognition_head (✅ ⟹ ❌)

--- Comment by vasqu at 2026-05-18T12:59:20Z ---
run-slow: pp_lcnet_v4, pp_ocrv5_mobile_det, pp_ocrv6_medium_det, pp_ocrv6_small_det, pp_ocrv6_small_rec, pp_ocrv6_tiny_rec

--- Comment by github-actions[bot] at 2026-05-18T13:00:41Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26035036441)

This comment contains `run-slow`, running the specified jobs:

models: ["models/pp_lcnet_v4", "models/pp_ocrv5_mobile_det", "models/pp_ocrv6_medium_det", "models/pp_ocrv6_small_det", "models/pp_ocrv6_small_rec", "models/pp_ocrv6_tiny_rec"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-18T13:09:14Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26035036441)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [2a9c8889](https://github.com/huggingface/transformers/commit/2a9c88897003aee57814e1bb88bbd1acaee9ac9a) | workflow commit (merge commit) |
| PR | [b063dd07](https://github.com/zhang-prog/transformers/commit/b063dd07e084a9dc3e6ae7f124baada7ed4a7332) | branch commit (from PR) |
| main | [ca80e957](https://github.com/huggingface/transformers/commit/ca80e95782220b009afbeec6bf864258a67d988b) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-05-18T13:21:31Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45838). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-05-19T12:42:40Z ---
run-slow: pp_lcnet_v4, pp_ocrv5_mobile_det, pp_ocrv6_medium_det, pp_ocrv6_small_det, pp_ocrv6_small_rec, pp_ocrv6_tiny_rec

--- Comment by github-actions[bot] at 2026-05-19T12:44:01Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26097891526)

This comment contains `run-slow`, running the specified jobs:

models: ["models/pp_lcnet_v4", "models/pp_ocrv5_mobile_det", "models/pp_ocrv6_medium_det", "models/pp_ocrv6_small_det", "models/pp_ocrv6_small_rec", "models/pp_ocrv6_tiny_rec"]
quantizations: []

--- Comment by github-actions[bot] at 2026-05-19T13:01:40Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/26097891526)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [efc07caa](https://github.com/huggingface/transformers/commit/efc07caa55a2b420b39919dccc1a6512ccde9f56) | workflow commit (merge commit) |
| PR | [b9ef1387](https://github.com/zhang-prog/transformers/commit/b9ef1387391569742b9b0c94b8f41bdb896667a0) | branch commit (from PR) |
| main | [0b25f8c4](https://github.com/huggingface/transformers/commit/0b25f8c49c37530ce9f8742d7a8c19ed8d254d7d) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by vasqu at 2026-05-19T13:03:03Z ---
Just as a note for onlookers: Weights and docs will be added a bit later after this PR

--- Comment by github-actions[bot] at 2026-05-19T13:14:13Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, pp_lcnet_v4, pp_ocrv5_mobile_det, pp_ocrv6_medium_det, pp_ocrv6_small_det, pp_ocrv6_small_rec, pp_ocrv6_tiny_rec

--- Comment by vasqu at 2026-05-08T11:01:19Z ---
Will be checking docs later, lmk know when they are ready 🫡

--- Comment by vasqu at 2026-05-08T11:03:51Z ---
Maybe we could add a validation for stem type here because strings are a bit brittle in this case

--- Comment by vasqu at 2026-05-08T11:09:49Z ---
Imo, doesn't really make much sense to be a module list, let's split into its actual vars --> conv1, act_fn, conv2

That will make it easier for future modular usage

--- Comment by vasqu at 2026-05-08T11:09:54Z ---
typing

--- Comment by vasqu at 2026-05-08T11:11:35Z ---
Could be combined with my previous comment possibly to have the conv->act->conv structure

--- Comment by vasqu at 2026-05-08T11:13:00Z ---
Yea same here, would avoid the module list here and just have the 2 attributes explicit, e.g. token_conv, token_squeeze_and_excitation

--- Comment by vasqu at 2026-05-08T11:13:06Z ---
typing

--- Comment by vasqu at 2026-05-08T11:14:43Z ---
we can directly append tbh

--- Comment by vasqu at 2026-05-08T11:15:08Z ---
maybe self.blocks makes more sense? above block_config instead of blocks

--- Comment by vasqu at 2026-05-08T11:15:28Z ---
typing (last time mentioning, polluting the review otherwise)

--- Comment by vasqu at 2026-05-08T11:18:09Z ---
So the only difference are some config values? Imo, it doesn't make sense to have different implementations as you can just use the same `PPOCRV6SmallRecForTextRecognition` code

Just needs to upload the corresponding different config

--- Comment by vasqu at 2026-05-08T11:21:14Z ---
```suggestion
```
super nit: I think modular should resolve these itself

--- Comment by vasqu at 2026-05-08T11:22:31Z ---
```suggestion
```
personal preference: also like to let modular resolve these instead (just remove all decorators, the parent's ones will be inherited)

Background is that manually keeping track of those is harder than the inheritance

--- Comment by vasqu at 2026-05-08T11:23:31Z ---
Let's highlight the diffs with small comments, e.g here

--- Comment by vasqu at 2026-05-08T11:23:36Z ---
and here

--- Comment by vasqu at 2026-05-08T11:32:30Z ---
```suggestion
```
hmm, should it not? I think we can still set it to true

--- Comment by vasqu at 2026-05-08T11:35:26Z ---
In essence, this is 2x conv->norm->act

Imo, we can use something along `PPOCRV5MobileDetConvBatchnormLayer` with an overriden init and add this as 2 attributes instead of a modulelist

--- Comment by vasqu at 2026-05-08T11:35:44Z ---
Could be a simple CLIPMLP like we inherit from and override the init again

--- Comment by vasqu at 2026-05-08T11:36:24Z ---
Imo, we don't need the wrapper structure, why not move the head modules directly here

--- Comment by zhang-prog at 2026-05-08T12:08:44Z ---
Yeah, the only differences are some config and backbone_config values. I agree that we can reuse the same `PPOCRV6SmallRecForTextRecognition` implementation and only keep the `PPOCRV6BaseRecConfig`. That makes more sense.

--- Comment by zhang-prog at 2026-05-09T08:29:34Z ---
Yeah, ofc

--- Comment by zhang-prog at 2026-05-09T08:29:35Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:36Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:37Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:38Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:41Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:44Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:45Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:46Z ---
Yeah, done

--- Comment by zhang-prog at 2026-05-09T08:29:47Z ---
Got it, done

--- Comment by zhang-prog at 2026-05-09T08:29:51Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:53Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:55Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:56Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:29:58Z ---
Because PP-OCRv6_tiny_rec has no gradient checkpointing layers

--- Comment by zhang-prog at 2026-05-09T08:30:00Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:30:03Z ---
Done

--- Comment by zhang-prog at 2026-05-09T08:30:04Z ---
Done

--- Comment by vasqu at 2026-05-11T14:45:55Z ---
Nit could we move this into `PPLCNetV4SmallStem`?

--- Comment by vasqu at 2026-05-11T14:47:53Z ---
Ah sorry I don't even think we need to have this at all - you can have a different hub config for sure but we don't need a new implementation on transformers side

--- Comment by vasqu at 2026-05-11T14:54:42Z ---
nit: would really make this a different module in nature of `PPOCRV5MobileDetConvBatchnormLayer` (conv1d, batch norm, act) but with conv1d

--- Comment by vasqu at 2026-05-11T14:56:05Z ---
we can move the integration test then as well

--- Comment by vasqu at 2026-05-11T14:57:23Z ---
Like mentioned not needed --> let's merge these into one implementation. Maybe we avoid the model sizes as suffix and have something more generic (for those 2 at least)

--- Comment by zhang-prog at 2026-05-12T09:10:28Z ---
Emm, this code belongs to the stages rather than the stem, so we can’t move it into `PPLCNetV4SmallStem`.

--- Comment by zhang-prog at 2026-05-12T09:10:30Z ---
Done

--- Comment by zhang-prog at 2026-05-12T09:10:31Z ---
Done

--- Comment by zhang-prog at 2026-05-12T09:10:56Z ---
I see, let me reuse it

--- Comment by zhang-prog at 2026-05-12T10:50:19Z ---
Unfortunately, `modular` doesn’t eliminate the if-else structure, so I can’t really reuse it.

<img width="968" height="608" alt="image" src="https://github.com/user-attachments/assets/5f25536c-8882-4d0b-95af-49b15a2f5e3e" />

--- Comment by vasqu at 2026-05-12T11:59:35Z ---
No no I meant as in a new module that has that structure, not exactly inheriting from it but in similar fashion. I would expect this to be useful for future models to inherit from modular

--- Comment by vasqu at 2026-05-12T12:00:15Z ---
I just thought it looked similar in structure so it could be wrapped as stem but no worries, probably was too ambitious here

--- Comment by vasqu at 2026-05-19T11:55:21Z ---
If this becomes common, might be nice as a flag (maybe cc @yonigozlan @molbap @guarin)

For now I think this is ok as is

--- Comment by vasqu at 2026-05-19T12:02:05Z ---
Let's also add the image processor tests please. Should be pretty much the same as from the parent image processor

--- Comment by zhang-prog at 2026-05-19T12:24:32Z ---
Done

--- Comment by vasqu at 2026-05-19T12:32:12Z ---
Oh it had a wrong name 👀 thanks for fixing

--- Comment by vasqu at 2026-05-19T12:36:03Z ---
Nit: imo we can remove the docstring, the small comments alone suffice


PULL REQUEST #45837: WIP: fix(deepseek_v4) correct CSA per-query block mask and causal top-k guard in eager attention
State: closed  |  Merged: False
Author: Prachi-kushwaha  |  Base: main
Labels:
Created: 2026-05-08T02:43:09Z

# What does this PR do?
Fixes two related correctness bugs in DeepseekV4CSACompressor / DeepseekV4Attention that cause multi-token prefill and training-style forwards to silently produce wrong attention patterns when Compressed Sparse Attention (CSA) layers are active.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # #45758

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by yiakwy-xpu-ml-framework-team at 2026-05-08T06:49:54Z ---
The huggingface implementation may interfere correctness in training process if Megatron implemented the function in the same way.

But it may not interfere infrence process since it intended for inference:

```
assert seq_len == 1, "CSA only supports single query in eager mode"
```

--- Comment by yiakwy-xpu-ml-framework-team at 2026-05-08T06:52:57Z ---
hi buddy @Prachi-kushwaha , you didn't commit any codes ?

--- Comment by Prachi-kushwaha at 2026-05-08T07:17:23Z ---
> hi buddy @Prachi-kushwaha , you didn't commit any codes ?

I’m working on it. Will make the commit once the implementation is complete.

--- Comment by Prachi-kushwaha at 2026-05-08T08:07:03Z ---
> The huggingface implementation may interfere correctness in training process if Megatron implemented the function in the same way.
>
> But it may not interfere infrence process since it intended for inference:
>
> ```
> assert seq_len == 1, "CSA only supports single query in eager mode"
> ```

it will hit at inference too specifcally during prefill phase


PULL REQUEST #45836: fix(minicpmv4_6): skip invalid failing tests
State: closed  |  Merged: True
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-05-08T01:24:20Z

This PR fixes failed test case:
`tests/models/minicpmv4_6/test_modeling_minicpmv4_6.py::MiniCPMV4_6ModelTest::test_generate_with_quant_cache`
that is not suitbale, just skip it.
@ydshieh pls help review.

--- Comment by github-actions[bot] at 2026-05-09T02:39:43Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: minicpmv4_6, qwen3_5, qwen3_5_moe, qwen3_next

--- Comment by zucchini-nlp at 2026-05-11T12:20:09Z ---
yeah, not super useful atm so can be skipped

--- Comment by HuggingFaceDocBuilderDev at 2026-05-11T12:28:22Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45836). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-05-08T14:42:18Z ---
Any idea why this is not skipped in base qwen 3.5 then?

--- Comment by kaixuanliu at 2026-05-09T01:36:20Z ---
Well, it fails as well. `tests/models/qwen3_5/test_modeling_qwen3_5.py::Qwen3_5ModelTest::test_generate_with_quant_cache`

--- Comment by kaixuanliu at 2026-05-09T02:39:51Z ---
I have skipped all other Qwen3.5 related `test_generate_with_quant_cache` cases as well.


PULL REQUEST #45835: fix: kosmos2.5: properly expand embeddings table
State: open  |  Merged: False
Author: nunq  |  Base: main
Labels:
Created: 2026-05-08T00:40:59Z

# What does this PR do?

Fixes #45834

when giving a large input file into kosmos2.5's `<ocr>` mode while using `max_new_tokens=4096`, it sometimes fails with an `index out of range in self` error. this is because the embedding table isn't properly expanded because `past_key_values_length=0` is hardcoded in the forward call. if the input is too large (=past expanded size), `return self.weights.index_select()` fails with the out of range error.

using the regression.py script provided in the issue, one can test that the embeddings table is properly expanded now:

```sh
# uv run regression.py
prompt_len=2052, padding_idx=1, initial_table=4098, final_table=6150
[PASS] all 4096 steps completed without crash
```

tests `tests/models/kosmos2_5/test_modeling_kosmos2_5.py` pass: 117 passed, 126 skipped, 26 warnings in 30.64s, same as before my changes.

i have also successfully tested some input files that previously failed with this error.

---

cc tbd, waiting for ci

--- Comment by github-actions[bot] at 2026-05-08T09:31:06Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: kosmos2_5

--- Comment by zucchini-nlp at 2026-05-11T17:24:08Z ---
shouldn't be like that, instead we have to pass the correct value fot `past_key_values` length. Can you change the hardcoded zero?

Also i am wondering, is model simply failing when generating with cache or is this an edge use-case? We'll need a test if not yet covered

--- Comment by nunq at 2026-05-11T19:58:43Z ---
yes, you're right. i will have a look tomorrow! btw there is an ai bot hijacking [the related issue](https://github.com/huggingface/transformers/issues/45834#issuecomment-4423037767) and [5 others in this repository](https://github.com/huggingface/transformers/issues?q=involves%3Ajshaofa-ui), just so you know :)


PULL REQUEST #45833: Add Qwen3.5 support for token classification
State: closed  |  Merged: True
Author: Ratwood98  |  Base: main
Labels:
Created: 2026-05-07T23:30:45Z

Adds token-classification support for Qwen3.5 in AutoModelForTokenClassification.

**What does this PR do?**
This PR enables loading Qwen3.5 checkpoints with `AutoModelForTokenClassification`, which previously failed with:
`ValueError: Unrecognized configuration class Qwen3_5Config for AutoModelForTokenClassification`.

**Changes**

- Adds Qwen3_5ForTokenClassification in:
  - `src/transformers/models/qwen3_5/modular_qwen3_5.py`
  - generated `src/transformers/models/qwen3_5/modeling_qwen3_5.py`
- Exports the new class via the Qwen3.5 modeling `__all__`.
- Registers auto-mapping entries in:
  - `src/transformers/models/auto/modeling_auto.py`
  - `("qwen3_5", "Qwen3_5ForTokenClassification")`
  - `("qwen3_5_text", "Qwen3_5ForTokenClassification")`
- Adds documentation in `docs/source/en/model_doc/qwen3_5.md`
- Adds test class registration in `tests/models/qwen3_5/test_modeling_qwen3_5.py`

**Why both mappings?**
Qwen3.5 uses a composite VLM config (qwen3_5) with a text sub-config (`qwen3_5_text`).
Registering **both** ensures classification works for direct text config usage and composite config loading paths.

**Before:**
Loading from e.g. Qwen/Qwen3.5-0.8B raises `ValueError: Unrecognized configuration class Qwen3_5Config for AutoModelForTokenClassification`.

**After this PR:**
Loading from `Qwen/Qwen3.5-0.8B` now resolves to `Qwen3_5ForTokenClassification`.

This PR follows the same pattern as #44406 (SequenceClassification support).

@Cyrilvallez @zucchini-nlp @ArthurZucker

--- Comment by github-actions[bot] at 2026-05-07T23:32:16Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, qwen3_5

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T11:44:18Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45833). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by zucchini-nlp at 2026-05-08T12:07:36Z ---
this is not correct, text-config should match to text-classfiier and the Config to general classifier, in case we want to have both classes

But ideally one class is enough because mm data is optional

--- Comment by zucchini-nlp at 2026-05-08T12:07:50Z ---
I will revert and fix it in linked PR and merge


PULL REQUEST #45830: fix: raise ValueError when num_beams × vocab_size exceeds torch.multinomial limit
State: closed  |  Merged: False
Author: YousefZahran1  |  Base: main
Labels: Code agent slop
Created: 2026-05-07T16:00:57Z

## What
`model.generate()` with `do_sample=True` and a large `num_beams` on a large-vocabulary model crashes with a cryptic `RuntimeError: number of categories cannot exceed 2^24` inside `_get_top_k_continuations`. The error comes from PyTorch's CUDA multinomial being limited to 16,777,216 (`2^24`) categories, while the flattened `num_beams × vocab_size` tensor can easily exceed this (e.g. 128 beams × 164,096 vocab = 21M categories).

## Why
The crash happens deep in CUDA internals with no indication of which generation parameters caused it or how to fix it. Users of large multilingual models (Qwen, Yi, Llama-3 variants with 128k+ vocab) are increasingly hitting this when experimenting with large beam counts.

## Reproduce
```python
import torch
# Simulate the limit check without a full model run:
num_beams, vocab_size = 128, 164_096
assert num_beams * vocab_size > 2**24  # 21,004,288 > 16,777,216
# RuntimeError: number of categories cannot exceed 2^24
```

## Fix
Add a `ValueError` in `_get_top_k_continuations` immediately before the `torch.multinomial` call. The error message names the offending parameters and tells the user the maximum safe `num_beams` for their model. This is a **validation-only change** — no behaviour change for valid configurations.

```
ValueError: Beam-search sampling requires torch.multinomial over 21,004,288 candidates
(num_beams=128 × vocab_size=164,096), but PyTorch's CUDA multinomial is limited to
16,777,216 categories. Reduce num_beams to at most 102 for this model's vocab size.
```

Previous PR #45251 was closed for being over-engineered; this PR is the minimal approach discussed in the issue thread.

Fixes #45245

--- Comment by github-actions[bot] at 2026-05-07T16:15:12Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45830&sha=2aa2fe


PULL REQUEST #45829: fix(rf_detr): correct paper URL and stale checkpoint references
State: closed  |  Merged: False
Author: omkar-334  |  Base: main
Labels:
Created: 2026-05-07T14:39:29Z

## Summary

1. The RF-DETR model doc (`docs/source/en/model_doc/rf_detr.md`) linked to https://huggingface.co/papers/2407.17140, which is actually the RT-DETRv2 paper ("RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer", Lv et al., Baidu). The correct RF-DETR paper is https://huggingface.co/papers/2511.09554 ("RF-DETR: Neural Architecture Search for Real-Time Detection Transformers", Robinson et al., Roboflow + CMU).
2. The intro called RF-DETR "Receptive Field Detection Transformer", which the paper never says. Rewrote it to match the actual paper: a NAS-based, DINOv2-initialized modernization of LW-DETR.
3. Fixed wrong checkpoint names in docstring examples (`stevenbucaille/rfdetr_small_60e_coco` -> `stevenbucaille/rf-detr-base`) in `modular_rf_detr.py`, and propagated to the generated `configuration_rf_detr.py` and `modeling_rf_detr.py`.

## Code Agent Policy
- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?

## Who can review?
@yonigozlan @sbucaille

--- Comment by github-actions[bot] at 2026-05-07T14:40:49Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: rf_detr

--- Comment by sbucaille at 2026-05-07T15:49:02Z ---
Hi @omkar-334 , thanks for catching these errors, the RF-DETR PR just got merged and still need some adjustments because we will move the checkpoint to another organization on the hub, so another PR will come later to tackle all that, but I'll include your suggestions


PULL REQUEST #45828: [Weight converter] Revert unnecessary changes to `rename_source_key`
State: closed  |  Merged: True
Author: yonigozlan  |  Base: main
Labels:
Created: 2026-05-07T13:24:00Z

Now that we correctly reverse "renamings -> converter" to "converter -> renamings" when saving weights, we can keep the separation of renamings and converters in `rename_source_key`

Also have the dummy models inherit from PreTrainedModel in core_model_loading tests

Cc @ArthurZucker @Cyrilvallez

--- Comment by HuggingFaceDocBuilderDev at 2026-05-07T13:37:55Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45828). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Delete dead code in qwen-vl series

PULL REQUEST #45827: Delete dead code in qwen-vl series
State: open  |  Merged: False
Author: zucchini-nlp  |  Base: main
Labels:
Created: 2026-05-07T12:52:24Z

# What does this PR do?

as per title:

- `rotary_pos_emb` is not accessed in forward and is actually a duplicate of `position_embeddings`
- `rope_deltas` also not accessed in forward and duplicates an instance attr `self.rope_deltas`
- returned ModelOutput doesn't need to have `output.rope_deltas` because it is fixed and also can be obtained from `model.base_mode.rope_deltas`
-

--- Comment by HuggingFaceDocBuilderDev at 2026-05-07T13:05:09Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45827). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by zucchini-nlp at 2026-05-19T06:44:25Z ---
cc @vasqu , this is ready now whenever you have time. Some light clean-up of unused code in qwen models

--- Comment by github-actions[bot] at 2026-05-19T06:45:05Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ernie4_5_vl_moe, exaone4_5, glm46v, glm4v, glm4v_moe, glm_image, glm_ocr, paddleocr_vl, qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_5, qwen3_5_moe, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe

--- Comment by vasqu at 2026-05-19T12:59:38Z ---
Can you look up when this was actually added or revert? Pretty sure it was just recently, max dec 2025

--- Comment by vasqu at 2026-05-19T13:15:54Z ---
I think we are missing a few `deprecate_kwarg` here

--- Comment by vasqu at 2026-05-19T13:18:29Z ---
Seems like some inheritance is not there so some decorators are missing now, e.g. the deprecator one

--- Comment by vasqu at 2026-05-19T13:18:48Z ---
here it seems

--- Comment by vasqu at 2026-05-19T13:19:13Z ---
or this might be an issue here instead that it was missing

--- Comment by vasqu at 2026-05-19T13:19:19Z ---
and here

--- Comment by vasqu at 2026-05-19T13:20:18Z ---
Imo, would like this to be a different PR, this seems like it could have a wider impact (and could reduce a few modular files)

--- Comment by zucchini-nlp at 2026-05-20T01:54:03Z ---
Yeah, in december, it says " on 2025-12-16.*" :)

This change comes from release data of actual paper on the hub, the date when the paper was published on the hub and is automatically added by "fix-repo"

The wording is weird though and imo it should be smth like "The model paper was published on HF papers on xxxx-xx-xx". I have it long in my TODO to check what happens and fix automatic dates, not the first time seeing random changes on it

--- Comment by zucchini-nlp at 2026-05-20T01:55:42Z ---
I added `deprecate_kwarg` on public classes only on purpose, not sure if we need to do with modules that are not high lvl imported given that `rotary_pos_emb` was never used anyway and will be consumed via kwargs if passed

--- Comment by zucchini-nlp at 2026-05-20T02:11:19Z ---
you mean on `rotary_embeddings`? I answered above

--- Comment by zucchini-nlp at 2026-05-20T02:24:45Z ---
it is directly linked to the changes here to pass CI, because I am inheriting the `ModelOutput` class and deleted the docstring

--- Comment by vasqu at 2026-05-20T07:39:39Z ---
Imo, even if it's more private modules youd be surprised how many people still use it 😢 it doesnt hurt to add it there for the cushion/security benefit we get

--- Comment by vasqu at 2026-05-20T07:40:45Z ---
Oh lol I completely overlooked the addition time on transformers but yea good point. Something messy is definitely happening from time to time here

--- Comment by vasqu at 2026-05-20T07:42:23Z ---
I think current models completely override the docstring in this case to pass ci?

I'd really just love something that cleans this up after this is merged. Also fine with me in a subsequent PR or at least an issue for us to keep track with this


Fix gemma4 with multi-gpu setup

PULL REQUEST #45826: Fix gemma4 with multi-gpu setup
State: closed  |  Merged: True
Author: zucchini-nlp  |  Base: main
Labels:
Created: 2026-05-07T12:51:35Z

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/45823

Sometimes embeddings end up in a different device than inputs, so we move the `mask` to the same device

--- Comment by github-actions[bot] at 2026-05-07T12:53:05Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4

--- Comment by HuggingFaceDocBuilderDev at 2026-05-07T13:03:44Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45826). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45825: [CI] Replace PAT with GitHub App token in repo-consistency-bot
State: open  |  Merged: False
Author: paulinebm  |  Base: main
Labels:
Created: 2026-05-07T12:24:16Z

## Summary

- Replaces static PAT (`HF_STYLE_BOT_ACTION`) with short-lived GitHub App token (`HF_BOT_STYLE_APP_ID` + `HF_BOT_STYLE_SECRET_PEM`) in the `commit-and-comment` job
- Drops `contents: write` from the job permissions — the push now uses the app token directly, not GITHUB_TOKEN
- Adds `lfs.locksverify=false` to avoid LFS lock verification errors on contributor forks

⚠ Requires `HF_BOT_STYLE_APP_ID` and `HF_BOT_STYLE_SECRET_PEM` secrets to be set in this repo's settings.

--- Comment by HuggingFaceDocBuilderDev at 2026-05-07T12:38:27Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45825). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by Rocketknight1 at 2026-05-08T11:17:00Z ---
cc @tarekziade @ydshieh


PULL REQUEST #45824: fix: correct typo 'seperate' -> 'separate' in comments across multipl…
State: closed  |  Merged: True
Author: Caromisc  |  Base: main
Labels:
Created: 2026-05-07T11:50:40Z

Fix typo: correct misspelling of 'seperate' -> 'separate' in comments and docstrings across 10 files in the following models: GLM46V, GLM4V, GLM4V-MoE, GLM-OCR, Qwen3.5, Qwen3.5-MoE, Qwen3-VL, and Qwen3-VL-MoE.

Note: `image_seperate.weight` in `convert_mm_grounding_dino_to_hf.py` was intentionally left unchanged as it is an exact key name from the original model checkpoint.

20 occurrences fixed across 10 files.

--- Comment by github-actions[bot] at 2026-05-07T11:52:01Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: glm46v, glm4v, glm4v_moe, glm_ocr, qwen3_5, qwen3_5_moe, qwen3_vl, qwen3_vl_moe

--- Comment by Rocketknight1 at 2026-05-08T11:16:02Z ---
My secondary school English teacher would have approved

--- Comment by HuggingFaceDocBuilderDev at 2026-05-08T11:26:36Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45824). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45822: Fix added-token prefix space in SentencePiece fast tokenizers
State: open  |  Merged: False
Author: qflen  |  Base: main
Labels:
Created: 2026-05-07T10:27:43Z

Fixes #28218

When a user adds a token to a SentencePiece-style fast tokenizer (e.g. NLLB), the tokens-trie splits the input
at the new token and runs the Metaspace pre_tokenizer on each chunk.

The default `prepend_scheme="always"` then prefixes every chunk with `▁`, so `"abcdgym"` tokenizes to `["abcd", "▁gym"]` and `decode(encode("abcdgym"))` returns `"abcd gym"`.

This PR mirrors what `SpmConverter.pre_tokenizer` already does for non-legacy spm tokenizers:
when a post-init, non-special `add_tokens` call lands on a standalone `Metaspace(prepend_scheme="always")`, the pre_tokenizer is swapped for one with `prepend_scheme="first"` so only the first chunk receives the prefix.

Sequence pipelines (WhitespaceSplit + Metaspace, used by T5, Pegasus, UDOP, LASR) are left untouched because they rely on `"always"` for whitespace handling.

With `tokenizer.add_tokens(["abcd"])` applied to a fresh NLLB tokenizer:

| Input | Before | After |
| --- | --- | --- |
| `"abcdgym"` | `["abcd", "▁gym"]` →  `"abcd gym"` | `["abcd", "gym"]` → `"abcdgym"` |
| `"I like to walk abcdgym along the beach"` | `[..., "▁walk", "▁", "abcd", "▁gym", "▁along", ...]` → `"I like to walk abcd gym along the beach"` | `[..., "▁walk", "▁", "abcd", "gym", "▁along", ...]` → `"I like to walk abcdgym along the beach"` |
| `"abcd gym"` (literal space already there) | `["abcd", "▁gym"]` | `["abcd", "▁gym"]` (unchanged) |

Inputs without any added tokens are unchanged (e.g. `"Hello world"` → `["▁Hello", "▁world"]` either way).

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the contributor guideline, Pull Request section?
- [x] Was this discussed or approved via a Github issue or the forum? Please add a link to it if that is the case.
  Issue: https://github.com/huggingface/transformers/issues/28218
  (`@ArthurZucker` proposed mirroring the SpmConverter approach)
- [ ] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

## Who can review?

`@ArthurZucker` `@itazap`

---

## AI assistance disclosure

I've used Opus 4.7 Max Effort for debugging and reviewed / double-checked every line.

- Coordination:
https://github.com/huggingface/transformers/issues/28218  (maintainer guidance to follow `SpmConverter.pre_tokenizer`

- Claimed issue in:
  https://github.com/huggingface/transformers/issues/28218#issuecomment-2050562552)

- Why this is not a duplicate:
The only related open PR (#31315) targets a different issue (#30824, plumbing `add_prefix_space` through the fast tokenizer); it does not touch the post-init alignment of `prepend_scheme` for added tokens.

- Tests run:
  - `make style` and `make typing` and `make fix-repo`, all green, no extra diff
  - `pytest tests/tokenization/test_tokenization_fast.py` (11 passed, 3 skipped)
  - `pytest tests/test_tokenization_common.py` (18 passed, 3 skipped)
  - `pytest tests/models/nllb/test_tokenization_nllb.py` (61 passed, 4 skipped)
  - `pytest tests/models/big_bird/test_tokenization_big_bird.py` (100 passed, 4 skipped)
  - `pytest tests/models/{reformer,t5,albert,udop,moshi,seamless_m4t}/test_tokenization_*.py` (all green)
  - The new regression test `test_added_token_does_not_get_metaspace_prefix` was verified to fail on main
    (without the fix) and pass with the fix.

--- Comment by Rocketknight1 at 2026-05-07T11:03:30Z ---
cc @arthurzucker @itazap

--- Comment by itazap at 2026-05-12T07:27:03Z ---
okay but since "always" and "first" are equivalent for normal text (without added tokens), the root fix is just: in `_get_prepend_scheme`, return "first" instead of "always" when a`dd_prefix_space=True`. That would fix new conversions to fast ! we just need to handle it in _add_tokens for the cases when loading a `.json` from the hub but should be a bit simpler without new flags

--- Comment by qflen at 2026-05-12T14:22:23Z ---
Thanks for the feedback. Pushed changes.

--- Comment by github-actions[bot] at 2026-05-12T14:22:35Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: big_bird, xlnet

--- Comment by itazap at 2026-05-12T07:29:32Z ---
I think we can directly apply the fix for meatspace without the function or flag!

--- Comment by itazap at 2026-05-12T07:30:16Z ---
great thanks! maybe we can use a 'real' tokenizer model (+pinned version)


[CB] [Major] Add tensor paralellism

PULL REQUEST #45821: [CB] [Major] Add tensor paralellism
State: closed  |  Merged: True
Author: remi-or  |  Base: main
Labels:
Created: 2026-05-07T09:32:38Z

This PR adds support for TP in continuous batching. The major changes required to do this were:
- add inter-process communication for the requests states
- add per TP group seeding
- add hints to prevent NCCL graph mixing
- change hash function to avoid python `hash` which is salted depending on the process

It also adds a mechanism to the benchmark script to make sure the generation is coherent.

## Performance

| Benchmark | TP1 main tok/s | TP1 tok/s | TP2 tok/s | Speedup | TP1 acc | TP2 acc |
|---|---|---|---|---|---|---|
| gsm8k_default | 2,454 | 2,454 | 3,657 | 1.49× | 0.822 | 0.819 |
| gsm8k_sampling | 1,940 | 1,942 | 2,616 | 1.35× | 0.792 | 0.775 |
| gsm8k_compile | 2,463 | 2,467 | 3,689 | 1.50× | 0.822 | 0.821 |
| gsm8k_no_fast_decode | 2,367 | 2,370 | 3,522 | 1.49× | 0.822 | 0.819 |
| gsm8k_bare_bones | 1,877 | 1,881 | 2,331 | 1.24× | 0.821 | 0.821 |
| ifeval_default | 7,890 | 7,898 | 15,135 | 1.92× | 0.442 | 0.455 |
| rollouts_1024 | 3,200 | 3,199 | 4,281 | 1.34× | — | — |
| rollouts_2048 | 3,048 | 3,049 | 4,194 | 1.38× | — | — |
| rollouts_4096 | 2,719 | 2,719 | 3,887 | 1.43× | — | — |
| rollouts_8192 | 2,209 | 2,211 | 3,345 | 1.51× | — | — |
| rollouts_16384 | 1,465 | 1,465 | 2,589 | 1.77× | — | — |
| few_blocks | 686 | 696 | 840 | 1.21× | — | — |
| multi_return_seq | 1,544 | 1,553 | 1,926 | 1.24× | — | — |

No perf regression, TP is faster.

## Tests

Added tests for TP, all tests run.

--- Comment by HuggingFaceDocBuilderDev at 2026-05-07T09:43:15Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45821). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ArthurZucker at 2026-05-12T05:09:03Z ---
only if the attention k and v are target of the tp plan tho

--- Comment by ArthurZucker at 2026-05-12T05:22:44Z ---
Okay will read the rest to see if all are drivers or not

--- Comment by ArthurZucker at 2026-05-12T05:23:22Z ---
okay

--- Comment by ArthurZucker at 2026-05-12T05:31:04Z ---
Let's make a func for easy overwrite? IDK if there can be an advantage but let's give freedom?

--- Comment by remi-or at 2026-05-12T10:14:50Z ---
What could be the other targets? Not familiar enough with the TP plan tbh

--- Comment by remi-or at 2026-05-12T10:15:17Z ---
?

--- Comment by remi-or at 2026-05-12T10:16:10Z ---
Why not, adding it.

--- Comment by remi-or at 2026-05-12T11:08:29Z ---
Ok, added a boolean `kv_is_tp = "layers.*.self_attn.k_proj" in config.tp_plan and "layers.*.self_attn.v_proj" in config.tp_plan` to condition this.


PULL REQUEST #45819: fix: use per-instance lru_cache in compile_compatible_method_lru_cache to prevent memory leak
State: open  |  Merged: False
Author: fyhon  |  Base: main
Labels:
Created: 2026-05-07T08:25:29Z

## Summary

Replace class-level `lru_cache` with per-instance cache in `compile_compatible_method_lru_cache` to prevent memory leaks when model instances are deleted.

Fixes #45412

## Changes

- Modified `src/transformers/pytorch_utils.py`: For methods (detected via first parameter named `self`), the decorator now stores a per-instance `lru_cache` in the instance's `__dict__` using a weakref destructor for cleanup. For standalone functions, the original shared cache behavior is preserved.

## Root Cause

`compile_compatible_method_lru_cache` wraps methods with a class-level `functools.lru_cache`. Since `self` is the first positional argument, `lru_cache` holds a strong reference to the model instance as a cache key, preventing garbage collection even after `del model` + `gc.collect()`. This affects RT-DETR, RT-DETR-v2, MaskFormer, Mask2Former, OneFormer, SAM2, and 20+ other models that use this decorator.

## Duplicate Check

- `gh pr list --state open --search "45412"` → no duplicate PRs
- `gh pr list --state open --search "memory leak lru_cache"` → no duplicate PRs
- Prior PRs #45708 and #45780 were closed without merge

## Testing

```python
import gc, weakref, functools, inspect

# Simulated test of the fix logic
class Model:
    @compile_compatible_method_lru_cache
    def compute(self):
        return [1, 2, 3]

m = Model()
ref = weakref.ref(m)
m.compute()  # populate cache
del m
gc.collect()
assert ref() is None  # instance is properly garbage collected
```

```
Test 1 PASSED: cache returns same object
Test 2 PASSED: instance is garbage collected after del
Test 3 PASSED: standalone function cache works
Test 4 PASSED: different instances have independent caches
Test 5 PASSED: deleting one instance does not affect another

ALL TESTS PASSED
```

## AI Disclosure

This PR was created with AI assistance (Claude). The human maintainer has reviewed and approved the changes before submission.