PULL REQUEST #45635: qa: speed up dtype regex weight load + reduce dtype tests to 3 random
State: open  |  Merged: False
Author: tarekziade  |  Base: main
Labels:
Created: 2026-04-24T14:39:03Z

# What does this PR do?

1. make sure we reuse the complied regex across calls
2. reduce the test to 3 dtypes only (picked randomly across all supported) so we speed up the tests but don't lose coverage over time.

on my m5 that drops

```
tests/models/d_fine/test_modeling_d_fine.py::DFineModelTest::test_bc_torch_dtype
```

from 7.13s to 2.17s.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T14:50:38Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45635). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by tarekziade at 2026-04-27T06:10:50Z ---
> Hey @tarekziade! I'm not really convinced that the regex compiling and the `get_submodule` calls are actual bottlenecks! This very very simple snippet:
>
> ```python
> import re
> import time
>
> N = 1000
>
> a = ["qwe|ui"] * 15
> # Construct the regex we will use to rename keys from the sources to the targets
> branches = []
> for i, source_pattern in enumerate(a):
>     group_name = f"g{i}"
>     pattern = source_pattern.replace(".*.", r"\..*\.")
>     branches.append(f"(?P<{group_name}>{pattern})")
>
> t0 = time.time()
> for _ in range(N):
>     compiled_sources = re.compile("|".join(branches))
> dt = time.time() - t0
> print(f"Took {dt/N:.2e} s")
> ```
>
> shows that the `compile` call is only of the order of 1e-6/1e-7. And for `get_submodule`, the complexity is bounded by the depth of the graph, which is super small as well (usually maximum 10 I'd say)

Sorry I should have been clearer, `re.compile` itself is not that slow, it's the fact that it's rarely needed, so by lazy loading it we get a small speedup because of the numerous `WeightRenaming` instances we create when loading weights. That class is a hot path and its constructor should stay as lightweight as possible.

Using ~18,000 keys and 9 repeats,  median times:

  - WeightRenaming init only
      - old: 0.589529s
      - new: 0.069270s
      - gain: 0.520258s (88.2%), 8.51x faster
  - WeightRenaming init + first rename_source_key()
      - old: 0.605083s
      - new: 0.602909s
      - gain: 0.002174s (0.4%), effectively flat
  - top-level rename_source_key() prefix remove path
      - old: 0.009985s
      - new: 0.005094s
      - gain: 0.004892s (49.0%), 1.96x faster
  - top-level rename_source_key() prefix add path
      - old: 0.004415s
      - new: 0.004897s
      - delta: -0.000483s (-10.9%)
      - this is a tiny absolute regression, about 0.48 ms over all 18,424 keys

Now with the tests being slow right now, random 3x dtypes solves it. I guess it's deciding if optimizing  `WeightRenaming` worth the added complexity.

--- Comment by tarekziade at 2026-04-27T06:22:38Z ---
Using the D-FINE `test_bc_torch_dtype` run as the baseline, the lazy regex compile is much smaller than cutting the dtype matrix, but it is still measurable.

- Full 7-dtype version of tests/test_modeling_common.py 7.13s
- 3-random-dtype version 2.17s
- Benefit from reducing dtypes 4.96s saved, about 69.6% faster

For the lazy regex change in src/transformers/core_model_loading.py, I benchmarked the exact number of weights loaded by the old 7-dtype D-FINE run:

- The test does 28 loads total: 14 x 650 weights and 14 x 666 weights
- That is 18,424 loaded weights total
- The isolated old-vs-new regex benchmark over 18,424 keys showed 0.520s saved from lazy regex compilation

So, at the whole-test level:
- Lazy regex benefit on the old 7-dtype run: about 0.52s
- 3-random-dtype benefit: 4.96s
- Lazy regex is about 10.5% of the dtype-reduction benefit

To recap, the lazy regex compile is a smaller but real secondary win. But in practice, when running in production, regex is quite a small win since we don't load but a single dtype at a time.

That said I would still recommend doing it and add a comment in that class to say it's instanciated a lot at load time and its consrtuctor should stay as thin as possible

--- Comment by Cyrilvallez at 2026-04-27T07:42:05Z ---
Alright I see, thanks for the added explanation!
Agreed that we can keep the lazy regex compile, since it's a very easy and sometimes noticeable improvement! Let's just add why we do it in the property - mostly the fact that every key will go through a fresh WeightRenaming if not matching any weight ops, for convenience (but those don't need to call `rename_source_key`, so the compiling is wasted)

--- Comment by ydshieh at 2026-04-28T12:35:22Z ---
Hi @tarekziade . Thanks for the work.

The origin of this optimization is for make some tests faster. The part of random 3x dtypes solves it works great.

For the changes in the core model loading file, so far the benefit is nit (in our tests, or if I load a single tiny model). However I am not saying it's not useful, but instead I am wondering:

  - if that change would show more benefit when we load a large (huge) model a single time?
  - and for small models (like in our CircleCI), the (absolute) amount of saving would become more visible if we load a model multiple times?
  - For the `deepcopy` part, is it really necessary?  Does `re.compile` already use some cache mechanism so the compile of the **same** pattern would not be slow (I am not sure and I might be wrong here).

Overall, I am fine as long as @Cyrilvallez is happy with the change. But it's nice if we can identify which part is really improving the loading, and limit the change to the smallest scope as possible.

--- Comment by tarekziade at 2026-04-28T13:03:24Z ---
@ydshieh Thanks for the thoughtful questions, that’s very helpful.

You’re right that most of the measurable speedup comes from reducing the number of dtypes we iterate over.

The changes in the core loading path are more about a design concern that surfaced while doing this: the weight class used for conversion does some preparation work in its constructor that is rarely used, while it sits directly on the critical path (one instance per key).

So the goal there is mainly to keep that constructor as lightweight as possible e.g. avoid creating objects (like regexps) that we don’t actually use, and prevent this from growing over time.

In terms of impact:

- On large key counts (~18k), this shaves ~0.5s (what we see in tests today)
- For typical models (~500 keys), the gain is indeed negligible on a single load

I haven’t tested large models yet, but I’d expect the benefit to scale with the number of keys rather than model size itself

So I agree this is more of a “keep the hot path clean” improvement than a big performance win.

On `deepcopy`: it’s needed because `collected_tensors`, `layer_targets`, and `_was_used` accumulate state during loading, so we need a fresh instance per target. This is orthogonal to `re.compile`.

And you’re right about `re.compile` it’s cached in the stdlib itself, so repeated calls are not a concern here.

Happy to reduce the scope if we feel this is too much for the current benefit 👍

--- Comment by ydshieh at 2026-04-28T14:28:20Z ---
OK, thanks for explaining. So `deepcopy` is needed (but not for the reason of `re.compile`).

 > Happy to reduce the scope if we feel this is too much for the current benefit 👍

 Since Cyril is convinced, so fine from my side after your above comment 👍

--- Comment by tarekziade at 2026-04-30T06:44:41Z ---
@ydshieh or @Cyrilvallez anything else to tweak?

--- Comment by Cyrilvallez at 2026-04-27T03:04:54Z ---
From my tests, compiling a regex is extremely fast, of the order of `1e-7`/`1e-6` s. Are you sure it's actually a bottleneck?

--- Comment by Cyrilvallez at 2026-04-27T03:05:36Z ---
Probably not needed to have an outer function here

--- Comment by Cyrilvallez at 2026-04-27T03:09:49Z ---
`get_submodule` should be extremely fast as well, see [here](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.get_submodule) where they say that the compelxity depends on the nesting, which is always small

--- Comment by Cyrilvallez at 2026-04-27T03:10:48Z ---
Not really nice IMO to define an inner function like this

--- Comment by tarekziade at 2026-04-28T07:06:03Z ---
it's used in two spots that's why I vectorized it here

--- Comment by tarekziade at 2026-04-28T07:06:42Z ---
agreed, I will extract it

--- Comment by Cyrilvallez at 2026-05-11T03:43:32Z ---
We don't need this as we already have the `compiled_sources` property avoiding to run `re.compile` on `__init__`!

--- Comment by Cyrilvallez at 2026-05-11T03:47:05Z ---
Would really much rather revert that as well, see previous comment - too hard to read for no gains IMO

--- Comment by Cyrilvallez at 2026-05-11T03:48:25Z ---
Same here, let's revert


PULL REQUEST #45634: DeepGEMM BF16 + mixed FP8/FP4 + MegaMoE + refactor
State: open  |  Merged: False
Author: IlyasMoutawwakil  |  Base: main
Labels: for patch
Created: 2026-04-24T14:38:32Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Based on #45621
GroupedLinear co-authored-by @sywangyi

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T14:54:22Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45634). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-05-19T01:17:22Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_v4, finegrained_fp8

--- Comment by ArthurZucker at 2026-05-12T00:34:17Z ---
needs to be done with tp plan instead 😉

--- Comment by ArthurZucker at 2026-05-12T09:20:32Z ---
mmmm this should be done in deepseek config maybe? weird to have here TBF!

--- Comment by ArthurZucker at 2026-05-12T09:21:03Z ---
keep_in_fp32 should prevent it frombeing targetted actually no?

--- Comment by ArthurZucker at 2026-05-12T09:21:43Z ---
Mmmm this is a but weird and comment is not on point

--- Comment by ArthurZucker at 2026-05-12T09:22:16Z ---
lgtm

--- Comment by ArthurZucker at 2026-05-12T09:22:23Z ---
better yep

--- Comment by ArthurZucker at 2026-05-12T09:22:33Z ---
flagging for the merge

--- Comment by ArthurZucker at 2026-05-12T13:03:21Z ---
As discussed offline:
```python
self.scorer(q, compressed_kv)
```

--- Comment by ArthurZucker at 2026-05-12T13:04:16Z ---
I wish we could avoid this! but if not okay, let's find closest model

--- Comment by ArthurZucker at 2026-05-12T13:05:06Z ---
this prevents the model and people from using the power feature of rope scaling tho!

--- Comment by ArthurZucker at 2026-05-12T13:05:44Z ---
`gate_up_proj_scale_inv`'s plan should implicetely inherit from its counter part!
```python
        "layers.*.self_attn.compressor.indexer.q_b_proj": "colwise",
        "layers.*.self_attn.compressor.indexer.weights_proj": "colwise",
        "layers.*.self_attn.compressor.indexer.scores_sync": "all_reduce",
```
🆙
LGTM

--- Comment by IlyasMoutawwakil at 2026-05-12T13:09:49Z ---
we can use an _apply_gate here as well if that makes sense

--- Comment by ArthurZucker at 2026-05-12T14:19:01Z ---
kinda brittle no? I don't know exactly how we usually decide which to replace... for now its super deepseekv4 specific.

Let's say its fine we can just check class being `GroupedLinear` in name?

--- Comment by ArthurZucker at 2026-05-12T14:20:17Z ---
same comment ! should not appear

--- Comment by ArthurZucker at 2026-05-12T14:21:29Z ---
Do we still need ot overwrite vs `LagunaRotaryEmbedding` ? maybe no longer but thats a nit

--- Comment by ArthurZucker at 2026-05-12T14:22:24Z ---
yep

--- Comment by ArthurZucker at 2026-05-12T14:22:45Z ---
I'd rather its explicit without apply gate TBH!

--- Comment by ArthurZucker at 2026-05-12T14:22:51Z ---
lgtm

--- Comment by IlyasMoutawwakil at 2026-05-12T14:46:33Z ---
aligned now

--- Comment by ArthurZucker at 2026-05-13T03:07:50Z ---
```suggestion
    "deep-gemm": {"repo_id": "adarshxs/deep-gemm", "revision": "v2", version="588a1fd47b4f8cd72e018d784d36fab013fb437d"},
```
can we pin a revision?

--- Comment by ArthurZucker at 2026-05-13T03:08:00Z ---
I can't merge until read here!


CircleCI with torch 2.11

PULL REQUEST #45633: CircleCI with torch 2.11
State: closed  |  Merged: True
Author: ydshieh  |  Base: main
Labels:
Created: 2026-04-24T14:04:39Z

# What does this PR do?

A bit late, but let's switch, because torch 2.12 is already in RC 1

--- Comment by github-actions[bot] at 2026-04-24T14:05:48Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: llava_onevision

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T14:16:06Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45633). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ydshieh at 2026-04-24T14:05:05Z ---
please ignore this part, I will revert before merge. It's just a quick way to check

--- Comment by ydshieh at 2026-04-24T14:08:14Z ---
This causes issues like below.

I have no context of why using this. But we are moving away from `torchvision` anyway as @zucchini-nlp told me, and we  also have

```python
        if not is_torchcodec_available():
            warnings.warn(
                "`torchcodec` is not installed and cannot be used to decode the video by default. "
                "Falling back to `torchvision`. Note that `torchvision` decoding is deprecated and will be removed in future versions. "
            )
            backend = "torchvision"
```

Anyway, the changes works.

### error log

```bash
a = (<tests.models.llava_onevision.test_processing_llava_onevision.LlavaOnevisionProcessorTest testMethod=test_apply_chat_template_video_1>,)
kw = {}

    @wraps(func)
    def standalone_func(*a, **kw):
>       return func(*(a + p.args), **p.kwargs, **kw)

/usr/local/lib/python3.10/site-packages/parameterized/parameterized.py:620:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_processing_common.py:1679: in test_apply_chat_template_video
    self._test_apply_chat_template(
tests/test_processing_common.py:1616: in _test_apply_chat_template
    out_dict = processor.apply_chat_template(
/usr/local/lib/python3.10/site-packages/transformers/processing_utils.py:1898: in apply_chat_template
    out = self(
/usr/local/lib/python3.10/site-packages/transformers/models/llava_onevision/processing_llava_onevision.py:139: in __call__
    video_inputs = self.video_processor(videos, **output_kwargs["videos_kwargs"])
/usr/local/lib/python3.10/site-packages/transformers/video_processing_utils.py:178: in __call__
    return self.preprocess(videos, **kwargs)
/usr/local/lib/python3.10/site-packages/transformers/video_processing_utils.py:354: in preprocess
    videos, video_metadata = self._decode_and_sample_videos(
/usr/local/lib/python3.10/site-packages/transformers/video_processing_utils.py:294: in _decode_and_sample_videos
    videos, video_metadata = self.fetch_videos(videos, sample_indices_fn=sample_indices_fn)
/usr/local/lib/python3.10/site-packages/transformers/video_processing_utils.py:836: in fetch_videos
    return list(zip(*[self.fetch_videos(x, sample_indices_fn=sample_indices_fn) for x in video_url_or_urls]))
/usr/local/lib/python3.10/site-packages/transformers/video_processing_utils.py:836: in <listcomp>
    return list(zip(*[self.fetch_videos(x, sample_indices_fn=sample_indices_fn) for x in video_url_or_urls]))
/usr/local/lib/python3.10/site-packages/transformers/video_processing_utils.py:838: in fetch_videos
    return load_video(video_url_or_urls, backend=backend, sample_indices_fn=sample_indices_fn)
/usr/local/lib/python3.10/site-packages/transformers/video_utils.py:726: in load_video
    video, metadata = video_decoder(file_obj, sample_indices_fn, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

video_path = '/root/project/sample_demo_1.mp4'
sample_indices_fn = functools.partial(<bound method BaseVideoProcessor.sample_frames of LlavaOnevisionVideoProcessor {
  "do_convert_rgb": true,
  "do_normalize": true,
  "do_rescale": true,
  "do_resize": true,
  "do_sample_frames": false,
  "image_mean": [
    0.48145466,
    0.4578275,
    0.40821073
  ],
  "image_std": [
    0.26862954,
    0.26130258,
    0.27577711
  ],
  "resample": 3,
  "rescale_factor": 0.00392156862745098,
  "return_metadata": false,
  "size": {
    "height": 384,
    "width": 384
  },
  "video_processor_type": "LlavaOnevisionVideoProcessor"
}
>, return_tensors='pt', num_frames=2, do_convert_rgb=True, do_resize=True, size=SizeDict(height=384, width=384, longest_edge=None, shortest_edge=None, max_height=None, max_width=None), default_to_square=False, resample=3, do_rescale=True, rescale_factor=0.00392156862745098, do_normalize=True, image_mean=(0.48145466, 0.4578275, 0.40821073), image_std=(0.26862954, 0.26130258, 0.27577711), do_center_crop=None, do_pad=None, crop_size=None, data_format=None, fps=None, return_metadata=False)
kwargs = {}

    def read_video_torchvision(
        video_path: Union["URL", "Path"],
        sample_indices_fn: Callable,
        **kwargs,
    ):
        """
        Decode the video with torchvision decoder.

        Args:
            video_path (`str`):
                Path to the video file.
            sample_indices_fn (`Callable`, *optional*):
                A callable function that will return indices at which the video should be sampled. If the video has to be loaded using
                by a different sampling technique than provided by `num_frames` or `fps` arguments, one should provide their own `sample_indices_fn`.
                If not provided, simple uniform sampling with fps is performed.
                Example:
                def sample_indices_fn(metadata, **kwargs):
                    return np.linspace(0, metadata.total_num_frames - 1, num_frames, dtype=int)

        Returns:
            tuple[`torch.Tensor`, `VideoMetadata`]: A tuple containing:
                - Torch tensor of frames in RGB (shape: [num_frames, height, width, 3]).
                - `VideoMetadata` object.
        """
        warnings.warn(
            "Using `torchvision` for video decoding is deprecated and will be removed in future versions. "
            "Please use `torchcodec` instead."
        )
>       video, _, info = torchvision_io.read_video(
            video_path,
            start_pts=0.0,
            end_pts=None,
            pts_unit="sec",
            output_format="TCHW",
        )
E       AttributeError: module 'torchvision.io' has no attribute 'read_video'. Did you mean: 'read_file'?

/usr/local/lib/python3.10/site-packages/transformers/video_utils.py:538: AttributeError
```

--- Comment by tarekziade at 2026-04-24T14:14:20Z ---
Maybe you remove that step and simply add `--upgrade` on the next one. It should work

--- Comment by tarekziade at 2026-04-24T14:14:32Z ---
ah ok

--- Comment by zucchini-nlp at 2026-04-24T14:43:18Z ---
lol, I don't know when that was added, and indeed it shouldn't be there in the first place

--- Comment by ydshieh at 2026-04-24T15:32:44Z ---
no worry, it was not you :-)


PULL REQUEST #45631: chore: bump doc-builder SHA for main doc build workflow
State: closed  |  Merged: True
Author: rtrompier  |  Base: main
Labels:
Created: 2026-04-24T12:27:14Z

Bump the pinned doc-builder SHA so that main documentation builds also sync to the HF bucket (dual-write introduced in doc-builder PR #780, 2026-04-15). The current pin predates that change, so release docs never land in the bucket — only in the legacy dataset.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T12:37:54Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45631). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Add new model: Kimi2-6

PULL REQUEST #45630: Add new model: Kimi2-6
State: open  |  Merged: False
Author: zucchini-nlp  |  Base: main
Labels:
Created: 2026-04-24T12:26:43Z

# What does this PR do?

The moonshot team is not very active and haven't yet replied, so I kept all the `model_type` naming to match the remote code. That allows us to load a model with local code. There are still a few problems such as tokenizer needing conversion and jinja chat template being non-consistent though

They have saved a tiktoken tokeninzer and tbh I am not sure if we can convert it, and nudge them to update. So as a workaround we convert from tiktoken at load-time

Tested with below script but uses a smaller/dummy model. Logits and eager generation are equivalent with official implementation.

```python
import torch
from transformers import AutoProcessor, AutoTokenizer, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained('moonshotai/Kimi-K2.6', trust_remote_code=False)
model = AutoModelForImageTextToText.from_pretrained(
    'moonshotai/Kimi-K2.6',
    dtype=torch.bfloat16,
    device_map="cuda:0",
    trust_remote_code=False,
    attn_implementation="eager",
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "image.png"},
            # {"type": "video", "video_url":{"url"  "video.mp4"}}, # note that videos need different layout due to jinja template
            {"type": "text", "text": "What is shown in this image?"},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(device=model.device, dtype=torch.bfloat16)

generated_ids = model.generate(**inputs, max_new_tokens=64)
generated_text = processor.batch_decode(generated_ids[:, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)[0]
print(generated_text)
```

--- Comment by zucchini-nlp at 2026-04-24T16:51:28Z ---
Oke, I think the model part looks same as the custom code, though I didn't test it. Also we need to check what is the exact chat template format and how to add `placeholder` tokens per modality, again with the real checkpoint

Kimi team might reply on Monday, coming back then

--- Comment by github-actions[bot] at 2026-05-19T06:18:42Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v3, kimi_k25, qwen2_vl

--- Comment by github-actions[bot] at 2026-05-19T06:34:22Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45630&sha=db102b

--- Comment by vasqu at 2026-05-07T13:10:29Z ---
```suggestion
# Kimi 2.6
```
nit

--- Comment by vasqu at 2026-05-07T13:15:43Z ---
Ah ok nvm, docs dont exist yet 😬

--- Comment by vasqu at 2026-05-07T13:17:07Z ---
```suggestion
                    config = AutoConfig.from_pretrained(
                        pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **kwargs
                    )
```
we already checked pretrained config above

--- Comment by vasqu at 2026-05-07T13:20:05Z ---
Would adding to `MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` work instead as well?

--- Comment by vasqu at 2026-05-07T13:21:20Z ---
Yea we should really request to fix that on the hub if possible, afaik we have on the fly conversion atm

--- Comment by vasqu at 2026-05-07T13:22:18Z ---
same here

--- Comment by vasqu at 2026-05-07T13:29:23Z ---
Not sure how deeply you have looked into tokenizers but imo we definitely should convert them tbh, e.g.
- https://github.com/huggingface/transformers/blob/ebac5e520c1831fff5083c7cd24f5300a7c5a19b/src/transformers/convert_slow_tokenizer.py#L1898
- https://github.com/huggingface/transformers/blob/ebac5e520c1831fff5083c7cd24f5300a7c5a19b/src/transformers/models/gpt_oss/convert_gpt_oss_weights_to_hf.py#L359

This could maybe be used to adapt this conversion to our tokenizers instead.

--- Comment by vasqu at 2026-05-07T13:30:58Z ---
👀

--- Comment by vasqu at 2026-05-07T13:31:51Z ---
Yea would avoid this, let's rewrite and avoid cross imports tbh

--- Comment by vasqu at 2026-05-07T13:32:31Z ---
Can we highlight what exactly is special with some small comments, e.g. here different resize

--- Comment by vasqu at 2026-05-07T13:33:40Z ---
qq: is kimi k25 the same as k26 arch wise?

--- Comment by vasqu at 2026-05-07T13:35:14Z ---
Dont forget the decorators ala autodoc + strict

--- Comment by vasqu at 2026-05-07T13:36:34Z ---
Shouldnt be needed, no? Default theta is already 10k and we "default to default" then

--- Comment by vasqu at 2026-05-07T13:36:48Z ---
same re decorators

--- Comment by vasqu at 2026-05-07T13:37:52Z ---
```suggestion
    projection_layer_norm_eps: float = 1e-5
```
to be more consistent

--- Comment by vasqu at 2026-05-07T13:39:45Z ---
In general a bit unsure about the projection prefix, some possible options maybe
1. Inferred from other values
2. Keep without prefix and change the autodocs explicitly
3. Another subconfig

--- Comment by vasqu at 2026-05-07T13:41:13Z ---
Could it happen that it is already a pretrained config from remote with the wrong model type?

--- Comment by vasqu at 2026-05-07T13:44:53Z ---
We should try to follow our naming conventions a bit more closely. This is nothing else but `inv_freq`

`grid_t` in this case acts as position ids or seq_len (if we look at qwen vl)

--- Comment by vasqu at 2026-05-07T13:46:17Z ---
This is the forward then, but we have fixed number of frames so that we can do this on the fly at init time. Still would be checking with Ilyas PR to adjust the compile friendliness

--- Comment by vasqu at 2026-05-07T13:47:43Z ---
In the end it might be smoother imo to have this as separate RoPE module but no too strong opinion as long as we keep closer to our conventions

--- Comment by vasqu at 2026-05-07T13:49:45Z ---
lets add the dim explicitly to cat

--- Comment by vasqu at 2026-05-07T13:51:45Z ---
Small docstring please to explain, essentially what happens
1. Extract height and width embeddings
    1.1. If they don't fit the perfect shape --> interpolate values to make them be
2. Add RoPE time embeddings (only for videos as frames > 1)
3. Flatten all embeddings into one sequence

--- Comment by vasqu at 2026-05-07T13:52:57Z ---
Maybe small comment for the 3, ig it's the thw

--- Comment by vasqu at 2026-05-07T14:24:47Z ---
This looks a bit more complicated then it should be imo.

```python
    def forward(self, x, position_ids):
        position_ids_expanded = position_ids.permute(1, 2, 0)[..., None].float()  # shape (bs, positions, 2, 1)
        inv_freq_expanded = self.inv_freq[None, None, None, :].float().expand(position_ids_expanded.shape[0], position_ids_expanded.shape[1], 2, -1).to(x.device) # shape (bs, positions, 2, freq_dim)

        device_type = x.device.type if isinstance(x.device.type, str) and x.device.type != "mps" else "cpu"
        with maybe_autocast(device_type=device_type, enabled=False):  # Force float32
            freqs = (inv_freq_expanded.float() * position_ids_expanded.float()).transpose(2, 3).flatten(2)
            emb = torch.cat([freqs, freqs], dim=-1)
            cos = emb.cos() * self.attention_scaling
            sin = emb.sin() * self.attention_scaling

        return cos, sin
```

Not 100% sure but it should be something around that. The whole interleave is implicitly integrated into the shapes itself on the mul.

--- Comment by vasqu at 2026-05-07T14:29:21Z ---
```suggestion
        query_states = self.q_proj(hidden_states).reshape(seq_length, -1, self.head_dim)
        key_states = self.k_proj(hidden_states).reshape(seq_length, -1, self.head_dim)
        value_states = self.v_proj(hidden_states).reshape(1, seq_length, -1, self.head_dim)
```
if we were to decide to use TP at some point, we should do this instead (TP slices along the num heads dim)

--- Comment by vasqu at 2026-05-07T14:31:17Z ---
no eps?

--- Comment by vasqu at 2026-05-07T14:31:49Z ---
maybe layers instead?

--- Comment by vasqu at 2026-05-07T14:34:04Z ---
Could we simplify a bit with Ilyas PR?

--- Comment by vasqu at 2026-05-07T14:34:53Z ---
docstring please

--- Comment by vasqu at 2026-05-07T14:35:32Z ---
Could be moved into clip MLP like?

--- Comment by vasqu at 2026-05-07T14:35:52Z ---
merge config with defaults is missing

--- Comment by vasqu at 2026-05-07T14:36:35Z ---
Any reason we dont inherit from some other model for the boilerplate functions, e.g. placeholder mask?

--- Comment by vasqu at 2026-05-07T14:37:58Z ---
probably new with prefix instead?

--- Comment by vasqu at 2026-05-07T14:40:02Z ---
Yea this is very specific to your vision counterpart then; we should specify it somewhere on which config it resides

--- Comment by vasqu at 2026-05-07T14:40:26Z ---
debug

--- Comment by zucchini-nlp at 2026-05-07T19:32:33Z ---
yep, i will fix the doc page at last

--- Comment by zucchini-nlp at 2026-05-07T19:33:47Z ---
it would though I think it will make it impossible for users to load a remote `TikTokenKimiTokenizer`. Anyways they are adding a fast tokenizer soon so this should be entirely removed

Dirty hack for a moment

--- Comment by zucchini-nlp at 2026-05-07T19:35:07Z ---
oke, but tbh both files have identical dependencies on torchvision, so we would be fine in any case

--- Comment by zucchini-nlp at 2026-05-07T19:36:18Z ---
yep, i am using k2.6 locally but the acrh is same

--- Comment by zucchini-nlp at 2026-05-07T19:36:43Z ---
ah right right, just being extra careful

--- Comment by zucchini-nlp at 2026-05-07T19:39:33Z ---
we have projection hidden size and in some models the activation already, e.g. llava

Wouldn't do another subconfig, because a config comes with its own PreTrainedModel. Also not a fan of deleting the prefix, kinda gets lost with hidden size of text/vision configs due to naming. We coudl infer hidden size imo, but not the activations/eps value, unless we just hardcode them in modeling

Some models actually do hardcode it, because we aren't really strict on that part

--- Comment by zucchini-nlp at 2026-05-07T19:40:40Z ---
not sure I got it, this is quite same as we did with exaone recently. Keep the remote config as is and make it work with local code by changing model type

--- Comment by zucchini-nlp at 2026-05-07T19:42:10Z ---
the input 3? It is the image RGB channel which is same in all vision models, don't agree that it needs a comment since that is a common vision thingy :)

--- Comment by zucchini-nlp at 2026-05-07T19:43:35Z ---
yep, taking a look. At first glance i think that `flatten` won't work, I tried smth along that line but `flatten` was stacking two tensors while `torch.stack()` would actually interleave as we want

So prob just change the op from flatten to stack, should work imo

--- Comment by zucchini-nlp at 2026-05-07T19:44:05Z ---
good point, defi will need TP and anything that helps to run this model

--- Comment by zucchini-nlp at 2026-05-07T19:44:34Z ---
same as default from torch, but will add explicitly to make it clear

--- Comment by zucchini-nlp at 2026-05-07T19:44:55Z ---
i hope so, this pattern of packing inputs of dynamic shapes became a trend in vision transformers haha

--- Comment by zucchini-nlp at 2026-05-07T19:46:59Z ---
ahhh my bad, tbh I was thinking if we can get the correct config from `module.config` but not sure how doable. Conversion iterates over state dict, and not the model itself, so prob try to get attr by name or anythgin like that

Will do smth with it so it just clicks with any backbone attention

--- Comment by vasqu at 2026-05-07T19:52:17Z ---
Note it's a torch mul (outer dot product) not matmul so I think the shape is something along `..., 2, freq_dim` and flatten should work but yea if it doesn't work any of these way, I can hop on and manually check :D

--- Comment by vasqu at 2026-05-07T19:53:01Z ---
Oki, yea fair enough no worries

--- Comment by vasqu at 2026-05-07T19:53:55Z ---
I thought that it might not be resolved into a dict and we directly have a passed PreTrainedConfig; then we dont override the model type. But probably fine as is

--- Comment by vasqu at 2026-05-07T19:54:31Z ---
Aya, caught me red handed with my lack of knowledge :D

--- Comment by zucchini-nlp at 2026-05-07T20:04:35Z ---
ah i see what you mean, i'll make sure it works with both: dict and config class

--- Comment by zucchini-nlp at 2026-05-11T12:51:04Z ---
we precompute once in init and slice during the forward, since each video can be sampled at different frame rate

--- Comment by zucchini-nlp at 2026-05-11T12:53:03Z ---
yep, this is just a default rope for time-grid tbh so will copy the code to make it clearer. Prob we can make a new module indeed, so we will do learned 2D embed for H/W grid and rope for T grid, then combine

--- Comment by zucchini-nlp at 2026-05-11T13:05:41Z ---
can't be moved to the MLP module we already have (due to different input/ouptut shapes), and I don;t think it's worth creating a new ProjectionMLP for it (because the whole proj itself is only MLP + norm)

--- Comment by zucchini-nlp at 2026-05-11T13:07:12Z ---
not needed, that one merges only `use_cache` which a vision model doesnt use. Other kwargs are merged within `capture_kwargs`

(which is why I was against `merge config with defaults` as a new decorator overall 😆 )

--- Comment by zucchini-nlp at 2026-05-11T13:17:52Z ---
doesn't really fit because we also remove the suffix `encoder`, so keeping as one rename is easier to read imo

--- Comment by zucchini-nlp at 2026-05-11T13:25:32Z ---
after some thought, i dont think there is a straightforward way to pass current module's config. So i'll mention this in docstring, and maybe it will be refactored later

Since we iterate over a state dict when loading, mapping back from `full_param_name` to the module, and trying to get its config seems like a "not worth" complication for now

--- Comment by zucchini-nlp at 2026-05-11T17:01:29Z ---
copied from `processing_auto.pt` btw. The idea is to infer local class name if possible from config, even if we found a remote code

The remote code will still be loaded with higher prio if users pass `trust_remote_code`, so nothing should break

--- Comment by zucchini-nlp at 2026-05-12T16:10:12Z ---
this worked! Low lvl api would force us to add all de-quant types, so inferring compressor and calling decompress is the best solution imo


PULL REQUEST #45629: Allow more artifacts to be download in CI
State: closed  |  Merged: True
Author: ydshieh  |  Base: main
Labels:
Created: 2026-04-24T12:20:16Z

# What does this PR do?

We have more than 1000 artifacts in a daily CI run ...

Without this change, some artifacts won't be downloaded, then either the report statistic is wrong or the check new failing tests job fails.

We might change all v4 to v8 to be consistent - but I will wait until @paulinebm back.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T12:31:04Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45629). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ydshieh at 2026-04-24T12:23:04Z ---
This `ACTIONS_ARTIFACT_MAX_ARTIFACT_COUNT` only works with `v8`


PULL REQUEST #45628: [MistralCommonBackend] Soften validation mode and apply_chat_template arguments check
State: closed  |  Merged: True
Author: juliendenize  |  Base: main
Labels:
Created: 2026-04-24T11:30:52Z

# What does this PR do?

This PR soften validation and apply_chat_template arguments check. This is to avoid upon new Mistral model releases the need to patch this backend and force users to install main.

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @itazap

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T13:03:57Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45628). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45627: Processing Utils: honor pre-built sub-processor kwargs in from_pretrained
State: open  |  Merged: False
Author: javierdejesusda  |  Base: main
Labels:
Created: 2026-04-24T09:48:25Z

## What does this PR do?

Addresses @ArthurZucker's follow-up on #44987. When a caller passes a pre-built sub-processor to `AutoProcessor.from_pretrained` — e.g. `tokenizer=tok` or `bpe_tokenizer=tok` — the instance is now used directly instead of being silently forwarded into the sub-loader calls.

Exact attribute names always take precedence (`bpe_tokenizer=`). For processors with a single sub-processor of a given modality, the canonical modality name (`tokenizer=`) is also accepted as an alias — this matches the reproducer in the issue comment, where the remote `UniversalActionProcessor` takes `bpe_tokenizer` but the caller passes `tokenizer=`.

The OP's zero-kwarg reproducer (`AutoProcessor.from_pretrained("physical-intelligence/fast", trust_remote_code=True)`) is a separate hub repo layout issue and is not addressed here.

## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)?
- [x] Did you write any new necessary tests?

## Who can review?

cc @ArthurZucker @itazap

--- Comment by github-actions[bot] at 2026-04-24T09:49:49Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto

--- Comment by HuggingFaceDocBuilderDev at 2026-04-28T02:59:59Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45627). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by zucchini-nlp at 2026-04-24T10:13:42Z ---
not really sure about this. The error from GH issue is due to old remote code, and we don't yet support the Pi0-FAST natively in transformers. also cc @yonigozlan ig you might have seen similar issue when refactoring processor loading

We're planning native support though, and waiting for lerobot team to test and convert the configs correctly

--- Comment by javierdejesusda at 2026-04-24T10:27:16Z ---
Thanks for taking a look, @zucchini-nlp!

Quick scope note: this PR isn't targeting the OP's zero-kwarg traceback (that's the hub layout / old remote code path you mentioned, which I agree is out of scope here and will be obsoleted by native support). It's targeting @ArthurZucker's follow-up comment on the issue:

> `p = AutoProcessor.from_pretrained("physical-intelligence/fast", tokenizer=tokenizer, trust_remote_code=True, use_fast=False)`
>
> *"this does not work and it should!"*

The underlying behavior is general to `ProcessorMixin`: when a caller supplies a pre-built sub-processor via kwargs (whether `tokenizer=` or the exact attribute name like `bpe_tokenizer=`), the instance is silently dropped and the loader tries to reload from disk anyway. Any processor with a non-primary tokenizer attribute runs into this, so native Pi0-FAST support wouldn't fix it on its own, it'd just mean one fewer processor hitting it.

That said, happy to defer fully. If you and @yonigozlan / @ArthurZucker feel this should wait (or be folded into the native support work, or handled differently), I'm glad to close or rescope, just let me know.

--- Comment by zucchini-nlp at 2026-04-27T13:18:05Z ---
yeah, totally get it. Personally, I think we can deliberately not support it as remote code and not-v5 compatible unless Arthur/Yoni have a different opinion


[Model] Add PP-FormulaNet Model Support

PULL REQUEST #45626: [Model] Add PP-FormulaNet Model Support
State: closed  |  Merged: True
Author: zhang-prog  |  Base: main
Labels: New model
Created: 2026-04-24T09:25:27Z

(no description)

--- Comment by zhang-prog at 2026-04-27T13:04:24Z ---
@vasqu I’ve restructured the PPFormulaNet into a VLM. Some unit tests are still failing and I’m fixing them, but that shouldn’t block you from reviewing the latest model structure code. PTAL.

--- Comment by vasqu at 2026-04-29T12:32:18Z ---
run-slow: pp_formulanet

--- Comment by github-actions[bot] at 2026-04-29T12:33:57Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25109002551)

This comment contains `run-slow`, running the specified jobs:

models: ["models/pp_formulanet"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-29T12:41:11Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25109002551)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [af86d363](https://github.com/huggingface/transformers/commit/af86d363fd43d0804e2af0c5bd890ab4a30a94ab) | workflow commit (merge commit) |
| PR | [74240ac5](https://github.com/zhang-prog/transformers/commit/74240ac5bcf4d148a3c740ec3a0fff578f2b7c20) | branch commit (from PR) |
| main | [a374d990](https://github.com/huggingface/transformers/commit/a374d990d422357a854ff7f4f5f1cf9f88308ade) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by vasqu at 2026-04-30T12:02:10Z ---
run-slow: pp_formulanet

--- Comment by github-actions[bot] at 2026-04-30T12:03:54Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25164246781)

This comment contains `run-slow`, running the specified jobs:

models: ["models/pp_formulanet"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-30T12:21:18Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25164246781)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [005dfbce](https://github.com/huggingface/transformers/commit/005dfbcef25ff9ca6e5905c83d9d4b6450af1d70) | workflow commit (merge commit) |
| PR | [e533c550](https://github.com/zhang-prog/transformers/commit/e533c5501acb9340a7cbd30230e5d27dd02edc7d) | branch commit (from PR) |
| main | [53b92b94](https://github.com/huggingface/transformers/commit/53b92b94ed7e48ff5db11b88a271cb8941c2df9e) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by HuggingFaceDocBuilderDev at 2026-04-30T12:42:20Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45626). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-30T12:42:44Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, pp_formulanet

--- Comment by github-actions[bot] at 2026-04-30T12:55:40Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45626&sha=5b3b5f

--- Comment by vasqu at 2026-04-24T13:08:31Z ---
nit: we started to use httpx instead - already removed in our code base for the most part

--- Comment by vasqu at 2026-04-24T13:11:31Z ---
```suggestion
result = processor.post_process_text_recognition(outputs.last_hidden_state)
```
I think it would make sense to have these consistent across all models that do text recognition

--- Comment by vasqu at 2026-04-24T13:13:36Z ---
```suggestion
model_path = "PaddlePaddle/PP-FormulaNet_plus-L_safetensors"  # or "PaddlePaddle/PP-FormulaNet-L_safetensors"
```
Not sure but in the docs 2 have been mentioned

--- Comment by vasqu at 2026-04-24T13:17:14Z ---
I think we need to override the decorator so in modular something along
```
@auto_docstring(checkpoint="PaddlePaddle/PPFormulaNet_plus-L_safetensors")
@strict
class PPFormulaNetVisionConfig(PreTrainedConfig):
    pass
```

--- Comment by vasqu at 2026-04-24T13:24:36Z ---
Imo, these would be suited to another subconfig --> text config

We can mimic a VLM more closely here  --> I've made a comment later on the (re)structure

--- Comment by vasqu at 2026-04-24T13:26:31Z ---
We should add that to the tokenization_auto mapping then re nougat tokenizer

--- Comment by vasqu at 2026-04-24T13:26:43Z ---
```suggestion
```
shouldn't be needed

--- Comment by vasqu at 2026-04-24T14:08:41Z ---
Ok, this is a bit more complicated - we should make this more like a VLM. I try to give the overall dummy structure
```python
class PPFormulaNetMultiModalProjector(nn.Module):
    def __init__(config):
        self.conv1 = ...  # post_conv1
        self.conv2 = ...  # post_conv2
        self.linear_1 = ...  # mm_projector_vary
        self.linear_2 = ...  # enc_to_dec_proj

class PPFormulaNetVisionModel(SLANeXtVisionEncoder):
    pass

class PPFormulaNetTextModel(MBartDecoder):
   pass

# Similar to other VLMs: Use a vision tower -> projector -> text backbone
class PPFormulaNetModel(GotOcr2Model):  # GotOcr2Model is just one possibility depending on what comes closer
    def __init__(self, config):
        super().__init__(config)

        self.language_model = PPFormulaNetTextModel.from_config(config.text_config)  # Using MbartDecoder inheritance  --> definitely make sure that config.is_encoder_decoder = True if that's how it is meant to be used and/or we overwrite `_prepare_cache_for_generation` there as well
        self.vision_tower = PPFormulaNetVisionModel.from_config(config.vision_config)
        self.multi_modal_projector = PPFormulaNetMultiModalProjector(config)

        # Initialize weights and apply final processing
        self.post_init()
```
With this the structure is much clearer

--- Comment by vasqu at 2026-04-24T14:14:00Z ---
Not needed then anymore, we follow the `ForConditionalGeneration` instead then or more like `GotOcr2ForConditionalGeneration` where we have a special VLM output

--- Comment by vasqu at 2026-04-24T14:17:38Z ---
Same here we can follow `GotOcr2ForConditionalGeneration` more closely then

We probably need to override `prepare_inputs_for_generation` (VLM style) and `_prepare_cache_for_generation` (force encoder-decoder cache)

--- Comment by vasqu at 2026-04-24T14:19:28Z ---
Yea, we don't do this then but support a more plain VLM that essentially ignores text input (which we can incorporate into the processor via the tokenizer imo)

Since we have encoder states, it definitely needs to be in the signature - t5gemma2 or bart in general could be a good reference. We essentially want an encoder-decoder model here but instead of text input we use pixel values

--- Comment by vasqu at 2026-04-24T14:20:05Z ---
This will likely change then to the normal VLM mapping instead

--- Comment by vasqu at 2026-04-24T14:20:36Z ---
```suggestion
            pattern = r"(\\[a-zA-Z]+)\s(?=\w)|\\[a-zA-Z]+\s(?=})"
```

--- Comment by vasqu at 2026-04-24T14:21:46Z ---
We should compile the regex outside the loops, probably similar above

--- Comment by vasqu at 2026-04-24T14:22:11Z ---
```suggestion
            logger.warning_once(
```

--- Comment by vasqu at 2026-04-24T14:22:40Z ---
Not a fan of an extra dependency tbh but ig it is too complicated/long to adopt here

--- Comment by vasqu at 2026-04-24T14:23:23Z ---
We can then use our VLM tester instead https://github.com/huggingface/transformers/blob/622b8e95c26b0b1bcb2595af4cd006e4634dd367/tests/vlm_tester.py#L36

--- Comment by zucchini-nlp at 2026-04-24T16:54:32Z ---
You might want to copy the same structure as `Florence2Model` and its generative class. Note that we needed to override one generation utility, to prepare encoder-decoder inputs correctly for Bart

https://github.com/huggingface/transformers/blob/c472755e79aac54d675845bff5e5c821c21260af/src/transformers/models/florence2/modular_florence2.py#L1427

--- Comment by vasqu at 2026-04-27T15:42:36Z ---
Rebump

--- Comment by vasqu at 2026-04-27T15:46:51Z ---
Imo these belong in the base config, it does not really belong to the text config as this is only meant for the mbart decode while this is all for the multimodal projector

--- Comment by vasqu at 2026-04-27T15:49:07Z ---
```suggestion
class PPFormulaNetTextConfig(MBartConfig):
```
We should inherit from Mbart directly, that way we don't have to think too much what is actually needed

--- Comment by vasqu at 2026-04-27T15:50:35Z ---
You might be searching for `max_position_embeddings` instead or at least it should not be part of the model but the tokenizer. Probably from the old model pattern you had where you manually called generate

--- Comment by vasqu at 2026-04-27T15:51:43Z ---
```suggestion
)
```
tbh, would mention it in the model docs (`model_doc/pp_formulanet.md`) but not here because the default values are valid for that checkpoint - we only search for one example here

--- Comment by vasqu at 2026-04-27T15:54:25Z ---
Yea that rename was bad on my side 😓 I expected to be more closely related to slanext before we noticed the VLM pattern, I think we can keep it simple as originally without the suffix

--- Comment by vasqu at 2026-04-27T15:54:33Z ---
Rebump

--- Comment by vasqu at 2026-04-27T15:56:41Z ---
```suggestion
```
missed from previous code

--- Comment by vasqu at 2026-04-27T15:58:05Z ---
```suggestion
```
shouldn't be needed iirc modular auto inherits them. Maybe if you need to overwrite `_can_record_outputs` it might make sense

--- Comment by vasqu at 2026-04-27T15:59:19Z ---
```suggestion
class PPFormulaNetTextModel(MBartDecoder):
    pass
```
We really should the base decoder model and not the wrapper (which is equivalent to ForCausalLM)

--- Comment by vasqu at 2026-04-27T16:11:04Z ---
Imo, we should check that either pixel values or encoder outputs exists (similar to input ids / embeds for text models) and raise an error if not

Then we can have a top-level check and less if else afterwards, so e.g. roughly
```python
if (encoder_outputs is None) ^ (pixel_values is not None):
    raise ValueError("You must specify exactly one of encoder_outputs or pixel_values")

if encoder_outputs is None:
    encoder_outputs = self.get_image_features(pixel_values, **kwargs).to(self.language_model.device, self.language_model.dtype)
elif encoder_outputs.pooler_output is None:
    encoder_outputs.pooler_output = self.multi_modal_projector(encoder_outputs.last_hidden_state)

image_features = encoder_outputs.pooler_output
```

--- Comment by vasqu at 2026-04-27T16:12:59Z ---
```suggestion
```
I think you don't need these 2, no? Only the language model is different because of auto model in florence2 (the mm projector will become the same with the config changes I mentioned earlier)

--- Comment by vasqu at 2026-04-27T16:15:04Z ---
```suggestion
        decoder_outputs = self.language_model(
            input_ids=decoder_input_ids,
            attention_mask=decoder_attention_mask,
            encoder_hidden_states=image_features,
            encoder_attention_mask=attention_mask,
            past_key_values=past_key_values,
            inputs_embeds=decoder_inputs_embeds,
            use_cache=use_cache,
            **kwargs,
        )
```
Like mentioned before would like to move away from the ForCausalLM model and use the decoder directly

--- Comment by vasqu at 2026-04-27T16:17:11Z ---
```suggestion
    def _prepare_encoder_decoder_kwargs_for_generation(self, *args, **kwargs):
        raise AttributeError()
```
I think you just don't want to inherit? That tells modular not to

--- Comment by vasqu at 2026-04-27T16:17:21Z ---
```suggestion
```

--- Comment by vasqu at 2026-04-27T16:21:18Z ---
Wouldn't this fit more to `image_last_hidden_state`? You want the last (pooled) feature, not the set of hidden states across all of this

Imo, we can even leave this completely out imo as the encoder is everything image-related. The output class should be new and explain that the encoder == vision encoder hence different expected shapes and all

--- Comment by vasqu at 2026-04-27T16:23:05Z ---
```suggestion
        pixel_values: torch.FloatTensor | None = None,
        attention_mask: torch.Tensor | None = None,  # TODO check if this is really used, likely to be removed as well
        decoder_input_ids: torch.LongTensor | None = None,
        decoder_attention_mask: torch.LongTensor | None = None,
        decoder_inputs_embeds: torch.FloatTensor | None = None,
        encoder_outputs: list[torch.FloatTensor] | None = None,
        past_key_values: Cache | None = None,
        use_cache: bool | None = None,
        **kwargs,
```
Noticing that we don't need those - we have pure images, no associated text in the encoder so we can leave/remove them

--- Comment by vasqu at 2026-04-27T16:32:17Z ---
main input name should be pixel values not sure if that is already the case within the pretrained model :D

--- Comment by vasqu at 2026-04-27T16:33:37Z ---
Atp because we have a special encoder-decoder which does not fit the standard VLM style, not sure if it really fits - maybe a classic encoder-decoder approach might be better

--- Comment by zucchini-nlp at 2026-04-27T16:37:39Z ---
this is weird imo, and doesn't really align with VLMs. `Encoder_output` if provided has contain `pooler_output`, i.e. the projected image features

--- Comment by zucchini-nlp at 2026-04-27T16:38:05Z ---
btw, I am currently drafting smth for these types of models, so keeping it consistent with other VLMs is a must

--- Comment by zucchini-nlp at 2026-04-27T16:38:16Z ---
`@auto_docstring` pls

--- Comment by vasqu at 2026-04-27T16:39:13Z ---
Ah gotcha, fair enough - I think this is still barebones so maybe some stuff needs to moved to some preparation functions

--- Comment by zucchini-nlp at 2026-04-27T16:39:46Z ---
same deletion here, `get_encoder` accepts a modality arg and is defined in parent

--- Comment by vasqu at 2026-04-27T16:51:59Z ---
For the tests, `dia` might be a good reference as you need to adjust the shape checks for most of them

--- Comment by zhang-prog at 2026-04-28T09:18:59Z ---
The parameters still need to remain in the argument list; otherwise, when calling self.language_model(..., **kwargs), it will raise errors like:
```text
got multiple values for keyword argument 'attention_mask'
got multiple values for keyword argument 'input_ids'
```

--- Comment by zhang-prog at 2026-04-28T09:19:02Z ---
removed

--- Comment by zhang-prog at 2026-04-28T09:23:06Z ---
It is used in the `_init_weights` method of `PPFormulaNetPreTrainedModel` to pass the `test_can_init_all_missing_weights` test.

--- Comment by zhang-prog at 2026-04-28T09:23:16Z ---
same as `PPFormulaNetVisionEncoder`

--- Comment by vasqu at 2026-04-28T09:35:00Z ---
Ah makes sense for the attention then but in this case it's a duplicate, we want `PPFormulaNetVisionModel` instead (this is a duplicate with the same inheritance)

--- Comment by zhang-prog at 2026-04-28T09:54:07Z ---
oh, i see

--- Comment by vasqu at 2026-04-28T14:59:51Z ---
auto model please

--- Comment by vasqu at 2026-04-28T15:09:56Z ---
nit: lets avoid short letter and just use text or similar

--- Comment by vasqu at 2026-04-28T15:10:50Z ---
On second thought, would it make sense to be more extreme and have those regex at init time once? Same below

--- Comment by vasqu at 2026-04-28T15:15:03Z ---
Can we mention with a small comment that we only keep this in the signature for generate compatibility?

--- Comment by vasqu at 2026-04-28T18:13:01Z ---
Imo, we shouldn't need this. Maybe we should either
1. Move the projector into the encoder as well
2. Adjust the generation pipeline where we prepare the encoder outputs to instead call get image features

--- Comment by vasqu at 2026-04-28T18:14:34Z ---
Maybe we could still inherit from florence here and raise an attribute error for stuff we dont need, wdyt?

--- Comment by vasqu at 2026-04-28T18:15:49Z ---
Imo we need processor tests at least, ideally img processor ones as well but less of a prio since its only different default values there

--- Comment by vasqu at 2026-04-28T18:16:17Z ---
```suggestion
```

--- Comment by vasqu at 2026-04-28T18:16:56Z ---
Can we add that this is not a normal VLM - still very nice that it worked out. Honestly was not expecting it to be this smooth kudos

--- Comment by vasqu at 2026-04-28T18:17:26Z ---
Could we try? :D

--- Comment by vasqu at 2026-04-28T18:18:44Z ---
Would be nice to fix but also not that big of a deal

--- Comment by vasqu at 2026-04-28T18:19:18Z ---
Hmm, should be fixed imo if possible - maybe overriding the test or something else

--- Comment by zhang-prog at 2026-04-29T06:44:58Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:44:59Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:45:03Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:45:04Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:45:06Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:45:08Z ---
ofc, you are right

--- Comment by zhang-prog at 2026-04-29T06:45:09Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:45:12Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:45:15Z ---
Done

--- Comment by zhang-prog at 2026-04-29T06:45:17Z ---
Done, passed

--- Comment by zhang-prog at 2026-04-29T06:45:21Z ---
Done, beam search tests are all passed

--- Comment by zhang-prog at 2026-04-29T06:45:24Z ---
I did try that, but it failed :(
I think it may be related to the model’s special architecture, so for now I kept it skipped.

<img width="788" height="220" alt="image" src="https://github.com/user-attachments/assets/4abef8f4-c55d-40e4-93ab-fac809a1a1d3" />

--- Comment by vasqu at 2026-04-29T12:23:53Z ---
This shouldn't be necessary and I'd rather adjust the values in the config from the get go

--- Comment by vasqu at 2026-04-29T12:28:19Z ---
Since we now follow the full encoder-decoder structure, it would be nicer to stay closer to them e.g. https://github.com/huggingface/transformers/blob/727741f533b1f47120db47ef5934296d07214d7e/src/transformers/models/bart/modeling_bart.py#L759-L771

We can still keep `get image features`, it just acts more as a nice utility then, not as core forward part

--- Comment by vasqu at 2026-04-29T12:28:38Z ---
Is it actually needed?

--- Comment by vasqu at 2026-04-29T12:29:25Z ---
Rebump, maybe missed to commit it :D

--- Comment by vasqu at 2026-04-29T12:30:55Z ---
Looks like the rtol/atol is maybe too low but yea no worries we can keep it skipped, not a high prio imo

--- Comment by zhang-prog at 2026-04-30T03:20:35Z ---
Done

--- Comment by zhang-prog at 2026-04-30T03:20:36Z ---
Removed


PULL REQUEST #45625: Add `supports_gradient_checkpointing` to `NemotronHPreTrainedModel`
State: closed  |  Merged: True
Author: sergiopaniego  |  Base: main
Labels:
Created: 2026-04-24T08:55:34Z

# What does this PR do?

Enables gradient checkpointing for `NemotronH` by setting `supports_gradient_checkpointing = True` on `NemotronHPreTrainedModel`. The idea comes from its usage in TRL.

`NemotronHBlock` already inherits from `GradientCheckpointingLayer`, so the infrastructure at the block level is in place. The only missing piece was the class-level flag, which currently defaults to `False` (inherited from `PreTrainedModel`). As a result, any call to `model.gradient_checkpointing_enable()` (including the one issued by `Trainer` when `gradient_checkpointing=True`) raises:

```
ValueError: NemotronHForCausalLM does not support gradient checkpointing.
```

This is a simple omission, not a limitation of the architecture. All sibling hybrid Mamba/attention models in the library already enable it:

| Model | `GradientCheckpointingLayer` | `supports_gradient_checkpointing` |
| --- | --- | --- |
| Bamba | ✅ | ✅ |
| GraniteMoeHybrid | ✅ | ✅ |
| Zamba2 | ✅ | ✅ |
| **NemotronH** | ✅ | ❌ (before this PR) |

GraniteMoeHybrid is the closest analogue (MoE + hybrid Mamba/attention, same layout as NemotronH).

The change is made in `modular_nemotron_h.py` and propagated to `modeling_nemotron_h.py` via `utils/modular_model_converter.py`.

Fixes the failure seen downstream in https://github.com/huggingface/trl/pull/5278, where NemotronH tests required a `gradient_checkpointing=False` workaround.

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? (n/a — class attribute only)
- [x] Did you write any new necessary tests? (n/a — covered by existing hybrid-model GC tests)

## Who can review?

@ArthurZucker @Cyrilvallez

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T09:06:29Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45625). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by github-actions[bot] at 2026-04-24T14:58:46Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: nemotron_h


Skip failing offloading tests

PULL REQUEST #45624: Skip failing offloading tests
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-04-24T08:28:53Z

# What does this PR do?

For future ref, it fails after https://github.com/huggingface/transformers/pull/45489 which does not change anything, only module order in the `__init__`...
cc @ydshieh

--- Comment by github-actions[bot] at 2026-04-24T08:30:01Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T08:40:40Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45624). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Glm5 change

PULL REQUEST #45623: Glm5 change
State: closed  |  Merged: False
Author: tarekziade  |  Base: main
Labels:
Created: 2026-04-24T07:58:54Z

# What does this PR do?

Testing the review agent

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T08:10:17Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45623). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Fix peft constructors

PULL REQUEST #45622: Fix peft constructors
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-04-24T07:53:35Z

# What does this PR do?

As per the title. Fixes https://github.com/huggingface/transformers/pull/45448#issuecomment-4281261317
cc @BenjaminBossan

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T08:03:28Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45622). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by BenjaminBossan at 2026-04-24T11:06:26Z ---
Thanks for working on a fix, this resolves the failing transformers test. In PEFT, the corresponding tests are still failing, but I will port the changes from here over to PEFT and that should do the trick.

Is there any way we can make all of this more robust? It's not the first time things break. I assume this is because we rely on Transformers-internal implementation details. Is there a public, more stable API that we can use instead?

--- Comment by Cyrilvallez at 2026-04-27T07:22:42Z ---
Those are indeed relying on internals, which is why it wasn't detected! If you can add a simple fast test performing one of those conversions, it would catch any regresions though!

--- Comment by BenjaminBossan at 2026-04-27T12:09:06Z ---
@Cyrilvallez I could convert `test_mixtral_lora_conversion` to a normal test instead of a slow test. It uses a tiny model (7.6 MB). Locally, with the model cached, it finishes in 1.5 sec. Would that be all right?

--- Comment by Cyrilvallez at 2026-04-28T04:38:44Z ---
Yes 100%


Better Grouped GEMM + EP

PULL REQUEST #45621: Better Grouped GEMM + EP
State: closed  |  Merged: True
Author: IlyasMoutawwakil  |  Base: main
Labels:
Created: 2026-04-24T07:48:02Z

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

The idea is that we shouldn't be clamping experts ids at all, clamping them makes their tokens get prjected as if they were routed to the last expert, instead, we should let the sentinels be:

- moved to the queue by the sorting (num_experts/sentinel is the biggest value)
- dropped by the histogram (because max=num_experts-1)
- and then the offsets created from this histogram will contain the right mapping : experts[i] takes tokens from (offsets[i-1] or 0 if i==0) to (offsets[i])

I micro-benchmarked the kernel with sentinel tokens and it is faster (it is skipping their compute as expected) :
- offsets[-1] = 16384 (100%)   22.7 ms/iter   1.00x
- offsets[-1] =  8192  (50%)   13.0 ms/iter   0.57x
- offsets[-1] =  2048 (12.5%)   4.5 ms/iter   0.20x

and still data independent / compatible with torch.compile / cuda graphs

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->

--- Comment by IlyasMoutawwakil at 2026-04-24T08:26:58Z ---
will ad bf16 deepgemm to testing as well

--- Comment by IlyasMoutawwakil at 2026-04-24T09:00:23Z ---
> Having TP code in moe is fine IMO let's not separate into TP since anything could be using sentinels actually!

yeah at first i created it because it was gonna be used everywhere but now only a clamp is needed in batched paths.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T09:39:03Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45621). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by IlyasMoutawwakil at 2026-04-24T11:19:59Z ---
reverted the bf16 deepgemm and isolation

--- Comment by IlyasMoutawwakil at 2026-04-24T11:25:33Z ---
Great that we have a couple models with TensorParallelTesterMixin they pass now !
the output of grouped mm can contain uninitialized values if not all tokens are routed to experts, because it uses torch.empty internally to initialize it.

--- Comment by AmineDiro at 2026-04-30T08:55:41Z ---
I benchmarked this PR's `grouped_mm_experts` end-to-end in my SFTrainer benchmark `Qwen/Qwen3-30B-A3B` on 2 H100 nodes (FSDP2 + EP=8 + sdpa, 16k ctx, 50 SFT steps). Tthe run trains with broken gradients:

| step | loss   | grad_norm | entropy | mean_token_acc | MFU |
|------|--------|-----------|---------|----------------|------------|
| 5    | 4.514  | 0         | **nan** | 0.157          | —          |
| 10   | **0**  | 0         | **nan** | 9.8e-05        | 27.44 %    |
| 15   | **0**  | 0         | **nan** | 2.1e-04        | 27.85 %    |
| 20   | **0**  | 0         | **nan** | 5.5e-05        | 27.41 %    |

Repro:
```python
import torch

torch.manual_seed(0)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
DTYPE = torch.bfloat16

T, TOP_K, H = 8, 2, 4
S = T * TOP_K   # 16 routing slots

def run_mode(label, *, sentinel_makes_nan, prezero_before_mul):
    sentinel_mask = torch.zeros(S, dtype=torch.bool, device=DEVICE)
    sentinel_mask[S // 2:] = True

    sample_weights = torch.randn(S, device=DEVICE, dtype=torch.float32)
    sample_weights[sentinel_mask] = 0.0
    sample_weights = sample_weights.to(DTYPE).clone().requires_grad_(True)

    proj_out = torch.randn(S, H, device=DEVICE, dtype=DTYPE)
    if sentinel_makes_nan:
        proj_out[sentinel_mask] = float("nan")     # simulate uninitialized rows
    proj_out = proj_out.clone().requires_grad_(True)

    proj_out_used = (
        proj_out.masked_fill(sentinel_mask.unsqueeze(-1), 0.0)
        if prezero_before_mul else proj_out
    )

    weighted = proj_out_used * sample_weights.unsqueeze(-1)
    weighted_zero = weighted.masked_fill(sentinel_mask.unsqueeze(-1), 0.0)
    out = weighted_zero.view(T, TOP_K, H).sum(dim=1)
    out.backward(torch.randn_like(out))

    sw_nan = torch.isnan(sample_weights.grad).sum().item()
    print(f"[{label}]")
    print(f"  forward out finite     : {torch.isfinite(out).all().item()}")
    print(f"  sample_weights.grad nan: {sw_nan} of {S}")
    print(f"  sw.grad sentinel slice : {sample_weights.grad[sentinel_mask].float().tolist()}")

run_mode("A: proj_out fully init (baseline)",
         sentinel_makes_nan=False, prezero_before_mul=False)
run_mode("B: NaN at sentinels, NO pre-zero  (this PR's pattern)",
         sentinel_makes_nan=True,  prezero_before_mul=False)
run_mode("C: NaN at sentinels, pre-zero before mul  (proposed fix)",
         sentinel_makes_nan=True,  prezero_before_mul=True)
```

Output:

```
[A: proj_out fully init (baseline)]
  forward out finite     : True
  sample_weights.grad nan: 0 of 16
  sw.grad sentinel slice : [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

[B: NaN at sentinels, NO pre-zero  (this PR's pattern)]
  forward out finite     : True             ← masked_fill makes forward finite
  sample_weights.grad nan: 8 of 16          ← but backward leaks NaN
  sw.grad sentinel slice : [nan, nan, nan, nan, nan, nan, nan, nan]

[C: NaN at sentinels, pre-zero before mul  (proposed fix)]
  forward out finite     : True
  sample_weights.grad nan: 0 of 16
  sw.grad sentinel slice : [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
```

Suggested fix:
```diff
     # Apply routing weights
+    # Zero sentinel rows of proj_out *before* the multiply so backward's
+    # `d_sample_weights_g = (d_weighted * proj_out).sum(-1)` doesn't read
+    # uninitialized memory at sentinel positions.
+    proj_out.masked_fill_((expert_ids_g >= self.num_experts).unsqueeze(-1), 0.0)
     weighted_out = proj_out * sample_weights_g.unsqueeze(-1)
-    weighted_out.masked_fill_((expert_ids_g >= self.num_experts).unsqueeze(-1), 0.0)
```

--- Comment by AmineDiro at 2026-04-30T09:57:33Z ---
Update on my earlier suggestion. I reran with`proj_out.masked_fill_(sentinel_mask, 0)` end-to-end on  same setup
**It's not sufficient*. Training still NaN'd at step 2 (`loss=0`, `entropy=NaN`).

After more digging, there are actually **three separate sentinel grad leaks**

```python
selected_hidden_states_g = hidden_states[perm // num_top_k]   # (S, H)
proj_out = _grouped_linear(selected_hidden_states_g, gate_up_proj, offsets)   # ←── leak 2
proj_out = proj_out.masked_fill(sentinel_mask, 0.0)
proj_out = self._apply_gate(proj_out)
proj_out = _grouped_linear(proj_out, down_proj, offsets)                      # ←── leak 3
proj_out.masked_fill_(sentinel_mask, 0.0)
weighted_out = proj_out * sample_weights_g.unsqueeze(-1)                      # ←── leak 1
```

**Leak 1** is what my first comment, grad Nan in`d(sample_weights)`. Closed by zeroing `proj_out` before the multiply.

**Leaks 2 & 3** are in the bwd path of `_grouped_mm`. The kernel writes only rows `[0, offsets[-1])` of `d_input` and leaves the sentinel tail uninitialized. Under production allocator pressure those rows are NaN.

Walking through what happens for the up-projection:
```python
# 1. Build the per-(token, slot) input tensor for the kernel by gathering rows from hidden_states:
selected_hidden_states_g = hidden_states[perm // num_top_k]   # shape (S, H)

# 2. Run the kernel. It only does work for rows [0, offsets[-1]) sentinels are excluded.
proj_out = _grouped_mm(selected_hidden_states_g, gate_up_proj, offsets)
```

In backward, autograd needs the gradient w.r.t. each input. The kernel produces `d_selected_hidden_states_g`. **The kernel writes only the valid rows of that gradient, sentinel rows are left as whatever uninitialized memory the buffer happened to be allocated with.** In a fresh process that's often zero; in production with a hot allocator, it's probably NaN. Then autograd has to undo step 1, the gather. The backward of `hidden_states[perm // num_top_k]` is the inverse: a *scatter-add* into `d_hidden_states`:  for every row of `d_selected_hidden_states_g`, it adds that row to `d_hidden_states` at token index `perm[r] // num_top_k`.

**Sentinel rows have NaN, so NaN gets added to `d_hidden_states` at every token a sentinel row was paired with.**

Under the ~94 % all-sentinel-token pattern EP=8 produces, almost every token has at least one sentinel slot, so almost every row of `d_hidden_states` ends up with NaN added to it. From there NaN spreads through the previous transformer block 😭 .

Suggested fix is to maskout at each stage to zerout NaN values for backward

--- Comment by ArthurZucker at 2026-04-24T08:25:54Z ---
is inplace OK for training?

--- Comment by ArthurZucker at 2026-04-24T08:27:12Z ---
ish not a fan of stateless, we can return them IMO

--- Comment by IlyasMoutawwakil at 2026-04-24T08:28:59Z ---
will test !

--- Comment by IlyasMoutawwakil at 2026-04-24T08:29:29Z ---
yes works !

--- Comment by IlyasMoutawwakil at 2026-04-24T11:18:17Z ---
works

--- Comment by IlyasMoutawwakil at 2026-04-26T10:45:07Z ---
for testing

--- Comment by AmineDiro at 2026-04-26T18:30:22Z ---

Quick sanity check, does this redirect mean that the currently-published kernels-community/sonic-moe does not yet have the metadata/sentinel handling?

Asking because I hit `cudaErrorIllegalAddress` reliably running `sonicmoe` with EP=8 (Qwen3-30B-A3B, 2 nodes × 8 H100, FSDP2 dp=2 ep=8) with the current hub kernel:

```bash
File "...sonic-moe/build/torch-cuda/quack/autotuner.py", line 84, in _gpu_warmup
    a = torch.randn(4096, 4096, device="cuda", dtype=torch.bfloat16)
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
```

Thats some sticky CUDA errors that ran asynchronously and faulted before the error propagated back.

with EP=8, `RouterParallel` produces sentinel expert_ids ≥ num_local_experts (16, since 128/8). The v1 sonic-moe kernel internally does `gate_up_proj[expert_ids[i]]` which is OOB ??

when I  add `expert_ids.clamp(0, num_experts-1)` and `masked_fill_(invalid_mask, 0.0)` in the wrapper before the kernel call everything works.
Removing the clamp brings the crash back 🥲 . So the v1 kernel really does need its inputs in-bounds, and the new build in your fork is what actually fixes it ??  Just want to confirm the plan is to republish the fixed build to kernels-community/sonic-moe and revert this redirect once that's done ?

--- Comment by ArthurZucker at 2026-04-28T09:41:57Z ---
nice if we write to all, which we do

--- Comment by ArthurZucker at 2026-04-28T09:42:23Z ---
to remove

--- Comment by ArthurZucker at 2026-04-28T09:43:50Z ---
loads of rep from moe vs fp8 cool if we re-use stuff, fine otherwise haha

--- Comment by IlyasMoutawwakil at 2026-04-29T07:41:38Z ---
yes !

--- Comment by IlyasMoutawwakil at 2026-04-29T07:42:12Z ---
yes and revert to the kernels-community one ?

--- Comment by ArthurZucker at 2026-05-04T03:41:47Z ---
yep!

--- Comment by IlyasMoutawwakil at 2026-05-04T09:13:54Z ---
done !


PULL REQUEST #45620: Fix TypeError in video_processor_class_from_name when torchvision is not installed
State: closed  |  Merged: False
Author: pratapyash  |  Base: main
Labels:
Created: 2026-04-24T07:14:52Z

## What does this PR do?

Adds a `None` guard in `video_processor_class_from_name` to prevent a `TypeError` when `torchvision` is not installed.

### The problem

When a user installs `transformers` without `torchvision` and tries to load a multimodal processor (e.g., `Qwen3OmniMoeProcessor.from_pretrained()`), they hit an opaque crash:

```
TypeError: argument of type 'NoneType' is not iterable
```

This happens because `VIDEO_PROCESSOR_MAPPING_NAMES` entries are overwritten to `None` when `torchvision` is unavailable (lines 69-76 of `video_processing_auto.py`). The lookup function `video_processor_class_from_name` then crashes at `class_name in extractors` because `in` cannot operate on `None`.

The lazy import system's `DummyObject` guard catches this in some code paths, but direct or internal calls to `video_processor_class_from_name` bypass that guard and produce the unhelpful `TypeError` instead of gracefully returning `None` (which would let the caller surface a clear "torchvision is required" message).

This differs from `image_processing_auto.py`, where entries are tuples `(slow_class, fast_class)` — nulling one element still leaves a valid tuple. Video processing entries are bare strings, so nulling produces `None` directly.

### The fix

```python
# Before
if class_name in extractors:

# After
if extractors is not None and class_name in extractors:
```


Remove unnecessary generate warnings

PULL REQUEST #45619: Remove unnecessary generate warnings
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-04-24T06:36:50Z

# What does this PR do?

Supersedes https://github.com/huggingface/transformers/pull/45559 as I messed up the branch 😅

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T07:00:06Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45619). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ArthurZucker at 2026-04-24T07:14:01Z ---
let's put this out of scope please

--- Comment by ArthurZucker at 2026-04-24T07:14:34Z ---
out of scope please

--- Comment by zucchini-nlp at 2026-04-24T09:34:20Z ---
thanks, this makes more sense!


PULL REQUEST #45618: Add MTP speculative decoding via MTPCandidateGenerator
State: open  |  Merged: False
Author: ArthurZucker  |  Base: main
Labels:
Created: 2026-04-24T06:24:21Z

Adds `use_mtp=True` to `generate` for DeepSeek-V3 / GLM-4 MoE. MTP modules live in a new `generation.candidate_generators.MTPCandidateGenerator` (loaded via its `from_pretrained`) — the base model stays clean.

--- Comment by github-actions[bot] at 2026-04-24T06:25:34Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_v3, glm4_moe, glm4v_moe, solar_open, youtu

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T06:35:46Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45618). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by curnane-lab at 2026-05-20T09:10:32Z ---
Hi @ArthurZucker, really excited about this PR! MTP speculative decoding is a huge win for inference throughput.

I noticed the branch currently has merge conflicts — would it help if I (or someone from the community) rebased it against latest main? Also, for awareness, I'm working on MTP training support for Qwen3.5 in [#45638](https://github.com/huggingface/transformers/pull/45638), which could complement this nicely if we align on the config naming (num_nextn_predict_layers vs mtp_num_hidden_layers).

No pressure, just offering help if it speeds things up! 🙏


PULL REQUEST #45617: Add Multi-Token Prediction (MTP) inference support
State: closed  |  Merged: False
Author: ArthurZucker  |  Base: main
Labels:
Created: 2026-04-24T05:58:59Z

## Summary

- Wires MTP speculative decoding into `generate()` for DeepSeek-V3 and GLM-4 MoE, loading the MTP modules that ship in the original checkpoints (previously ignored via `_keys_to_ignore_on_load_unexpected`).
- New `GenerationConfig.use_mtp=True` opt-in (default `False`; no behavior change when off). Routes to a new `GenerationMode.MTP_DECODING` / `_mtp_decoding` that uses the model's own MTP heads as the draft, verifies in a single forward, reuses the existing `_speculative_sampling` helper.
- Adds a tiny `MTPLayer` (RMSNorm + concat-and-project + decoder block + `shared_head`) mirroring vLLM's `deepseek_mtp.py`. Base `forward` keeps iterating only `self.layers[:num_hidden_layers]`; MTP is reached via a new `model.forward_mtp(...)` helper.
- `_assisted_decoding` / `candidate_generator.py` are untouched.

## Scope

- `generate(..., use_mtp=True)` is supported for DeepSeek-V3 / GLM-4 MoE at `batch_size=1` with dynamic cache.
- `generate_batch` with `use_mtp=True` raises `NotImplementedError` for now — continuous batching needs paged-cache slot reservation (K+1 per MTP request) plus per-request accept/reject in `_forward_process_and_sample`. That will come as a follow-up PR so it gets a clean review on its own.
- Training-time MTP loss is explicitly out of scope.

## Design decisions

| | |
|---|---|
| Draft | Model's own MTP heads (shared weights + shared KV cache). |
| Path | Dedicated `_mtp_decoding`, not routed through `_assisted_decoding`. |
| Activation | Explicit `use_mtp=True`; depth = `config.num_nextn_predict_layers`. |
| Layout | Tail of `self.layers` holds MTP modules (matches the checkpoint key scheme `model.layers.{num_base + k}.*`). |

## Files

- **Generation:** `generation/configuration_utils.py` (new `use_mtp` flag + mode), `generation/utils.py` (new `_mtp_decoding`, mode registration, validation), `generation/continuous_batching/continuous_api.py` (explicit refusal for now).
- **Models:** `deepseek_v3/` and `glm4_moe/` modular/config/modeling. Downstream variants (`glm4_moe_lite`, `glm4v_moe`, `glm_moe_dsa`, `longcat_flash`, `solar_open`, `youtu`) got regenerated with the new field propagated through the modular chain.
- **Tests:** new `tests/generation/test_mtp.py`.

## Test plan

- [x] `pytest tests/generation/test_mtp.py` — 9/9 passing:
  - `use_mtp=True` routes to `GenerationMode.MTP_DECODING`.
  - Greedy output matches plain `_sample` token-for-token for K=1/2/3 on DeepSeek-V3 and GLM-4 MoE (random-init, so drafts usually miss → verified bonus-token fallback path).
  - `use_mtp=True` with `num_nextn_predict_layers == 0` raises a clear `ValueError`.
  - Base forward output is unchanged when MTP layers are added (shared-weight copy + compare).
  - `model.forward_mtp` produces `(hidden, logits)` of the expected shapes.
  - `ContinuousBatchingManager(..., GenerationConfig(use_mtp=True))` raises `NotImplementedError`.
- [x] `pytest tests/generation/test_candidate_generator.py tests/generation/test_configuration_utils.py` — unchanged / green (regression check).
- [x] Import smoke-test on every modular-regenerated downstream model (`glm4_moe_lite`, `glm4v_moe`, `glm_moe_dsa`, `longcat_flash`, `solar_open`, `youtu`).
- [x] `make style` clean.
- [x] `make fix-repo` clean apart from a pre-existing `mlinter._using_rule_specs` / `check_modeling_structure.py --rules-toml` toolchain mismatch that also fails on an unmodified checkout — unrelated to this PR.

## Follow-ups

- `generate_batch` + MTP (paged-cache reservation, per-request accept/reject).
- Prefill through the MTP layers during the initial forward, to seed their KV caches for better draft quality (currently each step's MTP self-attn only sees the current query token; degraded acceptance vs vLLM but correct).
- Real-weight end-to-end test on `deepseek-ai/DeepSeek-V3` once GPU capacity is available.

--- Comment by github-actions[bot] at 2026-04-24T06:00:06Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: deepseek_v3, glm4_moe, glm4_moe_lite, glm4v_moe, glm_moe_dsa, longcat_flash, solar_open, youtu

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T06:10:04Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45617). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by ArthurZucker at 2026-04-24T06:24:17Z ---
Superseded by a cleaner refactor (MTP moved out of the model into MTPCandidateGenerator). Reopening from the origin branch.


Add DeepSeek V4

PULL REQUEST #45616: Add DeepSeek V4
State: closed  |  Merged: False
Author: ArthurZucker  |  Base: main
Labels:
Created: 2026-04-24T05:58:31Z

Draft moved to https://github.com/huggingface/transformers/pull/45643

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T06:08:35Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45616). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by 2020zyc at 2026-04-24T12:56:03Z ---
Hello, thank you for your adaptation work. May I ask if it is ready to use now? Or when is it expected to be available? Looking forward to your reply.

--- Comment by ArthurZucker at 2026-04-24T23:42:52Z ---
Had to take a small break but ETA is monday / tuesday

--- Comment by github-actions[bot] at 2026-04-24T23:43:22Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, deepseek_v4

--- Comment by ArthurZucker at 2026-04-25T00:05:05Z ---
Superseded by #45643 (same branch, hosted on origin).

--- Comment by github-actions[bot] at 2026-04-25T00:14:49Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45616&sha=9a4b9f

--- Comment by AmineDiro at 2026-04-24T20:04:02Z ---
base_ep_plan 🙈 🙈 or too soon ?

--- Comment by qgallouedec at 2026-04-24T23:21:03Z ---
```suggestion
    gate = gate.view(batch, length // ratio, ratio, head_dim) + ape.to(gate.dtype)
```

otherwise there is a dtype mismatch

--- Comment by qgallouedec at 2026-04-24T23:22:27Z ---
`ape` is kept in fp32 in the reference checkpoint, but promoting gate to fp32 via this addition would cascade into the pooled KV and break the downstream bf16 matmul.

--- Comment by ArthurZucker at 2026-04-24T23:46:26Z ---
```suggestion
        del self.gate_up_proj_bias
        del self.down_proj_bias
        del self.alpha
        self.limit = config.swiglu_limit
        self.act_fn = ACT2FN[config.hidden_act]
```

--- Comment by ArthurZucker at 2026-04-24T23:47:23Z ---
```suggestion
        for expert_idx in hit:
            expert_idx = expert_idx[0]
            # skip masking index
            if expert_idx == self.num_experts:
                continue
```

--- Comment by ArthurZucker at 2026-04-24T23:49:43Z ---
```suggestion
    def forward(self, hidden_streams: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
```

--- Comment by ArthurZucker at 2026-04-24T23:50:19Z ---
```suggestion
        pre, post, comb = self.attn_hc(hidden_states)
```

--- Comment by ArthurZucker at 2026-04-24T23:50:28Z ---
```suggestion
        pre, post, comb = self.ffn_hc(hidden_states)
```

--- Comment by ArthurZucker at 2026-04-24T23:51:10Z ---
```suggestion
        "attn_hc", "ffn_hc"
```


fix(qianfan_ocr): add XPU expectations

PULL REQUEST #45615: fix(qianfan_ocr): add XPU expectations
State: closed  |  Merged: True
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-04-24T05:14:21Z

@ydshieh pls help review, thx!

--- Comment by github-actions[bot] at 2026-04-24T05:15:26Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qianfan_ocr


PULL REQUEST #45614: Add missing requests dependency to transformers[serving]
State: open  |  Merged: False
Author: Oneirag  |  Base: main
Labels:
Created: 2026-04-24T04:00:01Z

Adds missing requests dependency. Otherwise installing just pip install transformers[serving] cannot launch transformers serve due to lack of requests

--- Comment by Rocketknight1 at 2026-04-24T11:15:45Z ---
cc @LysandreJik