[New Model] Add MiniCPM3 support

PULL REQUEST #45613: [New Model] Add MiniCPM3 support
State: open  |  Merged: False
Author: aliyevaladddin  |  Base: main
Labels:
Created: 2026-04-23T20:22:49Z

# What does this PR do?

  Adds native support for the **MiniCPM3** model architecture
  (`openbmb/MiniCPM3-4B`).

  MiniCPM3 uses Multi-head Latent Attention (MLA) from DeepSeek-V2 combined with
   a standard dense MLP (no MoE), plus three scaling mechanisms:
  - `scale_emb` — embedding scaling
  - `scale_depth / sqrt(num_hidden_layers)` — residual connection scaling
  - `hidden_size / dim_model_base` — logit scaling before the LM head

  The implementation uses the modular model pattern, inheriting from Llama
  (config, MLP, model structure) and DeepSeek-V2 (MLA attention, rotary
  embeddings).

  Fixes #41115

  ## Code Agent Policy

  - [x] I confirm that this is not a pure code agent PR.

  ## Before submitting
  - [ ] This PR fixes a typo or improves the docs (you can dismiss the other
  checks if that's the case).
  - [x] Did you read the [contributor guideline](https://github.com/huggingface/
  transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
        Pull Request section?
  - [x] Was this discussed/approved via a Github issue or the
  [forum](https://discuss.huggingface.co/)? Please add a link
        to it if that's the case. — #41115
  - [ ] Did you make sure to update the documentation with your changes?
  - [ ] Did you write any new necessary tests?

  ## Who can review?

  @ArthurZucker @Cyrilvallez

--- Comment by aliyevaladddin at 2026-04-23T20:40:56Z ---

  Add MiniCPM3 model support with configuration and modeling classes

  - Implement MiniCPM3 using modular pattern (inherits Llama + DeepSeek-V2 MLA
  attention)
  - Add embedding scaling (scale_emb), residual scaling (scale_depth), and logit
   scaling (dim_model_base)
  - Register model in AutoConfig, AutoModel, AutoModelForCausalLM,
  AutoModelForSequenceClassification
  - Verified forward pass with small config

--- Comment by Rocketknight1 at 2026-04-24T11:41:52Z ---
There's an existing PR at #41116, but it's quite stale at this point

--- Comment by aliyevaladddin at 2026-04-24T14:34:37Z ---

  Thanks @Rocketknight1! Since #41116 has been stale since October 2025, I'd like to take over
  and submit a fresh PR with MiniCPM3 support using the current modular pattern. I have a working
   implementation with tests ready — will open a PR shortly.

--- Comment by github-actions[bot] at 2026-04-30T21:54:07Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, minicpm3

--- Comment by github-actions[bot] at 2026-04-30T22:06:13Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45613&sha=558ee4


[docs] update model cards

PULL REQUEST #45612: [docs] update model cards
State: closed  |  Merged: True
Author: stevhliu  |  Base: main
Labels:
Created: 2026-04-23T19:59:06Z

backfill model cards for PE family (cc @eustlb) and Qwen3.5 (cc @vasqu)

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T20:18:46Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45612). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-04-30T16:55:46Z ---
Oh we dont do these anymore? Ig I should keep an eye out

--- Comment by vasqu at 2026-04-30T16:56:35Z ---
Qwen 3.6 are also included iirc, would be cool if you could cross check

--- Comment by vasqu at 2026-04-30T16:57:57Z ---
I would specify the causal conv1d to be from the Tri Dao team, not the fla integrated one

--- Comment by vasqu at 2026-04-30T16:58:17Z ---
Also not only slower but way more memory hungry

--- Comment by vasqu at 2026-04-30T16:59:08Z ---
Pretty much the same comments as for the moe
- To check: 3.6
- Specify the causal conv1d origin (fast path)

--- Comment by stevhliu at 2026-04-30T18:32:43Z ---
lets keep the other two but remove "PyTorch" as they're all PyTorch models now

--- Comment by stevhliu at 2026-04-30T18:33:03Z ---
correct, they have the same `model_type` ! 👍


PULL REQUEST #45611: Raise clear error for `problem_type="single_label_classification"` with `num_labels=1`
State: closed  |  Merged: True
Author: gaurav0107  |  Base: main
Labels:
Created: 2026-04-23T19:12:43Z

## What

Adds validation in `PreTrainedConfig.__post_init__` that rejects the combination `problem_type="single_label_classification"` + `num_labels=1` with a clear `ValueError` pointing users to the correct setup.

Closes #45479

## Why

Before this change, constructing a sequence-classification model with this combination silently produced a degenerate zero cross-entropy loss (softmax over a single logit is always `1`, so `CrossEntropyLoss` always returns `0`). Training appeared to run but accomplished nothing.

Reproducer (confirmed on `main`, both on shared-loss models like BERT and inlined-loss models like ModernBERT):

```python
import torch
from transformers import BertConfig, BertForSequenceClassification
config = BertConfig(vocab_size=100, hidden_size=32, num_hidden_layers=2,
                    num_attention_heads=2, intermediate_size=64,
                    num_labels=1, problem_type="single_label_classification")
model = BertForSequenceClassification(config)
model.eval()
out = model(input_ids=torch.tensor([[1, 2, 3, 4]]), labels=torch.tensor([0]))
print(out.loss)  # tensor(0.)
```

## Fix

`PreTrainedConfig.__post_init__` now raises `ValueError` on this combination after `num_labels` is resolved from `id2label`. Error message redirects the user to `num_labels=2` (binary classification) or `problem_type="regression"` (single-output regression).

The check sits at the config layer so it covers every sequence-classification model uniformly — both models that delegate to `ForSequenceClassificationLoss` in `src/transformers/loss/loss_utils.py` and models that inline the loss selection (e.g. ModernBERT, ~40 others).

## Coordination / approach

This matches the maintainer direction in [this comment on #45479](https://github.com/huggingface/transformers/issues/45479#issuecomment-4267713872):

> "maybe we could raise a clearer error if users set `num_labels=1` and `problem_type="single_label_classification"` to tell them not to do that?"

## Tests

New regression test `tests/utils/test_configuration_utils.py::ConfigTestUtils::test_single_label_classification_requires_more_than_one_label` asserts the degenerate combination raises, and verifies the three valid shapes (`num_labels=2 + single_label`, `num_labels=1 + regression`, `num_labels=1` with no explicit `problem_type`) still construct.

Verified that the test FAILS on `main` without the fix and PASSES with it.

Ran locally:

```
pytest tests/utils/test_configuration_utils.py
pytest tests/models/bert/test_modeling_bert.py -k "for_sequence or problem_type"
pytest tests/models/modernbert/test_modeling_modernbert.py -k "problem_type or sequence_classification"
pytest tests/models/roberta/test_modeling_roberta.py::RobertaModelTest::test_problem_types
pytest tests/models/albert/test_modeling_albert.py::AlbertModelTest::test_problem_types
```

All pass. `ruff check` and `ruff format --check` clean on the two touched files.

The existing common `test_problem_types` continues to pass: it assigns `problem_type` and `num_labels` as attributes on an already-constructed config, which bypasses `__post_init__` by design, and the test only asserts `loss.backward()` doesn't error.

## Compatibility

- Valid configurations are unaffected.
- Save/load round-trip of valid configs verified.
- A saved Hub config containing this exact bad combination will now fail to load with a clear error. Any such checkpoint can never have been trained successfully (loss was always 0), so no working workflow regresses.

## AI disclosure

AI-assisted: the fix was drafted with Claude, then human-reviewed end-to-end including the reproducer, diff, and test outputs listed above. Disclosed per `CONTRIBUTING.md` "AI-assisted and agentic contributions".

--- Comment by HuggingFaceDocBuilderDev at 2026-04-24T11:42:21Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45611). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by gaurav0107 at 2026-04-24T13:34:02Z ---
 Hi @Rocketknight1
would you mind taking a look at the CI results and helping me understand what caused the failure? The tests_processors job is reporting a worker crash (crashed and worker restarting disabled, exit status 1)

--- Comment by Rocketknight1 at 2026-04-24T16:36:59Z ---
@gaurav0107 probably just an intermittent CI error, I'll throw it into the queue again

--- Comment by Rocketknight1 at 2026-04-24T11:24:49Z ---
```suggestion
        if self.problem_type == "single_label_classification" and self.num_labels == 1:
            raise ValueError(
                '`problem_type="single_label_classification"` requires `num_labels > 1`. For binary "
                'classification use `num_labels=2`, or use `problem_type="regression"` for a '
                "single-output regression head."
            )
```

--- Comment by Rocketknight1 at 2026-04-24T11:25:14Z ---
```suggestion
```

--- Comment by Rocketknight1 at 2026-04-24T11:27:15Z ---
```suggestion
        if self.problem_type == "single_label_classification" and self.num_labels == 1:
            raise ValueError(
                '`problem_type="single_label_classification"` requires `num_labels > 1`. For binary '
                'classification use `num_labels=2`, or use `problem_type="regression"` for a '
                "single-output regression head."
            )
```


PULL REQUEST #45610: Fix configuration reading and error handling for kernels
State: closed  |  Merged: True
Author: hmellor  |  Base: main
Labels: for patch
Created: 2026-04-23T17:18:27Z

Fixes the following issues when trying to run Qwen3.6 FP8 checkpoints such as `Qwen/Qwen3.5-35B-A3B-FP8`. This PR fixes the following issues:

- The FP8 checkpoint stores the experts in a `nn.ModuleList` but the necessary `WeightConverter`s were missing
- When `kernels` was not installed it failed with a confusing attribute error
- The backup config names are eagerly read in `FP8Experts.__init__` so models which use the primary name fail with attribute errors

Closes https://github.com/huggingface/transformers/issues/44230

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T17:34:38Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45610). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by vasqu at 2026-04-23T17:53:15Z ---
Feel free to merge after the conversion mapping + getattr detail :hugs:

--- Comment by github-actions[bot] at 2026-04-23T18:17:21Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45610&sha=a18eb3

--- Comment by vasqu at 2026-04-23T17:25:29Z ---
Can we instead make this copy from qwen 3.5 text and qwen2 moe

--- Comment by vasqu at 2026-04-23T17:27:02Z ---
I'm not sure why we need this tbh, we have the error below, no? This should essentially be the same and it also tells to install kernels, same for below

```
    if missing:
        raise ImportError(
            f"finegrained-fp8 kernel is missing required functions: {', '.join(missing)}. "
            "Please update the `kernels` package (`pip install -U kernels`)."
        )
```

--- Comment by vasqu at 2026-04-23T17:29:03Z ---
Isn't it essentially the same? Now just that we can have even more attributes in theory we go through?

I'd like to revert tbh, as we really shouldn't add more possibilities

--- Comment by vasqu at 2026-04-23T17:31:58Z ---
Open to rephrase that a bit or change the order of the message slightly!

--- Comment by hmellor at 2026-04-23T17:35:51Z ---
We never get to that error because if kernels is not installed then `kernel` is `None` and `getattr(None, some_kernel)` fails with an unhelpful error

--- Comment by hmellor at 2026-04-23T17:38:26Z ---
It's not the same.

Before this PR, the default values get evaluated and if they don't exist we get an attribute error.

After this PR, the default value only gets evaluated if the first key wasn't found.

--- Comment by vasqu at 2026-04-23T17:38:45Z ---
Could we default to `None` explicitly, I think python should understand that

--- Comment by hmellor at 2026-04-23T17:39:10Z ---
Is there a mechanism to combine transforms from multiple other models? AFAIK we only have 1:1 copies

--- Comment by vasqu at 2026-04-23T17:41:39Z ---
Hmm, it seems it was changed on the gemma4 addition could we make the hasattr style instead

See https://github.com/huggingface/transformers/commit/91b1ab1fdfa81a552644a92fbe3e8d88de40e167#diff-faf0f03d7fcee1138b40be5d19010f2e143296983071d3cb31f5de3bc4f02b46

Not sure why they did that tbh

--- Comment by vasqu at 2026-04-23T17:43:00Z ---
I mean moreso to use `mapping["..."] = mapping["qwen2_moe"].copy()` statements, so that we know that they are not unique. Definitely need to improve on the composability

--- Comment by vasqu at 2026-04-23T17:44:47Z ---
So `getattr(kernel, "somestring", None)` should work :thinking:

--- Comment by hmellor at 2026-04-23T17:44:53Z ---
Ah ok, yeah we can do that

--- Comment by hmellor at 2026-04-23T17:45:22Z ---
That would work too, I can do that if you prefer

--- Comment by vasqu at 2026-04-23T17:47:16Z ---
Yea that would be nice, that was actually the intended way but it got messed up :cry:

--- Comment by hmellor at 2026-04-23T17:48:28Z ---
We could use `hasarttr` but it's not very nice to read.

vLLM has a very similar utility called `getattr_iter` for checking multiple names in configs.

--- Comment by vasqu at 2026-04-23T17:51:26Z ---
Yea no worries, this ones ok to keep. Sorry very nitpicky with this review lets keep this


PULL REQUEST #45609: make it possible to ser/deser HF MoE models with torchao
State: open  |  Merged: False
Author: vkuzo  |  Base: main
Labels:
Created: 2026-04-23T15:41:04Z

Summary:

TODO(before review): write me in more detail

Test Plan:

TODO write me

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [ ] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc
- kernels: @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Python code in model docs

PULL REQUEST #45608: Python code in model docs
State: closed  |  Merged: True
Author: zucchini-nlp  |  Base: main
Labels:
Created: 2026-04-23T15:17:28Z

# What does this PR do?

- Deletes all `torch_dtype` and `dtype`, since we load models in default dtype from config in v5
- Uses plain python code (find and delete all `>>> ` and `... `)

Batch updating because I see many comments in new model PRs asking users to fix it, though the best way is for us to fix old docs

@vasqu , do you have any other recurring/annoying patterns? Would be great to reduce number of nitty-picky PR comments, and save us time for more important tasks

--- Comment by vasqu at 2026-04-23T15:20:54Z ---
One quick one: missing device map auto on models and or missing movement from prepared input `.to(model.device)`

--- Comment by zucchini-nlp at 2026-04-23T15:24:15Z ---
This will need an agent help, not super regex friedly. Oke, noted

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T15:28:06Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by zucchini-nlp at 2026-04-24T09:46:44Z ---
@stevhliu this PR is generated by find-replace and code agent. As per title, eliminates most common patterns in new model reviews, so users don't copy a bad example

Whenever you have time, can you reveiew?

--- Comment by vasqu at 2026-04-24T13:08:10Z ---
@zucchini-nlp Another one maybe for a different PR: Using httpx vs requests. Sorry dropping stuff as I review other models atm :D

--- Comment by zucchini-nlp at 2026-04-24T13:10:22Z ---
No prob, gotta keep docs clean and anyway it is the LLM doing the work, I just re-check. Defi another PR opportunity :)

--- Comment by zucchini-nlp at 2026-04-27T13:19:00Z ---
Love vibe-coding, always have to check and clean manually afterwards 😄  I'll go over and fix those

--- Comment by zucchini-nlp at 2026-04-27T14:19:38Z ---
I think it is better now, applied ruff to fix indentation, reverted all `device=0` which were deleted incorrectly and fixed the merged Bnb names

--- Comment by zucchini-nlp at 2026-04-28T10:19:34Z ---
Yep, thanks, imo with the skill and a bit more requirements on using the provided doc-template, we can cut down time spent reviewing new model docs

--- Comment by stevhliu at 2026-04-24T16:21:14Z ---
i notice some examples keep `device=0` while others remove it (like the one immediately above)

--- Comment by stevhliu at 2026-04-24T16:21:54Z ---
i think this should still be `bnb_4bit_use_double_quant`

--- Comment by stevhliu at 2026-04-24T16:23:13Z ---
this should also just be `bnb_4bit_quant_type`

--- Comment by stevhliu at 2026-04-24T16:24:06Z ---
did we mean to remove this as well?

--- Comment by stevhliu at 2026-04-24T16:24:23Z ---
i think the indenting is off here

--- Comment by stevhliu at 2026-04-24T16:25:56Z ---
```suggestion
    model_name,
```

--- Comment by stevhliu at 2026-04-24T16:26:26Z ---
indenting may also be off here

--- Comment by stevhliu at 2026-04-24T16:26:42Z ---
indenting off here as well

--- Comment by stevhliu at 2026-04-24T16:26:54Z ---
same

--- Comment by stevhliu at 2026-04-24T16:27:25Z ---
```suggestion
    model="facebook/dinov3-vits16-pretrain-lvd1689m",
```

--- Comment by stevhliu at 2026-04-24T16:28:29Z ---
off here as well

--- Comment by stevhliu at 2026-04-24T16:28:52Z ---
```suggestion
    device_map="auto",
```

--- Comment by stevhliu at 2026-04-24T16:28:59Z ---
```suggestion
    device_map="auto",
```

--- Comment by stevhliu at 2026-04-24T16:29:08Z ---
```suggestion
    "baidu/ERNIE-4.5-VL-28B-A3B-PT",
```

--- Comment by stevhliu at 2026-04-24T16:29:22Z ---
```suggestion
    "baidu/ERNIE-4.5-VL-28B-A3B-PT",
```

--- Comment by stevhliu at 2026-04-24T16:29:47Z ---
```suggestion
    bnb_4bit_quant_dtype="nf4",
```

--- Comment by stevhliu at 2026-04-24T16:30:19Z ---
```suggestion
    bnb_4bit_quant_dtype="nf4",
```

--- Comment by stevhliu at 2026-04-24T16:30:39Z ---
```suggestion
    bnb_4bit_quant_dtype="nf4"
```

--- Comment by stevhliu at 2026-04-24T16:30:46Z ---
indenting off

--- Comment by stevhliu at 2026-04-24T16:30:54Z ---
indenting off

--- Comment by stevhliu at 2026-04-24T16:31:05Z ---
```suggestion
    model="zai-org/GLM-4.7-Flash",
```

--- Comment by stevhliu at 2026-04-24T16:31:17Z ---
```suggestion
    model="zai-org/GLM-5",
```

--- Comment by stevhliu at 2026-04-27T16:37:55Z ---
the `runnable:test_basic` should stay in the code fence

```suggestion
```python runnable:test_basic
```

--- Comment by stevhliu at 2026-04-27T16:38:06Z ---
```suggestion
```python runnable:test_advanced
```

--- Comment by stevhliu at 2026-04-27T16:38:15Z ---
```suggestion
```python runnable:test_audio_array
```

--- Comment by stevhliu at 2026-04-27T16:38:23Z ---
```suggestion
```python runnable:test_batched
```

--- Comment by stevhliu at 2026-04-27T16:39:48Z ---
still weird indents here

--- Comment by stevhliu at 2026-04-27T16:40:04Z ---
```suggestion
  `<|extratoken_1|>...<|extratoken_143|>`, so the `vocab_size` of tokenizer also becomes 50400.
```

--- Comment by stevhliu at 2026-04-27T16:41:51Z ---
```suggestion
```


PULL REQUEST #45607: Add regression test for Gemma4 audio relative positional range
State: closed  |  Merged: False
Author: mathceo  |  Base: main
Labels:
Created: 2026-04-23T15:10:59Z

## What does this PR do?

Adds a regression test for Gemma4AudioRelPositionalEncoding with a non-default audio config.

Follow-up to #45606.

--- Comment by mathceo at 2026-04-23T16:52:04Z ---
@vasqu small note: this follow-up test PR currently fails on main because it depends on the fix in #45606. Once #45606 is merged, I can update this branch so the regression test runs against the fixed behavior.

--- Comment by vasqu at 2026-04-23T17:02:57Z ---
Imo, it makes more sense to have it directly in the other PR before merge cc @eustlb

--- Comment by github-actions[bot] at 2026-04-24T09:18:37Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4

--- Comment by github-actions[bot] at 2026-04-24T09:25:35Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45607&sha=077c09

--- Comment by vasqu at 2026-04-27T12:21:38Z ---
Closing this in favor of #45606 (where this has been included and attributed :D)


PULL REQUEST #45606: [gemma4] infer from config instead of hardcoding
State: closed  |  Merged: True
Author: eustlb  |  Base: main
Labels:
Created: 2026-04-23T14:42:54Z

# What does this PR do?

As per title. Fix #45468

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T14:56:14Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45606). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by mathceo at 2026-04-23T14:57:43Z ---
@eustlb thanks for the clarification and the fix - context_size // 2 + 1 makes sense here given the blocked attention path and _rel_shift.

One thing that may still be useful is a small regression test for nondefault audio configs, so this does not silently stay tied to the default length again.

Something along these lines in tests/models/gemma4/test_modeling_gemma4.py:

```python
def test_audio_rel_pos_encoding_uses_context_size_from_config(self):
    from transformers.models.gemma4.configuration_gemma4 import Gemma4AudioConfig
    from transformers.models.gemma4.modeling_gemma4 import Gemma4AudioRelPositionalEncoding

    config = Gemma4AudioConfig(
        hidden_size=32,
        attention_chunk_size=6,
        attention_context_left=5,
        attention_context_right=1,
        use_clipped_linears=False,
    )

    module = Gemma4AudioRelPositionalEncoding(config)
    hidden_states = torch.zeros(1, 3, config.hidden_size)

    pos = module(hidden_states)

    context_size = config.attention_chunk_size + config.attention_context_left - 1 + config.attention_context_right
    expected_len = context_size // 2 + 1

    self.assertEqual(pos.shape, (1, expected_len, config.hidden_size))

    position_ids = torch.arange(context_size // 2, -1, -1, device=hidden_states.device)[..., None]
    scaled_time = position_ids * module.inv_timescales.to(device=hidden_states.device)
    expected = torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], dim=-1).to(hidden_states.dtype)

    torch.testing.assert_close(pos, expected)
```
Happy to open a follow-up PR for the test as well if that is preferred.

--- Comment by mathceo at 2026-04-23T17:16:52Z ---
I already implemented the regression test in #45607. Since #45606 is not my branch, I can't push directly there from my side. Feel free to cherry-pick the test commit from #45607
@eustlb

--- Comment by eustlb at 2026-04-24T09:23:10Z ---
thanks @mathceo! added you as a co-author here

--- Comment by vasqu at 2026-04-27T12:20:47Z ---
run-slow: gemma4

--- Comment by github-actions[bot] at 2026-04-27T12:21:27Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4

--- Comment by github-actions[bot] at 2026-04-27T12:22:12Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24994665509)

This comment contains `run-slow`, running the specified jobs:

models: ["models/gemma4"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-27T12:52:22Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24994665509)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [32ada79d](https://github.com/huggingface/transformers/commit/32ada79d5d64f6ee52cb9435e56cc6d7386519bf) | workflow commit (merge commit) |
| PR | [d404da97](https://github.com/huggingface/transformers/commit/d404da9777995ec33c518977331e1cb0d8bd2d9c) | branch commit (from PR) |
| main | [bbb51c83](https://github.com/huggingface/transformers/commit/bbb51c83c151c3285d7f0e63ea6a68165c61cc58) | base commit (on `main`) |

### Model CI Report

❌ **[1 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/81f8a8c1a831cf446bf25d05007a1211ccaa6aae/2026-04-27/runs/33760-24994665509/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [gemma4](https://github.com/huggingface/transformers/actions/runs/24994665509/job/73188732178):
    tests/models/gemma4/test_modeling_gemma4.py::Gemma4IntegrationTest::test_export_text_only (❌ ⟹ ❌)

--- Comment by vasqu at 2026-04-27T12:58:17Z ---
The different failure is a different PID in the test itself lol, merging


PULL REQUEST #45605: Processing Utils: continue when content is a string
State: closed  |  Merged: True
Author: RyanMullins  |  Base: main
Labels:
Created: 2026-04-23T14:11:12Z

# What does this PR do?

Fixes crash for the following `messages` structure when calling

```python
processor = AutoProcessor.from_pretrained(model_id)
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [{"type": "text", "text": "Write a short joke about saving RAM."}]},
]
processor.apply_chat_template(messages, tokenize=True)
```

```
Traceback (most recent call last):
  File "$HOME/git/transformers/local_testing/assisted_generation.py", line 18, in <module>
    inputs = processor.apply_chat_template(
        messages,
    ...<4 lines>...
        enable_thinking=False,
    )
  File "$HOME/git/transformers/src/transformers/processing_utils.py", line 1822, in apply_chat_template
    content_block for content_block in content if content_block["type"] in ["image", "video"]
                                                  ~~~~~~~~~~~~~^^^^^^^^
TypeError: string indices must be integers, not 'str'
```

## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
**we ask that new users do not submit pure code agent PRs** at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Discussed with @zucchini-nlp and @Rocketknight1 on Slack
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@zucchini-nlp

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T14:39:17Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Agent first cli with skill

PULL REQUEST #45604: Agent first cli with skill
State: open  |  Merged: False
Author: LysandreJik  |  Base: agent-first-cli
Labels:
Created: 2026-04-23T13:24:28Z

Adding a skill based on the CLI

--- Comment by github-actions[bot] at 2026-04-23T13:24:54Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45604&sha=59e475

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T13:36:16Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45604). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by tarekziade at 2026-04-23T14:41:52Z ---
semi-related, in case we want to pull out of the transformers package to speed-up agents-CLI interactions https://github.com/huggingface/transformers/pull/45181


PULL REQUEST #45603: [Auto] Pass kwargs to fixed_cross_entropy (cluster-43240-3): merged 2 of 2 PRs
State: closed  |  Merged: False
Author: evalstate  |  Base: main
Labels:
Created: 2026-04-23T11:41:51Z

Merged PRs:
- #43254: Merged cleanly; smaller implementation of the fixed_cross_entropy kwargs fix with passing CI.
- #43251: Merged after resolving one overlap in src/transformers/loss/loss_utils.py by taking the PR's fixed_cross_entropy signature/body changes.

Skipped PRs:
- none

Failed PRs:
- none

Notes:
- Cluster issue was #43240; both PRs targeted the same bug in fixed_cross_entropy.
- The two PRs are competing implementations and touch the same hunk in src/transformers/loss/loss_utils.py.
- Conflict between #43254 and #43251 was limited to adding weight/label_smoothing support plus kwargs naming/formatting differences.
- Final branch diff versus origin/main reflects the #43251-style result: explicit weight and label_smoothing parameters passed to nn.functional.cross_entropy with **_kwargs retained.

Next steps:
- If desired, run make style or targeted tests before any further handoff.
- Review whether keeping both merge commits is acceptable since the PRs are overlapping alternatives for the same fix.

--- Comment by evalstate at 2026-04-23T11:42:07Z ---
Trace for this mergeability run: https://huggingface.co/datasets/evalstate/transformers-merge-experiments/blob/main/2604231238-dX7qnR__dev__codex.jsonl

--- Comment by evalstate at 2026-04-23T11:43:15Z ---
Commit made by agent by mistake -- intended target was evalstate/transformers.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T11:52:40Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45603). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


[AMD CI] Fix expectations for Gemma3n

PULL REQUEST #45602: [AMD CI] Fix expectations for Gemma3n
State: closed  |  Merged: True
Author: Abdennacer-Badaoui  |  Base: main
Labels:
Created: 2026-04-23T10:34:16Z

As per title.

--- Comment by github-actions[bot] at 2026-04-23T10:35:30Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3n

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T10:46:35Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45602). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by tarekziade at 2026-04-23T10:48:46Z ---
run-slow: gemma3n

--- Comment by github-actions[bot] at 2026-04-23T10:50:04Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24831054954)

This comment contains `run-slow`, running the specified jobs:

models: ["models/gemma3n"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-23T11:21:12Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/24831054954)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [187a6cda](https://github.com/huggingface/transformers/commit/187a6cdae49c00d5e4bc48f625a480e520568244) | workflow commit (merge commit) |
| PR | [4090f240](https://github.com/Abdennacer-Badaoui/transformers/commit/4090f240a7bfbd45d77cdbfc1d7fa5ccbc6401b2) | branch commit (from PR) |
| main | [07e38311](https://github.com/huggingface/transformers/commit/07e3831133a37487ee6faf57aed965f39c962a43) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by tarekziade at 2026-04-23T10:51:00Z ---
looks like it's the same output than `cuda` - should we use the same text variable ?

--- Comment by tarekziade at 2026-04-23T10:51:40Z ---
same remark

--- Comment by Abdennacer-Badaoui at 2026-04-23T11:51:35Z ---
I don’t think so, since these tests also seem to be failing on NVIDIA. So maybe their expectations should be updated as well.


PULL REQUEST #45601: fix: compute auxiliary losses when denoising is disabled in D-FINE
State: closed  |  Merged: True
Author: Abineshabee  |  Base: main
Labels:
Created: 2026-04-23T09:56:08Z

Fixes #45593

### Problem

When `num_denoising=0`, `denoising_meta_values` is `None`, causing `DFineForObjectDetectionLoss` to skip auxiliary loss computation entirely since it is gated under the same condition.

However, auxiliary losses (deep supervision across decoder layers) are independent of denoising and should still be computed when `config.auxiliary_loss=True`. This leads to degraded training behavior when denoising is disabled.

### Fix

Decoupled denoising logic from auxiliary loss computation:

* Tensor splitting is performed only when `denoising_meta_values` is not `None`
* Auxiliary losses are always computed when `config.auxiliary_loss=True`
* Denoising-specific auxiliary losses are computed only when denoising is active

This follows the same pattern used in `loss_rt_detr.py`.

### Testing

* Verified that auxiliary loss keys (e.g., `loss_vfl_aux_*`, `loss_bbox_aux_*`) are present when `num_denoising=0`
* Confirmed no change in behavior when denoising is enabled

--- Comment by Rocketknight1 at 2026-04-23T11:48:43Z ---
cc @molbap @yonigozlan @nielsrogge maybe

--- Comment by Abineshabee at 2026-04-23T15:18:17Z ---
Thanks for the review! That makes sense — I’ll add a small test to verify auxiliary losses are computed when num_denoising=0.

--- Comment by Abineshabee at 2026-04-23T16:38:48Z ---
CI failed due to a CircleCI authentication error (not related to the code change). Could someone please re-run the workflow?

--- Comment by github-actions[bot] at 2026-04-23T17:05:00Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: d_fine

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T17:16:46Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45601). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


qa: more lazy loading

PULL REQUEST #45599: qa: more lazy loading
State: open  |  Merged: False
Author: tarekziade  |  Base: main
Labels:
Created: 2026-04-23T08:48:34Z

# What does this PR do?

Per https://github.com/huggingface/transformers/issues/44273 there are a few spots where we can do more lazy loading to speed up `import transformers` without complexifying the code too much

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T08:59:57Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45599). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by tarekziade at 2026-04-23T09:17:16Z ---
The public API of `transformers.utils` forces me to be very agressive on `hub.py` -- I am not sure if it's worth the added complexity.

The current patch reduces to 270 imports / and 0.5 seconds on my M5

@LysandreJik WDYT?

I feel like doing this can be quite fragile/painful to maintain..

--- Comment by github-actions[bot] at 2026-04-23T09:31:57Z ---
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45599&sha=3c10a2

--- Comment by LysandreJik at 2026-04-27T08:33:33Z ---
@albertvillanova can you confirm whether installing this branch results in dramatic speedups on your machine? not sure we want to guard numpy imports like this either

--- Comment by albertvillanova at 2026-04-28T04:20:13Z ---
Thanks for the ping, @LysandreJik. I'm checking it right now.

--- Comment by albertvillanova at 2026-04-28T04:31:41Z ---
Yes, I confirm I see a gain in speed:
```bash
time python -c "import transformers"

real	0m2.325s
user	0m0.518s
sys	0m0.150s
```

Before:
```bash
time python -c "import transformers"

real	0m2.836s
user	0m1.775s
sys	0m0.138s
```

--- Comment by albertvillanova at 2026-04-28T04:47:09Z ---
On another machine:
- Now:
```bash
time python -c "import transformers"

real	0m1.330s
user	0m1.230s
sys	0m0.100s
```
- Before:
```bash
time python -c "import transformers"

real	0m1.819s
user	0m3.204s
sys	0m0.176s
```

--- Comment by tarekziade at 2026-04-28T07:33:45Z ---
@albertvillanova thanks for the tests!

Could you share a bit more about the use case where startup time is critical here?
In some CLI / serving setups, transformers can be lazy-loaded, so it’d be great to understand how this fits your scenario.

--- Comment by albertvillanova at 2026-04-28T08:55:44Z ---
Lazy loading in transformers would be an ideal solution, but I think it is currently broken, e.g. importing anything from `transformers.utils`.

Indeed, my use case is improving the startup latency of the `trl` CLI. It imports transformers and this was a significant contributor to the overall startup time.

As a result, we had to introduce some workarounds in trl to avoid importing some transformers functions at CLI startup. See, e.g.:
- https://github.com/huggingface/trl/pull/5129
- https://github.com/huggingface/trl/pull/5208
- https://github.com/huggingface/trl/pull/5210
- https://github.com/huggingface/trl/pull/5257

If lazy loading in transformers is fixed and becomes more reliable, we could eventually revert these changes and simplify the current setup.

--- Comment by tarekziade at 2026-04-28T13:17:59Z ---
Transformers is a large codebase, so importing some modules will inevitably pull in dependencies. Lazy loading can help, but it also adds complexity because we want to keep APIs backward compatible (like what's exposed in `transformers.utils`), so there’s a tradeoff between startup speed and maintainability. And in practice, once you really use Transformers, you usually end up needing things like NumPy or Torch anyway.

For the TRL CLI, I totally understand the goal. That’s exactly why I worked on making startup faster and added a regression test to keep it that way. But I think we’re reaching a point where pushing this further makes some modules harder to maintain, so I’d rather defer more of that laziness to downstream projects, similar to what you did.

That said, I wouldn’t say Transformers lazy loading is “broken”; that feels a bit strong to me. It may be that some imports still have more overhead than expected.

What would be an acceptable import-time overhead for your use case?


PULL REQUEST #45598: Align latest model attention function dispatch
State: closed  |  Merged: True
Author: Cyrilvallez  |  Base: main
Labels:
Created: 2026-04-23T08:38:32Z

# What does this PR do?

Those were not up to date

--- Comment by github-actions[bot] at 2026-04-23T08:39:37Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma4, modernbert, moonshine_streaming

--- Comment by HuggingFaceDocBuilderDev at 2026-04-23T08:49:04Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45598). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


PULL REQUEST #45597: Add Granite 4.1 Vision (granite4_vision)
State: closed  |  Merged: True
Author: artem-spector  |  Base: main
Labels: New model
Created: 2026-04-23T06:51:56Z

## What does this PR do?

Adds built-in support for **Granite 4.1 Vision** (`granite4_vision`), IBM's multimodal vision-language model for enterprise document understanding.

### Architecture highlights
- **Vision encoder:** SigLIP2 (`google/siglip2-so400m-patch16-384`), tiled 384×384 patches
- **Window Q-Former projector:** 4×4 patch windows compressed to 2×2 query tokens via cross-attention (`downsample_rate="4/8"`)
- **DeepStack feature injection:** 8 vision-to-LLM injection points across two mechanisms:
  - *LayerDeepstack:* features from 4 vision encoder depths injected at 4 LLM layers (reversed order — deepest vision → earliest LLM)
  - *SpatialDeepstack:* deepest features split into 4 spatial offset groups (TL/TR/BL/BR), injected at 4 later LLM layers
- **Language model:** GraniteForCausalLM (3.5B) with a rank-256 LoRA adapter (same-repo, LM-only)

### Files added
| File | Purpose |
|---|---|
| `modular_granite4_vision.py` | Source of truth — inherits from LLaVA-Next, overrides novel components |
| `configuration_granite4_vision.py` | Config (generated) |
| `modeling_granite4_vision.py` | Model (generated) |
| `processing_granite4_vision.py` | Unified processor (generated) |
| `image_processing_granite4_vision.py` | Torchvision-based image processor |
| `image_processing_pil_granite4_vision.py` | PIL/NumPy image processor |
| `tests/models/granite4_vision/` | Modeling, image processing, and processor tests |
| `docs/source/en/model_doc/granite4_vision.md` | Model documentation |

### Auto-registration
- Config: auto-generated via `configuration_granite4_vision.py` model_type
- Modeling: `MODEL_MAPPING_NAMES` + `MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES`
- Processing + image processing: registered in respective auto files

### Tests
- Unit tests pass locally (`pytest tests/models/granite4_vision/ -x -q`)
- `@slow` integration tests load real checkpoint and assert outputs within tolerance
- `make style` and `make check-repo` pass (3 remaining failures are pre-existing upstream issues: `mlinter` version mismatch and `Sam3Lite` incomplete model)

## Before submitting
- [x] This PR is not a duplicate
- [x] I have read the [contributor guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md)
- [x] The documentation reflects the changes
- [x] The tests pass

## Related
- vLLM built-in support: https://github.com/vllm-project/vllm/pull/40282 (merged)
- Model: https://huggingface.co/ibm-granite/granite-vision-4.1-4b

--- Comment by artem-spector at 2026-04-23T08:36:05Z ---
I've traced the root cause of the check_repository_consistency and tests_torch failures to a specific upstream commit:

  [Sam3LiteText] Remove unnecessary modules/configs (#45535) (7439ac0459)

  This commit removed Sam3LiteTextViTConfig and Sam3LiteTextVisionConfig from the modeling file but left them referenced in auto_mappings.py, causing:
  - AttributeError: module transformers has no attribute Sam3LiteTextViTConfig (357 test failures)
  - check_repo failure: `Sam3LiteTextVisionConfig` appears in CONFIG_MAPPING_NAMES but is not defined

  This is reproducible on main independently of our PR.

  Question for reviewers: Should we include a fix for this in our PR (removing the stale entries from auto_mappings.py), or would you prefer to handle it in a separate hotfix? Happy to do either.

--- Comment by artem-spector at 2026-04-23T09:51:43Z ---
Opened a dedicated issue for the upstream regression: #45600

--- Comment by zucchini-nlp at 2026-04-23T14:22:51Z ---
LMK when ready for review, and ig this PR supersedes https://github.com/huggingface/transformers/pull/45350?

--- Comment by artem-spector at 2026-04-25T08:06:01Z ---
@zucchini-nlp - yes, this PR supersedes https://github.com/huggingface/transformers/pull/45350. Its our team that is responsible for producing/release IBM vision models.
This PR is ready for review from my side.

--- Comment by HuggingFaceDocBuilderDev at 2026-04-27T09:24:07Z ---
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45597). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

--- Comment by artem-spector at 2026-04-29T04:33:19Z ---
@zucchini-nlp I'm ready for the second round :)

--- Comment by artem-spector at 2026-05-01T08:43:22Z ---
Hi @zucchini-nlp, sorry for delay - I was chasing a performance problem, which turn out to be not real.

Summary of changes:
- Granite4VisionImageFeaturesOutput now inherits BaseModelOutputWithPooling instead of bare ModelOutput, which unblocks the common image-features tests
- Added @capture_outputs to Granite4VisionTextModel.forward so output_hidden_states propagates correctly through generation
- Removed the manual DynamicCache init from forward — GenerationMixin._prepare_cache_for_generation handles this
- get_image_features now properly passes output_hidden_states through (via kwarg or config fallback)
- qformer_config is now fully specified at __init__ time rather than patched post-super().__init__()
- _can_record_outputs and _deepstack_inject moved up to Granite4VisionPreTrainedModel
- One-letter variable names renamed for clarity; unused inherited config attrs now raise AttributeError with a clear message

Open question — test_can_init_all_missing_weights:

This test verifies that every weight can be re-initialized from meta device via _init_weights. It fails because WindowQFormerDownsampler embeds a Blip2QFormerModel (a full PreTrainedModel from a different family). When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized.
Is there a recommended approach here, or is skipping acceptable for embedded third-party PreTrainedModel submodules?

--- Comment by zucchini-nlp at 2026-05-01T09:15:53Z ---
> When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized.

That is weird, usually our initializer walks down each sub-module and initializes its weights. Unless Qformer has special buffers that need to be init, it should be fine. I can check it on Monday and do another review pass

--- Comment by artem-spector at 2026-05-01T10:44:30Z ---
> > When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized.
>
> That is weird, usually our initializer walks down each sub-module and initializes its weights. Unless Qformer has special buffers that need to be init, it should be fine. I can check it on Monday and do another review pass

will re-check too

--- Comment by artem-spector at 2026-05-03T08:15:00Z ---
> > > When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized.
> >
> >
> > That is weird, usually our initializer walks down each sub-module and initializes its weights. Unless Qformer has special buffers that need to be init, it should be fine. I can check it on Monday and do another review pass
>
> will re-check too

you right, `test_can_init_all_missing_weights` works

--- Comment by zucchini-nlp at 2026-05-04T13:53:34Z ---
run-slow: granite4_vision

--- Comment by artem-spector at 2026-05-05T07:14:43Z ---
@zucchini-nlp , thanks for the approval!
All suggestions have been implemented, except one for _init_weights: the Embedding/LayerNorm/RMSNorm branches are necessary, since the parent LlavaNextPreTrainedModel._init_weights doesn't handle these module types.

--- Comment by zucchini-nlp at 2026-05-05T07:35:52Z ---
@artem-spector not the parent llava-next, it is handled in base `PreTrainedModel` class :)

--- Comment by zucchini-nlp at 2026-05-05T07:41:28Z ---
A couple comments only, I think we had a misunderstanding with prev round. And requested review from a core maintainer (@vasqu) , then we can merge

--- Comment by artem-spector at 2026-05-05T11:05:06Z ---
@zucchini-nlp thanks for clarifying! All three addressed and pushed.

--- Comment by zucchini-nlp at 2026-05-05T13:44:44Z ---
@artem-spector we are releasing 5.8.0 today so if you could address all comments until the evening, we are very glad to add granite-vision in the release

If you need help, or you are afk, just ping me and I can fix the rest. It is all just nits about styling and better modular usage

--- Comment by artem-spector at 2026-05-05T15:10:54Z ---
@zucchini-nlp, thank you! I'm working on it, hope to finish soon. BTW what hour is "until evening"?

--- Comment by artem-spector at 2026-05-05T15:29:24Z ---
@zucchini-nlp Few questions:

  Re __post_init__ super order:
  The dict→typed conversion for qformer_config must happen before super().__post_init__ because the super call triggers the _attn_implementation setter which walks sub_configs — if qformer_config is still a raw dict at that point it would fail.
  We could clean this up by removing qformer_config from sub_configs (it doesn't support SDPA anyway so propagating _attn_implementation into it is a no-op) — happy to do that if preferred.

  Re Fraction and torch-compile:
  Fraction is only used during __init__ to parse the config string once — the result is stored as plain int attributes (query_side, window_side) that are used in forward. So it's not in the compiled path.

  Re _init_weights super call:
  The fix requires touching LlavaNextPreTrainedModel — just want to confirm you're okay with us touching LlavaNext code as part of this PR.

--- Comment by zucchini-nlp at 2026-05-05T15:32:20Z ---
> Re post_init super order:

Already fixed no? All subconfigs have to be an obj before calling super(), and we can do so by completely overriding `post-init`, i.e. no copy from llava. Copying had weird side-effects

>  you're okay with us touching LlavaNext

Yes, ofc. I should've also fixed that but feel free if anything is left :)

--- Comment by vasqu at 2026-05-05T15:33:32Z ---
Just pushed some small fixes for CI but feel to add onto them if it did not catch everything

--- Comment by vasqu at 2026-05-05T15:41:33Z ---
@artem-spector can you wait a second, the last force push deleted the last commits. Let me push them

--- Comment by vasqu at 2026-05-05T15:47:53Z ---
Should be good now

--- Comment by github-actions[bot] at 2026-05-05T16:01:01Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, granite4_vision, llava_next, llava_next_video, llava_onevision, ovis2

--- Comment by zucchini-nlp at 2026-04-27T10:03:07Z ---
nit: I think we dont need a bibtext entry and as long as there is a link to HF papers/arxiv, that is enought

--- Comment by zucchini-nlp at 2026-04-27T10:03:47Z ---
nit: these two are by default "auto" so we dont need to manually set

--- Comment by zucchini-nlp at 2026-04-27T10:04:07Z ---
same here, torch_dtype is "auto" by default and can be deleted

--- Comment by zucchini-nlp at 2026-04-27T10:04:52Z ---
lets move this block as `Usage Tips` section, before the usage example code snippets

--- Comment by zucchini-nlp at 2026-04-27T10:05:56Z ---
following "one-model - one-file" philosophy, it is better put inside modular/modeling files

--- Comment by zucchini-nlp at 2026-04-27T10:11:00Z ---
do we have both options in released ckpt? Lets drop any branching code, that is not released

--- Comment by zucchini-nlp at 2026-04-27T10:14:33Z ---
configs have to be pre-defined in `configuration_xx.py`. Let's move it as `config.qformer_config` which will default to blip model type

Similar to how you already have vision/text configs

--- Comment by zucchini-nlp at 2026-04-27T10:20:04Z ---
ig the `embed_std` is only for randomly init weight, so it has to be in `self._init_weights(module)` of the pretrained model class

For ex, in gemma3n

https://github.com/huggingface/transformers/blob/bbb51c83c151c3285d7f0e63ea6a68165c61cc58/src/transformers/models/gemma3n/modeling_gemma3n.py#L1388-L1389

--- Comment by zucchini-nlp at 2026-04-27T10:23:02Z ---
let's name this differently, maybe `self._windowed_raster`. Same for variables, can you rename the one-letter vars to smth more informative/longer in the whole file?

--- Comment by zucchini-nlp at 2026-04-27T10:25:44Z ---
imo the downsampling has to be a simple fn, as it has no weights. We can add two fn `def interpolate_downsample` and `def spatial_offset_downsample`

In the forward of Qformer, we will need to check which one is needed and call it directly with image and original/new sizes

--- Comment by zucchini-nlp at 2026-04-27T10:26:58Z ---
`Copyright 2026 IBM and HuggingFace Inc. team. All rights reserved.` ;)

--- Comment by zucchini-nlp at 2026-04-27T10:28:28Z ---
for the identical image processing, we don't need to re-define it. Instead just add a proper entry in `models/auto/image_processing_auto.py`

--- Comment by zucchini-nlp at 2026-04-27T10:31:01Z ---
nit: processor have no model type :)

--- Comment by zucchini-nlp at 2026-04-27T10:33:24Z ---
hmm, can we not delete it, when creating config in L99

Like this:

```
projector_hidden_act = AttributeError()
multimodal_projector_bias = AttributeError()
{new_attr}: bool = {default_value}
```

--- Comment by zucchini-nlp at 2026-04-27T10:34:28Z ---
does GraniteVision use new-line in any of released ckpt? We could delete it completely if not

--- Comment by zucchini-nlp at 2026-04-27T10:34:49Z ---
pad toke will be in `config.text_config` :)

--- Comment by zucchini-nlp at 2026-04-27T10:37:40Z ---
will this not lead to a size-error down the line, so we can raise error instead of warning?

--- Comment by zucchini-nlp at 2026-04-27T10:38:45Z ---
can we add `kwargs` and `@can_return_tuple` to align wth API

--- Comment by zucchini-nlp at 2026-04-27T10:39:29Z ---
we need to return a `ModelOutput` dict, since this method is used by 3rd party libs and they expect a dict output

--- Comment by zucchini-nlp at 2026-04-27T10:41:24Z ---
same as `get_placeholder_mask` from parent

--- Comment by zucchini-nlp at 2026-04-27T10:42:14Z ---
these args are not needed, instead consumed by kwargs and passed to LLM:

`kwargs: Unpack[TransformersKwargs]`

--- Comment by zucchini-nlp at 2026-04-27T10:44:38Z ---
hmm so this is Granite LLM backbone but with deepstack feature support. In that case, we should add a new text-class `Granite4VisionTextModel` and inherit with modular

--- Comment by zucchini-nlp at 2026-04-27T10:45:59Z ---
can you return deepstack features as extra output, as in qwen3-vl? That way the `vision_output.pooler_output` will be a tensor which is merged in input embeddings

--- Comment by zucchini-nlp at 2026-04-27T10:52:54Z ---
doesn't really match our API. IIUC you re-use the same weights for LLM and have trained weights for vision+qformer

Currently we dont support peft-style merging for such cases, and ckpt on the hub contain all weights already

--- Comment by zucchini-nlp at 2026-04-27T10:54:12Z ---
same here, part of the LLM class that needs to be defined with modular

--- Comment by zucchini-nlp at 2026-04-27T10:54:26Z ---
a few bad rebases :)

--- Comment by zucchini-nlp at 2026-04-27T10:55:03Z ---
I think it is not needed anymore, we added `PrefixWeights` recently and fixed all llava models

--- Comment by zucchini-nlp at 2026-04-27T10:55:11Z ---
2026 :)

--- Comment by zucchini-nlp at 2026-04-27T10:55:51Z ---
we need only this tester, since processing is identical to llava-next. Thanks for using `VLMTester` 🤩

--- Comment by zucchini-nlp at 2026-04-27T10:56:20Z ---
also bad rebase

--- Comment by artem-spector at 2026-04-28T06:14:23Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-28T06:14:49Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-28T06:14:51Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-28T06:14:52Z ---
Done — moved to a "Usage Tips" section before the code examples.

--- Comment by artem-spector at 2026-04-28T06:14:54Z ---
Done — `downsampling_granite4_vision.py` deleted, all contents inlined into the modular.

--- Comment by artem-spector at 2026-04-28T06:14:56Z ---
Done — moved to `Granite4VisionPreTrainedModel._init_weights`.

--- Comment by artem-spector at 2026-04-28T06:14:57Z ---
Done — renamed to `_windowed_raster`/`_unwindowed_raster`, and one-letter variables expanded throughout.

--- Comment by artem-spector at 2026-04-28T06:14:59Z ---
Done — `Copyright 2026 IBM and The HuggingFace Team. All rights reserved.`

--- Comment by artem-spector at 2026-04-28T06:15:01Z ---
Done — removed all re-definitions, added entries in `image_processing_auto.py` pointing to `LlavaNextImageProcessor` and `LlavaNextImageProcessorPil`.

--- Comment by artem-spector at 2026-04-28T06:15:02Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-28T06:15:04Z ---
Done — removed completely (not used in any released checkpoint).

--- Comment by artem-spector at 2026-04-28T06:15:05Z ---
Done — `pad_token_id = self.config.text_config.pad_token_id if self.config.text_config.pad_token_id is not None else -1`.

--- Comment by artem-spector at 2026-04-28T06:15:07Z ---
Done — raises `ValueError`.

--- Comment by artem-spector at 2026-04-28T06:15:09Z ---
Done — removed the stale entries introduced by bad rebases.

--- Comment by artem-spector at 2026-04-28T06:15:10Z ---
Done — removed the `granite4_vision` entry from `conversion_mapping.py`.

--- Comment by artem-spector at 2026-04-28T06:15:12Z ---
Done — `Copyright 2026 IBM and The HuggingFace Team`.

--- Comment by artem-spector at 2026-04-28T06:15:14Z ---
Done — removed `test_image_processing_granite4_vision.py` entirely (processing is identical to LlavaNext, no re-definition needed).

--- Comment by artem-spector at 2026-04-28T06:15:33Z ---
Both paths are active in the released checkpoint. `granite-vision-4.1-4b/config.json` has `deepstack_layer_map: [[-19,9],[-13,6],[-7,3],[-1,0]]` (4 interpolate-downsampled injection points) **and** `use_spatial_sampling: true` (4 spatial-offset groups injected at LLM layers 12/15/18/21). Together they account for all 8 deepstack injection points described in the model card.

Both run on every forward pass, but on different vision layers and LLM targets. The `if self._spatial_offset is not None` branching is per-projector — each of the 8 `WindowQFormerDownsampler` instances uses either interpolate or spatial, determined at init time from the config. So there is no dead code.

--- Comment by artem-spector at 2026-04-28T06:15:43Z ---
Done — added `qformer_config` as a sub-config on `Granite4VisionConfig` (same pattern as `Blip2Config`). `WindowQFormerDownsampler` now reads it as `config.qformer_config`.

--- Comment by artem-spector at 2026-04-28T06:15:53Z ---
Done — added `interpolate_downsample(image_features, config)` and `spatial_offset_downsample(image_features, config, offset)` as module-level functions. `WindowQFormerDownsampler.forward` calls them directly.

--- Comment by artem-spector at 2026-04-28T06:16:03Z ---
We tried this but the modular converter crashes on `projector_hidden_act = AttributeError()` at class level — it tries to look up `projector_hidden_act` in the parent's `__init__` and hits a `KeyError`. We kept `del self.multi_modal_projector` in `__init__` instead. Happy to revisit if there's a supported way to express this.

--- Comment by artem-spector at 2026-04-28T06:16:14Z ---
`get_image_features` already has `@can_return_tuple` and returns `Granite4VisionImageFeaturesOutput` (a `ModelOutput` subclass). Did you mean a different method? The line number may have shifted — happy to check if you point me at the specific code.

--- Comment by artem-spector at 2026-04-28T06:16:26Z ---
Done — returns `Granite4VisionImageFeaturesOutput(ModelOutput)` with a `deepstack_features` field.

--- Comment by artem-spector at 2026-04-28T06:16:27Z ---
Done — replaced `get_image_token_mask` with `self.get_placeholder_mask(...)`.

--- Comment by artem-spector at 2026-04-28T06:16:29Z ---
Done — removed from the explicit signature of `Granite4VisionModel.forward`, now extracted via `kwargs.pop("output_attentions", None)` before forwarding to the language model.

--- Comment by artem-spector at 2026-04-28T06:16:31Z ---
Done — added `Granite4VisionTextModel` inheriting from `GraniteModel` via modular, with only `forward` overridden for deepstack injection.

--- Comment by artem-spector at 2026-04-28T06:16:46Z ---
Done — `Granite4VisionModel.forward` returns `Granite4VisionModelOutputWithPast` which includes a `deepstack_features` field. Note that `pooler_output` isn't applicable here — deepstack features are injected layer-by-layer inside `Granite4VisionTextModel.forward` (not merged at embedding time), so the features are consumed during the LLM forward pass rather than returned as a single tensor.

--- Comment by artem-spector at 2026-04-28T06:16:48Z ---
Done — removed all PEFT adapter-toggling and `merge_lora_adapters` logic. The released checkpoint already contains merged weights (base + adapter), so no runtime merging is needed.

--- Comment by artem-spector at 2026-04-28T06:16:49Z ---
Done — moved to `Granite4VisionTextModel.forward` which is defined via modular inheriting from `GraniteModel`.

--- Comment by zucchini-nlp at 2026-04-29T10:13:57Z ---
nit: `[...], device_map="auto")` without eval, shoudl be already in eval mode when loaded

--- Comment by zucchini-nlp at 2026-04-29T10:14:54Z ---
can delete this, already mapped to 'LlavaNext'

--- Comment by zucchini-nlp at 2026-04-29T10:17:28Z ---
ig you forgot to commit and push 😄

--- Comment by zucchini-nlp at 2026-04-29T10:19:16Z ---
i think same as `qwen3_vl.BaseModelOutputWithDeepstackFeatures`

--- Comment by zucchini-nlp at 2026-04-29T10:20:09Z ---
not sure I am following, are we not supposed to inherit from `GraniteConfig`? Current class has no attributes

--- Comment by zucchini-nlp at 2026-04-29T10:20:20Z ---
nit: commented dead code

--- Comment by zucchini-nlp at 2026-04-29T10:23:24Z ---
we can't import from other models like this. instead lets' do:

```
if self.qformer_config is None:
    self.qformer_config = CONFIG_MAPPING['blip_2_qformer']()
elif isinstance(self.qformer_config, dict):
    model_type = self.qformer_config.get('model_type', 'blip_2_qformer')
    self.qformer_config = CONFIG_MAPPING[model_type](**self.qformer_config)
```

--- Comment by zucchini-nlp at 2026-04-29T10:24:24Z ---
hmm, does the saved 'config.json'' have incorrect fields? Personally I don't like hidden magic and overriding fields at run-time dynamically

--- Comment by zucchini-nlp at 2026-04-29T10:26:11Z ---
nit: lets rename all one-latter vars in this file to anything longer/informative

--- Comment by zucchini-nlp at 2026-04-29T10:27:02Z ---
both fn needs config to infer original size. Can we pass the `original_size` as arg instead, and remove `config`?

--- Comment by zucchini-nlp at 2026-04-29T10:27:55Z ---
class naming needs to start with model name: `class Granite4VisionWindowQFormerDownsampler`

--- Comment by zucchini-nlp at 2026-04-29T10:28:57Z ---
same here about import, Instead we can:

`self.qformer = AutoModel.from_config(config.qformer_config)`

Prob we need to add an entry in 'modeling_auto' for Qformer, to make this work.

--- Comment by zucchini-nlp at 2026-04-29T10:30:05Z ---
can you rewrite it as `raise ValueError('message explaining why failed and what a user can do to fix it')`

--- Comment by zucchini-nlp at 2026-04-29T10:30:21Z ---
same here for error msg

--- Comment by zucchini-nlp at 2026-04-29T10:31:37Z ---
`super()` already does the same, no? Shouldn't be failing if we just delete this block

--- Comment by zucchini-nlp at 2026-04-29T10:31:48Z ---
dangling `pass`

--- Comment by zucchini-nlp at 2026-04-29T10:33:27Z ---
both attr have to be defined above, inside `Granite4VisionPreTrainedModel`

--- Comment by zucchini-nlp at 2026-04-29T10:34:19Z ---
need to delete these args, the method already will be decorated with `capture_outputs` when auto-generated

--- Comment by zucchini-nlp at 2026-04-29T10:40:38Z ---
this is also not needed, intermediate output are collected and added at the end automatically, by decorators

--- Comment by zucchini-nlp at 2026-04-29T10:41:10Z ---
`BaseModelOutputWithPast` no? I don't see anything special returned

--- Comment by zucchini-nlp at 2026-04-29T10:41:48Z ---
also should be defined in `PreTrainedModel` class

--- Comment by zucchini-nlp at 2026-04-29T10:43:14Z ---
oh indeed, modular added it. Can we add `**kwargs` not and pass it over to vision tower?

--- Comment by zucchini-nlp at 2026-04-29T10:44:37Z ---
amd here as well to delete, all these `output_xxx` and `return_dict` are handled by decorators. We don't explicitly add it anywhere, except for when calling `get_image_features(return_dict=True)`

--- Comment by zucchini-nlp at 2026-04-29T10:51:09Z ---
I wonder about this part: `inputs_embeds = inputs_embeds.masked_fill(vision_mask_3d, 0.0)`

So we have a huge block of zeros not filled with anything, is that expected or should we not add placeholders in that case? I saw it as scattering `image_features` last time, sorry

--- Comment by zucchini-nlp at 2026-04-29T10:51:45Z ---
same for output-xxx and return-dict/use-cache args, to delete all mentions and let it be consumed in kwargs

--- Comment by zucchini-nlp at 2026-04-29T10:53:21Z ---
the only diff from llava-next is returning `deepstack_features`?

--- Comment by zucchini-nlp at 2026-04-29T10:55:47Z ---
we should be able to delete this line to inti cache. Correct cache should be init by `generationMixin._prepare_cache_for_generation` automatically

--- Comment by zucchini-nlp at 2026-04-29T10:56:47Z ---
these two should be fixed when we add `**kwargs` and return a `BaseModelOutputWithDeepstackFeatures`

--- Comment by zucchini-nlp at 2026-04-29T10:57:39Z ---
these also should be fixable, I don't think this is a valid reason to skip

--- Comment by zucchini-nlp at 2026-04-29T10:58:36Z ---
if actually not used, lets just delete them from config class. When inheriting config, you can define

```
multimodal_projector_bias = AtributeErro()
```

and it will not copy this field

--- Comment by avihu111 at 2026-04-29T11:45:47Z ---
@zucchini-nlp good question! Happy to clarify this part:
During the LLM forward pass, in some decoder layers, we inject different visual features (sum into the hidden states) before different decoder layers (see `_deepstack_inject()` and lines ~448-449).
The zeros initialization of the visual features is followed by multiple injections to those locations (including before the first decoder layer).

--- Comment by zucchini-nlp at 2026-04-29T11:55:26Z ---
ahh I just saw that part, interesting design. Oke, thanks for clarifying, that makes sense then :)

so do we have image features for all hidden layers, or are some layers forwarded as 'zeros'?

--- Comment by avihu111 at 2026-04-29T15:08:44Z ---
the logic is pretty much:
```
# looping over LLM decoder layers ..
if layer_idx in llm_layer_idx_to_projected_features:
   hidden_states[visual_indices] += layer_idx_to_projected_features[layer_idx]
# running the decoder layer ..
```
so injection doesn't happen for all decoder layers, only a few, but all visual tokens are used (injections sum-in information into the existing visual tokens)

--- Comment by artem-spector at 2026-04-30T15:04:33Z ---
Done — removed `.eval()`.

--- Comment by artem-spector at 2026-04-30T15:04:39Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-30T15:04:45Z ---
Sorry about that — pushed now.

--- Comment by artem-spector at 2026-04-30T15:04:52Z ---
Done — `Granite4VisionImageFeaturesOutput` now inherits `BaseModelOutputWithPooling` (same approach as `qwen3_vl.BaseModelOutputWithDeepstackFeatures`), with a `deepstack_features` field added.

--- Comment by artem-spector at 2026-04-30T15:05:05Z ---
Done — `Granite4VisionTextConfig` inherits `GraniteConfig` directly. It has no additional attributes because the text config is fully specified by `GraniteConfig`; the subclass exists only to set `model_type = "granite4_vision_text"` and `base_config_key = "text_config"`.

--- Comment by artem-spector at 2026-04-30T15:05:12Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-30T15:05:19Z ---
Done — `__post_init__` now uses `CONFIG_MAPPING["blip_2_qformer"]` exactly as suggested.

--- Comment by artem-spector at 2026-04-30T15:05:27Z ---
The patching was removed. `qformer_config` is now fully constructed before `super().__post_init__()` runs, so no fields are mutated after the fact.

--- Comment by artem-spector at 2026-04-30T15:05:33Z ---
Done — all one-letter variables renamed to descriptive names.

--- Comment by artem-spector at 2026-04-30T15:05:41Z ---
Done — both `interpolate_downsample` and `spatial_offset_downsample` now take explicit `original_size` / `new_size` args rather than reading from config.

--- Comment by artem-spector at 2026-04-30T15:05:50Z ---
Done — renamed to `Granite4VisionWindowQFormerDownsampler`.

--- Comment by artem-spector at 2026-04-30T15:05:56Z ---
Done — `self.qformer = AutoModel.from_config(config.qformer_config)`.

--- Comment by artem-spector at 2026-04-30T15:06:03Z ---
Done — replaced with `raise ValueError(...)` with a descriptive message.

--- Comment by artem-spector at 2026-04-30T15:06:05Z ---
Done — same here.

--- Comment by artem-spector at 2026-04-30T15:06:16Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-30T15:06:18Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-30T15:06:24Z ---
Done — both `_can_record_outputs` and `_deepstack_inject` moved to `Granite4VisionPreTrainedModel`.

--- Comment by artem-spector at 2026-04-30T15:06:33Z ---
Done — removed from the explicit signature; handled by `@capture_outputs` and `**kwargs`.

--- Comment by artem-spector at 2026-04-30T15:06:40Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-30T15:06:48Z ---
Done — `Granite4VisionTextModel.forward` now returns `BaseModelOutputWithPast`.

--- Comment by artem-spector at 2026-04-30T15:06:50Z ---
Done — moved to `Granite4VisionPreTrainedModel`.

--- Comment by artem-spector at 2026-04-30T15:06:57Z ---
Done — `**kwargs` already present and forwarded to `self.vision_tower(...)`.

--- Comment by artem-spector at 2026-04-30T15:07:04Z ---
Done — removed; handled by `**kwargs`.

--- Comment by artem-spector at 2026-04-30T15:07:06Z ---
Done — removed.

--- Comment by artem-spector at 2026-04-30T15:07:15Z ---
Yes — the main differences from `LlavaNextForConditionalGeneration` are: (1) the model uses `Granite4VisionModel` which has deepstack injection, (2) logits are scaled by `text_config.logits_scaling`, and (3) `deepstack_features` is threaded through the output. Everything else is inherited.

--- Comment by artem-spector at 2026-04-30T15:07:30Z ---
Done — removed; `GenerationMixin._prepare_cache_for_generation` handles cache initialization.

--- Comment by artem-spector at 2026-04-30T15:07:39Z ---
Done — `Granite4VisionImageFeaturesOutput` now inherits `BaseModelOutputWithPooling` so all five tests pass. `skip_test_image_features_output_shape = True` remains because `last_hidden_state` isn't meaningful for this output type, but `hidden_states`, `pooler_output`, and field presence checks all pass.

--- Comment by artem-spector at 2026-04-30T15:07:51Z ---
Done for `test_training` — the skip was stale; `Granite4VisionForConditionalGeneration` computes a loss, and the framework skips `Granite4VisionModel` automatically via `MODEL_MAPPING_NAMES`.\n\n`test_can_init_all_missing_weights` remains skipped: `Blip2QFormerModel` submodules aren't initialized from meta device by our `_init_weights` — `Blip2QFormerPreTrainedModel._init_weights` only handles `Blip2ForConditionalGeneration` instances, not the standalone `Blip2QFormerModel`. Happy to investigate further if you'd like.

--- Comment by artem-spector at 2026-04-30T15:08:01Z ---
Done — `multimodal_projector_bias` and `projector_hidden_act` are shadowed with `AttributeError()` at class level on `Granite4VisionConfig`, so the framework sees them as intentionally inaccessible. The `SPECIAL_CASES_TO_ALLOW` entry was removed.

--- Comment by zucchini-nlp at 2026-05-04T13:28:58Z ---
this belong in `image_processing_auto.py`. Was it auto-generated or added manually?

--- Comment by zucchini-nlp at 2026-05-04T13:34:58Z ---
vision/text configs cannot be None, no? In other words, we create a default config if not passes, just like with qformer below. Then we won't need a magical `1152` hardcoded here

If that is about the order, you can call `super()` at the beginning and it will unwrap first. Also we can completely override `__post_init__` by calling `PreTrainedConfig.__post_init__(**kwargs)` instead of super

--- Comment by zucchini-nlp at 2026-05-04T13:38:32Z ---
ultra nit: for sake of readability I prefer to split into several lines. Same comment for `unwindowed_raster`

--- Comment by zucchini-nlp at 2026-05-04T13:40:13Z ---
didn't push commit? 😄  Same for LN and embed modules, calling super is usually enough unless model requires special weight init

--- Comment by zucchini-nlp at 2026-05-04T13:41:32Z ---
same comment, I think you have one local commit unpushed

--- Comment by zucchini-nlp at 2026-05-04T13:42:19Z ---
let's move it under `TextModel` since it is not needed for other classes

--- Comment by zucchini-nlp at 2026-05-04T13:42:33Z ---
?

--- Comment by zucchini-nlp at 2026-05-04T13:43:28Z ---
btw, forgot to ask prev, do we have spatial sampling in all released weights? If yes, we don't need to init under condition

--- Comment by zucchini-nlp at 2026-05-04T13:46:37Z ---
we can also delete these and decorate `merge_defaults_with_config`. Also you can delete the decorator in L669 and it will be copied from llava by modular

--- Comment by zucchini-nlp at 2026-05-04T13:47:18Z ---
bad copy from llava-next, checking `size(0)` shoulnd't be needed

--- Comment by zucchini-nlp at 2026-05-04T13:48:19Z ---
i see, could you add a small inline comment that model relies only on `deepstack_features` and assigning `0` value is on purpose

--- Comment by zucchini-nlp at 2026-05-04T13:49:23Z ---
same here re decorarots and `x if x else self.x`

--- Comment by zucchini-nlp at 2026-05-04T13:51:01Z ---
looks weird, is that smth from `modular` or simply calling `self.language_model` has an overhead? Do you have an intuition what could be reason, I don't think we can keep it this way

--- Comment by artem-spector at 2026-05-05T07:07:17Z ---
Done — removed. Not sure why it was added, but `image_processing_auto.py` already has it. Duplicate deleted.

--- Comment by artem-spector at 2026-05-05T07:07:23Z ---
Done — split into separate assignment lines in both methods.

--- Comment by artem-spector at 2026-05-05T07:07:30Z ---
Done — moved to `Granite4VisionTextModel`.

--- Comment by artem-spector at 2026-05-05T07:07:37Z ---
These branches are needed — `test_can_init_all_missing_weights` fails without them. The parent `LlavaNextPreTrainedModel._init_weights` doesn't handle `nn.Embedding`, `nn.LayerNorm`, or RMSNorm, so without these branches meta-device initialization leaves them with garbage values. Keeping them.

--- Comment by artem-spector at 2026-05-05T07:07:43Z ---
`_can_record_outputs` is on `Granite4VisionPreTrainedModel` - as requested.

--- Comment by artem-spector at 2026-05-05T07:07:45Z ---
`_can_record_outputs` is on `Granite4VisionPreTrainedModel` - as requested.

--- Comment by artem-spector at 2026-05-05T07:07:51Z ---
Yes — `use_spatial_sampling: true` in the released config. Removed the conditional; `spatial_projectors` is now always initialized.

--- Comment by artem-spector at 2026-05-05T07:07:57Z ---
Done — removed.

--- Comment by artem-spector at 2026-05-05T07:08:03Z ---
Done — added inline comment.

--- Comment by artem-spector at 2026-05-05T07:08:09Z ---
Done — added `@merge_with_config_defaults`, removed manual fallback blocks. Kept `@can_return_tuple` explicitly since we override the full `forward` body.

--- Comment by artem-spector at 2026-05-05T07:08:14Z ---
Done.

--- Comment by artem-spector at 2026-05-05T07:08:21Z ---
Done — call `super().__post_init__(**kwargs)` first so `vision_config` is already a typed object when we build `qformer_config`.

--- Comment by artem-spector at 2026-05-05T07:08:26Z ---
Good catch — this was added when chasing a performance bug that turned out to be noise. Replaced with a normal `self.language_model(...)` call.

--- Comment by zucchini-nlp at 2026-05-05T07:37:23Z ---
not really what I meant. Vision config has to be a config obj by this moment, so you only call `vision_hidden_size = self.vision_config.hidden_size`

Ordering correctly should do, move viision config code block before and that is all

--- Comment by zucchini-nlp at 2026-05-05T07:38:28Z ---
I mean `base_model_prefix` and `_no_split_modules`. We can define all cls attr once in `Granite4VisionPreTrainedModel`

Children classes just inherit whatever they might need

--- Comment by zucchini-nlp at 2026-05-05T07:39:41Z ---
lets delete all, so it is copied completely. Current state is missing auto docstring

--- Comment by artem-spector at 2026-05-05T11:01:54Z ---
Done — three clean blocks: dict→typed conversion before `super()`, `super().__post_init__` deserializes `vision_config`, then default `qformer_config` built from `self.vision_config.hidden_size`. No fallbacks or hardcoded values.

--- Comment by artem-spector at 2026-05-05T11:01:56Z ---
Done — moved both to `Granite4VisionPreTrainedModel`.

--- Comment by artem-spector at 2026-05-05T11:01:58Z ---
The `get_image_features` body is entirely custom (deepstack multi-layer extraction + spatial projectors, different return type) so the method needs to stay in modular — deleting it would copy the LlavaNext version and break the model. Added `@auto_docstring` and `@can_return_tuple`.

--- Comment by vasqu at 2026-05-05T12:20:30Z ---
```suggestion
```
not on you but since we started to only support torch either way, that badge doesn't make much sense anymore

--- Comment by vasqu at 2026-05-05T12:22:26Z ---
Shouldn't this be the default in the configs itself? I think you can just add `"padding_side": "left"` to the tokenizer json and it should always adopt left padding then

--- Comment by vasqu at 2026-05-05T12:28:15Z ---
```suggestion
model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
```
otherwise we will (always) run on cpu here

--- Comment by vasqu at 2026-05-05T12:29:43Z ---
```suggestion
# Copyright 2026 IBM. All rights reserved.
```

--- Comment by vasqu at 2026-05-05T12:35:47Z ---
```suggestion
```
I think we need to remove the decorator (so modular inherits everything) or add auto docstring itself so
```
@auto_docstring
@dataclass
```
(the order matters)

Same for the output classes below

--- Comment by vasqu at 2026-05-05T12:36:54Z ---
We have 3 different descriptions, could we sync them?

--- Comment by vasqu at 2026-05-05T12:40:42Z ---
Why not the super call on top? A bit weird to have it in the middle

--- Comment by vasqu at 2026-05-05T12:43:29Z ---
would change the order here again to have the super at the end

--- Comment by vasqu at 2026-05-05T12:44:25Z ---
Let's highlight this difference with a small comment

--- Comment by vasqu at 2026-05-05T12:53:48Z ---
imo no need to have them private

--- Comment by vasqu at 2026-05-05T12:54:33Z ---
this is a bit weird, why did we not save them as a list of ints directly?

--- Comment by vasqu at 2026-05-05T12:55:37Z ---
```suggestion
    def forward(self, image_features: torch.Tensor) -> torch.Tensor:
```

--- Comment by vasqu at 2026-05-05T13:01:24Z ---
Ah ok I see that you use Fraction below. Tbh, might be a dumb question but is it torch-compile friendly?

--- Comment by vasqu at 2026-05-05T13:02:05Z ---
lets avoid the abbreviations, e.g. `interp`, `enc`

--- Comment by vasqu at 2026-05-05T13:03:11Z ---
yea here same re abbreviations, just to avoid confusion doesn't hurt to expand w -> window (same below)

--- Comment by vasqu at 2026-05-05T13:05:07Z ---
```suggestion
```
shouldn't be needed. Would it make sense to pass kwargs here as well?

--- Comment by vasqu at 2026-05-05T13:05:39Z ---
```suggestion
```

--- Comment by vasqu at 2026-05-05T13:08:46Z ---
We need to add a super call to `LlavaNextPreTrainedModel` because we shouldn't need most of this as @zucchini-nlp mentioned. This fails because llava didnt have the super call. Might be a good new TRF rule to have a super call in any pretrained model class 🤔 wdyt Raushan?

--- Comment by vasqu at 2026-05-05T13:10:35Z ---
Hmm, imo this is small enough that we can keep it in the main forward

--- Comment by vasqu at 2026-05-05T13:11:04Z ---
```suggestion
```
let modular inherit the decorators

--- Comment by vasqu at 2026-05-05T13:13:40Z ---
```suggestion
        deepstack_features: dict[int, torch.Tensor] | None = None,
```

--- Comment by vasqu at 2026-05-05T13:18:12Z ---
```suggestion
    @merge_with_config_defaults
    @can_return_tuple
```
we need to add this for the vision_feature args - then you don't need the manual prep within

--- Comment by vasqu at 2026-05-05T13:19:10Z ---
Can we add a comment where the actual difference happens and highlight it?

--- Comment by vasqu at 2026-05-05T13:24:26Z ---
Let's follow llava next more closely and avoid all these manual checks, it was due to the missing decorator

--- Comment by vasqu at 2026-05-05T13:25:54Z ---
Yea this should be done the same as in llava next imo, not with the manual checks but keeping in the signature to ignore

--- Comment by vasqu at 2026-05-05T13:27:36Z ---
This feels like it should be maybe its own module, the forward get really clustered here

--- Comment by vasqu at 2026-05-05T13:28:51Z ---
Maybe we could use `filter_output_hidden_states` from backbone utils cc @zucchini-nlp

--- Comment by vasqu at 2026-05-05T13:29:26Z ---
```suggestion
```
let modular handle the decorators

--- Comment by vasqu at 2026-05-05T13:31:24Z ---
```suggestion
```
really needed?

--- Comment by vasqu at 2026-05-05T13:31:33Z ---
```suggestion
```

--- Comment by vasqu at 2026-05-05T13:33:50Z ---
```suggestion
        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
        logits = self.lm_head(hidden_states[:, slice_indices, :])
        logits = logits / self.config.text_config.logits_scaling

        loss = None
        if labels is not None:
            loss = self.loss_function(
                logits=logits, labels=labels, vocab_size=self.config.text_config.vocab_size, **kwargs
            )
```
let's follow the original more closely, the current way doesnt really make use of logits to keep because we calculate everything either way

--- Comment by vasqu at 2026-05-05T13:35:24Z ---
`test_all_params_have_gradient=False` instead - I assume the text backbone is a MoE or similar? That can easily cause this

--- Comment by vasqu at 2026-05-05T13:38:28Z ---
please use `url_to_local`
1. https://github.com/huggingface/transformers/blob/2c432d73c1428640163c0ff3dc29f2fa4890b928/utils/fetch_hub_objects_for_ci.py#L40 (check if it is already there re url)
2. Use `from ...test_processing_common import url_to_local_path` on the url which resolves it to a cache path (if available else normal url)

--- Comment by vasqu at 2026-05-05T13:38:38Z ---
```suggestion
# Copyright 2026 IBM. All rights reserved.
```

--- Comment by zucchini-nlp at 2026-05-05T14:35:52Z ---
guess we need to delete it in docs overall, will check if we have it anywhere still

--- Comment by zucchini-nlp at 2026-05-05T14:39:07Z ---
forcing dict is needed tbh, in case the config was saved with `return_dict=False`

Re kwargs, not much of a use imo, since we don't return qformer output anywhere

--- Comment by zucchini-nlp at 2026-05-05T14:45:59Z ---
yep, hybrid granite backbone with linear attention

--- Comment by zucchini-nlp at 2026-05-05T14:50:14Z ---
I will leave it to IBM team as we don't have much control over hub-configs

--- Comment by zucchini-nlp at 2026-05-05T14:58:47Z ---
> `Overrides the parent to apply downsample_rate to height/width calculations.`

It has this in docsring, imo that explains it

--- Comment by zucchini-nlp at 2026-05-05T14:59:48Z ---
lets keep it for now, to align with released weights. We can move to a new modulr with conversion mapping later, if needed


fix failed test cases for blt model

PULL REQUEST #45596: fix failed test cases for blt model
State: closed  |  Merged: True
Author: kaixuanliu  |  Base: main
Labels:
Created: 2026-04-23T06:41:09Z

@ydshieh pls help review, thx!

--- Comment by ydshieh at 2026-04-28T14:57:41Z ---
run-slow: blt

--- Comment by github-actions[bot] at 2026-04-28T14:59:21Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25060431508)

This comment contains `run-slow`, running the specified jobs:

models: ["models/blt"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-28T15:14:47Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25060431508)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [2cd36be9](https://github.com/huggingface/transformers/commit/2cd36be9e9301a835e9970d0cfccd8d50063412f) | workflow commit (merge commit) |
| PR | [95a47816](https://github.com/kaixuanliu/transformers/commit/95a478167cdc9a9e3e6c2c5afe97e061de44c7a5) | branch commit (from PR) |
| main | [31469146](https://github.com/huggingface/transformers/commit/3146914673a4ebff2f5e06ace68fbddacdebe90b) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by ydshieh at 2026-04-28T15:33:05Z ---
I update some expected values. @kaixuanliu You might want to run again with XPU, and update with `Expectation` class if necessary 🙏

--- Comment by ydshieh at 2026-04-28T16:23:04Z ---
run-slow: blt

--- Comment by github-actions[bot] at 2026-04-28T16:24:39Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25064787642)

This comment contains `run-slow`, running the specified jobs:

models: ["models/blt"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-28T16:32:56Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25064787642)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [9325696e](https://github.com/huggingface/transformers/commit/9325696e27e48f61d3e5daaa969a25c53bf38d76) | workflow commit (merge commit) |
| PR | [9a394f6b](https://github.com/kaixuanliu/transformers/commit/9a394f6b0083a706490d77c8ed71cf441e16da51) | branch commit (from PR) |
| main | [4aba7167](https://github.com/huggingface/transformers/commit/4aba7167e328965caadcdfc6834b982037889f86) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by ydshieh at 2026-04-28T17:16:46Z ---
run-slow: blt

--- Comment by github-actions[bot] at 2026-04-28T17:18:05Z ---
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25067330471)

This comment contains `run-slow`, running the specified jobs:

models: ["models/blt"]
quantizations: []

--- Comment by github-actions[bot] at 2026-04-28T17:24:20Z ---
## CI Results
[Workflow Run ⚙](https://github.com/huggingface/transformers/actions/runs/25067330471)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [dc34d2f6](https://github.com/huggingface/transformers/commit/dc34d2f629243b1f9dcd3cbc72569b3eb29fa65e) | workflow commit (merge commit) |
| PR | [dc624a04](https://github.com/kaixuanliu/transformers/commit/dc624a04988c3cdb30c66e35718821d26d2e1431) | branch commit (from PR) |
| main | [f3d2b68b](https://github.com/huggingface/transformers/commit/f3d2b68b619cc2960146ac74a8cc6e9c9834bc9e) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !

--- Comment by github-actions[bot] at 2026-04-29T02:46:06Z ---
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: blt

--- Comment by kaixuanliu at 2026-04-29T03:10:16Z ---
Thx @ydshieh , I submitted another commit, and now all cases for blt can pass on XPU.

--- Comment by ydshieh at 2026-04-28T16:18:52Z ---
it's not clear why self.patcher doesn't handle the input device well.

--- Comment by ydshieh at 2026-04-28T16:26:52Z ---
Likely the BltPatcher itself is BltPreTrainedModel