ISSUE #43937: [GLM-5] ValueError: GenerationConfig is invalid
State: closed  |  Labels: bug
Author: xin3he  |  Created: 2026-02-12T08:41:21Z

### System Info

transformers 5.2.0.dev0

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Saving GLM-5 returns ValueError: GenerationConfig is invalid.

To reproduce quickly:
1. Download this model https://huggingface.co/tiny-random/glm-4.7-flash
2. Replace generation_config.json with https://huggingface.co/zai-org/GLM-5/blob/main/generation_config.json
3. load the model and then save with `save_pretrianed`, Error will happen since do_sample is not set in this generation_config.json

### Expected behavior

It would be nice to automatically handle this case and keep the original generation_config.json when saving.

--- Comment by xin3he at 2026-02-12T08:41:57Z ---
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 3245, in save_pretrained
    model_to_save.generation_config.save_pretrained(save_directory)
  File "/usr/local/lib/python3.12/dist-packages/transformers/generation/configuration_utils.py", line 785, in save_pretrained
    raise ValueError(str(exc) + "\n\nFix these issues to save the configuration.")
ValueError: GenerationConfig is invalid:
- `top_p`: `do_sample` is set not to set `True`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
If you're using a pretrained model, note that some of these attributes may be set through the model's `generation_config.json` file.
```

--- Comment by zucchini-nlp at 2026-02-12T11:03:04Z ---
The generation config is indeed not valid, as the message says. We can't have `top_p` set to any value except for `1` when sampling is turned off because top-p will not take effect without sampling

The only way is to update the generation config before saving and set `do_sample=True` or `top_p = 1.0`. Could you also open a PR in model repo page so the authors can fix it? No idea how the config was saved in the first place, we've had the validation for a while now and it should have errored out, unless the config was created manually

--- Comment by zRzRzRzRzRzRzR at 2026-02-12T14:03:00Z ---
Oh, is do_sample set to false now?

--- Comment by zucchini-nlp at 2026-02-12T14:19:56Z ---
By default it is `False`, yes. Tbh it was `False` in v4 as well :)

--- Comment by zRzRzRzRzRzRzR at 2026-02-13T09:12:58Z ---
Note: the GLM-5 implementation in Transformers is not fully complete yet. Some remaining work is still in progress in this [PR](https://github.com/huggingface/transformers/pull/43912).

If you need to run inference in the meantime, please refer to the sglang implementation first.

--- Comment by ArthurZucker at 2026-02-13T14:18:15Z ---
Yep really sorry everyone !

--- Comment by stargazerwh at 2026-02-21T14:34:30Z ---
After loading the model, proactively fill in the missing do_sample field in the generation_config, and then save the model.

--- Comment by zucchini-nlp at 2026-02-23T09:41:29Z ---
The model was merged, and I believe that glm team will fix the config on the hub. Closing as it's not really a bug in `transformers`


ISSUE #43935: Add `eval_on_end` flag (analogous to `eval_on_start`)
State: closed  |  Labels: Feature request
Author: MarkusSpanring  |  Created: 2026-02-12T08:11:18Z

### Feature request

### Feature request

#### Background
There is already a convenient, switch to evaluate **before** training starts: `eval_on_start=True`.

There’s a symmetric need at the other end of training: evaluate **after** training finishes, regardless of whether the last `global_step` lands exactly on an `eval_steps` boundary.

This has been a recurring pain point in the context of:

- **Issue #28539**: `load_best_model_at_end` can be inconsistent with evaluation/save behavior at the end of training
  https://github.com/huggingface/transformers/issues/28539
- **PR #30160**: improves end-of-run behavior so the model can be saved at the last step
  https://github.com/huggingface/transformers/pull/30160

However, even if the model is saved on the last step, it would also be great to have an option to **evaluate at the end** by setting a flag and use it for `load_best_model_at_end`.

---

#### Proposal
Add a new boolean argument to `TrainingArguments`:

- `eval_on_end: bool = False`

---

#### Expected behavior (should work like `eval_on_start`)

When `eval_on_end=True`:
1. `Trainer` runs one final evaluation after training completes (same `eval_dataset`, same `compute_metrics`, same logging).
2. It runs **even if** the final `global_step` is not divisible by `eval_steps`.
3. It integrates cleanly with the existing evaluation strategies:
   - `eval_strategy="steps"`: periodic evaluation continues as usual, plus a final one at end.
   - `eval_strategy="epoch"`: end-of-epoch eval already happens; `eval_on_end` can be a no-op if the last action already evaluated (or can still run once more—either way should be well-defined).

#### Interaction with `load_best_model_at_end`
Goal: `eval_on_end=True` should be compatible with `load_best_model_at_end=True` so that the final evaluation is not silently excluded from best-model selection—especially now that end-of-run save behavior is being improved via PR #30160.

### Motivation

#### Why this is needed (vs callbacks)
Users can implement this with a callback today, but `eval_on_start` demonstrates that Hugging Face is willing to provide “one flag, no boilerplate” for common evaluation flow control. `eval_on_end` is the natural counterpart.

### Your contribution

The complexity of the feature seems rather low however I would still need initial pointers for a PR on how to best integrate it.

--- Comment by Rocketknight1 at 2026-02-12T14:09:00Z ---
cc @sunmarc maybe?

--- Comment by SunMarc at 2026-02-13T14:55:48Z ---
Since we are saving the checkpoint for the last step when saving_strategy="step", I don't mind enforcing by default that we also evaluate the last checkpoint. cc @winglian wdyt ?

--- Comment by khushali9 at 2026-02-18T23:03:09Z ---
seems like a fair ask, let me know wdyt @winglian and I can take this up and make a quick PR out. cc @SunMarc @Rocketknight1

I’m thinking of implementing this by forcing a final call to _maybe_log_save_evaluate rather than manually invoking _evaluate and _determine_best_metric, so that callback, logging, and best-model logic remain centralized. Does that sound good?

--- Comment by khushali9 at 2026-02-19T05:57:36Z ---
created a quick PR, let me know what you think, will add some test cases soon. Thanks

--- Comment by mvanhorn at 2026-03-09T15:03:34Z ---
I've submitted an implementation in #44548. Added `eval_on_end` to `TrainingArguments` and the corresponding logic in `Trainer.train()`, mirroring the existing `eval_on_start` pattern.


ISSUE #43931: Model loading error: weight shapes mismatch of Qwen3-VL-30B-A3B-Instruct
State: closed  |  Labels: bug
Author: jonbakerfish  |  Created: 2026-02-12T02:24:57Z

### System Info

- `transformers` version: 5.1.0
- Platform: Linux-5.15.0-168-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 1.4.1
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0 (CUDA)
- Using distributed or parallel set-up in script?: single node multi-GPUs
- Using GPU in script?: yes
- GPU type: NVIDIA GeForce RTX 4090

### Who can help?

 @zucchini-nlp @CyrilVallez

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi,

I'm running the `transformers serving`, when I use the `Qwen/Qwen3-VL-30B-A3B-Instruct` model to chat with, it outputs the weight shapes mismatch erros.

### Expected behavior

```log
Qwen3VLMoeForConditionalGeneration LOAD REPORT from: Qwen/Qwen3-VL-30B-A3B-Instruct
Key                                                           | Status   |
--------------------------------------------------------------+----------+---------------------------------------------------------------------------------------------------------
model.language_model.layers.{0...47}.mlp.experts.gate_up_proj | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([128, 2048, 1536]) vs model:torch.Size([128, 1536, 2048])
model.language_model.layers.{0...47}.mlp.experts.down_proj    | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([128, 768, 2048]) vs model:torch.Size([128, 2048, 768])

Notes:
- MISMATCH      :ckpt weights were loaded, but they did not match the original empty weight shapes.
```

--- Comment by Rocketknight1 at 2026-02-12T13:52:02Z ---
Can you paste the exact command you used?

--- Comment by zucchini-nlp at 2026-02-12T13:55:08Z ---
I reproduced it by simply loading the model. Maybe smth changed recently in our model impl, didn't have time to investigate. Will take a look later, but seems to be affected by https://github.com/huggingface/transformers/pull/42697

--- Comment by zucchini-nlp at 2026-02-12T14:29:34Z ---
Oke, I think we forgot to revert transpose in conversion mapping after standardizing experts in Qwen3VLMoE, the conversion should be same as in qwen2-moe or other qwen models. Will open a PR shortly

--- Comment by Cyrilvallez at 2026-02-16T17:50:13Z ---
Was fixed in https://github.com/huggingface/transformers/pull/44037!


ISSUE #43927: [BUG] DiaConfig loses custom token IDs after save / load and causes IndexError during generation
State: closed  |  Labels: bug
Author: harshaljanjani  |  Created: 2026-02-11T19:57:48Z

### System Info

* `transformers` version: `5.0.0.dev0`
* Platform: `Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39`
* Python version: `3.12.3`
* `huggingface_hub` version: `1.3.2`
* `safetensors` version: `0.7.0`
* `accelerate` version: `1.12.0`
* Accelerate config: `not installed`
* DeepSpeed version: `not installed`
* PyTorch version (accelerator?): `2.9.1+cu128 (CUDA)`
* GPU type: `NVIDIA L4`
* NVIDIA driver version: `550.90.07`
* CUDA version: `12.4`

### Who can help?

@Rocketknight1

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import tempfile
from transformers import DiaConfig

config = DiaConfig(eos_token_id=97, pad_token_id=98, bos_token_id=99)
print("Before save eos_token_id:", config.decoder_config.eos_token_id)
with tempfile.TemporaryDirectory() as tmpdir:
  config.save_pretrained(tmpdir)
  loaded_config = DiaConfig.from_pretrained(tmpdir)
  print("After load eos_token_id:", loaded_config.decoder_config.eos_token_id)
# Expected: 97, Actual: 1024
```

[DiaConfig](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dia/configuration_dia.py#L267-L275) sets `eos_token_id`, `pad_token_id`, and `bos_token_id` only on the `decoder_config` sub-config, but not as direct attrs of `DiaConfig`. When the config is saved and reloaded, these values are reset to defaults (1024, 1025, 1026) breaking custom token ID tests with smaller vocabularies.

**Current Output:**

### Expected behavior

→ Custom token IDs should be preserved across save and load.
→ `test_modeling_dia.py::DiaModelTest::test_eager_matches_sdpa_generate` should pass without `IndexError`

**Output After the Fix:**

--- Comment by zucchini-nlp at 2026-02-12T08:29:37Z ---
Indeed, and imo it is not a good practice to allow passing encoder/decoder fields to the main config and force-set it. I've seen this pattern in a few other multimodal models iirc, so we might want to fix it there as well

Taking a look at your PR 👁


Add LFM2.5 Audio 1.5B

ISSUE #43909: Add LFM2.5 Audio 1.5B
State: open  |  Labels: New model, Audio
Author: MHRDYN7  |  Created: 2026-02-11T09:20:30Z

### Model description

I would like to add LFM2.5 Audio, which is a highly versatile multimodal audio-text model for its small size of 1.5b. Just ran the model on cpu and it's really good. Should be a good candidate for transformers integration. @eustlb @ebezzam

The modular implementation should not be very long since the original repo directly uses lfm2 text model from transformers and they use mimi from the original moshi repo (should be possible to use the transformer native module). They use fastconformer for the encoder (slightly modified scripts from Nemo), but I'm not very sure if there's a module for that in transformers (parakeet might have it).

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

authors: @haerski, @ArthurBook
[original repo](https://github.com/Liquid4All/liquid-audio)
[weights](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B)

If permitted, I'll go ahead and open a PR.

--- Comment by eustlb at 2026-02-11T13:17:07Z ---
Hey @MHRDYN7, it is indeed a really good candidate for transformer integration.
You should indeed be able to use Parakeet's encoder for the fastconformer encoder and Mimi as an audio_tokenizer directly in the processor. Happy to help along the process

I did a breakdown of its architecture [here](https://x.com/eustachelb/status/1974129698960453847) it that helps!

--- Comment by MHRDYN7 at 2026-02-11T13:18:50Z ---
Alright. Didn't know you had tweeted, should be very helpful.

--- Comment by haerski at 2026-02-11T15:10:21Z ---
Hey, author here! Thanks for the interest! This is certainly something we considered when releasing the model, but it seemed rather complicated due to the custom generation routines that didn't play nicely with the `transformers` API. In particular the vocabulary size and token output dimensionality can change during generation, which caused some issues with the standard `LogitsProcessor`s. In addition, one might want to specify different generation parameters for audio and text generation (both of which may happen in a single turn).

While a text-only implementation would likely be straightforward to implement, we felt that shipping a feature-limited version within `transformers` would not provide a great user experience.

Ultimately, we decided it would be more maintainable to develop and support a lightweight, dedicated library (`liquid-audio`), along with integration in llama.cpp. This approach is fairly common for audio models, for example Qwen-TTS, Moshi, and NVIDIA NeMo audio models follow a similar pattern.

That said, I’d be happy to review a pull request if @MHRDYN7, @eustlb, or anyone else decides to explore a `transformers` integration!

--- Comment by eustlb at 2026-02-16T14:21:17Z ---
Hey @haerski, thanks a lot for the detailed info, it totally makes sense.

Multimodality, especially for audio, is a major area of focus for the future of `transformers`, and we’re working toward making sure those refiners disappear as much as possible. There’s strong demand from the community to have native support for these models (eg they’re already familiar with the libs paradigm, for finetuning etc)

For future releases, please don’t hesitate to reach out to us (we have a dedicated slack collaboration channel 😉) so we can discuss these points early on. For the reasons explained above, we would even usually handle the PR ourselves ahead of the release to ensure day-0 support in the library, and adapt it to your needs.


ISSUE #43908: What is the process of adding a new hardware backend for Trainer?
State: closed  |  Labels:
Author: quic-meetkuma  |  Created: 2026-02-11T08:32:48Z

I work in Qualcomm and we have a hardware backend like cuda / mps. I want to add it in the Trainer so that we can use Trainer class to perform training on our stack. What is the process of adding it?

We have a branch which currently adds the backend specific changes: https://github.com/quic-meetkuma/transformers/tree/qaic_support

Please guide on this. Will it be fine to directly raise a PR or there will be subsequent steps that I should be taking care of?

--- Comment by Rocketknight1 at 2026-02-11T12:52:35Z ---
cc @sunmarc

--- Comment by SunMarc at 2026-02-11T13:20:45Z ---
Hey @quic-meetkuma ! In order to add a new hardware backend, you need to add support first in accelerate then transformers. For example, you can check the MUSA integration https://github.com/huggingface/accelerate/pull/2917 and https://github.com/huggingface/transformers/pull/31913

--- Comment by quic-meetkuma at 2026-02-11T13:52:21Z ---
Hey @SunMarc
Thanks for quick response. We already have added our backend in one of our accelerate forks. We will plan the to raise PR for accelerate repo as well.

Ref: https://github.com/quic-meetkuma/accelerate/tree/v1.12.0-release-shubham-changes

Apart from raising PR on transformer and accelerate, anything else I should be taking care of?

--- Comment by SunMarc at 2026-02-11T14:56:20Z ---
No that should be it !

--- Comment by github-actions[bot] at 2026-03-14T08:04:25Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43906: Isolated reproduction of https://github.com/huggingface/transformers/issues/38071
State: closed  |  Labels: Good First Issue, bug
Author: willxxy  |  Created: 2026-02-11T08:16:24Z

### System Info

name = "accelerate"
version = "1.12.0"
name = "transformers"
version = "4.57.3"

Python 3.11

### Who can help?

@gante  @ArthurZucker  Related to warning from https://github.com/huggingface/transformers/issues/38071 for `Qwen/Qwen3-Next-80B-A3B-Instruct` model

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import pipeline
from tqdm import tqdm
LANGUAGE_MODEL = "Qwen/Qwen3-Next-80B-A3B-Instruct"
def main():

    pipe = pipeline("text-generation", model=LANGUAGE_MODEL, device_map="auto")
    messages = [[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hdi"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hddi"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hasdasi"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hiasdsad"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hiasd"},
    ],[
        {"role": "system", "content": "hi"},
        {"role": "user", "content": "hiasd"},
    ]]

    BATCH_SIZE = 3
    for out in tqdm(
        pipe(messages, max_new_tokens=4, batch_size=BATCH_SIZE),
        total=len(messages),
        desc="Batched inference",
    ):
        response = out[0]["generated_text"][-1]["content"].strip()
        print(response)

if __name__ == "__main__":
    main()
```

Adding

```
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct", padding = "left")
pipe = pipeline("text-generation", model=LANGUAGE_MODEL, tokenizer=tokenizer, device_map="auto")
```
Does not make the warning go away.

### Expected behavior

```
(ecg-preprocess) (ecg-encoder) -bash-4.4$ CUDA_VISIBLE_DEVICES=4,5,6,7 uv run src/test.py
The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Loading checkpoint shards: 100%|█████████████████| 41/41 [00:31<00:00,  1.28it/s]
Device set to use cuda:0
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
Batched inference:   0%|                                   | 0/6 [00:00<?, ?it/s])
Hi!
Hi! It looks
Hello! It seems
Hello! It seems
)
) I'm here
Batched inference: 100%|████████████████████████| 6/6 [00:00<00:00, 82782.32it/s]
```

--- Comment by ArthurZucker at 2026-02-11T10:36:31Z ---
@willxxy this should be "easy" to fix, do you want to open a PR ?

--- Comment by ChiragSW at 2026-02-11T10:54:20Z ---
Hi @willxxy there seems to be a common mistake.
while adding this:
```
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct", padding = "left")
```

use this instead:

```
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct", padding_side = "left")
```

please let me know if this fixes the issue.

--- Comment by Jignesh03-T at 2026-02-12T13:41:55Z ---
Hi, I would like to work on this issue.
Can I try to reproduce and submit a fix?

--- Comment by nguyencongtuyenlp at 2026-02-12T17:07:21Z ---
Hi! I can take this. I’ll open a PR that (1) clarifies the warning to use padding_side="left" and (2) ensures the text-generation pipeline uses left-padding for decoder-only models in batched generation (with a regression test).

--- Comment by ManasVardhan at 2026-02-15T21:46:08Z ---
I've opened a PR to fix this: #44021

**Root cause:** Two issues combine to produce this warning:

1. `TextGenerationPipeline` doesn't set `padding_side='left'` for decoder-only models, so the tokenizer's default `'right'` padding is used during batch collation — meaning shorter sequences get right-padded, which is incorrect for decoder-only generation.

2. The right-padding detection in `generate()` uses a heuristic that checks if the last token equals `pad_token_id`. For Qwen3, `pad_token_id == bos_token_id == 151643` (`<|endoftext|>`), which can cause false positives even in non-padded scenarios.

**Fix:**
- Set `padding_side='left'` automatically in `TextGenerationPipeline.__init__`
- Use `attention_mask` (when available) for more reliable right-padding detection in `generate()`

Note: Setting `padding='left'` (as mentioned in the issue) doesn't work because the correct parameter is `padding_side='left'` — but even with the correct parameter, the pipeline overrides this during collation, so fixing it at the pipeline level is the proper solution.


ISSUE #43901: TextClassificationPipeline docs still mention return_all_scores, but behavior differs
State: closed  |  Labels: bug
Author: math-hiyoko  |  Created: 2026-02-11T01:34:44Z

### System Info

- `transformers` version: 5.2.0.dev0
- Platform: macOS-26.2-arm64-arm-64bit-Mach-O
- Python version: 3.14.3
- Huggingface_hub version: 1.4.1
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0 (NA)
- Using distributed or parallel set-up in script?: parallel set-up

### Who can help?

@Rocketknight1 @stevhliu

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

The documentation for TextClassificationPipeline still mentions the return_all_scores argument.
https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.TextClassificationPipeline.return_all_scores
However, return_all_scores has already been removed, and the current pipeline behavior does not match the docs.
As a result, users may expect return_all_scores=True to return all label scores, but it currently returns only the top label.
```python
>>> from transformers import pipeline
>>>
>>> classifier = pipeline(model="distilbert/distilbert-base-uncased-finetuned-sst-2-english", return_all_scores=True)
>>> classifier("This movie is disgustingly good !")
[{'label': 'POSITIVE', 'score': 0.9998536109924316}]
```

### Expected behavior

The mention of return_all_scores should be removed.


ISSUE #43899: `sync_each_batch` has no effect when using FSDP
State: closed  |  Labels:
Author: ojh31  |  Created: 2026-02-10T23:41:28Z

I can corroborate the finding of @zch0414 below that there is no way to configure the trainer to force sync when using FSDP. As explained [here](https://huggingface.co/docs/accelerate/en/concept_guides/gradient_synchronization#nosync-requires-additional-gpu-memory-when-using-fsdp), this is a big problem for memory intensive workloads. I include a comparison below for llama 70b, FSDP1 across 8 gpus, 4bit frozen params, lora rank 256, PDTBS=1, GAS=2, with and without the monkeypatch, showing that forcing sync leads to a significant reduction in the memory spike from the first backward pass.

> Does this still work? I saw that this PR: https://github.com/huggingface/transformers/pull/34511/files#diff-ed55888e6665791fe92cc8fc0c499da54f4ace6738551cd9a2591881cda076deR4965 got rid of setting accelerator's gradient_accumulation_plugin kwargs.
>
> It seems that the Trainer explicitly avoids relying on `gradient_accumulation_plugin` for handling gradient accumulation. Additionally, FSDP 2 uses `set_requires_gradient_sync` to control gradient synchronization.
>
> For FSDP 1, I found that adding a small monkey-patch at the very beginning of the project to globally disable no_sync() appears to work for me.
>
> ```
> import contextlib, importlib, types
> def _always_sync(self):
>     """Replacement for .no_sync that does nothing."""
>     return contextlib.nullcontext()
> def disable_no_sync():
>     """Monkey-patch FSDP so .no_sync returns a null-context."""
>     targets = [
>         ("torch.distributed.fsdp", "FullyShardedDataParallel")
>     ]
>     for mod_path, cls_name in targets:
>         try:
>             cls = getattr(importlib.import_module(mod_path), cls_name)
>             cls.no_sync = types.MethodType(_always_sync, cls)
>         except (ImportError, AttributeError):
>             continue
> disable_no_sync()
> ```
>
>

 _Originally posted by @zch0414 in [#29425](https://github.com/huggingface/transformers/issues/29425#issuecomment-3169638337)_

<img width="3012" height="1610" alt="Image" src="https://github.com/user-attachments/assets/bb2a30f9-d18a-4dda-961d-1175792a2d56" />

--- Comment by Rocketknight1 at 2026-02-11T12:48:09Z ---
cc @sunmarc

--- Comment by SunMarc at 2026-02-11T15:41:56Z ---
Hey @ojh31, thanks for the report ! Can you try this PR to fix if it fixes your issue ? Also, can you check that is the impact on speed ? I'm thinking if it will be better to always sync

--- Comment by ojh31 at 2026-02-11T18:07:00Z ---
@SunMarc Thanks for the quick response! The impact on speed was negligible in my case. I agree that when using FSDP the better default is to always sync.

--- Comment by SunMarc at 2026-02-12T11:16:00Z ---
Thanks ! With sync, how much memory did you save ?
cc @qgallouedec wdyt ? Do you want to enable this by default ?

--- Comment by qgallouedec at 2026-02-12T14:57:51Z ---
Great! Yes, I think it would be wise to enable it by default. I'm just wondering if there are any unfavourable cases that would confirm that the impact on speed remains minimal? Perhaps a large model + a large world size + numerous accumulation steps.

also, a limitation to be aware of (see [doc](https://docs.pytorch.org/docs/stable/fsdp.html) :

> FSDP currently does not support gradient accumulation outside `no_sync()` when using CPU offloading. This is because FSDP uses the newly-reduced gradient instead of accumulating with any existing gradient, which can lead to incorrect results.


ISSUE #43885: [Refactor] Confusing naming: input_embeds vs inputs_embeds
State: closed  |  Labels:
Author: JiangJiaWei1103  |  Created: 2026-02-10T10:00:22Z

There is currently inconsistent usage of the variable names `input_embeds` and `inputs_embeds` across the codebase, which can be confusing for contributors.

Based on a quick survey:
- `inputs_embeds` is used in 10,000+ occurrences across ~700 files
- `input_embeds` is used in ~700 occurrences across ~300 files

From a semantics perspective, both names refer to the same concept: a tensor containing token embeddings that are passed as model inputs. Having two near-identical names for the same concept makes code search harder and complicates model porting and refactoring efforts.

It seems reasonable to standardize on a single name. My suggestion would be to unify on `input_embeds`, which is concise and commonly used in other frameworks (e.g., `sglang`).

I’m opening this issue to ask:
- Is there a preferred naming convention between `input_embeds` and `inputs_embeds`?
- Would maintainers be open to standardizing on one name across the codebase to improve consistency and contributor experience?

Happy to help with a follow-up PR if this direction makes sense.

--- Comment by zucchini-nlp at 2026-02-10T12:52:57Z ---
Looks the `input_embeds` version appeared after the new attn mask API. We use `inputs_embeds` across the codebase and it model forward calls

I am 100% of using a single name, and I think it's better to keep `inputs_embeds` for BC purposes. Since this is a bit core cc @Cyrilvallez @vasqu

--- Comment by vasqu at 2026-02-10T15:13:52Z ---
`inputs_embeds` is indeed more consistently used across the code base (historically and nowadays) so if we align then `inputs_embeds` <- `input_embeds`

The issue is now which signatures are affected: It seems it's only the new mask API? Unsure how many remote models now use it so I'm a bit hesitant and we should add some form of BC compatibility. Wdyt @Cyrilvallez

--- Comment by Cyrilvallez at 2026-02-11T10:04:42Z ---
Ha indeed! Let's use `inputs_embeds` everywhere. We can simply do `@deprecate_kwarg("input_embeds", new_name="inputs_embeds")` on top of every affected signature

--- Comment by Cyrilvallez at 2026-02-11T10:10:38Z ---
Though actually it's almost mixed everywhere due to the mask (though it's only changing the name as an internal, so no need for any deprecation warning when it's only passing the "wrong" name to the mask) - so now that I think about it, maybe we should take the opportunity to make it better and use `input_embeds` (which is the correct english version).
Also, we use `input_ids` without the final S, so would make sense to align both!

--- Comment by Cyrilvallez at 2026-02-11T10:12:15Z ---
We can live with the decorator on all model's signatures for a few releases I guess, same as we did when I unified `past_key_value` -> `past_key_values`

--- Comment by Cyrilvallez at 2026-02-11T10:12:46Z ---
(At some point I would love a `past_key_values` -> `cache` as well, but that's another topic hahaha)

--- Comment by zucchini-nlp at 2026-02-11T10:24:33Z ---
> At some point I would love a past_key_values -> cache as well

It also helps us to get rid of `cache_params` used in linear-only models like mamba. Also had the same thought, it can be part of creating a base linear cache object instead of per-model cache

--- Comment by vasqu at 2026-02-11T12:11:45Z ---
Pro the cache renaming, it has to be broken at some point - while `input_embeds` is proper english I'd rather stick with the more BC version of `inputs_embeds` --> we cannot estimate how many have built upon our library standards and the mask is a lower barrier imo than the entire generate/forwards etc

--- Comment by Cyrilvallez at 2026-02-11T12:21:15Z ---
Note that with the `@deprecate_kwargs`, both names stay valid, so not BC issues! It's the point of the decorator!

--- Comment by vasqu at 2026-02-11T12:23:16Z ---
So we never want to fully deprecate, i.e. keep both? ;) it feels like we put a bandaid on we don't intend to lift because it will have huge ripple effects

Imo we can allow both imo but then don't call it a deprecation

--- Comment by Cyrilvallez at 2026-02-11T12:27:57Z ---
Oh it's just both are allowed during the deprecation cycle and raises a warning if the old name is used to tell users. Then we simply remove the decorator after a few versions and we only have the new one - basic deprecation cycle

--- Comment by zucchini-nlp at 2026-02-11T12:29:42Z ---
Not sure if we can really delete after few versions, it's been there for quite long and too many ppl rely on it as the main entry for forward pass

I think that might backfire us with more code to maintain and not really worth if the aim is only to `"use correct english version"`, we can deprecate easily mask API as it's a new code and keep old-style `inputs_embeds`

--- Comment by vasqu at 2026-02-11T12:30:41Z ---
Yes, that's my sentiment: We have too many that rely on these conventions it won't be a "simple" deprecation as people will complain (for a good reason)

--- Comment by Cyrilvallez at 2026-02-11T12:32:17Z ---
Not sure that many people use `inputs_embeds` tbh, but alright I hear you both haha, let's go with old name and fix the masking primitives then!

--- Comment by Cyrilvallez at 2026-02-11T12:33:08Z ---
(but it's the same as `past_key_values` -> `cache` in essence, except maybe less people pass the cache explicitly than inputs_embeds?)

--- Comment by Cyrilvallez at 2026-02-11T12:34:01Z ---
Any of you want to open a PR/already have a PR? Otherwise I can quickly do it?

--- Comment by vasqu at 2026-02-11T12:36:16Z ---
Nope, go ahead!

Re caching: This seems more of a necessary evil than `inputs_embeds`. In one case, it's essentially convention/style but on the other one it's overloading the meaning and creates a maintenance burden where we shouldn't have one

--- Comment by vasqu at 2026-02-19T11:08:09Z ---
Closing since it got handled by #43916

tl;dr: we rename to all `inputs_embeds` due to legacy reasons (+ eventually we will also rename caches at some point)


ISSUE #43883: AttributeError: 'MolmoForCausalLM' object has no attribute 'all_tied_weights_keys'
State: closed  |  Labels: bug
Author: Kaihui-intel  |  Created: 2026-02-10T08:48:56Z

### System Info

transformers==5.1.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

code
```
from transformers import AutoModelForCausalLM
model_name = "allenai/Molmo-7B-D-0924"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", trust_remote_code=True)
```
error
```bash
../../.local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:365: in from_pretrained
    return model_class.from_pretrained(
../../.local/lib/python3.11/site-packages/transformers/modeling_utils.py:4110: in from_pretrained
    load_info = cls._finalize_load_state_dict(model, load_config, load_info)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../.local/lib/python3.11/site-packages/transformers/modeling_utils.py:4264: in _finalize_load_state_dict
    model.mark_tied_weights_as_initialized()
../../.local/lib/python3.11/site-packages/transformers/modeling_utils.py:4587: in mark_tied_weights_as_initialized
    for tied_param in self.all_tied_weights_keys.keys():
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = MolmoForCausalLM(
  (model): Molmo(
    (transformer): ModuleDict(
      (wte): Embedding()
      (emb_drop): Dropout(...ropout): Dropout(p=0.0, inplace=False)
      )
      (image_feature_dropout): Dropout(p=0.0, inplace=False)
    )
  )
)
name = 'all_tied_weights_keys'

    def __getattr__(self, name: str) -> Union[Tensor, "Module"]:
        if "_parameters" in self.__dict__:
            _parameters = self.__dict__["_parameters"]
            if name in _parameters:
                return _parameters[name]
        if "_buffers" in self.__dict__:
            _buffers = self.__dict__["_buffers"]
            if name in _buffers:
                return _buffers[name]
        if "_modules" in self.__dict__:
            modules = self.__dict__["_modules"]
            if name in modules:
                return modules[name]
>       raise AttributeError(
            f"'{type(self).__name__}' object has no attribute '{name}'"
        )
E       AttributeError: 'MolmoForCausalLM' object has no attribute 'all_tied_weights_keys'

../../miniforge3/envs/py311/lib/python3.11/site-packages/torch/nn/modules/module.py:1964: AttributeError
```

### Expected behavior

The code should be work.

--- Comment by Rocketknight1 at 2026-02-10T14:23:06Z ---
cc @cyrilvallez remote models affected by weight tying PRs maybe?

--- Comment by lordaarush at 2026-02-12T13:30:10Z ---
Hey @Rocketknight1, I think i found the problem.
After #42270, `all_tied_weights_keys` is initialized in `post_init()`, but remote models (`trust_remote_code=True`) don't call this properly. I'm thinking of initializing it on-the-fly in `_adjust_tied_keys_with_tied_pointers()` by detecting tied weights via data pointers, with a safety check in `mark_tied_weights_as_initialized()`. Can I work on this and submit a PR?

--- Comment by Rocketknight1 at 2026-02-12T14:19:13Z ---
I'd prefer confirmation from a core maintainer who's worked with this first! cc @cyrilvallez @ArthurZucker

--- Comment by Cyrilvallez at 2026-02-12T15:38:14Z ---
Hey @lordaarush! The issue here is that tied_weights won't be handled properly if `post_init` is not called, and thus `all_tied_weights_keys` does not exist. For this reason I fully decided on the time to NOT check for the existence, so that it would crash for remote code that does not call `post_init`.
This is in my opinion much safer for several reasons, as `post_init` is very important in general. That way, while inconvenient on the moment, people can very quickly update their remote code to instead call `post_init`.

--- Comment by lordaarush at 2026-02-12T16:00:12Z ---
Thanks for explaining @Cyrilvallez! I understand now that it's by design to enforce proper `post_init()` usage.

I was just wondering about the user experience - since only model authors can update the remote code on the Hub, for example users are blocked from using Molmo until AllenAI updates it. Is there a way to notify model authors proactively about these issues, or do they typically find out through user reports?

Just trying to understand the workflow for these situations. Thanks!

--- Comment by Cyrilvallez at 2026-02-16T17:52:24Z ---
The best would be to directly open a PR for their code on the hub repo! They should in general be happy to merge!

--- Comment by Cyrilvallez at 2026-02-16T17:53:11Z ---
(we cannot force merge ourselves of course, even if it's really to "fix" something...)


glm-4v-9b loading failed

ISSUE #43881: glm-4v-9b loading failed
State: closed  |  Labels: bug
Author: Kaihui-intel  |  Created: 2026-02-10T08:31:10Z

### System Info

transformers v5.1.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

code
```
from transformers import AutoModelForCausalLM
model_name = "zai-org/glm-4v-9b"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", trust_remote_code=True)
```
error
```bash
../../.local/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py:365: in from_pretrained
    return model_class.from_pretrained(
../../.local/lib/python3.11/site-packages/transformers/modeling_utils.py:4072: in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/models/huggingface/modules/transformers_modules/glm_hyphen_4v_hyphen_9b/modeling_chatglm.py:1076: in __init__
    self.max_sequence_length = config.max_length
                               ^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = ChatGLMConfig {
  "add_bias_linear": false,
  "add_qkv_bias": true,
  "apply_query_key_layer_scaling": true,
  "apply_...en_layers": 63,
    "num_positions": 6401,
    "patch_size": 14,
    "scaling_factor": 8
  },
  "vocab_size": 151552
}

key = 'max_length'

    def __getattribute__(self, key):
        if key != "attribute_map" and key in super().__getattribute__("attribute_map"):
            key = super().__getattribute__("attribute_map")[key]
>       return super().__getattribute__(key)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AttributeError: 'ChatGLMConfig' object has no attribute 'max_length'

../../.local/lib/python3.11/site-packages/transformers/configuration_utils.py:164: AttributeError
```

### Expected behavior

`ChatGLMConfig` should be consistent with the modeling parameters.

--- Comment by zucchini-nlp at 2026-02-10T09:26:13Z ---
The glm family models are all supported in transformers ig, so you should be able to load without `trust_remote_code`. Custom code stored in the hub is not updated regularly and it's expected fail if that case

Maybe the HF equivalent of the model you want is https://huggingface.co/zai-org/GLM-4.1V-9B-Base?

--- Comment by zucchini-nlp at 2026-02-23T09:46:02Z ---
I will assume that the issue is resolved and close, so it doesn't attract code agents. If you still see the issue with official HF weights, feel free to re-open and provide a small reproducer

--- Comment by xin3he at 2026-03-16T05:15:43Z ---
Hi @zucchini-nlp
I'm using transformers 5.3.0 and this mode is not supported without trust_remote_code=True.

--- Comment by zucchini-nlp at 2026-03-16T08:05:20Z ---
We did several big refactors for v5 and deleted deprecated stuff, so the model owner would need to update custom code. cc @zRzRzRzRzRzRzR but you can also open a discussion in model repo page


ISSUE #43878: Save + loading from pretrained raises: out_features is not supported by TimmBackbone. Please use out_indices instead.
State: closed  |  Labels: bug
Author: jveitchmichaelis  |  Created: 2026-02-10T05:58:00Z

### System Info

Test in Colab - see notebook below. `transformers` 5.1.0

### Overview

A recent update (5.1?) seems to have broken model save + load for some architectures that use a ResNet-50(?) timm backbone (we noticed as our CI started to fail while v5.0 seemed to be fine). ViT backbones seem unaffected. To replicate, call save_pretrained and load_pretrained. Default/random init is affected as well as pretrained models, so it seems to be a bug in saving/loading rather than some deprecated key in the state dict.

The cause seems to be the presence of `out_features` in the saved checkpoint, which is then loaded (there is a [commented line ](https://github.com/huggingface/transformers/blob/b2028e775a52bf57ac2b6bd71b49ce61fa3adde6/src/transformers/models/timm_backbone/configuration_timm_backbone.py#L81)to pop it). May affect other models, but I did some grepping to find the list below.

```bash
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

[/tmp/ipython-input-2121964053.py](https://localhost:8080/#) in <cell line: 0>()
      7 model.save_pretrained(checkpoint_path)
      8
----> 9 _ = DeformableDetrForObjectDetection.from_pretrained(
     10     checkpoint_path,
     11 )

7 frames

[/usr/local/lib/python3.12/dist-packages/transformers/models/timm_backbone/modeling_timm_backbone.py](https://localhost:8080/#) in __init__(self, config, **kwargs)
     47
     48         if hasattr(config, "out_features") and config.out_features is not None:
---> 49             raise ValueError("out_features is not supported by TimmBackbone. Please use out_indices instead.")
     50
     51         # We just take the final layer by default. This matches the default for the transformers models.

ValueError: out_features is not supported by TimmBackbone. Please use out_indices instead.
```

### Who can help?

cc @yonigozlan

Thanks!

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Tested this on [Colab](https://colab.research.google.com/drive/1zFLB2i0rx6K9Clf5ukn7vtpyz44AYN-O?usp=sharing) with a default environment + updated transformers.

```python
import tempfile
import traceback
import transformers

MODELS = [
    ("DetrForObjectDetection", "DetrConfig"),
    ("ConditionalDetrForObjectDetection", "ConditionalDetrConfig"),
    ("DabDetrForObjectDetection", "DabDetrConfig"),
    ("TableTransformerForObjectDetection", "TableTransformerConfig"),
    ("DeformableDetrForObjectDetection", "DeformableDetrConfig"),
    ("GroundingDinoForObjectDetection", "GroundingDinoConfig"),
    ("MaskFormerForInstanceSegmentation", "MaskFormerConfig"),
    ("Mask2FormerForUniversalSegmentation", "Mask2FormerConfig"),
    ("OneFormerForUniversalSegmentation", "OneFormerConfig"),
    ("UperNetForSemanticSegmentation", "UperNetConfig"),
    ("DPTForSemanticSegmentation", "DPTConfig"),
    ("DepthAnythingForDepthEstimation", "DepthAnythingConfig"),
    ("ZoeDepthForDepthEstimation", "ZoeDepthConfig"),
    ("VitPoseForPoseEstimation", "VitPoseConfig"),
    ("VitMatteForImageMatting", "VitMatteConfig"),
    ("TvpForVideoGrounding", "TvpConfig"),
    ("OmDetTurboForObjectDetection", "OmDetTurboConfig"),
]

for model_cls_name, config_cls_name in MODELS:
    model_cls = getattr(transformers, model_cls_name)
    config_cls = getattr(transformers, config_cls_name)
    try:
        model = model_cls(config_cls())
        with tempfile.TemporaryDirectory() as d:
            model.save_pretrained(d)
            model_cls.from_pretrained(d)
        print(f"OK:    {model_cls_name}")
    except ValueError as e:
        if "out_features" in str(e):
            print(f"FAIL:   {model_cls_name}: {e}")
        else:
            print(f"OTHER: {model_cls_name}: {e}")
    except Exception as e:
        print(f"ERROR: {model_cls_name}: {type(e).__name__}: {e}")
    finally:
        del model

```

```bash
 Writing model shards: 100%
 1/1 [00:00<00:00,  3.76it/s]

FAIL:   DetrForObjectDetection: out_features is not supported by TimmBackbone. Please use out_indices instead.

Writing model shards: 100%
 1/1 [00:00<00:00,  3.86it/s]

FAIL:   ConditionalDetrForObjectDetection: out_features is not supported by TimmBackbone. Please use out_indices instead.

Writing model shards: 100%
 1/1 [00:02<00:00,  2.94s/it]

FAIL:   DabDetrForObjectDetection: out_features is not supported by TimmBackbone. Please use out_indices instead.

Writing model shards: 100%
 1/1 [00:00<00:00,  3.77it/s]

FAIL:   TableTransformerForObjectDetection: out_features is not supported by TimmBackbone. Please use out_indices instead.

Writing model shards: 100%
 1/1 [00:00<00:00,  4.22it/s]

FAIL:   DeformableDetrForObjectDetection: out_features is not supported by TimmBackbone. Please use out_indices instead.

Writing model shards: 100%
 1/1 [00:03<00:00,  3.92s/it]
Loading weights: 100%
 990/990 [00:01<00:00, 772.35it/s, Materializing param=model.text_projection.weight]

OK:    GroundingDinoForObjectDetection

Writing model shards: 100%
 1/1 [00:02<00:00,  2.95s/it]
Loading weights: 100%
 648/648 [00:01<00:00, 672.09it/s, Materializing param=model.transformer_module.queries_embedder.weight]

OK:    MaskFormerForInstanceSegmentation

Writing model shards: 100%
 1/1 [00:00<00:00,  1.24it/s]
Loading weights: 100%
 782/782 [00:01<00:00, 736.53it/s, Materializing param=model.transformer_module.queries_features.weight]

OK:    Mask2FormerForUniversalSegmentation

Writing model shards: 100%
 1/1 [00:00<00:00,  2.27it/s]
Loading weights: 100%
 610/610 [00:01<00:00, 616.13it/s, Materializing param=model.transformer_module.queries_embedder.weight]

OK:    OneFormerForUniversalSegmentation

Writing model shards: 100%
 1/1 [00:04<00:00,  4.48s/it]
Loading weights: 100%
 400/400 [00:00<00:00, 659.06it/s, Materializing param=decode_head.psp_modules.3.1.conv.weight]

OK:    UperNetForSemanticSegmentation

Writing model shards: 100%
 1/1 [00:06<00:00,  6.72s/it]
Loading weights: 100%
 280/280 [00:00<00:00, 518.98it/s, Materializing param=neck.reassemble_stage.readout_projects.3.0.weight]

OK:    DPTForSemanticSegmentation

Writing model shards: 100%
 1/1 [00:00<00:00,  5.23it/s]
Loading weights: 100%
 287/287 [00:00<00:00, 642.54it/s, Materializing param=neck.reassemble_stage.layers.3.resize.weight]

OK:    DepthAnythingForDepthEstimation

Writing model shards: 100%
 1/1 [01:02<00:00, 62.26s/it]
Loading weights: 100%
 553/553 [00:00<00:00, 770.34it/s, Materializing param=relative_head.conv3.weight]

OK:    ZoeDepthForDepthEstimation

Writing model shards: 100%
 1/1 [00:06<00:00,  6.32s/it]
Loading weights: 100%
 199/199 [00:00<00:00, 477.93it/s, Materializing param=head.conv.weight]

OK:    VitPoseForPoseEstimation

Writing model shards: 100%
 1/1 [00:02<00:00,  2.46s/it]
Loading weights: 100%
 198/198 [00:00<00:00, 319.44it/s, Materializing param=decoder.matting_head.matting_convs.3.weight]

OK:    VitMatteForImageMatting

Writing model shards: 100%
 1/1 [00:03<00:00,  3.03s/it]
Loading weights: 100%
 533/533 [00:00<00:00, 590.15it/s, Materializing param=video_grounding_head.layer_1.weight]

OK:    TvpForVideoGrounding

Writing model shards: 100%
 1/1 [00:11<00:00, 11.80s/it]

FAIL:   OmDetTurboForObjectDetection: out_features is not supported by TimmBackbone. Please use out_indices instead.

```

### Expected behavior

Models save + load without error using standard methods.

--- Comment by zucchini-nlp at 2026-02-10T09:28:20Z ---
Taking a look 👀  We recently refactored backbone models which might have affected it


ISSUE #43874: [GLM46V] Glm46VImageProcessorFast missing get_number_of_image_patches breaks _get_num_multimodal_tokens (AttributeError)
State: closed  |  Labels: bug
Author: baonudesifeizhai  |  Created: 2026-02-10T01:32:32Z

### System Info

When `use_fast=True`, `Glm46VProcessor._get_num_multimodal_tokens` calls:
`self.image_processor.get_number_of_image_patches(...)`,
but `Glm46VImageProcessorFast` does not implement this method.

This raises:
`AttributeError: 'Glm46VImageProcessorFast' object has no attribute 'get_number_of_image_patches'`

https://github.com/vllm-project/vllm/issues/34156

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/vllm-project/vllm/issues/34156

### Expected behavior

https://github.com/vllm-project/vllm/issues/34156

--- Comment by weiguangli-io at 2026-02-10T04:50:29Z ---
Opened a fix in #43877.\n\nThis adds get_number_of_image_patches to Glm46VImageProcessorFast, regenerates the fast processor file from modular source, and adds a regression test to keep fast and slow patch counting aligned.


ISSUE #43873: Offloading not working as expected with quantization
State: open  |  Labels: bug
Author: edmcman  |  Created: 2026-02-09T23:04:43Z

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-6.8.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.11.12
- Huggingface_hub version: 0.36.2
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4070 Laptop GPU

### Who can help?

@sunmarc @MekkCyber

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Run with `uvx --with protobuf,sentencepiece --with "transformers==4.57.3" --with "accelerate==1.12.0" --with "bitsandbytes==0.49.1" python bug3.py`
(The versions are needed because of #43872)
``` python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

MODEL_ID = "ejschwartz/decaf-v1-22b"

bnb_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_enable_fp32_cpu_offload=True,
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    low_cpu_mem_usage=True,
)

prompt = "Hello!"
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=1024)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=32, do_sample=False)

print(tokenizer.decode(out[0], skip_special_tokens=True))
```
2. See error `torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 36.00 MiB. GPU 0 has a total capacity of 7.62 GiB of which 75.75 MiB is free. Including non-PyTorch memory, this process has 6.46 GiB memory in use. Of the allocated memory 6.20 GiB is allocated by PyTorch, and 92.81 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)`

### Expected behavior

Not to run out of memory.  Note that commenting out the `quantization_config=bnb_config` line counter-intuitively does *not* result in a CUDA error though it's very slow, as expected.

I have also tried `max_memory={0: "3GiB", "cpu": "64GiB"},` which did not help.  I have about 7 GiB of free vram, so this should have easily fit.

--- Comment by SunMarc at 2026-02-10T15:11:17Z ---
cc @Cyrilvallez, are you fine with cleaning `_is_hf_initialized` flag after we finished loading the model or there could be unexpected use case ? The issue happens later on here.

--- Comment by Cyrilvallez at 2026-02-11T09:45:58Z ---
Humm, I don't see any link to the `_is_hf_initialized` flag @SunMarc?? Moreover, this is on v4 so the flag was only applied to modules there, and loading has changed tremenduously.
The real reason here @edmcman is that you seem to be trying to load a 22B model into a 8GB gpu, so it will never fit even with quantization. Not sure why the device_map is not being respected with bnb config though, could you try on latest v5 instead?

--- Comment by SunMarc at 2026-02-11T10:34:38Z ---
Oh sorry, tagged you in the incorrect issue as OP opened two issues. Here's the right one where he tested on v5 : https://github.com/huggingface/transformers/issues/43872

--- Comment by github-actions[bot] at 2026-03-12T08:08:48Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by edmcman at 2026-03-12T13:51:03Z ---
I think this is still an issue.  It's still unclear if quantization +
offloading is supported, and if so, when?  Always?

On Thu, Mar 12, 2026 at 4:09 AM github-actions[bot] -
***@***.*** <github.edmcman.99c9f1b9d0.notifications#
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZJK2546PUXKNOWQGFT4QJWCNAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANBUG44DMNBYGA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

--- Comment by michaelanderson01826-glitch at 2026-03-12T16:05:38Z ---
Yes I agree this is still an issue going on

On Thu, Mar 12, 2026, 1:51 PM Edward J. Schwartz ***@***.***>
wrote:

> *edmcman* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4046936878>
> I think this is still an issue. It's still unclear if quantization +
> offloading is supported, and if so, when? Always?
>
> On Thu, Mar 12, 2026 at 4:09 AM github-actions[bot] -
> ***@***.*** <github.edmcman.99c9f1b9d0.notifications#
> ***@***.***> wrote:
>
> > *github-actions[bot]* left a comment (huggingface/transformers#43873)
> > <
> https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>
>
> >
> > This issue has been automatically marked as stale because it has not had
> > recent activity. If you think this still needs to be addressed please
> > comment on this thread.
> >
> > Please note that issues that do not follow the contributing guidelines
> > <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> > are likely to be ignored.
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
> https://github.com/huggingface/transformers/issues/43873#issuecomment-4044786480>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AAHYKZJK2546PUXKNOWQGFT4QJWCNAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANBUG44DMNBYGA>
>
> > .
> > You are receiving this because you were mentioned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4046936878>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEJQXLDWHX3RVZJC6OT4QK6GJAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANBWHEZTMOBXHA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

--- Comment by github-actions[bot] at 2026-04-06T08:28:27Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by edmcman at 2026-04-06T12:36:12Z ---
not stale

On Mon, Apr 6, 2026 at 4:28 AM github-actions[bot] -
***@***.*** <github.edmcman.99c9f1b9d0.notifications#
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4191140253>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4191140253>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZOHHEMAJMMTWC2DIK34UNTD7AVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOJRGE2DAMRVGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

--- Comment by github-actions[bot] at 2026-05-01T08:35:52Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by edmcman at 2026-05-01T09:52:54Z ---
Not stale

On Fri, May 1, 2026, 4:36 AM github-actions[bot] - ***@***.***
***@***.***> wrote:

> *github-actions[bot]* left a comment (huggingface/transformers#43873)
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4358524904>
>
> This issue has been automatically marked as stale because it has not had
> recent activity. If you think this still needs to be addressed please
> comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines
> <https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md>
> are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43873#issuecomment-4358524904>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAHYKZJASZQHZLKZ5M3HTPD4YROYDAVCNFSM6AAAAACURH7UVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DGNJYGUZDIOJQGQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


ISSUE #43872: bitsandbytes incompatibility: TypeError: Int8Params.__new__() got an unexpected keyword argument '_is_hf_initialized'
State: closed  |  Labels: bug
Author: edmcman  |  Created: 2026-02-09T22:32:43Z

### System Info

Well that's interesting...
```
ed@banana /tmp [127]> uvx --with huggingface-hub,ipython,transformers,torch,bitsandbytes,accelerate transformers env
Traceback (most recent call last):
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/bin/transformers", line 6, in <module>
    from transformers.cli.transformers import main
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/transformers/cli/transformers.py", line 22, in <module>
    from transformers.cli.serve import Serve
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/transformers/cli/serve.py", line 360, in <module>
    class Serve:
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/transformers/cli/serve.py", line 588, in Serve
    validator: TypeAdapter,
               ^^^^^^^^^^^
NameError: name 'TypeAdapter' is not defined
```

I guess that's a bug too?

Here are the major versions:
torch 2.10.0+cu128 transformers 5.1.0 bitsandbytes 0.49.1 accelerate 1.12.0

### Who can help?

@sunmarc @MekkCyber

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Run with `uvx --with huggingface-hub,ipython,transformers,torch,bitsandbytes,accelerate python bug.py`
``` python
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
MODEL_ID = "ejschwartz/decaf-v1-22b"
bnb_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_enable_fp32_cpu_offload=True,
)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    max_memory={0: "3GiB", "cpu": "64GiB"},  # tighter GPU cap than before
)
```
2. Receive exception
```
Traceback (most recent call last):
  File "/tmp/bug.py", line 17, in <module>
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 374, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4083, in from_pretrained
    accelerate_dispatch(model, hf_quantizer, device_map, offload_folder, disk_offload_index, offload_buffers)
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/transformers/integrations/accelerate.py", line 389, in accelerate_dispatch
    dispatch_model(model, **device_map_kwargs)
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/big_modeling.py", line 426, in dispatch_model
    attach_align_device_hook_on_blocks(
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 676, in attach_align_device_hook_on_blocks
    attach_align_device_hook_on_blocks(
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 639, in attach_align_device_hook_on_blocks
    attach_align_device_hook(
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 530, in attach_align_device_hook
    attach_align_device_hook(
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 530, in attach_align_device_hook
    attach_align_device_hook(
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 521, in attach_align_device_hook
    add_hook_to_module(module, hook, append=True)
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in add_hook_to_module
    module = hook.init_hook(module)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/hooks.py", line 313, in init_hook
    set_module_tensor_to_device(module, name, "meta")
  File "/home/ed/.cache/uv/archive-v0/93vgXQSK-rYWDyEU_W9k9/lib/python3.11/site-packages/accelerate/utils/modeling.py", line 363, in set_module_tensor_to_device
    new_value = param_cls(new_value, requires_grad=old_value.requires_grad, **kwargs).to(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Int8Params.__new__() got an unexpected keyword argument '_is_hf_initialized'
```

### Expected behavior

No exception

--- Comment by Cyrilvallez at 2026-02-11T11:09:05Z ---
Hey @SunMarc! Ha yes, I remember that the bnb quantizer was first removing the flag before adding it back indeed! However I must say in general I'm unsure if bnb works with offloading?? I.e. I don't think accelerate is correctly serializing everything to reload correctly?

--- Comment by SunMarc at 2026-02-11T13:25:58Z ---
>  However I must say in general I'm unsure if bnb works with offloading?

It should work but I don't recommend offloading to CPU in general as it is super slow

--- Comment by edmcman at 2026-02-11T14:08:37Z ---
@Cyrilvallez The [documentation says that offloading is possible with 8-bit quantization](https://huggingface.co/docs/transformers/en/quantization/bitsandbytes#offloading).

But part of the reason I opened this issue is that the overall support for offloading with bnb is very unclear.  It seems like offloading with 4-bit is not supported, but I don't see that explicitly stated in the documentation.  And I've seen an exception when using 4-bit quants - I think it is [this one](https://github.com/huggingface/transformers/blob/b52b6631a26fd362e49968af14ed77ff359006a0/src/transformers/quantizers/quantizer_bnb_4bit.py#L72) - that at first glance suggests that offloading with 4-bit quants should be possible.  After closer inspection, the `int8` references probably indicate that message should only be shown for 8-bit quants?

The larger picture of what I'm trying to do is make two large models at least *usable* for people with consumer GPUs like mine.  I've had very good luck with the 22B model using llama.cpp and Q4_K_M quants.  With partial offloading using my 8GiB GPU, I get about 6 tokens per second, which is fine for my purposes.  The 32B model is a qwen3 model with a classification head on it, which llama.cpp/GGML does not support.  This is why I am interested in understanding whether transformers/bitsandbytes could possibly help.

Obviously in an ideal world, everyone would have access to enough VRAM, but they don't.  IMHO if CPU offloading makes sense anywhere, it's with 4-bit quants, as llama.cpp/Ollama and related projects are showing.

@SunMarc When you say CPU offloading is super slow, do you mean 100% offloading, or partial offloading?  For the 22B quantized model on llama.cpp, I found that 100% offloading is completely unusable, but partial offloading works really well.

--- Comment by edmcman at 2026-02-11T14:11:11Z ---
Also, now I've gotten a little confused about my two issues.  This issue #43872 seems like a transformers v5 bug.  #43873 was supposed to be the more general "offloading in bnb is not working" issue when avoiding the `_is_hf_initialized` problem.

--- Comment by Cyrilvallez at 2026-02-16T17:37:13Z ---
cc @SunMarc, we should make sure the flag is not passed to the constructor when reinstantiating params with bnb

--- Comment by github-actions[bot] at 2026-03-13T08:08:42Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by edmcman at 2026-03-13T12:22:12Z ---
Not stale

On Fri, Mar 13, 2026 at 4:09 AM github-actions[bot] -
***@***.***
***@***.***> wrote:
>
> github-actions[bot] left a comment (huggingface/transformers#43872)
>
> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
>
> Please note that issues that do not follow the contributing guidelines are likely to be ignored.
>
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.Message ID: ***@***.***>

--- Comment by dorpxam at 2026-03-15T18:39:14Z ---
@edmcman I confirm that it's a bug. 4.57.6 no problem, 5.0.0 problem. Same configuration, same models.

## EDIT (Monkey Patch)

```python
import bitsandbytes as bnb

original_new = bnb.nn.Int8Params.__new__

def patched_new(cls, data=None, requires_grad=False, has_fp16_weights=False, **kwargs):
    kwargs.pop('_is_hf_initialized', None)
    return original_new(cls, data, requires_grad, has_fp16_weights, **kwargs)

bnb.nn.Int8Params.__new__ = patched_new
```

Just before any import ...

--- Comment by codybarrett108-pixel at 2026-03-17T14:19:53Z ---
Thanks for reporting this I'll take a look

On Sun, Mar 15, 2026, 1:39 PM Max Prod ***@***.***> wrote:

> *dorpxam* left a comment (huggingface/transformers#43872)
> <https://github.com/huggingface/transformers/issues/43872#issuecomment-4063624606>
>
> @edmcman <https://github.com/edmcman> I confirm that it's a bug. 4.57.6
> no problem, 5.0.0 problem. Same configuration, same models.
>
> File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\transformers\modeling_utils.py", line 4129, in from_pretrained    accelerate_dispatch(  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\transformers\integrations\accelerate.py", line 353, in accelerate_dispatch    dispatch_model(model, **device_map_kwargs)  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\big_modeling.py", line 426, in dispatch_model    attach_align_device_hook_on_blocks(  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 676, in attach_align_device_hook_on_blocks    attach_align_device_hook_on_blocks(  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 676, in attach_align_device_hook_on_blocks    attach_align_device_hook_on_blocks(  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 676, in attach_align_device_hook_on_blocks    attach_align_device_hook_on_blocks(  [Previous line repeated 1 more time]  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 639, in attach_align_device_hook_on_blocks    attach_align_device_hook(  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 530, in attach_align_device_hook    attach_align_device_hook(  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 530, in attach_align_device_hook    attach_align_device_hook(  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 521, in attach_align_device_hook    add_hook_to_module(module, hook, append=True)  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 166, in add_hook_to_module    module = hook.init_hook(module)             ^^^^^^^^^^^^^^^^^^^^^^  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\hooks.py", line 313, in init_hook    set_module_tensor_to_device(module, name, "meta")  File "C:\anaconda3\envs\ltx2caption\Lib\site-packages\accelerate\utils\modeling.py", line 363, in set_module_tensor_to_device    new_value = param_cls(new_value, requires_grad=old_value.requires_grad, **kwargs).to(                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^TypeError: Int8Params.__new__() got an unexpected keyword argument '_is_hf_initialized'
>
> return BitsAndBytesConfig(
>     load_in_8bit=True,
>     llm_int8_enable_fp32_cpu_offload=True,
>     llm_int8_threshold=0.0, # 6.0 default
> )
>
> With that kind of problem, I can't work with two wonderful models :
>
>    - Step3-VL-10B : works with transformers 4.57.6 (latest 4.x)
>    - Music Flamingo : work with transformers 5.x
>
> The problem is that I need to offload part of the Step3-VL-10B, 8bit quant
> is not enough.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43872#issuecomment-4063624606>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HX4LXAOGJFKS7JP3NFA3L4Q32G7AVCNFSM6AAAAACURFCIXCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANRTGYZDINRQGY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

--- Comment by SunMarc at 2026-03-17T15:23:31Z ---
Fixed on accelerate side, please install the version from main for now until I release a new version


load model error when state_dict sorted

ISSUE #43867: load model error when state_dict sorted
State: closed  |  Labels: bug
Author: nursery42  |  Created: 2026-02-09T17:03:21Z

### System Info

```
my_model.from_pretraine('path_to_model')

state_dict = sorted(state_dict.items(), key=lambda kv: dot_natural_key(kv[0]))
TypeError: '<' not supported between instances of 'str' and 'int'
```
dot_natural_key splits model parameter names into a list composed of several strings or integers. However, in some models, there may be both integers and strings at the same position in this list, which seems to result in an error.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model from_pretrained when loading a portion of a model that has sub-models or contains multiple built-in models. dot_natural_key causes elements at different positions in the list to have different types. if will sort state_dict name such as
['model', 'layers', 26, 'self_attn', 'o_proj', 'weight']
['desc_model', 'roberta', 'encoder', 'layers', 7, 'norm1', 'weight'] .

### Expected behavior

fix dot_natural_key

--- Comment by Rocketknight1 at 2026-02-10T12:46:07Z ---
Hi @enze5088, can you give us an example checkpoint where you get this error?

--- Comment by Rocketknight1 at 2026-02-10T14:27:03Z ---
Seems real though - cc @arthurzucker @cyrilvallez, the problem is that `state_dict = sorted(state_dict.items(), key=lambda kv: dot_natural_key(kv[0]))` in `core_model_loading.py` calls `dot_natural_key`, but when `kv[0]` is sometimes numerical and sometimes not, this causes a comparison failure and a crash.

--- Comment by ArthurZucker at 2026-02-10T15:38:04Z ---
Mmmm I don't see when it can be non numerical but happy to have a quick fix 😉

--- Comment by Sai-Suraj-27 at 2026-02-13T10:46:12Z ---
> Hi [@enze5088](https://github.com/enze5088), can you give us an example checkpoint where you get this error?

> Mmmm I don't see when it can be non numerical but happy to have a quick fix 😉

Hey, @Rocketknight1 @ArthurZucker

I was trying to load this model: `jinaai/jina-embeddings-v3` with the latest source code and faced this issue. I did some debugging, The value returned by `dot_natural_key(kv[0])`  is a list right now and the list can contain both strings and integers which is causing the issue.

Example (https://huggingface.co/jinaai/jina-embeddings-v3/blob/main/model.safetensors), If we see for this model weights,
```py
# kv[0]: roberta.embeddings.token_type_embeddings.parametrizations.weight.0.lora_A
dot_natural_key(kv[0]):  ["roberta", "embeddings", "token_type_embeddings", "parametrizations", "weight", 0, "lora_A"]

# kv[0]:  roberta.embeddings.token_type_embeddings.parametrizations.weight.0.lora_B
dot_natural_key(kv[0]): ["roberta", "embeddings", "token_type_embeddings", "parametrizations", "weight", 0, "lora_B"]

# kv[0]: roberta.embeddings.token_type_embeddings.parametrizations.weight.original
dot_natural_key(kv[0]): ["roberta", "embeddings", "token_type_embeddings", "parametrizations", "weight", "original"]
```

So, when we compare these returned lists for sorting then **0** (`int`) get's compared with **original** (`string`) as till that index both lists have the same prefix which is causing the issue. From a quick glance, this https://github.com/huggingface/transformers/pull/43871 feels better than https://github.com/huggingface/transformers/pull/43966 (Account contains 100's of spam PR's to openclaw 🥲)

--- Comment by Rocketknight1 at 2026-02-13T14:14:18Z ---
Yeah, since we see some clear examples, I'm going to merge the fix at #43966


Ovis2 1B checkpoint corrupted

ISSUE #43866: Ovis2 1B checkpoint corrupted
State: closed  |  Labels: bug
Author: EnricoBeltramo  |  Created: 2026-02-09T15:27:57Z

### System Info

transformers env
Traceback (most recent call last):
  File "/root/miniconda3/bin/transformers", line 3, in <module>
    from transformers.cli.transformers import main
  File "/root/miniconda3/lib/python3.12/site-packages/transformers/cli/transformers.py", line 22, in <module>
    from transformers.cli.serve import Serve
  File "/root/miniconda3/lib/python3.12/site-packages/transformers/cli/serve.py", line 360, in <module>
    class Serve:
  File "/root/miniconda3/lib/python3.12/site-packages/transformers/cli/serve.py", line 588, in Serve
    validator: TypeAdapter,
               ^^^^^^^^^^^
NameError: name 'TypeAdapter' is not defined

manual versions:
transformers 5.1.0
Ubuntu 24
python 3.12

### Who can help?

I'm loading AIDC-AI/Ovis2-1B using this script (derived from official example):
`
from PIL import Image
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from accelerate import Accelerator

MODEL_NAME = "AIDC-AI/Ovis2-1B"

device = Accelerator().device

model = AutoModelForImageTextToText.from_pretrained(
    MODEL_NAME,
    dtype=torch.bfloat16,
).eval().to(device)
processor = AutoProcessor.from_pretrained(MODEL_NAME)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "OCR:"},
        ],
    },
]
# Load image in memory
image = Image.open("53061_image.jpg").convert("RGB")
messages = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(messages)

inputs = processor(
    images=[image],
    text=messages,
    return_tensors="pt",
)
inputs = inputs.to(model.device)
inputs['pixel_values'] = inputs['pixel_values'].to(torch.bfloat16)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
    generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
    output_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
    print(output_text)
`

But I have this error:

model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████| 3.54G/3.54G [00:23<00:00, 152MB/s]
Loading weights: 100%|███████████████| 122/122 [00:00<00:00, 5319.57it/s, Materializing param=model.vision_tower.transformer.rms_norm.weight]
This checkpoint seem corrupted. The tied weights mapping for this model specifies to tie model.language_model.embed_tokens.weight to lm_head.weight, but both are absent from the checkpoint, and we could not find another related tied weight for those keys
Ovis2ForConditionalGeneration LOAD REPORT from: thisisiron/Ovis2-1B-hf
Key                                                                              | Status     |
---------------------------------------------------------------------------------+------------+-
vision_tower.transformer.encoder.layers.{0...23}.ffn.fc2.weight                  | UNEXPECTED |
language_model.model.layers.{0...23}.self_attn.q_proj.bias                       | UNEXPECTED |
vision_tower.transformer.encoder.layers.{0...23}.attention.proj_out.weight       | UNEXPECTED |
language_model.model.layers.{0...23}.self_attn.o_proj.weight                     | UNEXPECTED |
language_model.model.layers.{0...23}.input_layernorm.weight                      | UNEXPECTED |
language_model.model.layers.{0...23}.self_attn.q_proj.weight                     | UNEXPECTED |
vision_tower.head.{0, 1}.weight                                                  | UNEXPECTED |
visual_table.weight                                                              | UNEXPECTED |
vision_tower.transformer.encoder.layers.{0...23}.ffn.fc1.weight                  | UNEXPECTED |
language_model.model.layers.{0...23}.post_attention_layernorm.weight             | UNEXPECTED |
language_model.model.layers.{0...23}.mlp.up_proj.weight                          | UNEXPECTED |
language_model.model.layers.{0...23}.mlp.gate_proj.weight                        | UNEXPECTED |
language_model.model.layers.{0...23}.self_attn.k_proj.weight                     | UNEXPECTED |
language_model.model.layers.{0...23}.mlp.down_proj.weight                        | UNEXPECTED |
vision_tower.transformer.encoder.layers.{0...23}.ffn.fc3.weight                  | UNEXPECTED |
language_model.model.layers.{0...23}.self_attn.v_proj.bias                       | UNEXPECTED |
language_model.model.layers.{0...23}.self_attn.v_proj.weight                     | UNEXPECTED |
language_model.model.layers.{0...23}.self_attn.k_proj.bias                       | UNEXPECTED |
vision_tower.transformer.embeddings.patch_embed.weight                           | UNEXPECTED |
vision_tower.transformer.embeddings.patch_embed.bias                             | UNEXPECTED |
vision_tower.transformer.embeddings.position_embeddings.weight                   | UNEXPECTED |
vision_tower.head.1.bias                                                         | UNEXPECTED |
language_model.model.embed_tokens.weight                                         | UNEXPECTED |
language_model.model.norm.weight                                                 | UNEXPECTED |
model.language_model.layers.{0...23}.self_attn.q_proj.weight                     | MISSING    |
model.language_model.layers.{0...23}.self_attn.k_proj.weight                     | MISSING    |
model.vision_tower.head_norm.bias                                                | MISSING    |
model.vision_tower.transformer.encoder.layers.{0...23}.attention.out_proj.weight | MISSING    |
model.language_model.layers.{0...23}.self_attn.q_proj.bias                       | MISSING    |
model.language_model.layers.{0...23}.mlp.down_proj.weight                        | MISSING    |
model.vision_tower.transformer.encoder.layers.{0...23}.ffn.up_proj.weight        | MISSING    |
model.language_model.layers.{0...23}.self_attn.v_proj.bias                       | MISSING    |
model.language_model.layers.{0...23}.self_attn.k_proj.bias                       | MISSING    |
model.language_model.layers.{0...23}.self_attn.o_proj.weight                     | MISSING    |
model.language_model.layers.{0...23}.post_attention_layernorm.weight             | MISSING    |
model.vision_tower.transformer.encoder.layers.{0...23}.ffn.gate_proj.weight      | MISSING    |
model.language_model.layers.{0...23}.input_layernorm.weight                      | MISSING    |
model.language_model.layers.{0...23}.mlp.gate_proj.weight                        | MISSING    |
model.language_model.layers.{0...23}.mlp.up_proj.weight                          | MISSING    |
model.vision_tower.head_linear.weight                                            | MISSING    |
model.language_model.layers.{0...23}.self_attn.v_proj.weight                     | MISSING    |
model.vision_tower.transformer.encoder.layers.{0...23}.ffn.down_proj.weight      | MISSING    |
model.visual_embeddings_table.weight                                             | MISSING    |
model.language_model.embed_tokens.weight                                         | MISSING    |
model.vision_tower.transformer.embeddings.position_embedding.weight              | MISSING    |
lm_head.weight                                                                   | MISSING    |
model.vision_tower.head_norm.weight                                              | MISSING    |
model.vision_tower.transformer.embeddings.patch_embedding.weight                 | MISSING    |
model.vision_tower.transformer.embeddings.patch_embedding.bias                   | MISSING    |
model.language_model.norm.weight                                                 | MISSING    |

Notes:
- UNEXPECTED    :can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING       :those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

and the model doesn't work (emit a very long string of casual characters). If I use 2b it's works fine

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the script above

### Expected behavior

A regular ocr detection, no error when download weigths

--- Comment by Saad-Mallebhari at 2026-02-28T10:19:33Z ---
@EnricoBeltramo
The issue is a weight key mismatch , the 1B checkpoint at `thisisiron/Ovis2-1B-hf` has keys like `language_model.model.layers.*` but the `Ovis2ForConditionalGeneration` architecture expects `model.language_model.layers.*`. The `model.` prefix is missing from the converted checkpoint, which is why every key shows as UNEXPECTED/MISSING.

The 2B works because the official HF conversion for that size has the correct key prefix.

Workaround , load using the official AIDC-AI checkpoint directly instead of the community conversion:
```python
MODEL_NAME = "AIDC-AI/Ovis2-1B-hf"  # use official, not thisisiron/
```
If that's unavailable, you can remap keys manually before loading:
```python
from safetensors.torch import load_file, save_file

weights = load_file("model.safetensors")
remapped = {"model." + k: v for k, v in weights.items()}
save_file(remapped, "model_fixed.safetensors")
```
This appears to be a conversion issue specific to that community checkpoint, rather than an issue in transformers itself.

--- Comment by github-actions[bot] at 2026-03-25T08:11:10Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43864: GlmMoeDsaConfig: mlp_layer_types default overwritten by inlined parent init
State: closed  |  Labels:
Author: joninco  |  Created: 2026-02-09T14:30:13Z

## Bug Description

`GlmMoeDsaConfig` ends up with the wrong default `mlp_layer_types`. The intended default is `["dense"]*3 + ["sparse"]*75` (3 initial dense layers), but the actual default at runtime is `["dense"] + ["sparse"]*77` (1 dense layer).

## Root Cause

The generated `configuration_glm_moe_dsa.py` inlines both the child (`GlmMoeDsaConfig`) and parent (`Glm4MoeLiteConfig`) `__init__` bodies sequentially. Both contain this pattern:

```python
self.mlp_layer_types = mlp_layer_types  # from __init__ param, which is None
if self.mlp_layer_types is None:
    self.mlp_layer_types = ["dense"] * N + ["sparse"] * M
```

The child's block (lines ~211-215) correctly computes `["dense"]*3 + ["sparse"]*75`. But then the parent's inlined block (lines ~254-256) runs afterward, resets `self.mlp_layer_types` back to `None` (from the original parameter), and recomputes it as `["dense"] + ["sparse"]*77`.

The modular source (`modular_glm_moe_dsa.py`) has the same issue — the child sets the correct value at lines 214-217, then `super().__init__(**kwargs)` at line 250 calls the parent which overwrites it.

## Reproduction

```python
from transformers import GlmMoeDsaConfig

config = GlmMoeDsaConfig()
print(config.mlp_layer_types[:5])
# Actual:   ['dense', 'sparse', 'sparse', 'sparse', 'sparse']
# Expected: ['dense', 'dense', 'dense', 'sparse', 'sparse']
```

## Impact

Layers 1 and 2 will use sparse MoE instead of dense MLP when using the default config. This likely does not affect pretrained model loading (where `mlp_layer_types` is read from `config.json`), but it affects anyone instantiating a default config.

## Affected Files

- `src/transformers/models/glm_moe_dsa/modular_glm_moe_dsa.py` (source of truth)
- `src/transformers/models/glm_moe_dsa/configuration_glm_moe_dsa.py` (generated, runtime)

## Suggested Fix

In `modular_glm_moe_dsa.py`, move the `mlp_layer_types` default computation to **after** the `super().__init__()` call, so the child's value is set last:

```python
def __init__(self, ..., mlp_layer_types=None, **kwargs):
    ...
    super().__init__(**kwargs)
    # Set after super() so parent doesn't overwrite
    if mlp_layer_types is None:
        mlp_layer_types = ["dense"] * 3 + ["sparse"] * (self.num_hidden_layers - 3)
    self.mlp_layer_types = mlp_layer_types
```

Then regenerate `configuration_glm_moe_dsa.py`.

Introduced in #43858.

--- Comment by joninco at 2026-02-09T14:30:50Z ---
cc @zRzRzRzRzRzRzR

--- Comment by zucchini-nlp at 2026-02-09T20:06:04Z ---
Thanks for reporting, indeed modular auto-generated it incorrectly. We need to call `PreTrainedConfig.__init__(**kwargs)` in modular file instead of `super().__init__(**kwargs)`, to make sure that the content of init isn't copied

Would you like to make a PR?

--- Comment by weiguangli-io at 2026-02-10T04:29:10Z ---
Opened a fix in #43876.\n\nIt updates the GlmMoeDsaConfig modular init so parent init logic is not inlined and cannot overwrite mlp_layer_types defaults, regenerates the config file, and adds a regression test for the expected default layer pattern.


huggingface

ISSUE #43859: huggingface
State: closed  |  Labels:
Author: Pediboi666  |  Created: 2026-02-09T10:30:38Z

https://github.com/likaia/nginxpulse/issues/54

--- Comment by Rocketknight1 at 2026-02-09T12:35:48Z ---
huggingface