FSDP_CPU_RAM_EFFICIENT_LOADING broken

ISSUE #43749: FSDP_CPU_RAM_EFFICIENT_LOADING broken
State: closed  |  Labels: bug
Author: kmod  |  Created: 2026-02-05T00:36:21Z

### System Info

- `transformers` version: 5.0.0
- Platform: Linux-6.14.0-37-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 1.3.4
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.1+cu129 (CUDA)
- Using distributed or parallel set-up in script?: manual launcher
- Using GPU in script?: yes
- GPU type: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Load a large model using FSDP2 and FSDP_CPU_RAM_EFFICIENT_LOADING=True. I don't think this is model-specific, but I ran into this with Qwen/Qwen3-30B-A3B-Instruct-2507
2. Watch the ram usage of all of the ranks
3. See that all ranks temporarily use an amount of CPU RAM that's the size of the model being loaded
4. See that the system gets stuck for a long time right after the weights are loaded

I think there are two separate but related issues happening here:
- The model loading code doesn't check for FSDP_CPU_RAM_EFFICIENT_LOADING until _move_missing_keys_from_meta_to_device, which is after the model is already loaded. At this point it discards the loaded weights and inserts new empty tensors
- These empty tensors are seen as uninitialized and will all undergo random initialization. Since they're loaded on the CPU this can take a very long time

This diff causes both of these issues to go away for me. Setting _is_hf_initialized on the new tensors seems like it might be the intended way of handling the second. I'm not sure about the first; I just used a kludge where it skips all of the loading entirely. This works but causes a lot of loud warnings about missing keys. I had tried changing get_device() to return "meta" but this crashed quickly and I didn't investigate further.

```diff
--- modeling_utils.py.orig      2026-02-04 06:59:59.107266669 +0000
+++ modeling_utils.py   2026-02-02 05:50:43.453427661 +0000
@@ -497,10 +497,11 @@
 def _load_parameter_into_model(model: "PreTrainedModel", param_name: str, tensor: torch.Tensor):
     """Cast a single parameter or buffer `param_name` into the `model`, with value `tensor`."""
     parent, param_type = get_module_from_name(model, param_name)
     if param_type in parent._parameters and not isinstance(tensor, nn.Parameter):
         tensor = nn.Parameter(tensor, requires_grad=tensor.is_floating_point())
+        tensor._is_hf_initialized = True
     # We need to use setattr here, as we set non-persistent buffers as well with this function (`load_state_dict`
     # does not allow to do it)
     setattr(parent, param_type, tensor)
--- core_model_loading.py.orig  2026-02-04 07:03:09.751218636 +0000
+++ core_model_loading.py       2026-02-04 07:02:54.937066957 +0000
@@ -1134,10 +1134,15 @@

     pattern_to_converter = {k: converter for converter in converters for k in converter.source_patterns}

     state_dict = sorted(state_dict.items(), key=lambda kv: dot_natural_key(kv[0]))

+    from .integrations import is_fsdp_enabled
+    from .modeling_utils import is_local_dist_rank_0
+    if is_fsdp_enabled() and not is_local_dist_rank_0() and hf_quantizer is None:
+        state_dict = []
+
     for original_key, tensor in state_dict:
         # 1. Rename the key according to all renaming pattern and optional weight converter patterns
         renamed_key, source_pattern = rename_source_key(
             original_key, renamings, converters, prefix, meta_model_state_dict
         )
```

### Expected behavior

- Only rank0 allocates a large amount of cpu ram
- The system moves quickly from weights loading to the rest of the initialization

--- Comment by ArthurZucker at 2026-02-06T11:17:38Z ---
cc @SunMarc or @3outeille as well. I reviewed the PR (not mergeable) but might need a proper fix indeed.

Ty for the report

--- Comment by SunMarc at 2026-02-06T14:30:46Z ---
cc @winglian did you see any issue on your CI ?

--- Comment by github-actions[bot] at 2026-03-07T08:02:41Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by SunMarc at 2026-03-10T15:16:06Z ---
We merged this PR which fixes the issue described here https://github.com/huggingface/transformers/pull/44473#event-23400935204 @kmod ! closing this issue

--- Comment by kmod at 2026-03-11T21:14:28Z ---
Hi, thanks for the fix! I just tested main, and I see that the weight re-initialization does not happen, which is great. But I still see that all ranks fully load the model into host RAM, and since I don't have enough system ram to load four separate copies of the model the workers end up getting killed by the oom killer, so I think this feature is still not quite working out of the box.

I think the initial loading in convert_and_load_state_dict_in_model() needs to be skipped somehow, rather than loading the weights and later discarding them. I've been running with the workaround I put in the initial report and so far haven't run into any issues.

--- Comment by Cyrilvallez at 2026-03-12T09:57:21Z ---
@SunMarc is this something that's possible? @winglian said on the PR that the behavior is now the same as before on v4 etc, but maybe we can do better?

--- Comment by SunMarc at 2026-03-12T14:52:06Z ---
Hmmm can you share a minimal reproducer @kmod ? From @winglian experiment, there shouldn't be any issues with the ram. The model weights on the other ranks should be empty even if they are on the cpu.

--- Comment by michaelanderson01826-glitch at 2026-03-12T16:05:10Z ---
SunMarc is it that's possible?  said on the PR that the behavior is now the
same as before on v4 etc

On Thu, Mar 12, 2026, 2:52 PM Marc Sun ***@***.***> wrote:

> *SunMarc* left a comment (huggingface/transformers#43749)
> <https://github.com/huggingface/transformers/issues/43749#issuecomment-4047383489>
>
> Hmmm can you share a minimal reproducer @kmod <https://github.com/kmod> ?
> From @winglian <https://github.com/winglian> experiment, there shouldn't
> be any issues with the ram. The model weights on the other ranks should be
> empty even if they are on the cpu.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43749#issuecomment-4047383489>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEENZO6WUWDPBWHXEZL34QLFLLAVCNFSM6AAAAACUA4DOYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANBXGM4DGNBYHE>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

--- Comment by kmod at 2026-03-12T21:45:27Z ---
Sure thing: [fsdp_test.py](https://github.com/user-attachments/files/25953158/fsdp_test.py)

I ran these tests on a server with 4x RTX Pro 6000 Blackwell gpus, and 768GB of ram, on transformers commit 852f785

Here's it running on Qwen/Qwen3-30B-A3B-Instruct-2507, a 57GiB model. The server has enough ram to load 4 copies of this model, and the peak memory usage is 218GiB, which is pretty close to 4x the model size

Qwen/Qwen3-30B-A3B-Instruct-2507

Transformers version: 5.3.0.dev0
Loading model: Qwen/Qwen3-30B-A3B-Instruct-2507
RAM: 18.3 GB used, 736.6 GB free
RAM: 18.8 GB used, 736.0 GB free
Loading weights:   1%|▏                          | 4/531 [00:00<00:14, 36.70it/s]
RAM: 22.4 GB used, 732.4 GB free
Loading weights:   4%|▉                         | 20/531 [00:00<00:25, 20.34it/s]
RAM: 34.2 GB used, 720.6 GB free
Loading weights:   9%|██▍                       | 49/531 [00:02<00:20, 23.85it/s]
RAM: 45.8 GB used, 709.0 GB free
Loading weights:  13%|███▍                      | 71/531 [00:02<00:17, 25.56it/s]
RAM: 56.0 GB used, 698.9 GB free
Loading weights:  20%|████▉                    | 104/531 [00:04<00:15, 26.92it/s]
RAM: 68.7 GB used, 686.1 GB free
Loading weights:  24%|█████▉                   | 126/531 [00:05<00:15, 26.86it/s]
RAM: 79.3 GB used, 675.6 GB free
Loading weights:  30%|███████▍                 | 159/531 [00:06<00:13, 27.42it/s]
RAM: 91.1 GB used, 663.7 GB free
Loading weights:  34%|████████▌                | 181/531 [00:07<00:12, 27.01it/s]
RAM: 101.5 GB used, 653.3 GB free
Loading weights:  40%|██████████               | 214/531 [00:08<00:11, 27.56it/s]
RAM: 113.1 GB used, 641.7 GB free
Loading weights:  44%|███████████              | 236/531 [00:09<00:10, 27.14it/s]
RAM: 123.7 GB used, 631.1 GB free
Loading weights:  51%|████████████▋            | 269/531 [00:10<00:09, 26.75it/s]
RAM: 135.0 GB used, 619.9 GB free
Loading weights:  55%|█████████████▋           | 291/531 [00:11<00:08, 26.92it/s]
RAM: 145.1 GB used, 609.7 GB free
Loading weights:  61%|███████████████▎         | 324/531 [00:12<00:07, 27.58it/s]
RAM: 157.4 GB used, 597.5 GB free
Loading weights:  65%|████████████████▎        | 346/531 [00:13<00:06, 27.40it/s]
RAM: 168.3 GB used, 586.5 GB free
Loading weights:  69%|█████████████████▎       | 368/531 [00:13<00:05, 27.44it/s]
RAM: 180.4 GB used, 574.5 GB free
Loading weights:  76%|██████████████████▉      | 401/531 [00:15<00:04, 26.36it/s]
RAM: 190.8 GB used, 564.1 GB free
Loading weights:  80%|███████████████████▉     | 423/531 [00:16<00:04, 26.72it/s]
RAM: 202.3 GB used, 552.6 GB free
Loading weights:  86%|█████████████████████▍   | 456/531 [00:17<00:02, 26.91it/s]
RAM: 212.9 GB used, 541.9 GB free
Loading weights:  90%|██████████████████████▌  | 478/531 [00:18<00:01, 27.10it/s]
RAM: 224.4 GB used, 530.5 GB free
Loading weights:  96%|████████████████████████ | 511/531 [00:19<00:00, 27.53it/s]
RAM: 236.5 GB used, 518.3 GB free
Loading weights: 100%|█████████████████████████| 531/531 [00:19<00:00, 27.30it/s]
Loading weights: 100%|█████████████████████████| 531/531 [00:19<00:00, 27.34it/s]
Loading weights: 100%|█████████████████████████| 531/531 [00:19<00:00, 27.18it/s]
Loading weights: 100%|█████████████████████████| 531/531 [00:19<00:00, 27.10it/s]
RAM: 150.9 GB used, 603.9 GB free
RAM: 75.5 GB used, 679.3 GB free
RAM: 75.5 GB used, 679.4 GB free
RAM: 75.5 GB used, 679.4 GB free
RAM: 75.4 GB used, 679.4 GB free
RAM: 75.4 GB used, 679.4 GB free
RAM: 75.4 GB used, 679.4 GB free
RAM: 38.9 GB used, 716.0 GB free

Here's a model that is too large to fit 4 times into ram:

zai-org/GLM-4.5-Air

Transformers version: 5.3.0.dev0
Loading model: zai-org/GLM-4.5-Air
RAM: 19.1 GB used, 735.7 GB free
RAM: 19.7 GB used, 735.1 GB free
Loading weights:   0%|                                  | 0/735
RAM: 23.6 GB used, 731.3 GB free
Loading weights:   2%|▌                        | 16/735 [00:00<00:16, 43.64it/s]
RAM: 41.7 GB used, 713.1 GB free
Loading weights:   4%|█                        | 32/735 [00:01<00:43, 16.27it/s]
RAM: 47.9 GB used, 706.9 GB free
Loading weights:   5%|█▏                       | 35/735 [00:03<01:21,  8.61it/s]
RAM: 56.4 GB used, 698.5 GB free
Loading weights:   7%|█▋                       | 48/735 [00:03<00:47, 14.37it/s]
RAM: 75.0 GB used, 679.9 GB free
Loading weights:   9%|██▏                      | 64/735 [00:04<00:47, 14.09it/s]
RAM: 83.1 GB used, 671.7 GB free
Loading weights:   9%|██▎                      | 67/735 [00:06<01:15,  8.88it/s]
RAM: 90.8 GB used, 664.0 GB free
Loading weights:  11%|██▋                      | 80/735 [00:06<00:46, 14.15it/s]
RAM: 111.3 GB used, 643.5 GB free
Loading weights:  13%|███▎                     | 96/735 [00:07<00:45, 13.97it/s]
RAM: 119.4 GB used, 635.5 GB free
Loading weights:  15%|███▋                    | 112/735 [00:09<00:45, 13.83it/s]
RAM: 127.3 GB used, 627.6 GB free
RAM: 143.4 GB used, 611.5 GB free
Loading weights:  17%|████▏                   | 128/735 [00:10<00:48, 12.46it/s]
RAM: 147.8 GB used, 607.1 GB free
Loading weights:  18%|████▎                   | 131/735 [00:12<01:17,  7.83it/s]
RAM: 153.0 GB used, 601.9 GB free
Loading weights:  20%|████▋                   | 144/735 [00:12<00:49, 11.84it/s]
RAM: 166.7 GB used, 588.2 GB free
Loading weights:  20%|████▊                   | 147/735 [00:14<01:19,  7.39it/s]
RAM: 171.3 GB used, 583.6 GB free
Loading weights:  22%|█████▏                  | 160/735 [00:14<00:52, 11.05it/s]
RAM: 184.6 GB used, 570.3 GB free
Loading weights:  22%|█████▎                  | 163/735 [00:15<01:20,  7.07it/s]
RAM: 189.4 GB used, 565.5 GB free
Loading weights:  24%|█████▋                  | 176/735 [00:16<00:52, 10.70it/s]
RAM: 202.6 GB used, 552.3 GB free
Loading weights:  24%|█████▊                  | 178/735 [00:17<01:22,  6.71it/s]
RAM: 207.5 GB used, 547.3 GB free
Loading weights:  26%|██████▎                 | 192/735 [00:18<00:51, 10.64it/s]
RAM: 221.2 GB used, 533.6 GB free
Loading weights:  28%|██████▊                 | 208/735 [00:20<00:49, 10.63it/s]
RAM: 225.6 GB used, 529.3 GB free
RAM: 240.2 GB used, 514.6 GB free
Loading weights:  30%|███████▎                | 224/735 [00:22<00:47, 10.68it/s]
RAM: 243.4 GB used, 511.4 GB free
RAM: 258.9 GB used, 496.0 GB free
Loading weights:  33%|███████▊                | 240/735 [00:23<00:46, 10.60it/s]
RAM: 262.1 GB used, 492.8 GB free
RAM: 277.8 GB used, 477.1 GB free
Loading weights:  35%|████████▎               | 256/735 [00:25<00:45, 10.48it/s]
RAM: 279.5 GB used, 475.3 GB free
RAM: 292.3 GB used, 462.5 GB free
Loading weights:  37%|████████▉               | 272/735 [00:27<00:45, 10.17it/s]
RAM: 294.6 GB used, 460.2 GB free
RAM: 311.1 GB used, 443.8 GB free
Loading weights:  39%|█████████▍              | 288/735 [00:29<00:42, 10.41it/s]
RAM: 313.2 GB used, 441.7 GB free
Loading weights:  39%|█████████▍              | 290/735 [00:31<01:06,  6.66it/s]
RAM: 321.1 GB used, 433.8 GB free
Loading weights:  41%|█████████▉              | 304/735 [00:31<00:40, 10.54it/s]
RAM: 330.8 GB used, 424.0 GB free
Loading weights:  42%|█████████▉              | 306/735 [00:33<01:04,  6.68it/s]
RAM: 335.9 GB used, 418.9 GB free
Loading weights:  44%|██████████▍             | 320/735 [00:33<00:39, 10.63it/s]
RAM: 349.2 GB used, 405.6 GB free
Loading weights:  44%|██████████▌             | 323/735 [00:34<00:58,  7.01it/s]
RAM: 354.0 GB used, 400.8 GB free
Loading weights:  46%|██████████▉             | 336/735 [00:35<00:37, 10.53it/s]
RAM: 367.4 GB used, 387.5 GB free
Loading weights:  46%|███████████             | 338/735 [00:36<00:59,  6.63it/s]
RAM: 372.2 GB used, 382.7 GB free
Loading weights:  48%|███████████▍            | 352/735 [00:37<00:36, 10.54it/s]
RAM: 385.2 GB used, 369.6 GB free
Loading weights:  50%|████████████            | 368/735 [00:39<00:34, 10.51it/s]
RAM: 390.0 GB used, 364.9 GB free
RAM: 403.9 GB used, 351.0 GB free
Loading weights:  52%|████████████▌           | 384/735 [00:41<00:33, 10.60it/s]
RAM: 407.7 GB used, 347.1 GB free
RAM: 422.8 GB used, 332.0 GB free
Loading weights:  54%|█████████████           | 400/735 [00:43<00:31, 10.52it/s]
RAM: 425.6 GB used, 329.2 GB free
RAM: 440.6 GB used, 314.2 GB free
Loading weights:  57%|█████████████▌          | 416/735 [00:44<00:30, 10.41it/s]
RAM: 442.8 GB used, 312.0 GB free
RAM: 459.0 GB used, 295.9 GB free
Loading weights:  59%|██████████████          | 432/735 [00:46<00:28, 10.49it/s]
RAM: 460.8 GB used, 294.1 GB free
RAM: 476.7 GB used, 278.1 GB free
Loading weights:  61%|██████████████▋         | 448/735 [00:48<00:27, 10.59it/s]
RAM: 478.8 GB used, 276.0 GB free
Loading weights:  61%|██████████████▋         | 450/735 [00:50<00:43,  6.62it/s]
RAM: 485.5 GB used, 269.3 GB free
Loading weights:  63%|███████████████▏        | 464/735 [00:50<00:25, 10.53it/s]
RAM: 497.0 GB used, 257.8 GB free
Loading weights:  63%|███████████████▏        | 466/735 [00:52<00:40,  6.61it/s]
RAM: 502.4 GB used, 252.5 GB free
Loading weights:  65%|███████████████▋        | 480/735 [00:52<00:24, 10.45it/s]
RAM: 514.6 GB used, 240.2 GB free
Loading weights:  66%|███████████████▋        | 482/735 [00:54<00:38,  6.61it/s]
RAM: 518.9 GB used, 235.9 GB free
Loading weights:  67%|████████████████▏       | 496/735 [00:54<00:22, 10.42it/s]
RAM: 531.7 GB used, 223.2 GB free
Loading weights:  68%|████████████████▎       | 498/735 [00:55<00:35,  6.61it/s]
RAM: 536.3 GB used, 218.6 GB free
Loading weights:  70%|████████████████▋       | 512/735 [00:56<00:21, 10.47it/s]
RAM: 549.4 GB used, 205.5 GB free
Loading weights:  70%|████████████████▊       | 514/735 [00:57<00:33,  6.62it/s]
RAM: 554.3 GB used, 200.5 GB free
Loading weights:  72%|█████████████████▏      | 528/735 [00:58<00:19, 10.54it/s]
RAM: 562.3 GB used, 192.6 GB free
RAM: 563.6 GB used, 191.2 GB free
RAM: 563.9 GB used, 190.9 GB free
RAM: 564.1 GB used, 190.8 GB free
RAM: 564.3 GB used, 190.5 GB free
RAM: 564.6 GB used, 190.3 GB free
RAM: 570.3 GB used, 184.5 GB free
Loading weights:  74%|█████████████████▊      | 544/735 [01:05<00:43,  4.39it/s]
RAM: 569.0 GB used, 185.9 GB free
RAM: 572.2 GB used, 182.6 GB free
RAM: 572.6 GB used, 182.3 GB free
RAM: 573.0 GB used, 181.8 GB free
RAM: 573.4 GB used, 181.4 GB free
RAM: 578.1 GB used, 176.8 GB free
Loading weights:  74%|█████████████████▊      | 546/735 [01:11<01:28,  2.14it/s]
RAM: 579.9 GB used, 175.1 GB free
Loading weights:  76%|██████████████████▎     | 560/735 [01:12<00:46,  3.77it/s]
RAM: 591.6 GB used, 163.2 GB free
Loading weights:  76%|██████████████████▎     | 562/735 [01:14<00:55,  3.13it/s]
RAM: 594.0 GB used, 161.0 GB free
Loading weights:  78%|██████████████████▊     | 576/735 [01:14<00:29,  5.39it/s]
RAM: 605.4 GB used, 149.4 GB free
RAM: 619.4 GB used, 135.5 GB free
Loading weights:  81%|███████████████████▎    | 592/735 [01:16<00:21,  6.74it/s]
RAM: 619.7 GB used, 135.2 GB free
RAM: 630.7 GB used, 124.2 GB free
Loading weights:  83%|███████████████████▊    | 608/735 [01:19<00:16,  7.49it/s]
RAM: 631.0 GB used, 123.8 GB free
RAM: 641.5 GB used, 113.3 GB free
RAM: 647.4 GB used, 107.4 GB free
Loading weights:  85%|████████████████████▍   | 624/735 [01:21<00:14,  7.77it/s]
RAM: 652.1 GB used, 102.8 GB free
RAM: 660.7 GB used, 94.1 GB free
RAM: 668.1 GB used, 86.7 GB free
Loading weights:  87%|████████████████████▉   | 640/735 [01:25<00:14,  6.75it/s]
RAM: 664.3 GB used, 90.5 GB free
RAM: 675.1 GB used, 79.8 GB free
RAM: 684.1 GB used, 70.8 GB free
Loading weights:  87%|████████████████████▉   | 642/735 [01:27<00:23,  4.02it/s]
RAM: 682.3 GB used, 72.5 GB free
Loading weights:  89%|█████████████████████▍  | 656/735 [01:28<00:11,  6.72it/s]
RAM: 690.0 GB used, 64.8 GB free
RAM: 697.1 GB used, 57.8 GB free
RAM: 694.6 GB used, 60.2 GB free
Loading weights:  91%|█████████████████████▉  | 672/735 [01:31<00:09,  6.31it/s]
RAM: 703.9 GB used, 51.0 GB free
RAM: 711.3 GB used, 43.5 GB free
RAM: 720.1 GB used, 34.7 GB free
RAM: 720.7 GB used, 34.1 GB free
Loading weights:  94%|██████████████████████▍ | 688/735 [01:35<00:08,  5.59it/s]
RAM: 717.9 GB used, 37.0 GB free
RAM: 729.4 GB used, 25.4 GB free
Loading weights:  94%|██████████████████████▌ | 690/735 [01:38<00:11,  3.87it/s]
RAM: 728.2 GB used, 26.7 GB free
Loading weights:  96%|██████████████████████▉ | 704/735 [01:38<00:04,  6.44it/s]
RAM: 738.6 GB used, 16.2 GB free
RAM: 746.9 GB used, 8.0 GB free
Loading weights:  96%|███████████████████████ | 706/735 [01:41<00:07,  3.94it/s]
RAM: 741.3 GB used, 13.6 GB free
Loading weights:  98%|███████████████████████▌| 720/735 [01:41<00:02,  6.55it/s]
RAM: 753.4 GB used, 1.5 GB free
Timeout, server 192.168.0.226 not responding.

ie it got killed by the oom killer and disconnected my ssh

While it's not visible in the final terminal output I pasted here, if you look closely while it runs you can see that there are 4 separate versions of the progress bar that are being displayed at the same time and will overwrite each other with very slightly different it/s values.

My guess is that the discrepancy with @winglian's numbers is that they were looking at final ram usage, but I'm not sure. The rank>0 weights are empty once from_pretrained() finishes, but I think the issue is that _load_pretrained_model() will load all of the weights on all of the ranks, and the logic in _move_missing_keys_from_meta_to_device() that discards the weights is running too late to prevent the oom.

--- Comment by winglian at 2026-03-16T16:53:51Z ---
Try something like this with explicit device mapping. https://gist.github.com/winglian/daba274b1fe7d8680f08e2f06e9bc598 (below was on a machine with 2x5090s)

With your original script, it was using `RAM: 128.9 GB used, 122.4 GB free`

```
Loading weights:  92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                | 488/531 [00:35<00:02, 16.42it/s]
RAM: 70.0 GB used, 181.4 GB free
Loading weights:  94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏           | 499/531 [00:36<00:01, 16.53it/s]
RAM: 71.7 GB used, 179.7 GB free
Loading weights:  98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎   | 521/531 [00:37<00:00, 16.63it/s]
RAM: 72.8 GB used, 178.5 GB free
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 531/531 [00:38<00:00, 13.82it/s]
/home/wing/env-cu130/venv-cu130/lib/python3.12/site-packages/torch/distributed/device_mesh.py:603: UserWarning: Slicing a flattened dim from root mesh will be deprecated in PT 2.11. Users need to bookkeep the flattened mesh directly.
  sliced_mesh_layout = self._get_slice_mesh_layout(mesh_dim_names)
RAM: 74.0 GB used, 177.4 GB free
RAM: 74.5 GB used, 176.8 GB free
RAM: 74.5 GB used, 176.8 GB free
```

--- Comment by kmod at 2026-03-16T19:42:36Z ---
Ah ok, yes explicitly assigning rank>0 to the meta device does seem to address the issue.

--- Comment by michaelanderson01826-glitch at 2026-03-17T14:12:23Z ---
Thanks for the detailed report very helpful

On Mon, Mar 16, 2026, 7:42 PM Kevin Modzelewski ***@***.***>
wrote:

> *kmod* left a comment (huggingface/transformers#43749)
> <https://github.com/huggingface/transformers/issues/43749#issuecomment-4070139787>
>
> Ah ok, yes explicitly assigning rank>0 to the meta device does seem to
> address the issue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/43749#issuecomment-4070139787>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/B7HTEEND2MSOIOQO4DJUW2D4RBKMFAVCNFSM6AAAAACUA4DOYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANZQGEZTSNZYG4>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


ISSUE #43746: [GraniteSpeechForConditionalGeneration] Models with PEFT adapters won't load from local checkpoints (from_pretrained)
State: closed  |  Labels: bug
Author: gabe-l-hart  |  Created: 2026-02-04T17:55:19Z

### System Info

- `transformers` version: 4.57.6
- Platform: macOS-26.2-arm64-arm-64bit
- Python version: 3.11.14
- Huggingface_hub version: 0.36.1
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@eustlb @ebezzam @vasqu

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Download the model locally

```sh
hf download ibm-granite/granite-speech-3.3-2b --local-dir ibm-granite/granite-speech-3.3-2b
```

2. Run a lightly-modified version of the example script, pointing at the local path on disk

```py
import torch
import torchaudio
from pathlib import Path
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
#from huggingface_hub import hf_hub_download

device = "cuda" if torch.cuda.is_available() else ("mps" if torch.mps.is_available() else "cpu")

#model_name = "ibm-granite/granite-speech-3.3-2b"
model_name = "/Users/ghart/models/ibm-granite/granite-speech-3.3-2b"
processor = AutoProcessor.from_pretrained(model_name)
tokenizer = processor.tokenizer
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_name, device_map=device, torch_dtype=torch.bfloat16
)
# load audio
#audio_path = hf_hub_download(repo_id=model_name, filename="10226_10111_000000.wav")
audio_path = str(Path(model_name) / "10226_10111_000000.wav")
wav, sr = torchaudio.load(audio_path, normalize=True)
assert wav.shape[0] == 1 and sr == 16000  # mono, 16khz

# create text prompt
system_prompt = "Knowledge Cutoff Date: April 2024.\nToday's Date: April 9, 2025.\nYou are Granite, developed by IBM. You are a helpful AI assistant"
user_prompt = "<|audio|>can you transcribe the speech into a written format?"
chat = [
    dict(role="system", content=system_prompt),
    dict(role="user", content=user_prompt),
]
prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

# run the processor+model
model_inputs = processor(prompt, wav, device=device, return_tensors="pt").to(device)
model_outputs = model.generate(**model_inputs, max_new_tokens=200, do_sample=False, num_beams=1)

# Transformers includes the input IDs in the response.
num_input_tokens = model_inputs["input_ids"].shape[-1]
new_tokens = torch.unsqueeze(model_outputs[0, num_input_tokens:], dim=0)
output_text = tokenizer.batch_decode(
    new_tokens, add_special_tokens=False, skip_special_tokens=True
)
print(f"STT output = {output_text[0].upper()}")
```

3. The model will still be pulled from the hub, despite already being downloaded

### Expected behavior

The model should use the existing local directory checkpoint rather than downloading from the hub.

## Analysis

The bug is caused by [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/auto_factory.py#L305C37-L305C37) where the value of `pretrained_model_name_or_path` is overridden by the value pulled from the adapter config ([here](https://huggingface.co/ibm-granite/granite-speech-3.3-2b/blob/main/adapter_config.json#L4) for `granite-speech-3.3-2b`).

## Solution Ideas

I'm not sure the cleanest way to handle this since the adapter is theoretically only valid against the specific checkpoint it was trained with. What we really need is a way to validate that the absolute-path to the local copy is the same checkpoint as the upstream parent. I'm not aware of any kind of checksum behavior that exists to do this currently (though I could easily just not be aware of it).

Lacking the ability to do a full verification, I would propose that we simply check if `pretrained_model_name_or_path` is a path-on-disk and skip the override logic if so (possibly with a warning).

--- Comment by gabe-l-hart at 2026-02-04T18:05:45Z ---
I'll work up a quick PR with the conditional check as a proposed fix.

--- Comment by gabe-l-hart at 2026-02-04T20:11:19Z ---
Looking a little deeper, I think the conditional needs a little more nuance. I think the _goal_ of this line is to allow a model artifact _on disk_ that only contains the adapter and not the base model to fetch its parent model automatically. We'd want to preserve this, so we need to distinguish this case from the case where it's a unified model that bundles an adapter with it.


ISSUE #43742: Key error when loading facebook/MobileLLM-125M
State: closed  |  Labels: bug
Author: pahrendt-semron  |  Created: 2026-02-04T16:19:57Z

### System Info

- `transformers` version: 5.0.0
- Platform: Linux-4.18.0-553.el8_10.x86_64-x86_64-with-glibc2.28
- Python version: 3.12.12
- Huggingface_hub version: 1.3.7
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA H200 NVL

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

>>from transformers import AutoModelForCausalLM, AutoTokenizer
>>model = AutoModelForCausalLM.from_pretrained("facebook/MobileLLM-125M", trust_remote_code=True)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/.venv/lib64/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 365, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.venv/lib64/python3.12/site-packages/transformers/modeling_utils.py", line 4072, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hf/modules/transformers_modules/facebook/MobileLLM_hyphen_125M/f6820a829b347aaa6674a705c82be3db47b1f9ee/modeling_mobilellm.py", line 1132, in __init__
    self.model = MobileLLMModel(config)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/hf/modules/transformers_modules/facebook/MobileLLM_hyphen_125M/f6820a829b347aaa6674a705c82be3db47b1f9ee/modeling_mobilellm.py", line 883, in __init__
    [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hf/modules/transformers_modules/facebook/MobileLLM_hyphen_125M/f6820a829b347aaa6674a705c82be3db47b1f9ee/modeling_mobilellm.py", line 682, in __init__
    self.self_attn = LLAMA_ATTENTION_CLASSES[config._attn_implementation](config=config, layer_idx=layer_idx)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hf/modules/transformers_modules/facebook/MobileLLM_hyphen_125M/f6820a829b347aaa6674a705c82be3db47b1f9ee/modeling_mobilellm.py", line 273, in __init__
    self._init_rope()
  File "/hf/modules/transformers_modules/facebook/MobileLLM_hyphen_125M/f6820a829b347aaa6674a705c82be3db47b1f9ee/modeling_mobilellm.py", line 283, in _init_rope
    scaling_type = self.config.rope_scaling["type"]
                   ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'type'

### Expected behavior

No error.

--- Comment by Rocketknight1 at 2026-02-05T12:59:02Z ---
This is likely an issue with the remote model code not being updated for v5! You'll need to either ask the maintainers to update it, or make a PR to add the model into `transformers`, and do the changes there

--- Comment by GuangLun2000 at 2026-02-07T16:47:05Z ---
I got the same problem. That's because the update of v5. You can return to v4.

--- Comment by redpanda1995 at 2026-02-07T20:28:55Z ---
hey folks, sent a PR for this. Thanks.

--- Comment by vasqu at 2026-02-09T11:49:18Z ---
See my comment at https://github.com/huggingface/transformers/pull/43823#issuecomment-3871253078

It's a remote code issue, either stick to the v4 versions that work or it has to be updated on the hub

--- Comment by github-actions[bot] at 2026-03-07T08:02:42Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43726: Help: Transformer v5 managed to break HF models
State: closed  |  Labels:
Author: TomLucidor  |  Created: 2026-02-04T06:57:52Z

There are some models that are rendered broken because of `from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode` and `ImportError: cannot import name 'bytes_to_unicode' from 'transformers.models.gpt2.tokenization_gpt2'`
https://huggingface.co/NexVeridian/Kimi-Linear-REAP-35B-A3B-Instruct-4bit/discussions/1 https://huggingface.co/nightmedia/Kimi-Linear-REAP-35B-A3B-Instruct-mxfp4-mlx/discussions/1

--- Comment by Rocketknight1 at 2026-02-04T13:03:53Z ---
Hey, can you paste the code you're running that causes this error? I'd guess that the cause is a custom model was depending on a function that got moved around for `v5`, in which case the custom model in that repo might need an update to remain compatible.

--- Comment by TomLucidor at 2026-02-04T14:07:19Z ---
`tokenization_kimi.py` hidden somewhere under `transformers_modules` according to the HF testers, I have similar experiences

--- Comment by jiangwu300 at 2026-02-05T23:06:40Z ---
I can also confirm that transformers v5 breaks Kimi K2.5. It launches as expected on 4.57.6, breaks in 5.0.0 and 5.1.0.

--- Comment by arch-btw at 2026-02-06T12:30:41Z ---
```
self.byte_encoder = bytes_to_unicode()
self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
```

https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/blob/main/tokenization_kimi.py#L132-L133

--- Comment by hmellor at 2026-02-06T14:23:34Z ---
FYI I have updated this import in a bunch of Kimi checkpoints

If you're using any of these, try updating your downloaded models. If not, make a similar PR to the checkpoint you're using!

--- Comment by TomLucidor at 2026-02-06T15:20:42Z ---
@hmellor Kimi-Linear-REAP is a Cerebas project, so they would also need to update the methods.

--- Comment by jiangwu300 at 2026-02-06T15:47:38Z ---
> FYI I have updated this import in a bunch of Kimi checkpoints
>
>
>
> If you're using any of these, try updating your downloaded models. If not, make a similar PR to the checkpoint you're using!

Where can I see these changes? Couldn't find the PRs.

--- Comment by hmellor at 2026-02-06T16:35:43Z ---
Here is one of them https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Base/discussions/3/files

You navigate to the `community` tab and check `view closed` to view them.

--- Comment by github-actions[bot] at 2026-03-06T08:05:34Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


Quantization model behavior changed

ISSUE #43725: Quantization model behavior changed
State: closed  |  Labels: bug
Author: jiqing-feng  |  Created: 2026-02-04T03:29:00Z

### System Info

torch 2.10.0
peft 0.18.2.dev0
bitsandbytes 0.49.1
The only variable is transformers.

### Who can help?

@ArthurZucker

### Reproduction

The regression was found in peft tests:
https://github.com/jiqing-feng/peft/blob/8bit/tests/test_gpu_examples.py#L2901
`RUN_SLOW=1 pytest tests/test_gpu_examples.py::TestLoftQ::test_bloomz_loftq_8bit`

### Expected behavior

The previous tests could pass before; after the PR https://github.com/huggingface/transformers/pull/42805:
```
FAILED tests/test_gpu_examples.py::TestLoftQ::test_bloomz_loftq_8bit[cuda] - AssertionError: assert tensor(3.6478e-09, device='cuda:0', grad_fn=<MeanBackward0>) < (tensor(3.2703e-09, device='cud...
FAILED tests/test_gpu_examples.py::TestLoftQ::test_bloomz_loftq_8bit[cpu] - assert tensor(2.7105e-09, grad_fn=<MeanBackward0>) < (tensor(2.0073e-09, grad_fn=<MeanBackward0>) / 1.005)
```

However, I suppose it's a correct change. Before this change, the quantized models were always loaded as float16 model (embedding and lm_head weight type if no dtype specified). After this change, the quantized models are loaded as float32 model if no dtype is specified. I just want to make sure we have aligned with it.
After we have aligned and agreed with the PR https://github.com/huggingface/transformers/pull/42805, I will update the peft tests.

cc @BenjaminBossan @matthewdouglas

--- Comment by ArthurZucker at 2026-02-06T08:08:47Z ---
cc @SunMarc as well

--- Comment by SunMarc at 2026-02-10T14:34:29Z ---
> The quantized models were always loaded as float16 model (embedding and lm_head weight type if no dtype specified). After this change, the quantized models are loaded as float32 model if no dtype is specified. I just want to make sure we have aligned with it.

If the dtype is not specified, it should be loaded using the dtype stored in the config.json or if the dtype was overwritten by the quantizer. The regression you saw was from this PR but I think this is the right behavior that we should keep : https://github.com/huggingface/transformers/pull/42882/changes#diff-f8f1cbbbc1e4d884ff472530314eb600f7ed84a38fabf5ab381962e34e329c9a

--- Comment by github-actions[bot] at 2026-03-07T08:02:44Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43724: [integration] add pluto experiment tracker callback.
State: open  |  Labels: Feature request
Author: asaiacai  |  Created: 2026-02-04T01:19:47Z

### Feature request

add `PlutoCallback` as an option for natively logging to the pluto experiment tracker.

### Motivation

We recently launched a new OSS experiment tracker ([client code](https://github.com/Trainy-ai/pluto), [server code](https://github.com/Trainy-ai/pluto-server)), and we're hoping to get integrated as a logging callback option for those using `transformers`. We're currently offering a hosted version as well in the link below.

https://pluto.trainy.ai/

hoping to hear back if it is okay if we open a PR for this. Quite a few of our users moved from Neptune given the shutdown so we want to close the gap so people have to time to try it out before the Neptune shutdown date.

### Your contribution

I will be drafting the PR and managing it until it is merged.

--- Comment by vasanthrpjan1-boop at 2026-02-08T14:25:35Z ---
I can work on this feature request


ISSUE #43723: Issue loading tokenizer using AutoTokenizer.from_pretrained in v5
State: closed  |  Labels: bug
Author: msubrayada  |  Created: 2026-02-04T00:03:20Z

### System Info

There is an issue when loading the tokenizer from the model guillermoruiz/bilmaLAT and others in transformers v5.

when I run the code:
```
bilmaLAT = AutoTokenizer.from_pretrained('guillermoruiz/bilmaLAT')
bilmaLAT('hola')
```

the output is:
`
{'input_ids': [2, 76, 83, 80, 69, 3], 'attention_mask': [1, 1, 1, 1, 1, 1]}
`

When I use Transformers v4.55.0 the output is
`
{'input_ids': [2, 14518, 3], 'attention_mask': [1, 1, 1]}
`
which is the correct output.

I think the problem is when loading the normalizing functions of the tokenizer.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import transformers
from transformers import AutoTokenizer

bilmaLAT = AutoTokenizer.from_pretrained('guillermoruiz/bilmaLAT')
bilmaLAT('hola')
```

### Expected behavior

The output should be

`{'input_ids': [2, 14518, 3], 'attention_mask': [1, 1, 1]}`

--- Comment by Rocketknight1 at 2026-02-04T13:02:00Z ---
cc @arthurzucker @itazap

--- Comment by itazap at 2026-02-04T16:12:24Z ---
hey @msubrayada ! `bilmaLAT's` [tokenizer_config.json file on the hub](https://huggingface.co/guillermoruiz/bilmaLAT/blob/main/tokenizer_config.json) should be updated to:
 `tokenzier_class: "PreTrainedTokenizerFast"` (instead of `RobertaTokenizer` )

since `bilmaLAT's` uses a WordPiece tokenizer and RobertaTokenizer is BPE. This flew under the radar in v4 while in v5 we respect this mapping more strictly!

Thanks for the clear issue / reproducer BTW 👍

--- Comment by msubrayada at 2026-02-04T16:39:39Z ---
The problem was solved. Thank you for your response. 🤗


ISSUE #43720: [BUG][CI] BitNet AutoBitLinear fails when packed weights aren’t unpacked during accelerate loading
State: closed  |  Labels: bug
Author: harshaljanjani  |  Created: 2026-02-03T19:58:35Z

### System Info

* `transformers` version: `5.0.0.dev0`
* Platform: `Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39`
* Python version: `3.12.3`
* `huggingface_hub` version: `1.3.2`
* `safetensors` version: `0.7.0`
* `accelerate` version: `1.12.0`
* Accelerate config: `not installed`
* DeepSpeed version: `not installed`
* PyTorch version (accelerator?): `2.9.1+cu128 (CUDA)`
* GPU type: `NVIDIA L4`
* NVIDIA driver version: `550.90.07`
* CUDA version: `12.4`

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import BitNetForCausalLM

model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T")
input_ids = torch.tensor([[1, 2, 3]])
with torch.no_grad():
    output = model(input_ids)
print(output.logits.shape)
```

When loading `microsoft/bitnet-b1.58-2B-4T` with `device_map="auto"`, `AutoBitLinear.load_hook` is bypassed by accelerate's loading process; leaving weights in packed format (shape `[out_features//4, in_features]`). This materializes as inference + CI failures with `RuntimeError: shape '[1, 3, -1, 128]' is invalid for input of size 480`.

**CI Failure:**

**Current Output:**

### Expected behavior

→ The model should load and run inference successfully.
→ `tests/models/bitnet/test_modeling_bitnet.py::BitNetIntegrationTest::test_model_generation && tests/models/bitnet/test_modeling_bitnet.py::BitNetIntegrationTest::test_model_logits` integration tests pass without regressions

**Output After the Fix:**


init_weights usage in Mamba and Mamba-2

ISSUE #43717: init_weights usage in Mamba and Mamba-2
State: closed  |  Labels:
Author: kevinli573  |  Created: 2026-02-03T17:53:47Z

Hi,

`dt_bias` initialization has a pretty non-trivial impact on Mamba training/performance.

In the current implementation, it looks like `dt_bias` is [initialized to all ones](https://github.com/huggingface/transformers/blob/v5.0.0/src/transformers/models/mamba2/modeling_mamba2.py#L253) in the mixer class and then later [reinitialized to the standard range](https://github.com/huggingface/transformers/blob/v5.0.0/src/transformers/models/mamba2/modeling_mamba2.py#L730) in the parent PreTrainedModel class.

We were wondering if this is the right way to structure the code for Mamba models, as those who directly import the layer/mixer without using the full model will have the incorrect initialized `dt_bias` values. If this is not the case, would it be possible to move the initialization into the mixer?

Thanks!

--- Comment by tridao at 2026-02-03T18:03:05Z ---
I'd like to advocate calling the weight_init code when initializing the Mamba2Mixer class. There were multiple cases of folks using the wrong initialization (for dt_bias) for months because they only import Mamba2Mixer where dt_bias has the wrong initialization (all ones).
If this sounds fine, I'll work with @kevinli573 on a PR.

--- Comment by ArthurZucker at 2026-02-04T13:23:40Z ---
Sounds great actually, @Cyrilvallez was gonna refactor `transformers` in general to have a much more granular, module level init scheme!

Super happy if you can open a PR! 🤗

--- Comment by Cyrilvallez at 2026-02-04T13:54:24Z ---
Indeed! Weight init schemes have historically been centralized in the `_init_weights` of the `XXXPreTrainedModel`, but we've recently discussed a lot about it and we all feel that it would be much easier/readable to have them at the module-level!

--- Comment by github-actions[bot] at 2026-03-06T08:05:37Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43716: Mistral-3: dtype mismatch between image preprocessor and model
State: closed  |  Labels: bug
Author: egallouis  |  Created: 2026-02-03T17:47:11Z

### System Info

transformers\=\=5.0.0
pytorch\=\=2.10.0
python\=\=3.12.3
Platform: WSL

### Who can help?

@zucchini-nlp @Rocketknight1

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

### Description

When using mistralai/Ministral-3-3B-Reasoning-2512 for image-text-to-text,
The dtype of the processed images is float32 and the dtype of the model weights bfloat16. Because of that, the Inference pipeline crashes as soon as the processed images are given to the model.
The problem has to do with the use of the rescale function of the image preprocessing:
https://github.com/huggingface/transformers/blob/v5.0.0/src/transformers/image_transforms.py#L89-L95

Here, the standard dtype is defined as float32. the Pixtral Processor apparently does not use this argument and the model does not transform the dtype of the pixels values beforehand.

### Minimal Reproducible Example
```python
import torch
from transformers import pipeline

if __name__ == '__main__':
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
                {"type": "text", "text": "What are these?"},
            ],
        },
    ]
    pipe = pipeline(
        "image-text-to-text",
        model="mistralai/Ministral-3-3B-Reasoning-2512",  # same for the instruction version
        device="cpu",  # also "cpu", "cuda", and device_map="auto" have the same problem
        dtype=torch.bfloat16,  # does not change, if left out
    )

    result = pipe(messages)
    print(result)
```

### Traceback
```
Traceback (most recent call last):
  File "/home/user/project/issue.py", line 24, in <module>
    result = pipe(messages)
             ^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/pipelines/image_text_to_text.py", line 293, in __call__
    return super().__call__(Chat(images), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1274, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1281, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1173, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/pipelines/image_text_to_text.py", line 373, in _forward
    generated_sequence = self.model.generate(**model_inputs, **generate_kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2669, in generate
    result = decoding_method(
             ^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2864, in _sample
    outputs = self._prefill(input_ids, generation_config, model_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 3853, in _prefill
    return self(**model_inputs, return_dict=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 1002, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/models/mistral3/modeling_mistral3.py", line 446, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 1002, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/models/mistral3/modeling_mistral3.py", line 313, in forward
    image_features = self.get_image_features(
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 1002, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/models/mistral3/modeling_mistral3.py", line 232, in get_image_features
    image_outputs = self.vision_tower(
                    ^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 835, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/transformers/models/pixtral/modeling_pixtral.py", line 496, in forward
    patch_embeds = self.patch_conv(pixel_values)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 553, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/project/.venv/lib/python3.12/site-packages/torch/nn/modules/conv.py", line 548, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^
RuntimeError: Input type (torch.FloatTensor) and weight type (CPUBFloat16Type) should be the same or input should be a MKLDNN tensor and weight is a dense tensor
```

### Monkey-Patched working version
```python
import torch
from transformers import pipeline

if __name__ == '__main__':
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "url": "http://images.cocodataset.org/val2017/000000039769.jpg"},
                {"type": "text", "text": "What are these?"},
            ],
        },
    ]
    pipe = pipeline(
        "image-text-to-text",
        model="mistralai/Ministral-3-3B-Reasoning-2512",  # same for the instruction version
        device="cpu",  # also "cpu", "cuda", and device_map="auto" have the same problem
        dtype=torch.bfloat16,
    )

    # Patch the vision tower's forward method to convert inputs to bfloat16
    original_forward = pipe.model.model.vision_tower.forward

    def patched_forward(pixel_values, *args, **kwargs):
        # Convert pixel_values to bfloat16 to match the model weights
        pixel_values = pixel_values.to(torch.bfloat16)
        return original_forward(pixel_values, *args, **kwargs)

    pipe.model.model.vision_tower.forward = patched_forward

    result = pipe(messages)
    print(result)
```

### Expected behavior

Printed output dictionary with generated text tokens.

--- Comment by zucchini-nlp at 2026-02-04T09:20:43Z ---
The pipeline isn't casting the inputs to `dtype` when we take `chat_template` route. Will fix it, thanks for pointing out


ISSUE #43708: Trainer `resume_from_checkpoint` incorrectly calculates `max_steps` when changing `per_device_train_batch_size` with same global batch size
State: closed  |  Labels: bug
Author: KHPan  |  Created: 2026-02-03T14:18:09Z

### System Info

- `transformers` version: 4.57.3
- Platform: linux
- Python version: 3.11.14
- PyTorch version: 2.9.1

### Who can help?

@SunMarc

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

## Bug Description

When resuming training from a checkpoint with a **different `per_device_train_batch_size`** but the **same global batch size** (by adjusting `gradient_accumulation_steps`), the Trainer incorrectly recalculates `max_steps` based on the new batch size configuration, causing:

1. **Incorrect total training steps**: The resumed training runs more (or fewer) steps than expected
2. **Incorrect learning rate schedule**: The linear scheduler recalculates based on wrong `max_steps`

## Minimal Reproducible Example

### Script 1: Initial Training (will be interrupted)

```python
import torch
from torch import nn
from datasets import Dataset
from transformers import PreTrainedModel, PretrainedConfig, Trainer, TrainingArguments
import time

class TinyConfig(PretrainedConfig):
    model_type = "tiny"
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

data_dim = 32
data_cnt = 32

class TinyModel(PreTrainedModel):
    config_class = TinyConfig
    def __init__(self, config):
        super().__init__(config)
        self.fc = nn.Linear(data_dim, data_dim)

    def forward(self, input_ids, attention_mask=None, labels=None, **kwargs):
        time.sleep(1)
        loss = self.fc(input_ids.float()).mean(dim=-1)
        return {"loss": loss}

dataset = Dataset.from_dict({
    "input_ids": torch.zeros(data_cnt, data_dim, dtype=torch.long).tolist(),
    "attention_mask": torch.ones(data_cnt, data_dim, dtype=torch.long).tolist(),
    "labels": torch.zeros(data_cnt, data_dim, dtype=torch.long).tolist(),
})

model = TinyModel(TinyConfig())
trainer_args = TrainingArguments(
    output_dir="./issue_output",
    num_train_epochs=1,
    report_to="none",
    logging_steps=1,
    save_steps=1,
    learning_rate=0.08,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    lr_scheduler_type="linear",
)
trainer = Trainer(
    model=model,
    args=trainer_args,
    train_dataset=dataset,
)
trainer.train()
```

### Script 2: Resume Training with Different Batch Config (same global batch)

```python
import torch
from torch import nn
from datasets import Dataset
from transformers import PreTrainedModel, PretrainedConfig, Trainer, TrainingArguments
import time

class TinyConfig(PretrainedConfig):
    model_type = "tiny"
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

data_dim = 32
data_cnt = 32

class TinyModel(PreTrainedModel):
    config_class = TinyConfig
    def __init__(self, config):
        super().__init__(config)
        self.fc = nn.Linear(data_dim, data_dim)

    def forward(self, input_ids, attention_mask=None, labels=None, **kwargs):
        time.sleep(1)
        loss = self.fc(input_ids.float()).mean(dim=-1)
        return {"loss": loss}

dataset = Dataset.from_dict({
    "input_ids": torch.zeros(data_cnt, data_dim, dtype=torch.long).tolist(),
    "attention_mask": torch.ones(data_cnt, data_dim, dtype=torch.long).tolist(),
    "labels": torch.zeros(data_cnt, data_dim, dtype=torch.long).tolist(),
})

model = TinyModel(TinyConfig())
trainer_args = TrainingArguments(
    output_dir="./issue_output",
    num_train_epochs=1,
    report_to="none",
    logging_steps=1,
    save_steps=1,
    learning_rate=0.08,
    per_device_train_batch_size=4,   # Changed from 1 to 4
    gradient_accumulation_steps=1,    # Changed from 4 to 1
    lr_scheduler_type="linear",
)
trainer = Trainer(
    model=model,
    args=trainer_args,
    train_dataset=dataset,
)
trainer.train(resume_from_checkpoint=True)
```

## Steps to Reproduce

1. Run Script 1 and **interrupt it at 50%** (around step 4/8)
2. Run Script 2 to resume from the checkpoint

## Observed Behavior

### Script 1 Log (interrupted at 50%):
```
{'loss': -0.0153, 'grad_norm': 0.7071067690849304, 'learning_rate': 0.08, 'epoch': 0.12}
{'loss': -0.3353, 'grad_norm': 0.7071067690849304, 'learning_rate': 0.07, 'epoch': 0.25}
{'loss': -0.6153, 'grad_norm': 0.7071067690849304, 'learning_rate': 0.06, 'epoch': 0.38}
{'loss': -0.8553, 'grad_norm': 0.7071067690849304, 'learning_rate': 0.05, 'epoch': 0.5}
 50%|██████████████████████████████████████████▌                                          | 4/8 [00:16<00:16,  4.07s/it]
```

- Total steps: **8** (32 samples / (batch_size=1 × gradient_accumulation=4) = 8)
- Completed: **4 steps** (50%)

### Script 2 Log (resumed training):
```
{'loss': -0.2638, 'grad_norm': 0.1767766922712326, 'learning_rate': 0.04, 'epoch': 0.16}
{'loss': -0.3001, 'grad_norm': 0.1767766922712326, 'learning_rate': 0.0675, 'epoch': 0.19}
{'loss': -0.3568, 'grad_norm': 0.1767766922712326, 'learning_rate': 0.065, 'epoch': 0.22}
... (many more steps)
{'loss': -0.9351, 'grad_norm': 0.1767766922712326, 'learning_rate': 0.0025, 'epoch': 1.0}
100%|███████████████████████████████████████████████████████████████████████████████████| 32/32 [00:28<00:00,  1.11it/s]
```

- **max_steps was recalculated to 32** instead of maintaining the original 8
- The training ran **28 additional steps** (32 - 4 = 28) instead of **4** (8 - 4 = 4)
- The learning rate schedule was recalculated based on wrong max_steps

## Analysis

| Configuration | Script 1 | Script 2 |
|---------------|----------|----------|
| `per_device_train_batch_size` | 1 | 4 |
| `gradient_accumulation_steps` | 4 | 1 |
| **Global batch size** | **4** | **4** |
| Calculated `max_steps` | 8 | 32 (incorrect!) |

Since the global batch size remains the same (4), the total number of optimization steps should remain **8**. However, when resuming, the Trainer recalculates `max_steps` based on the new `per_device_train_batch_size` without considering that the global batch size is unchanged.

The learning rate schedule also gets recalculated based on the new (incorrect) `max_steps`, causing the linear decay to behave incorrectly.

### Expected behavior

When resuming from a checkpoint with the same global batch size (but different `per_device_train_batch_size` and `gradient_accumulation_steps`), the Trainer should:

1. **Maintain the original `max_steps`** from the initial training run, or at least calculate based on the global batch size
2. **Continue the learning rate schedule correctly** based on the original `max_steps`

In this case:
- Original `max_steps` = 8
- Resumed at step 4
- Expected remaining steps = 4
- Expected final step = 8

The resumed training should complete at step 8, not step 32.

--- Comment by lordaarush at 2026-02-03T14:43:03Z ---
I've investigated this issue and can confirm the root cause. The problem appears to be in the `set_initial_training_values` method in `trainer.py`.

When resuming from a checkpoint, the Trainer recalculates `num_update_steps_per_epoch` based on the new dataloader length and gradient_accumulation_steps, without considering:

1. The original training's `max_steps` should be preserved
2. The global batch size is unchanged, so total optimization steps should remain the same

The fix should involve checking if we're resuming from a checkpoint and preserving the original `max_steps` from the `TrainerState`, rather than recalculating it.

I'm interested in working on a fix. @SunMarc, could you confirm this is the right approach?

--- Comment by SunMarc at 2026-02-03T15:39:15Z ---
Feel free to work on it @lordaarush ! I think there might be an issue with how max_steps is calculated maybe, but I don't think we should just use the value from TrainerState

--- Comment by lordaarush at 2026-02-03T18:54:31Z ---
Thanks @SunMarc for the guidance! I dug deeper and found the actual root cause.

**The issue wasn't in `set_initial_training_values` or the `max_steps` calculation itself** - that formula is correct. The problem was earlier in the `train()` method at line 2150-2151.

When resuming from a checkpoint, `self._train_batch_size` was unconditionally restored from the checkpoint state, causing the dataloader to be created with the OLD batch size. This led to incorrect `len_dataloader` and ultimately wrong `max_steps`.

**The fix:** Only restore the checkpoint's `train_batch_size` when `auto_find_batch_size` is enabled (since that feature needs the auto-discovered batch size). Otherwise, respect the user's current configuration.

```python
# Before:
if state.train_batch_size is not None:
    self._train_batch_size = state.train_batch_size

# After:
if state.train_batch_size is not None and args.auto_find_batch_size:
    self._train_batch_size = state.train_batch_size

--- Comment by SunMarc at 2026-02-04T13:04:59Z ---
We only save the `train_batch_size` if `auto_find_batch_size` was used I think. So it shouldn't happen for this issue. Can you try to confirm ?

--- Comment by lordaarush at 2026-02-04T13:47:00Z ---
> We only save the `train_batch_size` if `auto_find_batch_size` was used I think. So it shouldn't happen for this issue. Can you try to confirm ?

I checked this and it looks like `train_batch_size` is actually saved unconditionally, not just when `auto_find_batch_size` is used.

There are two places in `_inner_training_loop` where `state.train_batch_size` is set:

1. **Line 2251** - Inside the `if self.args.auto_find_batch_size:` block
2. **Line 2308** - Unconditionally: `self.state.train_batch_size = self._train_batch_size`

That second assignment runs every training, regardless of the `auto_find_batch_size` setting.

I verified this by running training with `auto_find_batch_size=False` and checking the saved checkpoint:

```python
args = TrainingArguments(
    ...,
    per_device_train_batch_size=2,
    auto_find_batch_size=False,
)
trainer.train()
# Then check checkpoint:
state = TrainerState.load_from_json("checkpoint-2/trainer_state.json")
print(state.train_batch_size)  # Output: 2

```
So the checkpoint contains `train_batch_size = 2` even though `auto_find_batch_size=False`. This causes the bug when resuming with a different batch configuration.

--- Comment by SunMarc at 2026-02-05T13:54:25Z ---
Arf indeed, thanks for finding this @ ! We should remove this then. Would you like to open a PR for that ?
`Line 2308 - Unconditionally: self.state.train_batch_size = self._train_batch_size`
Would this fix your issue @KHPan ?

--- Comment by lordaarush at 2026-02-05T14:27:42Z ---
Sure @SunMarc, I have opened a new PR (#43770) which simply removes the unconditional `self.state.train_batch_size = self._train_batch_size` assignment.


ISSUE #43704: Qwen3ForCausalLM leaks VRAM if used in multiple dataloader threads
State: closed  |  Labels: bug
Author: dxqb  |  Created: 2026-02-03T10:37:40Z

### System Info

reopening, please see https://github.com/huggingface/transformers/issues/42673
this is still an issue, but auto-closed by `github-actions`. That it expects a reaction within a little over a week isn't great.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

see above

### Expected behavior

see above

--- Comment by Rocketknight1 at 2026-02-03T14:00:27Z ---
Closing this to reopen the original


resume_from_checkpoint key mismatch

ISSUE #43701: resume_from_checkpoint key mismatch
State: closed  |  Labels: bug
Author: zenosai  |  Created: 2026-02-03T08:21:39Z

### System Info

- `transformers` version: 4.57.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.39
- Python version: 3.10.19
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.11.0
- Accelerate config:    not found
- DeepSpeed version: 0.17.6
- PyTorch version (accelerator?): 2.8.0+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no matter
- Using GPU in script?: yes
- GPU type: NVIDIA A800-SXM4-80GB

### Who can help?

@SunMarc @CyrilVallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

With `transformers >= 4.52`, the model uses `_checkpoint_conversion_mapping` (for example, Qwen2.5VL).
1. Train Qwen2.5-VL to obtain a checkpoint.
2. Resume training from this checkpoint using `resume_from_checkpoint`.

### Expected behavior

Because resume_from_checkpoint loads the model using load_from_state_dict instead of applying _checkpoint_conversion_mapping to convert the weights, this results in key mismatches (I guess):

--- Comment by SunMarc at 2026-02-03T12:33:44Z ---
Can you check with the latest version of transformers ? Otherwise, we think we might indeed need to load transformers model using `from_pretrained` in order to resume training due to some ops that happens in between.

--- Comment by zenosai at 2026-02-04T06:40:10Z ---
I've tried with transformers 5.0.0 and get the same result

--- Comment by antznette1 at 2026-02-04T17:35:25Z ---
I jumped on the issue, and my fix applies conversion mapping during resume, aligning with the model-loading utilities, without requiring a full `from_pretrained()` re-instantiation.

--- Comment by github-actions[bot] at 2026-03-06T08:05:39Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43700: Lack of FastTokenizer for Internlm/Intern-S1
State: closed  |  Labels:
Author: jiosephlee  |  Created: 2026-02-03T08:00:01Z

Hi,

I'm currently trying to use `return_assistant_token_masks` with Intern's models, but it's a Python-based, slow tokenizer, which is incompatible with this feature. I can't seem to find any docs on converting slow tokenizers to fast tokenizers.

Any pointers would be very much appreciated. Thanks!

--- Comment by Saad-Mallebhari at 2026-02-28T10:22:27Z ---
@jiosephlee
`return_assistant_token_masks` requires a Fast tokenizer because it depends on token offset mappings only available via the Rust-based `tokenizers` library.

For InternLM, you can try converting the slow tokenizer to fast using:
```python
from transformers.convert_slow_tokenizer import convert_slow_tokenizer
from transformers import AutoTokenizer

slow = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b", use_fast=False)
fast = convert_slow_tokenizer(slow)
fast.save_pretrained("./internlm_fast_tokenizer")
```
Then load it back with `use_fast=True`. This works for most SentencePiece-based tokenizers but may not work perfectly if InternLM uses custom special token handling.

If the conversion fails, a workaround is to manually compute the assistant token mask by finding assistant turn token positions after tokenization:
```python
input_ids = tokenizer(text, return_tensors="pt").input_ids
assistant_token_id = tokenizer.convert_tokens_to_ids("<|im_start|>")
# find positions and build mask manually
```
A native Fast tokenizer for InternLM would need to be contributed upstream , worth opening a request on the InternLM repo directly.

--- Comment by github-actions[bot] at 2026-03-25T08:11:16Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43698: SwanLab integration uses outdated swanlab.init() signature
State: closed  |  Labels: Good First Issue, Feature request
Author: TianjiaoTsao  |  Created: 2026-02-03T07:26:39Z

### Feature request

https://github.com/huggingface/transformers/blob/aefa23ad1c52de9c115f3d762fe1a1eda643275a/src/transformers/integrations/integration_utils.py#L2305

In `src/transformers/integrations/integration_utils.py`, the SwanLab integration calls `swanlab.init(**init_args)` but the current implementation does not expose important init options like `id` and `resume`.

As a result, when resuming training with `Trainer.train(resume_from_checkpoint=...)`, SwanLab cannot resume the same experiment/run and instead creates a new one, because there is no way to pass `id`/`resume` through the callback.

SwanLab docs for `init` (showing `id` / `resume` support):
https://docs.swanlab.cn/en/api/py-init.html

Request:
Please update the SwanLab integration to support passing `id` and `resume`, then forwarded into `swanlab.init(...)`).

Thanks.

### Motivation

When using `Trainer` with the SwanLab integration, resuming training (`trainer.train(resume_from_checkpoint=...)`) starts a new SwanLab experiment instead of continuing the existing one. This breaks experiment continuity: metrics/history are split across multiple runs, comparisons become confusing, and automation workflows that expect a single run per training job (e.g., restart on preemption/SLURM requeue) are harder to manage.

The root cause is that the current integration calls `swanlab.init()` without exposing `id` and `resume`, even though SwanLab supports them. Adding support for these parameters would allow seamless run continuation across restarts.

### Your contribution

None

--- Comment by Rocketknight1 at 2026-02-03T13:59:46Z ---
cc @sunmarc @MekkCyber

--- Comment by SunMarc at 2026-02-03T15:52:20Z ---
Thanks for the report @TianjiaoTsao ! Would you like to submit a PR to fix it ? Otherwise, we can let the community fix it !


ISSUE #43697: RTDetrV2ForObjectDetection produces different outputs in Transformers v5 with identical inputs
State: closed  |  Labels: bug
Author: MorganFujimaka  |  Created: 2026-02-03T07:15:55Z

### System Info

After upgrading from Transformers 4.57.6 to 5.0.0, RTDetrV2ForObjectDetection produces different logits and pred_boxes for identical pixel_values tensors.
	•	The same saved pixel_values tensor is reused across versions.
	•	Model is in eval() mode.
	•	Weights/config are identical (checkpoint trained with Transformers 4.51.3).
	•	Differences are non-trivial, not just numeric noise.

This suggests a behavioral change in the RTDetrV2ForObjectDetection forward implementation between Transformers 4.x and 5.0.0.

Happy to provide a minimal reproduction with a fixed input tensor and checkpoint if helpful.
[config.json](https://github.com/user-attachments/files/25035123/config.json)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import RTDetrV2ForObjectDetection

pixel_values = torch.randn(1, 3, 640, 640, dtype=torch.float32)
torch.save(pixel_values, "test_pixel_values.pt")

# Switch to transformers 4.57.6
model = RTDetrV2ForObjectDetection.from_pretrained(model_path)
pred_boxes_v4 = model(pixel_values=pixel_values).pred_boxes

# Switch to transformers 5.0.0
pixel_values = torch.load("test_pixel_values.pt")

model = RTDetrV2ForObjectDetection.from_pretrained(model_path)
pred_boxes_v5 = model(pixel_values=pixel_values).pred_boxes

torch.equal(pred_boxes_v4, pred_boxes_v5) => False

# transformers 4.57.6
tensor([[[0.0803, 0.1549, 0.0167, 0.0167],
         [0.3277, 0.0824, 0.0288, 0.0235],
         [0.5289, 0.8680, 0.0158, 0.0149],
         ...,
         [0.1689, 0.2689, 0.0320, 0.0193],
         [0.4674, 0.4583, 0.0376, 0.0227],
         [0.7807, 0.9875, 0.0544, 0.0248]]])

# transformers 5.0.0
tensor([[[0.0789, 0.1558, 0.0198, 0.0225],
         [0.3244, 0.0851, 0.0417, 0.0465],
         [0.5289, 0.8686, 0.0192, 0.0195],
         ...,
         [0.1615, 0.2735, 0.0409, 0.0342],
         [0.4569, 0.4622, 0.0590, 0.0432],
         [0.7627, 0.9902, 0.0575, 0.0225]]])
```

### Expected behavior

With identical pixel_values and the same checkpoint, RTDetrV2ForObjectDetection should produce identical (or numerically equivalent) outputs across Transformers versions.

--- Comment by NielsRogge at 2026-02-06T07:49:15Z ---
Hi @MorganFujimaka can you check whether my PR above fixes the issue?

--- Comment by yonigozlan at 2026-02-06T16:11:09Z ---
Hello @MorganFujimaka ! It'd be great if you could provide a checkpoint to reproduce the issue. @NielsRogge I don't think the issue comes form sin pos embeddings. The ones in v5 should actually be correct for non square image inputs (see https://github.com/huggingface/transformers/pull/41380), and they are equivalent to the old ones for square image inputs (rt_detr_v2 checkpoints on the hub resize the image to squares so this should make no differences).
What could be different is that we switched to using fast image processors (torch/torchvision based) by default which results in very slight numerical differences in processed images outputs, which can be amplified by the forward pass. Can you try to check if you get the old results back by instantiating the processor using `use_fast=False`? Otherwise can you provide the checkpoints you're using to reproduce?
Out of curiosity, do the new results make sense or are worse than the old ones?

--- Comment by MorganFujimaka at 2026-02-09T02:51:51Z ---
@yonigozlan Looks like it’s resolved in v5.1.0. Answering your questions:
1. Switching to `use_fast=False` didn’t help.
2. I haven’t tested it extensively, only a couple of test cases failed. The updated bounding boxes still looked reasonable, just slightly different, which was enough to break our tests. That instability made me hesitant to roll out Transformers 5.0.0 to production.
3. For reproduction, I used a model fine-tuned with Transformers 4.51.3. It behaved as expected on 4.57.6, but started failing tests after upgrading to 5.0.0.

--- Comment by github-actions[bot] at 2026-03-05T08:07:05Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by NielsRogge at 2026-03-05T08:33:22Z ---
Hi, let me take a look at this one.

--- Comment by NielsRogge at 2026-03-05T09:15:46Z ---
Hi @MorganFujimaka, I had Codex investigate this issue:

Key result: the v5.0.0 regression is caused by RT-DETR v2 _tied_weights_keys mapping direction, which prevents v4-style model.decoder.* head weights from loading correctly (even though missing/unexpected key counters stay zero). This was corrected by #41549 (present in v5.1.0), and outputs return to near-identical parity with v4.57.6.


ISSUE #43696: GPT-oss-20b: torch.OutOfMemoryError: CUDA out of memory.
State: closed  |  Labels:
Author: Decadz  |  Created: 2026-02-03T06:33:14Z

Hi,

I am trying to perform a distributed training run of [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) on x8 A100s (40gb); however, I am running into memory issues when trying to load the model into memory using the code below. I am aware that for GPT-OSS the Mxfp4 is only supported for Hopper generation and greater; however, even when dequantizing the model to float16/bfloat16 I should still be well within the required memory since I have 8 x 40gb = 320 GB, when doing stage-3 sharding.

**Question:** Do you have any suggestions as to why this might be the case? Am I performing the loading of the model correctly?

### Debugging Steps
- I have checked all devices are visible.
- Tried manually setting the device_map={'':local_rank}
- Inference on the model works when quantization_config = Mxfp4Config(dequantize=False)

### Dependencies
```
torch: 2.10.0
transformers: 4.57.3
trl: 0.24.0
```

### Accelerate Config
```python
compute_environment: LOCAL_MACHINE
distributed_type: DEEPSPEED
deepspeed_config:
  deepspeed_multinode_launcher: standard
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
debug: false
```

### Shell Script
```shell
module purge
module load GCCcore/9.3.0 CUDA/12.2.0 cuDNN/8.9.2.26-CUDA-12.2.0

# Ensuring all GPUs on the node are visible.
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# Path to accelerate config file.
CONFIG="accelerate_config.yaml"

# Launching the script with accelerate
accelerate launch --config_file ${CONFIG} script.py
```

### Python Code
```python
from transformers import AutoTokenizer, AutoModelForCausalLM, Mxfp4Config
import torch

# Setting up the model configuration and telling HF to dequantize the model.
quantization_config = Mxfp4Config(dequantize=True)
model_kwargs = dict(
    attn_implementation="eager",
    dtype=torch.bfloat16,
    quantization_config=quantization_config,
    use_cache=True,
    device_map="auto",  # {'':local_rank}
    max_memory={i: "40GiB" for i in range(8)},
    offload_folder="offload"
)

# Loading the model and tokeniser into memory.
model = AutoModelForCausalLM.from_pretrained("openai/gpt-oss-20b", **model_kwargs)
tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-20b")
```

### nvidia-smi

### Error Trace

--- Comment by Ashx098 at 2026-02-04T09:01:47Z ---
I think you’re accidentally forcing a per rank full or near-full model materialization on GPU first, so each process OOMs before DeepSpeed can partition anything.

[Look at Open AI page ](https://developers.openai.com/cookbook/articles/gpt-oss/run-transformers/?utm_source=chatgpt.com
)
Both are MXFP4 quantized by default. Please, note that MXFP4 is supported in Hopper or later architectures. This includes data center GPUs such as H100 or GB200, as well as the latest RTX 50xx family of consumer cards.

If you use bfloat16 instead of MXFP4, memory consumption will be larger (~48 GB for the 20b parameter model).

--- Comment by github-actions[bot] at 2026-03-05T08:07:06Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #43688: Incorrect normalization of auxiliary loss in OLMoE and GPT Oss
State: closed  |  Labels: bug
Author: andresnowak  |  Created: 2026-02-02T15:59:33Z

### System Info

- `transformers` version: 4.57.6
- Platform: macOS-15.7.3-arm64-arm-64bit
- Python version: 3.10.14
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: not installed
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.1 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

### Expected behavior

From what I understand I think the balancing loss used in OLMoE and GPT-Oss in the huggingface implementation has an error.

```py
 expert_attention_mask = (
            attention_mask[None, :, :, None, None]
            .expand((num_hidden_layers, batch_size, sequence_length, top_k, num_experts))
            .reshape(-1, top_k, num_experts)
            .to(compute_device)
        )

        # Compute the percentage of tokens routed to each experts
        tokens_per_expert = torch.sum(expert_mask.float() * expert_attention_mask, dim=0) / torch.sum(
            expert_attention_mask, dim=0
        )
```

Here in this code the formula being used is:

$$
\begin{aligned}
f_i &= \frac{N}{T} \sum_{t=1}^{T} \mathbb{1} (s_{i,t} \in \text{Topk}(\{s_{j,t} \mid 1 \leqslant j \leqslant N\}, K)), \quad \sum^N_i f_i = K \\
P_i &= \frac{1}{T} \sum_{t=1}^{T} s_{i,t}
\end{aligned}
$$

But this is incorrect as $f_i$ should be normalized by $K$, in Switch transformer this isn't done because the model uses a Top-k of 1, but if we see how deepseek does like in [Deepseek-MoE](https://arxiv.org/abs/2401.06066), we need to normalize $f_i$ by $K$, if we don't do this then $f_i$ and $P_i$ represent two different distributions, where $f_i$ will have $\frac{K}{N}$ and $P_i$ will have $\frac{1}{N}$.
And for OLMoE they mention that they use megablocks implementation for the balancing and Grouped GEMM and in [megablocks](https://github.com/Muennighoff/megablocks/blob/4a25bc7b5665bcb9da93d72d5ad0c14d41e1a351/megablocks/layers/moe.py#L31) we can see that it uses the correct normalized formula by $K$.

$$
\begin{aligned}
f_i &= \frac{N}{T K} \sum_{t=1}^{T} \mathbb{1} (s_{i,t} \in \text{Topk}(\{s_{j,t} \mid 1 \leqslant j \leqslant N\}, K)), \quad \sum^N_i f_i = 1 \\
P_i &= \frac{1}{T} \sum_{t=1}^{T} s_{i,t}
\end{aligned}
$$

With this both $f_i$ and $P_i$ can represent the same distribution.

--- Comment by github-actions[bot] at 2026-03-05T08:07:08Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by jiosephlee at 2026-03-15T05:17:44Z ---
This seems still relevant?

--- Comment by martinjaggi at 2026-03-30T07:58:48Z ---
aux losses are still wrong currently, because the PR has not been merged yet.

i can confirm aux losses for olmoe, qwen3, deepseek are all too high by a factor of top_k


ISSUE #43684: Add Qwen3-Omni model registration to AutoModel and AutoModelForConditionalGeneration classes
State: closed  |  Labels: Feature request
Author: samudraneel05  |  Created: 2026-02-02T14:15:36Z

### Feature request

When attempting to load Qwen3-Omni using Auto classes:
```python
from transformers import AutoModel
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    trust_remote_code=True,
)
```

I get an error:
```
ValueError:
Unrecognized configuration class <class 'transformers.models.qwen3_omni_moe.configuration_qwen3_omni_moe.Qwen3OmniMoeConfig'> for this kind of AutoModel: AutoModel.
Model type should be one of .......
```

Optimally, the model should load successfully through Auto classes, similar to other Qwen models.

### Motivation

I am working towards integrating Qwen3-Omni in the Keras-Hub library

### Your contribution

I am happy to take this up, if anyone else is not interested in this issue.

--- Comment by zucchini-nlp at 2026-02-02T16:01:32Z ---
The model has no base model, i.e. a model that outputs raw hidden states before the generative head. We have `Qwen3OmniMoeForConditionalGeneration` model class available with `AutoModelForMultimodalLM`  for generation

--- Comment by samudraneel05 at 2026-02-03T05:50:01Z ---
Hi, thank you for responding @zucchini-nlp ! I understand.

On a separate note, while going through the codebase files of Qwen3-Omni-Moe, in [configuration_qwen3_omni_moe](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_omni_moe/configuration_qwen3_omni_moe.py) , inside the class `Qwen3OmniMoeTalkerCodePredictorConfig(PreTrainedConfig):`, the `__init__` has `self.sliding_window`defined two times leading to issues when I'm trying to work with it. Line 584 has it as         `self.sliding_window = sliding_window`, and 591 as `self.sliding_window = sliding_window if self.use_sliding_window else None`.

Is this intended behaviour, and I am missing something, or is it a bug and only one instance needs to be there?

--- Comment by samudraneel05 at 2026-02-03T06:03:26Z ---
Reproduction script:
```
from transformers import AutoConfig

config = AutoConfig.from_pretrained(
    "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    trust_remote_code=True
)
```

Result: `AttributeError: 'Qwen3OmniMoeTalkerCodePredictorConfig' object has no attribute 'use_sliding_window'. Did you mean: 'sliding_window'?`

--- Comment by zucchini-nlp at 2026-02-03T09:40:48Z ---
Supposed to be fixed by https://github.com/huggingface/transformers/pull/43593

--- Comment by zucchini-nlp at 2026-02-04T14:37:15Z ---
Fix was just merged! Closing as resolved

--- Comment by samudraneel05 at 2026-02-04T14:38:45Z ---
thank you for all the follow ups!


ISSUE #43682: `torch.compile` and `torch.export` helper function to be moved from `import_utils`
State: open  |  Labels: WIP
Author: IlyasMoutawwakil  |  Created: 2026-02-02T13:42:53Z

a tracker per @vasqu's request


ISSUE #43680: Add Nvidia NitroGen to huggingface transformers
State: open  |  Labels: Feature request
Author: AffanBinFaisal  |  Created: 2026-02-02T11:51:53Z

### Feature request

NVIDIA's NitroGen is a groundbreaking AI model designed to play hundreds of video games autonomously across various genres. This model, developed in collaboration with MineDojo, is trained on over 40,000 hours of publicly available gameplay videos, allowing it to perform across 1,000+ games without relying on proprietary tools or custom datasets.

### Motivation

This model should be added to Hugging Face Transformers so that developers can use it through the HF interface and fine-tune it to play other games. Doing so will also help researchers build further on this work.

### Your contribution

Sure, I’ll try to help with some modules, though I’m not entirely sure yet.

--- Comment by zucchini-nlp at 2026-02-02T12:40:50Z ---
I see that it is a vision-language-action model with a DiT used to predict next actions. We don't have any VLAs in the core library yet, and I don't think we want to host a diffusion model

NitroGen might be more suited in `diffusers` imo, cc @sayakpaul if you have better suggestions.

Model repo: https://github.com/MineDojo/NitroGen
Model page on HF hub has no download stats, but it seems to be a quite interesting one to experiment

--- Comment by AffanBinFaisal at 2026-02-02T12:56:17Z ---
@zucchini-nlp I understand that NitroGen isn’t a natural fit for the core Transformers library, but I’m curious about where it might best fit in the ecosystem perhaps in diffusers. Since it’s so new, it could be interesting to explore its potential value for Hugging Face.

--- Comment by zucchini-nlp at 2026-02-25T09:25:33Z ---
Linking a loosely related PR adding Pi0 to transformers, which is another VLA (https://github.com/huggingface/transformers/pull/44160)


ISSUE #43676: `test_apply_chat_template_video_frame_sampling` fails with `num_frames` and `fps` are mutually exclusive
State: closed  |  Labels: bug
Author: tarekziade  |  Created: 2026-02-02T10:10:35Z

### System Info

================================================================================= short test summary info =================================================================================
FAILED tests/models/qwen3_vl/test_processing_qwen3_vl.py::Qwen3VLProcessorTest::test_apply_chat_template_video_frame_sampling - ValueError: `num_frames` and `fps` are mutually exclusive arguments, please use only one!

### Reproduction

pytest -sv tests/models/qwen3_vl/test_processing_qwen3_vl.py::Qwen3VLProcessorTest::test_apply_chat_template_video_frame_sampling

### Expected behavior

passing test

### Why it happens

1. The test passes fps=None explicitly to signal "don't use fps, use num_frames instead"
2. The current code filters out None values at processing_utils.py:1728, so fps=None doesn't get added to mm_load_kwargs
3. The video processor has a default fps=2 in its configuration
4. When fps is not explicitly passed to the video processor, it falls back to self.fps=2 (see video_processing_utils.py:263)
5. Both num_frames=3 and fps=2 end up being non-None, triggering the mutual exclusivity check

We can explicitely set kwargs to None in processing_utils.py but I don't like that fix I find it fragile.

@ArthurZucker WDYT

--- Comment by zucchini-nlp at 2026-02-02T10:48:46Z ---
This is a rather old branch in the PR. We've refactored and moved video loading into each model's own `video_processor` class, and passing an explicit `fps=None` should work. The video processor will not set a default value if `fps` is already present in input kwargs

--- Comment by tarekziade at 2026-02-02T10:57:01Z ---
oops, yes, sorry for the noise! it passes in `main`


ISSUE #43673: GenerationMixin cache missing in v5.0.0 during chunked_prefill
State: closed  |  Labels: bug
Author: SmerkyG  |  Created: 2026-02-02T07:23:31Z

### System Info

The removal of v4.57.6's default insertion of DynamicCache https://github.com/huggingface/transformers/blob/v4.57.6/src/transformers/generation/utils.py#L2034
vs
https://github.com/huggingface/transformers/blob/v5.0.0/src/transformers/generation/utils.py#L2042
has made it so that in v5.0.0 chunked_prefill no longer works, complaining of
`Cannot use prefill chunking without a cache`
due to line
https://github.com/huggingface/transformers/blob/v5.0.0/src/transformers/generation/utils.py#L3862
which checks that a cache exists. But per the old comment:
```
        # TODO (joao): remove this `else` when we remove the last traces of the legacy cache format (v4.58.0, search
        # for `instance(past_key_values, Cache)` as well). In general, if `cache_implementation` is unset, cache
        # initialization should happen inside the model at prefill time.
```
the cache no longer should exist until the model is first called.

I'm not sure what the most proper fix is, but I would suggest maybe testing for `if "past_key_values" not in model_kwargs and not model_kwargs.get('use_cache'):` instead?

### Who can help?

 @gante

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

use any model with chunked_prefill but do not pass an initial cache object, just use_cache=True

### Expected behavior

should work, but gives the error listed

--- Comment by zucchini-nlp at 2026-02-02T09:12:54Z ---
Indeed! I am not sure what was the plan for cache though I don't agree that initializing default cache should happen inside model's `forward` call. I think the best solution is to bring it back, lemme see


Proposal to add Qwen3-TTS support

ISSUE #43671: Proposal to add Qwen3-TTS support
State: open  |  Labels: New model
Author: ShahVandit  |  Created: 2026-02-02T02:33:32Z

### Model description

Model: Qwen3-TTS
Repository: https://github.com/QwenLM/Qwen3-TTS
Paper/Blog: https://huggingface.co/papers/2601.15621
License: Apache 2.0

Qwen3-TTS is currently available as a standalone package (qwen-tts) but is not integrated into transformers. The model uses `PreTrainedModel` and `GenerationMixin`, making it compatible with the transformers architecture.

Would the maintainers be open to this addition? I'm happy to submit a PR if this aligns with transformers' roadmap?

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

_No response_

--- Comment by Rocketknight1 at 2026-02-03T13:40:56Z ---
cc @eustlb @ebezzam

--- Comment by farzanehnakhaee70 at 2026-02-04T12:14:28Z ---
I am also intersted on adding this model to transformer library. vllm-omni currently has been added this model to his online-inference server. However, as it is not supported by the transformers, it is not useable yet.

--- Comment by ShahVandit at 2026-02-05T05:23:33Z ---
Hi! I'm implementing the Qwen3-TTS integration into transformers. Configuration, modeling, and processing utilities are complete. Currently working on tests to ensure everything works correctly.

--- Comment by ebezzam at 2026-02-05T17:18:34Z ---
@ShahVandit that's great! please open a PR when it's ready. Here are a few resources to help you with the Transformers integration:
- Every model integration should use `modular` to define (at least) the modeling, and sometimes configuration/processing: https://huggingface.co/docs/transformers/en/modular_transformers
- A few recent audio model additions so you can get an idea of the files involved and coding style: [GLM-ASR](https://github.com/huggingface/transformers/pull/42875) and [VibeVoice](https://github.com/huggingface/transformers/pull/40546/changes) (_ongoing but also a TTS model, and conventions are often changing so best to see latest approaches_)

--- Comment by pjgao at 2026-03-02T13:29:04Z ---
Any progress for this model?

--- Comment by ShahVandit at 2026-03-07T18:50:24Z ---
Qwen3-TTS implementation is complete and ready for review.
https://github.com/huggingface/transformers/pull/44517

--- Comment by ShahVandit at 2026-03-19T17:41:43Z ---
Hi @ebezzam , just checking in on the PR when you get a chance. Happy to make any changes based on your feedback!

--- Comment by ebezzam at 2026-03-19T18:05:09Z ---
Hi @ShahVandit thanks for the message! I haven't forgotten, there's a few ongoing model additions so I kept postponing my review 🙃 I'll try to drop some pointers tomorrow or early next week. Thanks for your contribution 🙏


ISSUE #43668: ModernBERTConfig `norm_eps` type hint is incorrect
State: closed  |  Labels:
Author: fschlatt  |  Created: 2026-02-01T09:46:55Z

The type hint for `norm_eps` is int, but should be float.

https://github.com/huggingface/transformers/blob/78bb85146c59258a0710c8d08311d98d52303c38/src/transformers/models/modernbert/configuration_modernbert.py#L158