ISSUE #44403: Unnecessary noise when loading a transformer
State: open  |  Labels: bug
Author: AngledLuffa  |  Created: 2026-03-02T23:43:47Z

### System Info

python: 3.13.5

torch: 2.7.1+cu118

transformers: 5.2.0

tokenizers: 0.22.2

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The latest version of transformers is extremely noisy, and there seems to be no reason or benefit for any of it.

```
>>> from transformers import AutoModel
>>> bert_model = AutoModel.from_pretrained('hfl/chinese-electra-180g-large-discriminator')
Loading weights: 100%|████████████████████████████████████████████| 389/389 [00:00<00:00, 5460.73it/s, Materializing param=encoder.layer.23.output.dense.weight]
ElectraModel LOAD REPORT from: hfl/chinese-electra-180g-large-discriminator
Key                                               | Status     |  |
--------------------------------------------------+------------+--+-
electra.embeddings.position_ids                   | UNEXPECTED |  |
discriminator_predictions.dense_prediction.bias   | UNEXPECTED |  |
discriminator_predictions.dense.weight            | UNEXPECTED |  |
discriminator_predictions.dense_prediction.weight | UNEXPECTED |  |
discriminator_predictions.dense.bias              | UNEXPECTED |  |

Notes:
- UNEXPECTED    :can be ignored when loading from different task/architecture; not ok if you expect identical arch.
>>>
Exception in Thread-auto_conversion:
Traceback (most recent call last):
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/huggingface_hub/utils/_http.py", line 720, in hf_raise_for_status
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://huggingface.co/api/models/hfl/chinese-electra-180g-large-discriminator/discussions?p=0'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/threading.py", line 994, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/safetensors_conversion.py", line 117, in auto_conversion
    raise e
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/safetensors_conversion.py", line 96, in auto_conversion
    sha = get_conversion_pr_reference(api, pretrained_model_name_or_path, **cached_file_kwargs)
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/safetensors_conversion.py", line 69, in get_conversion_pr_reference
    pr = previous_pr(api, model_id, pr_title, token=token)
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/safetensors_conversion.py", line 14, in previous_pr
    for discussion in get_repo_discussions(repo_id=model_id, token=token):
                      ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/huggingface_hub/hf_api.py", line 6455, in get_repo_discussions
    discussions, has_next = _fetch_discussion_page(page_index=page_index)
                            ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/huggingface_hub/hf_api.py", line 6444, in _fetch_discussion_page
    hf_raise_for_status(resp)
    ~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/huggingface_hub/utils/_http.py", line 802, in hf_raise_for_status
    raise _format(HfHubHTTPError, message, response) from e
huggingface_hub.errors.HfHubHTTPError: (Request ID: Root=1-69a62017-4041fa5968ff8ee13d64fe3d;47725ac8-64d6-4947-ad16-4f88ef89391c)

403 Forbidden: Discussions are disabled for this repo.
Cannot access content at: https://huggingface.co/api/models/hfl/chinese-electra-180g-large-discriminator/discussions?p=0.
Make sure your token has the correct permissions.

>>>
KeyboardInterrupt
```

### Expected behavior

Expected behavior would be nice and quiet!

```
>>> from transformers import AutoModel
>>> bert_model = AutoModel.from_pretrained('hfl/chinese-electra-180g-large-discriminator')
>>>
```

--- Comment by Adarsh0047 at 2026-03-03T10:32:55Z ---
I will check and take up this issue

--- Comment by Rocketknight1 at 2026-03-03T14:16:36Z ---
To be fair, most of that noise is Python throwing an error because you don't actually have repo access. The other warnings about missing/unexpected weights are important - they let users know that they're transferring a checkpoint to a different task, and may need to fine-tune rather than just assuming inference will immediately work out of the box.

--- Comment by ArthurZucker at 2026-03-03T14:24:50Z ---
mmm its weird that weights did load tho!

--- Comment by ArthurZucker at 2026-03-03T14:25:29Z ---
cc @LysandreJik its the safetensor auto bot conversion here:
```python
  File "/nlp/scr/horatio/miniconda3/lib/python3.13/site-packages/transformers/safetensors_conversion.py", line 14, in previous_pr
    for discussion in get_repo_discussions(repo_id=model_id, token=token):
```

--- Comment by AngledLuffa at 2026-03-03T17:19:23Z ---
> To be fair, most of that noise is Python throwing an error because you don't actually have repo access.

We have the ability to download that model, though, so it's weird that we can't get any other necessary information.

--- Comment by github-actions[bot] at 2026-04-02T08:17:52Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by AngledLuffa at 2026-04-02T15:15:49Z ---
@ArthurZucker was there a fix for the repo not having access leaving a long stack trace?  It was happening every time I loaded this model, although the model still successfully loaded.

--- Comment by github-actions[bot] at 2026-04-27T08:46:41Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

--- Comment by dhruv7477 at 2026-05-03T12:04:03Z ---
The previous attempt (#44440) was closed because it was opened while discussion was still ongoing. The issue has since gone stale. The fix is straightforward: the background thread passes ignore_errors_during_conversion=False, causing the 403 to surface as an unhandled thread exception. Since this thread is fire-and-forget, changing that flag to True is sufficient. Opening a minimal PR now.


ISSUE #44402: "rmihaylov/bert-base-bg" model has pad and unk tokens outside the tokenizer vocab_size
State: closed  |  Labels: bug
Author: AngledLuffa  |  Created: 2026-03-02T21:49:00Z

### System Info

python: 3.13.5

torch: 2.7.1+cu118

transformers: 5.2.0

tokenizers: 0.22.2

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

This is using the Stanza (1.11.1) integration with transformers.  Any task using the "rmihaylov/bert-base-bg" or "rmihaylov/bert-base-theseus-bg" models, such as training on a BG Universal Dependencies dataset, fails with the following error.

When tokenizing multiple sentences of BG text using the model "rmihaylov/bert-base-bg", the <pad> and <unk> token is outside the expected dimensions of the embedding.  This results in a crash when the transformer embeds the input text at the bottom layer.

Result of loading the tokenizer:

```
TokenizersBackend(name_or_path='rmihaylov/bert-base-bg', vocab_size=119547, model_max_length=512, padding_side='right', truncation_side='right', special_tokens={'bos_token': '[CLS]', 'eos_token': '[SEP]', 'unk_token': '<unk>', 'sep_token': '[SEP]', 'pad_token': '<pad>', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}, added_tokens_decoder={
        2: AddedToken("[CLS]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        3: AddedToken("[SEP]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        4: AddedToken("[MASK]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        119547: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        119548: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
)
```

Example token IDs and attention matrix from this model after processing multiple lines at once.  Note that the padded tokens (attn mask 0) have ID 119547.

```
tensor([     2,     57,  38910,     86,   1525,     36,    243,   2474,    518,
           392,     19,      8,  38020,     21,   3312, 100681,     32,    985,
            27,    153,  23214,    402,    701,   4275,   1448,     19,     20,
            51,    207,   2995,     21,  11080,     40,   1506,     25,   1710,
            19,     20,   7970,     21,   3061,     19,      7,  37849,   1144,
            22,   3738,     19,      7,     25,   5558,  25814,   3230,     62,
          2824,  39769,  33752,     19,     20,     51,  10517,     26,  16506,
            31,   1506,    304,   6080,     19,     20,  32034,     21,  84886,
            27,    153,   2702,   8380,     19,     20,     51,     24,   2663,
            25,    316,     22,    877,   3438,    443,     19,      9,      3,
        119547, 119547, 119547, 119547, 119547], device='cuda:0')
tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0],
       device='cuda:0')
```

This leads to an exception / assertion:

```
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:1553: indexSelectLargeIndex: block: [79,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
  File ".../miniconda3/lib/python3.13/site-packages/torch/nn/modules/sparse.py", line 190, in forward
    return F.embedding(
           ~~~~~~~~~~~^
        input,
        ^^^^^^
    ...<5 lines>...
        self.sparse,
        ^^^^^^^^^^^^
    )
    ^
  File ".../miniconda3/lib/python3.13/site-packages/torch/nn/functional.py", line 2551, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
```

Patching the tokenizer myself by setting the pad_token and unk_token to some ID in the expected range makes the processing work, but that solution is extremely hacky

### Expected behavior

Ideally the model itself would be patched with correct PAD and UNK indices / embeddings, the tokenizers / transformers packages could compensate for this error, or maybe the model just needs to be deprecated or removed until it is properly usable.

--- Comment by Rocketknight1 at 2026-03-03T14:02:04Z ---
Hmmn, I'm not sure we can auto-fix a broken model config! Have you tried opening an issue on the model repo instead?

--- Comment by AngledLuffa at 2026-03-03T17:48:17Z ---
I have done that!  I can't recall ever seeing a model updated / fixed after reporting such an issue, though.  I have a growing list of models with broken max sequence lengths, but this is only the second time I've seen a broken pad token

--- Comment by Cyrilvallez at 2026-03-09T10:49:35Z ---
Hey @AngledLuffa! Did you use to be able to use the model properly on any v4 version? May be an issue with the tokenizer refactors, cc @ArthurZucker

--- Comment by AngledLuffa at 2026-03-09T18:21:28Z ---
I did not have any version of transformers where model worked, either the 4.57 versions or the latest 5 version.

I hacked it as follows in our wrapper around the model loading.  I assume it's not ideal, and it makes me wonder if the model ever saw any pad or unk tokens when training or how it was handled, but OTOH this model was more accurate on downstream tasks than the other BG models I found

```
        if model_name in ('rmihaylov/bert-base-bg', 'rmihaylov/bert-base-theseus-bg'):
            if max(bert_tokenizer.added_tokens_decoder.keys()) > bert_tokenizer.vocab_size:
                logger.debug("Tokenizer for %s has a bug in its pad/unk tokens, in that they are outside the embedding range of the tokenizer.  tokenizer.vocab_size %s, tokenizer.added_tokens_decoder %s - fixing the pad_token and unk_token by setting them to match mask_token (%s)", model_name, bert_tokenizer.vocab_size, bert_tokenizer.added_tokens_decoder, bert_tokenizer.mask_token)
                bert_tokenizer.pad_token = bert_tokenizer.mask_token
                bert_tokenizer.unk_token = bert_tokenizer.mask_token
```

--- Comment by Cyrilvallez at 2026-03-10T09:29:18Z ---
Alright, if model has always been broken, then as @Rocketknight1 said it's the config itself, and there's nothing we can do! They will have to update on the hub!


ISSUE #44393: Qwen3-VL: Halucination/Error with 2D bounding box output
State: closed  |  Labels: bug
Author: L-Reichardt  |  Created: 2026-03-02T14:39:22Z

### System Info

Not quite sure if this is the proper place, but could in theory be handled via preprocessing. This issue is more as a type of documentation in case other face this in the future.

Detecting 2D bounding boxes does not work if inputting images with an aspect ratio similar to KITTI (3.3:1), as the model defaults to 3D bounding boxes. This can be fixed by changing the aspect ratio to a size similar during 2D training and the rescaling the output bounding boxes.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Ask Qwen3 to detect 2D Boxes in KITTI images

### Expected behavior

JSON output with 9 values per box (3D bbox)

--- Comment by Saad-Mallebhari at 2026-03-03T09:17:26Z ---
@L-Reichardt
This may be related to how extreme aspect ratios influence visual feature encoding and task interpretation in multimodal models , the exact mechanism isn't confirmed, but the aspect ratio appears to affect how the model interprets the detection task.

The workaround is to resize before inference and scale back after:
```python
from PIL import Image

def resize_for_2d_detection(image, target_ratio=1.5):
    w, h = image.size
    if w / h > target_ratio:
        new_w = int(h * target_ratio)
        image = image.resize((new_w, h))
    return image

# Scale x-coordinates back after inference:
# x_original = x_predicted * (original_width / resized_width)
# y_original = y_predicted * (original_height / resized_height)
```
Also worth trying , explicitly conditioning the prompt with the expected output format:
```
"Detect all objects and return 2D bounding boxes in
[x1, y1, x2, y2] format only."
```
This can help override the behavior at the prompt level without needing to resize.

--- Comment by L-Reichardt at 2026-03-03T10:18:03Z ---
- Yeah I did run some tests and confirmed that resizing works prior to posting the issue (my text is a bit unclear on that), specifically .resize((1008, 756) seems to work best.
- Prompt conditioning was not as reliable


ISSUE #44387: Increased CUDA reserved memory in Transformers 5.x under int4 quantization leads to OOM
State: closed  |  Labels: bug
Author: tangefly  |  Created: 2026-03-02T11:16:43Z

### System Info

- `transformers` version: 5.2.0
- Platform: Linux-6.12.57+deb13-amd64-x86_64-with-glibc2.41
- Python version: 3.12.12
- Huggingface_hub version: 0.36.2
- Safetensors version: 0.7.0
- Accelerate version: 1.11.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA GeForce RTX 5090

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

### Problem Description

I encountered a CUDA OOM error when using LlamaFactory to load the Qwen3.5-35B-A3B model with int4 quantization for inference.

My hardware setup consists of 2 × RTX 5090 32GB GPUs. Under normal circumstances, this configuration should be sufficient to load the model without OOM when using int4 quantization.

To further investigate the issue, I used a minimal reproduction script and observed that under Transformers 5.2.0, the CUDA reserved memory appears to be allocated as if the model were not quantized. This results in excessive reserved memory usage, leading to memory waste and potential OOM, even though the actual allocated memory remains consistent with int4 quantization expectations.

This behavior was not observed in earlier versions, where the model could be loaded successfully under the same hardware conditions.

```
import torch
import gc
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

def print_mem(tag):
    torch.cuda.synchronize()
    allocated = torch.cuda.memory_allocated() / 1024**3
    reserved = torch.cuda.memory_reserved() / 1024**3
    print(f"{tag}")
    print(f"  allocated: {allocated:.2f} GB")
    print(f"  reserved : {reserved:.2f} GB")

# Clean up GPU memory
gc.collect()
torch.cuda.empty_cache()

print_mem("Before loading")

model_name = "Qwen2.5-7B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)

print_mem("After loading")

```
### Output Result

- Transformers 5.2.0
```
Before loading
  allocated: 0.00 GB
  reserved : 0.00 GB
Loading weights: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 339/339 [00:02<00:00, 155.08it/s, Materializing param=model.norm.weight]
After loading
  allocated: 5.46 GB
  reserved : 13.83 GB
```
- Transformers 4.57.5
```
Before loading
  allocated: 0.00 GB
  reserved : 0.00 GB
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05<00:00,  1.43s/it]
After loading
  allocated: 5.46 GB
  reserved : 6.70 GB
```

### Expected behavior

- In an environment with **2 × RTX 5090 32GB GPUs**, loading the **Qwen3.5-35B-A3B** model with **int4 quantization** using LlamaFactory should not result in an OOM error.
- The CUDA **reserved memory** usage during model loading in **Transformers 5.2.0** should be consistent with that of **Transformers 4.57.5**.

--- Comment by Rocketknight1 at 2026-03-02T14:26:42Z ---
Hi @tangefly, can you try doing a `git bisect` to identify the exact commit that causes the issue? This will help a lot in figuring out the cause and possibly fixing it!

--- Comment by tangefly at 2026-03-03T09:54:54Z ---
@Rocketknight1

### Root Cause Analysis
Through version bisect, I identified that the problematic behavior was introduced by Pull Request #42882.

### Verification Steps:

1. Installed transformers at the specific commit from PR #42882 (commit b05d2c4309ba28cecaa64acf14521fea5c829523).
2. Observed the same excessive CUDA reserved memory issue (similar to v5.2.0).
3. Installed the immediate parent commit of that PR.
4. Observed that the issue does not occur, and memory usage is similar to v4.57.5.

This confirms that the changes in PR #42882 (which aims to “Fix dtype quantizer”) are highly likely the cause of the abnormal memory reservation behavior.

--- Comment by Rocketknight1 at 2026-03-03T14:25:40Z ---
cc @sunmarc @mekkcyber any idea what's happening here, in that case?

--- Comment by SunMarc at 2026-03-03T16:55:42Z ---
This is most likely an issue with the cache allocator as we might be allocating too much memory. I'll have a look. Thanks for the reproducer. Also, the memory is `reserved` but you can still use it no since not everything is allocated? How does the oom happens

--- Comment by tangefly at 2026-03-04T09:00:30Z ---
@SunMarc For example, when loading the Qwen3-14B model with int4 quantization on a GPU with 16GB of VRAM, Transformers 5.2.0 will run out of memory (OOM), whereas Transformers 4.57.5 will not.

--- Comment by davedgd at 2026-03-05T20:00:22Z ---
@SunMarc: This is also causing issues with unsloth using newer versions of transformers:

https://github.com/unslothai/unsloth/issues/4108#issuecomment-3977750643

--- Comment by SunMarc at 2026-03-05T23:55:46Z ---
Thanks @tangefly for bisecting the faulty PR ! I'll fix it asap.

--- Comment by tangefly at 2026-03-06T02:33:56Z ---
@SunMarc

Setting `HF_DEACTIVATE_ASYNC_LOAD=1` resolved the issue on my side. After disabling async loading, the model loads successfully and the previous memory problem no longer occurs.

It does seem that async loading was materializing the tensors before quantization, which caused the temporary memory spike. Thanks for the suggestion!

--- Comment by tangefly at 2026-03-06T02:35:37Z ---
You're welcome!

--- Comment by SunMarc at 2026-03-06T10:03:41Z ---
Hmmm what about he PR you linked then @tangefly ? Was the real issue the async loading or the pr you linked or both somehow ?

For users who night face this issue, try setting `HF_DEACTIVATE_ASYNC_LOAD=1` as it deactivate the async loading of tensors.

--- Comment by SunMarc at 2026-03-06T10:58:24Z ---
I think the issue if with the async load:

  ```
┌───────────────────────────┬────────────────┬───────────┬──────────┐
  │          Version          │     Async      │ Allocated │ Reserved │
  ├───────────────────────────┼────────────────┼───────────┼──────────┤
  │ v4.57.6                   │ N/A (old path) │ 5.46 GB   │ 6.70 GB  │
  ├───────────────────────────┼────────────────┼───────────┼──────────┤
  │ parent of PR #42882       │ ON             │ 5.47 GB   │ 13.28 GB │
  ├───────────────────────────┼────────────────┼───────────┼──────────┤
  │ parent of PR #42882       │ OFF            │ 5.45 GB   │ 7.20 GB  │
  ├───────────────────────────┼────────────────┼───────────┼──────────┤
  │ PR #42882 (b05d2c4)       │ ON             │ 5.47 GB   │ 13.28 GB │
  ├───────────────────────────┼────────────────┼───────────┼──────────┤
  │ PR #42882 (b05d2c4)       │ OFF            │ 5.45 GB   │ 7.20 GB  │
  ├───────────────────────────┼────────────────┼───────────┼──────────┤
  │ current HEAD (5.3.0.dev0) │ ON             │ 5.46 GB   │ 13.50 GB │
  ├───────────────────────────┼────────────────┼───────────┼──────────┤
  │ current HEAD (5.3.0.dev0) │ OFF            │ 5.45 GB   │ 5.63 GB  │
  └───────────────────────────┴────────────────┴───────────┴──────────┘
```

--- Comment by davedgd at 2026-03-06T23:06:00Z ---
> Hmmm what about he PR you linked then [@tangefly](https://github.com/tangefly) ? Was the real issue the async loading or the pr you linked or both somehow ?
>
> For users who night face this issue, try setting `HF_DEACTIVATE_ASYNC_LOAD=1` as it deactivate the async loading of tensors.

@SunMarc: I can confirm you are correct -- adding `HF_DEACTIVATE_ASYNC_LOAD=1` does fix this in unsloth using the exact same code without the need for `dtype` workarounds.

I'll note this finding in the unsloth thread. Also, while this workaround does fix the problem (thank you for pointing it out!), I assume it's fair to say that it would be ideal if the OOM could be avoided without the need for setting this variable on every call moving forward.

--- Comment by SunMarc at 2026-03-10T15:13:49Z ---
Can you give this a PR @davedgd  ? https://github.com/huggingface/transformers/pull/44576
It should be better than having to set `HF_DEACTIVATE_ASYNC_LOAD=1`

--- Comment by davedgd at 2026-03-10T23:44:20Z ---
> Can you give this a PR [@davedgd](https://github.com/davedgd) ? [#44576](https://github.com/huggingface/transformers/pull/44576) It should be better than having to set `HF_DEACTIVATE_ASYNC_LOAD=1`

Done -- LGTM!

--- Comment by Cyrilvallez at 2026-03-12T10:02:02Z ---
Ha yes indeed, if quantization is to be done on-the-fly, indeed async loading will load param faster ⚡ than they are quantized usually!

--- Comment by Cyrilvallez at 2026-03-12T10:02:18Z ---
Fun to see boths responding to themselves though haha


ISSUE #44384: Qwen3.5 model: When data is not padding, an error is reported, indicating that the shape does not match.
State: closed  |  Labels:
Author: Vectorwh  |  Created: 2026-03-02T09:37:31Z

commit id：fc9137225880a9d03f130634c20f9dbe36a7b8bf
Qwen3_5 Whether the position_ids input when the text model invokes the decoder_layer is text_position_ids

--- Comment by vvvdwbvvv at 2026-03-02T11:56:14Z ---
It seems like related to issue #44322

--- Comment by pjgao at 2026-03-02T13:26:27Z ---
In my test, change this line https://github.com/huggingface/transformers/blob/8468dbf7d0f97550e70755550abf0862978918ec/src/transformers/models/qwen3_5/modular_qwen3_5.py#L681  `position_ids` to `text_position_ids` can fix.

In `Qwen3VLTextModel`,  it is `text_position_ids`  too. https://github.com/huggingface/transformers/blob/8468dbf7d0f97550e70755550abf0862978918ec/src/transformers/models/qwen3_vl/modular_qwen3_vl.py#L819,  Now why using `position_ids`  instead of `text_position_ids` in `Qwen3_5TextModel`?

--- Comment by zucchini-nlp at 2026-03-02T16:30:02Z ---
Not sure what is the error @Vectorwh is reporting, a small reproducer would be nice

@pjgao huh indeed, must have been a typo. It might cause some issues in FA2, do you want to open a PR to fix?

--- Comment by weiguangli-io at 2026-03-02T17:29:09Z ---
Opened #44399 to fix the `position_ids` → `text_position_ids` typo in `Qwen3_5TextModel.forward`, as identified by @pjgao and confirmed by @zucchini-nlp.

--- Comment by JakobJBauer at 2026-03-03T16:43:15Z ---
I ran into the same issue here:
```py
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer
from peft import LoraConfig

dataset = load_dataset("json", data_files=str(data_path), split="train")

training_args = SFTConfig(
    output_dir=args.output_dir,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    num_train_epochs=3,
    learning_rate=2e-5,
)

peft_config = LoraConfig(
    task_type="CAUSAL_LM",
    r=8,
    lora_alpha=64,
    lora_dropout=0.05,
    bias="none",
    target_modules=["q_proj", "k_proj"],
)

trainer = SFTTrainer(
    model=args.model,
    args=training_args,
    train_dataset=dataset,
    peft_config=peft_config,
)

trainer.train()
trainer.save_model(args.output_dir)
```

This gives the error:
```bash
Traceback (most recent call last):
  File "/workspace/writeable/axiom-guided-structured-reasoning/scripts/train_sft_codebook_qa.py", line 97, in <module>
    main()
  File "/workspace/writeable/axiom-guided-structured-reasoning/scripts/train_sft_codebook_qa.py", line 92, in main
    trainer.train()
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/trainer.py", line 1412, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/trainer.py", line 1742, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/trl/trainer/sft_trainer.py", line 1320, in training_step
    return super().training_step(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/trainer.py", line 1951, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/trl/trainer/sft_trainer.py", line 1216, in compute_loss
    (loss, outputs) = super().compute_loss(
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/trainer.py", line 2022, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/accelerate/utils/operations.py", line 819, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/accelerate/utils/operations.py", line 807, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/peft/peft_model.py", line 1923, in forward
    return self.base_model(
           ^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/peft/tuners/tuners_utils.py", line 311, in forward
    return self.model.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 841, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 1938, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 841, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 1694, in forward
    outputs = self.language_model(
              ^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/utils/generic.py", line 915, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/utils/output_capturing.py", line 253, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py", line 1354, in forward
    causal_mask = create_causal_mask(
                  ^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 171, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/masking_utils.py", line 933, in create_causal_mask
    causal_mask = mask_interface(
                  ^^^^^^^^^^^^^^^
  File "/workspace/writeable/axiom-guided-structured-reasoning/venv/lib/python3.12/site-packages/transformers/masking_utils.py", line 509, in sdpa_mask
    attention_mask = attention_mask.expand(batch_size, -1, q_length, kv_length)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expand(torch.cuda.LongTensor{[4, 1, 1, 2075, 2075]}, size=[4, -1, 1, 2075]): the number of sizes provided (4) must be greater or equal to the number of dimensions in the tensor (5)
```

Environment:
```
- python: 3.12.3 (main, Jan 22 2026, 20:57:42) [GCC 13.3.0]
- platform: Ubuntu 24.04 (Linux-6.8.0-55-generic-x86_64-with-glibc2.39)
- GPU: NVIDIA A100-SXM4-40GB (1x), CUDA available: True
- torch: 2.10.0+cu128
- transformers: 5.2.0
- trl: 0.29.0
- peft: 0.18.1
- accelerate: 1.12.0
- datasets: 4.6.1
- wandb: 0.25.0
- Installation: pip (inside virtualenv)
```

I also tried installing the Qwen3.5 position_ids fix from PR #44399, but this did not change the behavior; the error in sdpa_mask remains.

--- Comment by zucchini-nlp at 2026-03-03T17:31:03Z ---
Hmm, interesting, why would the mask be 5D in `forward`. Might be an issue with peft or trl, I'll check it out tomorrow

--- Comment by zucchini-nlp at 2026-03-04T10:57:38Z ---
@JakobJBauer Seems like the fix for `mm_token_type_ids` didn't make it to the release (https://github.com/huggingface/trl/pull/5178). I tried to reproduce the issue with `Qwen/Qwen3.5-9B` and a random VLM dataset but got a different issue with the latest TRL version

I believe you need to install TRL from main and try, and prob report it in trl repo. cc @albertvillanova

--- Comment by albertvillanova at 2026-03-04T11:43:58Z ---
Thanks for the ping, @zucchini-nlp.

Yes, if the issue reported here is the same as in TRL:
- https://github.com/huggingface/trl/issues/5177

then the fix is already merged to main but not released yet.


ISSUE #44380: GPT2 attention scaling config is ignored when using SDPA / FlashAttention backends
State: closed  |  Labels: bug
Author: Qi-Zhan  |  Created: 2026-03-02T03:31:15Z

### System Info

None

### Who can help?

@ArthurZucker Hi, I'm new to LLMs and currently learning GPT2 model. I found that

The GPT2 attention configuration options:
	•	scale_attn_weights
	•	scale_attn_by_inverse_layer_idx

are respected in eager attention mode but silently ignored when using AttentionInterface backends such as "sdpa" or "flash_attention_2".

In eager mode:
the scaling logic is applied inside eager_attention_forward:
	•	division by sqrt(head_dim) if scale_attn_weights=True
	•	division by (layer_idx+1) if scale_attn_by_inverse_layer_idx=True

However, when using sdpa:
`torch._C._nn.scaled_dot_product_attention(query_states_3, key_states_3, value_states_3, attn_mask = attention_mask_1, dropout_p = 0.0, scale = None, is_causal = False)` which seems to ignore the above config.

I realize that the default configuration (`scale_attn_weights =True, scale_attn_by_inverse_layer_idx=False`) produces the same results, so I’m not sure whether this is intentional or should be considered a bug.

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

Different attention implementations should produce semantically equivalent results and respect model configuration parameters.

--- Comment by Saad-Mallebhari at 2026-03-02T08:46:09Z ---
@Qi-Zhan
You're right that this is a real inconsistency. The `scale` parameter in `scaled_dot_product_attention` is being passed as `None`, which defaults to `1/sqrt(head_dim)` , matching `scale_attn_weights=True`. But `scale_attn_by_inverse_layer_idx` has no equivalent in the SDPA call, so it's silently ignored when using SDPA or FlashAttention backends.

The fix would be to compute the scale explicitly before calling SDPA and pass it in:
```python
scale = 1.0 / (head_dim ** 0.5) if config.scale_attn_weights else 1.0
if config.scale_attn_by_inverse_layer_idx:
    scale = scale / (layer_idx + 1)

torch._C._nn.scaled_dot_product_attention(..., scale=scale)
```
This ensures all three backends produce semantically equivalent results regardless of config. The default config (`scale_attn_weights=True`, `scale_attn_by_inverse_layer_idx=False`) happens to match SDPA's default behavior, which is why it's not immediately obvious.

--- Comment by DhruvTilva at 2026-03-02T09:07:41Z ---
@Saad-Mallebhari  @Qi-Zhan  will try to resolve this, On it..

--- Comment by vasqu at 2026-03-02T19:11:24Z ---
Heya just seeing this now, coming from the PR here #44397

Not to double the effort @DhruvTilva, you can also help out on the review or similar if you want to. Really appreciate digging into this


Wrong docstring for position_ids

ISSUE #44373: Wrong docstring for position_ids
State: closed  |  Labels: bug
Author: RmZeta2718  |  Created: 2026-03-01T15:46:43Z

### System Info

latest commit, see link below

### Who can help?

@stevhliu

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Wrong docstring:

https://github.com/huggingface/transformers/blob/11b1906d5c0dae39c13270e47cc02c4cde70e548/src/transformers/modeling_flash_attention_utils.py#L359-L360

https://github.com/huggingface/transformers/blob/11b1906d5c0dae39c13270e47cc02c4cde70e548/src/transformers/modeling_flash_attention_utils.py#L412-L413

The docstring is for attention_mask, not position_ids

https://github.com/huggingface/transformers/blob/11b1906d5c0dae39c13270e47cc02c4cde70e548/src/transformers/modeling_flash_attention_utils.py#L201

### Expected behavior

Expected docstring:

https://github.com/huggingface/transformers/blob/11b1906d5c0dae39c13270e47cc02c4cde70e548/src/transformers/models/layoutlmv2/modeling_layoutlmv2.py#L897-L901

or

https://github.com/huggingface/transformers/blob/11b1906d5c0dae39c13270e47cc02c4cde70e548/src/transformers/masking_utils.py#L715-L716

--- Comment by ManasVardhan at 2026-03-01T17:07:04Z ---
I'll take this - fixing the docstring now.

--- Comment by mvanhorn at 2026-03-09T14:59:48Z ---
I've submitted a fix for this in #44547. Corrected the position_ids docstring to accurately describe position indices instead of attention mask semantics.

--- Comment by github-actions[bot] at 2026-04-03T08:13:57Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


<spam>

ISSUE #44371: <spam>
State: closed  |  Labels:
Author: HansElze  |  Created: 2026-03-01T13:04:25Z

<spam>


ISSUE #44370: [i18n-<languageCode>] Translating docs to <languageName>
State: closed  |  Labels: WIP
Author: j6n5nwwmx9-cpu  |  Created: 2026-03-01T08:57:27Z

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the <languageName>-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `<languageCode>` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<languageCode>/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.md) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.md) (waiting for initial PR to go through)
- [ ] [installation.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.md).

## Tutorial section
- [ ] [pipeline_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.md)
- [ ]  [autoclass_tutorial.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/autoclass_tutorial.md)
- [ ]  [preprocessing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.md)
- [ ]  [training.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.md)
- [ ]  [accelerate.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.md)
- [ ]  [model_sharing.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.md)
- [ ]  [multilingual.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.md)

<!--
Keep on adding more as you go 🔥
-->

--- Comment by j6n5nwwmx9-cpu at 2026-03-01T08:57:51Z ---
L


ISSUE #44368: when using ms-swift lora fine-tuning Qwen3.5-27B, each layer emits warning:You should update the config with `tie_word_embeddings=False` to silence this warning
State: closed  |  Labels: bug
Author: huangy3881  |  Created: 2026-03-01T07:25:46Z

### System Info

transformers==5.2.0
torch==2.8.0
deepspeed==0.18.6
python==3.10
ms-swift==4.0.0.dev0

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

# 4 * 30GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
MAX_PIXELS=1003520 \
VIDEO_MAX_PIXELS=50176 \
FPS_MAX_FRAMES=12 \
CUDA_VISIBLE_DEVICES=0,1 \
swift sft \
    --model Qwen3.5-27B \
    --tuner_type lora \
    --dataset alpaca-gpt4-data-zh \
    --load_from_cache_file true \
    --add_non_thinking_prefix true \
    --split_dataset_ratio 0.01 \
    --torch_dtype bfloat16 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --learning_rate 1e-4 \
    --lora_rank 8 \
    --lora_alpha 16 \
    --target_modules all-linear \
    --gradient_accumulation_steps 1 \
    --output_dir output \
    --report_to tensorboard \
    --eval_steps 50 \
    --save_steps 50 \
    --save_total_limit 2 \
    --logging_steps 1 \
    --max_length 2048 \
    --warmup_ratio 0.05 \
    --dataloader_num_workers 4 \
    --deepspeed zero3 \
> output_lora.log 2>&1

### Expected behavior

a warning is emitted for every layer:

The tied weights mapping and config for this model specifies to tie model.visual.patch_embed.proj.weight to model.language_model.norm.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning

--- Comment by redpanda1995 at 2026-03-01T22:57:56Z ---
https://github.com/huggingface/transformers/pull/44378

--- Comment by Rocketknight1 at 2026-03-02T15:03:11Z ---
Hmmn, this is unusual - I guess we can do the fix in #44378, but I don't think that warning should fire here. The model has `tie_word_embeddings=False`. Is `ms-swift` doing something weird?

--- Comment by huangy3881 at 2026-03-03T03:11:12Z ---
<img width="2256" height="1210" alt="Image" src="https://github.com/user-attachments/assets/0b2f341b-70a4-4a74-87e7-8bcf4f0dec47" />

I can't tell if it's related with ms-swift from the log above.

--- Comment by github-actions[bot] at 2026-03-31T08:19:10Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44367: Unauthorized error for non gated models also
State: closed  |  Labels:
Author: Swatikkar  |  Created: 2026-03-01T06:32:25Z

Hi Team,

I am continuously getting the same unauthorized error for every model i am trying to use.
I have created a token access recently with read as token type.
I have already tried with multiple models but same issue is coming up for all of those.
Please look into it and give me a solution.
userId - SwatikX

This is the error coming for every model.
STILL ERRORING? Check this: 404 Client Error: Not Found for url: https://router.huggingface.co/hf-inference/models/microsoft/Phi-3-mini-4k-instruct/v1/chat/completions (Request ID: Root=1-69a3dacb-11cd4511013222d66bfd0a28;6ec1e299-5834-4924-8898-82dab3e982e0)

--- Comment by Saad-Mallebhari at 2026-03-02T08:54:19Z ---
@Swatikkar
The error is a 404, not a 401 , so this isn't actually an authorization issue, it's that the model endpoint isn't available via the Inference API router at that URL.

A few things to check:
1. The free Inference API has limited model availability.  `microsoft/Phi-3-mini-4k-instruct` may not be currently hosted ,   verify at https://huggingface.co/microsoft/Phi-3-mini-4k-instruct   and check if the "Inference API" widget is visible on the model page.

2. If you're using `InferenceClient`, try specifying the provider explicitly:
```python
from huggingface_hub import InferenceClient

client = InferenceClient(
    model="microsoft/Phi-3-mini-4k-instruct",
    token="your_token_here"
)
```
3. For models not hosted on the free tier, you'd need to use the  Serverless Inference API with a PRO account, or run the model locally.

--- Comment by Swatikkar at 2026-03-03T05:12:27Z ---
@Saad-Mallebhari
But i have tried with other models also. same error was coming for all of those.

can you suggest some  free models which can be used now.
I can check if it works or else i will let you know.

--- Comment by Saad-Mallebhari at 2026-03-03T09:07:31Z ---
@Swatikkar
If you're getting the same 404 for every model, this is very unlikely to be model-specific.

The URL in your error:
`https://router.huggingface.co/hf-inference/models/.../v1/chat/completions`

is the **Serverless Inference API (OpenAI-compatible router)**.
A 404 from that endpoint usually means:
- The router service is not enabled for your account
- The model is not available through that router path
- Or the endpoint is being called incorrectly

## Quick Test , Try This First
Before trying more models, test with `gpt2` which is publicly
hosted and should work on the free tier:
```python
from huggingface_hub import InferenceClient

client = InferenceClient(
    model="gpt2",
    token="your_hf_token"
)

print(client.text_generation("Hello world"))
```

If this returns a 404 too, the issue is definitively with your endpoint setup or account , not the model choice.

--- Comment by Swatikkar at 2026-03-04T05:49:41Z ---
I tried that, now it's showing stopiteration error.

--- Comment by Saad-Mallebhari at 2026-03-04T09:10:07Z ---
@Swatikkar
Great the `StopIteration` error means the previous 404 is resolved and your request is now reaching the API. This is a different issue.

With `InferenceClient`, `StopIteration` usually happens when the response iterator is empty or when the call is made in a way that expects streaming results.

Try this minimal non-streaming example:
```python
from huggingface_hub import InferenceClient

client = InferenceClient(
    model="gpt2",
    token="your_hf_token"
)

result = client.text_generation(
    "Hello world",
    max_new_tokens=50
)

print(result)
```

If this still raises `StopIteration`, please share:
- The full code snippet you're running
- The complete traceback of the error

That will help determine whether the issue is coming from the client call, the model endpoint, or how the response is being consumed.

--- Comment by github-actions[bot] at 2026-03-31T08:19:12Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44361: [BUG] MLukeTokenizer fails with AttributeError on tasks
State: closed  |  Labels: bug
Author: harshaljanjani  |  Created: 2026-02-28T19:58:16Z

### System Info

* `transformers` version: `5.0.0.dev0`
* Platform: `Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39`
* Python version: `3.12.3`
* `huggingface_hub` version: `1.3.2`
* `safetensors` version: `0.7.0`
* `accelerate` version: `1.12.0`
* Accelerate config: `not installed`
* DeepSpeed version: `not installed`
* PyTorch version (accelerator?): `2.9.1+cu128 (CUDA)`
* GPU type: `NVIDIA L4`
* NVIDIA driver version: `550.90.07`
* CUDA version: `12.4`

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import MLukeTokenizer

try:
    tokenizer = MLukeTokenizer.from_pretrained(
      "studio-ousia/mluke-base", task="entity_classification"
    )
    sentence = "Japanese is an East Asian language spoken by about 128 million people, primarily in Japan."
    span = (15, 34)
    encoding = tokenizer(sentence, entity_spans=[span])
    print(tokenizer.decode(encoding["input_ids"], spaces_between_special_tokens=False))
except Exception as e:
    print(e)
```

[MIGRATION_GUIDE_V5.md](https://github.com/harshaljanjani/transformers/blob/main/MIGRATION_GUIDE_V5.md) states that v5 renamed `additional_special_tokens` to `extra_special_tokens` internally as part of the tokenizer refactor; but [tokenization_mluke.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mluke/tokenization_mluke.py#L1016) (amongst other instances) still references `self.additional_special_tokens_ids`, which is the only remaining call site under the old name that needs fixing :)

**Current Output:**

### Expected behavior

→ Entity classification task should pass without `AttributeError`.

**Output After the Fix:**


ISSUE #44360: [Bug/Discussion] The DSA indexer lacks a ReLU
State: closed  |  Labels: bug
Author: yangdsh  |  Created: 2026-02-28T19:25:43Z

### System Info

The model structure of the GLM-MOE-DSA indexer lacks a ReLU here (https://github.com/zRzRzRzRzRzRzR/transformers/blob/4ca30213c6f7aa84b55c280e02730fe14d33dac5/src/transformers/models/glm_moe_dsa/modular_glm_moe_dsa.py#L403) compared to the reference implementation (https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/inference/kernel.py#L241)

### Who can help?

@JaredforReal

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

N/A

### Expected behavior

Add ReLU

--- Comment by yangdsh at 2026-03-05T00:41:42Z ---
Hi @Rocketknight1. I noticed that you close the related PR but I believe the bug is real. Is #43912 still WIP? Could we treat GlmMoeDsa as a golden standard now? Thank you!

--- Comment by Rocketknight1 at 2026-03-05T15:24:23Z ---
Hi @yangdsh yes, the issue is real, but we're still closing most code agent PRs that show no signs of human review. They're just pointless - the bottleneck for us is reviewing code, and we can run code agents ourselves if we need to, so opening loads of code agent PRs just spams the notifications of the maintainers and wastes our time.

I'll try to get a proper review for this soon!


ISSUE #44355: Errors occur when running compiled Python files.
State: closed  |  Labels: bug
Author: HuaC-Z  |  Created: 2026-02-28T09:49:16Z

### System Info

linux
transformers >=4.51.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

error code:TypeError: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method

file: transformers/utils/doc.py

Code snippet with the error
def get_docstring_indentation_level(func):
    """Return the indentation level of the start of the docstring of a class or function (or method)."""
    # We assume classes are always defined in the global scope
    if inspect.isclass(func):
        return 4
    source = inspect.getsource(func)
    first_line = source.splitlines()[0]
    function_def_level = len(first_line) - len(first_line.lstrip())
    return 4 + function_def_level

### Expected behavior

Fix this error

--- Comment by Saad-Mallebhari at 2026-02-28T10:11:06Z ---
@HuaC-Z
This happens because `inspect.getsource()` can't handle Cython-compiled functions , it only works on pure Python objects. When running compiled `.pyc` files, some functions become `cython_function_or_method` which triggers the `TypeError`.

The fix is to guard the `getsource` call in `get_docstring_indentation_level()`:
```python
def get_docstring_indentation_level(func):
    if inspect.isclass(func):
        return 4
    try:
        source = inspect.getsource(func)
    except (TypeError, OSError):
        return 4  # fallback for compiled/Cython functions
    first_line = source.splitlines()[0]
    function_def_level = len(first_line) - len(first_line.lstrip())
    return 4 + function_def_level
```
Catching both `TypeError` and `OSError` covers Cython functions and cases where source files aren't available on disk.

--- Comment by Rocketknight1 at 2026-03-02T14:01:38Z ---
Why are you running those utils on `.pyc` files..?

--- Comment by github-actions[bot] at 2026-03-31T08:19:14Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44351: cannot import name 'HybridCache' from 'transformers
State: closed  |  Labels: bug
Author: wade0604  |  Created: 2026-02-28T03:46:26Z

### System Info

transformers 5.2.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import Cache, DynamicCache, EncoderDecoderCache, HybridCache, PreTrainedModel

### Expected behavior

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'HybridCache' from 'transformers' (/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/transformers/__init__.py)
>>>

--- Comment by ArjunPimpale at 2026-02-28T07:27:17Z ---
HybridCache function was depricated
Check #39106


ISSUE #44336: Some ANSI codes are generated by utils/loading_report even when not connected to terminal
State: closed  |  Labels: bug
Author: foxik  |  Created: 2026-02-27T17:31:28Z

### System Info

The bug does not depend on system info, it is obvious in sources.

### Who can help?

@Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Hi @Cyrilvallez ,

the 0fa2c2f changed ANSI handling in `utils/loading_report.py` from `ANSI` class to `PALETTE` plus `_color`. However, the `bold` and `italic` are now used without any check that we are connected to terminal, notably:

https://github.com/huggingface/transformers/blob/46d09b53ec4f38ff3e6d893c802e5f9af98fba68/src/transformers/utils/loading_report.py#L266

https://github.com/huggingface/transformers/blob/46d09b53ec4f38ff3e6d893c802e5f9af98fba68/src/transformers/utils/loading_report.py#L182

https://github.com/huggingface/transformers/blob/46d09b53ec4f38ff3e6d893c802e5f9af98fba68/src/transformers/utils/loading_report.py#L191

https://github.com/huggingface/transformers/blob/46d09b53ec4f38ff3e6d893c802e5f9af98fba68/src/transformers/utils/loading_report.py#L200

https://github.com/huggingface/transformers/blob/46d09b53ec4f38ff3e6d893c802e5f9af98fba68/src/transformers/utils/loading_report.py#L214

https://github.com/huggingface/transformers/blob/46d09b53ec4f38ff3e6d893c802e5f9af98fba68/src/transformers/utils/loading_report.py#L230

so these are now printed even when redirecting the output.

### Expected behavior

Do not print ANSI codes for bold and italics when `sys.stdout.isatty()` is false, probably by using `_color`. But `_color` always also prints the ANSI reset at the end, and it is not clear to me when should some of the `_PALETTE['italics']` end.

--- Comment by Rocketknight1 at 2026-03-04T14:10:15Z ---
cc @cyrilvallez I suspect this bug is real! There are plenty of code agent fixes to choose from if you agree 😅


ISSUE #44327: decode_spans in QA pipeline crashes with ValueError: kth out of bounds when len(scores_flat) == top_k
State: closed  |  Labels:
Author: jakipradip-patra  |  Created: 2026-02-27T15:05:31Z

### System Info

- `transformers` version: 4.39.0 (also verified present in 4.53.3 and `main` branch)
- Python version: 3.10
- NumPy version: 1.x
- OS: Linux (AWS SageMaker)

### Who can help?

@Narsil

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder
- [ ] My own task or dataset

### Reproduction

The `decode_spans` function in `src/transformers/pipelines/question_answering.py` has an off-by-one boundary check on line 81:

```python
scores_flat = candidates.flatten()
if topk == 1:
    idx_sort = [np.argmax(scores_flat)]
elif len(scores_flat) < topk:          # <-- should be <=
    idx_sort = np.argsort(-scores_flat)
else:
    idx = np.argpartition(-scores_flat, topk)[0:topk]  # crashes when len == topk
    idx_sort = idx[np.argsort(-scores_flat[idx])]
```

`np.argpartition(array, kth)` requires `0 <= kth < len(array)`. When `len(scores_flat) == topk`, the guard `len(scores_flat) < topk` evaluates to `False`, and `argpartition` is called with `kth == len(array)`, which is out of bounds.

**Minimal reproduction:**

```python
import numpy as np

def decode_spans_bug(seq_len=10, topk=100, max_answer_len=15):
    """Reproduces the bug when seq_len² == topk"""
    start = np.random.rand(1, seq_len)
    end = np.random.rand(1, seq_len)
    outer = np.matmul(np.expand_dims(start, -1), np.expand_dims(end, 1))
    candidates = np.tril(np.triu(outer), max_answer_len - 1)
    scores_flat = candidates.flatten()

    print(f"len(scores_flat)={len(scores_flat)}, topk={topk}")
    print(f"len < topk: {len(scores_flat) < topk}")

    # This is what decode_spans does:
    idx = np.argpartition(-scores_flat, topk)[0:topk]  # CRASH

decode_spans_bug()  # ValueError: kth(=100) out of bounds (100)
```

Since `scores_flat` has `seq_len²` elements (from the outer product matrix), this triggers when `topk` is a perfect square and `seq_len = sqrt(topk)`. For example:
- `top_k=100` crashes when `seq_len=10`
- `top_k=400` crashes when `seq_len=20`

This occurs in practice when using `top_k > 1` with very short input contexts (e.g., a short question with minimal context text).

### Expected behavior

The function should handle the boundary case where `len(scores_flat) == topk` without crashing. The fix is changing `<` to `<=` on line 81:

```python
elif len(scores_flat) <= topk:    # fixed: catches the boundary case
    idx_sort = np.argsort(-scores_flat)
```

When `len(scores_flat) == topk`, we want ALL elements anyway, so `argsort` is both correct and sufficient — no need for `argpartition`.

--- Comment by Rocketknight1 at 2026-03-02T13:03:51Z ---
The `question-answering` pipeline has been deprecated in `v5` and is no longer available! For QA tasks like this we recommend just asking the question to a standard causal LM.

--- Comment by mvanhorn at 2026-03-10T23:53:01Z ---
I've submitted a fix for this in PR #44584. Changed the boundary check from `<` to `<=` so `argsort` is used when `len(scores_flat) == topk`, preventing the `argpartition` out-of-bounds crash.


ISSUE #44322: AttributeError: 'Qwen3_5Config' object has no attribute 'num_attention_heads'
State: closed  |  Labels: bug
Author: zzc0430  |  Created: 2026-02-27T10:52:57Z

### System Info

- transfomers: 5.3.0.dev0

### Who can help?

@remi-or @ArthurZucker @McPatate

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

- use transformers serve-cli to deploy
```transformers serve --force-model Qwen/Qwen3.5-27B --port 9016 --continuous-batching```
- logs
```
2026-02-27T18:44:43.002681005+08:00 stdout F INFO:     ::1:46248 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2026-02-27T18:44:43.002993914+08:00 stderr F Error in generation loop: 'Qwen3_5Config' object has no attribute 'num_attention_heads'
2026-02-27T18:44:43.002996802+08:00 stderr F Traceback (most recent call last):
2026-02-27T18:44:43.002998847+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/generation/continuous_batching/continuous_api.py", line 799, in _run_generation_loop
2026-02-27T18:44:43.003000958+08:00 stderr F     paged_attention_cache = PagedAttentionCache(
2026-02-27T18:44:43.003002582+08:00 stderr F                             ^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.003004339+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/generation/continuous_batching/cache.py", line 144, in __init__
2026-02-27T18:44:43.003006276+08:00 stderr F     self.num_key_value_heads: int = kv_heads if kv_heads is not None else config.num_attention_heads
2026-02-27T18:44:43.003007921+08:00 stderr F                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.003009773+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/configuration_utils.py", line 164, in __getattribute__
2026-02-27T18:44:43.003011403+08:00 stderr F     return super().__getattribute__(key)
2026-02-27T18:44:43.003013125+08:00 stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.003016089+08:00 stderr F AttributeError: 'Qwen3_5Config' object has no attribute 'num_attention_heads'
2026-02-27T18:44:43.007934561+08:00 stderr F ERROR:    Exception in ASGI application
2026-02-27T18:44:43.00795328+08:00 stderr F   + Exception Group Traceback (most recent call last):
2026-02-27T18:44:43.007956234+08:00 stderr F   |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
2026-02-27T18:44:43.007958168+08:00 stderr F   |     yield
2026-02-27T18:44:43.007960098+08:00 stderr F   |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 186, in __call__
2026-02-27T18:44:43.007961889+08:00 stderr F   |     async with anyio.create_task_group() as task_group:
2026-02-27T18:44:43.007963958+08:00 stderr F   |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 763, in __aexit__
2026-02-27T18:44:43.007965588+08:00 stderr F   |     raise BaseExceptionGroup(
2026-02-27T18:44:43.007967316+08:00 stderr F   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2026-02-27T18:44:43.007968864+08:00 stderr F   +-+---------------- 1 ----------------
2026-02-27T18:44:43.007970393+08:00 stderr F     | Traceback (most recent call last):
2026-02-27T18:44:43.007972218+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
2026-02-27T18:44:43.007973802+08:00 stderr F     |     result = await app(  # type: ignore[func-returns-value]
2026-02-27T18:44:43.007975466+08:00 stderr F     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.007977074+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2026-02-27T18:44:43.007978615+08:00 stderr F     |     return await self.app(scope, receive, send)
2026-02-27T18:44:43.007980263+08:00 stderr F     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.007982039+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2026-02-27T18:44:43.007983675+08:00 stderr F     |     await super().__call__(scope, receive, send)
2026-02-27T18:44:43.007985308+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
2026-02-27T18:44:43.007986948+08:00 stderr F     |     await self.middleware_stack(scope, receive, send)
2026-02-27T18:44:43.007988502+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
2026-02-27T18:44:43.007990248+08:00 stderr F     |     raise exc
2026-02-27T18:44:43.007991832+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
2026-02-27T18:44:43.007993575+08:00 stderr F     |     await self.app(scope, receive, _send)
2026-02-27T18:44:43.007995117+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 185, in __call__
2026-02-27T18:44:43.007996831+08:00 stderr F     |     with collapse_excgroups():
2026-02-27T18:44:43.007998751+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/contextlib.py", line 158, in __exit__
2026-02-27T18:44:43.008000468+08:00 stderr F     |     self.gen.throw(typ, value, traceback)
2026-02-27T18:44:43.008002057+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
2026-02-27T18:44:43.008003849+08:00 stderr F     |     raise exc
2026-02-27T18:44:43.0080053+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 187, in __call__
2026-02-27T18:44:43.008006815+08:00 stderr F     |     response = await self.dispatch_func(request, call_next)
2026-02-27T18:44:43.008008509+08:00 stderr F     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008021307+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/cli/serve.py", line 551, in get_or_set_request_id
2026-02-27T18:44:43.008023938+08:00 stderr F     |     response = await call_next(request)
2026-02-27T18:44:43.008026277+08:00 stderr F     |                ^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008028766+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
2026-02-27T18:44:43.008031295+08:00 stderr F     |     raise app_exc
2026-02-27T18:44:43.008033757+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
2026-02-27T18:44:43.008037063+08:00 stderr F     |     await self.app(scope, receive_or_disconnect, send_no_error)
2026-02-27T18:44:43.008039529+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2026-02-27T18:44:43.008042034+08:00 stderr F     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2026-02-27T18:44:43.008044727+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2026-02-27T18:44:43.008047494+08:00 stderr F     |     raise exc
2026-02-27T18:44:43.008049912+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-02-27T18:44:43.008052506+08:00 stderr F     |     await app(scope, receive, sender)
2026-02-27T18:44:43.008054965+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
2026-02-27T18:44:43.008057432+08:00 stderr F     |     await self.middleware_stack(scope, receive, send)
2026-02-27T18:44:43.008059883+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
2026-02-27T18:44:43.008062259+08:00 stderr F     |     await route.handle(scope, receive, send)
2026-02-27T18:44:43.008067601+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
2026-02-27T18:44:43.008069285+08:00 stderr F     |     await self.app(scope, receive, send)
2026-02-27T18:44:43.008072004+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
2026-02-27T18:44:43.008074277+08:00 stderr F     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2026-02-27T18:44:43.008076397+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2026-02-27T18:44:43.008078576+08:00 stderr F     |     raise exc
2026-02-27T18:44:43.008080726+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-02-27T18:44:43.008082688+08:00 stderr F     |     await app(scope, receive, sender)
2026-02-27T18:44:43.008084702+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
2026-02-27T18:44:43.008086675+08:00 stderr F     |     response = await f(request)
2026-02-27T18:44:43.008089112+08:00 stderr F     |                ^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008091164+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
2026-02-27T18:44:43.008093467+08:00 stderr F     |     raw_response = await run_endpoint_function(
2026-02-27T18:44:43.008095807+08:00 stderr F     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008098262+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
2026-02-27T18:44:43.008103772+08:00 stderr F     |     return await run_in_threadpool(dependant.call, **values)
2026-02-27T18:44:43.008105888+08:00 stderr F     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008108133+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
2026-02-27T18:44:43.008111537+08:00 stderr F     |     return await anyio.to_thread.run_sync(func, *args)
2026-02-27T18:44:43.008113878+08:00 stderr F     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008116457+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2026-02-27T18:44:43.008118569+08:00 stderr F     |     return await get_async_backend().run_sync_in_worker_thread(
2026-02-27T18:44:43.00812082+08:00 stderr F     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008123449+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
2026-02-27T18:44:43.00812586+08:00 stderr F     |     return await future
2026-02-27T18:44:43.008128231+08:00 stderr F     |            ^^^^^^^^^^^^
2026-02-27T18:44:43.008130589+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 943, in run
2026-02-27T18:44:43.008133+08:00 stderr F     |     result = context.run(func, *args)
2026-02-27T18:44:43.008135637+08:00 stderr F     |              ^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008138117+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/cli/serve.py", line 504, in chat_completion
2026-02-27T18:44:43.008140634+08:00 stderr F     |     return self.continuous_batching_chat_completion(body, request.state.request_id)
2026-02-27T18:44:43.008143826+08:00 stderr F     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008146686+08:00 stderr F     |   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/cli/serve.py", line 828, in continuous_batching_chat_completion
2026-02-27T18:44:43.008148308+08:00 stderr F     |     ).to(model.device)["input_ids"][0]
2026-02-27T18:44:43.008150004+08:00 stderr F     |       ^^
2026-02-27T18:44:43.008151596+08:00 stderr F     | AttributeError: 'str' object has no attribute 'to'
2026-02-27T18:44:43.008153096+08:00 stderr F     +------------------------------------
2026-02-27T18:44:43.008154576+08:00 stderr F
2026-02-27T18:44:43.008156224+08:00 stderr F During handling of the above exception, another exception occurred:
2026-02-27T18:44:43.008157577+08:00 stderr F
2026-02-27T18:44:43.008159094+08:00 stderr F Traceback (most recent call last):
2026-02-27T18:44:43.008160837+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
2026-02-27T18:44:43.00816237+08:00 stderr F     result = await app(  # type: ignore[func-returns-value]
2026-02-27T18:44:43.008163893+08:00 stderr F              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008165505+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2026-02-27T18:44:43.008167042+08:00 stderr F     return await self.app(scope, receive, send)
2026-02-27T18:44:43.008168552+08:00 stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008170161+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2026-02-27T18:44:43.008171651+08:00 stderr F     await super().__call__(scope, receive, send)
2026-02-27T18:44:43.008173262+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
2026-02-27T18:44:43.008177248+08:00 stderr F     await self.middleware_stack(scope, receive, send)
2026-02-27T18:44:43.008178717+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
2026-02-27T18:44:43.008180261+08:00 stderr F     raise exc
2026-02-27T18:44:43.008181747+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
2026-02-27T18:44:43.00818324+08:00 stderr F     await self.app(scope, receive, _send)
2026-02-27T18:44:43.008188175+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 185, in __call__
2026-02-27T18:44:43.008190636+08:00 stderr F     with collapse_excgroups():
2026-02-27T18:44:43.008193214+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/contextlib.py", line 158, in __exit__
2026-02-27T18:44:43.008195609+08:00 stderr F     self.gen.throw(typ, value, traceback)
2026-02-27T18:44:43.008198108+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
2026-02-27T18:44:43.008201128+08:00 stderr F     raise exc
2026-02-27T18:44:43.00820262+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 187, in __call__
2026-02-27T18:44:43.008204139+08:00 stderr F     response = await self.dispatch_func(request, call_next)
2026-02-27T18:44:43.008205637+08:00 stderr F                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008207155+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/cli/serve.py", line 551, in get_or_set_request_id
2026-02-27T18:44:43.008208639+08:00 stderr F     response = await call_next(request)
2026-02-27T18:44:43.008210188+08:00 stderr F                ^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008211761+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 163, in call_next
2026-02-27T18:44:43.008213272+08:00 stderr F     raise app_exc
2026-02-27T18:44:43.008214914+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/base.py", line 149, in coro
2026-02-27T18:44:43.008216424+08:00 stderr F     await self.app(scope, receive_or_disconnect, send_no_error)
2026-02-27T18:44:43.008218014+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2026-02-27T18:44:43.00821985+08:00 stderr F     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2026-02-27T18:44:43.008221537+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2026-02-27T18:44:43.008223062+08:00 stderr F     raise exc
2026-02-27T18:44:43.00822454+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-02-27T18:44:43.008226123+08:00 stderr F     await app(scope, receive, sender)
2026-02-27T18:44:43.008227586+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
2026-02-27T18:44:43.008229068+08:00 stderr F     await self.middleware_stack(scope, receive, send)
2026-02-27T18:44:43.008230552+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
2026-02-27T18:44:43.008232061+08:00 stderr F     await route.handle(scope, receive, send)
2026-02-27T18:44:43.008233752+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
2026-02-27T18:44:43.008235227+08:00 stderr F     await self.app(scope, receive, send)
2026-02-27T18:44:43.008238047+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
2026-02-27T18:44:43.00824318+08:00 stderr F     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2026-02-27T18:44:43.008244715+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2026-02-27T18:44:43.008246386+08:00 stderr F     raise exc
2026-02-27T18:44:43.008247856+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2026-02-27T18:44:43.008249355+08:00 stderr F     await app(scope, receive, sender)
2026-02-27T18:44:43.008251094+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
2026-02-27T18:44:43.008252601+08:00 stderr F     response = await f(request)
2026-02-27T18:44:43.008254103+08:00 stderr F                ^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008255743+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
2026-02-27T18:44:43.008257325+08:00 stderr F     raw_response = await run_endpoint_function(
2026-02-27T18:44:43.008259113+08:00 stderr F                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.00826063+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
2026-02-27T18:44:43.008262108+08:00 stderr F     return await run_in_threadpool(dependant.call, **values)
2026-02-27T18:44:43.008263768+08:00 stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008265222+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
2026-02-27T18:44:43.008266775+08:00 stderr F     return await anyio.to_thread.run_sync(func, *args)
2026-02-27T18:44:43.00826834+08:00 stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008269891+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2026-02-27T18:44:43.008271492+08:00 stderr F     return await get_async_backend().run_sync_in_worker_thread(
2026-02-27T18:44:43.008273117+08:00 stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.00827473+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2441, in run_sync_in_worker_thread
2026-02-27T18:44:43.008276317+08:00 stderr F     return await future
2026-02-27T18:44:43.008277785+08:00 stderr F            ^^^^^^^^^^^^
2026-02-27T18:44:43.008279313+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 943, in run
2026-02-27T18:44:43.008280816+08:00 stderr F     result = context.run(func, *args)
2026-02-27T18:44:43.008282353+08:00 stderr F              ^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008287887+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/cli/serve.py", line 504, in chat_completion
2026-02-27T18:44:43.008290487+08:00 stderr F     return self.continuous_batching_chat_completion(body, request.state.request_id)
2026-02-27T18:44:43.008293356+08:00 stderr F            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T18:44:43.008296116+08:00 stderr F   File "/home/work/.conda/envs/swift/lib/python3.11/site-packages/transformers/cli/serve.py", line 828, in continuous_batching_chat_completion
2026-02-27T18:44:43.008299386+08:00 stderr F     ).to(model.device)["input_ids"][0]
2026-02-27T18:44:43.008300914+08:00 stderr F       ^^
2026-02-27T18:44:43.008302548+08:00 stderr F AttributeError: 'str' object has no attribute 'to'
```

### Expected behavior

support deploy qwen3.5

--- Comment by Rocketknight1 at 2026-02-27T14:41:12Z ---
Seems like a continuous batching thing, so cc @remi-or @mcpatate!

--- Comment by vvvdwbvvv at 2026-03-02T09:27:00Z ---
@zzc0430 Thanks, I am also struggle with this.

--- Comment by remi-or at 2026-03-02T11:10:55Z ---
Thanks for bringing this to our attention! The underlying issue is that continuous batching does not yet support linear attention nor multi-modal models.
I will open a PR to better highlight this, and it is in our road map. Closing this issue until then!


ISSUE #44315: Liger Kernel is not applied when creating the model with `model_init`
State: closed  |  Labels: bug
Author: linfeng-du  |  Created: 2026-02-27T02:57:03Z

### System Info

N/A

### Who can help?

@SunMarc In `Trainer.train`, the Liger Kernel is not applied when the model is instantiated via `call_model_init`. As a result, hyperparameter search runs cannot leverage this kernel for acceleration.

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

N/A

### Expected behavior

Liger Kernel should be configurable via `args.use_liger_kernel` when the model is instantiated via `model_init`.

--- Comment by SunMarc at 2026-02-27T11:50:29Z ---
Oh yeah, we don't test that much hypermarameter search as this is not a feature we feel like is being used a lot. But yeah we should probably move function that are applied to the model after we call `call_model_init`. Can you check if moving
```python
        if self.args.use_liger_kernel:
            apply_liger_kernel(model, self.args.liger_kernel_config)
```
just after this code in train() would fix your issue ?
```python
  # Model re-init
        if self.model_init is not None:
            # Seed must be set before instantiating the model when using model_init.
            enable_full_determinism(args.seed) if args.full_determinism else set_seed(args.seed)
            self.model = self.call_model_init(trial)
            # Reinitializes optimizer and scheduler
            self.optimizer, self.lr_scheduler = None, None
            if self.place_model_on_device:
                self._move_model_to_device(self.model, args.device)
            self.model_wrapped = self.model
```

--- Comment by linfeng-du at 2026-02-27T11:59:13Z ---
Yes that would fix it. Thanks!

--- Comment by SunMarc at 2026-02-27T13:29:22Z ---
Would you like to open a PR to add this ?


ISSUE #44303: Less verbose `tqdm` weight loading (`Loading weights:  38% ... Materializing param=....]` log)
State: closed  |  Labels: Feature request
Author: fxmarty-amd  |  Created: 2026-02-26T16:20:08Z

### Feature request

Hello,

Currently when redirecting any `PretrainedModel.from_pretrained` to log file, we get a huge:

```
Loading weights:  38%|███▊      | 74/197 [00:00<00:00, 11546.82it/s, Materializing param=model.decoder.layers.4.final_layer_norm.bias]
Loading weights:  38%|███▊      | 74/197 [00:00<00:00, 11490.82it/s, Materializing param=model.decoder.layers.4.final_layer_norm.bias]
Loading weights:  38%|███▊      | 75/197 [00:00<00:00, 11550.31it/s, Materializing param=model.decoder.layers.4.final_layer_norm.weight]
Loading weights:  38%|███▊      | 75/197 [00:00<00:00, 11495.44it/s, Materializing param=model.decoder.layers.4.final_layer_norm.weight]
Loading weights:  39%|███▊      | 76/197 [00:00<00:00, 11556.23it/s, Materializing param=model.decoder.layers.4.self_attn.k_proj.bias]
Loading weights:  39%|███▊      | 76/197 [00:00<00:00, 11500.36it/s, Materializing param=model.decoder.layers.4.self_attn.k_proj.bias]
Loading weights:  39%|███▉      | 77/197 [00:00<00:00, 11549.60it/s, Materializing param=model.decoder.layers.4.self_attn.k_proj.weight]
Loading weights:  39%|███▉      | 77/197 [00:00<00:00, 11494.52it/s, Materializing param=model.decoder.layers.4.self_attn.k_proj.weight]
```

with many lines.

This comes from https://github.com/huggingface/transformers/blob/af5dfb6c5cb98091aa76c2c768bd4b7a38891b01/src/transformers/core_model_loading.py#L1203

This makes log files huge, especially in CI which loads many models.

Could we consider making this less verbose?

Thank you!

### Motivation

Several tens of hundreds of thousands lines CI log files

### Your contribution

maybe


ISSUE #44297: [BUG] tokenizer.save_pretrained: tokenizer_class in tokenizer_config.json doesn't match the original
State: closed  |  Labels: bug
Author: Jintao-Huang  |  Created: 2026-02-26T11:37:49Z

### System Info

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3.5-27B')
tokenizer.save_pretrained('output')
```

->

### Who can help?

-

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

--- Comment by Rocketknight1 at 2026-02-26T14:17:54Z ---
cc @itazap @arthurzucker

--- Comment by cynricfu at 2026-02-27T08:39:09Z ---
I guess this is caused by string mismatch between Qwen3.5 tokenizer config file's `tokenizer_class` and `TOKENIZER_MAPPING_NAMES["qwen3_5"]`.

In Qwen3.5 [tokenizer_config.json](https://huggingface.co/Qwen/Qwen3.5-27B/blob/main/tokenizer_config.json#L292):
```
...
    "tokenizer_class": "Qwen2Tokenizer",
...
```

While in `TOKENIZER_MAPPING_NAMES`:
https://github.com/huggingface/transformers/blob/24db9db27e3a360149136ac7ba8ba253bb2d899f/src/transformers/models/auto/tokenization_auto.py#L267-L268

--- Comment by JJJYmmm at 2026-03-02T05:57:23Z ---
Hi! This is a temporary fix (setting `Qwen2Tokenizer` in tokenizer_config.json) for sglang/vllm, since they only support transformers v4 for now. We will update the ckpts to the correct Qwen3_5Tokenizer after they upgrade to v5.
The difference between `Qwen2Tokenizer` and `Qwen3_5Tokenizer` is the vocab size and the pre-tokenizer rules. Currently, `TokenizersBackend` should also work.

--- Comment by ArthurZucker at 2026-03-02T14:47:02Z ---
Yes! @cynricfu is right!
Because there is a difference between the `TOKENIZER_MAPPING_NAMES [model_type]` and ` "tokenizer_class": "Qwen2Tokenizer"` in v5 this means its safer to load the `tokenizer.json` directly because for us it means the authors probably just want this. So we use `TokenizersBackend` which is the equivalent of what would have been loaded with the previous `Qwen2TokenizerFast` class -> it was just reading the json files

--- Comment by github-actions[bot] at 2026-03-29T08:07:51Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #44295: An error occurs when reading position_ids after registering it as a buffer.
State: closed  |  Labels: bug
Author: nursery42  |  Created: 2026-02-26T08:54:10Z

### System Info

transformers==5.2.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

after
```
self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)),persistent=False)
```

```
position_ids = self.position_ids[:, :seq_length].expand(batch_size, -1)
```
ids is

### Expected behavior

It should have been a sequence from 1 to n.

--- Comment by Rocketknight1 at 2026-02-26T14:02:46Z ---
I'm not sure what exactly you did here - can you send us the script or the model you're using? You're reading uninitialized memory somehow, it seems like

--- Comment by vasqu at 2026-02-26T14:22:22Z ---
I suppose something custom is done; for buffers you need to copy them in the init, e.g. see RoPE https://github.com/huggingface/transformers/blob/d4cb841690bd963f2de7119d5f569ff139b3d989/src/transformers/modeling_utils.py#L2321-L2322

It is random memory (torch empty) atm if it is not define in `_init_weights`

--- Comment by nursery42 at 2026-02-27T03:24:34Z ---
> I'm not sure what exactly you did here - can you send us the script or the model you're using? You're reading uninitialized memory somehow, it seems like

like this model [gte-reranker](https://huggingface.co/Alibaba-NLP/new-impl/blob/main/modeling.py#L308).

The same model file exhibits inconsistent behavior across two different versions (v4 and v5): the position_ids in RoPE embedding in the latter version behaves abnormally, and an error is thrown when it forward. I have checked and it does not seem to be an issue with their implementation itself.

--- Comment by vasqu at 2026-02-27T07:35:01Z ---
Yea, it's like I said their buffer needs to be properly initialized in https://huggingface.co/Alibaba-NLP/new-impl/blob/main/modeling.py#L811

It allows us to properly init models on meta devices

--- Comment by github-actions[bot] at 2026-03-29T08:07:53Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.