ISSUE #45800: incompatiblity between torch 2.4.1 and transformers 5.8.0
State: closed  |  Labels: bug
Author: nickjyj  |  Created: 2026-05-06T03:19:54Z

### System Info

torch 2.4.1
transformers 5.8.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

dockerfile:
```docker
FROM nvcr.io/nvidia/tensorrt:25.04-py3
ARG DEBIAN_FRONTEND=noninteractive

# Install dependencies
RUN apt-get update && apt-get install libgl1 -y
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu124
RUN pip install rfdetr
```

python code:
```python
from rfdetr import RFDETRMedium
model = RFDETRMedium()
```

error logs:
```
predict  | Traceback (most recent call last):
predict  |   File "/app/tmp.py", line 1, in <module>
predict  |     from rfdetr import RFDETRMedium
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/__init__.py", line 7, in <module>
predict  |     from rfdetr.detr import (
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/detr.py", line 52, in <module>
predict  |     from rfdetr.models import PostProcess, build_model
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/__init__.py", line 17, in <module>
predict  |     from rfdetr.models.lwdetr import build_model
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/lwdetr.py", line 30, in <module>
predict  |     from rfdetr.models.backbone import build_backbone
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/__init__.py", line 15, in <module>
predict  |     from rfdetr.models.backbone.backbone import Backbone
predict  |   File "/usr/local/lib/python3.12/dist-packages/rfdetr/models/backbone/backbone.py", line 22, in <module>
predict  |     from peft import PeftModel
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/__init__.py", line 17, in <module>
predict  |     from .auto import (
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/auto.py", line 31, in <module>
predict  |     from .config import PeftConfig
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/config.py", line 30, in <module>
predict  |     from .utils import CONFIG_NAME, PeftType, TaskType
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/utils/__init__.py", line 15, in <module>
predict  |     from .constants import ALLOWED_COMPUTE_DTYPES, UPCAST_DTYPES
predict  |   File "/usr/local/lib/python3.12/dist-packages/peft/utils/constants.py", line 16, in <module>
predict  |     from transformers import BloomPreTrainedModel
predict  |   File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2226, in __getattr__
predict  |     module = self._get_module(self._class_to_module[name])
predict  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2460, in _get_module
predict  |     raise e
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/utils/import_utils.py", line 2458, in _get_module
predict  |     return importlib.import_module("." + module_name, self.__name__)
predict  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
predict  |     return _bootstrap._gcd_import(name[level:], package, level)
predict  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/models/bloom/modeling_bloom.py", line 26, in <module>
predict  |     from ...modeling_layers import GradientCheckpointingLayer
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py", line 27, in <module>
predict  |     from .processing_utils import Unpack
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 79, in <module>
predict  |     from .modeling_utils import PreTrainedAudioTokenizerBase
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py", line 69, in <module>
predict  |     from .integrations.finegrained_fp8 import ALL_FP8_EXPERTS_FUNCTIONS
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/finegrained_fp8.py", line 30, in <module>
predict  |     from .moe import ExpertsInterface, use_experts_implementation
predict  |   File "/usr/local/lib/python3.12/dist-packages/transformers/integrations/moe.py", line 250, in <module>
predict  |     torch.library.custom_op("transformers::grouped_mm_fallback", _grouped_mm_fallback, mutates_args=())
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 142, in custom_op
predict  |     return inner(fn)
predict  |            ^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 119, in inner
predict  |     schema_str = torch._custom_op.impl.infer_schema(fn, mutates_args)
predict  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 42, in infer_schema
predict  |     error_fn(
predict  |   File "/usr/local/lib/python3.12/dist-packages/torch/_library/infer_schema.py", line 21, in error_fn
predict  |     raise ValueError(
predict  | ValueError: infer_schema(func): Parameter input has unsupported type torch.Tensor. The valid types are: dict_keys([<class 'torch.Tensor'>, typing.Optional[torch.Tensor], typing.Sequence[torch.Tensor], typing.List[torch.Tensor], typing.Sequence[typing.Optional[torch.Tensor]], typing.List[typing.Optional[torch.Tensor]], <class 'int'>, typing.Optional[int], typing.Sequence[int], typing.List[int], typing.Optional[typing.Sequence[int]], typing.Optional[typing.List[int]], <class 'float'>, typing.Optional[float], typing.Sequence[float], typing.List[float], typing.Optional[typing.Sequence[float]], typing.Optional[typing.List[float]], <class 'bool'>, typing.Optional[bool], typing.Sequence[bool], typing.List[bool], typing.Optional[typing.Sequence[bool]], typing.Optional[typing.List[bool]], <class 'str'>, typing.Optional[str], typing.Union[int, float, bool], typing.Union[int, float, bool, NoneType], typing.Sequence[typing.Union[int, float, bool]], typing.List[typing.Union[int, float, bool]], <class 'torch.dtype'>, typing.Optional[torch.dtype], <class 'torch.device'>, typing.Optional[torch.device]]). Got func with signature (input: 'torch.Tensor', weight: 'torch.Tensor', offs: 'torch.Tensor') -> 'torch.Tensor')
```

### Expected behavior

it should not generate any errors

--- Comment by rraghavkaushik at 2026-05-06T10:53:47Z ---
I tried reproducing this issue and found the cause.

The integrations/moe.py file has `from __future__ import annotations`  at the top, which makes all type annotations strings, and hence torch.Tensor is stored as "torch.Tensor" rather than the actual class object at definition time.

`torch.library.custom_op` (in PyTorch 2.4.1) fails as it sees the string and not the type.

If we pass an explicit schema string to `torch.library.custom_op`, PyTorch skips the annotation inference step entirely and the import succeeds:

```
torch.library.custom_op(
    "transformers::grouped_mm_fallback",
    _grouped_mm_fallback,
    mutates_args=(),
    schema="(Tensor input, Tensor weight, Tensor offs) -> Tensor",
)
```


[BUG] HF Hub is DOWN on AWS in Xet mode

ISSUE #45797: [BUG] HF Hub is DOWN on AWS in Xet mode
State: closed  |  Labels: bug
Author: michaelroyzen  |  Created: 2026-05-05T23:41:44Z

### System Info

# Xet-backed model download hangs midway for `Qwen/Qwen3.6-35B-A3B`

## Summary

Downloading `Qwen/Qwen3.6-35B-A3B` on a p5en.48xlarge in us-east-2 on AWS hangs partway through when using the default Hugging Face download path. Clearing `~/.cache` does not fix it.

The evidence suggests the hang is in the Hugging Face Xet-backed download path (`hf-xet` / CAS), not local disk, cache corruption, or general connectivity. Disabling Xet with `HF_HUB_DISABLE_XET=1` works around the issue.

## Reproduction

```bash
uv run hf download Qwen/Qwen3.6-35B-A3B
```

The dry run succeeds immediately:

```bash
uv run hf download Qwen/Qwen3.6-35B-A3B --dry-run
```

Output:

```text
[dry-run] Will download 40 files (out of 40) totalling 71.9G.
```

## Observed Behavior

The real download starts, writes several GB to the Hugging Face cache, and then stalls indefinitely with no visible progress.

In one reproduction:

- The cache grew to about `4.6G`.
- The Python process stayed alive.
- CPU usage continued, but disk writes stopped.
- TCP sockets to Hugging Face / Xet endpoints stayed open but idle.
- `strace` showed the process mostly waking on timers, with no meaningful network reads/writes.
- The Xet log showed repeated `403 Forbidden` responses from signed `us.aws.cdn.hf.co/xorbs/...` range URLs, followed by retrieval URL refresh/retry loops.

Representative Xet log lines:

```text
Received CAS response ... status_code=403
Refreshing expired retrieval URLs
Retry on 403 (Forbidden) enabled): "s3::get_range" api call failed ... HTTP status client error (403 Forbidden) for url (https://us.aws.cdn.hf.co/xorbs/...)
Retrieval URLs refreshed successfully
```

## Expected Behavior

The model download should either complete successfully or fail with a clear error. It should not stall indefinitely without surfacing the repeated `403 Forbidden` failures to the CLI.

## Workaround

Disabling Xet allows the download path to work:

```bash
HF_HUB_DISABLE_XET=1 uv run hf download Qwen/Qwen3.6-35B-A3B
```

A smaller shard test completed successfully with Xet disabled:

```bash
HF_HUB_DISABLE_XET=1 uv run hf download Qwen/Qwen3.6-35B-A3B \
  --include model-00007-of-00026.safetensors \
  --local-dir /tmp/qwen-no-xet-test
```

That downloaded a `1.1G` safetensors shard successfully in a few seconds.

## Environment

All tests run on a p5en.48xlarge instance in us-east-2 on AWS.

```text
OS: Linux 6.8.0-1050-aws
Python: 3.12.13
huggingface_hub: 1.13.0
hf-xet: 1.5.0rc0
requests: 2.34.0.dev1
urllib3: 2.6.3
CLI: hf 1.13.0
```

No relevant `HF_*`, `HUGGINGFACE_*`, `HTTP_PROXY`, or `HTTPS_PROXY` environment variables were set during the test.

Disk space was not the issue:

```text
Filesystem: /dev/root
Size: 1.5T
Available: ~1.5T
Use: 3%
Inodes used: 1%
```

## Additional Context

The CLI warns that the request is unauthenticated:

```text
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
```

UPDATE: logging in did not fix it, this was not the issue.

However, the observed behavior looks like a Xet/CAS transfer stall rather than ordinary rate limiting:

- Metadata/dry-run requests succeed quickly.
- Downloads proceed partially before stalling.
- The Xet backend logs repeated `403 Forbidden` responses from signed range URLs.
- Disabling Xet makes the same repository downloadable.

This may belong in `huggingface_hub` or `xet-core` rather than `transformers`, but it affects model loading/downloading workflows that Transformers users will hit by default.

### Who can help?

@ArthurZucker @SunMarc

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```bash
uv run hf download Qwen/Qwen3.6-35B-A3B
```

on a p5en.48xlarge instance in us-east-2 on AWS.

### Expected behavior

Xet download works in us-east-2.

--- Comment by Rocketknight1 at 2026-05-06T10:57:02Z ---
Is anyone else encountering this..? Xet downloads are working fine for me

--- Comment by michaelroyzen at 2026-05-06T18:04:50Z ---
Hi @Rocketknight1 this seems to be an AWS-specific issue and it's possibly related to how Cloudfront is configured for HF Hub downloads with Xet on AWS. Can you spin up an EC2 instance in us-east-2 and try from there to reproduce?

I tried again today on multiple distinct p5en.48xlarge in us-east-2 and each time it makes it part way through the download until it tries fetching the remaining files and hangs on infinite 403s. It is a real issue, it seems. I'm not familiar with the internals of the HF Hub CDN but it seems that it has a Cloudfront CDN for downloads to EC2 instances on AWS and that seems to be having problems here.

--- Comment by ThomasTosik at 2026-05-07T13:31:14Z ---
We got the same issue on eu-central-1

--- Comment by Rocketknight1 at 2026-05-07T15:29:12Z ---
Hi @michaelroyzen @ThomasTosik this should now be fixed! The error was indeed real but was internal to the HF Xet systems. If you retry now it should work.

Closing the issue since I think it should be fixed, but if you still encounter the same error, please ping me and I'll reopen


grootN_training

ISSUE #45796: grootN_training
State: closed  |  Labels:
Author: zeic1  |  Created: 2026-05-05T21:46:46Z

VLA model trainng for academic purposes. Requires Flash Attention 2.


ISSUE #45779: [Windows] RTX 5070 Ti (Blackwell sm_120) - setup and deployment notes
State: open  |  Labels:
Author: loongmiaow-pixel  |  Created: 2026-05-05T03:54:09Z

### Environment
- GPU: NVIDIA GeForce RTX 5070 Ti Laptop GPU (Blackwell, compute capability 12.0)
- Driver: 595.79 (CUDA 13.2)
- OS: Windows 11
- Python: 3.14
- transformers: [latest]

### Problem
transformers on RTX 5070 Ti requires workarounds:

- **TORCH_CUDA_ARCH_LIST=12.0** required for Blackwell
- Model loading may fail or fall back to CPU without explicit arch
- **CUDA_VISIBLE_DEVICES=-1** system env var blocks GPU detection

### Solution
```powershell
$env:TORCH_CUDA_ARCH_LIST = "12.0"
$env:CUDA_VISIBLE_DEVICES = "0"
$env:CUDA_MODULE_LOADING = "LAZY"
```

### Question
Would it be helpful to add RTX 50 series notes to the GPU documentation? I can contribute a PR with Windows troubleshooting.

--- Comment by ArjunSrivastava1 at 2026-05-20T09:42:10Z ---
hmm, a compatibility issue? dunno whom to get for this one

@stevhliu , looks to be more docs related so imma leave it up to u, but feel free to get anyone else too

--- Comment by Rocketknight1 at 2026-05-20T11:12:24Z ---
I don't think this is docs related! It's a surprising issue because I think most hardware that works with PyTorch should work with Transformers, and I don't think we've seen other people saying that 50XX cards don't work. Are you sure this isn't some weird local environment issue?

--- Comment by ArjunSrivastava1 at 2026-05-20T17:32:51Z ---
yeah, well, i didnt wanna sound like an alarmist of sorts so i was hoping it to be docs related and went in that line - see the last para in issue, am looking into the repro of it rn cus this better be a local issue

tho there maybe breakage cus pytorch rolled out there updates just a few days ago? no, if that was the case would have come up earlier, if i get anything from those places then i will put it here


Macro Privitera Pitbull 24-10-2025

ISSUE #45768: Macro Privitera Pitbull 24-10-2025
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:51:36Z

(no description)


Privitera Fabrizio 16-12-1967

ISSUE #45767: Privitera Fabrizio 16-12-1967
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:40:34Z

(no description)


Verduci Caterina 21-07-1964

ISSUE #45766: Verduci Caterina 21-07-1964
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:40:01Z

(no description)


Erika Privitera 14-08 -2003

ISSUE #45765: Erika Privitera 14-08 -2003
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:39:07Z

(no description)


Daniele Privitera 14-05-1998

ISSUE #45764: Daniele Privitera 14-05-1998
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:36:14Z

(no description)


Ivan privitera 02-09-1993

ISSUE #45763: Ivan privitera 02-09-1993
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:35:29Z

(no description)


Mirko Privitera 30-09-1990

ISSUE #45762: Mirko Privitera 30-09-1990
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:33:28Z

(no description)


Veneto

ISSUE #45761: Veneto
State: closed  |  Labels:
Author: mirko772  |  Created: 2026-05-03T19:32:27Z

(no description)


ISSUE #45759: `AutoModelForCausalLM.from_config` does not unwrap `text_config` for composite Qwen 3.5 and 3.6 multimodal configs
State: closed  |  Labels: bug
Author: jamesbraza  |  Created: 2026-05-03T18:54:41Z

### System Info

- `transformers` version: 5.6.2
- Platform: Linux-6.8.0-1043-nvidia-x86_64-with-glibc2.35
- Python version: 3.12.13
- Huggingface_hub version: 1.13.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu129 (CUDA)
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@Cyrilvallez @zucchini-nlp @ArthurZucker

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("Qwen/Qwen3.6-35B-A3B")
# How SkyRL meta-initializes its FSDP workers
# SEE: https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/workers/model_wrapper.py#L137-L139
with torch.device("meta"):
    model = AutoModelForCausalLM.from_config(config)
```

This throws:

```none
Traceback (most recent call last):
  File "repro.py", ...
    model = AutoModelForCausalLM.from_config(config)
  File ".../transformers/models/auto/auto_factory.py", line 241, in from_config
    return model_class._from_config(config, **kwargs)
  File ".../transformers/modeling_utils.py", line 1542, in _from_config
    model = cls(config, **kwargs)
  File ".../transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py", line 1906, in __init__
    self.model = Qwen3_5MoeTextModel(config)
  File ".../transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py", line 1339, in __init__
    self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, config.pad_token_id)
  File ".../transformers/configuration_utils.py", line 425, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'Qwen3_5MoeConfig' object has no attribute 'vocab_size'
```

For reference, `from_pretrained("Qwen/Qwen3.6-35B-A3B")` works fine, only `from_config` is broken.

This affects the Qwen3.5-MoE and Qwen3.6 family of models.

### Expected behavior

`from_pretrained` knows to unwrap (`config = config.get_text_config()`) before dispatching, see [`auto_factory.py:L383-L396`](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/models/auto/auto_factory.py#L383-L396).

However, `from_config` does not have this unwrap, see [`auto_factory.py:L239-L241`](https://github.com/huggingface/transformers/blob/v5.6.2/src/transformers/models/auto/auto_factory.py#L239-L241).

So the request is symmetry inside `auto_factory.py`: the same unwrap that exists for `from_pretrained` should exist for `from_config`.

Related issues:

- Analogous case for Gemma 3: https://github.com/huggingface/transformers/issues/36683
- Related vLLM issue: https://github.com/vllm-project/vllm/issues/36236


ISSUE #45758: DeepSeek-V4 CSA eager path may not preserve per-query top-k masking for S > 1
State: closed  |  Labels:
Author: kekmodel  |  Created: 2026-05-03T18:52:35Z

### System Info

Observed on `main` after DeepSeek-V4 support was added in #45643.

Relevant file:

- `src/transformers/models/deepseek_v4/modeling_deepseek_v4.py`
- `DeepseekV4CSACompressor.forward`
- `DeepseekV4Attention.forward`

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder
- [X] My own task or dataset

### Reproduction

This is a code-reading report rather than a failing runtime script. I may be missing an intended invariant in the eager path, but the current tensor shapes look non-equivalent to per-query CSA sparse attention when `seq_len > 1`.

In `DeepseekV4CSACompressor.forward`, CSA first obtains per-query top-k compressed KV indices:

```python
topk = self.indexer(hidden_states, q_residual, position_ids, past_key_values, layer_idx)  # [B, S, k]
expanded = compressed_kv.unsqueeze(2).expand(-1, -1, seq_len, -1, -1)                    # [B, 1, S, T, D]
idx = topk.unsqueeze(1).unsqueeze(-1).expand(-1, 1, -1, -1, self.head_dim)               # [B, 1, S, k, D]
return torch.gather(expanded, 3, idx).reshape(batch, 1, -1, self.head_dim)               # [B, 1, S*k, D]
```

Conceptually, the gathered tensor before reshape is per-query selected compressed KV:

```text
selected[b, 0, t, :, :] = compressed entries selected for query t
shape: [B, 1, S, k, D]
```

After `.reshape(batch, 1, -1, D)`, the selected entries for all query positions are flattened into one KV axis:

```text
shape: [B, 1, S*k, D]
```

Then `DeepseekV4Attention.forward` concatenates those entries after the sliding-window KV branch:

```python
compressed_kv = self.compressor(...)
kv = torch.cat([kv, compressed_kv], dim=2)

if isinstance(attention_mask, torch.Tensor) and kv.shape[2] > attention_mask.shape[-1]:
    attention_mask = F.pad(attention_mask, (0, kv.shape[2] - attention_mask.shape[-1]), value=0.0)
```

The dense mask is right-padded with `0.0`, which appears to make the entire flattened compressed segment visible to every query. For `S > 1`, that means query `t0` can attend to compressed entries that were selected for query `t1`, `t2`, etc.

A minimal shape example:

```text
B = 1, S = 3, k = 2

logical selected entries:
  q0 -> [A0, A1]
  q1 -> [B0, B1]
  q2 -> [C0, C1]

after flatten:
  [A0, A1, B0, B1, C0, C1]

if the compressed segment is mask-padded with 0.0:
  q0 can attend to A*, B*, C*
  q1 can attend to A*, B*, C*
  q2 can attend to A*, B*, C*
```

That seems different from query-specific sparse attention unless an additional block mask maps each query `t` to only its own flattened segment `[t*k : (t+1)*k]`.

There is a second related question around causal visibility. The paper describes the index score between a query token `t` and a preceding compressed block `s` (`s < floor(t / m)`). In the current eager code, I do not see an explicit query-dependent visible-range mask before:

```python
return index_scores.topk(topk, dim=-1).indices
```

So for multi-token prefill/training-style forwards, it is not obvious where future-containing compressed blocks are excluded from top-k selection.

### Expected behavior

For CSA with `seq_len > 1`, each query position should attend only to the compressed KV entries selected for that same query, plus the intended sliding-window KV entries.

Equivalently, one of these should hold:

1. the selected compressed KV remains logically shaped as `[B, H_kv, S, k, D]` and the attention kernel consumes it per query;
2. the flattened `[B, H_kv, S*k, D]` layout is paired with a query-dependent block mask so query `t` only sees its own segment; or
3. the eager path is documented/guarded as decode-only for CSA (`S == 1`) if that is the intended supported usage.

If the current implementation relies on some invariant that makes this equivalent, could you point me to where that masking/visibility constraint is enforced?

--- Comment by Rocketknight1 at 2026-05-05T14:08:21Z ---
cc @arthurzucker this seems real, although I'm also uncertain.

--- Comment by ArthurZucker at 2026-05-07T12:43:59Z ---
yep having a look as its important!

--- Comment by ArthurZucker at 2026-05-12T09:27:50Z ---
#45892 fixes it


ISSUE #45753: Qwen3_5 goes into infinite loop for a specific image
State: open  |  Labels: bug
Author: MHRDYN7  |  Created: 2026-05-03T13:47:50Z

### System Info

Colab T4 GPU

### Who can help?

I tried to run the colab notebooks for qwen3_5 models which are auto-generated. The inference takes forever. I figured out this issue is due to the specific image that is automatically inserted => "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG". The inference works fine for other images (like the one used for model test).

Can someone from the community please investigate and fix the exact issue. It should not be a very difficult as it's most likely related to how this model handles images of different dimensions.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Go to the auto-generated colab link from the model repo
https://colab.research.google.com/#fileId=https%3A//huggingface.co/Qwen/Qwen3.5-0.8B.ipynb

### Expected behavior

Inference will not loop forever for this specific image.

--- Comment by pdjoshi-30 at 2026-05-03T15:30:14Z ---
Hi, I was able to reproduce the issue on Colab (T4 GPU).

From my investigation, the slowdown is caused by the high-resolution image (`candy.JPG`). Large images generate a significantly higher number of visual tokens, which makes inference very slow and appear as if it’s hanging. When I replaced the image with a smaller one, the pipeline worked correctly and returned results quickly. I also verified that resizing the same image before passing it to the pipeline resolves the issue.

Since this notebook is auto-generated, I wanted to confirm the best way to address this:

* Should this be handled by updating the example by adding resizing before inference
* Or would you prefer adding guidance in the documentation (for example, in `docs/source/en/tasks/image_text_to_text.md`) to recommend resizing high-resolution images?

I’d be happy to open a PR once I have your guidance. Thanks!

--- Comment by MHRDYN7 at 2026-05-03T15:43:32Z ---
That's interesting, really thanks a lot. I think @yonigozlan can decide the right policy for  this kinda situation. But still kinda mysterious as the time is too very long, so proper profiling might help to get some more insights. Also, the thing is I ran the example code for lfm2.5 vl 450B yesterday and it ran totally fine (I guess the same image was also used there), so this again points fingers to qwen3.5 implementation

--- Comment by pdjoshi-30 at 2026-05-04T13:41:17Z ---
That is interesting , I guess the different Image tokenization techniques might be the reason for difference between the two models. Let us hear @yonigozlan  opinion as well

--- Comment by iavinas at 2026-05-05T05:37:58Z ---
I profiled this on a Colab T4 to understand what's going on. Here's what I found:

The `candy.JPG` is 4032×3024 (12MP). With the default processor settings, it produces **11,844 visual tokens**, and a single forward pass over an 11,865-token sequence never completes in any reasonable time on T4.

With `max_pixels=1024*1024` set when loading the processor, the same image drops to **972 visual tokens** and runs fine (~27s per forward pass, output: "The animal on the candy is a turtle.").

Here's the profiling code if anyone wants to reproduce:

```python
import time
import requests
from PIL import Image
from io import BytesIO
from contextlib import contextmanager

@contextmanager
def timer(label):
    start = time.perf_counter()
    yield
    elapsed = time.perf_counter() - start
    print(f"[{elapsed:6.2f}s] {label}")

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"

# inspect the image
with timer("Download image"):
    resp = requests.get(url, stream=True)
    img = Image.open(BytesIO(resp.content))
print(f"  Size: {img.size[0]}×{img.size[1]}  {img.size[0]*img.size[1]/1e6:.1f}MP\n")

# load processor with pixel constraints
from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

with timer("Load processor + model"):
    processor = AutoProcessor.from_pretrained(
        "Qwen/Qwen3.5-0.8B",
        min_pixels=224 * 224,      # comment out to see default behavior
        max_pixels=1024 * 1024,    # comment out to see default behavior
    )
    model = AutoModelForImageTextToText.from_pretrained(
        "Qwen/Qwen3.5-0.8B", dtype=torch.bfloat16
    ).cuda()

ip = processor.image_processor
print(f"  shortest_edge: {ip.size['shortest_edge']}")
print(f"  longest_edge:  {ip.size['longest_edge']}")
print(f"  merge_size:    {ip.merge_size}\n")

messages = [
    {"role": "user", "content": [
        {"type": "image", "url": url},
        {"type": "text", "text": "What animal is on the candy?"},
    ]},
]

# encode
with timer("encode"):
    inputs = processor.apply_chat_template(
        messages, add_generation_prompt=True,
        tokenize=True, return_dict=True, return_tensors="pt",
    ).to(model.device)

grid = inputs.get("image_grid_thw")
if grid is not None:
    t, h, w = grid[0].tolist()
    vt = (t * h * w) // (ip.merge_size ** 2)
    print(f"  image_grid_thw: [{t}, {h}, {w}]")
    print(f"  visual tokens:  {vt}")
    print(f"  total seq len:  {inputs['input_ids'].shape[1]}\n")

# forward + generate
print("--- Forward pass ---")
for i in range(3):
    torch.cuda.synchronize()
    with timer(f"  run {i+1}"):
        with torch.no_grad():
            _ = model(**inputs, num_logits_to_keep=1)

torch.cuda.synchronize()
with timer("generate"):
    gen = model.generate(**inputs, max_new_tokens=20)

out = processor.decode(gen[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
print(f"  Output: {out}")
```

The root cause: `Qwen2VLImageProcessor` defaults to `longest_edge=16777216` (~16MP), so high-res images pass through barely downscaled. VLMs with dynamic resolution generate visual tokens proportionally to image area, and 11k+ tokens kills T4 latency.

Two suggestions for the fix:

1. **In `huggingface.js`**: the auto-generated notebook pulls its snippet from `model-libraries-snippets.ts`. Either swap `candy.JPG` for a smaller image there, or generate the snippet with `min_pixels`/`max_pixels` set on the processor.

2. **In `transformers`**: `docs/source/en/tasks/image_text_to_text.md` has the same issue — it uses `bee.jpg` (5184×3456, 18MP) in 5 places, which would hit the same problem for anyone running that auto-generated notebook. Should be swapped for a smaller image.

Two workarounds for now:

**Option A — create processor first (cleanest):**
```python
from transformers import AutoProcessor, pipeline

processor = AutoProcessor.from_pretrained(
    "Qwen/Qwen3.5-0.8B",
    min_pixels=224 * 224,
    max_pixels=1024 * 1024,
)
pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.5-0.8B", processor=processor)
pipe(text=messages)
```

**Option B — patch after creation (hacky):**
```python
pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.5-0.8B")
pipe.processor.image_processor.size['longest_edge'] = 1024 * 1024
pipe.processor.image_processor.size['shortest_edge'] = 224 * 224
pipe(text=messages)
```

Happy to open a PR for either or both.

--- Comment by iavinas at 2026-05-05T07:04:48Z ---
@MHRDYN7

LFM2 VL has a default cap built in for image tokens:

Lfm2VlImageProcessor (lines 224-225)

min_image_tokens = 64
max_image_tokens = 256    # ← NEVER exceeds 256 visual tokens

--- Comment by jshaofa-ui at 2026-05-12T13:37:42Z ---
## Fix: Qwen3.5 Infinite Loop with High-Resolution Images

### Root Cause (confirmed by @iavinas profiling)
The `candy.JPG` image is 4032×3024 (12MP), producing **11,844 visual tokens**. With default processor settings, this creates an 11,865-token sequence that never completes inference on T4 GPU.

**Comparison**: LFM2 VL has `max_image_tokens = 256` built into its image processor, which caps visual tokens and prevents this issue.

### Fix: Add `max_pixels` default to Qwen3.5 image processor

**File**: `src/transformers/models/qwen3_5/processing_qwen3_5.py` (or the relevant processor file)

Add default pixel constraints to the image processor configuration:

```python
# In the image processor config or processing_qwen3_5.py
image_processor = Qwen2VLImageProcessor(
    ...
    max_pixels=1024 * 1024,  # Cap at ~1M pixels (~972 visual tokens)
    min_pixels=224 * 224,
)
```

### Alternative Fix: Update the auto-generated Colab notebook

Since the notebook is auto-generated, add a resize step before inference:

```python
from transformers import AutoProcessor

# Add max_pixels constraint when loading processor
processor = AutoProcessor.from_pretrained(
    "Qwen/Qwen3.5-0.8B",
    min_pixels=224 * 224,
    max_pixels=1024 * 1024,  # ← Add this
)
```

### Recommended Approach

1. **Short-term**: Update the auto-generated notebook template to include `max_pixels` constraint
2. **Long-term**: Add default `max_pixels` to the Qwen3.5 model config, similar to LFM2 VLs

--- Comment by ArjunSrivastava1 at 2026-05-13T15:22:02Z ---
@zucchini-nlp thoughts? looks to be more a documentation issue but thats just my opinion

--- Comment by zucchini-nlp at 2026-05-18T01:27:58Z ---
Replied under the PR, imo this doesn't really need fixing as qwen models are known to produce a long seq of image tokens. ARAIR the model doc on qwen mentions how to regulate min/max pixels

--- Comment by ArjunSrivastava1 at 2026-05-19T05:35:31Z ---
@zucchini-nlp the boss has spoken, its fine as is then, checked out ur reply under the pr - yeah that tracks,
thx for replying when tagged


ISSUE #45750: `Qwen3VLVisionPatchEmbed.proj` (`nn.Conv3d` with `stride == kernel`) is ~50,000× slower than equivalent `nn.Linear` on Blackwell + bf16
State: open  |  Labels: bug
Author: WangYuHang-cmd  |  Created: 2026-05-03T07:27:10Z

### System Info

```
transformers version: 5.0.0.dev0
PyTorch:              2.9.0+cu128
CUDA:                 12.8
cuDNN:                9.10.0.2 (91002)
Python:               3.14.0
flash-attn:           2.8.3 (installed)
GPU:                  NVIDIA GeForce RTX 5090 (Blackwell, compute capability 12.0, sm_120)
OS:                   Linux 6.8.0-110-generic, glibc 2.39
```

### Who can help?

@yonigozlan  @molbap  @zucchini-nlp

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

##

1. **Confirm GPU is healthy.** RTX 5090 should hit ~100–209 TFLOPS bf16 dense matmul.

   ```python
   import torch, time
   for size in [4096, 8192]:
       a = torch.randn(size, size, dtype=torch.bfloat16, device="cuda")
       b = torch.randn(size, size, dtype=torch.bfloat16, device="cuda")
       for _ in range(3): _ = a @ b
       torch.cuda.synchronize(); t0 = time.time()
       for _ in range(10): c = a @ b
       torch.cuda.synchronize()
       e = time.time() - t0
       print(f"matmul {size}x{size}: {2 * size**3 * 10 / e / 1e12:.1f} TFLOPS")
   ```

   Output:

   ```
   matmul 4096x4096: 182.3 TFLOPS
   matmul 8192x8192: 223.7 TFLOPS
   ```

   Hardware is fine.

2. **Run a full vision-tower forward at multiple batch sizes**, all with default settings.

   ```python
   import time, torch
   from PIL import Image
   from transformers import AutoModelForImageTextToText, AutoProcessor

   torch.set_grad_enabled(False)
   proc = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-4B-Instruct")
   model = AutoModelForImageTextToText.from_pretrained(
       "Qwen/Qwen3-VL-4B-Instruct", dtype=torch.bfloat16,
   ).cuda().eval()

   def make_clip():
       return [Image.fromarray(torch.randint(0, 256, (720, 1280, 3),
               dtype=torch.uint8).numpy()) for _ in range(8)]

   def time_forward(bs):
       texts, images = [], []
       for _ in range(bs):
           frames = make_clip()
           msgs = [{"role": "user", "content":
                    [{"type": "image", "image": img} for img in frames]
                    + [{"type": "text", "text": "Describe."}]}]
           texts.append(proc.apply_chat_template(
               msgs, tokenize=False, add_generation_prompt=True))
           images.append(frames)
       inputs = proc(text=texts, images=images,
                     return_tensors="pt", padding=True)
       inputs = {k: (v.cuda() if isinstance(v, torch.Tensor) else v)
                 for k, v in inputs.items()}
       inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16)
       keys = ("input_ids","attention_mask","pixel_values","image_grid_thw")
       args = {k: inputs[k] for k in keys if k in inputs}
       for rep in range(2):
           torch.cuda.synchronize(); t0 = time.time()
           with torch.amp.autocast("cuda", dtype=torch.bfloat16):
               _ = model.model(**args, use_cache=False, return_dict=True)
           torch.cuda.synchronize()
           e = time.time() - t0
           print(f"  bs={bs} rep={rep}: {e:.2f}s ({e/bs*1000:.0f} ms/sample)")

   for bs in [1, 4, 8, 16]: time_forward(bs)
   ```

   Output:
1.

   ```
   bs=1  rep=0: 16.70s (16700 ms/sample)
   bs=1  rep=1: 16.46s (16458 ms/sample)
   bs=4  rep=0: 65.30s (16325 ms/sample)
   bs=8  rep=0: 148.05s (18506 ms/sample)
   bs=8  rep=1: 148.01s (18501 ms/sample)
   bs=16 rep=0: 148.78s (9299 ms/sample)
   bs=16 rep=1: 148.34s (9271 ms/sample)
   ```

   Per-sample time is ~16 s regardless of batch — rules out DataLoader,
   collate, padding bugs.

3. **Eliminate `attn_implementation` as the cause.** Test all three.

   ```python
   for impl in ["sdpa", "flash_attention_2", "eager"]:
       model = AutoModelForImageTextToText.from_pretrained(
           "Qwen/Qwen3-VL-4B-Instruct", dtype=torch.bfloat16,
           attn_implementation=impl).cuda().eval()
       # ... same time_forward(bs=8) ...
   ```

   Output:

   ```
   sdpa              bs=8: 148.05s (18506 ms/sample)
   flash_attention_2 bs=8: 147.64s (18455 ms/sample)
   eager             bs=8: 148.20s (18525 ms/sample)
   ```

   All three implementations are identically slow → attention is not the cause.

4. **Per-component timing of `Qwen3VLVisionModel.forward`** (bs=1, 8 frames).

   ```python
   import torch.nn.functional as F
   pv = inputs["pixel_values"]; grid_thw = inputs["image_grid_thw"]
   vt = model.visual

   torch.cuda.synchronize(); t0 = time.time()
   h = vt.patch_embed(pv); torch.cuda.synchronize()
   print(f"patch_embed:  {(time.time()-t0)*1000:.1f} ms, shape={tuple(h.shape)}")

   t0 = time.time(); pos = vt.fast_pos_embed_interpolate(grid_thw)
   torch.cuda.synchronize(); print(f"pos_embed:    {(time.time()-t0)*1000:.1f} ms")
   h = h + pos

   t0 = time.time(); rope = vt.rot_pos_emb(grid_thw)
   torch.cuda.synchronize(); print(f"rot_pos_emb:  {(time.time()-t0)*1000:.1f} ms")

   seq_len = h.size(0); h = h.reshape(seq_len, -1)
   rope = rope.reshape(seq_len, -1)
   emb = torch.cat((rope, rope), dim=-1)
   pos_emb = (emb.cos(), emb.sin())
   cu = torch.repeat_interleave(grid_thw[:,1]*grid_thw[:,2],
        grid_thw[:,0]).cumsum(0, dtype=torch.int32)
   cu = F.pad(cu, (1,0), value=0)

   times = []
   for i, blk in enumerate(vt.blocks):
       torch.cuda.synchronize(); t0 = time.time()
       h = blk(h, cu_seqlens=cu, position_embeddings=pos_emb)
       torch.cuda.synchronize()
       times.append((time.time()-t0)*1000)
   print(f"24 blocks total: {sum(times):.1f} ms (mean {sum(times)/24:.1f} ms)")

   t0 = time.time(); _ = vt.merger(h)
   torch.cuda.synchronize(); print(f"merger:       {(time.time()-t0)*1000:.1f} ms")
   ```

   Output:

   ```
   patch_embed:  16111.3 ms, shape=(6080, 1024)   ← 96% of total forward
   pos_embed:    22.8 ms
   rot_pos_emb:  20.7 ms
   24 blocks total: 56.4 ms (mean 2.3 ms)
   merger:       0.5 ms
   ```

   The 24-layer ViT runs in 56 ms total. The single `patch_embed` takes
   16,111 ms — 287× more than the rest combined.

5. **Inspect `Qwen3VLVisionPatchEmbed`** (file:
   `transformers/models/qwen3_vl/modeling_qwen3_vl.py`, lines 59–76).

   ```python
   class Qwen3VLVisionPatchEmbed(nn.Module):
       def __init__(self, config) -> None:
           super().__init__()
           self.patch_size = config.patch_size                  # 16
           self.temporal_patch_size = config.temporal_patch_size  # 2
           self.in_channels = config.in_channels                 # 3
           self.embed_dim = config.hidden_size                   # 1024
           kernel_size = [self.temporal_patch_size, self.patch_size, self.patch_size]
           self.proj = nn.Conv3d(
               self.in_channels, self.embed_dim,
               kernel_size=kernel_size, stride=kernel_size, bias=True,
           )

       def forward(self, hidden_states):
           target_dtype = self.proj.weight.dtype
           hidden_states = hidden_states.view(
               -1, self.in_channels, self.temporal_patch_size,
               self.patch_size, self.patch_size,
           )
           hidden_states = self.proj(
               hidden_states.to(dtype=target_dtype)
           ).view(-1, self.embed_dim)
           return hidden_states
   ```

   `kernel_size == stride`, no padding, no dilation → output windows are
   disjoint → mathematically equivalent to `flatten + nn.Linear`.

6. **Isolated benchmark: Conv3d vs equivalent Linear** (no checkpoint needed).

   ```python
   import time, torch, torch.nn as nn
   torch.set_grad_enabled(False)

   conv = nn.Conv3d(3, 1024, kernel_size=(2,16,16),
                    stride=(2,16,16), bias=True).cuda().to(torch.bfloat16)

   out_dim, in_dim = 1024, 3*2*16*16  # 1536
   lin = nn.Linear(in_dim, out_dim, bias=True)
   lin.weight.data.copy_(conv.weight.detach().reshape(out_dim, in_dim))
   lin.bias.data.copy_(conv.bias.detach())
   lin = lin.cuda().to(torch.bfloat16)

   N = 6080  # patches in one 8-frame Qwen3-VL clip
   x_5d  = torch.randn(N, 3, 2, 16, 16, dtype=torch.bfloat16, device="cuda")
   x_flat = x_5d.reshape(N, -1).contiguous()

   for _ in range(3): _ = conv(x_5d); _ = lin(x_flat)

   torch.cuda.synchronize(); t0 = time.time()
   for _ in range(5): y_conv = conv(x_5d).view(N, -1)
   torch.cuda.synchronize(); t_conv = (time.time()-t0)/5

   torch.cuda.synchronize(); t0 = time.time()
   for _ in range(5): y_lin = lin(x_flat)
   torch.cuda.synchronize(); t_lin = (time.time()-t0)/5

   print(f"Conv3d:  {t_conv*1000:8.2f} ms")
   print(f"Linear:  {t_lin*1000:8.2f} ms")
   print(f"Speedup: {t_conv/t_lin:8.1f}x")
   diff = (y_conv.float() - y_lin.float()).abs().max().item()
   cos = torch.nn.functional.cosine_similarity(
       y_conv.float().flatten().unsqueeze(0),
       y_lin.float().flatten().unsqueeze(0)).item()
   print(f"max abs diff (bf16): {diff:.2e}")
   print(f"cosine similarity:   {cos:.6f}")
   ```

   Output:

   ```
   Conv3d:    16111.30 ms
   Linear:        0.30 ms
   Speedup:    53704.3x
   max abs diff (bf16): 1.56e-02
   cosine similarity:   0.999500
   ```

7. **Verify mathematical equivalence in fp32** (rules out numerical accident).

```python
import torch, torch.nn as nn
torch.manual_seed(0); N = 100; C, T, P = 3, 2, 16; out_dim = 1024
conv = nn.Conv3d(C, out_dim, (T,P,P), stride=(T,P,P), bias=True)
in_dim = C*T*P*P
lin = nn.Linear(in_dim, out_dim, bias=True)
lin.weight.data.copy_(conv.weight.detach().reshape(out_dim, in_dim))
lin.bias.data.copy_(conv.bias.detach())

x_5d = torch.randn(N, C, T, P, P, dtype=torch.float32)
x_flat = x_5d.reshape(N, -1).contiguous()
with torch.no_grad():
    o_conv = conv(x_5d).view(N, -1)
    o_lin  = lin(x_flat)

abs_diff = (o_conv - o_lin).abs()
print(f"fp32 max abs diff:  {abs_diff.max().item():.2e}")
print(f"fp32 mean abs diff: {abs_diff.mean().item():.2e}")
```

Output:

```
fp32 max abs diff:  4.77e-07
fp32 mean abs diff: 7.61e-08
```

`Conv3d` with `kernel == stride` is **exactly** equivalent to `Linear`
over reshaped weights — fp32 difference is single-multiplication
round-off (~5e-7).

8. **Apply the fix** (lazy in-place `Conv3d → Linear` via monkey-patch on
`Qwen3VLVisionPatchEmbed.forward`) and re-benchmark.

```python
import time, torch, torch.nn as nn
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
from transformers.models.qwen3_vl.modeling_qwen3_vl import (
    Qwen3VLVisionPatchEmbed,
)

def _fast_forward(self, hidden_states):
    target_dtype = self.proj.weight.dtype
    if isinstance(self.proj, nn.Conv3d):
        conv = self.proj
        out_dim = conv.out_channels
        in_dim = (conv.in_channels * conv.kernel_size[0]
                  * conv.kernel_size[1] * conv.kernel_size[2])
        w_flat = conv.weight.detach().reshape(out_dim, in_dim).contiguous()
        bias = conv.bias.detach().clone() if conv.bias is not None else None
        new_proj = nn.Linear(in_dim, out_dim, bias=bias is not None)
        new_proj.weight.data.copy_(w_flat)
        if bias is not None: new_proj.bias.data.copy_(bias)
        new_proj.to(device=conv.weight.device, dtype=conv.weight.dtype)
        self.proj = new_proj
    if hidden_states.dim() > 2 \
            or hidden_states.shape[-1] != self.proj.in_features:
        hidden_states = hidden_states.reshape(-1, self.proj.in_features)
    return self.proj(hidden_states.to(dtype=target_dtype))

Qwen3VLVisionPatchEmbed.forward = _fast_forward

# Reload model and run step-2-style timing again.
```

Output:

```
bs=1  rep=0: 0.27s (270 ms/sample)
bs=1  rep=1: 0.29s (290 ms/sample)
bs=8  rep=0: 2.16s (270 ms/sample)
bs=8  rep=1: 2.18s (273 ms/sample)
```

Speedup vs step 2: **62× at bs=1, 68× at bs=8.** VRAM unchanged.
Patch embedding goes from 96% of total forward time to <1%.

### Expected behavior

<!DOCTYPE html><p cid="n71" mdtype="paragraph" class="md-end-block md-p md-focus" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Qwen3VLVisionPatchEmbed.forward should run in ~0.3 ms (the time of the
equivalent nn.Linear), not ~16 s.
<h3 cid="n72" mdtype="heading" class="md-end-block md-heading" style="box-sizing: border-box; white-space: pre-wrap; break-after: avoid-page; break-inside: avoid; orphans: 4; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; font-weight: bold; line-height: 1.43; cursor: text; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Proposed fix
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="diff" cid="n73" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: pre; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"> class Qwen3VLVisionPatchEmbed(nn.Module):
     def __init__(self, config) -> None:
         super().__init__()
         self.patch_size = config.patch_size
         self.temporal_patch_size = config.temporal_patch_size
         self.in_channels = config.in_channels
         self.embed_dim = config.hidden_size

-        kernel_size = [self.temporal_patch_size, self.patch_size, self.patch_size]
-        self.proj = nn.Conv3d(
-            self.in_channels, self.embed_dim,
-            kernel_size=kernel_size, stride=kernel_size, bias=True,
-        )
+        in_dim = (self.in_channels * self.temporal_patch_size
+                  * self.patch_size * self.patch_size)
+        self.proj = nn.Linear(in_dim, self.embed_dim, bias=True)

     def forward(self, hidden_states):
         target_dtype = self.proj.weight.dtype
-        hidden_states = hidden_states.view(
-            -1, self.in_channels, self.temporal_patch_size,
-            self.patch_size, self.patch_size,
-        )
-        hidden_states = self.proj(hidden_states.to(dtype=target_dtype)).view(-1, self.embed_dim)
+        hidden_states = hidden_states.reshape(-1, self.proj.in_features)
+        hidden_states = self.proj(hidden_states.to(dtype=target_dtype))
         return hidden_states
<h3 cid="n74" mdtype="heading" class="md-end-block md-heading" style="box-sizing: border-box; white-space: pre-wrap; break-after: avoid-page; break-inside: avoid; orphans: 4; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; font-weight: bold; line-height: 1.43; cursor: text; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Backward compatibility for existing checkpoints
<p cid="n75" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Pretrained Qwen3-VL-*-Instruct checkpoints save proj.weight in
5-D Conv3d shape (out, in, k_t, k_h, k_w). To load them into the new
Linear layer ((out, in·k_t·k_h·k_w)), add a _load_from_state_dict
hook:
<pre class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" lang="python" cid="n76" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: pre; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit; color: rgb(51, 51, 51); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">def _load_from_state_dict(self, state_dict, prefix, *args, **kwargs):
    key = prefix + "proj.weight"
    if key in state_dict and state_dict[key].dim() == 5:
        out_dim = state_dict[key].shape[0]
        state_dict[key] = state_dict[key].reshape(out_dim, -1).contiguous()
    super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
<p cid="n77" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">This makes the change transparent to existing public checkpoints.
<h3 cid="n78" mdtype="heading" class="md-end-block md-heading" style="box-sizing: border-box; white-space: pre-wrap; break-after: avoid-page; break-inside: avoid; orphans: 4; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; font-weight: bold; line-height: 1.43; cursor: text; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Numerical equivalence verified
<figure class="md-table-fig table-figure" cid="n79" mdtype="table" style="box-sizing: border-box; margin: 1.2em 0px; overflow-x: auto; max-width: calc(100% + 16px); padding: 0px; cursor: default; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">
check | tolerance | observed
-- | -- | --
fp32 max abs diff (proj output) | < 1e-5 | < 1e-7
bf16 cosine similarity (proj output) | > 0.999 | 0.9995
bf16 cosine similarity (full 24-layer vision tower) | > 0.99 | > 0.999 per sample

<h3 cid="n96" mdtype="heading" class="md-end-block md-heading" style="box-sizing: border-box; white-space: pre-wrap; break-after: avoid-page; break-inside: avoid; orphans: 4; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; font-weight: bold; line-height: 1.43; cursor: text; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Same fix applies to Qwen2-VL and Qwen2.5-VL
<p cid="n97" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">The same Conv3d-with-kernel_size == stride pattern exists in
Qwen2VLVisionPatchEmbed and Qwen2_5_VLVisionPatchEmbed. Both should
be patched identically.
<h3 cid="n98" mdtype="heading" class="md-end-block md-heading" style="box-sizing: border-box; white-space: pre-wrap; break-after: avoid-page; break-inside: avoid; orphans: 4; font-size: 1.5em; margin-top: 1rem; margin-bottom: 1rem; position: relative; font-weight: bold; line-height: 1.43; cursor: text; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Why this matters
<p cid="n99" mdtype="paragraph" class="md-end-block md-p" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Anyone running Qwen-VL inference on a Blackwell GPU in bf16 silently
pays a ~50,000× cost on the patch projection. For 30,000-sample feature
extraction, this is the difference between 6 days and ~2 hours.
<p cid="n100" mdtype="paragraph" class="md-end-block md-p md-focus" style="box-sizing: border-box; line-height: inherit; orphans: 4; margin: 0.8em 0px; white-space: pre-wrap; position: relative; color: rgb(51, 51, 51); font-family: "Open Sans", "Clear Sans", "Helvetica Neue", Helvetica, Arial, "Segoe UI Emoji", "SF Pro", sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Happy to send a PR with the rewrite, the backward-compat
_load_from_state_dict hook, a unit test, and a benchmark script.

--- Comment by vasqu at 2026-05-04T07:33:30Z ---
Hey, we already enabled this over here #45041

You do need to pass this explicitly tho. While it does not change the output too much on small inputs, it can change the behavior slightly just fyi. Conversions like these are never 1:1 matching with the original version

--- Comment by vadimkantorov at 2026-05-13T16:26:59Z ---
Also, maybe should be reported in core PyTorch for fix upstream?

--- Comment by vasqu at 2026-05-13T16:28:49Z ---
Oh yea, that's a good point! Could you submit one and reference this? Can also cc me if you want to 🤗

--- Comment by WangYuHang-cmd at 2026-05-13T16:47:11Z ---
> Oh yea, that's a good point! Could you submit one and reference this? Can also cc me if you want to 🤗

Good point — I agree this is probably worth reporting upstream to core PyTorch as well.

I’d be happy to help with the upstream report, either by contributing the isolated Conv3d vs Linear benchmark / Blackwell + bf16 reproduction details, or by co-writing it if that’s useful. @vadimkantorov  I can also help make sure this Transformers issue is referenced and cc you there.


ISSUE #45736: Please update `tokenizers` version check
State: open  |  Labels: Feature request
Author: lalala-233  |  Created: 2026-05-01T12:15:45Z

### Feature request

Several days before, `tokenizers` released a new version of `0.23.1`, but `transformers` doesn't allow us to use it.

https://github.com/huggingface/transformers/blob/ecc3d0da0f8e9c2c54676345b816db29f842792a/setup.py#L150
https://github.com/huggingface/transformers/blob/ecc3d0da0f8e9c2c54676345b816db29f842792a/src/transformers/dependency_versions_table.py#L77

And if you import `transformers`, you will get an error.

```python
ImportError: tokenizers>=0.22.0,<=0.23.0 is required for a normal functioning of this module, but found tokenizers==0.23.1.
```

`0.23.1` includes some break changes, if you have any plan to remove `<=0.23.0` check?

### Motivation

In Arch Linux, we use the latest version of `tokenizers`, making it impossible to use current version of `transformers`.

And `0.23.1` have improved performance that will improve `transformers`' performance

### Your contribution

I have made a patch to remove this check, but running `make fix-repo` as it suggested generates a lot of changes. And I missed some of check because of the lack of some python package.

Or, I just need to edit `setup.py` and `src/transformers/dependency_versions_table.py`?

Btw, I have not tested if `0.23.1` breaks something down yet.

--- Comment by Rocketknight1 at 2026-05-01T12:55:36Z ---
We'll probably bump that version with the next version of `transformers`! cc @itazap

--- Comment by lalala-233 at 2026-05-02T06:40:40Z ---
`0.23.1` includes some breaking changes, e.g., `cls` was renamed to `cls_token` in `BertProcessing` and `RobertaProcessing`.

These following line have been affected

https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/models/herbert/tokenization_herbert.py#L107

https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/models/roberta/tokenization_roberta.py#L172

https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/models/layoutlmv3/tokenization_layoutlmv3.py#L230

https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/models/clip/tokenization_clip.py#L119

https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/convert_slow_tokenizer.py#L485

https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/convert_slow_tokenizer.py#L556

https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/convert_slow_tokenizer.py#L1458


ISSUE #45735: Bug with detecting cache positions in sdpa_mask
State: closed  |  Labels: bug
Author: davidmezzetti  |  Created: 2026-05-01T10:42:56Z

### System Info

Transformers v5.7.0

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

It looks like https://github.com/huggingface/transformers/pull/44181 introduced the following issue.

```
File transformers/models/modernbert/modeling_modernbert.py:472, in ModernBertModel.forward(self, input_ids, attention_mask, position_ids, inputs_embeds, **kwargs)
    465 if not isinstance(attention_mask_mapping := attention_mask, dict):
    466     mask_kwargs = {
    467         "config": self.config,
    468         "inputs_embeds": hidden_states,
    469         "attention_mask": attention_mask,
    470     }
    471     attention_mask_mapping = {
--> 472         "full_attention": create_bidirectional_mask(**mask_kwargs),
    473         "sliding_attention": create_bidirectional_sliding_window_mask(**mask_kwargs),
    474     }
    476 position_embeddings = {}
    477 for layer_type in set(self.config.layer_types):

File transformers/utils/deprecation.py:171, in deprecate_kwarg.<locals>.wrapper.<locals>.wrapped_func(*args, **kwargs)
    167 elif minimum_action in (Action.NOTIFY, Action.NOTIFY_ALWAYS) and not is_torchdynamo_compiling():
    168     # DeprecationWarning is ignored by default, so we use FutureWarning instead
    169     warnings.warn(message, FutureWarning, stacklevel=2)
--> 171 return func(*args, **kwargs)

File transformers/masking_utils.py:1071, in create_bidirectional_mask(config, inputs_embeds, attention_mask, encoder_hidden_states, past_key_values, or_mask_function, and_mask_function)
   1068     use_vmap = True
   1070 # We now create the mask
-> 1071 attention_mask = mask_interface(
   1072     batch_size=batch_size,
   1073     q_length=q_length,
   1074     kv_length=kv_length,
   1075     q_offset=q_offset,
   1076     kv_offset=kv_offset,
   1077     mask_function=mask_factory_function,
   1078     attention_mask=attention_mask,
   1079     # Additional kwargs for sdpa
   1080     allow_is_causal_skip=False,
   1081     allow_is_bidirectional_skip=allow_is_bidirectional_skip,
   1082     dtype=dtype,  # Additional kwarg for eager
   1083     config=config,  # Pass the config as well, in case someone wants to easily have their own mask_interface
   1084     use_vmap=use_vmap,  # Short-circuit to non-vmap expansions for the mask
   1085     device=device,
   1086 )
   1087 return attention_mask

File transformers/masking_utils.py:492, in sdpa_mask(batch_size, q_length, kv_length, q_offset, kv_offset, mask_function, attention_mask, local_size, allow_is_causal_skip, allow_is_bidirectional_skip, allow_torch_fix, use_vmap, device, **kwargs)
    487 if isinstance(q_length, torch.Tensor):
    488     logger.warning_once(
    489         "`cache_position` is deprecated as an arg, and will be removed in Transformers v5.6. Please use `q_length` and "
    490         "`q_offset` instead, similarly to `kv_length` and `kv_offset`"
    491     )
--> 492     q_length, q_offset = q_length.shape[0], q_length[0].to(device)
    494 # Potentially pad the 2D mask
    495 padding_mask = prepare_padding_mask(attention_mask, kv_length, kv_offset)

IndexError: tuple index out of range
```

The following code reproduces this.

```python
import torch
from transformers import AutoModel, AutoTokenizer

# Note that using bert-base-uncased creates another separate ERROR
path = "answerdotai/ModernBERT-base"

model = AutoModel.from_pretrained(path)
tokenizer = AutoTokenizer.from_pretrained(path)

# Generate dummy inputs
dummy = dict(tokenizer(["test inputs"], return_tensors="pt"))

torch.onnx.export(
    model,
    (dummy,),
    dynamo=False
)
```

The following patch seems to fix the issue.

```diff
diff --git a/transformers/masking_utils.py b/tmp/patch.py
index 45e43fd..3d9f496 100644
--- a/transformers/masking_utils.py
+++ b/tmp/patch.py
@@ -484,7 +484,7 @@ def sdpa_mask(

     """
     # For BC on `cache_positions` that used to be an arg at the position of `q_length`
-    if isinstance(q_length, torch.Tensor):
+    if isinstance(q_length, torch.Tensor) and q_length.ndim > 0:
         logger.warning_once(
             "`cache_position` is deprecated as an arg, and will be removed in Transformers v5.6. Please use `q_length` and "
             "`q_offset` instead, similarly to `kv_length` and `kv_offset`"
@@ -689,7 +689,7 @@ def flex_attention_mask(
             An optional device to create the mask on.
     """
     # For BC on `cache_positions` that used to be an arg at the position of `q_length`
-    if isinstance(q_length, torch.Tensor):
+    if isinstance(q_length, torch.Tensor) and q_length.ndim > 0:
         logger.warning_once(
             "`cache_position` is deprecated as an arg, and will be removed in Transformers v5.6. Please use `q_length` and "
             "`q_offset` instead, similarly to `kv_length` and `kv_offset`"
```

### Expected behavior

The export works.

--- Comment by Rocketknight1 at 2026-05-01T13:07:01Z ---
cc @cyrilvallez for #44181!


ISSUE #45727: fix(generation): correct spelling mistake in continuous_api docstring
State: closed  |  Labels:
Author: lohdptm  |  Created: 2026-04-30T16:03:07Z

Hi team,

While reviewing the continuous batching generation logic for a local deployment, I noticed a minor spelling mistake (`usefull` -> `useful`) in the docstring of `continuous_api.py`.

It's a tiny detail, but keeping the core API docs pristine is always good. Here is the patch:

```diff
--- a/src/transformers/generation/continuous_batching/continuous_api.py
+++ b/src/transformers/generation/continuous_batching/continuous_api.py
@@ -102,7 +102,7 @@ class ContinuousGenerationApi:
         self.tokenizer = tokenizer

     def add_request(self, request: Dict[str, Any]) -> int:
-        """Adds a request to the queue. Returns the request ID. This is usefull if you want to track the request later."""
+        """Adds a request to the queue. Returns the request ID. This is useful if you want to track the request later."""
         return self.queue.add_request(request)
```

Cheers!

--- Comment by nightcityblade at 2026-05-02T03:11:08Z ---
Hi, I'd like to work on this. I'll submit a PR shortly.

--- Comment by Dhruv908615 at 2026-05-03T06:02:43Z ---
Hi, I would like to work on this issue. Can you assign it to me?

--- Comment by Caromisc at 2026-05-07T11:04:43Z ---
Hi, I was looking into this issue and can confirm the fix has already been merged via PR #45749 (commit aaec1092d4 by @Dhruv908615). The typo is no longer present in the current main branch. This issue can be closed.


ISSUE #45721: Pass library_name="transformers" to HfApi/Hub functions in push paths so commits attribute correctly
State: open  |  Labels:
Author: davanstrien  |  Created: 2026-04-30T12:40:51Z

Per the repo's agentic contribution policy, opening this issue first to coordinate before sending the PR.

### Summary

`transformers` downloads correctly report a User-Agent of `transformers/<version>; python/...; session_id/...` (built by `http_user_agent()` in `src/transformers/utils/hub.py`). Pushes do not — every `HfApi(...)` instantiation and every top-level call to `create_repo` / `upload_folder` / `create_commit` in `src/transformers/` is bare, so commits made by `model.push_to_hub(...)`, `Trainer.push_to_hub(...)`, and the model conversion scripts attribute as `unknown/None; hf_hub/X.Y.Z; ...` rather than `transformers/<version>; ...`.

### Proposal

Add `library_name="transformers", library_version=__version__` to the bare `HfApi(...)` instantiations and to top-level `create_repo` / `upload_folder` / `create_commit` calls across `src/transformers/` (~25 call sites: core push helper in `utils/hub.py`, the per-class `push_to_hub` / `save_pretrained` methods, `trainer.py`, and a handful of `convert_*_to_hf.py` scripts).

### Precedent

Same gap in `datasets` was just closed by huggingface/datasets#8161. Motivated by Hub-side UA-attribution work.

### Coordination

Happy to open the PR if the approach sounds good. cc @Wauplin for the convention check.

--- Comment by Wauplin at 2026-04-30T13:02:26Z ---
Yes agree with this approach! I thought it was already the case but if not, let's do it!

--- Comment by Kaz-scr at 2026-04-30T19:24:42Z ---
Hi @davanstrien and @Wauplin! I am a new contributor and I would like to take this on. Ill start adding libbrary_name="transformers" and library_version=__version__ to the HfApi and top level hub calls across the src/transformers/ directory as proposed. I'll open a PR shortly!


ISSUE #45715: PreTrainedTokenizer.convert_ids_to_tokens(skip_special_tokens=True) rebuilds all_special_ids on every iteration of the per-id loop
State: closed  |  Labels:
Author: longlee0622  |  Created: 2026-04-30T09:08:07Z

### System Info

- transformers version: 5.3.0 (also reproduces on `main`)
- Python: 3.12
- OS: Linux
- Affected class: `transformers.tokenization_python.PreTrainedTokenizer` (renamed to `PythonBackend` on `main`) — the slow-tokenizer base class
- **Not** affected: `TokenizersBackend` (the fast subclass) — it already has the fix

### Who can help?

Tokenizers: @ArthurZucker

### Reproduction

In `src/transformers/tokenization_python.py`, the slow tokenizer's `convert_ids_to_tokens` evaluates `self.all_special_ids` *inside* the per-id loop:

```python
# tokenization_python.py (current)
def convert_ids_to_tokens(self, ids, skip_special_tokens=False):
    ...
    tokens = []
    for index in ids:
        index = int(index)
        if skip_special_tokens and index in self.all_special_ids:   # ← per-iter property access
            continue
        tokens.append(...)
    return tokens
```

`all_special_ids` is an `@property` that **rebuilds the list on every access**:

```python
# tokenization_utils_base.py
@property
def all_special_ids(self) -> list[int]:
    return self.convert_tokens_to_ids(self.all_special_tokens)
```

So the loop is effectively `O(N_tokens × cost_to_rebuild_special_ids)` instead of `O(N_tokens)`.

The fast subclass — `TokenizersBackend.convert_ids_to_tokens` in `tokenization_utils_tokenizers.py` — already hoists this out of the loop (and a comment in that file explicitly documents the problem):

```python
# tokenization_utils_tokenizers.py:735
tokens = []
# self.all_special_ids is an @property which may be slow, so only compute it once before the loop
ids_to_skip = set(self.all_special_ids) if skip_special_tokens else set()
for index in ids:
    index = int(index)
    if index in ids_to_skip:
        continue
    ...
```

The slow base class wasn't updated when the fast subclass got this fix.

#### Minimal repro on Kimi-K2.5 (TikTokenTokenizer, slow, loaded via `trust_remote_code=True`)

```python
import time, random
from transformers import AutoTokenizer

# Any local checkout of moonshotai/Kimi-K2-* with tokenization_kimi.py
MODEL = "/path/to/Kimi-K2.5-NVFP4"

tok = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)
print(f"is_fast={tok.is_fast}, len(special_ids)={len(tok.all_special_ids)}")
# is_fast=False, len(special_ids)=11

random.seed(0)
ids = [random.randint(0, 100_000) for _ in range(512)]

t0 = time.time()
for _ in range(100):
    tok.convert_ids_to_tokens(ids, skip_special_tokens=True)
print(f"512 ids skip=True : {(time.time()-t0)/100*1e6:.0f} us/call")

t0 = time.time()
for _ in range(100):
    tok.convert_ids_to_tokens(ids, skip_special_tokens=False)
print(f"512 ids skip=False: {(time.time()-t0)/100*1e6:.0f} us/call")
```

#### Output (transformers 5.3.0, unmodified)

```
512 ids skip=True : 48009 us/call
512 ids skip=False:   105 us/call
```

A single boolean kwarg flips the call cost by **~450×**, even though the tokenizer only has 11 special tokens. The cost scales with `len(ids)` × `cost_to_evaluate_self.all_special_ids`, not with the number of special tokens.

This isn't specific to Kimi — any model that ships a slow custom tokenizer (e.g. via `trust_remote_code=True`) and is used for streaming detokenization (which calls `convert_ids_to_tokens(..., skip_special_tokens=True)` per generated token batch) will hit it. We discovered this when a TensorRT-LLM streaming benchmark on Kimi-K2.5 showed an 80% TPOT regression on the transformers 4.57 → 5.3 upgrade — DeepSeek-V3 family models (which use a fast tokenizer) on the same upgrade were unaffected.

### Expected behavior

`convert_ids_to_tokens(ids, skip_special_tokens=True)` should be ~the same cost as `skip_special_tokens=False`, plus a single set-membership check per id.

### Suggested fix

Mirror the fix already in `TokenizersBackend.convert_ids_to_tokens` — it's the same pattern:

```diff
         tokens = []
+        # self.all_special_ids is an @property which may be slow, so only compute it once before the loop
+        ids_to_skip = set(self.all_special_ids) if skip_special_tokens else set()
         for index in ids:
             index = int(index)
-            if skip_special_tokens and index in self.all_special_ids:
+            if index in ids_to_skip:
                 continue
             tokens.append(
                 self._added_tokens_decoder[index].content
                 if index in self._added_tokens_decoder
                 else self._convert_id_to_token(index)
             )
         return tokens
```

#### After this fix, on the same Kimi-K2.5 repro

```
512 ids skip=True :   227 us/call   (was 48009)  →  ~211× faster
512 ids skip=False:   129 us/call   (unchanged)
1   id  skip=True :    95 us/call   (unchanged — see below)
```

The single-id case is *not* improved by this change because computing `set(self.all_special_ids)` itself still pays one property access per call. A follow-up that caches the property output on the tokenizer instance (with appropriate invalidation on `add_special_tokens` / `add_tokens`) would address that, but it has a wider blast radius and seemed worth keeping out of this minimal fix. Happy to file a separate issue for that if it's of interest.

Thanks!

--- Comment by Rocketknight1 at 2026-04-30T11:13:24Z ---
cc @itazap as well

--- Comment by ArthurZucker at 2026-04-30T15:24:20Z ---
do you want to open a pr for the fix? 🤗


ISSUE #45712: Six leftover dummy classes in `dummy_pt_objects.py` fail `check_repo.py` and leak into `dir(transformers)` without torch
State: closed  |  Labels: bug
Author: jw9603  |  Created: 2026-04-30T07:47:24Z

### System Info

transformers 5.7.0.dev0 (main).

`utils/check_repo.py` raises on a fresh install without torch:

```
Exception: The following objects are in the public init, but not in the docs:
 - BeamScorer
 - ConstrainedBeamSearchScorer
 - Constraint
 - ConstraintListState
 - DisjunctiveConstraint
 - PhrasalConstraint
```

All six are leftover entries in `src/transformers/utils/dummy_pt_objects.py`. The real implementations were removed in v5 (#41223) but the dummies stayed. but the dummies stayed. None of the six is referenced anywhere under `src/`, `tests/`, `docs/`, or in `MIGRATION_GUIDE_V5.md`.

This only fires without torch because `src/transformers/__init__.py` registers `dummy_pt_objects`'s contents into `_import_structure` only inside the `except OptionalDependencyNotAvailable:` branch. With torch installed, that branch never runs and the six names never appear in `dir(transformers)`, so the docs check passes. Without torch, they leak into the public namespace and the check sees them.

A side effect of the dummies still being there is that `from transformers import BeamScorer` returns a dummy in a no-torch env and only fails with `requires backend torch` on use. Installing torch wouldn't help either because the real class is gone, so the message is misleading. With torch, the import already raises `ImportError`.

### Who can help?

@Rocketknight1 @ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```bash
python -m venv .venv && source .venv/bin/activate
pip install -e ".[quality]"
python utils/check_repo.py
```

### Expected behavior

`check_repo.py` should pass on a clean install without torch, and `dir(transformers)` shouldn't expose objects whose backing implementations were removed.

--- Comment by Rocketknight1 at 2026-04-30T11:35:58Z ---
@jw9603 this looks legit to me, would you be willing to make a PR to remove the dummies now they don't have an actual class anymore?


ISSUE #45710: Fix wrong repo link in Dinov2ForImageClassification doc example
State: closed  |  Labels:
Author: Milan-Bhimani  |  Created: 2026-04-30T06:10:00Z

The link to the config.json for google/dinov2-base-patch16-224 in the Dinov2ForImageClassification example is returning a 404. It should be updated to the correct path.

--- Comment by pcuenca at 2026-04-30T07:27:26Z ---
Can you please provide a link for the page where this happens? I didn't see a reference to that model in [this example](https://huggingface.co/docs/transformers/model_doc/dinov2#transformers.Dinov2ForImageClassification.forward.example).


ISSUE #45706: Hive Civilization — x402-native services for transformers Agents (notification)
State: closed  |  Labels:
Author: srotzin  |  Created: 2026-04-30T00:41:28Z

Notification post — Hive Civilization runs 52 x402-wired services on Base mainnet (treasury 0x15184bf50b3d3f52b60434f8942b7d52f2eb436e, USDC settlement, x402 spec from coinbase/x402).

Why it might be relevant to transformers/Agents:

- transformers Agents tools can wrap Hive services (evaluator, classifier, summarizer, kycOracle, riskScorer, 47 more) with on-chain USDC settlement.
- Discovery is free (`hive_discover_services` returns the catalog with no payment).
- Per-call USDC on Base — no API keys, no auth tokens, no intermediary.

Open to PR'ing a `transformers-agents-hive` integration following existing Tool conventions, or a HuggingFace community Space, whichever the team prefers. Partner posture.

Two open PRs adopting the same pattern on neighboring SDKs:
- coinbase/agentkit#1157 — HiveActionProvider
- goat-sdk/goat#575 — @goat-sdk/plugin-hive

41 public MCP shims at github.com/srotzin (all v1.0.0). 51 entries on Anthropic MCP Registry. MEV leaderboard: https://hive-a2amev.onrender.com/leaderboard. Spectral receipt sample: rcpt_76fceca973da4ec0.

Steve Rotzin, Hive
github.com/srotzin


ISSUE #45704: T5 silently uses apex.FusedRMSNorm which has a memory leak (NVIDIA/apex#1999)
State: closed  |  Labels: bug
Author: dustnehowl  |  Created: 2026-04-29T18:16:58Z

### System Info

- `transformers` version: 4.57.6
- Platform: Linux-6.8.0-100-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.36.2
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: 0.18.9
- PyTorch version (accelerator?): 2.11.0+cu130 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No(single-GPU minimal repro)
- Using GPU in script?: Yes (single NVIDIA B200, cuda:0)
- GPU type: NVIDIA B200

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When `apex` is importable in the current environment, `transformers` silently uses `apex.normalization.FusedRMSNorm` (or `FusedLayerNorm`) as the layer-norm implementation in T5. There is a known memory leak in `apex.FusedRMSNorm` that I have separately reported here:

> NVIDIA/apex#1999 — `FusedRMSNorm` leaks 2 CUDA tensors per forward call under `torch.no_grad()`

Because T5-XXL has 49 layer-norm calls per `forward(...)`, this means **98 CUDA tensors are leaked per `T5EncoderModel.forward(...)`** in a typical `transformers` environment where apex is installed.

This is hard to discover from the user side because:

1.  The apex usage is conditional and silent — there is no log message, no warning, no flag.
2. The leak is invisible in single-shot inference (it's only ~0.1 GB per call), and only becomes visible after many calls.
3. The user's pipeline code (e.g. `FluxPipeline.encode_prompt(...)`) does not mention apex at all.

In our case, this surfaced as an OOM after ~90 batches of FLUX inference on a single B200 (180GB VRAM):

```python
# This loop OOMs after ~90 iterations with batch_size=4 on a 180GB B200,
# because every encode_prompt call leaks ~0.1 GB.
from diffusers import FluxPipeline
import torch

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16,
).to("cuda:0")

for i in range(1000):
    with torch.no_grad():
        pipe.encode_prompt(
            prompt="test", prompt_2="test", device="cuda:0",
            num_images_per_prompt=1, max_sequence_length=256,
        )
    # gc.collect() and torch.cuda.empty_cache() do NOT free the leaked tensors
```

We confirmed that the leak originates from apex's `FusedRMSNorm` by isolating each pipeline stage:

| Component | CUDA tensors leaked per call |
|---|---|
| `pipe.encode_prompt(...)` (uses T5)  | **+98** |
| `pipe.scheduler.set_timesteps(...)`  | 0 |
| `pipe.vae.decode(...)`               | 0 |
| `pipe.transformer(...)`              | 0 |

We further confirmed it by blocking apex's import before transformers is loaded — see workaround below.

### Workaround
```python
import sys
sys.modules['apex'] = None
sys.modules['apex.normalization'] = None
sys.modules['apex.normalization.fused_layer_norm'] = None

# now import transformers / diffusers / ...
```
This forces T5 to fall back to the native PyTorch implementation of RMSNorm, and the leak disappears completely (verified across thousands of calls).

### Expected behavior

I'd like to suggest one of the following (in order of how invasive they are):

1. **An opt-out mechanism** — for example, an environment variable like `TRANSFORMERS_NO_APEX=1` or `USE_APEX_LAYER_NORM=0`, that forces T5 (and any other model that conditionally imports apex) to use the native PyTorch path even when apex is importable. This seems like the lowest-friction option: it doesn't change any default behavior, doesn't risk regressions for users who rely on apex's speed, and gives users a documented way to avoid the leak without resorting to `sys.modules` hacks.
2. **A warning / log** — at minimum, log once at model load time which layer- norm implementation is being used, so that users in environments with pre-installed apex (e.g. NVIDIA NGC containers, ML clusters) can at least notice it.
3. **A config flag on the model** — e.g. `T5Config(use_apex_layernorm=False)`, for users who construct models programmatically.

I'd be happy to put together a PR for option 1 (env var) if maintainers think this is the right direction. Please let me know.


ISSUE #45701: transformers version changes the tokenization
State: open  |  Labels: bug
Author: PiRom1  |  Created: 2026-04-29T12:19:16Z

### System Info

```
Python: 3.12.3
transformers: 5.7.0
torch       : 2.11.0+cu130
sentencepiece: 0.2.1
```

________________

- **Platform:** Linux-6.17.0-22-generic-x86_64-with-glibc2.39
- **Python version:** 3.12.3
- **Huggingface_hub version:** 0.36.2
- **Safetensors version:** 0.7.0
- **Accelerate version:** 1.12.0
- **Accelerate config:** not found
- **DeepSpeed version:** not installed
- **Tensorflow version (GPU?):** not installed (NA)
- **Flax version (CPU?/GPU?/TPU?):** not installed (NA)
- **Jax version:** not installed
- **JaxLib version:** not installed
- **Using distributed or parallel set-up in script?:** `<fill in>`
- **Using GPU in script?:** `<fill in>`
- **GPU type:** NVIDIA GeForce RTX 5060 Laptop GPU

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Tokenization differs according to the transformers version.

- **In 4.X:** Tokenization is normal
- **In 5.X:** Tokenization is at the character level

Please find below an example toy code I used in 2 different venv. The only difference was the transformers version used (`5.7.0` vs `4.57.6`):

```python
import transformers, torch, sentencepiece
print("transformers:", transformers.__version__)
print("torch       :", torch.__version__)
print("sentencepiece:", sentencepiece.__version__)

MODEL_DIR = "almanach/camembertv2-base"
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained(MODEL_DIR)
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)

ids = tokenizer("This is a text example, You could have written anything else if necessary !", return_tensors="pt")["input_ids"]
print("Token IDs:", ids.tolist())
print("Tokens   :", tokenizer.convert_ids_to_tokens(ids[0]))
```

#### Outputs with transformers 5.7.0

```
transformers: 5.7.0
torch       : 2.11.0+cu130
sentencepiece: 0.2.1
Token IDs: [[1, 233, 59, 79, 80, 90, 233, 80, 90, 233, 72, 233, 91, 76, 95, 91, 233, 76, 95, 72, 84, 87, 83, 76, 19, 233, 64, 86, 92, 233, 74, 86, 92, 83, 75, 233, 79, 72, 93, 76, 233, 94, 89, 80, 91, 91, 76, 85, 233, 72, 85, 96, 91, 79, 80, 85, 78, 233, 76, 83, 90, 76, 233, 80, 77, 233, 85, 76, 74, 76, 90, 90, 72, 89, 96, 233, 8, 2]]
Tokens   : ['[CLS]', 'Ġ', 'T', 'h', 'i', 's', 'Ġ', 'i', 's', 'Ġ', 'a', 'Ġ', 't', 'e', 'x', 't', 'Ġ', 'e', 'x', 'a', 'm', 'p', 'l', 'e', ',', 'Ġ', 'Y', 'o', 'u', 'Ġ', 'c', 'o', 'u', 'l', 'd', 'Ġ', 'h', 'a', 'v', 'e', 'Ġ', 'w', 'r', 'i', 't', 't', 'e', 'n', 'Ġ', 'a', 'n', 'y', 't', 'h', 'i', 'n', 'g', 'Ġ', 'e', 'l', 's', 'e', 'Ġ', 'i', 'f', 'Ġ', 'n', 'e', 'c', 'e', 's', 's', 'a', 'r', 'y', 'Ġ', '!', '[SEP]']
```

#### Outputs with transformers 4.57.6

```
transformers: 4.57.6
torch       : 2.10.0+cu128
sentencepiece: 0.2.1
Token IDs: [[1, 13711, 5806, 72, 15532, 28108, 19, 9619, 26650, 13592, 94, 5152, 4954, 22118, 5259, 4985, 10723, 4766, 16562, 25007, 8346, 8, 2]]
Tokens   : ['[CLS]', 'This', 'is', 'a', 'text', 'example', ',', 'You', 'could', 'have', 'w', '##rit', '##ten', 'any', '##th', '##ing', 'el', '##se', 'if', 'necess', '##ary', '!', '[SEP]']
```

### Expected behavior

This tokenization:

```
['[CLS]', 'This', 'is', 'a', 'text', 'example', ',', 'You', 'could', 'have', 'w', '##rit', '##ten', 'any', '##th', '##ing', 'el', '##se', 'if', 'necess', '##ary', '!', '[SEP]']
```

Should appear with the 5.X transformers version.

--- Comment by zeel2104 at 2026-04-29T14:43:38Z ---
Open for contribution?

--- Comment by Rocketknight1 at 2026-04-30T10:16:19Z ---
cc @arthurzucker @itazap

--- Comment by itazap at 2026-05-13T03:20:43Z ---
```python

tok_roberta = TokenizersBackend.from_pretrained("FacebookAI/roberta-base")
tok_camembertv2 = TokenizersBackend.from_pretrained("almanach/camembertv2-base")

```
```
Pre-tokenizers differ:
{"type": "ByteLevel", "add_prefix_space": false, "trim_offsets": true, "use_regex": true}
{"type": "Sequence", "pretokenizers": [{"type": "Split", "pattern": {"Regex": "[ ]{1,4}"}, "behavior": "Isolated", "invert": false}, {"type": "Split", "pattern": {"Regex": "^ $"}, "behavior": "Removed", "invert": false}, {"type": "Split", "pattern": {"Regex": "(?i:\\b[cmnjstdl]|qu|puisqu|lorsqu|quelqu|presqu|quoiqu|jusqu…"}, "behavior": "Isolated", "invert": false}, {"type": "Split", "pattern": {"Regex": "[🍆🥖🌪⚗🕰🍭🕎🦈🦵🤍🍯🧦🥙🔖🥛⚰📿📌🇦🏓👫👦🐴🎄🍁🦡🤽🏫🔛🪔🥮🐷⚾🆘🥴🍦🪲🥼🍠🎍🐚⤴🦹🚰🍓💜🍾⏹📳📗👕🎴🗼🇧🤮🥂⏩🎓🔁…"}, "behavior": "Isolated", "invert": false}]}

Decoders differ:
{"type": "ByteLevel", "add_prefix_space": true, "trim_offsets": true, "use_regex": true}
{"type": "WordPiece", "prefix": "##", "cleanup": true}

Post-processors differ:
{"type": "RobertaProcessing", "sep": ["</s>", 2], "cls": ["<s>", 0], "trim_offsets": true, "add_prefix_space": false}
{"type": "RobertaProcessing", "sep": ["[SEP]", 2], "cls": ["[CLS]", 1], "trim_offsets": true, "add_prefix_space": true}
(uv_env) (base) itazaporozhets@Huggingfaces-MacBook-Pro-2 transformers %

```

I'm not sure why this was ever configured as a roberta tokenizer tbh! need a PR to update the class here to `PreTrainedTokenizerFast`

https://huggingface.co/almanach/camembertv2-base/blob/main/tokenizer_config.json#L54

(can load correctly with TokenizersBackend.from_pretrained in the meantime!)