ISSUE #45305: Gradients not averaged by GAS when using DeepSpeed + model_accepts_loss_kwargs=True (Qwen3, Llama3, etc.)
State: closed  |  Labels: bug
Author: florian6973  |  Created: 2026-04-07T23:31:43Z

### System Info

- `transformers` version: 5.3.0
- Platform: Linux-5.14.0-427.33.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.15
- Huggingface_hub version: 1.6.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: True
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- DeepSpeed version: 0.18.9
- PyTorch version (accelerator?): 2.6.0+cu124 (CUDA)
- Using distributed or parallel set-up in script?: Yes
- Using GPU in script?: Yes
- GPU type: NVIDIA L40S

### Who can help?

@SunMarc, @3outeille, @ArthurZucker

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

## Bug description

When using the HuggingFace `Trainer` with DeepSpeed and any modern model where `model_accepts_loss_kwargs=True` (e.g., Qwen3 [I tested], Llama3 [I tested], and most recent HuggingFace models), gradients are **summed instead of averaged** across gradient accumulation steps. This causes ~GAS× higher grad norms compared to training the same model **without** DeepSpeed, leading to divergent training dynamics.

## Root cause

The issue is a conflict between two code paths in `Trainer.training_step()` (`trainer.py`, lines 1925–1933 in v5.3.0):

```python
# Line 1925-1927: Conditional loss normalization
if (not self.model_accepts_loss_kwargs or num_items_in_batch is None) and self.compute_loss_func is None:
    loss = loss / self.current_gradient_accumulation_steps

# Line 1931-1933: Unconditional scale_wrt_gas=False for DeepSpeed (added by PR #35808)
if self.accelerator.distributed_type == DistributedType.DEEPSPEED:
    kwargs["scale_wrt_gas"] = False

self.accelerator.backward(loss, **kwargs)
```

**PR [#35808](https://github.com/huggingface/transformers/pull/35808)** added `scale_wrt_gas=False` to prevent double-scaling when the Trainer already divides loss by GAS. This fix is correct when `model_accepts_loss_kwargs=False` (Trainer divides on line 1927, and `scale_wrt_gas=False` prevents DeepSpeed from dividing again).

However, for modern models where `model_accepts_loss_kwargs=True`:
1. **Trainer skips dividing** (line 1925 condition is False)
2. **DeepSpeed's gradient scaling is disabled** (`scale_wrt_gas=False` → `_backward_prologue_per_tensor` hook is a no-op)
3. **Nobody divides by GAS** → gradients are summed, not averaged

### Trace of the bug

| Step | `model_accepts_loss_kwargs=True` (Qwen3, etc.) | `model_accepts_loss_kwargs=False` (older models) |
|---|---|---|
| Trainer divides loss by GAS? | ❌ No (line 1925 skipped) | ✅ Yes (line 1927) |
| `scale_wrt_gas=False` passed? | ✅ Yes (line 1932) | ✅ Yes (line 1932) |
| DeepSpeed hook divides grads? | ❌ No (`_scale_wrt_gas=False`) | ❌ No (`_scale_wrt_gas=False`) |
| **Net GAS scaling** | **❌ None — gradients summed** | **✅ 1/GAS from Trainer — correct** |

## Minimal reproduction

**Important:** Run each configuration as a **separate process** so the model is freshly loaded each time.

```python
"""
min_repro.py — Run separately for each configuration:
  python min_repro.py --use_ds false
  deepspeed --num_gpus=1 min_repro.py --use_ds true

Compare grad_norm at equivalent steps between the two runs.
With GAS=8 and the bug present, DS grad_norm will be ~8x higher.
"""
import argparse, json, os, random, tempfile
import torch
from datasets import Dataset
from transformers import (
    AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments, set_seed,
)

DS_CONFIG = {
    "bf16": {"enabled": "auto"},
    "zero_optimization": {"stage": 3,
        "offload_optimizer": {"device": "cpu", "pin_memory": True},
        "offload_param": {"device": "cpu", "pin_memory": True},
        "overlap_comm": True, "contiguous_gradients": True,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto"},
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
}

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--use_ds", type=str, default="false")
    parser.add_argument("--local_rank", type=int, default=-1)
    args = parser.parse_args()
    use_ds = args.use_ds.lower() in ("true", "1", "yes")

    set_seed(42)

    model_name = "Qwen/Qwen3-0.6B-Base"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token

    model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
    model.resize_token_embeddings(len(tokenizer))

    # Random tokens so loss does not trivially collapse
    random.seed(42)
    vocab, seq_len, n = len(tokenizer), 128, 200
    data = {
        "input_ids": [[random.randint(0, vocab-1) for _ in range(seq_len)] for _ in range(n)],
        "labels":    [[random.randint(0, vocab-1) for _ in range(seq_len)] for _ in range(n)],
    }
    dataset = Dataset.from_dict(data)

    ds_path = None
    if use_ds:
        ds_path = os.path.join(tempfile.mkdtemp(), "ds.json")
        with open(ds_path, "w") as f: json.dump(DS_CONFIG, f)

    training_args = TrainingArguments(
        output_dir=f"./test_gas_{'ds' if use_ds else 'no_ds'}",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        max_steps=20, logging_steps=1,
        learning_rate=5e-5, lr_scheduler_type="cosine",
        bf16=True, deepspeed=ds_path,
        report_to="none", save_strategy="no",
        max_grad_norm=0.0,  # Disable clipping so raw grad norms are visible
    )
    trainer = Trainer(model=model, args=training_args,
                      train_dataset=dataset, processing_class=tokenizer)

    print(f"\nDeepSpeed={use_ds}, model_accepts_loss_kwargs={trainer.model_accepts_loss_kwargs}")
    trainer.train()

if __name__ == "__main__":
    main()
```

Run as **two separate invocations** and compare the `grad_norm` column:
```bash
python min_repro.py --use_ds false          # baseline
deepspeed --num_gpus=1 min_repro.py --use_ds true   # ~8x higher grad_norm expected
```

## Evidence from real training

Training Qwen3-1.7B-Base with identical configs (GAS=8, lr=2e-5, cosine schedule, bf16), with and without DeepSpeed ZeRO-3:

- **train/grad_norm**: DeepSpeed run shows ~5-7× higher grad norms (compressed from theoretical 8× by `max_grad_norm=1.0` clipping)
- **train/loss**: DeepSpeed run plateaus ~0.5 higher — model underfits because effective LR is ~8× too large (unaveraged gradients × clipped norm still produce larger updates)
- **train/learning_rate**: Schedules diverge because the optimizer behaves differently under inflated gradients

The ratio of grad norms matches the `gradient_accumulation_steps=8` value, confirming that no GAS averaging is happening in the DeepSpeed path.

### Expected behavior

Gradient norms and loss curves should be **identical** (up to numerical precision) regardless of whether DeepSpeed is enabled, given the same effective batch size and hyperparameters.

The fix should condition `scale_wrt_gas` on whether the Trainer already normalized the loss:

```python
# Proposed fix in training_step():
if self.accelerator.distributed_type == DistributedType.DEEPSPEED:
    if not self.model_accepts_loss_kwargs or num_items_in_batch is None:
        # Trainer already divided by GAS above → disable DeepSpeed's scaling
        kwargs["scale_wrt_gas"] = False
    else:
        # Trainer did NOT divide → let DeepSpeed handle it
        kwargs["scale_wrt_gas"] = True
```

Alternatively, always divide in the Trainer and always set `scale_wrt_gas=False`:

```python
# Always normalize, regardless of model_accepts_loss_kwargs
loss = loss / self.current_gradient_accumulation_steps

if self.accelerator.distributed_type == DistributedType.DEEPSPEED:
    kwargs["scale_wrt_gas"] = False
```

## Workaround

Until this is fixed, users can subclass `Trainer`:

```python
class FixedGASTrainer(Trainer):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.model_accepts_loss_kwargs = False
```

This forces the Trainer to always divide loss by GAS (line 1927), and combined with the existing `scale_wrt_gas=False`, produces correct gradient averaging.

## Related issues

- #35808 — PR that introduced `scale_wrt_gas=False` (correct for old models, breaks new ones)
- #40564 — Loss scales incorrectly with GAS (Gemma3 and other models)
- #34694 — Discrepancy in training loss with GA using DeepSpeed
- #37110 — Loss and gradient explosion caused by the Trainer
- DeepSpeed [#7773](https://github.com/deepspeedai/DeepSpeed/issues/7773) — Related gradient accumulation issue in pipeline parallelism

## Affected models

Any model where `model_accepts_loss_kwargs=True`, which includes most models added since late 2024: **Qwen3, Gemma3, Llama 3, Phi-4**, and any model whose `forward()` accepts `**kwargs` or has `accepts_loss_kwargs = True`.

_LLM Disclaimer: I used Anthropic Claude 4.6 Opus Extended to structure the bug report based on my observations_

--- Comment by SunMarc at 2026-04-09T15:47:04Z ---
Hmmmm for deepspeed, the loss should be averaged correctly if it supports `num_items_in_batch` no ? But indeed there might be some false positive in models that leads to sum the grad instead of averaging it correctly and I need to fix this at some point

--- Comment by florian6973 at 2026-04-09T16:11:23Z ---
Hi SunMarc!

Thanks so much for your reply!

Actually this is very true, I expanded the script to compare with the Pytorch groundtruth (see below).

When running it, I find that DS is correct, so I assume `num_items_in_batch` works as intended for Qwen3. The discrepancy could be in the non-DS path: Accelerate's `backward()` unconditionally dividing by GAS ([here](https://github.com/huggingface/accelerate/blob/94f975df709723e1bd71ba34433fab222cd86305/src/accelerate/accelerator.py#L2827)), which is redundant when `num_items_in_batch` is active. So the no-DS path would over-divide by GAS, making gradients 8× too small. Should I open an issue on the `accelerate` repo?

In accelerate, their own [gradient_accumulation_for_autoregressive_models.py](https://github.com/huggingface/accelerate/blob/94f975df709723e1bd71ba34433fab222cd86305/examples/by_feature/gradient_accumulation_for_autoregressive_models.py#L250) works around it by `multiplying loss * gradient_accumulation_steps` before calling `accelerator.backward()` to cancel the spurious `/GAS`. The Trainer doesn't seem to apply this compensation for the non-DS path, which could be why no-DS gradients are GAS× too small when `num_items_in_batch` is active.

> Ground truth results (GAS=8, Qwen3-0.6B):
> Method A (sum/total_tokens, no /GAS): grad_norm = 57.75 ← correct
> Method B (sum/total_tokens, then /GAS): grad_norm = 7.22 ← 8× too small
> Trainer + DS: grad_norm = 51.28 ≈ Method A ✓
> Trainer + no-DS: grad_norm = 6.41 ≈ Method B ✗

```python
"""
min_repro.py — Three modes to identify which path computes correct gradients:

  python min_repro.py --mode ground_truth   # plain PyTorch, full-batch, no Trainer
  python min_repro.py --mode no_ds          # Trainer without DeepSpeed
  deepspeed --num_gpus=1 min_repro.py --mode ds  # Trainer with DeepSpeed

Compare grad_norm at step 1 across all three.
The ground truth gives the mathematically correct answer.
"""
import argparse, json, os, random, tempfile
import torch
import torch.nn.functional as F
from datasets import Dataset
from transformers import (
    AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments, set_seed,
)

GAS = 8
SEQ_LEN = 128
N_SAMPLES = 64  # must be divisible by GAS
LR = 5e-5
MODEL_NAME = "Qwen/Qwen3-0.6B-Base"

DS_CONFIG = {
    "bf16": {"enabled": "auto"},
    "zero_optimization": {"stage": 0},
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
}

def make_dataset(tokenizer):
    random.seed(42)
    vocab = len(tokenizer)
    return {
        "input_ids": [[random.randint(0, vocab - 1) for _ in range(SEQ_LEN)] for _ in range(N_SAMPLES)],
        "labels": [[random.randint(0, vocab - 1) for _ in range(SEQ_LEN)] for _ in range(N_SAMPLES)],
    }

def ground_truth_run(model, data):
    """
    Plain PyTorch: accumulate GAS micro-batches manually, compute grad norm.
    This is the mathematically unambiguous reference.

    We do what num_items_in_batch SHOULD do:
      loss_i = sum(token_losses_i) / total_tokens_across_all_GAS_microbatches
    Then sum gradients across micro-batches. No extra /GAS.
    """
    model = model.cuda().train()
    model.zero_grad()

    # First GAS micro-batches
    micro_batches = []
    for i in range(GAS):
        input_ids = torch.tensor([data["input_ids"][i]], device="cuda")
        labels = torch.tensor([data["labels"][i]], device="cuda")
        micro_batches.append((input_ids, labels))

    # Total tokens across all micro-batches (analogous to num_items_in_batch)
    total_tokens = sum((lb != -100).sum().item() for _, lb in micro_batches)

    # --- Method A: num_items_in_batch style (sum / total_tokens, no /GAS) ---
    model.zero_grad()
    for input_ids, labels in micro_batches:
        logits = model(input_ids).logits
        shift_logits = logits[..., :-1, :].contiguous()
        shift_labels = labels[..., 1:].contiguous()
        loss = F.cross_entropy(
            shift_logits.view(-1, shift_logits.size(-1)),
            shift_labels.view(-1),
            ignore_index=-100,
            reduction="sum",
        )
        loss = loss / total_tokens
        loss.backward()

    norm_a = torch.nn.utils.clip_grad_norm_(model.parameters(), float("inf")).item()

    # --- Method B: same but with extra /GAS (what Accelerate does for non-DS) ---
    model.zero_grad()
    for input_ids, labels in micro_batches:
        logits = model(input_ids).logits
        shift_logits = logits[..., :-1, :].contiguous()
        shift_labels = labels[..., 1:].contiguous()
        loss = F.cross_entropy(
            shift_logits.view(-1, shift_logits.size(-1)),
            shift_labels.view(-1),
            ignore_index=-100,
            reduction="sum",
        )
        loss = loss / total_tokens / GAS
        loss.backward()

    norm_b = torch.nn.utils.clip_grad_norm_(model.parameters(), float("inf")).item()

    # --- Method C: classic mean + /GAS (old-style, no num_items_in_batch) ---
    model.zero_grad()
    for input_ids, labels in micro_batches:
        logits = model(input_ids).logits
        shift_logits = logits[..., :-1, :].contiguous()
        shift_labels = labels[..., 1:].contiguous()
        loss = F.cross_entropy(
            shift_logits.view(-1, shift_logits.size(-1)),
            shift_labels.view(-1),
            ignore_index=-100,
            reduction="mean",
        )
        loss = loss / GAS
        loss.backward()

    norm_c = torch.nn.utils.clip_grad_norm_(model.parameters(), float("inf")).item()

    print(f"\n{'='*60}")
    print(f"GROUND TRUTH (plain PyTorch, GAS={GAS}, total_tokens={total_tokens})")
    print(f"  Method A (sum/total_tokens, no /GAS):      grad_norm = {norm_a:.6f}")
    print(f"  Method B (sum/total_tokens, then /GAS):     grad_norm = {norm_b:.6f}")
    print(f"  Method C (mean per micro-batch, then /GAS): grad_norm = {norm_c:.6f}")
    print(f"")
    print(f"  A/B ratio = {norm_a/norm_b:.1f}x   (expect {GAS}x)")
    print(f"  A/C ratio = {norm_a/norm_c:.2f}x   (expect ~1x if tokens balanced)")
    print(f"")
    print(f"  → Match Trainer+DS     grad_norm to Method A or B?")
    print(f"  → Match Trainer+no-DS  grad_norm to Method A or B?")
    print(f"{'='*60}\n")

def trainer_run(use_ds, data):
    """Run with HF Trainer, with or without DeepSpeed."""
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token

    model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.bfloat16)
    model.resize_token_embeddings(len(tokenizer))

    dataset = Dataset.from_dict(data)

    ds_path = None
    if use_ds:
        ds_path = os.path.join(tempfile.mkdtemp(), "ds.json")
        with open(ds_path, "w") as f:
            json.dump(DS_CONFIG, f)

    training_args = TrainingArguments(
        output_dir=f"./test_gas_{'ds' if use_ds else 'no_ds'}",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=GAS,
        max_steps=1,
        logging_steps=1,
        learning_rate=LR,
        lr_scheduler_type="cosine",
        bf16=True,
        deepspeed=ds_path,
        report_to="none",
        save_strategy="no",
        max_grad_norm=0.0,
    )
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=dataset,
        processing_class=tokenizer,
    )

    label = "DeepSpeed" if use_ds else "No DeepSpeed"
    print(f"\n{'='*60}")
    print(f"TRAINER ({label}, GAS={GAS})")
    print(f"  model_accepts_loss_kwargs={trainer.model_accepts_loss_kwargs}")
    print(f"{'='*60}")
    trainer.train()
    # grad_norm is in the logged output above

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--mode", type=str, required=True, choices=["ground_truth", "no_ds", "ds"])
    parser.add_argument("--local_rank", type=int, default=-1)
    args = parser.parse_args()

    set_seed(42)

    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token

    data = make_dataset(tokenizer)

    if args.mode == "ground_truth":
        model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.bfloat16)
        model.resize_token_embeddings(len(tokenizer))
        ground_truth_run(model, data)
    elif args.mode == "no_ds":
        trainer_run(use_ds=False, data=data)
    elif args.mode == "ds":
        trainer_run(use_ds=True, data=data)

if __name__ == "__main__":
    main()
```

--- Comment by SunMarc at 2026-04-09T16:21:04Z ---
>  which is redundant when num_items_in_batch is active. So the no-DS path would over-divide by GAS, making gradients 8× too small. Should I open an issue on the accelerate repo?

We don't rely on accelerate for GAS for the no-DS path normally when using Trainer, if you check the `gradient_accumulation_step` value over there, it should be at 1. If not, there is this probably where the issue is.

--- Comment by florian6973 at 2026-04-09T16:27:54Z ---
I printed the value of GAS in accelerate before `trainer.train()` with `    print(f"  accelerator.gradient_accumulation_steps={trainer.accelerator.gradient_accumulation_steps}")
`. And I get 8, so I assume this is issue.

I don't know what happens in the logic here:
https://github.com/huggingface/transformers/blob/6546bec441db2c27e9962c8e1835735f6e76ba98/src/transformers/trainer.py#L758

--- Comment by SunMarc at 2026-04-09T16:37:19Z ---
Ok indeed I see the issue. This is because of my PR: https://github.com/huggingface/transformers/pull/43919
The fix would be to remove

```python
        else:
            grad_acc_kwargs["num_steps"] = self.args.gradient_accumulation_steps
```

and hardcode `grad_acc_kwargs["num_steps"]=1` after that. Thanks for finding this !

--- Comment by SunMarc at 2026-04-09T16:39:01Z ---
Would you be willing to open a PR for that and add a test for a simple model that takes `model_accepts_loss_kwargs=True` ? We do have a GA test, not sure why we didn't manage to catch this regression.


ISSUE #45304: NexusQuant: training-free KV cache compression (10-33x) via DynamicCache hooks
State: closed  |  Labels:
Author: jagmarques  |  Created: 2026-04-07T22:50:31Z

Hi,

Sharing a training-free KV cache compression approach we've been developing that hooks into DynamicCache. Might be useful for folks running into memory limits with long contexts.

**NexusQuant** compresses the KV cache by 10-33x by combining attention-based token eviction with E8 lattice vector quantization. It monkey-patches `DynamicLayer.update` to intercept KV writes — same pattern as kvpress.

Some recent GPU results across 3 models:
- Mistral-7B: 9x compression, essentially zero PPL loss with our real attention scorer
- Phi-3-mini (head_dim=96): works via zero-padding, 9x at +0.59%
- Qwen2.5-7B: needs first/last 2 layers at FP16 (boundary protection) — otherwise catastrophic

One thing that surprised us: **3-bit keys + 2-bit values dramatically outperforms symmetric 2-bit** on all models. The softmax amplifies key quantization noise across all positions, so keys deserve more precision. This is consistent with what the TurboQuant+ project found on Apple Silicon.

The API is a context manager:
```python
from nexusquant.integrations.huggingface import nexusquant_evict

with nexusquant_evict(model, quality="high"):
    output = model.generate(input_ids, max_new_tokens=200)
```

We also added physical KV truncation (actually remove evicted tokens from tensors, not just mask them) and asymmetric K/V as options.

Code: https://github.com/jagmarques/nexusquant

Would welcome any feedback, especially on the DynamicCache hook pattern.

--- Comment by jagmarques at 2026-04-08T08:08:03Z ---
We ran two more experiments since our initial post:

**Graduated layer profile (positive result):** Giving boundary layers (first/last 15%) 3-bit V instead of 2-bit V while keeping middle layers at K3V2 improves quality by 0.02-0.05pp consistently across eviction rates. Small but free.

**Soft eviction (negative result):** Quantizing evicted tokens to 1-bit instead of masking them out is worse, not better (+2.24% vs +0.82% at 35%). The 1-bit tokens corrupt attention patterns. Masking completely is strictly better.

**Updated Cerebrium A10 results (Mistral-7B, 1742-tok prefix):**

| Config | 35% evict | 60% evict |
|--------|-----------|-----------|
| K2V2 hard (baseline) | +0.91% | +1.64% |
| K3V2 hard | +0.82% | +1.22% |
| K3V2 graduated | +0.80% | +1.17% |
| K3V2 soft eviction | +2.24% | +2.39% |

Updated code + negative results blog: https://github.com/jagmarques/nexusquant

--- Comment by github-actions[bot] at 2026-05-08T08:27:55Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #45295: Support Sequence Classification for Gemma 4 Models
State: closed  |  Labels: Feature request
Author: jesperschlegel  |  Created: 2026-04-07T17:52:26Z

### Feature request

Add Gemma4ForSequenceClassification

### Motivation

Without this class, fine-tuning Gemma 4 on classification tasks requires manually adding a classification head, losing compatibility with AutoModelForSequenceClassification, Trainer, and the standard pipeline workflow.

### Your contribution

#45294

--- Comment by Charly21r at 2026-04-14T16:22:54Z ---
This looks like a duplicate of #45373 , which has more context. I’ll proceed there to avoid duplication.

--- Comment by Rocketknight1 at 2026-04-15T10:43:34Z ---
Yes, closing in favour of #45373!


ISSUE #45292: resize_token_embeddings does not effect to output_embeddings
State: closed  |  Labels: bug
Author: KoichiYasuoka  |  Created: 2026-04-07T14:36:45Z

### System Info

- `transformers` version: 5.5.0
- Platform: Linux-6.6.113+-x86_64-with-glibc2.35
- Python version: 3.12.13
- Huggingface_hub version: 1.8.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cpu (NA)
- Using distributed or parallel set-up in script?: <fill in>

(Google Colaboratory)

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Quick reproduce:

```
from transformers import AutoModelForMaskedLM
mdl = AutoModelForMaskedLM.from_pretrained("google-bert/bert-base-uncased")
e = mdl.get_input_embeddings()
g = mdl.get_output_embeddings()
print(e.num_embeddings, g.out_features)
assert e.num_embeddings == g.out_features
e = mdl.resize_token_embeddings(e.num_embeddings + 1)
g = mdl.get_output_embeddings()
print(e.num_embeddings, g.out_features)
assert e.num_embeddings == g.out_features
```

### Expected behavior

Both e.num_embeddings and g.out_features should be increased to 30523 (as transformers 4.57.6 did).

--- Comment by KoichiYasuoka at 2026-04-07T14:50:33Z ---
partially duplicates #45276 but seems different in debug

--- Comment by KoichiYasuoka at 2026-04-08T15:37:47Z ---
I've just found `_tied_weights_keys` at https://github.com/huggingface/transformers/blob/7f6cc4b3c540a0836f0aad5921111e73205dd4e9/src/transformers/models/bert/modeling_bert.py#L915 is conflicted with `resize_token_embeddings` after #41580. Umm...

--- Comment by KoichiYasuoka at 2026-04-25T09:16:04Z ---
Thank you all I've just confirmed that this bug is eliminated in transformers 5.6.x.


ISSUE #45290: `apply_chat_template(tokenize=True)` crashes on assistant messages with tool calls and no content
State: closed  |  Labels: bug
Author: qgallouedec  |  Created: 2026-04-07T13:29:49Z

### System Info

Transformers 5.5..0

### Who can help?

@zucchini-nlp @Rocketknight1

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`ProcessorMixin.apply_chat_template` raises `KeyError: 'content'` when `tokenize=True` and the conversation contains an assistant message with `tool_calls` but no `content` key.

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

messages = [
    [
        {"role": "user", "content": [{"type": "text", "text": "dummy"}]},
        {"role": "assistant", "tool_calls": [{"type": "function", "function": {"name": "foo", "arguments": {}}}]},
    ]
]

processor.apply_chat_template(messages, tokenize=True)
```

```
KeyError: 'content'
  File "processing_utils.py", line 1807, in apply_chat_template
    visuals = [content for content in message["content"] if content["type"] in ["image", "video"]]
                                      ~~~~~~~^^^^^^^^^^^
```

It comes from these lines

https://github.com/huggingface/transformers/blob/52cb0653b48fcb0737a74546911df77034b61732/src/transformers/processing_utils.py#L1801-L1807

where it's assumed that all turns contain a content key. However in the codebase, AFAICT, content is always assumed to be optional for assistant turns with tool calls.

## Possible fix

Guard the access with `.get("content")` or skip messages without content:

```python
for message in conversation:
    content = message.get("content") or []
    visuals = [c for c in content if isinstance(c, dict) and c.get("type") in ["image", "video"]]
```

### Expected behavior

to pass with no content

--- Comment by zucchini-nlp at 2026-04-07T13:38:01Z ---
Yep, if content is optional I don't mind safe accessing it. @Rocketknight1  do you agree that the "content" is optional?

--- Comment by Rocketknight1 at 2026-04-09T13:51:57Z ---
@zucchini-nlp yes! Some models definitely have templates with messages that can have `tool_calls` but no `content`


ISSUE #45278: Many import errors after update from 4.57.0 to 5.5.0
State: closed  |  Labels: bug
Author: marchcat69  |  Created: 2026-04-07T05:28:47Z

### System Info

After updating Transformers from version 4.57.0 to version 5.5.0, I get the following import errors:
ImportError: cannot import name 'HybridCache' from 'transformers'
ImportError: cannot import name 'AutoModelForVision2Seq' from 'transformers'
ImportError: cannot import name 'PretrainedConfig' from 'transformers.modeling_utils'

What should I do to avoid them?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers.modeling_utils import PretrainedConfig, PreTrainedModel
from transformers import Cache, DynamicCache, EncoderDecoderCache, HybridCache, PreTrainedModel

### Expected behavior

No import errors

--- Comment by ibrahim1023 at 2026-04-07T13:42:26Z ---
I reproduced this in clean installs.

On `transformers==5.5.0` with `torch`, these fail:
- `from transformers import HybridCache`
- `from transformers import AutoModelForVision2Seq`
- `from transformers.modeling_utils import PretrainedConfig`

On `transformers==4.57.0`, the same imports succeed, so this does look like a real compatibility break.

This seems mixed rather than one bug:
- `AutoModelForVision2Seq` looks tied to the `vision2seq` -> `image-text-to-text` rename (`AutoModelForImageTextToText` works in `5.5.0`).
- `PretrainedConfig` still imports from `transformers` and `transformers.configuration_utils`, but not from `transformers.modeling_utils`.
- `HybridCache` does not import from `transformers` in `5.5.0`.

I also found likely overlap with `#40695` and `#37623`.

Would maintainers want a focused compatibility fix for the `PretrainedConfig` import path only, or is this issue expected fallout from the rename/removal work already in progress?

--- Comment by Rocketknight1 at 2026-04-08T14:36:24Z ---
The upgrade to version `5` did cause compatibility breaks at some points. This was part of a deliberate cleanup. We try to avoid these breaks most of the time, but on occasion the cruft builds up enough to force a major version with breaking changes.

--- Comment by github-actions[bot] at 2026-05-07T08:50:36Z ---
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.


ISSUE #45276: [gemma4] resize_token_embeddings does not effect to embed_tokens_per_layer or output_embeddings
State: closed  |  Labels: bug
Author: KoichiYasuoka  |  Created: 2026-04-07T02:55:03Z

### System Info

- `transformers` version: 5.5.0
- Platform: Linux-6.6.113+-x86_64-with-glibc2.35
- Python version: 3.12.13
- Huggingface_hub version: 1.8.0
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.10.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: Tesla T4

(Google Colaboratory GPU)

### Who can help?

@zucchini-nlp @Cyrilvallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Reproduce the behavior:

```
from transformers import Gemma4ForConditionalGeneration
mdl = Gemma4ForConditionalGeneration.from_pretrained("google/gemma-4-E2B-it")
e = mdl.get_input_embeddings()
f = mdl.model.language_model.embed_tokens_per_layer
g = mdl.get_output_embeddings()
print(e.num_embeddings, f.num_embeddings, g.out_features)
assert e.num_embeddings == f.num_embeddings == g.out_features
e = mdl.resize_token_embeddings(e.num_embeddings + 1)
f = mdl.model.language_model.embed_tokens_per_layer
g = mdl.get_output_embeddings()
print(e.num_embeddings, f.num_embeddings, g.out_features)
assert e.num_embeddings == f.num_embeddings == g.out_features
```

### Expected behavior

All `e.num_embeddings`, `f.num_embeddings` and `g.out_features` should be increased to 262145.

--- Comment by KoichiYasuoka at 2026-04-07T02:56:46Z ---
I made PR #45236 but failed in test.

--- Comment by zucchini-nlp at 2026-04-07T09:58:02Z ---
I remember seeing a similar issue in gemma3n in the past which couldn't be reproduced unfortunately https://github.com/huggingface/transformers/issues/39921. Looking at prev gemma3n I see that they zero out all token beyond `vocab_size_per_layer` which I believe was supposed to be added in Gemma4 as well 🤔  Not really sure if resizing per-layer-embeddings is intended for FT tbh

https://github.com/huggingface/transformers/blob/b9f0fbf532c124ff836466d896a716e26dbe4722/src/transformers/models/gemma3n/modeling_gemma3n.py#L2048-L2052

--- Comment by KoichiYasuoka at 2026-04-08T01:15:39Z ---
Thank you @zucchini-nlp for the information about #39921.

```
>>> from transformers import pipeline
>>> nlp=pipeline("image-text-to-text","google/gemma-3n-E2B-it",token="hf_...")
>>> nlp.model.get_input_embeddings()
Gemma3nTextScaledWordEmbedding(262400, 2048, padding_idx=0)
>>> nlp.model.get_output_embeddings()
Linear(in_features=2048, out_features=262400, bias=False)
>>> nlp.model.model.language_model.embed_tokens_per_layer
Gemma3nTextScaledWordEmbedding(262144, 7680, padding_idx=0)
```

Oops, `embed_tokens_per_layer` differs in its sizes...

```
>>> nlp.model.model.language_model.embed_tokens_per_layer.weight
Parameter containing:
tensor([[ 0.0613,  0.0200,  0.0603,  ..., -0.0703,  0.0552, -0.0486],
        [ 0.0150, -0.0282,  0.0469,  ..., -0.0530, -0.0645, -0.1309],
        [ 0.0049, -0.0160, -0.0308,  ..., -0.0562,  0.0176, -0.0562],
        ...,
        [ 0.0564,  0.0076, -0.0435,  ...,  0.0410, -0.0223,  0.0042],
        [ 0.0282, -0.0075, -0.0503,  ...,  0.0264,  0.0254, -0.0294],
        [-0.0064,  0.0013,  0.0262,  ..., -0.1357, -0.0215, -0.0679]],
       dtype=torch.bfloat16, requires_grad=True)
>>> nlp.model.model.language_model.embed_tokens_per_layer.weight.size()
torch.Size([262144, 7680])
>>> len(nlp.tokenizer)
262400
```

Umm, it's different from `embed_tokens_per_layer` in gemma4.

```
>>> nlp.model.config.vision_config
Gemma3nVisionConfig {
  "architecture": "mobilenetv5_300m_enc",
  "do_pooling": false,
  "dtype": "bfloat16",
  "hidden_size": 2048,
  "initializer_range": 0.02,
  "label_names": [
    "LABEL_0",
    "LABEL_1"
  ],
  "model_args": null,
  "model_type": "gemma3n_vision",
  "num_classes": 2,
  "rms_norm_eps": 1e-06,
  "transformers_version": "5.5.0",
  "vocab_offset": 262144,
  "vocab_size": 128
}
>>> nlp.model.model.embed_vision
Gemma3nMultimodalEmbedder(
  (embedding): Embedding(128, 2048)
  (hard_embedding_norm): Gemma3nRMSNorm()
  (soft_embedding_norm): Gemma3nRMSNorm()
  (embedding_projection): Linear(in_features=2048, out_features=2048, bias=False)
  (embedding_post_projection_norm): Gemma3nRMSNorm()
)
>>> nlp.tokenizer.convert_ids_to_tokens(262144)
'<end_of_image>'
>>> nlp("<end_of_image>")
Both `max_new_tokens` (=256) and `max_length`(=20) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
[{'input_text': '<end_of_image>', 'generated_text': "<end_of_image>\n,\nI have a project where I'd like to use a RESTful API.  I am unsure where to start.  Can you help me break down the process?\n\nOkay, let's break down the process of using a RESTful API, from understanding the basics to getting it working in practice. Here's a structured guide.\n\n**1. Understanding the Basics of REST (Representational State Transfer)**\n\n*   **What is REST?** REST is an architectural style for designing networked applications. It's based on using standard HTTP methods (like GET, POST, PUT, DELETE) to interact with resources over a network.\n*   **Key Concepts:**\n    *   **Resources:**  Representable things (data, objects, etc.) on a server.  Examples: a user, a product, a blog post.\n    *   **HTTP Methods:**  Define the actions you can perform on a resource.\n        *   `GET`:  Retrieve a resource. (e.g., `GET /users/123`)\n        *   `POST`:  Create a new resource. (e.g., `POST /users`)\n        *   `PUT`:  Update an existing resource completely"}]
```

Well, I think that google/gemma-3n-E2B-it uses token ids 0-262399 in Gemma3nTextModel, while `embed_tokens_per_layer` uses 0-262143 and `embed_vision` uses 262144-262271. Token ids 262272-262399 seem to be used in `audio_tower` but I'm not sure. But now I understand `resize_token_embeddings` is too difficult for gemma3n.

--- Comment by KoichiYasuoka at 2026-04-08T01:23:03Z ---
On the other hand, gemma4 uses token ids easier way.

--- Comment by KoichiYasuoka at 2026-04-08T15:24:12Z ---
I've just found this bug partially comes from `_tied_weights_keys` at https://github.com/huggingface/transformers/blob/7f6cc4b3c540a0836f0aad5921111e73205dd4e9/src/transformers/models/gemma4/modeling_gemma4.py#L2365 and `g.out_features` can be fixed by PR #45311. But how do I fix `f.num_embeddings`?

--- Comment by zucchini-nlp at 2026-04-08T15:35:57Z ---
Haha, is that an agent reasoning output? Yeah, that makes sense if gemma3n already has reserved tokens in `per_layer_input` which is bigger than model's vocab size

My point is that Gemma4 might not need resizing the per-layer input when FT, which would be consistent with 3N release. Anyway, lemme ask the research team about intended behavior

We'll either override `resize_token_embeddings` or mask-out overflowing tokens

--- Comment by KoichiYasuoka at 2026-04-08T16:47:29Z ---
> Haha, is that an agent reasoning output?

No, it's by my head and hands. Through checking #39921 (and #45292) I've found that this bug partially occurs from the confliction between `resize_token_embeddings` and #41580. And I've also found that Gemma3N has `vocab_offset` but Gemma4 not.

--- Comment by zucchini-nlp at 2026-04-08T17:07:40Z ---
Yep, I asked the Gemma team and we need to resize all per-layer embeddings that are not in vision/audio offset range. Linked a PR fixing Gemma4


ISSUE #45265: More permissive config parsing and validation
State: closed  |  Labels: Feature request
Author: vadimkantorov  |  Created: 2026-04-06T16:28:50Z

### Feature request

Make more permissive `config.json`/`params.json` parsing / validation: cast int constants as float without warnings

### Motivation

E.g. when loading Leanstral (cf https://huggingface.co/mistralai/Leanstral-2603/discussions/7#69cfde05abe040f5323c6390):

```
Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'apply_yarn_scaling'}
`rope_parameters`'s factor field must be a float >= 1, got 128
`rope_parameters`'s beta_fast field must be a float, got 32
`rope_parameters`'s beta_slow field must be a float, got 1
Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'apply_yarn_scaling'}
`rope_parameters`'s factor field must be a float >= 1, got 128
`rope_parameters`'s beta_fast field must be a float, got 32
`rope_parameters`'s beta_slow field must be a float, got 1
```

1, 32, 128 should be auto-promoted to float

### Your contribution

N/A

cc @juliendenize

--- Comment by zucchini-nlp at 2026-04-07T10:23:14Z ---
Hey, i think we have to differentiate here between the validation in RoPE params and validation for other config fields, since we recently added a strict type validation for configs

So for RoPE, I agree that we could just check (`float or int`) and raise warning if not as there are many models in the hub with int values saved. Casting shouldn't be needed, the downstream code doesn't crash/perform bad with `int` iirc. If you wish, feel free to submit a PR

Though if you also see an error when loading a config that types do not match (e.g. "dropout has to be float but int provided"), it's better to adjust the type hints. We are not currently force-casting to keep type validation light-weight

--- Comment by vadimkantorov at 2026-04-07T10:26:49Z ---
Sure, adjusting type hints would also work, but IMO just allowing silent int->float promotion would be more future-proof (and have simpler typing). I'd say  should either give a proper error, or just silently auto-promote int to float (as typically using int instead of float is not hiding any errors)

--- Comment by zucchini-nlp at 2026-04-07T11:38:13Z ---
To clarify, are you talking about RoPE params only? I don't mind casting, though i don't think it gives any diff in downstream usage. But yeah, if it is about config typing in general, it is a new addition and we'll be adjusting as we go from user feedback for sure

--- Comment by vadimkantorov at 2026-04-07T11:42:22Z ---
Well, I only hit this with this new Leanstral docker.
It looks quite strange warning messages (and unclear whether they indicate an error, and if not I'd prefer silent auto-promotion from int to float)

So I don't know if it's widespread, but if yes, maybe it would be good not only for rope

--- Comment by zucchini-nlp at 2026-04-07T12:50:26Z ---
oke, I'll fix that today for RoPE then

--- Comment by vadimkantorov at 2026-04-07T15:05:03Z ---
Also, it could be good if `Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'apply_yarn_scaling'}` line was prefixed as `[WARNING] transformers:` or `[INFO] transformers:` to indicate that it's not an error and that it originates from the transformers library (e.g. it can be hard to understand if warning comes out from HF or from VLLM), maybe it can be good to have built-in logging package used for this (and also replace all tqdm with basic logging usage - e.g. using logging methods to print some percentages - tqdm is not very friendly with output redirected to file/captured). Like so, it can be configured/filtered/disabled with core python knobs.

--- Comment by zucchini-nlp at 2026-04-07T15:41:33Z ---
I like the prefixing of logged messages, and imo that is a good idea to apply for all warnings emitted in transformers. I will ping core maintainers internally so we could discuss it

--- Comment by vadimkantorov at 2026-04-07T15:49:12Z ---
Yeah, and also if it uses python's logging package, prefixing/formatting can be nicely handled, although may require either having a global variable (like it also happens with warnings package) or passing the logger object (which is cleaner, but sometimes more intrusive)

--- Comment by juliendenize at 2026-04-07T20:42:59Z ---
Hey regarding these error they were originating from vLLM side when we convert Mistral config to HF. With @hmellor we caught them I believe so i think it should be fine now upstream, including the missing key by levering the api of RoPE Transformers that allow to ignore some keys validation.


ISSUE #45259: How can I use Gemma4's variable image resolution feature?
State: closed  |  Labels:
Author: ejlee95  |  Created: 2026-04-06T05:14:28Z

Hi,

I'd like to adjust the vision token budget. It appears that `default_output_length` in vision_config controls the length of the vision output tokens, but modifying it in `config.json` or loading the processor with `default_output_length=560` don't work.

Could you advise how to change the number of vision outputs?

--- Comment by zucchini-nlp at 2026-04-07T10:38:46Z ---
The config file shouldn't have the `default_output_length` param as it is not a valid param and is not used anywhere in modeling code. The variable length can be tweaked via image/video preprocessors by passing `max_soft_tokens`

For ex:

```python

inputs = processor.apply_chat_template(convo, return_dict=True, tokenize=True, return_tensors="pt", processing_kwargs={"max_soft_tokens": 560})
```

Could you check that it works? Prob it is worth adding a small example in docs 😅


Flash Attention 2.0

ISSUE #45250: Flash Attention 2.0
State: closed  |  Labels:
Author: nicholasdudek  |  Created: 2026-04-05T11:04:24Z

: NemotronHForCausalLM does not support Flash Attention 2.0 yet. Please request to add support where the model is hosted, on its model
   hub page: https://huggingface.co//kaggle/input/models/metric/nemotron-3-nano-30b-a3b-bf16/transformers/default/1/discussions/new or in the
   Transformers GitHub repo: https://github.com/huggingface/transformers/issues/new

--- Comment by zucchini-nlp at 2026-04-07T10:43:33Z ---
The model is not yet supported in transformers (see https://github.com/huggingface/transformers/issues/44863). For remote code it's better to ask under model repo discussions, and you can check-out the linked issue for shipping NemotronH in transformers. Not sure if anyone is working on it currently

I will close the issue as it is remote code