Bvicii Claude Code commited on
Commit
bb2c9bb
·
verified ·
1 Parent(s): 1cd6565

Keep broadcast mask dims in per-image vision attention; pass BlockMask through

Browse files

- Per-image eager/SDPA vision attention sliced both mask dims, so a broadcastable `[B, 1, 1, L]` padding mask got an empty query dim for every image after the first and failed with a size mismatch (e.g. `cu_seqlens=[0, 4, 8]`). Size-1 dims are now kept and only real sequence dims are sliced.
- `_create_causal_mask` read `attention_mask.ndim`, which a flex-attention `BlockMask` does not have. The dense 4D conversion now applies only to `torch.Tensor`, so a BlockMask reaches the native `create_causal_mask`, which returns it unchanged.

Tested with transformers 4.57.6, 5.12.1 and 5.22.1: packed attention with `[B,1,1,L]` and `[B,1,L,L]` padding masks matches running each image alone (eager, SDPA), a BlockMask is returned as the same object, and the existing CPU fp32 / A100 bf16 (eager, SDPA, FA2) regression checks still pass.

Co-Authored-By: Claude Code <noreply@anthropic.com>

Files changed (1) hide show
  1. modeling_paddleocr_vl.py +11 -5
modeling_paddleocr_vl.py CHANGED
@@ -104,7 +104,8 @@ def _init_tensor(tensor, operation, *args, **kwargs):
104
 
105
 
106
  def _create_causal_mask(config, inputs_embeds, attention_mask, past_key_values, cache_position):
107
- if attention_mask is not None and attention_mask.ndim == 4:
 
108
  mask = attention_mask.to(device=inputs_embeds.device)
109
  if mask.dtype == torch.bool:
110
  additive = torch.zeros(mask.shape, device=mask.device, dtype=inputs_embeds.dtype)
@@ -1241,10 +1242,15 @@ class PaddleOCRAttention(nn.Module):
1241
  if output_attentions else None
1242
  )
1243
  for start, end in zip(boundaries[:-1], boundaries[1:]):
1244
- local_mask = (
1245
- attention_mask[..., start:end, start:end]
1246
- if attention_mask is not None else None
1247
- )
 
 
 
 
 
1248
  output, weights = attention_interface(
1249
  self, queries[:, :, start:end], keys[:, :, start:end],
1250
  values[:, :, start:end], local_mask,
 
104
 
105
 
106
  def _create_causal_mask(config, inputs_embeds, attention_mask, past_key_values, cache_position):
107
+ # Only dense tensors need conversion; flex BlockMask is passed through to the native function.
108
+ if isinstance(attention_mask, torch.Tensor) and attention_mask.ndim == 4:
109
  mask = attention_mask.to(device=inputs_embeds.device)
110
  if mask.dtype == torch.bool:
111
  additive = torch.zeros(mask.shape, device=mask.device, dtype=inputs_embeds.dtype)
 
1242
  if output_attentions else None
1243
  )
1244
  for start, end in zip(boundaries[:-1], boundaries[1:]):
1245
+ local_mask = None
1246
+ if attention_mask is not None:
1247
+ # Keep size-1 broadcast dims, e.g. a [B, 1, 1, L] padding mask.
1248
+ span = slice(start, end)
1249
+ local_mask = attention_mask[
1250
+ ...,
1251
+ span if attention_mask.shape[-2] > 1 else slice(None),
1252
+ span if attention_mask.shape[-1] > 1 else slice(None),
1253
+ ]
1254
  output, weights = attention_interface(
1255
  self, queries[:, :, start:end], keys[:, :, start:end],
1256
  values[:, :, start:end], local_mask,