Fix rotary/kvcache OOB with speaker_audio + CUDA/CPU device mix

#13
by multimodalart HF Staff - opened

Two bugs that surface on Blackwell + torch 2.10:

  1. setup_cache was allocated to seq_len, but delayed_codes (after apply_delay_pattern) is 9 frames longer. Generation overruns the rotary table / kv cache on the last few steps — usually invisible with the upstream flash_attn (its kvcache kernel silently clamped), but explodes as vectorized_gather_kernel: index out of bounds on Blackwell where the SDPA shim does explicit cos = rotary_cos[positions].

  2. remaining_steps[mask] = torch.minimum(remaining_steps[mask], torch.tensor(9)) mixes a CUDA tensor with a CPU torch.tensor(9). torch 2.10 device-asserts on this. .clamp(max=9) is equivalent and stays on-device.

Verified end-to-end via dev-mode SSH and an outside smoke test.

multimodalart changed pull request status to merged

Sign up or log in to comment