delete autocast, an unnecessary thorn in my side, manage dtypes directly
Andrej Karpathy
1076f97
also update comments
svlandeg
535f664
Fix SDPA KV-cache decode to respect sliding window (#456)
Andrej Karpathy+2: KartikVashishta, Claude Opus 4.5
3ba42e8
SDPA fallback now respects sliding window during single-token KV-cache
decode by slicing K/V to the last (window + 1) tokens.
Also simplifies the mask building for chunk inference to properly apply
sliding window in that path as well.
Fixes #452
Co-Authored-By: Kartik Vashishta <kartikv776@gmail.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
implement flash attention 3 fallback to pytorch sdpa by touching as few lines of code as possible in main files and keeping all implementation to a single file. add tests. add helpful warning messages for the user.
Andrej Karpathy +2: KartikVashishta, Claude Opus 4.5 SDPA fallback now respects sliding window during single-token KV-cache decode by slicing K/V to the last (window + 1) tokens. Also simplifies the mask building for chunk inference to properly apply sliding window in that path as well. Fixes #452 Co-Authored-By: Kartik Vashishta <kartikv776@gmail.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>3ba42e8