File size: 5,740 Bytes
ede5883
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2f1ec11
 
 
ede5883
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2f1ec11
ede5883
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2f1ec11
 
ede5883
 
 
 
2f1ec11
 
 
 
 
 
 
 
 
 
 
 
ede5883
 
 
 
 
2f1ec11
ede5883
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
---
license: apache-2.0
library_name: transformers
tags:
  - ricursive
  - ripple
  - transformers
  - patch
---

# Ripple-65B-Preview — Transformers 5.x compatibility patch

Drop-in replacement for `modeling_ricursive.py` from
[`ricursiveAI/Ripple-65B-Preview`](https://huggingface.co/ricursiveAI/Ripple-65B-Preview).

The upstream custom code was written against the Transformers 4.x cache API.
Under Transformers 5.x, `generate()` failed on the very first prefill, and even
after that single line was fixed, padded batches decoded incorrectly and
generations longer than `sliding_window` crashed.

This repository distributes the fix only. It contains **no weights**; use it
with the original checkpoint.

## Apply

Copy the patched file over the upstream one:

```bash
cp modeling_ricursive.py /path/to/Ripple-65B-Preview/modeling_ricursive.py
```

Or apply the unified diff from the root of a checkout of the model repository:

```bash
git apply PATCH.diff
```

Then load as usual:

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "ricursiveAI/Ripple-65B-Preview",
    trust_remote_code=True,
)
```

## What was broken upstream

1. **Spurious axis on prefill.** `prepare_inputs_for_generation` does
   `input_ids[:, cache_position]` whenever `input_ids.shape[1] != 1`.
   Transformers 5 pre-creates an empty `DynamicCache` and passes
   `cache_position=None` for the prefill call, so this becomes
   `input_ids[:, None]`: `(batch, seq)` turns into `(batch, 1, seq)`. The
   embedding lookup returns 4-D hidden states and the first decoder layer dies
   on `bsz, q_len, _ = hidden_states.shape`.

2. **Every decode step fully masked.** `_update_causal_mask` seeds the mask with
   `finfo.min` and only writes the visible diagonal via `triu` when
   `sequence_length != 1`. At decode `sequence_length == 1`, so the whole row
   stays masked and the model emits a constant token. The function also writes
   `1.0` (not `0.0`) for visible keys, which biases every score, and calls
   `past_key_values.get_max_cache_shape()`, which Transformers 5 removed.

3. **`position_ids` ignore padding.** They are overwritten with
   `cache_position` for every row, which is only correct when nothing is padded.
   With left padding, a short row's real tokens sit at higher indices, so a
   shared position vector shifts its RoPE phases and the row decodes to garbage.

4. **Sliding window uses the wrong absolute positions.**
   `_apply_sliding_window_mask` builds `arange(kv_len)` and assumes the keys sit
   at positions `0..kv_len-1`. Once the hybrid cache evicts from the left they
   sit at `[cp-kv_len+1 .. cp]`, so the mask excludes progressively more valid
   keys (e.g. window 16 at position 32 leaves 0 of 16 keys visible).

5. **Mask shape crash past the window.** Transformers 5 uses a hybrid cache:
   sliding layers hold at most `sliding_window` keys while full-attention layers
   keep growing. The model builds one mask per forward, so the shape no longer
   matches the sliding layer's key tensor
   (`RuntimeError: expanded size (1024) must match existing size (1025)`).

6. **`position_ids` left on CPU.** Transformers derives them from the caller's
   tensors before the generation hook runs, so a CPU prompt yields CPU positions
   even after the inputs are moved, and RoPE fails with `mat2 is on cpu`.

7. **Missing `GenerationMixin`.** Transformers 5 no longer injects it into
   remote-code classes, so `generate` can be missing entirely and
   `generation_config` is never built.

## What the patch changes

* `prepare_inputs_for_generation` rebuilds a missing `cache_position`, slices
  only when the cache is warm, derives per-row `position_ids` from the padding
  mask, and moves `input_ids` / `attention_mask` / `inputs_embeds` /
  `position_ids` onto the model device.
* `_update_causal_mask` sizes the key axis from `cache_position`, emits `0.0`
  for visible keys and `finfo.min` elsewhere, and preserves visible keys during
  single-token decode.
* `_apply_sliding_window_mask` trims the shared mask to the keys the layer
  holds, anchors key positions at the first key the cache still holds, and is
  skipped when the window cannot bind.
* `RicursiveForCausalLM` inherits `GenerationMixin` explicitly.
* Cached generation also handles new-token blocks, continuation buffers,
  embeddings-only prompts, and explicit position advancement.
* Attention forwards static-cache write positions, trims unwritten capacity,
  preserves caller 4D masks, and returns requested attention weights without
  changing the SDPA output. Tensor-parallel heads use local projection widths.
* `logits_to_keep` limits the LM-head input during inference, and MLP row
  chunking bounds prefill activation memory. Labeled calls retain full-sequence
  FP32 loss and honor accumulation-wide token counts.
* Training implements gradient checkpointing with value-residual gradients;
  tied embeddings work with Transformers 4 and 5.
* Default caches use hybrid layers on Transformers 5 and a crop-compatible
  full cache on Transformers 4, preserving assisted decoding after the window.

The patch is backward compatible with Transformers 4.x.

## Validation

* `generate()` works under Transformers 5.22.2.
* Cached and uncached generation are bit-identical, including past
  `sliding_window` (the eviction path).
* Left-padded batches decode bit-identically to the same rows generated
  unpadded.
* The patched implementation is bit-identical to an independently written
  reference patch across prefill logits, decode, padded and uncached paths.

## Files

```
modeling_ricursive.py   patched drop-in replacement
PATCH.diff              unified diff against the upstream file
```