Instructions to use ModelCloud/Ripple-65B-Preview-tf5-patch with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ModelCloud/Ripple-65B-Preview-tf5-patch with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ModelCloud/Ripple-65B-Preview-tf5-patch", device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 5,740 Bytes
ede5883 2f1ec11 ede5883 2f1ec11 ede5883 2f1ec11 ede5883 2f1ec11 ede5883 2f1ec11 ede5883 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | ---
license: apache-2.0
library_name: transformers
tags:
- ricursive
- ripple
- transformers
- patch
---
# Ripple-65B-Preview — Transformers 5.x compatibility patch
Drop-in replacement for `modeling_ricursive.py` from
[`ricursiveAI/Ripple-65B-Preview`](https://huggingface.co/ricursiveAI/Ripple-65B-Preview).
The upstream custom code was written against the Transformers 4.x cache API.
Under Transformers 5.x, `generate()` failed on the very first prefill, and even
after that single line was fixed, padded batches decoded incorrectly and
generations longer than `sliding_window` crashed.
This repository distributes the fix only. It contains **no weights**; use it
with the original checkpoint.
## Apply
Copy the patched file over the upstream one:
```bash
cp modeling_ricursive.py /path/to/Ripple-65B-Preview/modeling_ricursive.py
```
Or apply the unified diff from the root of a checkout of the model repository:
```bash
git apply PATCH.diff
```
Then load as usual:
```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"ricursiveAI/Ripple-65B-Preview",
trust_remote_code=True,
)
```
## What was broken upstream
1. **Spurious axis on prefill.** `prepare_inputs_for_generation` does
`input_ids[:, cache_position]` whenever `input_ids.shape[1] != 1`.
Transformers 5 pre-creates an empty `DynamicCache` and passes
`cache_position=None` for the prefill call, so this becomes
`input_ids[:, None]`: `(batch, seq)` turns into `(batch, 1, seq)`. The
embedding lookup returns 4-D hidden states and the first decoder layer dies
on `bsz, q_len, _ = hidden_states.shape`.
2. **Every decode step fully masked.** `_update_causal_mask` seeds the mask with
`finfo.min` and only writes the visible diagonal via `triu` when
`sequence_length != 1`. At decode `sequence_length == 1`, so the whole row
stays masked and the model emits a constant token. The function also writes
`1.0` (not `0.0`) for visible keys, which biases every score, and calls
`past_key_values.get_max_cache_shape()`, which Transformers 5 removed.
3. **`position_ids` ignore padding.** They are overwritten with
`cache_position` for every row, which is only correct when nothing is padded.
With left padding, a short row's real tokens sit at higher indices, so a
shared position vector shifts its RoPE phases and the row decodes to garbage.
4. **Sliding window uses the wrong absolute positions.**
`_apply_sliding_window_mask` builds `arange(kv_len)` and assumes the keys sit
at positions `0..kv_len-1`. Once the hybrid cache evicts from the left they
sit at `[cp-kv_len+1 .. cp]`, so the mask excludes progressively more valid
keys (e.g. window 16 at position 32 leaves 0 of 16 keys visible).
5. **Mask shape crash past the window.** Transformers 5 uses a hybrid cache:
sliding layers hold at most `sliding_window` keys while full-attention layers
keep growing. The model builds one mask per forward, so the shape no longer
matches the sliding layer's key tensor
(`RuntimeError: expanded size (1024) must match existing size (1025)`).
6. **`position_ids` left on CPU.** Transformers derives them from the caller's
tensors before the generation hook runs, so a CPU prompt yields CPU positions
even after the inputs are moved, and RoPE fails with `mat2 is on cpu`.
7. **Missing `GenerationMixin`.** Transformers 5 no longer injects it into
remote-code classes, so `generate` can be missing entirely and
`generation_config` is never built.
## What the patch changes
* `prepare_inputs_for_generation` rebuilds a missing `cache_position`, slices
only when the cache is warm, derives per-row `position_ids` from the padding
mask, and moves `input_ids` / `attention_mask` / `inputs_embeds` /
`position_ids` onto the model device.
* `_update_causal_mask` sizes the key axis from `cache_position`, emits `0.0`
for visible keys and `finfo.min` elsewhere, and preserves visible keys during
single-token decode.
* `_apply_sliding_window_mask` trims the shared mask to the keys the layer
holds, anchors key positions at the first key the cache still holds, and is
skipped when the window cannot bind.
* `RicursiveForCausalLM` inherits `GenerationMixin` explicitly.
* Cached generation also handles new-token blocks, continuation buffers,
embeddings-only prompts, and explicit position advancement.
* Attention forwards static-cache write positions, trims unwritten capacity,
preserves caller 4D masks, and returns requested attention weights without
changing the SDPA output. Tensor-parallel heads use local projection widths.
* `logits_to_keep` limits the LM-head input during inference, and MLP row
chunking bounds prefill activation memory. Labeled calls retain full-sequence
FP32 loss and honor accumulation-wide token counts.
* Training implements gradient checkpointing with value-residual gradients;
tied embeddings work with Transformers 4 and 5.
* Default caches use hybrid layers on Transformers 5 and a crop-compatible
full cache on Transformers 4, preserving assisted decoding after the window.
The patch is backward compatible with Transformers 4.x.
## Validation
* `generate()` works under Transformers 5.22.2.
* Cached and uncached generation are bit-identical, including past
`sliding_window` (the eviction path).
* Left-padded batches decode bit-identically to the same rows generated
unpadded.
* The patched implementation is bit-identical to an independently written
reference patch across prefill logits, decode, padded and uncached paths.
## Files
```
modeling_ricursive.py patched drop-in replacement
PATCH.diff unified diff against the upstream file
```
|