Hakureirm/rwkv7-2.9b-hf

RWKV-7 "Goose" World 2.9B — ready for the in-tree transformers implementation

This is not a new model. It is a format conversion of BlinkDL/rwkv-7-world → RWKV-x070-World-2.9B-v3-20250211-ctx4096.pth, laid out as a transformers directory so the rwkv7 implementation can load it with from_pretrained. All credit for the weights belongs to BlinkDL / the RWKV project; they are redistributed here under the Apache-2.0 licence they were released under.

The conversion is a rename, not a transformation: every one of the 1059 tensors in the source .pth is bit-identical here (verified tensor by tensor). The only additions are three all-zero placeholders for layer 0's value-residual LoRA, which that layer never reads (layer 0 produces v_first rather than mixing towards it). bfloat16, the dtype the source is stored in. Training context length 4096.

Usage

The implementation ships inside this repo (auto_map remote code), so stock transformers is all you need — no fork, no extra package:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("RWKV/RWKV7-Goose-World2.8-0.1B-HF", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Hakureirm/rwkv7-2.9b-hf", trust_remote_code=True, dtype=torch.bfloat16)

inputs = tokenizer("The Eiffel Tower is located in the city of", return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=20)[0]))

(The tokenizer is the RWKV "world" tokenizer, shared by every model in the family, so the 0.1B repo above is just a convenient place to load it from.)

The same model class is also open as an in-tree PR (rwkv-rs/transformers-rwkv#2); this repo is the way to use it today on released transformers.

Runs on

Pure-PyTorch portable path, no CUDA-only code on it. The implementation is verified greedy 20/20 against BlinkDL's own runtime on CPU, Apple Silicon MPS and CUDA (see Hakureirm/rwkv7-0.1b-hf for that matrix and the logit-level comparison). Optional Triton kernels engage only on CUDA and fall back to the portable path everywhere else.

RWKV-7 is attention-free and fully recurrent: the state is a fixed-size matrix per head, so there is no KV cache, memory is constant in context length, and each new token costs the same as the first.

What was verified for this size

• all 1059 source tensors bit-identical after conversion, none missing, none extra beyond the three zero placeholders named above;

• the converter's shape check: every tensor matches the shape the inferred config implies (32 layers, hidden 2560, 40 heads x 64, ffn 10240, LoRA ranks w/a/v/g 96/96/64/320);

• a bf16 CUDA greedy smoke run. "The Eiffel Tower is located in the city of" continues:

  The Eiffel Tower is located in the city of Paris, France. It was built in 1889 and is named after the engineer

The logit-level check against BlinkDL's own runtime was run on the 0.1B conversion (same converter, same implementation), not repeated per size.

Reproducing the conversion

huggingface-cli download BlinkDL/rwkv-7-world RWKV-x070-World-2.9B-v3-20250211-ctx4096.pth --local-dir .
python src/transformers/models/rwkv7/convert_rwkv7_checkpoint_to_hf.py \
    --checkpoint RWKV-x070-World-2.9B-v3-20250211-ctx4096.pth \
    --flavour native --dtype bfloat16 --output_dir ./rwkv7-2.9b-hf

The converter compares the checkpoint's tensor shapes against the ones the config implies and refuses a disagreement, so a config that names a different model fails rather than producing something that loads and generates noise.

Citation

The model is RWKV-7 "Goose" by Bo Peng (BlinkDL) and the RWKV community. Reference implementation: BlinkDL/RWKV-LM.