rooster-coder-rwkv7-1.5b

LoRA adapter for RWKV/RWKV7-G1k-1.5B-20260930 (Apache-2.0), SFT on the coder profile of the Qwen4 Max reasoning corpus.

This repo ships the adapter and both merged deployment formats:

path what size
adapter_model.safetensors + adapter_config.json PEFT LoRA adapter (needs the base) 56.7 MB
gguf/rooster-coder-rwkv7-1.5b-Q6_K.gguf merged, llama.cpp (rwkv7 arch) 1.34 GB
gguf/rooster-coder-rwkv7-1.5b-Q4_K_M.gguf merged, llama.cpp (rwkv7 arch) 990 MB
mlx-4bit/ merged, MLX 4-bit (4.972 bpw) 907 MB

All four are the same weights: the adapter applied to the Apache-2.0 base, not a new pretrain.

Data

70 rows (1_code 32, 2_cs 29, 4_stats 9) from AJ-Gonzalez/qwen4-max-reasoning-250x, traces generated by qwen/qwen4-max-20261115 via OpenRouter. Each assistant reasoning is stored as <think + reasoning + </think + content, which is the checkpoint's own template form (verified against chat_template.jinja).

Rendered with the RWKV template, no generation prompt, <|endoftext|> appended per conversation, chunked at 2048 tokens with 256 overlap:

split rows chunks tokens
train 63 356 667,970
valid 7 46 88,854

Held-out corpus ids: 10, 19, 74, 76, 137, 169, 210. Dataset: aliciajgonzalez/rooster-coder-rwkv7-sft-data.

Training

Plain LoRA (not QLoRA): r=16, alpha=32, dropout 0.05, task_type=CAUSAL_LM, targets receptance, key, value, output. That name match also hits the FFN key/value projections, so the adapter is 14,155,776 parameters (0.9183%). 3 epochs, batch 1 x grad-accum 8, lr 1e-4, warmup 10, AdamW, gradient checkpointing, max sequence 2048.

fp32, deliberately. The checkpoint is bf16-native and fp16 training produced NaN logits and grad_norm: nan from the first step on a T4; T4 has no bf16, so the run is fp32. Peak 11.43 GiB, 2.46 h for 135 optimizer steps on one Tesla T4 (Kaggle).

Benchmarks

Same harness for both rows: one T4, fp32, RWKV template with thinking=True, greedy, 768 new tokens, evalplus 0.3.1 sanitize + evaluate. NLL is token-level cross-entropy; the held-out chunks are the 46 validation chunks (88,677 tokens) and SWE20 is the same 20 SWE-bench statements used in training (15,427 tokens).

model HumanEval HumanEval+ held-out NLL SWE20 NLL
base RWKV7-G1k-1.5B 0.341 0.329 2.2417 1.6097
+ this adapter 0.201 0.195 2.1179 1.7227

The adapter improves in-distribution NLL (2.2417 -> 2.1179) and degrades HumanEval+ by 13.4 points (0.329 -> 0.195); SWE20 NLL is also worse. 70 rows at lr 1e-4 for 3 epochs overfit the trace style and cost general code ability. Published as a measured negative result, not as a recommended coder model.

Training diagnostics: train loss 1.6735, eval loss 1.9399 (46 chunks), 135 optimizer steps.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AJ-Gonzalez/rooster-coder-rwkv7-1.5b"
base = AutoModelForCausalLM.from_pretrained(
    "RWKV/RWKV7-G1k-1.5B-20260930", trust_remote_code=True, dtype=torch.float32)
model = PeftModel.from_pretrained(base, repo, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)

text = tok.apply_chat_template(
    [{"role": "user", "content": "Write a Python LRU cache."}],
    tokenize=False, add_generation_prompt=True, thinking=True)
ids = torch.tensor([tok.encode(text)])
out = model.generate(ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.shape[1]:]))

The tokenizer, chat template and remote-code files (configuration_rwkv7.py, modeling_rwkv7.py, tokenization_rwkv7.py) are shipped beside the adapter so it loads without the base repository; the weights still come from the base id in adapter_config.json.

Caveats

  • Base checkpoint, not instruction-tuned; the adapter teaches the coder trace style, not general chat.
  • One corpus, 70 rows. Expect the style and not much else.
  • fp16 inference of this architecture is unreliable (that is why training is fp32); use fp32 in PyTorch. The existing RWKV7-G1k-1.5B GGUFs are the base model and do not contain this adapter.
  • The reasoning corpus prompts derive from TeichAI/MiMo; the traces are Qwen4 Max. Upstream terms apply.

Merged artefacts

The adapter is also shipped fused into the base, for inference without PEFT (files listed at the top).

llama-cli -m gguf/rooster-coder-rwkv7-1.5b-Q6_K.gguf \
  -p 'User: Write a Python LRU cache.\n\nAssistant: <think' -n 256 --temp 0
from mlx_lm import load, generate
model, tok = load("AJ-Gonzalez/rooster-coder-rwkv7-1.5b/mlx-4bit")
print(generate(model, tok, prompt="User: Write a Python LRU cache.\n\nAssistant: <think",
               max_tokens=256, temp=0.0))

Measured on the M1 Air (Metal, greedy):

artefact prompt generation peak
Q6_K GGUF 173.6 t/s 32.0 t/s โ€”
Q4_K_M GGUF 173.8 t/s 36.1 t/s โ€”
MLX 4-bit 19.6 t/s 46.0 t/s 1.076 GB

Q6_K and Q4_K_M load and answer coherently; MLX was checked on the same prompt. These are smoke results, not benchmarks.

How the merge was done: a streaming LoRA merge (scale 2.0) into the bf16 base, then convert_hf_to_gguf.py from llama.cpp master. The GGUF has the same 678 tensors and shapes as the published base GGUF: att.ln1 -> attn_norm, att.ln2 -> attn_norm_2 (llama.cpp uses attn_norm_2 as the FFN norm), att.ln_x -> time_mix_ln, the six att.x_* vectors stack into time_mix_lerp_fused, and the vocabulary is the same 65,536-token list. MLX maps att.ln_x to g_norm after reshaping to (heads, head_dim) and transposes the low-rank gate matrices, since the HF form applies them as x @ W.

Downloads last month
91
MLX
Hardware compatibility
Log In to add your hardware

Quantized

GGUF
Model size
2B params
Architecture
rwkv7
Hardware compatibility
Log In to add your hardware

4-bit

6-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AJ-Gonzalez/rooster-coder-rwkv7-1.5b

Adapter
(1)
this model

Dataset used to train AJ-Gonzalez/rooster-coder-rwkv7-1.5b