veyra-ai/Kairo-30M-RWKV7-Base

Kairo 30M RWKV7 Base !!GENERATION IS BROKEN RIGHT NOW!!

Kairo 30M RWKV7 Base is an experimental recurrent language model from Veyra AI. It uses an RWKV7-style architecture through the FLA implementation and was trained as part of the Kairo research series, which explores efficient alternatives to dense Transformer models.

This model is not intended to replace the Veyra2 stable line. Veyra2 remains the compatibility-first family. Kairo is for experimental architectures, recurrent models, long-context behavior, and unusual efficiency tradeoffs.

Training loss over 1B tokens

Model details

• Model family: Kairo
• Architecture: RWKV7
• Parameters: ~34M
• Training tokens: 1B
• Context length: 2048 tokens
• Vocabulary size: 8192
• Tokenizer: Veyra 8k tokenizer
• Precision: bfloat16
• Embeddings: untied input/output embeddings
• Training objective: causal language modeling
• Base model: trained from scratch
• License: Apache-2.0

Training data

The model was pretrained on a mixed dataset:

• 80% Cosmopedia v2 from HuggingFaceTB/smollm-corpus
• 10% Python code from codeparrot/github-code-clean
• 10% chat-style data from HuggingFaceH4/ultrachat_200k

The data was streamed and packed into fixed-length sequences during training.

Training setup

The model was trained for 1 billion tokens at a sequence length of 2048.

Training used AdamW with a warmup and cosine decay schedule. The run was performed in bfloat16 on an NVIDIA RTX PRO 6000 Blackwell Server Edition.

Final training loss:

Tokens	Training loss
1B	~2.380

Evaluation

On BLiMP, Kairo 30M RWKV7 Base scored:

Benchmark	Score
BLiMP	63.98%

For comparison, Veyra2 30M Base 2B scored 63.89% on the same BLiMP evaluation setup.

This is a small margin and should not be overinterpreted, but it suggests that the RWKV7 architecture learned grammatical preferences efficiently despite being trained on fewer tokens than the Veyra2 30M 2B baseline.

Notes

Kairo RWKV7 showed strong early sample efficiency during training and reached useful loss levels quickly. However, it trained slower than the Veyra2 dense Transformer-style models and did not beat Veyra2 30M on final training loss.

The model may require stronger decoding controls than Veyra2. Repetition penalty and lower temperature are recommended for sampling.

Example:

This model uses an RWKV7 implementation from Flash Linear Attention. You may need compatible versions of transformers and flash-linear-attention.

pip install -U transformers flash-linear-attention

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "veyra-ai/kairo-30m-rwkv7-base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
).cuda()

prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        temperature=0.4,
        top_k=30,
        repetition_penalty=1.2,
        do_sample=True,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))