i3-lab/i3-4096ctx-chat

i3 Hybrid Chat Model

This is a chat-tuned version of the i3 hybrid architecture with latent context compression.

Model Details

• Architecture: RWKV + Attention Hybrid with Latent Compression
• Parameters: ~342.4M
• Context Window: 4096 tokens (via compression)
• Inference Window: 4096 tokens
• Kernel Size: 512 tokens
• Training Data: HuggingFaceH4/ultrachat_200k

Usage

import torch
from tokenizers import Tokenizer

# Load model
model = torch.load("pytorch_model.bin")
tokenizer = Tokenizer.from_file("tokenizer.json")

# Format conversation
conversation = "<BOS><|user|>\nHello!\n<|assistant|>\n"
tokens = torch.tensor([tokenizer.encode(conversation).ids])

# Generate
output = model.generate(tokens, max_new_tokens=100, temperature=0.8)
response = tokenizer.decode(output[0].tolist())

Capabilities

• Multi-turn conversations
• Long context understanding via latent compression
• Efficient inference with RWKV base layers
• Ready for chain-of-thought fine-tuning

Training

Fine-tuned on UltraChat 200k dataset with:

• Learning rate: 1e-05
• Batch size: 4 × 4 accumulation
• Sequence length: 512