metadata
library_name: transformers
pipeline_tag: text-generation
tags:
- conversational
i3 Hybrid Chat Model
This is a chat-tuned version of the i3 hybrid architecture with latent context compression.
Model Details
- Architecture: RWKV + Attention Hybrid with Latent Compression
- Parameters: ~342.4M
- Context Window: 4096 tokens (via compression)
- Inference Window: 4096 tokens
- Kernel Size: 512 tokens
- Training Data: HuggingFaceH4/ultrachat_200k
Usage
import torch
from tokenizers import Tokenizer
# Load model
model = torch.load("pytorch_model.bin")
tokenizer = Tokenizer.from_file("tokenizer.json")
# Format conversation
conversation = "<BOS><|user|>\nHello!\n<|assistant|>\n"
tokens = torch.tensor([tokenizer.encode(conversation).ids])
# Generate
output = model.generate(tokens, max_new_tokens=100, temperature=0.8)
response = tokenizer.decode(output[0].tolist())
Capabilities
- Multi-turn conversations
- Long context understanding via latent compression
- Efficient inference with RWKV base layers
- Ready for chain-of-thought fine-tuning
Training
Fine-tuned on UltraChat 200k dataset with:
- Learning rate: 1e-05
- Batch size: 4 × 4 accumulation
- Sequence length: 512