Upload folder using huggingface_hub

README.md

@@ -0,0 +1,45 @@
+# i3 Hybrid Chat Model
+
+This is a chat-tuned version of the i3 hybrid architecture with latent context compression.
+
+## Model Details
+
+- **Architecture**: RWKV + Attention Hybrid with Latent Compression
+- **Parameters**: ~342.4M
+- **Context Window**: 4096 tokens (via compression)
+- **Inference Window**: 4096 tokens
+- **Kernel Size**: 512 tokens
+- **Training Data**: HuggingFaceH4/ultrachat_200k
+
+## Usage
+
+```python
+import torch
+from tokenizers import Tokenizer
+
+# Load model
+model = torch.load("pytorch_model.bin")
+tokenizer = Tokenizer.from_file("tokenizer.json")
+
+# Format conversation
+conversation = "<BOS><|user|>\nHello!\n<|assistant|>\n"
+tokens = torch.tensor([tokenizer.encode(conversation).ids])
+
+# Generate
+output = model.generate(tokens, max_new_tokens=100, temperature=0.8)
+response = tokenizer.decode(output[0].tolist())
+```
+
+## Capabilities
+
+- Multi-turn conversations
+- Long context understanding via latent compression
+- Efficient inference with RWKV base layers
+- Ready for chain-of-thought fine-tuning
+
+## Training
+
+Fine-tuned on UltraChat 200k dataset with:
+- Learning rate: 1e-05
+- Batch size: 4 × 4 accumulation
+- Sequence length: 512

config.json

@@ -0,0 +1,23 @@
+{
+  "architectures": [
+    "i3HybridChatModel"
+  ],
+  "model_type": "i3-chat",
+  "d_model": 1180,
+  "n_layers": 14,
+  "rwkv_layers": 12,
+  "attn_layers": 2,
+  "vocab_size": 32000,
+  "kernel_size": 512,
+  "max_latent_context": 4096,
+  "inference_context_window": 4096,
+  "compression_enabled": true,
+  "num_latent_tokens": 32,
+  "task": "chat",
+  "special_tokens": {
+    "bos_token": "<BOS>",
+    "eos_token": "<EOS>",
+    "user_token": "<|user|>",
+    "assistant_token": "<|assistant|>"
+  }
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c80e15b5de785422713557ae4cd724c198f0e2fba5716c65beb2f7d4fab8ada6
+size 1369667759