Initial commit: caca-150M untrained model

README.md

@@ -0,0 +1,1343 @@
+---
+license: apache-2.0
+language:
+- id
+- en
+tags:
+- text-generation
+- pytorch
+- causal-lm
+- transformer
+- untrained
+- gqa
+- rope
+- swiglu
+- rmsnorm
+- flash-attention
+- indonesian
+- bilingual
+library_name: transformers
+pipeline_tag: text-generation
+widget:
+  - text: "Jakarta adalah ibu kota"
+    example_title: "🇮🇩 Pelengkapan Teks (ID)"
+  - text: |
+      Pertanyaan: Apa itu kecerdasan buatan?
+      Jawaban:
+    example_title: "🇮🇩 Tanya Jawab (ID)"
+  - text: |
+      Tulis cerita pendek tentang robot yang belajar mencintai.
+    example_title: "🇮🇩 Penulisan Kreatif (ID)"
+  - text: "The capital of Indonesia is"
+    example_title: "🇬🇧 Text Completion (EN)"
+  - text: |
+      Question: What is artificial intelligence?
+      Answer:
+    example_title: "🇬🇧 Question Answering (EN)"
+  - text: |
+      def fibonacci(n):
+          """Hitung bilangan fibonacci ke-n"""
+    example_title: "💻 Pelengkapan Kode"
+  - text: |
+      # Fungsi untuk mengurutkan array
+      def sort_array(arr):
+    example_title: "💻 Generasi Kode"
+  - text: |
+      User: Halo! Siapa kamu?
+      Assistant:
+    example_title: "💬 Format Chat (ID)"
+  - text: |
+      User: Jelaskan tentang machine learning dalam 2 kalimat.
+      Assistant:
+    example_title: "💬 Conversational (ID)"
+inference:
+  parameters:
+    max_new_tokens: 100
+    temperature: 0.7
+    top_p: 0.9
+    top_k: 50
+    do_sample: true
+    repetition_penalty: 1.1
+    num_beams: 1
+datasets: []
+metrics:
+- perplexity
+- accuracy
+model-index:
+- name: caca-150M
+  results: []
+---
+
+<div align="center">
+
+<img src="https://i.postimg.cc/MTSj073X/logo.png" width="400" alt="caca-150M"/>
+
+# 🤖 caca-150M
+
+### Arsitektur Transformer Modern dengan Fitur Canggih
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-red.svg)](https://pytorch.org/)
+[![Transformers](https://img.shields.io/badge/🤗%20Transformers-4.35+-yellow.svg)](https://github.com/huggingface/transformers)
+[![Model Type](https://img.shields.io/badge/Model-Causal%20LM-green.svg)]()
+[![Parameters](https://img.shields.io/badge/Parameters-145.24M-orange.svg)]()
+[![Status](https://img.shields.io/badge/Status-Untrained-red.svg)]()
+
+**145,244,672** parameters • **145.24M** • **29 layers** • **2,048 tokens**
+
+[📚 Documentation](#-dokumentasi) • [💻 Usage](#-cara-penggunaan) • [⚙️ Configuration](#️-konfigurasi-detail) • [🔬 Architecture](#-arsitektur)
+
+</div>
+
+---
+
+## ⚠️ PENTING: Model Belum Dilatih (Untrained)
+
+<div style="background: #fff3cd; border-left: 4px solid #ffc107; padding: 12px; margin: 16px 0;">
+<strong>⚠️ PERHATIAN</strong>: Ini adalah model yang <strong>belum melalui proses training</strong>. Bobot model masih dalam kondisi <strong>random initialization</strong>. Output yang dihasilkan akan <strong>tidak bermakna dan acak</strong>.
+</div>
+
+**Status Model:**
+- 🔴 **Belum dilatih** - Bobot masih random (Kaiming/Xavier init)
+- 🟡 **Untuk riset & eksperimen** - Arsitektur sudah siap, tinggal train
+- 🟢 **Production-ready architecture** - Teruji dan optimal
+
+Widget di atas hanya menunjukkan **format input yang diharapkan**. Setelah model dilatih dengan dataset yang tepat, format yang sama akan menghasilkan output berkualitas tinggi.
+
+### 🎯 Apa yang Bisa Dilakukan?
+
+| ✅ Bisa | ❌ Belum Bisa |
+|---------|----------------|
+| Load model architecture | Generate teks bermakna |
+| Test forward pass | Menjawab pertanyaan |
+| Measure memory & speed | Reasoning & understanding |
+| Start training | Production deployment |
+| Fine-tuning experiments | Real-world applications |
+
+---
+
+## 📋 Deskripsi
+
+**Caca** adalah arsitektur Large Language Model (LLM) generasi terbaru yang menggabungkan berbagai teknik state-of-the-art dalam deep learning. Model ini dirancang dengan fokus pada **efisiensi komputasi**, **skalabilitas**, dan **performa tinggi**.
+
+<blockquote style="border-left: 4px solid #4A90E2; padding-left: 16px; margin: 16px 0; background: #f8f9fa; padding: 12px;">
+<p><strong>📖 Tentang Project Caca</strong></p>
+<p><em>Caca</em> adalah eksperimen open-source Indonesian LLM yang dibuat dari nol secara individual dan bertahap. Bukan kompetitor siapa-siapa, cuma pengen eksplorasi apa yang bisa dilakukan dengan budget terbatas, passion unlimited, dan mindset collaborative.</p>
+<p>Kalau berguna buat orang lain, alhamdulillah. Kalau enggak, ya tetap fun kok. Ini proyek eksplorasi, jadi kalau gagal ya bagian dari proses belajar. Kalau berhasil, itu bonus.</p>
+<p>— <strong>Lyon</strong>, Creator</p>
+</blockquote>
+
+### 🌟 Mengapa Caca?
+
+1. **🇮🇩 Fokus pada Bahasa Indonesia** - Dirancang dengan mempertimbangkan karakteristik bahasa Indonesia
+2. **⚡ Efisiensi Tinggi** - GQA & Flash Attention untuk inferensi 3-5x lebih cepat
+3. **💾 Memory Efficient** - Hemat 80% memory untuk KV cache
+4. **🔧 Modular & Extensible** - Mudah dikustomisasi untuk berbagai use case
+5. **🌐 Bilingual** - Support optimal untuk Indonesia & English
+
+### 🎯 Keunggulan vs Model Lain
+
+| Fitur | Caca caca-150M | LLaMA-2 145.24M | GPT-3 145.24M |
+|-------|------------|-----------|----------|
+| **Attention Type** | GQA | GQA | MHA |
+| **Position Encoding** | RoPE + ALiBI | RoPE | Learned |
+| **Activation** | SwiGLU | SwiGLU | GELU |
+| **Flash Attention** | ✅ v2 | ✅ v1/v2 | ❌ |
+| **Long Context** | Sliding Window + Sink | ✅ | Limited |
+| **MoE Support** | ✅ Optional | ❌ | ❌ |
+| **Multimodal** | ✅ Optional | ❌ | ❌ |
+| **Quantization** | 4/8-bit | 4/8-bit | Limited |
+
+---
+
+## 🎯 Use Cases & Applications
+
+### ✅ Cocok Untuk
+
+<table>
+<tr>
+<td width="50%">
+
+**🔬 Research & Development**
+- Eksperimen arsitektur transformer
+- Ablation studies
+- Novel training techniques
+- Architecture search
+
+**📚 Academic & Education**
+- Thesis & research papers
+- Teaching materials
+- Student projects
+- LLM internals understanding
+
+</td>
+<td width="50%">
+
+**🚀 Base Model for Fine-tuning**
+- Task-specific models
+- Domain adaptation
+- Instruction tuning
+- RLHF experiments
+
+**💡 Prototyping**
+- Proof of concept
+- Feature testing
+- A/B testing architectures
+- Benchmark comparisons
+
+</td>
+</tr>
+</table>
+
+### ❌ Tidak Cocok Untuk
+
+<div style="background: #ffe6e6; border-left: 4px solid #ff4444; padding: 12px; margin: 16px 0;">
+
+- 🚫 **Production Applications** - Model belum dilatih, output random
+- 🚫 **Real-world Deployment** - Perlu training & safety alignment dulu
+- 🚫 **Safety-critical Systems** - Tidak ada safety guardrails
+- 🚫 **Direct User-facing Apps** - Output tidak dapat diprediksi
+- 🚫 **Commercial Use (as-is)** - Harus dilatih terlebih dahulu
+
+</div>
+
+---
+
+## 📊 Spesifikasi Model
+
+<table>
+<tr>
+<td><strong>Parameter</strong></td>
+<td><strong>Value</strong></td>
+<td><strong>Parameter</strong></td>
+<td><strong>Value</strong></td>
+</tr>
+<tr>
+<td>Total Parameters</td>
+<td><code>145,244,672</code></td>
+<td>Vocab Size</td>
+<td><code>16,000</code></td>
+</tr>
+<tr>
+<td>Hidden Size</td>
+<td><code>640</code></td>
+<td>Intermediate Size</td>
+<td><code>1728</code></td>
+</tr>
+<tr>
+<td>Num Layers</td>
+<td><code>29</code></td>
+<td>Attention Heads</td>
+<td><code>10</code></td>
+</tr>
+<tr>
+<td>KV Heads (GQA)</td>
+<td><code>2</code></td>
+<td>Head Dimension</td>
+<td><code>64</code></td>
+</tr>
+<tr>
+<td>Max Context Length</td>
+<td><code>2,048</code></td>
+<td>RoPE Base (θ)</td>
+<td><code>10,000</code></td>
+</tr>
+<tr>
+<td>Model Size (FP16)</td>
+<td><code>0.29 GB</code></td>
+<td>Formatted Size</td>
+<td><code>145.24M</code></td>
+</tr>
+</table>
+
+---
+
+### 🎯 Core Features
+
+<details open>
+<summary><b>🔍 Klik untuk expand/collapse</b></summary>
+
+- ✅ **Grouped Query Attention (GQA)** - Efisiensi memori dan komputasi superior
+  - Query heads: **10**
+  - KV heads: **2**
+  - Ratio: **5:1** (hemat ~80% memory KV cache)
+  - **Benefit**: Inferensi lebih cepat dengan memory footprint lebih kecil
+
+- ✅ **Rotary Position Embeddings (RoPE)** - Generalisasi konteks panjang lebih baik
+  - Theta (θ): **10,000**
+  - Support extrapolation untuk konteks > training length
+  - **Benefit**: Performa stabil pada sequence length yang belum pernah dilihat saat training
+
+- ✅ **RMSNorm** - Normalisasi lebih stabil dan ~50% lebih cepat dari LayerNorm
+  - Epsilon: **1e-06**
+  - **Benefit**: Training lebih stabil, inference lebih cepat, gradient flow lebih baik
+
+- ✅ **SwiGLU Activation** - Performa 10-15% lebih baik dari ReLU/GELU
+  - Intermediate size: **1,728** (2.7x hidden)
+  - **Benefit**: Kapasitas model lebih besar tanpa menambah parameter signifikan
+
+- ✅ **Flash Attention 2** - Akselerasi hingga 3x dengan memory efficiency
+  - Otomatis aktif jika tersedia CUDA device
+  - IO-aware algorithm untuk minimal HBM access
+  - **Benefit**: Training & inference jauh lebih cepat, support batch size lebih besar
+
+</details>
+
+### 🔥 Advanced Features
+
+### 🎯 Mekanisme Attention
+
+- ⚡ **Flash Attention v2** - Algoritma IO-aware yang 3x lebih cepat dari attention standar
+- 🔑 **Grouped Query Attention (GQA)** - 10 Query heads : 2 KV heads
+  - Rasio kompresi: **5:1** (hemat ~80% memory KV cache)
+- 🚀 **xFormers Support** - Fallback memory-efficient attention
+- 🎯 **PyTorch SDPA** - Native scaled dot product attention
+
+### 📍 Position Encodings
+
+- 🔄 **RoPE (Rotary Position Embeddings)** - Base frequency θ=10,000
+  - Generalisasi lebih baik untuk sequence panjang dibanding absolute PE
+
+### 🎓 Optimisasi Training
+
+- 💾 **Gradient Checkpointing** - Trade compute for memory (support model hingga 100B+ params)
+- 🎯 **Mixed Precision Training** - Support FP16, BF16, dan TF32
+- 📉 **Dropout Regularization**
+  - Hidden dropout: 0.1
+  - Attention dropout: 0.0
+  - Residual dropout: 0.1
+
+### 📦 Dukungan Quantization
+
+- 4️⃣ **4-bit Quantization** - NF4 & FP4 via bitsandbytes
+  - Memory reduction: ~**75%** (4GB → 1GB)
+  - Accuracy loss: <2% pada kebanyakan tasks
+  - Support double quantization untuk kompresi maksimal
+- 8️⃣ **8-bit Quantization** - LLM.int8() dengan outlier handling
+  - Memory reduction: ~**50%** (4GB → 2GB)
+  - Accuracy loss: <1%
+- 🔄 **Dynamic Quantization** - Runtime quantization tanpa calibration
+
+### 🔬 Advanced Features
+
+- 📊 **Automatic Mixed Precision (AMP)** - Dynamic loss scaling
+- 🎯 **Gradient Clipping** - Stabilitas training dengan max norm clipping
+- 📈 **Learning Rate Scheduling** - Support cosine, linear, warmup
+- 💡 **Smart Memory Management** - Auto cache clearing & monitoring
+- 🔍 **Metrics Tracking** - Real-time perplexity, loss, gradient norms
+- 🛡️ **NaN/Inf Detection** - Automatic recovery dari numerical instability
+
+---
+
+## 💾 Kebutuhan Memory
+
+### Training Requirements
+
+<table>
+<tr>
+<th>Configuration</th>
+<th>Model Weights</th>
+<th>+ Optimizer States</th>
+<th>Total Training</th>
+</tr>
+<tr>
+<td><strong>FP32 (AdamW)</strong></td>
+<td>0.58 GB</td>
+<td>+1.74 GB</td>
+<td><strong>2.32 GB</strong></td>
+</tr>
+<tr>
+<td><strong>Mixed Precision</strong></td>
+<td>0.29 GB</td>
+<td>+2.03 GB</td>
+<td><strong>2.32 GB</strong></td>
+</tr>
+<tr>
+<td><strong>+ Gradient Checkpointing</strong></td>
+<td colspan="2">Menghemat ~30-50% activation memory</td>
+<td><strong>~1.39 GB</strong></td>
+</tr>
+</table>
+
+### Inference Requirements
+
+<table>
+<tr>
+<th>Precision</th>
+<th>Model Size</th>
+<th>KV Cache (2K ctx)</th>
+<th>Total Memory</th>
+<th>Memory Saving</th>
+</tr>
+<tr>
+<td><strong>FP16 / BF16</strong></td>
+<td>0.29 GB</td>
+<td>0.03 GB</td>
+<td><strong>0.32 GB</strong></td>
+<td>Baseline</td>
+</tr>
+<tr>
+<td><strong>INT8</strong></td>
+<td>0.15 GB</td>
+<td>0.03 GB</td>
+<td><strong>0.18 GB</strong></td>
+<td>~50% ↓</td>
+</tr>
+<tr>
+<td><strong>INT4 (NF4)</strong></td>
+<td>0.07 GB</td>
+<td>0.03 GB</td>
+<td><strong>0.10 GB</strong></td>
+<td>~75% ↓</td>
+</tr>
+</table>
+
+> 💡 **Note**: KV cache bertambah secara linear dengan panjang sequence. Untuk context 8K, kalikan nilai KV cache dengan 4.
+
+### Performance Estimates
+
+<table>
+<tr>
+<th>Metric</th>
+<th>Value</th>
+<th>Notes</th>
+</tr>
+<tr>
+<td><strong>FLOPs per Token</strong></td>
+<td>290,489,344</td>
+<td>Forward pass only</td>
+</tr>
+<tr>
+<td><strong>TFLOPs per Token</strong></td>
+<td>0.0003</td>
+<td>≈ 6× untuk backward</td>
+</tr>
+<tr>
+<td><strong>Bandwidth (FP16)</strong></td>
+<td>0.29 GB/token</td>
+<td>Memory bandwidth requirement</td>
+</tr>
+</table>
+
+---
+
+### 📐 Struktur Arsitektur Lengkap
+
+<details>
+<summary><b>🔍 Klik untuk lihat detail arsitektur</b></summary>
+
+```
+CacaForCausalLM (145.24M)
+│
+├─ Embedding: 16,000 × 640
+│
+├─ Transformer Layers (29x)
+│  ├─ RMSNorm
+│  ├─ Attention (GQA)
+│  │  ├─ Q: 10 heads × 64 dim
+│  │  ├─ KV: 2 heads × 64 dim
+│  │  ├─ RoPE (θ=10,000)
+│  │  └─ Flash Attention v2
+│  ├─ Residual
+│  ├─ RMSNorm
+│  ├─ FFN (SwiGLU)
+│  │  ├─ Gate: 640 → 1728
+│  │  ├─ Up: 640 → 1728
+│  │  └─ Down: 1728 → 640
+│  └─ Residual
+│
+├─ Final RMSNorm
+└─ LM Head: 640 → 16,000
+
+═══════════════════════════════════════════════════════════
+📊 PARAMETER BREAKDOWN:
+═══════════════════════════════════════════════════════════
+Embeddings:                10,240,000 (  7.1%)
+Transformer Layers:       124,723,200 ( 85.9%)
+  ├─ Attention:            28,508,160
+  └─ FFN:                  96,215,040
+Final Norm:                       640 (  0.0%)
+───────────────────────────────────────────────────────────
+TOTAL:                    145,244,672 (100.0%)
+═══════════════════════════════════════════════════════════
+```
+
+**Key Design Decisions:**
+
+1. **GQA over MHA**: Hemat 80% KV cache memory dengan minimal accuracy loss
+2. **SwiGLU over GELU**: ~10% better performance pada language modeling
+3. **RMSNorm over LayerNorm**: Lebih cepat & stabil, tanpa bias term
+4. **RoPE over Learned**: Better extrapolation untuk sequence length > training
+5. **No Bias in Linear**: Mengikuti modern LLM best practices (LLaMA-style)
+
+</details>
+
+---
+
+## 📚 Dokumentasi
+
+### 📦 Instalasi Dependencies
+
+```bash
+# Core dependencies (REQUIRED)
+pip install torch>=2.0.0 transformers>=4.35.0 accelerate safetensors
+
+# Optional: Untuk performa maksimal
+pip install flash-attn --no-build-isolation  # Flash Attention 2 (3x speedup)
+pip install xformers                          # Memory efficient attention
+pip install bitsandbytes                      # 4/8-bit quantization
+
+# Optional: Untuk monitoring & profiling
+pip install tensorboard wandb               # Training monitoring
+pip install gputil psutil                   # Resource monitoring
+```
+
+**Compatibility Matrix:**
+
+| Component | Version | Note |
+|-----------|---------|------|
+| Python | 3.8 - 3.11 | 3.11 recommended |
+| PyTorch | ≥ 2.0.0 | 2.1+ untuk SDPA optimal |
+| CUDA | 11.8 / 12.1 | Untuk Flash Attention |
+| Transformers | ≥ 4.35.0 | Untuk AutoModel support |
+
+### Cara Penggunaan
+
+#### 1️⃣ Basic Loading
+
+```python
+from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
+import torch
+
+# Load configuration
+config = AutoConfig.from_pretrained(
+    "Lyon28/caca-150M-untrained",
+    trust_remote_code=True
+)
+
+# Load model (FP16 untuk efisiensi)
+model = AutoModelForCausalLM.from_pretrained(
+    "Lyon28/caca-150M-untrained",
+    config=config,
+    trust_remote_code=True,
+    torch_dtype=torch.float16,
+    device_map="auto"  # Automatic device placement
+)
+
+# Model ini UNTRAINED - butuh training dulu!
+print(f"Model loaded: {model.num_parameters():,} parameters")
+print("⚠️  Model ini belum dilatih dan belum bisa digunakan untuk inference")
+```
+
+#### 2️⃣ Quantized Loading (4-bit/8-bit)
+
+```python
+from transformers import AutoModelForCausalLM, BitsAndBytesConfig
+import torch
+
+# 4-bit quantization config
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16,
+    bnb_4bit_use_double_quant=True
+)
+
+# Load model dengan quantization
+model = AutoModelForCausalLM.from_pretrained(
+    "Lyon28/caca-150M-untrained",
+    trust_remote_code=True,
+    quantization_config=bnb_config,
+    device_map="auto"
+)
+
+print(f"Memory footprint: ~0.07GB (4-bit)")
+```
+
+#### 3️⃣ Training Setup
+
+```python
+from transformers import TrainingArguments, Trainer
+
+# Training configuration
+training_args = TrainingArguments(
+    output_dir="./output",
+    per_device_train_batch_size=1,
+    gradient_accumulation_steps=16,
+    learning_rate=2e-4,
+    max_steps=10000,
+    lr_scheduler_type="cosine",
+    warmup_steps=500,
+    logging_steps=10,
+    save_steps=500,
+    fp16=True,  # Mixed precision
+    gradient_checkpointing=True,  # Memory efficient
+)
+
+# Initialize trainer
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=train_dataset,
+)
+
+# Start training
+trainer.train()
+```
+
+### Advanced Usage
+
+#### Gradient Checkpointing (Memory Efficient)
+
+```python
+model.gradient_checkpointing_enable()
+print("✅ Gradient checkpointing enabled - saves ~40% memory")
+```
+
+#### Custom Training Loop
+
+```python
+from torch.optim import AdamW
+from torch.cuda.amp import autocast, GradScaler
+
+optimizer = AdamW(model.parameters(), lr=2e-4)
+scaler = GradScaler()
+
+for batch in dataloader:
+    # Mixed precision forward
+    with autocast(dtype=torch.bfloat16):
+        outputs = model(**batch)
+        loss = outputs.loss
+
+    # Backward with gradient scaling
+    scaler.scale(loss).backward()
+    scaler.step(optimizer)
+    scaler.update()
+    optimizer.zero_grad()
+```
+
+#### Multi-GPU Training (DDP)
+
+```python
+import torch.distributed as dist
+from torch.nn.parallel import DistributedDataParallel
+
+# Initialize process group
+dist.init_process_group(backend="nccl")
+
+# Wrap model
+model = DistributedDataParallel(
+    model,
+    device_ids=[local_rank],
+    find_unused_parameters=False
+)
+```
+
+---
+
+## ⚙️ Konfigurasi Detail
+
+### Full Configuration JSON
+
+```json
+{
+  "architectures": ["CacaForCausalLM"],
+  "model_type": "caca",
+  "vocab_size": 16000,
+  "hidden_size": 640,
+  "intermediate_size": 1728,
+  "num_hidden_layers": 29,
+  "num_attention_heads": 10,
+  "num_key_value_heads": 2,
+  "head_dim": 64,
+  "max_position_embeddings": 2048,
+  "rope_theta": 10000,
+  "rms_norm_eps": 1e-06,
+  "use_cache": true,
+  "use_qk_norm": true,
+  "use_flash_attn": true,
+  "attention_dropout": 0.0,
+  "hidden_dropout": 0.1,
+  "torch_dtype": "float16"
+}
+```
+
+### Custom Configuration
+
+```python
+from transformers import AutoConfig
+
+# Load dan modifikasi config
+config = AutoConfig.from_pretrained("Lyon28/caca-150M-untrained")
+
+# Custom modifications
+config.max_position_embeddings = 16384  # Extend context
+config.rope_scaling = {"type": "linear", "factor": 2.0}
+config.use_flash_attn = True
+config.hidden_dropout = 0.05
+
+# Save custom config
+config.save_pretrained("./custom_config")
+```
+
+---
+
+## 🔬 Arsitektur
+
+### Layer Structure
+
+```
+Input Tokens
+    ↓
+Embedding Layer (16,000 → 640)
+    ↓
+┌─────────────────────────────────────┐
+│ Decoder Block × 29                  │
+│                                     │
+│  ┌─ RMSNorm                        │
+│  ├─ Multi-Head Attention (GQA)     │
+│  │  - Flash Attention v2           │
+│  │  - 10 Query heads, 2 KV heads       │
+│  │  - RoPE position encoding       │
+│  ├─ Residual Connection            │
+│  │                                  │
+│  ├─ RMSNorm                        │
+│  ├─ Feed-Forward Network (SwiGLU)  │
+│  │  - Gate: 640 → 1728     │
+│  │  - Up:   640 → 1728     │
+│  │  - Down: 1728 → 640     │
+│  └─ Residual Connection            │
+│                                     │
+└─────────────────────────────────────┘
+    ↓
+RMSNorm (Final)
+    ↓
+LM Head (640 → 16,000)
+    ↓
+Output Logits
+```
+
+### Attention Mechanism (GQA)
+
+```
+Query:  [10 heads × 64 dim] = 640
+Key:    [2 heads × 64 dim] = 128
+Value:  [2 heads × 64 dim] = 128
+
+Grouped Query Attention:
+- Setiap 5 query heads berbagi 1 KV head
+- Memory KV cache: 80% lebih kecil dari Multi-Head Attention
+- Kualitas mendekati MHA, speed mendekati MQA
+```
+
+### Feed-Forward Network (SwiGLU)
+
+```
+FFN(x) = (SiLU(xW_gate) ⊙ xW_up) W_down
+
+Where:
+- W_gate: 640 × 1728
+- W_up:   640 × 1728
+- W_down: 1728 × 640
+- SiLU(x) = x · sigmoid(x)
+- ⊙ = element-wise multiplication
+```
+
+## 💬 Format Chat & Prompt Engineering
+
+### 📝 Chat Template
+
+Model mendukung format chat standar untuk conversational AI:
+
+```python
+# Format chat template bawaan
+chat_template = """
+{% for message in messages %}
+{% if message['role'] == 'system' %}
+System: {{ message['content'] }}
+
+{% elif message['role'] == 'user' %}
+User: {{ message['content'] }}
+
+{% elif message['role'] == 'assistant' %}
+Assistant: {{ message['content'] }}
+
+{% endif %}
+{% endfor %}
+{% if add_generation_prompt %}Assistant:{% endif %}
+"""
+
+# Contoh penggunaan
+messages = [
+    {"role": "system", "content": "Kamu adalah asisten AI yang membantu dan ramah."},
+    {"role": "user", "content": "Jelaskan tentang fotosintesis"},
+    {"role": "assistant", "content": "Fotosintesis adalah proses di mana tumbuhan mengubah cahaya matahari menjadi energi kimia..."},
+    {"role": "user", "content": "Apa manfaatnya bagi manusia?"},
+]
+
+# Apply template
+formatted = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+
+print(formatted)
+# Output:
+# System: Kamu adalah asisten AI yang membantu dan ramah.
+#
+# User: Jelaskan tentang fotosintesis
+# Assistant: Fotosintesis adalah proses di mana tumbuhan...
+# User: Apa manfaatnya bagi manusia?
+# Assistant:
+```
+
+---
+
+## 🎯 Use Cases
+
+Model ini dirancang untuk berbagai aplikasi NLP setelah melalui proses training:
+
+### Text Generation
+- ✍️ Creative writing & storytelling
+- 📰 Article generation
+- 💬 Conversational AI
+- 🔄 Text completion
+
+### Language Understanding
+- 📊 Text classification
+- 🏷️ Named Entity Recognition (NER)
+- ❓ Question Answering
+- 📝 Summarization
+
+### Code Generation
+- 💻 Code completion
+- 🐛 Bug fixing suggestions
+- 📚 Documentation generation
+- 🔄 Code translation
+
+### Multilingual Tasks
+- 🌏 Translation (ID ↔ EN)
+- 🗣️ Cross-lingual understanding
+- 🌐 Multilingual classification
+
+---
+
+## 📈 Benchmark & Evaluation
+
+> ⚠️ Model belum melalui evaluasi karena status untrained
+
+Setelah training, model akan dievaluasi pada:
+
+### Indonesian Benchmarks
+- **IndoNLU**: Comprehensive Indonesian NLU tasks
+- **IndoQA**: Indonesian Question Answering
+- **IndoSum**: Summarization
+- **IndoNER**: Named Entity Recognition
+
+### Multilingual Benchmarks
+- **MMLU**: Massive Multitask Language Understanding
+- **HellaSwag**: Common sense reasoning
+- **ARC**: Science QA
+- **TruthfulQA**: Truthfulness evaluation
+
+### Generation Quality
+- **Perplexity**: Language modeling quality
+- **BLEU/ROUGE**: Translation & summarization
+- **Human Evaluation**: Fluency, coherence, factuality
+
+---
+
+## 🛠️ Development & Training Tips
+
+### Optimal Batch Size
+
+```python
+# Rule of thumb untuk 145.24M model
+# GPU Memory → Batch size per device
+
+if gpu_memory >= 80:  # A100 80GB
+    batch_size = 110
+    gradient_accumulation = 1
+elif gpu_memory >= 40:  # A100 40GB
+    batch_size = 55
+    gradient_accumulation = 1
+elif gpu_memory >= 24:  # RTX 3090/4090
+    batch_size = 1
+    gradient_accumulation = 1
+
+# Effective batch size = batch_size × gradient_accumulation × num_gpus
+```
+
+### Learning Rate Scheduling
+
+```python
+# Recommended untuk 145.24M model
+learning_rate = 0.0005  # Base LR
+warmup_ratio = 0.05  # 5% of total steps
+lr_scheduler = "cosine"  # atau "linear"
+
+# Learning rate scaling rule:
+# LR ∝ sqrt(batch_size)
+# Untuk batch size 256: LR = 0.0005
+# Untuk batch size 512: LR = 7.07e-04
+```
+
+### Gradient Clipping
+
+```python
+# Prevent gradient explosion
+max_grad_norm = 1.0  # Clip at 1.0
+
+# Monitor gradients
+from torch.nn.utils import clip_grad_norm_
+
+grad_norm = clip_grad_norm_(model.parameters(), max_grad_norm)
+if grad_norm > 10.0:
+    print(f"⚠️ High gradient norm: {grad_norm:.2f}")
+```
+
+### Training Stability
+
+```python
+# Tips untuk stable training:
+
+1. **Warmup**: Mulai dengan LR rendah
+2. **Gradient Checkpointing**: Kurangi memory footprint
+3. **Mixed Precision**: Gunakan BF16 jika tersedia (lebih stable dari FP16)
+4. **Batch Size**: Start small, increase gradually
+5. **Monitor**: Track loss, perplexity, gradient norms
+```
+
+---
+
+## 🔧 Troubleshooting
+
+### Out of Memory (OOM)
+
+```python
+# Solusi OOM saat training:
+
+✅ 1. Enable gradient checkpointing
+model.gradient_checkpointing_enable()
+
+✅ 2. Reduce batch size
+per_device_train_batch_size = 1
+
+✅ 3. Increase gradient accumulation
+gradient_accumulation_steps = 32
+
+✅ 4. Use quantization
+load_in_8bit = True  # atau load_in_4bit
+
+✅ 5. Reduce sequence length
+max_length = 2048  # Start dengan ini
+
+✅ 6. CPU offloading (jika perlu)
+device_map = "auto"
+offload_folder = "offload"
+```
+
+### Slow Training
+
+```python
+# Optimasi kecepatan training:
+
+✅ 1. Flash Attention
+config.use_flash_attn = True  # 2-3x speedup
+
+✅ 2. Compile model (PyTorch 2.0+)
+model = torch.compile(model, mode="reduce-overhead")
+
+✅ 3. DataLoader optimization
+dataloader = DataLoader(
+    dataset,
+    batch_size=batch_size,
+    num_workers=4,  # Parallel data loading
+    pin_memory=True,  # Faster GPU transfer
+    prefetch_factor=2
+)
+
+✅ 4. Mixed precision
+use_fp16 = True  # atau bf16
+
+✅ 5. Optimize communication (multi-GPU)
+find_unused_parameters = False
+gradient_as_bucket_view = True
+```
+
+### NaN Loss
+
+```python
+# Jika loss menjadi NaN:
+
+✅ 1. Reduce learning rate
+learning_rate = learning_rate * 0.1
+
+✅ 2. Check gradient norms
+clip_grad_norm_(model.parameters(), 1.0)
+
+✅ 3. Use BF16 instead of FP16
+torch_dtype = torch.bfloat16  # Lebih stable
+
+✅ 4. Add epsilon to RMSNorm
+rms_norm_eps = 1e-5  # Increase jika perlu
+
+✅ 5. Check data
+# Pastikan tidak ada inf/nan di dataset
+assert not torch.isnan(input_ids).any()
+assert not torch.isinf(attention_mask).any()
+```
+
+---
+
+### 🚫 Prohibited Uses
+
+<div style="background: #ffebee; border-left: 4px solid #f44336; padding: 12px; margin: 16px 0;">
+
+Model ini **TIDAK BOLEH** digunakan untuk:
+
+- 🚫 **Harmful content generation** (violence, self-harm, illegal acts)
+- 🚫 **Misinformation/disinformation campaigns**
+- 🚫 **Harassment or hate speech**
+- 🚫 **Impersonation or identity theft**
+- 🚫 **Child safety violations** (CSAM, grooming, exploitation)
+- 🚫 **Privacy violations** (doxxing, stalking, surveillance abuse)
+- 🚫 **Malicious code generation** (malware, exploits, etc)
+- 🚫 **Spam or manipulation** (fake reviews, astroturfing)
+- 🚫 **Medical/legal advice** (tanpa disclaimer & expert review)
+- 🚫 **Financial fraud** (scams, market manipulation)
+
+**Violation consequences:** Model access revocation + legal action jika applicable
+
+</div>
+
+---
+
+## 📜 License & Citation
+
+### 📄 License
+
+<div style="background: #e8f5e9; border-left: 4px solid #4caf50; padding: 12px; margin: 16px 0;">
+
+Model ini dirilis di bawah **Apache License 2.0**
+
+✅ **Anda BEBAS untuk:**
+- ✔️ Gunakan secara komersial
+- ✔️ Modifikasi sesuka hati
+- ✔️ Distribusi ulang
+- ✔️ Patent use
+- ✔️ Private use
+
+⚠️ **Dengan syarat:**
+- 📄 Include license & copyright notice
+- 📝 State changes yang dibuat
+- 📋 Disclaimer of warranty
+
+❌ **Tanpa jaminan apapun** (use at your own risk)
+
+</div>
+
+**Full license text**: [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+## 📖 Citation
+
+Jika Anda menggunakan model ini dalam penelitian, mohon sitasi:
+
+```bibtex
+@misc{cacacaca150m,
+  author = {Lyon},
+  title = {Caca-caca-150M: Modern Transformer Architecture with Grouped Query Attention},
+  year = {2026},
+  publisher = {Hugging Face},
+  journal = {Hugging Face Model Hub},
+  howpublished = {\url{https://huggingface.co/Lyon28/caca-150M-untrained}},
+  note = {Untrained model with 145,244,672 parameters}
+}
+```
+
+**APA Style:**
+```
+Lyon. (2026). Caca-caca-150M: Modern Transformer Architecture with Grouped
+Query Attention [Untrained model]. Hugging Face.
+https://huggingface.co/Lyon28/caca-150M-untrained
+```
+
+**MLA Style:**
+```
+Lyon. "Caca-caca-150M: Modern Transformer Architecture with Grouped Query Attention."
+Hugging Face, 2026, huggingface.co/Lyon28/caca-150M-untrained.
+```
+
+---
+
+### 🙏 Acknowledgments
+
+Model ini berdiri di pundak para raksasa! Terima kasih kepada:
+
+<details>
+<summary><b>🏛️ Klik untuk daftar lengkap acknowledgments</b></summary>
+
+#### 🏗️ **Core Architecture**
+- **LLaMA/LLaMA 2** (Meta AI, 2023) - Decoder-only architecture, RMSNorm, SwiGLU
+  - Paper: [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971)
+  - Authors: Hugo Touvron et al.
+- **GPT-3** (OpenAI, 2020) - Transformer language modeling paradigm
+- **PaLM** (Google, 2022) - SwiGLU activation insights
+
+#### 🎯 **Attention Mechanisms**
+- **Flash Attention v2** (Tri Dao et al., Stanford, 2023)
+  - Paper: [FlashAttention-2: Faster Attention with Better Parallelism](https://arxiv.org/abs/2307.08691)
+  - 3x speedup dengan IO-aware algorithm
+- **Grouped Query Attention** (Joshua Ainslie et al., Google, 2023)
+  - Paper: [GQA: Training Generalized Multi-Query Transformer](https://arxiv.org/abs/2305.13245)
+  - Memory-efficient KV cache
+- **Multi-Query Attention** (Noam Shazeer, Google, 2019)
+  - Fast inference dengan shared K/V
+- **xFormers** (Meta AI, 2022) - Memory efficient attention
+- **PyTorch SDPA** (PyTorch Team, 2023) - Native attention optimization
+
+#### 📍 **Position Encodings**
+- **RoPE** (Jianlin Su et al., EleutherAI, 2021)
+  - Paper: [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/abs/2104.09864)
+  - Superior length extrapolation
+- **ALiBI** (Ofir Press et al., 2022)
+  - Paper: [Train Short, Test Long: Attention with Linear Biases](https://arxiv.org/abs/2108.12409)
+  - Length generalization without retraining
+- **YaRN** (Bowen Peng et al., 2023)
+  - Paper: [YaRN: Efficient Context Window Extension](https://arxiv.org/abs/2309.00071)
+
+#### 🪟 **Long Context & Efficiency**
+- **Sliding Window Attention** (Albert Gu et al., Mistral AI, 2023)
+  - Paper: [Mistral 7B](https://arxiv.org/abs/2310.06825)
+- **StreamingLLM** (Guangxuan Xiao et al., MIT, 2023)
+  - Paper: [Efficient Streaming Language Models with Attention Sinks](https://arxiv.org/abs/2309.17453)
+  - Infinite sequence length!
+- **Logit Softcapping** (Google Gemma Team, 2024)
+  - Paper: [Gemma: Open Models Based on Gemini](https://arxiv.org/abs/2403.08295)
+
+#### 🧠 **Mixture of Experts**
+- **Mixtral 8x7B** (Albert Jiang et al., Mistral AI, 2024)
+  - Paper: [Mixtral of Experts](https://arxiv.org/abs/2401.04088)
+  - State-of-the-art sparse MoE
+- **Switch Transformers** (William Fedus et al., Google, 2021)
+  - Paper: [Switch Transformers: Scaling to Trillion Parameter Models](https://arxiv.org/abs/2101.03961)
+  - Expert scaling insights
+- **GLaM** (Nan Du et al., Google, 2021) - Generalist Language Model
+- **Expert Choice Routing** (Yanqi Zhou et al., Google, 2022)
+  - Better load balancing
+
+#### 🎓 **Training Optimizations**
+- **Layer Scale** (Hugo Touvron et al., Meta, 2021)
+  - Paper: [Going Deeper with Image Transformers](https://arxiv.org/abs/2103.17239)
+  - Training stability untuk deep networks
+- **Stochastic Depth** (Gao Huang et al., 2016)
+  - Paper: [Deep Networks with Stochastic Depth](https://arxiv.org/abs/1603.09382)
+- **Mixture of Depths** (David Raposo et al., DeepMind, 2024)
+  - Paper: [Mixture-of-Depths: Dynamically allocating compute](https://arxiv.org/abs/2404.02258)
+  - Dynamic compute allocation
+- **Gradient Checkpointing** (Tianqi Chen et al., 2016)
+
+#### 📦 **Quantization**
+- **LLM.int8()** (Tim Dettmers et al., 2022)
+  - Paper: [LLM.int8(): 8-bit Matrix Multiplication for Transformers](https://arxiv.org/abs/2208.07339)
+- **QLoRA** (Tim Dettmers et al., 2023)
+  - Paper: [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)
+  - 4-bit efficient fine-tuning
+- **bitsandbytes** (Tim Dettmers) - Quantization library
+
+#### 🎨 **Multimodal**
+- **Vision Transformer** (Alexey Dosovitskiy et al., Google, 2020)
+  - Paper: [An Image is Worth 16x16 Words](https://arxiv.org/abs/2010.11929)
+- **Flamingo** (Jean-Baptiste Alayrac et al., DeepMind, 2022)
+  - Paper: [Flamingo: a Visual Language Model](https://arxiv.org/abs/2204.14198)
+  - Perceiver Resampler
+- **BLIP-2** (Junnan Li et al., Salesforce, 2023)
+  - Paper: [BLIP-2: Bootstrapping Language-Image Pre-training](https://arxiv.org/abs/2301.12597)
+  - Q-Former architecture
+- **Whisper** (Alec Radford et al., OpenAI, 2022) - Audio encoding
+
+#### 🛠️ **Normalization & Activations**
+- **RMSNorm** (Biao Zhang, Rico Sennrich, 2019)
+  - Paper: [Root Mean Square Layer Normalization](https://arxiv.org/abs/1910.07467)
+- **SwiGLU** (Noam Shazeer, Google, 2020)
+  - Paper: [GLU Variants Improve Transformer](https://arxiv.org/abs/2002.05202)
+
+#### 🔧 **Tools & Frameworks**
+- **🤗 Hugging Face** - Transformers, Accelerate, PEFT
+  - Making NLP accessible to everyone
+- **PyTorch** - Deep learning framework
+  - Facebook AI Research team
+- **Safetensors** - Secure serialization
+  - Hugging Face team
+- **DeepSpeed** - Distributed training
+  - Microsoft Research
+- **Flash Attention Implementation** - Tri Dao & team
+
+#### 🇮🇩 **Indonesian NLP Community**
+Special thanks to Indonesian NLP researchers & practitioners yang telah membangun foundation untuk Indonesian language AI.
+
+</details>
+
+---
+
+## 📄 License
+
+Model ini dirilis di bawah **Apache License 2.0**.
+
+### Ketentuan Penggunaan:
+- ✅ **Bebas digunakan** untuk keperluan komersial dan non-komersial
+- ✅ **Modifikasi** diperbolehkan
+- ✅ **Distribusi** diperbolehkan dengan attribution
+- ⚠️ **No Warranty** - model disediakan "as is"
+- 📝 **Attribution Required** - sertakan copyright notice
+
+Lihat [LICENSE](LICENSE) untuk detail lengkap.
+
+---
+
+## 🤝 Contributing
+
+Kami sangat terbuka untuk kontribusi! Berikut cara Anda bisa berkontribusi:
+
+### Training & Fine-tuning
+- 🎓 Train model ini dengan dataset Anda
+- 📊 Share benchmark results
+- 🔬 Experiment dengan hyperparameters
+
+### Code & Architecture
+- 🐛 Report bugs atau issues
+- 💡 Suggest improvements
+- 🔧 Submit pull requests
+
+### Documentation
+- 📚 Improve documentation
+- 🌐 Add translations
+- ✍️ Write tutorials & guides
+
+### Dataset & Evaluation
+- 📝 Contribute training data
+- 🧪 Create evaluation benchmarks
+- 🎯 Share fine-tuned versions
+
+---
+
+## 👥 Team & Acknowledgments
+
+### Core Team
+- **LyonPoy** - Architecture design & implementation
+
+### Special Thanks
+- 🤗 **Hugging Face** - Infrastructure & community
+- ⚡ **FlashAttention Team** - Efficient attention implementation
+- 🧠 **Anthropic, Google, Meta** - Research inspirations
+
+### Community
+Terima kasih kepada komunitas open-source yang telah berkontribusi pada:
+- Transformers library
+- PyTorch framework
+- Datasets & evaluation tools
+
+---
+
+## 📞 Contact & Support
+
+### Community
+- 💬 [Discussions](https://huggingface.co/Lyon28/caca-150M-untrained/discussions) - Ask questions
+- 🐛 [Issues](https://github.com/lyon28/caca-transformers/issues) - Report bugs
+- 📧 Email : cacatransformers@gmail.com
+
+---
+
+## 🌟 Star History
+
+<div align="center">
+
+[![Star History Chart](https://api.star-history.com/svg?repos=Lyon-28/caca-transformers&type=Date)](https://star-history.com/#Lyon-28/caca-transformers&Date)
+
+</div>
+
+## 💝 Dibuat dengan ❤️ untuk Komunitas AI Indonesia
+
+<img src="https://i.postimg.cc/MTSj073X/logo.png" width="200" alt="Caca Logo"/>
+
+### **Terima kasih telah menggunakan Caca!**
+
+Jika model ini berguna, jangan lupa ⭐ repository kami!
+
+<div align="center">
+
+<table>
+<tr>
+<td align="center">⭐<br/><b>Star Repo</b><br/><sub>Show your support</sub></td>
+<td align="center">🔗<br/><b>Share</b><br/><sub>Tell your friends</sub></td>
+<td align="center">💬<br/><b>Join Discussion</b><br/><sub>Ask questions</sub></td>
+<td align="center">🤝<br/><b>Contribute</b><br/><sub>Make it better</sub></td>
+</tr>
+</table>
+
+### 🚀 Happy Training! 🚀
+
+**Model ini menunggu untuk dilatih dan menjadi foundation untuk aplikasi AI Anda.**
+
+[📥 Download Model](#) • [📖 Read Docs](https://caca-transformers.ai) • [💬 Join Community](https://discord.gg/cacatransformers)
+
+</div>
+
+---
+
+### 📊 Model Statistics
+
+<img src="https://img.shields.io/badge/Parameters-145.24M-blue?style=for-the-badge" alt="Parameters"/>
+<img src="https://img.shields.io/badge/Status-Untrained-orange?style=for-the-badge" alt="Status"/>
+<img src="https://img.shields.io/badge/License-Apache%202.0-green?style=for-the-badge" alt="License"/>
+
+<img src="https://img.shields.io/badge/Architecture-Transformer-purple?style=for-the-badge" alt="Architecture"/>
+<img src="https://img.shields.io/badge/Type-Causal%20LM-red?style=for-the-badge" alt="Type"/>
+<img src="https://img.shields.io/badge/Context-2,048%20tokens-cyan?style=for-the-badge" alt="Context"/>
+
+---
+
+### 🎨 Daily Inspiration
+
+<div align="center">
+  <img src="https://quotes-caca.vercel.app/api/SsQuote" alt="Daily Quote" width="600" />
+</div>
+
+---
+
+### 📈 Quick Stats
+
+| Metric | Value |
+|--------|-------|
+| 💎 Total Parameters | 145,244,672 |
+| 🏗️ Layers | 29 |
+| 🎯 Attention Heads | 10 |
+| 📖 Max Context | 2,048 tokens |
+| 💾 Size (FP16) | 0.29 GB |
+| 💾 Size (INT4) | 0.07 GB |
+
+---
+
+<sub>
+Model ini adalah bagian dari <b>Caca Project</b> - Open source initiative untuk membangun Indonesian LLM ecosystem.<br/>
+Created with 💻 by <a href="https://huggingface.co/Lyon28">@Lyon28</a> |
+Licensed under <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache 2.0</a> |
+Built with <a href="https://huggingface.co">🤗 HuggingFace</a>
+</sub>
+
+<br/><br/>
+
+**🌟 "Dari nol, untuk semua" 🌟**
+
+<sub>Last updated: january 2026</sub>
+
+</div>
+
+---
+
+<div align="center">
+<sub>Built with ❤️ by Caca Transformers Team</sub><br>
+<sub>Powered by 🤗 Transformers • ⚡ PyTorch • 🔥 Flash Attention</sub>
+</div>

config.json

@@ -0,0 +1,107 @@
+{
+  "architectures": [
+    "CacaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attention_pattern": "all_global",
+  "attention_sink_size": 4,
+  "attention_sink_window": 1024,
+  "attention_temperature": 1.0,
+  "attn_logit_softcapping": null,
+  "audio_config": {},
+  "auto_map": {
+    "AutoConfig": "caca_transformers.CacaConfig",
+    "AutoModel": "caca_transformers.CacaModel",
+    "AutoModelForCausalLM": "caca_transformers.CacaForCausalLM"
+  },
+  "bos_token_id": 1,
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}System: {{ message['content'] }}\n{% elif message['role'] == 'user' %}User: {{ message['content'] }}\n{% elif message['role'] == 'assistant' %}Assistant: {{ message['content'] }}\n{% endif %}{% endfor %}{% if add_generation_prompt %}Assistant:{% endif %}",
+  "cross_attention_frequency": 4,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "expert_capacity_factor": 1.0,
+  "expert_choice_k": 0.125,
+  "expert_load_monitoring": true,
+  "expert_load_warn_threshold": 0.3,
+  "final_logit_softcapping": null,
+  "global_attention_every_n_layers": 2,
+  "gradient_checkpointing_granularity": "full",
+  "head_dim": 64,
+  "hidden_dropout": 0.1,
+  "hidden_size": 640,
+  "initializer_range": 0.02,
+  "intermediate_size": 1728,
+  "label_smoothing": 0.0,
+  "layer_scale_init": 1e-05,
+  "lm_logit_softcapping": 30.0,
+  "longformer_attention_window": 512,
+  "lora_alpha": 32.0,
+  "lora_bias": "none",
+  "lora_dropout": 0.05,
+  "lora_rank": 16,
+  "lora_target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "max_position_embeddings": 2048,
+  "merge_threshold": 0.5,
+  "mlp_bias": false,
+  "mod_capacity_factor": 0.5,
+  "mod_route_method": "learned",
+  "model_type": "caca",
+  "moe_layer_frequency": 2,
+  "mup_base_width": 256,
+  "nan_recovery_level": 1,
+  "num_attention_heads": 10,
+  "num_expert_groups": 1,
+  "num_experts": 8,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 29,
+  "num_key_value_heads": 2,
+  "pad_token_id": null,
+  "pipeline_parallel_size": 1,
+  "pretraining_tp": 1,
+  "projector_hidden_size": 640,
+  "qk_norm_eps": 1e-06,
+  "residual_dropout": 0.1,
+  "rms_norm_eps": 1e-06,
+  "rope_ntk_alpha": 1.0,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "rope_theta": 10000.0,
+  "rope_type": "default",
+  "router_aux_loss_coef": 0.01,
+  "router_z_loss_coef": 0.001,
+  "sliding_window": null,
+  "stochastic_depth_prob": 0.1,
+  "tensor_parallel_size": 1,
+  "tie_word_embeddings": false,
+  "token_dropout": 0.0,
+  "transformers_version": "5.14.1",
+  "use_alibi": false,
+  "use_attention_sink": false,
+  "use_cache": true,
+  "use_cross_attention": false,
+  "use_expert_choice": false,
+  "use_flash_attn": true,
+  "use_grouped_moe": false,
+  "use_grouped_query_attention": false,
+  "use_layer_scale": false,
+  "use_longformer_attention": false,
+  "use_lora": false,
+  "use_mixture_of_depths": false,
+  "use_moe": false,
+  "use_multi_query_attention": false,
+  "use_multimodal": false,
+  "use_mup": false,
+  "use_qk_norm": true,
+  "use_rotary_embeddings": true,
+  "use_soft_merging": false,
+  "use_spectral_norm": false,
+  "use_stochastic_depth": false,
+  "vision_config": {},
+  "vocab_size": 16000
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.14.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c916e68026851198b7991d66cd4a2d9079cb11dd5b0c15e4e7e9f7c316f352fd
+size 581014360

training_stats.json

@@ -0,0 +1,179 @@
+{
+  "metrics": {},
+  "cache_stats": {
+    "layer_0": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_1": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_2": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_3": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_4": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_5": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_6": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_7": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_8": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_9": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_10": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_11": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_12": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_13": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_14": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_15": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_16": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_17": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_18": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_19": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_20": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_21": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_22": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_23": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_24": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_25": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_26": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_27": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    },
+    "layer_28": {
+      "hits": 0,
+      "misses": 0,
+      "hit_rate": 0,
+      "cache_size": 0
+    }
+  }
+}