README.md

---
license: apache-2.0
language:
- id
- en
tags:
- text-generation
- pytorch
- causal-lm
- transformer
- untrained
- mla
- multi-token-prediction
- qk-norm
- rope
- yarn
- swiglu
- rmsnorm
- sliding-window-attention
- indonesian
- bilingual
library_name: transformers
pipeline_tag: text-generation
widget:
  - text: "Jakarta adalah ibu kota"
    example_title: "🇮🇩 Pelengkapan Teks (ID)"
  - text: |
      Pertanyaan: Apa itu kecerdasan buatan?
      Jawaban:
    example_title: "🇮🇩 Tanya Jawab (ID)"
  - text: "The capital of Indonesia is"
    example_title: "🇬🇧 Text Completion (EN)"
  - text: |
      User: Halo! Siapa kamu?
      Assistant:
    example_title: "💬 Format Chat (ID)"
inference:
  parameters:
    max_new_tokens: 100
    temperature: 0.7
    top_p: 0.9
    top_k: 50
    do_sample: true
    repetition_penalty: 1.1
datasets: []
metrics:
- perplexity
model-index:
- name: caca-650M-untrained
  results: []
---

<div align="center">

<img src="https://i.postimg.cc/MTSj073X/logo.png" width="400" alt="caca-650M-untrained"/>

# 🤖 caca-650M-untrained

### Arsitektur Transformer dengan Multi-head Latent Attention (MLA)

![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)
![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)
![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-red.svg)
![Transformers](https://img.shields.io/badge/🤗%20Transformers-4.35+-yellow.svg)
![Model Type](https://img.shields.io/badge/Model-Causal%20LM-green.svg)
![Parameters](https://img.shields.io/badge/Parameters-527M-orange.svg)
<img src="https://img.shields.io/badge/Status-Untrained%20(Weights%20Ready)-orange?style=for-the-badge" alt="Status"/>

**526,860,267** parameters • **527M** • **20 layers** • **8,192 tokens** (efektif **32,768** dgn YaRN)

[📚 Dokumentasi](#-dokumentasi) • [💻 Usage](#-cara-penggunaan) • [⚙️ Konfigurasi](#-spesifikasi-model) • [🔬 Arsitektur](#-struktur-arsitektur-lengkap)

</div>

---

## ⚠️ PENTING: Model Belum Dilatih (Untrained)

<div style="background:#fff3cd;border-left:4px solid #ffc107;padding:12px;margin:16px 0;"><strong>⚠️ PERHATIAN</strong>: Bobot model ini adalah <strong>random initialization</strong> (belum melalui proses training apa pun). Output yang dihasilkan akan <strong>tidak bermakna dan acak</strong>.</div>

**Status Model:**
- 🔴 **Belum dilatih** — Bobot masih random init (`initializer_range=0.02`)
- 🟡 **Untuk riset & eksperimen** — Arsitektur sudah siap, tinggal train dari nol
- 🟢 **Base architecture release** — Titik awal pretraining

| ✅ Bisa | ❌ Belum Bisa |
|---------|----------------|
| Load model architecture | Generate teks bermakna |
| Test forward pass | Menjawab pertanyaan |
| Measure memory & speed | Reasoning & understanding |
| Mulai pretraining dari nol | Production deployment |

---

## 📋 Deskripsi

**Caca** adalah arsitektur Large Language Model (LLM) yang menggabungkan **Multi-head Latent Attention (MLA)** ala DeepSeek-V2/V3, **Multi-Token Prediction (MTP)**, **QK-Normalization**, dan **sliding window attention** berselang-seling — dirancang untuk efisiensi KV cache dan skalabilitas dari model kecil (~650M) sampai sangat besar (1T+).

<blockquote style="border-left:4px solid #4A90E2;padding-left:16px;margin:16px 0;background:#f8f9fa;padding:12px;">
<p><strong>📖 Tentang Project Caca</strong></p>
<p><em>Caca</em> adalah eksperimen open-source Indonesian LLM yang dibuat dari nol secara individual dan bertahap. Bukan kompetitor siapa-siapa, cuma pengen eksplorasi apa yang bisa dilakukan dengan budget terbatas, passion unlimited, dan mindset collaborative.</p>
<p>Kalau berguna buat orang lain, alhamdulillah. Kalau enggak, ya tetap fun kok.</p>
<p>— <strong>Lyon</strong>, Creator</p>
</blockquote>

### 🌟 Mengapa Caca?

1. **🇮🇩 Fokus pada Bahasa Indonesia** — dirancang bilingual ID/EN
2. **💾 KV Cache Sangat Hemat** — MLA menekan KV cache ~81% dibanding attention standar setara
3. **🎯 Multi-Token Prediction** — prediksi 1 token ke depan sekaligus, potensi throughput lebih baik
4. **📏 Long Context via YaRN** — RoPE scaling dari 8,192 ke 32,768 token
5. **🪟 Sliding Window Berselang-seling** — 10 full-attention layer + 10 sliding-window layer

---

## 📊 Spesifikasi Model

<table>
<tr><td><strong>Parameter</strong></td><td><strong>Value</strong></td><td><strong>Parameter</strong></td><td><strong>Value</strong></td></tr>
<tr><td>Total Parameters</td><td><code>526,860,267</code></td><td>Vocab Size</td><td><code>32,000</code></td></tr>
<tr><td>Hidden Size</td><td><code>1280</code></td><td>Intermediate Size</td><td><code>5120</code></td></tr>
<tr><td>Num Layers</td><td><code>20</code></td><td>Attention Heads</td><td><code>10</code></td></tr>
<tr><td>Max Context Length</td><td><code>8,192</code></td><td>Efektif (YaRN)</td><td><code>32,768</code></td></tr>
<tr><td>RoPE Base (θ)</td><td><code>10,000</code></td><td>Tie Embeddings</td><td><code>True</code></td></tr>
<tr><td>Model Size (FP16)</td><td><code>1.05 GB</code></td><td>Formatted Size</td><td><code>527M</code></td></tr>
</table>

---

## 💾 Kebutuhan Memory

### Training Requirements

<table>
<tr><th>Configuration</th><th>Model Weights</th><th>+ Optimizer States</th><th>Total Training</th></tr>
<tr><td><strong>FP32 (AdamW)</strong></td><td>2.11 GB</td><td>+6.32 GB</td><td><strong>8.43 GB</strong></td></tr>
<tr><td><strong>Mixed Precision</strong></td><td>1.05 GB</td><td>+7.38 GB</td><td><strong>8.43 GB</strong></td></tr>
<tr><td><strong>+ Gradient Checkpointing</strong></td><td colspan="2">Menghemat ~30-50% activation memory</td><td><strong>~5.06 GB</strong></td></tr>
</table>

### Inference Requirements (MLA KV Cache)

<table>
<tr><th>Precision</th><th>Model Size</th><th>KV Cache (2K ctx)</th><th>Total Memory</th></tr>
<tr><td><strong>FP16 / BF16</strong></td><td>1.05 GB</td><td>0.020 GB</td><td><strong>1.07 GB</strong></td></tr>
<tr><td><strong>INT8</strong></td><td>0.53 GB</td><td>0.020 GB</td><td><strong>0.55 GB</strong></td></tr>
<tr><td><strong>INT4 (NF4)</strong></td><td>0.26 GB</td><td>0.020 GB</td><td><strong>0.28 GB</strong></td></tr>
</table>

> 💡 KV cache MLA jauh lebih kecil drpd attention standar karena cuma nyimpen `kv_lora_rank + qk_rope_head_dim` (245 dim) per token, bukan `num_heads × head_dim` penuh. Pada 8K context, KV cache ~0.080 GB.

### Performance Estimates

<table>
<tr><th>Metric</th><th>Value</th><th>Notes</th></tr>
<tr><td><strong>FLOPs per Token</strong></td><td>1,053,720,534</td><td>Forward pass only</td></tr>
<tr><td><strong>TFLOPs per Token</strong></td><td>0.0011</td><td>≈ 6× untuk backward</td></tr>
</table>

---

## 💻 Cara Penggunaan

### Basic Loading

```python
from transformers import AutoConfig, AutoModelForCausalLM
import torch

config = AutoConfig.from_pretrained("Lyon28/caca-650M-untrained", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "Lyon28/caca-650M-untrained",
    config=config,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

print(f"Model loaded: {model.num_parameters():,} parameters")
print("⚠️  Model ini UNTRAINED — output belum bermakna")
```

### Quantized Loading (4-bit)

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    "Lyon28/caca-650M-untrained",
    trust_remote_code=True,
    quantization_config=bnb_config,
    device_map="auto"
)
print(f"Memory footprint: ~0.28GB (4-bit)")
```

### Mulai Pretraining

```python
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./output",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=3e-4,
    max_steps=10000,
    lr_scheduler_type="cosine",
    warmup_steps=500,
    bf16=True,
    gradient_checkpointing=True,
)

trainer = Trainer(model=model, args=training_args, train_dataset=train_dataset)
trainer.train()
```

---

## 📦 Isi Repo

Weight tersimpan dalam satu file <code>model.safetensors</code>.

## 🏗️ Status Training

Tidak ada metrik training (loss, wandb, tensorboard) — ini adalah **base architecture release** untuk dipakai sebagai titik awal pretraining dari nol.

---

## 📄 License

Model ini dirilis di bawah **Apache License 2.0** — bebas dipakai, dimodifikasi, dan didistribusikan (dengan attribution), disediakan "as is" tanpa warranty.

---

## 👥 Team & Contact

- **Lyon** — Architecture design & implementation
- 📧 cacatransformers@gmail.com
- 💬 [Discussions](https://huggingface.co/Lyon28/caca-650M-untrained/discussions)

---

<div align="center">
<sub>Model ini adalah bagian dari <b>Caca Project</b> — eksplorasi Indonesian LLM dari nol.<br/>
Dibuat oleh <a href="https://huggingface.co/Lyon28">@Lyon28</a> | Licensed under Apache 2.0 | Built with 🤗 Transformers</sub>

**🌟 "Dari nol, untuk semua" 🌟**
</div>