|
|
--- |
|
|
title: README |
|
|
emoji: 🐨 |
|
|
colorFrom: purple |
|
|
colorTo: indigo |
|
|
sdk: static |
|
|
pinned: true |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
<!-- markdownlint-disable first-line-h1 --> |
|
|
<!-- markdownlint-disable html --> |
|
|
|
|
|
<div align="center"> |
|
|
<img src="https://huggingface.co/spaces/Caca-AI/README/resolve/main/logo.png" width="50%" alt="caca-AI" /> |
|
|
</div> |
|
|
<hr> |
|
|
<div align="center"> |
|
|
<a href="https://huggingface.co/Lyon28"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Lyon28-ffc107?color=ffc107&logoColor=white"/></a> |
|
|
<br> |
|
|
<a href="LICENSE"> |
|
|
<img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"/> |
|
|
</a> |
|
|
</div> |
|
|
|
|
|
## 📋 Deskripsi |
|
|
|
|
|
**Caca** adalah arsitektur Large Language Model (LLM) generasi terbaru yang menggabungkan berbagai teknik state-of-the-art dalam deep learning. Model ini dirancang dengan fokus pada **efisiensi**, **skalabilitas**, dan **performa tinggi**. |
|
|
|
|
|
<blockquote style="border-left: 4px solid #4A90E2; padding-left: 16px; margin: 16px 0; color: #555;"> |
|
|
<p><strong>Caca</strong> itu eksperimen open-source Indonesian LLM yang dibuat dari nol secara individual dan bertahap. Bukan kompetitor siapa-siapa, cuma pengen eksplorasi apa yang bisa dilakukan dengan budget terbatas, passion unlimited, dan mindset collaborative. Kalau berguna buat orang lain, alhamdulillah. Kalau enggak, ya tetap fun kok.</p> |
|
|
<p>Ini proyek eksplorasi, jadi kalau gagal ya bagian dari proses belajar. Kalau berhasil, itu bonus.</p> |
|
|
</blockquote> |
|
|
|
|
|
<!-- COMPARISON --> |
|
|
<h2>📊 Perbandingan dengan Arsitektur Lain</h2> |
|
|
|
|
|
<table width="100%"> |
|
|
<thead> |
|
|
<tr style="background-color: #f0f0f0;"> |
|
|
<th width="25%">Fitur</th> |
|
|
<th width="15%">Caca</th> |
|
|
<th width="15%">LLaMA 2</th> |
|
|
<th width="15%">Mistral</th> |
|
|
<th width="15%">IndoGPT</th> |
|
|
<th width="15%">GPT-2</th> |
|
|
</tr> |
|
|
</thead> |
|
|
<tbody> |
|
|
|
|
|
<!-- Basic Architecture --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>🏗️ Arsitektur Dasar</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Status</b></td> |
|
|
<td bgcolor="#fff3cd">⚠️ Untrained</td> |
|
|
<td bgcolor="#d4edda">✅ Trained</td> |
|
|
<td bgcolor="#d4edda">✅ Trained</td> |
|
|
<td bgcolor="#d4edda">✅ Trained</td> |
|
|
<td bgcolor="#d4edda">✅ Trained</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Ukuran Model</b></td> |
|
|
<td bgcolor="#d4edda">60+ variant<br><small>1M - 1T (semoga)</small></td> |
|
|
<td bgcolor="#d4edda">7B / 13B / 70B</td> |
|
|
<td bgcolor="#d4edda">7B</td> |
|
|
<td bgcolor="#fff3cd">117M</td> |
|
|
<td bgcolor="#fff3cd">117M - 1.5B</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Tipe Arsitektur</b></td> |
|
|
<td>Decoder-only</td> |
|
|
<td>Decoder-only</td> |
|
|
<td>Decoder-only</td> |
|
|
<td>Decoder-only</td> |
|
|
<td>Decoder-only</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Fungsi Aktivasi</b></td> |
|
|
<td bgcolor="#d4edda">SwiGLU</td> |
|
|
<td bgcolor="#d4edda">SwiGLU</td> |
|
|
<td bgcolor="#d4edda">SwiGLU</td> |
|
|
<td bgcolor="#f8d7da">GELU</td> |
|
|
<td bgcolor="#f8d7da">GELU</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Normalisasi</b></td> |
|
|
<td bgcolor="#d4edda">RMSNorm</td> |
|
|
<td bgcolor="#d4edda">RMSNorm</td> |
|
|
<td bgcolor="#d4edda">RMSNorm</td> |
|
|
<td bgcolor="#f8d7da">LayerNorm</td> |
|
|
<td bgcolor="#f8d7da">LayerNorm</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Tahun Release</b></td> |
|
|
<td>2025</td> |
|
|
<td>2023</td> |
|
|
<td>2023</td> |
|
|
<td>2020</td> |
|
|
<td>2019</td> |
|
|
</tr> |
|
|
|
|
|
<!-- Attention Mechanisms --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>👁️ Mekanisme Attention</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Tipe Attention</b></td> |
|
|
<td bgcolor="#d4edda">GQA (configurable)</td> |
|
|
<td bgcolor="#d4edda">GQA</td> |
|
|
<td bgcolor="#d4edda">GQA</td> |
|
|
<td bgcolor="#f8d7da">MHA</td> |
|
|
<td bgcolor="#f8d7da">MHA</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Position Encoding</b></td> |
|
|
<td bgcolor="#d4edda">RoPE + variants</td> |
|
|
<td bgcolor="#d4edda">RoPE</td> |
|
|
<td bgcolor="#d4edda">RoPE</td> |
|
|
<td bgcolor="#f8d7da">Learned</td> |
|
|
<td bgcolor="#f8d7da">Learned</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Max Context</b></td> |
|
|
<td bgcolor="#d4edda">8K - 16K</td> |
|
|
<td bgcolor="#fff3cd">4K</td> |
|
|
<td bgcolor="#d4edda">32K</td> |
|
|
<td bgcolor="#f8d7da">1K</td> |
|
|
<td bgcolor="#f8d7da">1K</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Sliding Window</b></td> |
|
|
<td bgcolor="#d4edda">✅ Optional</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#d4edda">✅ 4K window</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Flash Attention</b></td> |
|
|
<td bgcolor="#d4edda">✅ Flash Attn 2</td> |
|
|
<td bgcolor="#d4edda">✅ Supported</td> |
|
|
<td bgcolor="#d4edda">✅ Supported</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>KV Cache Efficiency</b></td> |
|
|
<td bgcolor="#d4edda">75% reduction<br><small>(GQA 4:1)</small></td> |
|
|
<td bgcolor="#d4edda">~60% reduction</td> |
|
|
<td bgcolor="#d4edda">75% reduction</td> |
|
|
<td bgcolor="#f8d7da">No optimization</td> |
|
|
<td bgcolor="#f8d7da">No optimization</td> |
|
|
</tr> |
|
|
|
|
|
<!-- Advanced Features --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>🚀 Fitur Lanjutan</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Mixture of Experts</b></td> |
|
|
<td bgcolor="#d4edda">✅ Optional<br><small>TopK + ExpertChoice</small></td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌<br><small>(Mixtral variant)</small></td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Multimodal</b></td> |
|
|
<td bgcolor="#d4edda">✅ Native<br><small>Vision + Audio</small></td> |
|
|
<td bgcolor="#f8d7da">❌<br><small>(LLaVA separate)</small></td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Config Flexibility</b></td> |
|
|
<td bgcolor="#d4edda">✅ 50+ parameters<br><small>Toggle semua fitur</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ Limited</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Limited</td> |
|
|
<td bgcolor="#f8d7da">❌ Fixed</td> |
|
|
<td bgcolor="#f8d7da">❌ Fixed</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Layer Scale</b></td> |
|
|
<td bgcolor="#d4edda">✅ Optional</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Stochastic Depth</b></td> |
|
|
<td bgcolor="#d4edda">✅ Optional</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
<td bgcolor="#f8d7da">❌</td> |
|
|
</tr> |
|
|
|
|
|
<!-- Performance & Optimization --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>⚡ Performa & Optimisasi</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Inference Speed</b><br><small>(7B model, A100)</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ TBD<br><small>(belum trained)</small></td> |
|
|
<td bgcolor="#d4edda">~75 tok/s</td> |
|
|
<td bgcolor="#d4edda">~78 tok/s</td> |
|
|
<td bgcolor="#d4edda">~150 tok/s<br><small>(jauh lebih kecil)</small></td> |
|
|
<td bgcolor="#d4edda">~120 tok/s<br><small>(jauh lebih kecil)</small></td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Memory Footprint</b><br><small>(7B, BF16)</small></td> |
|
|
<td bgcolor="#d4edda">~14GB<br><small>(dengan GQA)</small></td> |
|
|
<td bgcolor="#fff3cd">~14GB</td> |
|
|
<td bgcolor="#d4edda">~14GB</td> |
|
|
<td bgcolor="#d4edda">~500MB</td> |
|
|
<td bgcolor="#d4edda">~500MB</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Gradient Checkpointing</b></td> |
|
|
<td bgcolor="#d4edda">✅ Full support</td> |
|
|
<td bgcolor="#d4edda">✅ Supported</td> |
|
|
<td bgcolor="#d4edda">✅ Supported</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Manual</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Manual</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Quantization</b></td> |
|
|
<td bgcolor="#d4edda">✅ 8-bit/4-bit built-in</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Via external tools</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Via external tools</td> |
|
|
<td bgcolor="#f8d7da">❌ Limited support</td> |
|
|
<td bgcolor="#f8d7da">❌ Limited support</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Multi-Backend Support</b></td> |
|
|
<td bgcolor="#d4edda">✅ 4 backends<br><small>Flash/xFormers/SDPA/Standard</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ 2 backends</td> |
|
|
<td bgcolor="#fff3cd">⚠️ 2 backends</td> |
|
|
<td bgcolor="#f8d7da">❌ Standard only</td> |
|
|
<td bgcolor="#f8d7da">❌ Standard only</td> |
|
|
</tr> |
|
|
|
|
|
<!-- Language Support --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>🌏 Dukungan Bahasa</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Bahasa Indonesia</b></td> |
|
|
<td bgcolor="#fff3cd">⚠️ Belum trained<br><small>Designed for ID</small></td> |
|
|
<td bgcolor="#f8d7da">❌ Poor<br><small>English-heavy</small></td> |
|
|
<td bgcolor="#f8d7da">❌ Poor<br><small>English-heavy</small></td> |
|
|
<td bgcolor="#d4edda">✅ Native</td> |
|
|
<td bgcolor="#f8d7da">❌ Minimal</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>English</b></td> |
|
|
<td bgcolor="#fff3cd">⚠️ TBD<br><small>Bilingual design</small></td> |
|
|
<td bgcolor="#d4edda">✅ Excellent</td> |
|
|
<td bgcolor="#d4edda">✅ Excellent</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Limited</td> |
|
|
<td bgcolor="#d4edda">✅ Good</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Training Data</b></td> |
|
|
<td bgcolor="#fff3cd">⚠️ To be trained<br><small>User's choice</small></td> |
|
|
<td bgcolor="#d4edda">2T tokens<br><small>English-heavy</small></td> |
|
|
<td bgcolor="#d4edda">Unknown<br><small>English-heavy</small></td> |
|
|
<td bgcolor="#d4edda">23GB<br><small>Indonesian</small></td> |
|
|
<td bgcolor="#d4edda">40GB<br><small>WebText</small></td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Vocab Size</b></td> |
|
|
<td bgcolor="#d4edda">32K<br><small>(configurable)</small></td> |
|
|
<td bgcolor="#d4edda">32K</td> |
|
|
<td bgcolor="#d4edda">32K</td> |
|
|
<td bgcolor="#fff3cd">50K</td> |
|
|
<td bgcolor="#fff3cd">50K</td> |
|
|
</tr> |
|
|
|
|
|
<!-- Developer Experience --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>👨💻 Developer Experience</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Error Messages</b></td> |
|
|
<td bgcolor="#d4edda">✅ Helpful + solutions<br><small>Detailed debugging</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ Standard PyTorch</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Standard PyTorch</td> |
|
|
<td bgcolor="#f8d7da">❌ Basic errors</td> |
|
|
<td bgcolor="#f8d7da">❌ Basic errors</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Config Validation</b></td> |
|
|
<td bgcolor="#d4edda">✅ Comprehensive<br><small>Auto-check conflicts</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ Basic</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Basic</td> |
|
|
<td bgcolor="#f8d7da">❌ Minimal</td> |
|
|
<td bgcolor="#f8d7da">❌ Minimal</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Documentation</b></td> |
|
|
<td bgcolor="#d4edda">✅ Extensive<br><small>ID + EN, with examples</small></td> |
|
|
<td bgcolor="#d4edda">✅ Good<br><small>Official docs</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ Medium<br><small>Community-driven</small></td> |
|
|
<td bgcolor="#f8d7da">❌ Limited<br><small>Minimal docs</small></td> |
|
|
<td bgcolor="#d4edda">✅ Extensive<br><small>OpenAI docs</small></td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Code Examples</b></td> |
|
|
<td bgcolor="#d4edda">✅ 50+ examples<br><small>Training to deployment</small></td> |
|
|
<td bgcolor="#d4edda">✅ Many examples</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Some examples</td> |
|
|
<td bgcolor="#f8d7da">❌ Few examples</td> |
|
|
<td bgcolor="#d4edda">✅ Many examples</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>HuggingFace Integration</b></td> |
|
|
<td bgcolor="#d4edda">✅ Full native<br><small>Auto-registered</small></td> |
|
|
<td bgcolor="#d4edda">✅ Official</td> |
|
|
<td bgcolor="#d4edda">✅ Official</td> |
|
|
<td bgcolor="#d4edda">✅ Available</td> |
|
|
<td bgcolor="#d4edda">✅ Standard</td> |
|
|
</tr> |
|
|
|
|
|
<!-- Availability & License --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>🌍 Ketersediaan & Lisensi</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>License</b></td> |
|
|
<td bgcolor="#d4edda">✅ Apache 2.0<br><small>Fully permissive</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ LLaMA 2 License<br><small>Commercial OK</small></td> |
|
|
<td bgcolor="#d4edda">✅ Apache 2.0</td> |
|
|
<td bgcolor="#d4edda">✅ MIT</td> |
|
|
<td bgcolor="#d4edda">✅ MIT</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Commercial Use</b></td> |
|
|
<td bgcolor="#d4edda">✅ Allowed<br><small>No restrictions</small></td> |
|
|
<td bgcolor="#d4edda">✅ Allowed</td> |
|
|
<td bgcolor="#d4edda">✅ Allowed</td> |
|
|
<td bgcolor="#d4edda">✅ Allowed</td> |
|
|
<td bgcolor="#d4edda">✅ Allowed</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Weights Available</b></td> |
|
|
<td bgcolor="#f8d7da">❌ Not trained<br><small>Architecture only</small></td> |
|
|
<td bgcolor="#d4edda">✅ All sizes<br><small>7B/13B/70B</small></td> |
|
|
<td bgcolor="#d4edda">✅ 7B</td> |
|
|
<td bgcolor="#d4edda">✅ 117M</td> |
|
|
<td bgcolor="#d4edda">✅ All sizes</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Self-Hosting</b></td> |
|
|
<td bgcolor="#d4edda">✅ Designed for it<br><small>Full control</small></td> |
|
|
<td bgcolor="#d4edda">✅ Yes</td> |
|
|
<td bgcolor="#d4edda">✅ Yes</td> |
|
|
<td bgcolor="#d4edda">✅ Yes</td> |
|
|
<td bgcolor="#d4edda">✅ Yes</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Training Required</b></td> |
|
|
<td bgcolor="#f8d7da">❌ Yes<br><small>From scratch</small></td> |
|
|
<td bgcolor="#d4edda">✅ No<br><small>Ready to use</small></td> |
|
|
<td bgcolor="#d4edda">✅ No<br><small>Ready to use</small></td> |
|
|
<td bgcolor="#d4edda">✅ No<br><small>Ready to use</small></td> |
|
|
<td bgcolor="#d4edda">✅ No<br><small>Ready to use</small></td> |
|
|
</tr> |
|
|
|
|
|
<!-- Use Cases --> |
|
|
<tr> |
|
|
<td colspan="6" style="background-color: #e3f2fd; padding: 10px;"> |
|
|
<b>🎯 Use Cases</b> |
|
|
</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Production Ready</b></td> |
|
|
<td bgcolor="#f8d7da">❌ Not yet<br><small>After training</small></td> |
|
|
<td bgcolor="#d4edda">✅ Yes</td> |
|
|
<td bgcolor="#d4edda">✅ Yes</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Limited<br><small>Too small</small></td> |
|
|
<td bgcolor="#fff3cd">⚠️ Limited<br><small>Outdated</small></td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Research</b></td> |
|
|
<td bgcolor="#d4edda">✅ Excellent<br><small>Modular design</small></td> |
|
|
<td bgcolor="#d4edda">✅ Good</td> |
|
|
<td bgcolor="#d4edda">✅ Good</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Limited</td> |
|
|
<td bgcolor="#d4edda">✅ Classic baseline</td> |
|
|
</tr> |
|
|
|
|
|
<tr bgcolor="#f8f9fa"> |
|
|
<td><b>Indonesian NLP</b></td> |
|
|
<td bgcolor="#fff3cd">⚠️ After training<br><small>High potential</small></td> |
|
|
<td bgcolor="#f8d7da">❌ Poor<br><small>Needs fine-tuning</small></td> |
|
|
<td bgcolor="#f8d7da">❌ Poor<br><small>Needs fine-tuning</small></td> |
|
|
<td bgcolor="#d4edda">✅ Native<br><small>But limited</small></td> |
|
|
<td bgcolor="#f8d7da">❌ Poor</td> |
|
|
</tr> |
|
|
|
|
|
<tr> |
|
|
<td><b>Education</b></td> |
|
|
<td bgcolor="#d4edda">✅ Excellent<br><small>Learn modern LLMs</small></td> |
|
|
<td bgcolor="#d4edda">✅ Good</td> |
|
|
<td bgcolor="#fff3cd">⚠️ Medium</td> |
|
|
<td bgcolor="#d4edda">✅ Good<br><small>Simple architecture</small></td> |
|
|
<td bgcolor="#d4edda">✅ Classic<br><small>Well-documented</small></td> |
|
|
</tr> |
|
|
|
|
|
</tbody> |
|
|
</table> |
|
|
|
|
|
<div style="margin-top: 20px; padding: 15px; background-color: #fff3cd; border-left: 4px solid #ffc107;"> |
|
|
<p><b>📝 Catatan Penting:</b></p> |
|
|
<ul style="margin: 10px 0;"> |
|
|
<li><b>Caca</b> adalah arsitektur modern yang <b>belum dilatih</b> - perlu training dari nol dengan dataset Indonesian</li> |
|
|
<li><b>LLaMA 2 & Mistral</b> sangat bagus untuk English, tapi <b>poor untuk Indonesian</b> tanpa fine-tuning</li> |
|
|
<li><b>IndoGPT</b> adalah satu-satunya dedicated Indonesian LLM, tapi <b>arsitektur sudah outdated</b> (GPT-2 era)</li> |
|
|
<li><b>GPT-2</b> dimasukkan sebagai baseline klasik - arsitektur yang sudah proven tapi tidak modern</li> |
|
|
</ul> |
|
|
</div> |
|
|
|
|
|
<div style="margin-top: 15px; padding: 15px; background-color: #d4edda; border-left: 4px solid #28a745;"> |
|
|
<p><b>✨ Keunggulan Unik Caca:</b></p> |
|
|
<ul style="margin: 10px 0;"> |
|
|
<li>🎯 <b>Modular Design</b>: Toggle 50+ fitur tanpa rewrite code</li> |
|
|
<li>🔧 <b>Developer-Friendly</b>: Error messages helpful + config validation</li> |
|
|
<li>🚀 <b>Modern Architecture</b>: GQA + Flash Attention + SwiGLU + RMSNorm</li> |
|
|
<li>🎨 <b>Multimodal Native</b>: Vision & Audio built-in (bukan add-on)</li> |
|
|
<li>📚 <b>Extensive Docs</b>: Bahasa Indonesia + English dengan banyak contoh</li> |
|
|
<li>⚡ <b>Optimization Focus</b>: 4 attention backends, auto-fallback, quantization ready</li> |
|
|
<li>🔬 <b>Research-Oriented</b>: MoE, Mixture of Depths, Layer Scale, dll.</li> |
|
|
</ul> |
|
|
</div> |
|
|
|
|
|
<div style="margin-top: 15px; padding: 15px; background-color: #f8d7da; border-left: 4px solid #dc3545;"> |
|
|
<p><b>⚠️ Keterbatasan Realistis:</b></p> |
|
|
<ul style="margin: 10px 0;"> |
|
|
<li>❌ <b>Belum trained</b> - output akan random sampai di-training</li> |
|
|
<li>❌ <b>Belum ada tokenizer</b> - perlu training tokenizer sendiri untuk Indonesian</li> |
|
|
<li>❌ <b>Butuh resources besar</b> - training 7B model perlu GPU kelas A100</li> |
|
|
<li>❌ <b>Belum teruji</b> - perlu extensive evaluation setelah training</li> |
|
|
<li>❌ <b>Community masih kecil</b> - tidak sebesar LLaMA/Mistral ecosystem</li> |
|
|
</ul> |
|
|
</div> |
|
|
|
|
|
<!-- Quotes --> |
|
|
<div align="center"> |
|
|
<img src="https://quotes-caca.vercel.app/api/SsQuote" alt="Daily Quote" width="100%" /> |
|
|
</div> |
|
|
<!-- Akhir Quotes --> |
|
|
|
|
|
## 🔗 Links |
|
|
|
|
|
- PyPI: https://pypi.org/project/caca-transformers/ |
|
|
- GitHub: https://github.com/Lyon-28/caca-transformers |
|
|
- Email: cacatransformers@gmail.com |
