Update README.md

README.md

@@ -1,3 +1,171 @@
+# Ling-3.0 Tiny — Apex Quant GGUF Models
+
+This repository contains 4 quantized GGUF profiles of the **inclusionAI/Ling-3.0-tiny** model, produced using Apex-Quant technology.
+
+## 📦 Model Profile Summary
+
+| Profile | Size | BPW | Use Case |
+|---------|------|-----|----------|
+| **i-quality** | 5.4 GB | 5.83 | Highest quality, production environments |
+| **i-balanced** | 5.6 GB | 6.02 | Balanced quality and performance |
+| **i-compact** | 3.7 GB | 4.03 | Compact deployment, low RAM |
+| **i-mini** | — | — | ⚠️ Requires `imatrix`, not yet quantized |
+
+> **BPW** = Bits Per Weight. Higher value = better quality.
+
+## 📁 File Structure
+
+```
+models/
+├── Ling-3.0-tiny-i-quality.gguf   # 5.4 GB — Highest quality
+├── Ling-3.0-tiny-i-balanced.gguf  # 5.6 GB — Balanced
+└── Ling-3.0-tiny-i-compact.gguf   # 3.7 GB — Compact
+```
+
+## 🔗 Source Model
+
+These models were created based on the **inclusionAI/Ling-3.0-tiny** model from HuggingFace.
+
+- **Model Page:** https://huggingface.co/inclusionAI/Ling-3.0-tiny
+- **Architecture:** BailingMoeV3ForCausalLM (Mixture-of-Experts)
+- **Parameter Count:** 128×1.0B (128 experts, each with 1B parameters)
+- **Context Length:** 131,072 tokens
+- **Vocabulary:** 157,184 tokens
+- **License:** MIT
+
+## 🛠️ Technology
+
+These quantized models were produced using **Apex-Quant** technology.
+
+- **Apex-Quant:** MoE-aware mixed-precision quantization
+- **Infrastructure:** llama.cpp (`llama-quantize`)
+- **Quantize Script:** `apex-quant/scripts/quantize.sh`
+
+## 🙏 Acknowledgments
+
+- **[localai-org/apex-quant](https://github.com/localai-org/apex-quant)** — Apex-Quant MoE-aware mixed-precision quantization framework
+- **[ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)** — GGUF format and quantization engine
+- **[inclusionAI](https://huggingface.co/inclusionAI)** — Original Ling-3.0-tiny model creators
+
+## 📋 Technical Details
+
+### Architecture Information
+- **Architecture:** `bailingmoe3`
+- **Block Count:** 24 layers
+- **Expert Count:** 128 experts
+- **Expert Used Count:** 8 experts/token
+- **Expert Group Count:** 8
+- **Expert Group Used Count:** 4
+- **Expert Gating Function:** Top-K (k=8)
+- **Hidden Size:** 1,536
+- **Feed Forward Size:** 4,608
+- **Attention Heads:** 16
+- **Attention Head Count KV:** [0, 0, 0, 1, ...] (grouped query attention)
+- **Rope Frequency Base:** 6,000,000
+- **Layer Norm Epsilon:** 1e-6
+
+### Quantize Profile Details
+
+#### i-quality (Q6_K/Q5_K/IQ4_XS)
+- **Expert FFN:** Q6_K / Q5_K / IQ4_XS (mixed)
+- **Shared FFN:** Q8_0
+- **Attention:** Q6_K
+- **BPW:** 5.83
+- **File Size:** 5.4 GB
+
+#### i-balanced (Q6_K/Q5_K)
+- **Expert FFN:** Q6_K / Q5_K (mixed)
+- **Shared FFN:** Q8_0
+- **Attention:** Q6_K
+- **BPW:** 6.02
+- **File Size:** 5.6 GB
+
+#### i-compact (Q4_K/Q3_K)
+- **Expert FFN:** Q4_K / Q3_K (mixed)
+- **Shared FFN:** Q6_K
+- **Attention:** Q4_K
+- **BPW:** 4.03
+- **File Size:** 3.7 GB
+
+#### i-mini (Q3_K/IQ2_S) — ⚠️ Not Quantized
+- **Expert FFN:** Q3_K / IQ2_S (mixed)
+- **Shared FFN:** Q5_K / Q4_K
+- **Attention:** Q4_K / Q3_K
+- **BPW:** ~3.5 (estimated)
+- **Status:** Requires `imatrix` (importance matrix). Inference must be run on the model to generate the importance matrix before quantization.
+
+## 💻 Usage
+
+### With llama.cpp
+
+```bash
+# Run with i-quality profile
+./main -m models/Ling-3.0-tiny-i-quality.gguf -n 128 -p "Hello, how are you?"
+
+# Run with i-compact profile
+./main -m models/Ling-3.0-tiny-i-compact.gguf -n 128 -p "Hello, how are you?"
+```
+
+### With Ollama
+
+```bash
+# Create Dockerfile or Ollamafile
+FROM llama.cpp
+COPY models/Ling-3.0-tiny-i-quality.gguf /model.gguf
+```
+
+### With Python (llama-cpp-python)
+
+```python
+from llama_cpp import Llama
+
+llm = Llama(
+    model_path="models/Ling-3.0-tiny-i-quality.gguf",
+    n_ctx=4096,
+    n_threads=8
+)
+
+output = llm(
+    "Hello, how are you?",
+    max_tokens=128
+)
+print(output["choices"][0]["text"])
+```
+
+## 📊 Model Comparison
+
+| Criterion | i-quality | i-balanced | i-compact |
+|-----------|-----------|------------|-----------|
+| **Quality** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐ |
+| **Speed** | ⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
+| **RAM** | High | Medium | Low |
+| **Size** | 5.4 GB | 5.6 GB | 3.7 GB |
+| **BPW** | 5.83 | 6.02 | 4.03 |
+
+## 📝 Notes
+
+- All models are in **GGUF v3** format.
+- The **BailingMoeV3** architecture uses Mixture-of-Experts (MoE) technology.
+- The model uses **grouped query attention (GQA)** and **rope** positional embeddings.
+- The `i-mini` profile cannot be quantized without `imatrix`. ~100-200 inference samples must be run on the model to generate the importance matrix.
+
+## 📄 License
+
+The original model is distributed under the **MIT** license. The quantized models are shared under the same license.
+
+## 🔗 Related Links
+
+- **Original Model:** https://huggingface.co/inclusionAI/Ling-3.0-tiny
+- **Apex-Quant:** https://github.com/localai-org/apex-quant
+- **llama.cpp:** https://github.com/ggerganov/llama.cpp
+- **GGUF Format:** https://github.com/ggerganov/ggml/blob/master/docs/gguf.md
+
+---
+
+**Note:** These models are quantized for local use. Check the original model's license for commercial use.
+
+
+
 ---
 license: mit
 ---