Add Tals-coder-01 Model Card with Ternary Bonsai PQ2 lineage & custom quant documentation

README.md

@@ -0,0 +1,167 @@
+---
+language:
+- en
+- multilingual
+tags:
+- ternary
+- speculative-decoding
+- mtp
+- vision
+- multimodal
+- gguf
+- custom-quant
+- qwen35
+- code
+- reasoning
+- agentic
+base_model:
+- ukisai/Swift-Bonsai-2-GGUF
+- killy369/Ternary-Bonsai-2-27B-MTP-drafter-GGUF
+license: apache-2.0
+pipeline_tag: text-generation
+---
+
+# ⚡ Tals-coder-01 (Custom GGUF Quant: Ternary Bonsai 2 + Grafted Native MTP)
+
+**Tals-coder-01** is a custom GGUF quantization and architectural graft of **Ternary Bonsai 2 (27B)**. Engineered specifically for consumer 16GB VRAM GPUs (RTX 4060 Ti 16GB, RTX 4070 Ti Super, RTX 4080/4090), it delivers **55–60 tokens/second** on a budget 128-bit memory bus (288 GB/s), scaling to **120–150+ tokens/second** on high-bandwidth hardware (RTX 3090, 4090, A100).
+
+---
+
+## 🧬 Architectural Pedigree & Custom GGUF Quantization
+
+This model is a custom GGUF quant built upon and fused from:
+1. **Base Model:** Based on **[Swift-Bonsai-2-PQ2_0.gguf](https://huggingface.co/ukisai/Swift-Bonsai-2-GGUF)** by Ukisai — a 27B parameter Ternary model using Walsh-Hadamard rotation representation (`PQ2_0` 142 quantization).
+2. **Native MTP Drafter:** Integrated with weights from **[Ternary-Bonsai-2-27B-MTP-drafter-GGUF](https://huggingface.co/killy369/Ternary-Bonsai-2-27B-MTP-drafter-GGUF)** by killy369 — featuring a 1-layer native Multi-Token Prediction (MTP) draft head with a 64k pruned vocabulary.
+3. **The Custom Graft:** Rather than requiring users to configure and orchestrate separate target and draft models (`-m` and `-md`), **Tals-coder-01 grafts the 65th MTP layer and unrotated token embeddings directly into the core GGUF file** (`Tals-coder-01.gguf`), producing an all-in-one standalone model.
+4. **Multimodal Vision:** Paired with **`Tals-coder-01-mmproj.gguf`** (0.63 GB CLIP Q8_0 vision tower) for full image comprehension, UI analysis, and diagram inspection.
+
+---
+
+## 🌟 Why Tals-coder-01?
+
+* **All-in-One Model File:** The native MTP draft head is built directly into `Tals-coder-01.gguf` (`qwen35.nextn_predict_layers = 1`, `qwen35.block_count = 65`).
+* **Pruned Reasoning (No Overthinking):** Unlike verbose models that get trapped in endless monologue loops, Tals-coder-01 outputs direct, clean, and immediately functional code.
+* **55–60 tok/s on Budget 16GB Cards:** Breaks through the theoretical memory bandwidth barrier with verified **68%–85% MTP speculative draft acceptance**.
+* **100% VRAM Execution:** Runs 27B parameters, 128k context, and vision completely on GPU (~13.8–14.1 GB / 16.0 GB total) with **zero CPU/PCIe RAM spillover**.
+* **Massive Context Scaling on 16GB VRAM:**
+  * **64k Context:** Full FP16 KV Cache (`-ctk f16 -ctv f16`) for zero precision degradation.
+  * **128k Context:** Balanced Hybrid KV Cache (`-ctk q8_0 -ctv q4_0`) at ~14.1 GB total VRAM.
+  * **262k Full Native Context:** Ultra-long context with Q4_0 KV Cache (`-ctk q4_0 -ctv q4_0`) fitting 100% in VRAM!
+
+---
+
+## 📊 Benchmark & Performance
+
+Tested locally on **NVIDIA GeForce RTX 4060 Ti 16GB** (PCIe 4.0 x8, 288 GB/s bandwidth, WDDM):
+
+| Configuration | Speculative Acceptance | Generation Speed | Context Window |
+| :--- | :---: | :---: | :---: |
+| **Base Model Only** (No MTP) | N/A (0%) | 24.8 – 25.8 tok/s | Up to 262k |
+| **Tals-coder-01 (MTP `n_max=1`)** | 77.8% | 40.7 – 42.5 tok/s | Up to 262k |
+| **Tals-coder-01 (MTP `n_max=2`)** *(Optimal)* | **68.9% – 74.2%** | **55.0 – 60.5 tok/s** | Up to 262k |
+| **Boilerplate / Code Bursts** | **85.2%** | **Up to 76.7 tok/s** | Up to 262k |
+| *Estimated RTX 4090 / 3090 (1,008 GB/s)* | *75% – 85%* | **120 – 155 tok/s** | *Full 262k* |
+
+---
+
+## 📦 Model Files in Repository
+
+| Filename | Size | Description |
+| :--- | :---: | :--- |
+| `Tals-coder-01.gguf` | 8.25 GB | **Unified 27B Custom GGUF Quant** with Grafted Native MTP Head (All 65 layers). |
+| `Tals-coder-01-mmproj.gguf` | 0.63 GB | High-fidelity CLIP Q8_0 multimodal vision projector. |
+| `bonsai2-pascal.patch` | 259 KB | Source patch for custom Hadamard kernels & MTP timing. |
+
+---
+
+## 🛠️ Optimal llama.cpp Runtime Setup
+
+Because this model uses ternary quantization in a Hadamard-rotated basis, use the optimized `llama.cpp` build with Hadamard kernel support (Prism ML / Pascal branch, or apply `bonsai2-pascal.patch`).
+
+### Critical Environment Variables (Mandatory)
+Before starting the server, set these environment variables to enable Pascal CUDA kernels and prevent CPU busy-polling:
+
+#### Windows (PowerShell / CMD):
+```bat
+set GGML_CUDA_ROWLANE=1
+set GGML_CUDA_RL_N4_LONG=1
+set CUDA_DEVICE_SCHEDULE=BLOCKING_SYNC
+```
+
+#### Linux (Bash):
+```bash
+export GGML_CUDA_ROWLANE=1
+export GGML_CUDA_RL_N4_LONG=1
+export CUDA_DEVICE_SCHEDULE=BLOCKING_SYNC
+```
+
+---
+
+## ⚡ Recommended Run Configurations
+
+### 1. Default Production Mode (128k Context + Grafted MTP + Vision)
+Run the single unified model with vision on a 16GB GPU:
+```bash
+./llama-server \
+  -m "Tals-coder-01.gguf" \
+  --mmproj "Tals-coder-01-mmproj.gguf" \
+  --mmproj-offload \
+  --image-min-tokens 1024 \
+  -ngl 99 \
+  -c 128000 \
+  -ctk q8_0 -ctv q4_0 \
+  -fa on -np 1 -t 4 -tb 4 --poll 0 \
+  --load-mode mmap --reasoning-preserve \
+  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.2 \
+  --backend-sampling --jinja \
+  --host 127.0.0.1 --port 8080
+```
+
+### 2. High-Precision Coding Mode (64k Context + FP16 KV Cache)
+Maximum precision for complex mathematical derivations:
+```bash
+./llama-server \
+  -m "Tals-coder-01.gguf" \
+  --mmproj "Tals-coder-01-mmproj.gguf" \
+  -ngl 99 \
+  -c 65536 \
+  -ctk f16 -ctv f16 \
+  -fa on -np 1 -t 4 -tb 4 --poll 0 \
+  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.2 \
+  --backend-sampling --host 127.0.0.1 --port 8080
+```
+
+### 3. Extreme Context Mode (262k Native Context on 16GB VRAM)
+Process entire codebases and large book-length documents with zero CPU offload:
+```bash
+./llama-server \
+  -m "Tals-coder-01.gguf" \
+  -ngl 99 \
+  -c 262144 \
+  -ctk q4_0 -ctv q4_0 \
+  -fa on -np 1 -t 4 -tb 4 --poll 0 \
+  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.2 \
+  --backend-sampling --host 127.0.0.1 --port 8080
+```
+
+---
+
+## 💻 OpenCode / Cline / Continue Configuration
+
+Set your tool's API endpoint to:
+* **Base URL:** `http://127.0.0.1:8080/v1`
+* **Model Name:** `tals-coder-01`
+* **API Key:** `not-needed` (any string)
+* **Temperature:** `0.0` – `0.2` (Deterministic coding & reasoning)
+
+---
+
+## 🙏 Credits & Acknowledgements
+
+Special thanks to the open-source creators whose foundational work made this custom GGUF quant distribution possible:
+* **[Ukisai](https://huggingface.co/ukisai/Swift-Bonsai-2-GGUF):** Creator of the original **Swift-Bonsai-2-PQ2_0.gguf** (Ternary Bonsai 2) model weights and quantization basis.
+* **[killy369 / Kilian](https://huggingface.co/killy369/Ternary-Bonsai-2-27B-MTP-drafter-GGUF):** Creator of the **Ternary-Bonsai-2-27B-MTP-drafter-GGUF** native draft weights and pruned 64k vocabulary.
+* **[Prism ML / Pascal](https://github.com/prism-ml):** For pioneering ternary Hadamard kernels and custom llama.cpp acceleration.
+* **Qwen Team:** For the underlying Qwen 3.5 architecture.
+
+Distributed under the **Apache 2.0** license.