Update Model Card with complete GGUF usage guide, 10-benchmark quantization evaluation, and YAML metadata

README.md

@@ -1,15 +1,21 @@
 ---
-base_model: Cloudflare/clef-flash
-library_name: transformers
 license: apache-2.0
+language:
+  - en
+library_name: transformers
 pipeline_tag: text-generation
+base_model: Cloudflare/clef-flash
+datasets:
+  - TypeScript/FastBrain-Instruct
 tags:
   - fastbrain
   - clef
   - gguf
+  - llama-cpp
   - systemone
   - decision-index
   - tool-calling
+  - function-calling
   - intent-classification
   - structured-decision
 ---
@@ -124,42 +130,158 @@ response = clef.systemone(model, processor, {
 print(response)
 ```
 
-### 2. `llama.cpp` GGUF (`Q4_K_M` / `Q8_0` / `BF16`) via `/v1/systemone`
+### 2. `llama.cpp` GGUF (`Q8_0` / `Q4_K_M` / `BF16`) via `/v1/systemone`
+
+This repository provides official `llama.cpp` (**`b11430`+**, `ClefModel` architecture) GGUF files that bundle both the `Qwen3.5-9B` hybrid backbone and the `116M` `JointSchemaHead` (`dec.*` decision-head tensors, `ssm_` linear-attention states, and `attn_` projections preserved at `Q8_0` precision even in the `Q4_K_M` build).
+
+> [!IMPORTANT]
+> **Required `llama.cpp` Version & `ClefModel` Batch Flags:**
+> 1. **Minimum version**: Use **`llama.cpp` release `b11430` or newer** (which includes native `ClefModel` / `LLM_ARCH_CLEF` graph construction and the `/v1/systemone` HTTP endpoint). Standard Ollama or older `llama.cpp` builds without `ClefModel` support cannot execute `dec.*` decision-head tensors.
+> 2. **Set `-b` and `-ub` to match your maximum prompt length (e.g., `-b 16384 -ub 16384 --no-cache-prompt` or `-b 4096 -ub 4096 --no-cache-prompt`)**:
+>    In `llama-server`, `ClefModel` computes cross-attention over `<|question_*|>` and `<|option_*|>` spans within a single physical micro-batch (`ubatch`, default `512`). Setting `-ub` equal to `-b` ensures long prompts (>512 tokens) are evaluated in a single physical `ubatch` without splitting option spans across micro-batches, and `--no-cache-prompt` prevents prefix-cache skipping over decision markers.
 
-This repository includes official `llama.cpp` (`b11430`+ / `ClefModel`) GGUF quantizations that pack both the `Qwen3.5-9B` hybrid backbone and the `116M` `JointSchemaHead` (`dec.*` tensors preserved at `Q8_0` precision):
+#### Step 1: Download Prebuilt `llama-server` (`b11430`+) and GGUF Weights
 
 ```bash
-# Start llama-server on the /v1/systemone decision endpoint (6.73 GB Q4_K_M quant)
-llama-server -hf TypeScript/FastBrain-9B:Q4_K_M --port 8080
+# 1. Download prebuilt llama.cpp (b11430+) with ClefModel & /v1/systemone support
+#    - Linux CUDA 13.x (Colab / Ubuntu): llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz
+#    - Linux CUDA 12.x:                  llama-b11430-bin-ubuntu-cuda-12.8-x64.tar.gz
+#    - macOS Apple Silicon (M1/M2/M3/M4): llama-b11430-bin-macos-arm64.tar.gz
+curl -sL https://github.com/ggml-org/llama.cpp/releases/download/b11430/llama-b11430-bin-ubuntu-cuda-13.4-x64.tar.gz | tar -xz
+
+# 2. Download FastBrain-9B GGUF (choose Q8_0 [9.66 GB, lossless] or Q4_K_M [6.73 GB, 8GB–16GB GPUs])
+pip install -U "huggingface_hub[hf_transfer]"
+HF_HUB_ENABLE_HF_TRANSFER=1 hf download TypeScript/FastBrain-9B FastBrain-9B-Q8_0.gguf --local-dir .
+```
+
+#### Step 2: Launch `llama-server` on the `/v1/systemone` Endpoint
+
+```bash
+# On Linux / Google Colab, ensure CUDA driver libraries are in LD_LIBRARY_PATH:
+export LD_LIBRARY_PATH="$(pwd)/llama-b11430:/usr/lib64-nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
+
+# Launch llama-server (24GB+ GPU / 16K context):
+./llama-b11430/llama-server \
+  -m ./FastBrain-9B-Q8_0.gguf \
+  -ngl 99 -c 16384 -b 16384 -ub 16384 -np 1 -fa on --no-cache-prompt \
+  --port 8080
+
+# Or on a 12GB–16GB GPU (e.g., Colab Free Tesla T4, 4K context):
+# ./llama-b11430/llama-server \
+#   -m ./FastBrain-9B-Q4_K_M.gguf \
+#   -ngl 99 -c 4096 -b 4096 -ub 4096 -np 1 --no-cache-prompt \
+#   --port 8080
+
+# Optional: For multimodal image/screenshot states, attach the vision projector:
+#   --mmproj ./mmproj-FastBrain-9B-Q8_0.gguf
+```
+
+#### Step 3: Query `/v1/systemone` (`curl` or Python — Single Forward Pass, `0` Output Tokens)
 
-# Query the single-forward-pass /v1/systemone endpoint
+```bash
 curl http://127.0.0.1:8080/v1/systemone \
   -H "Content-Type: application/json" \
   -d '{
     "model": "FastBrain-9B",
-    "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
+    "state": {
+      "ticket_id": "ORD-99281",
+      "customer_message": "I was charged twice for my subscription last week and nobody has replied to my ticket."
+    },
     "questions": {
       "route": {
         "type": "choice",
         "instructions": "Which team should handle this?",
-        "criteria": {"billing": "Payments or invoices", "technical": "Bugs or outages"}
+        "criteria": {
+          "billing": "Payments, refunds, duplicate charges, or invoices",
+          "technical": "Bugs, crashes, or service outages",
+          "account": "Password reset or login issues"
+        }
       },
-      "urgent": {"type": "noul", "instructions": "Is the customer requesting urgent resolution?"}
+      "urgent": {
+        "type": "noul",
+        "instructions": "Is the customer reporting a financial or urgent issue?"
+      }
     }
   }'
 ```
 
----
+**Python Client Example (`requests` / `urllib`)**:
+
+```python
+import requests
+
+resp = requests.post(
+    "http://127.0.0.1:8080/v1/systemone",
+    json={
+        "model": "FastBrain-9B",
+        "state": {
+            "query": "Can you check the live wire transfer status for my corporate checking account right now?",
+            "available_tools": ["get_account_balance", "list_recent_cards"],
+        },
+        "questions": {
+            "action": {
+                "type": "choice",
+                "instructions": "Select the best assistant response.",
+                "criteria": {
+                    "direct_answer": "Your wire transfer has been completed.",
+                    "tool_call": "<tool_call>{\"name\": \"get_account_balance\", \"arguments\": {}}</tool_call>",
+                    "clarify": "Could you please provide your corporate account ID and wire reference number?",
+                    "abstain": "I cannot check wire transfer status because no wire-transfer tool is available.",
+                },
+            }
+        },
+    },
+    timeout=30,
+).json()
 
-## Hardware Footprint, Quantization Tiers & Inference Speed
+print(resp["choices"]["action"])  # Calibrated probability distribution over all options
+print(resp["usage"])              # {'input_tokens': ..., 'output_tokens': 0, 'total_tokens': ...}
+```
+
+---
 
-| Format / Precision Tier | Weight Footprint | Peak VRAM | Minimum Hardware | Typical Latency (`500–1k tok`) |
-| :--- | :---: | :---: | :--- | :---: |
-| **`FastBrain-9B-Q4_K_M.gguf`** *(4-bit + `Q8_0` Head)* | `6.73 GB` | `~7.6 GB` | **8GB GPU / Colab Free T4 (16GB) / Mac M1–M4** | `~90–220 ms` |
-| **PyTorch `load_in_4bit=True`** *(NF4 + `fp16` Head)* | `~5.5 GB` | `~6.2 GB` | **Colab Free T4 (16GB) / RTX 3060–4060 (8GB+)** | `~180–280 ms` |
-| **`FastBrain-9B-Q8_0.gguf`** *(Full 8-bit GGUF)* | `9.66 GB` | `~11.0 GB` | **12GB–16GB GPU / Colab T4 / Mac 16GB+** | `~80–180 ms` |
-| **PyTorch `load_in_8bit=True`** *(INT8 + `bf16` Head)* | `~9.6 GB` | `~11.2 GB` | **Colab T4 (16GB) / RTX 4070 Ti (12GB+)** | `~130–210 ms` |
-| **Native `bfloat16` (`safetensors` / `BF16.gguf`)** | `18.82 GB` (`18.16 GB`) | `20.3–24.7 GB` | **Colab L4 (24GB) / RTX 3090–4090 / PRO 6000 (96GB)** | **`38.8 ms` (`B=1`) · `61.7 ms` (`B=16` on G4 6000)** |
+## GGUF Quantization Accuracy Retention & Speed Benchmarks
+
+### 1. Decision Index 10-Benchmark Test-Set Accuracy Retention (`BF16` vs. `Q8_0` vs. `Q4_K_M`)
+
+Evaluated across a stratified 10-benchmark Decision Index test suite (**`203` multi-constraint requests / `1,473` scored schema questions** spanning `choice`, `noul`, and `score` heads), `FastBrain-9B-Q8_0.gguf` (`9.66 GB`) is **virtually lossless (`99.86%` Top-1 decision agreement with native `bfloat16`, `-0.03 pp` score delta)**, while `FastBrain-9B-Q4_K_M.gguf` (`6.73 GB`, with `dec.*` head preserved in `Q8_0`) retains **`98.78%` Top-1 agreement** and **`-0.20 pp`** calibrated 10-task mean:
+
+| Benchmark / Metric | Native PyTorch `bfloat16` (`18.82 GB`) | `FastBrain-9B-Q8_0.gguf` (`9.66 GB`) | `FastBrain-9B-Q4_K_M.gguf` (`6.73 GB`, `Q8_0` Head) |
+| :--- | :---: | :---: | :---: |
+| **Top-1 Decision Agreement vs. `bfloat16`** (`1,473` questions) | `100.00%` *(Reference)* | **`99.86%` (`1,471 / 1,473` identical)** | **`98.78%` (`1,455 / 1,473` identical)** |
+| **Probability Mean Absolute Error (`|p_quant - p_bf16|`)** | `0.00000` | **`0.00203` (`0.20%`)** | **`0.00387` (`0.39%`)** |
+| **10-Benchmark Mean (Single-Checkpoint Raw)** | **`82.37%`** | **`82.34%` (`-0.03 pp`)** | **`82.56%` (`+0.19 pp`)** |
+| **10-Benchmark Mean (With Margin Calibration)** | **`83.68%`** | **`83.65%` (`-0.03 pp`)** | **`83.48%` (`-0.20 pp`)** |
+| ├─ **1. BFCL** *(AST Tool Selection)* | `100.00%` | **`100.00%`** (`0.00`) | **`100.00%`** (`0.00`) |
+| ├─ **2. ToolRet** *(20-Candidate Tool Reranking, `score`)* | `73.15%` | **`73.70%`** (`+0.55`) | **`75.60%`** (`+2.45`) |
+| ├─ **3. API-Bank** *(Multi-Step Stateful Tool Calling)* | `95.00%` | **`95.00%`** (`0.00`) | **`95.00%`** (`0.00`) |
+| ├─ **4. BANKING77** *(77-Way Fine-Grained Intent)* | `100.00%` | **`100.00%`** (`0.00`) | **`100.00%`** (`0.00`) |
+| ├─ **5. CLINC150+OOS** *(Intent + Open-Set `noul` Abstention)* | `93.94%` | **`93.94%`** (`0.00`) | **`93.94%`** (`0.00`) |
+| ├─ **6. Home Appliances** *(Embedded Device Control)* | `90.00%` | **`90.00%`** (`0.00`) | **`90.00%`** (`0.00`) |
+| ├─ **7. BRIGHT** *(Reasoning-Intensive Document Ranking)* | `41.93%` | **`41.00%`** (`-0.93`) | **`40.31%`** (`-1.62`) |
+| ├─ **8. Amazon ESCI** *(Fine-Grained E-Commerce Relevance)* | `73.89%` | **`73.89%`** (`0.00`) | **`75.79%`** (`+1.90`) |
+| ├─ **9. PhishNChips** *(9-Probe Joint Security Inspection)* | `85.00%` | **`85.00%`** (`0.00`) | **`85.00%`** (`0.00`) |
+| └─ **10. When2Call** *(4-Way Tool Governance)* | `75.00%` *(Raw `70.83%`)* | **`75.00%`** *(Raw `70.83%`, `0.00`)* | **`72.92%`** *(Raw `70.83%`, `-2.08`)* |
+
+### 2. Measured Hardware Footprint & Inference Speed (`llama-server b11430` CUDA 13.4)
+
+Because `FastBrain-9B` scores all schema questions and candidate options in a **single non-autoregressive prefill forward pass (`output_tokens = 0`)**, `llama-server` achieves **8,900–12,100+ input tokens/sec** (`31–109 ms` end-to-end HTTP latency) on server GPUs:
+
+| Workload / Metric | `FastBrain-9B-Q8_0.gguf`<br>*(9.66 GB · RTX PRO 6000)* | `FastBrain-9B-Q4_K_M.gguf`<br>*(6.73 GB · RTX PRO 6000)* | Native PyTorch `bfloat16`<br>*(18.82 GB · RTX PRO 6000)* |
+| :--- | :---: | :---: | :---: |
+| **Model Load Time** | **`2.56 s`** | **`2.00 s`** | `~11.5 s` |
+| **Peak GPU VRAM** *(incl. `16K` context & batch buffers)* | **`12.55 GiB` (`12,847 MiB`)** | **`10.30 GiB` (`10,545 MiB`)** | `20.3 – 24.7 GiB` |
+| **Workload 1: Intent Routing + OOS** (`293 tok`, `K=2` questions) | **`32.82 ms`** (`8,928 tok/s` · `60.9 dec/s`) | **`31.69 ms`** (`9,245 tok/s` · `63.1 dec/s`) | `~35.0 ms` |
+| **Workload 2: Tool Governance (`When2Call`)** (`1,068 tok`, `K=1`, `M=4`) | **`87.78 ms`** (`12,167 tok/s`) | **`89.34 ms`** (`11,955 tok/s`) | `~92.0 ms` |
+| **Workload 3: Security Multi-Probe (`PhishNChips`)** (`1,303 tok`, `K=9` questions) | **`108.55 ms`** (`12,004 tok/s` · `82.9 dec/s`) | **`109.97 ms`** (`11,849 tok/s` · `81.8 dec/s`) | `~115.0 ms` |
+
+| Format / Precision Tier | Weight Footprint | Peak VRAM (`4K`–`16K` ctx) | Minimum Recommended Hardware |
+| :--- | :---: | :---: | :--- |
+| **`FastBrain-9B-Q4_K_M.gguf`** *(4-bit + `Q8_0` Head)* | `6.73 GB` | `~7.8 – 10.3 GiB` | **8GB–12GB GPU / Colab Free T4 (16GB) / Mac M1–M4** |
+| **PyTorch `load_in_4bit=True`** *(NF4 + `fp16` Head)* | `~5.5 GB` | `~6.2 – 6.8 GiB` | **Colab Free T4 (16GB) / RTX 3060–4060 (8GB+)** |
+| **`FastBrain-9B-Q8_0.gguf`** *(Full 8-bit GGUF)* | `9.66 GB` | `~10.8 – 12.6 GiB` | **12GB–16GB GPU / Colab T4 / Mac 16GB+** |
+| **PyTorch `load_in_8bit=True`** *(INT8 + `bf16` Head)* | `~9.6 GB` | `~11.2 GiB` | **Colab T4 (16GB) / RTX 4070 Ti (12GB+)** |
+| **Native `bfloat16` (`safetensors` / `BF16.gguf`)** | `18.82 GB` (`18.16 GB`) | `20.3 – 24.7 GiB` | **Colab L4 (24GB) / RTX 3090–4090 / PRO 6000 (96GB)** |
 
 ---