Document fused MLX and GGUF distributions

README.md

@@ -5,6 +5,8 @@ pipeline_tag: text-generation
 license: apache-2.0
 tags:
   - mlx
+  - gguf
+  - quantized
   - lora
   - qwen3-next
   - orchestrator
@@ -13,12 +15,14 @@ tags:
 
 # bankai-v1
 
-`bankai-v1` is an MLX LoRA adapter for
+`bankai-v1` is a one-step agent orchestration model based on
 [`Qwen/Qwen3-Coder-Next`](https://huggingface.co/Qwen/Qwen3-Coder-Next), trained
 to perform one-step agent orchestration. Given a task, it selects one specialist
 worker and emits a complete delegated instruction as strict JSON.
 
-The adapter was trained against
+This repository includes the original MLX LoRA, a llama.cpp GGUF LoRA, a fully
+fused 4-bit MLX model, and a fully fused Q4_K_M GGUF model. The adapter was
+trained against
 [`mlx-community/Qwen3-Coder-Next-4bit`](https://huggingface.co/mlx-community/Qwen3-Coder-Next-4bit).
 The upstream model is an 80B MoE with approximately 3B active parameters.
 
@@ -26,7 +30,7 @@ The upstream model is an 80B MoE with approximately 3B active parameters.
 
 | Property | Value |
 |---|---|
-| Model type | One-step agent orchestration adapter |
+| Model type | One-step agent orchestration model and adapters |
 | Upstream architecture | Qwen3-Next hybrid Gated DeltaNet / attention MoE |
 | Total base parameters | 79,674,391,296 (79.674B) |
 | Active parameters per token | Approximately 3B |
@@ -45,6 +49,19 @@ experts per token, and one shared expert. `bankai-v1` adapts only attention and
 gated-delta projections in the final 16 layers; it does not add LoRA matrices
 to the routed experts.
 
+## Repository Contents
+
+| Path | Type | Size | Standalone |
+|---|---|---:|---|
+| `adapters.safetensors` | Original MLX LoRA | 11,439,496 bytes | No |
+| `adapter_config.json` | MLX LoRA configuration | 1.3 KB | No |
+| `gguf/bankai-v1-f16.gguf` | llama.cpp F16 LoRA | 6,116,192 bytes | No |
+| `gguf/bankai-v1-Q4_K_M.gguf` | Fused Q4_K_M GGUF | 48,528,320,544 bytes | Yes |
+| `mlx/` | Fused 4-bit MLX model, nine shards | Approximately 44.9 GB | Yes |
+| `bankai-system.txt` | Exact training-time system prompt | 1 KB | N/A |
+| `training_config.yaml` | Reproducible training configuration | 746 bytes | N/A |
+| `evaluation.json` | Per-example behavioral evaluation | 30 KB | N/A |
+
 ## Output Contract
 
 ```json
@@ -74,6 +91,31 @@ Supported worker labels:
 
 ## Usage
 
+Download the exact system prompt:
+
+```bash
+hf download ApplauseLab/bankai-v1 bankai-system.txt --local-dir bankai-v1
+```
+
+### Fused MLX Model
+
+The fused MLX model does not require an adapter:
+
+```bash
+hf download ApplauseLab/bankai-v1 \
+  --include "mlx/*" \
+  --local-dir bankai-v1
+
+mlx_lm.generate \
+  --model bankai-v1/mlx \
+  --system-prompt "$(cat bankai-v1/bankai-system.txt)" \
+  --prompt "Design a GDPR-compliant CRM integration for a real-estate business." \
+  --max-tokens 1024 \
+  --temp 0
+```
+
+### MLX Adapter
+
 Install MLX LM on Apple Silicon:
 
 ```bash
@@ -89,14 +131,66 @@ Generate with the adapter:
 mlx_lm.generate \
   --model mlx-community/Qwen3-Coder-Next-4bit \
   --adapter-path bankai-v1-adapter \
-  --system-prompt "You are BankAI, a concise orchestration model. Select exactly one specialist worker for the user's task and return only the call_agent JSON contract." \
+  --system-prompt "$(cat bankai-system.txt)" \
   --prompt "Design and test a GDPR-compliant CRM integration for a real-estate business." \
   --max-tokens 1024 \
   --temp 0
 ```
 
-For best consistency, use the complete system prompt from `training_config.yaml`
-or the source repository.
+For best consistency, use the included `bankai-system.txt`.
+
+### Fused GGUF / llama.cpp
+
+The fused GGUF is standalone and does not require `--lora`:
+
+```bash
+hf download ApplauseLab/bankai-v1 \
+  gguf/bankai-v1-Q4_K_M.gguf \
+  --local-dir bankai-v1
+
+llama-cli \
+  --model bankai-v1/gguf/bankai-v1-Q4_K_M.gguf \
+  --system-prompt-file bankai-v1/bankai-system.txt \
+  --prompt "Design a GDPR-compliant CRM integration for a real-estate business." \
+  --temp 0 \
+  --n-predict 1024 \
+  --single-turn
+```
+
+### GGUF Adapter
+
+`bankai-v1-f16.gguf` is the same LoRA adapter converted for llama.cpp. It is
+not a standalone 80B model; load it alongside a compatible
+Qwen3-Coder-Next GGUF base:
+
+```bash
+llama-cli \
+  --model Qwen3-Coder-Next-Q4_K_M.gguf \
+  --lora bankai-v1/gguf/bankai-v1-f16.gguf \
+  --system-prompt-file bankai-v1/bankai-system.txt \
+  --prompt "Design a GDPR-compliant CRM integration for a real-estate business." \
+  --temp 0 \
+  --n-predict 1024
+```
+
+The GGUF adapter was converted through PEFT tensor orientation with
+`lora_alpha=128`, preserving the MLX effective scale of 16 at rank 8.
+
+Use the included `bankai-system.txt` for the exact worker vocabulary and JSON
+contract seen during fine-tuning. Shorter prompts may produce valid routing in
+a different schema.
+
+## Fusion Details
+
+The fused MLX model was produced by merging the LoRA into
+`mlx-community/Qwen3-Coder-Next-4bit` and saving the resulting quantized MLX
+weights.
+
+The standalone GGUF starts from the single-file Unsloth Q4_K_M conversion of
+Qwen3-Coder-Next. Untouched tensors are copied bit-for-bit. The 64 targeted
+attention and gated-delta matrices in layers 32-47 are dequantized to F32, the
+LoRA delta is applied at effective scale 16, and each matrix is requantized to
+its original Q4_K_M mixture type. The resulting GGUF contains 843 tensors.
 
 ## Training
 
@@ -140,6 +234,12 @@ Only assistant tokens contributed to the loss.
 The behavioral sample is small and measures formatting and category routing,
 not end-to-end quality of worker execution.
 
+Both fused formats received inference smoke tests with the full system prompt.
+The fused MLX output matched the JSON contract. The fused GGUF output also
+matched the contract, selected `compliance_specialist` for a held-out
+real-estate GDPR/CRM/accounting request, and generated at approximately 64
+tokens/second on an Apple M4 Max using Metal offload.
+
 ## Limitations
 
 This first version learns only single-step worker selection and prompt
@@ -179,8 +279,11 @@ used the checksummed final `training_set.jsonl` artifact.
 
 ## Integrity
 
-SHA-256 of `adapters.safetensors`:
+| File | SHA-256 |
+|---|---|
+| `adapters.safetensors` | `66df3ea1d9a9f0b1c3e07b1620a33dfb0b3bf36b6ea4b1e52fe2fc1bd8da67d9` |
+| `gguf/bankai-v1-f16.gguf` | `184b7855166c6c97fe366b4412de51cf98474dfc5df7caa95bda09f83c80e84d` |
+| `gguf/bankai-v1-Q4_K_M.gguf` | `75f8200b83756e6f32565b4579c9bb07aa6aa591fca380bc3f85af1138102296` |
 
-```text
-66df3ea1d9a9f0b1c3e07b1620a33dfb0b3bf36b6ea4b1e52fe2fc1bd8da67d9
-```
+The fused GGUF also passed llama.cpp per-tensor hashing with whole-file XXH64
+`c64819733a364e4d`.