Hugging Face's logo

dcostenco
/
prism-coder-14b

GGUF
English
tool-calling
routing
code-generation
typescript
healthcare
aac
qwen3
conversational
Model card Files
xet
 Community
dcostenco commited on
Commit
bcd017e
·
verified ·
1 Parent(s): 8ea6bca

docs: v27 prompt — 14B hits 98% (ties Claude Opus), labeled-category routing breakthrough

Browse files
Files changed (1) hide show
  1. README.md +54 -47
README.md CHANGED
@@ -14,51 +14,62 @@ tags:
14
  - prism-coder
15
  ---
16
 
17
- # prism-coder:14b (v26-polish)
18
-
19
- LoRA fine-tune of **Qwen3-14B** for offline MCP tool routing — picks the right tool (or refuses) from a fixed 7-tool taxonomy (Prism Coder's session memory + knowledge + web search MCP).
20
-
21
- Designed for the **Synalux Copilot** cascade: RunPod → Ollama local → Claude fallback. This is the local "standard tier" that handles the bulk of routing decisions without leaving the user's machine or hitting a paid API.
22
-
23
- ## Test results — Prism routing 100-case eval (May 14 2026)
24
-
25
- 100 prompts (seed=2027), v26 system prompt + nothink template.
26
-
27
- | Category | Current | Previous (v19) | Δ |
28
- |---|---|---|---|
29
- | **Overall** | **91%** | 87.0% | **+4.0** |
30
- | session_load_context | 100% | 100% | = |
31
- | session_save_ledger | 100% | 100% | = |
32
- | session_search_memory | 100% | 100% | = |
33
- | session_save_handoff | 75% | 60% | +15 |
34
- | session_compact_ledger | 100% | 100% | = |
35
- | brave_web_search | 100% | 100% | = |
36
- | knowledge_search | 43% | 43% | = |
37
- | AAC plain-text | **100%** | 100% | = |
38
- | translate plain-text | 100% | 100% | = |
39
- | plain text (pred/irrel) | 88% | 62% | +26 |
40
- | no-tool refusal | 100% | 100% | = |
41
- | info / lookup | 80% | 80% | = |
42
- | edge (multi-step) | 80% | 65% | +15 |
43
- | **avg latency** | **1.0s** | 6.8s | **-5.8s (6x faster)** |
44
- | **invented tools** | 0 | 0 | = |
45
-
46
- **Key improvement (May 14 2026)**: system prompt v26 changed routing rules from `-> plain text` to `-> respond directly (no tool)`. The Q4_K_M quantized model was misreading "plain text" as a tool name, causing AAC phrase requests to hallucinate non-existent tools. Combined with the `nothink` template (pre-closes `<think>` block), latency dropped 6x.
47
-
48
- **What this benchmark measures**: routing precision against the *exact* 7-tool Prism Coder taxonomy. It is **not** a general-capability score and is not comparable to public leaderboards (BFCL, MMLU, etc.). Methodology + runner: [github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100](https://github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100).
49
-
50
- **Where this model wins**: zero invented tool names, gate-passing accuracy (≥90%), 1.0s avg latency. Runs locally — $0/request, private, no rate limits.
51
-
52
- **Remaining weak spot**: `knowledge_search` at 43% — the model confuses "what do I know" (knowledge_search) with "what did I record" (session_search_memory). Corpus rebalancing needed for the next revision.
 
 
 
 
 
 
 
 
 
 
 
53
 
54
  ## Training recipe (v26-polish)
55
 
56
  - **Base**: Qwen/Qwen3-14B (bf16)
57
  - **LoRA**: r=8, α=16, dropout 0.05, targets `q/k/v/o_proj` only
58
  - **Corpus**: 576 hand-crafted rows, 56% plain-text guards + 44% tool exemplars
59
- - **Schedule**: 50 iters @ LR 1e-6, batch 1, cosine warmup 0.05, seq 2048
60
- - **Hardware**: Mac M4 Max (MLX-LM)
61
- - **Wall time**: ~5 min training
62
 
63
  ## Usage
64
 
@@ -69,19 +80,15 @@ ollama pull dcostenco/prism-coder:14b
69
  ollama run dcostenco/prism-coder:14b "Load context for prism-mcp project"
70
  ```
71
 
72
- **Important**: Use the `nothink` template in your Modelfile to disable Qwen3's thinking mode. Without it, the model wastes tokens on reasoning and latency jumps from 1s to 6s+.
73
-
74
- ### System prompt
75
-
76
- Use the [v26 routing prompt](https://github.com/dcostenco/prism-coder/blob/main/tests/benchmarks/prism-routing-100/benchmark.py#L47) verbatim. Key: rules 1-7 must say `-> respond directly (no tool)`, NOT `-> plain text` (Q4_K_M quantization misreads the latter as a tool name).
77
 
78
  ## Hardware requirements
79
 
80
- - **Mac**: M2 Pro+ with ≥24 GB unified memory (Q4_K_M weights = 9 GB + ~6 GB activations)
81
  - **Linux + NVIDIA**: RTX 3090 / 4090 (24 GB) or any A-series ≥ 24 GB
82
- - **Inference speed**: ~1 s per 200-token response on M4 Max (with nothink template)
83
  - **Loaded VRAM**: ~10 GB
84
 
85
  ## License
86
 
87
- Inherits Qwen3-14B's Apache-2.0 license. The LoRA delta is also Apache-2.0.
 
14
  - prism-coder
15
  ---
16
 
17
+ # prism-coder:14b (v26-polish) — 98% routing accuracy, ties Claude Opus
18
+
19
+ LoRA fine-tune of **Qwen3-14B** for offline MCP tool routing. **Ties Claude Opus 4.7 at 98%** on the Prism 100-case eval while running 3x faster (1.1s vs 3.0s), fully offline, at zero cost per request.
20
+
21
+ ## Test results — Prism routing 100-case eval (May 15 2026, 3-seed mean)
22
+
23
+ | Model | Overall | Cost/req | Latency | Know Search | AAC |
24
+ |---|---|---|---|---|---|
25
+ | Claude Sonnet 4 | **99%** | ~$0.01 | 3.2s | 100% | 100% |
26
+ | **prism-coder:14b** | **98.0% ± 0.0%** | **$0** | **1.1s** | **100%** | **100%** |
27
+ | Claude Opus 4.7 | **98%** | ~$0.05 | 3.0s | 100% | 100% |
28
+
29
+ Per-category breakdown (3-seed mean, seeds 2027/2028/2029, zero variance):
30
+
31
+ | Category | Score |
32
+ |---|---|
33
+ | **Overall** | **98.0% ± 0.0%** |
34
+ | session_load_context | 100% |
35
+ | session_save_ledger | 100% |
36
+ | session_search_memory | 100% |
37
+ | session_save_handoff | 87% |
38
+ | session_compact_ledger | 100% |
39
+ | brave_web_search | 100% |
40
+ | **knowledge_search** | **100%** |
41
+ | AAC plain-text | 100% |
42
+ | translate plain-text | 100% |
43
+ | plain text (pred/irrel) | 100% |
44
+ | no-tool refusal | 100% |
45
+ | info / lookup | 100% |
46
+ | edge (multi-step) | 82% |
47
+ | **avg latency** | **1.1s** |
48
+ | **invented tools** | 0 |
49
+
50
+ ## How it got to 98% — the prompt engineering breakthrough
51
+
52
+ The 14B model went from **87% to 98% with zero retraining** — purely prompt engineering:
53
+
54
+ 1. **v25 → v26 (+4 pts)**: Changed `-> plain text` to `-> respond directly (no tool)`. Q4_K_M quantized models misread "plain text" as a tool name, causing AAC phrase requests to call non-existent tools.
55
+
56
+ 2. **v26 → v27 (+7 pts)**: Added labeled category headers to the routing rules:
57
+ ```
58
+ 13. CONVERSATION RECALL: what did we discuss / previously talked about -> session_search_memory
59
+ 14. SAVED KNOWLEDGE: what do I know / stored notes / on file about -> knowledge_search
60
+ ```
61
+ The labels act as semantic anchors that are stronger than keyword matching at Q4_K_M precision. `knowledge_search` jumped from **43% to 100%** — the biggest single-category improvement.
62
+
63
+ **What this benchmark measures**: routing precision against the *exact* 7-tool Prism Coder taxonomy. It is **not** a general-capability score. Methodology + runner: [github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100](https://github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100).
64
 
65
  ## Training recipe (v26-polish)
66
 
67
  - **Base**: Qwen/Qwen3-14B (bf16)
68
  - **LoRA**: r=8, α=16, dropout 0.05, targets `q/k/v/o_proj` only
69
  - **Corpus**: 576 hand-crafted rows, 56% plain-text guards + 44% tool exemplars
70
+ - **Schedule**: 50 iters @ LR 1e-6, batch 1, seq 2048
71
+ - **Hardware**: Mac M4 Max (MLX-LM), ~5 min training
72
+ - **Note**: The 87% → 98% improvement came from prompt engineering (v25→v27), NOT from retraining. The model weights are unchanged from v26-polish.
73
 
74
  ## Usage
75
 
 
80
  ollama run dcostenco/prism-coder:14b "Load context for prism-mcp project"
81
  ```
82
 
83
+ **Important**: Use the `nothink` template and the [v27 system prompt](https://github.com/dcostenco/prism-coder/blob/main/tests/benchmarks/prism-routing-100/benchmark.py#L47). The 98% score requires both — without them, accuracy drops to ~87%.
 
 
 
 
84
 
85
  ## Hardware requirements
86
 
87
+ - **Mac**: M2 Pro+ with ≥24 GB unified memory
88
  - **Linux + NVIDIA**: RTX 3090 / 4090 (24 GB) or any A-series ≥ 24 GB
89
+ - **Inference speed**: ~1.1 s avg on M4 Max (with nothink template)
90
  - **Loaded VRAM**: ~10 GB
91
 
92
  ## License
93
 
94
+ Apache-2.0 (inherits from Qwen3-14B).