Hugging Face's logo

dcostenco
/
prism-coder-32b

Text Generation
GGUF
English
tool-calling
function-calling
prism
synalux
memory-augmented
LoRA
Q4_K_M
conversational
Model card Files
xet
 Community
1
dcostenco commited on
Commit
7b5e164
·
verified ·
1 Parent(s): 986e0d5

docs: updated benchmark scores — v26 system prompt + nothink template (May 14 2026)

Browse files
Files changed (1) hide show
  1. README.md +33 -52
README.md CHANGED
@@ -20,55 +20,46 @@ LoRA fine-tune of **Qwen/QwQ-32B** for offline MCP tool routing — Synalux Copi
20
 
21
  ## Test results — Prism routing 100-case eval (May 14 2026)
22
 
23
- 3-run mean across seeds 2027/2028/2029 — variance was essentially zero (±0.6%).
24
-
25
- | Category | Score |
26
- |---|---|
27
- | **Overall** | **93.7% ± 0.6%** |
28
- | session_load_context | 100% |
29
- | session_save_ledger | 100% |
30
- | session_search_memory | 100% |
31
- | session_save_handoff | 100% |
32
- | session_compact_ledger | 100% |
33
- | brave_web_search | 100% |
34
- | knowledge_search | 100% |
35
- | AAC plain-text | 79% |
36
- | translate plain-text | 83% |
37
- | static facts (pred) | 100% |
38
- | live-info refusal | 67% |
39
- | info / lookup | 100% |
40
- | edge (multi-step) | 82% |
41
- | **avg latency** | 2.3 s |
42
- | **invented tools** | 0 |
43
-
44
- **This is currently the only Prism Coder model that clears the internal 90% routing gate.**
45
-
46
- **What this benchmark measures**: routing precision against the *exact* 7-tool Prism Coder taxonomy. **Not** a general-capability score. Not comparable to public leaderboards. Methodology + runner: [github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100](https://github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100).
47
-
48
- **Where this model wins**:
49
- - **All 7 tools routed correctly 100%** of the time (perfect score on every tool category)
50
- - Zero invented tool names
51
- - ~2.3 s average latency on a Mac M4 Max — comparable to Claude Sonnet (3.2 s) and Opus (3.0 s) despite running locally
52
- - Free per-request, private, no rate limits
53
-
54
- **Where it underperforms vs Claude Sonnet 4 / Opus 4.7** (99% / 98% on the same eval):
55
- - `irrel` (live-info refusal) — 67% vs 100%. Sometimes calls `brave_web_search` for "I'm hungry" / "What time is it?" instead of replying in plain text.
56
- - `aac` — 79% vs 100%. Occasionally tries to route AAC phrase requests to a tool instead of generating phrases directly.
57
- - `translate` — 83% vs 100%. Same pattern — over-eager tool calls on plain-text intents.
58
-
59
- For production: the [Synalux router](https://github.com/dcostenco/prism-coder) routes complex prompts here and falls through to Claude when this model refuses or invokes the wrong tool.
60
 
61
  ## Notes on QwQ-32B base
62
 
63
- QwQ-32B is a reasoning-tuned variant that natively emits `<think>...</think>` blocks. For routing, that's overhead — the Ollama Modelfile uses a `nothink` template (empty `<think></think>` block in the assistant prefix) to skip reasoning and go straight to the tool call. Without `nothink`: 97% (single-seed); with `nothink` + surgical prompt disambiguation: 93.7% stable. See [`Modelfile.32b`](https://github.com/dcostenco/prism-coder/blob/main/tests/benchmarks/prism-routing-100/Modelfile.32b).
64
 
65
  ## Training recipe (v19)
66
 
67
  - **Base**: Qwen/QwQ-32B
68
- - **LoRA**: r=32, α=64, dropout 0.05, all 7 target modules (q/k/v/o + gate/up/down)
69
  - **Corpus**: ~14K-row composite (Phase 1 general + Phase 2 agentic + Phase 3 multi-turn XL)
70
  - **Hardware**: RunPod A100 80GB / RTX 6000 Ada
71
- - **Quantization**: published as Q4_K_M GGUF (~19 GB) and the merged HF safetensors
72
 
73
  ## Usage
74
 
@@ -79,25 +70,15 @@ ollama pull dcostenco/prism-coder:32b
79
  ollama run dcostenco/prism-coder:32b "Save handoff for prism-coder — deployment complete"
80
  ```
81
 
82
- ### HuggingFace (transformers + PEFT)
83
-
84
- ```python
85
- from transformers import AutoModelForCausalLM, AutoTokenizer
86
- from peft import PeftModel
87
- base = AutoModelForCausalLM.from_pretrained("Qwen/QwQ-32B", torch_dtype="auto")
88
- model = PeftModel.from_pretrained(base, "dcostenco/prism-coder-32b")
89
- tok = AutoTokenizer.from_pretrained("Qwen/QwQ-32B")
90
- ```
91
-
92
  ### System prompt
93
 
94
- Use the [v25 routing prompt](https://github.com/dcostenco/prism-coder/blob/main/tests/benchmarks/prism-routing-100/benchmark.py#L47) verbatim, with the `nothink` template (see `Modelfile.32b`).
95
 
96
  ## Hardware requirements
97
 
98
  - **Mac**: M2 Ultra+ with ≥48 GB unified memory (Q4_K_M = 19 GB + activations)
99
  - **Linux + NVIDIA**: A100 40GB+, H100, B200, or 2× RTX 4090
100
- - **Inference speed**: ~2–5 s per 200-token response (varies by hardware)
101
  - **Loaded VRAM**: ~22 GB
102
 
103
  ## License
 
20
 
21
  ## Test results — Prism routing 100-case eval (May 14 2026)
22
 
23
+ 100 prompts (seed=2027), v26 system prompt + nothink template.
24
+
25
+ | Category | Current | Previous (v19 old prompt) | Δ |
26
+ |---|---|---|---|
27
+ | **Overall** | **98%** | 93.7% | **+4.3** |
28
+ | session_load_context | 100% | 100% | = |
29
+ | session_save_ledger | 100% | 100% | = |
30
+ | session_search_memory | 100% | 100% | = |
31
+ | session_save_handoff | 100% | 100% | = |
32
+ | session_compact_ledger | 100% | 100% | = |
33
+ | brave_web_search | 100% | 100% | = |
34
+ | knowledge_search | 100% | 100% | = |
35
+ | AAC plain-text | **100%** | 79% | **+21** |
36
+ | translate plain-text | 83% | 83% | = |
37
+ | plain text (pred/irrel) | 100% | 67% | +33 |
38
+ | no-tool refusal | 100% | 100% | = |
39
+ | info / lookup | 100% | 100% | = |
40
+ | edge (multi-step) | 80% | 82% | -2 |
41
+ | **avg latency** | **2.7s** | 2.3s | +0.4s |
42
+ | **invented tools** | 0 | 0 | = |
43
+
44
+ **Key improvement (May 14 2026)**: system prompt v26 eliminates Q4_K_M quantization artifacts where "plain text" was misread as a tool name. AAC routing jumped from 79% to 100% — critical for the life-critical AAC use case.
45
+
46
+ **98% puts this model within 1 point of Claude Sonnet 4 (99%) on the same eval**, while running fully offline on a Mac.
47
+
48
+ Only 2 misroutes in 100 cases: "Convert 'good morning' to Japanese" → brave_web_search (edge case), and a multi-step ledger query.
49
+
50
+ **What this benchmark measures**: routing precision against the *exact* 7-tool Prism Coder taxonomy. **Not** a general-capability score. Methodology + runner: [github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100](https://github.com/dcostenco/prism-coder/tree/main/tests/benchmarks/prism-routing-100).
 
 
 
 
 
 
 
 
 
51
 
52
  ## Notes on QwQ-32B base
53
 
54
+ QwQ-32B natively emits `<think>...</think>` blocks. The Ollama Modelfile uses a `nothink` template (pre-closes the `<think>` block) to skip reasoning and go straight to the tool call.
55
 
56
  ## Training recipe (v19)
57
 
58
  - **Base**: Qwen/QwQ-32B
59
+ - **LoRA**: r=32, α=64, dropout 0.05, all 7 target modules
60
  - **Corpus**: ~14K-row composite (Phase 1 general + Phase 2 agentic + Phase 3 multi-turn XL)
61
  - **Hardware**: RunPod A100 80GB / RTX 6000 Ada
62
+ - **Quantization**: published as Q4_K_M GGUF (~19 GB) and merged HF safetensors
63
 
64
  ## Usage
65
 
 
70
  ollama run dcostenco/prism-coder:32b "Save handoff for prism-coder — deployment complete"
71
  ```
72
 
 
 
 
 
 
 
 
 
 
 
73
  ### System prompt
74
 
75
+ Use the [v26 routing prompt](https://github.com/dcostenco/prism-coder/blob/main/tests/benchmarks/prism-routing-100/benchmark.py#L47) with the `nothink` template.
76
 
77
  ## Hardware requirements
78
 
79
  - **Mac**: M2 Ultra+ with ≥48 GB unified memory (Q4_K_M = 19 GB + activations)
80
  - **Linux + NVIDIA**: A100 40GB+, H100, B200, or 2× RTX 4090
81
+ - **Inference speed**: ~2–3 s per 200-token response
82
  - **Loaded VRAM**: ~22 GB
83
 
84
  ## License