Automaticity Benchmark
MiniCPM got faster, but the v7 LoRA overcalls no-op prompts.
On the same 92-row automaticity benchmark, the incumbent FunctionGemma v7 Q8 remains the leader at 82/92 exact. MiniCPM5 base is surprisingly strong at 78/92 exact; the MiniCPM5 v7 LoRA variants fall behind because no-op recall drops sharply.
FunctionGemma_AUTOMATICITY_V7_Q8, 82/92 exact.
MiniCPM5_Base, 78/92 exact before fine-tuning.
Wall-clock for MiniCPM5 base + LoRA training + Q4 export. Trainer loop was 249.9 seconds; Q8 export-only was about 25 seconds.
Materialized with 1,070 train rows and 565 no-op rows for overcall hardening.
Comparison
| Run Label | Kind | Exact | Tool Name | Arguments | No-op Recall | p50 | p95 | Failures |
|---|---|---|---|---|---|---|---|---|
| FunctionGemma_AUTOMATICITY_V7_Q8 | Leader baked GGUF Q8 | 96.7% | 90.2% | 94.7% | 180 ms | 568 ms | 7 wrong args, 3 wrong tool | |
| MiniCPM5_Base | Best MiniCPM HF base | 92.4% | 87.0% | 86.8% | 701 ms | 2,070 ms | 7 wrong args, 7 wrong tool | |
| MiniCPM5_Base + AUTOMATICITY_V7_LORA_ADAPTER | LoRA Hat unmerged HF adapter | 67.4% | 75.0% | 21.1% | 316 ms | 478 ms | 3 wrong args, 30 wrong tool | |
| MiniCPM5_AUTOMATICITY_V7_Q8 | Merged baked GGUF Q8 | 69.6% | 75.0% | 26.3% | 151 ms | 248 ms | 4 wrong args, 28 wrong tool | |
| MiniCPM5_AUTOMATICITY_V7_Q4 | Merged baked GGUF Q4 | 64.1% | 67.4% | 13.2% | 141 ms | 226 ms | 5 wrong args, 33 wrong tool |
Interpretation
MiniCPM5 base is the better MiniCPM target today. The v7 LoRA learned to emit MiniCPM XML or compact XML fragments and became much faster in GGUF form, but it lost the base model's restraint on hypothetical, negated, deferred, and partial prompts.
Training Target
MiniCPM fine-tuning is using MiniCPM XML tool calls, not JSON. The parser accepts full XML and the compact fragments the exported model often emits. SGLang's native MiniCPM path should stay aligned with this XML convention.
Why The Earlier 52.5% Number Differed
The 52.5% MiniCPM result came from the older 120-row FunctionGemma spine benchmark. This report uses the newer 92-row automaticity-hard benchmark, so those percentages are not the same population.
Next Action
Train `AUTOMATICITY_V8` before promoting MiniCPM. The v8 dataset has added no-op-heavy contrastive rows and should be judged against the same 92 frozen rows with FunctionGemma v7 Q8 included as the leader row.
Artifacts
- /home/turnercore/automaticity-training-v8/automaticity-train-v8.jsonl · 1,070 rows, 565 no-op rows.
- /home/turnercore/automaticity-benchmark-v1/automaticity-hard-v1.jsonl · frozen 92-row benchmark.
- /tmp/ai-gateway/training/adapters/minicpm5-automaticity-v1 · unmerged LoRA adapter tested as the LoRA hat row.
- /tmp/ai-gateway/training/gguf/minicpm5-automaticity-v1-q8/q8_0_gguf/MiniCPM5-1B.Q8_0.gguf · MiniCPM5_AUTOMATICITY_V7_Q8.
- /tmp/ai-gateway/training/gguf/minicpm5-automaticity-v1-q4/q4_k_m_gguf/MiniCPM5-1B.Q4_K_M.gguf · MiniCPM5_AUTOMATICITY_V7_Q4.