turnercore's picture
Link benchmark dataset from model card
cf41862 verified
|
Raw
History Blame Contribute Delete
1.69 kB
metadata
license: apache-2.0
base_model: openbmb/MiniCPM5-1B
tags:
  - function-calling
  - tool-use
  - minicpm
  - automaticity

MiniCPM5 Automaticity V7

Public release of the MiniCPM5 automaticity V7 LoRA and merged GGUF exports.

The training data and benchmark repos are private because future rows/results may contain real tool-call traces. The included HTML report and benchmark dataset repo compare:

  • MiniCPM5_Base
  • MiniCPM5_Base + AUTOMATICITY_V7_LORA_ADAPTER
  • MiniCPM5_AUTOMATICITY_V7_Q4
  • MiniCPM5_AUTOMATICITY_V7_Q8
  • FunctionGemma_AUTOMATICITY_V7_Q8

MiniCPM tool-call fine-tuning targets MiniCPM XML function calls rather than JSON.

Automaticity benchmark v1

Frozen local benchmark: /home/turnercore/automaticity-benchmark-v1/automaticity-hard-v1.jsonl

Canonical benchmark repo: turnercore/automaticity-benchmark-v1 (private)

Run Exact Tool name Arguments No-op recall p50 latency p95 latency
FunctionGemma_AUTOMATICITY_V7_Q8 82/92 (89.1%) 96.7% 90.2% 94.7% 180 ms 568 ms
MiniCPM5_Base 78/92 (84.8%) 92.4% 87.0% 86.8% 701 ms 2,070 ms
MiniCPM5_Base + AUTOMATICITY_V7_LORA_ADAPTER 59/92 (64.1%) 67.4% 75.0% 21.1% 316 ms 478 ms
MiniCPM5_AUTOMATICITY_V7_Q8 60/92 (65.2%) 69.6% 75.0% 26.3% 151 ms 248 ms
MiniCPM5_AUTOMATICITY_V7_Q4 54/92 (58.7%) 64.1% 67.4% 13.2% 141 ms 226 ms

The V7 MiniCPM fine-tune is not promoted. It learned the XML output shape and is fast when baked to GGUF, but it overcalls no-op, negated, hypothetical, deferred, and incomplete prompts. AUTOMATICITY_V8 adds no-op hardening rows for the next run.