πŸ› οΈ MiniCPM5-2B-Supertool

RACER IS OP

A 2.5B tool-calling model: openbmb/MiniCPM5-2B fine-tuned to drive the Claude Code tool surface β€” Bash, Read, Write, Edit, Glob, Grep, WebFetch, WebSearch, TodoWrite, TaskUpdate, ToolSearch β€” emitting calls in the base model's native XML form:

<function name="Bash"><param name="command">ls tasks/</param></function>

Weights are full precision with the LoRA already merged. Load it directly:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("saidutta69/MiniCPM5-2B-Supertool")
model = AutoModelForCausalLM.from_pretrained("saidutta69/MiniCPM5-2B-Supertool")

No PEFT, no adapter step. Pass your tool schemas to apply_chat_template(tools=...) as you would with the base model.

Benchmarks

FC-Eval β€” independent, held-out APIs

saidutta69/fc-eval-benchmark, eval split: 89 APIs, none seen in training. 150 tasks sampled round-robin across APIs, greedy decoding, both models at fp16. This is the one result here that does not come from the training corpus.

Metric Base Supertool
Task success (exact call) 49.3% 77.3%
Tool name correct 50.0% 90.0%
Argument keys correct 50.0% 87.3%
Format failures 50.0% 8.7%
Wall time, 150 tasks 1071 s 886 s

The base is all-or-nothing: it emits a parseable call exactly half the time, and when it does the call is exactly right (success == name == arg-keys == 0.50). Supertool parses 91.3% of the time and is exact on 77.3%. Most of the gain is format compliance and tool choice; argument values improve less.

In-domain agent traces

120 examples from the training dataset's own validation split, never trained on. Greedy, max_new=384, both models 4-bit on one Tesla T4.

Metric Base Supertool
Completion loss 1.5691 1.1246
Tool name correct 47.5% 54.2%
Argument names correct 31.7% 46.7%
Full match 0.0% 4.2%
Parseable call emitted 88.3% 75.8%

That last row looks like a regression and is not one. Every failing Supertool example is a Write call cut off inside its content argument by the generation cap: this model writes an actual file body, while the base emits a shorter, vaguer call that happens to fit. Raising the cap from 160 to 384 lifted the base from 81.7% to 88.3% while Supertool moved 64.2% to 75.8% β€” truncation, not non-compliance. Among calls it does emit:

Base Supertool
Tool name correct 53.8% 71.4%
Argument names correct 35.8% 61.5%

Training

Base openbmb/MiniCPM5-2B
Method QLoRA β€” NF4 4-bit, LoRA r=16 alpha=32 dropout 0
LoRA targets q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Data saidutta69/fable-5.1-premium-v2, claude_code split only
Examples 42,514 β€” one per assistant tool-calling turn
Coverage ~3,500 of 4,132 trajectories, 12 examples each
Validated calls 347,136 β€” zero bad JSON, zero missing required fields
Supervision final assistant turn only, via a prompt/completion split
Schedule 240 steps, effective batch 16, seq 4096, lr 2e-4 cosine, warmup 3%
Loss 1.107 β†’ 0.276
Hardware 2Γ— Tesla T4 (sm_75), fp16, 7 h 40 m, peak 8.92 GB per card

Data

Traces originate from MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x; credit for them belongs to MoreThought. They were repackaged into saidutta69/fable-5.1-premium-v2 and then adapted for this model:

  • kept only the claude_code share; dropped the anonymised half (unknown_tool)
  • dropped tool_calls with no matching tool response, and orphan tool messages
  • re-serialised messages canonically and de-duplicated by SHA-256 over messages plus tool names
  • reconstructed the 14-tool Claude Code schema surface, validated every call against it, dropped rows that did not fit
  • excluded the fable_code configuration: 203,750 of its 203,794 calls carry the schema rendered as the value ({"path": "<file_path>"}), which teaches a model to emit placeholders
  • sampled 12 examples per trajectory so coverage spans ~3,500 rows instead of 212

Limitations

  • Not a general assistant. A 2.5B specialist that emits tool calls; it is weaker than the base at open-ended conversation.
  • One tool family. Trained on Claude Code's surface, not arbitrary schemas. The FC-Eval result shows real cross-schema transfer, not generalisation to any API.
  • Exact match is low in-domain β€” 5 of 120. Long Write payloads are approximated.
  • One pass, never looped. 240 steps is roughly three quarters of a single epoch.
  • No GGUF / llama.cpp build published yet.
  • Held-out loss is worse than an earlier narrower run of the same recipe (1.125 vs 0.976) while tool accuracy is better: wider data memorises less and generalises more.

Responsible use

No guardrails. This model was not trained or evaluated for prompt-injection resistance, and it will follow instructions that arrive inside tool output. Validate its calls before letting it act on anything destructive, and do not expose it to untrusted tool results without a policy layer in front.

Licence

Apache-2.0.

openbmb/MiniCPM5-2B states in its card that the repository and MiniCPM model weights are Apache-2.0, with a link to the LICENSE in the OpenBMB GitHub repository. The training data declares Apache-2.0 in saidutta69/fable-5.1-premium-v2, inherited from MoreThought, which declares the same.

Both declarations are made in cards: neither dataset repository ships a standalone LICENSE file, and the MoreThought source carries the tag with no separate terms or attribution section. The Apache-2.0 claim for these weights therefore rests on those upstream declarations. Stated plainly here rather than implied.

Apache-2.0 requires keeping the licence and notices and indicating changes β€” the change list above is provided for that purpose.

πŸ”— Related

Made with care by RACER IS OP (saidutta69).

Downloads last month
87
Safetensors
Model size
3B params
Tensor type
F16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for saidutta69/MiniCPM5-2B-Supertool

Finetuned
(93)
this model
Finetunes
1 model
Quantizations
2 models

Datasets used to train saidutta69/MiniCPM5-2B-Supertool

Collection including saidutta69/MiniCPM5-2B-Supertool