| --- |
| base_model: Qwen/Qwen3-8B-Base |
| library_name: peft |
| license: gpl-3.0 |
| datasets: |
| - Salesforce/xlam-function-calling-60k |
| model-index: |
| - name: tool-concierge |
| results: |
| - task: |
| type: text-generation |
| dataset: |
| type: gorilla-llm/Berkeley-Function-Calling-Leaderboard |
| name: BFCL |
| metrics: |
| - type: accuracy |
| name: call_acc |
| value: 0.67 |
| - task: |
| type: text-generation |
| name: MCP Tool-Call Generation |
| dataset: |
| type: Nexusflow/NexusRaven_API_evaluation |
| name: Nexus |
| metrics: |
| - type: accuracy |
| name: call_acc |
| value: 0.71 |
| language: |
| - en |
| tags: |
| - mcp |
| - model-context-protocol |
| - function-calling |
| - tool-use |
| - lora |
| - qwen3 |
| --- |
| |
| # Tool Concierge (Qwen3-8B-Base + LoRA for MCP Tool-Call Generation) |
|
|
| ## 1. Introduction |
|
|
| The Model Context Protocol (MCP) became a cross-vendor standard "in a very short time span" ([Anthropic, 2024](https://www.anthropic.com/news/model-context-protocol)). Models that hallucinate tool names or malform arguments turn these integrations into silent failures ([Patil et al., 2023](https://arxiv.org/abs/2305.15334)). Routing every tool-use decision through a frontier model is also costly at scale, and small open-weight models fine-tuned on curated function-calling data can match or beat larger ones ([Erdogan et al., 2024](https://arxiv.org/abs/2409.00608)). I fine-tuned [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base) with LoRA to generate a single MCP tool call, or an explicit `no_applicable_tool` abstention, from a plain-English request and a set of MCP tool definitions. Gorilla ([Patil et al., 2023](https://arxiv.org/abs/2305.15334)) trains against ML model-hub APIs, and TinyAgent ([Erdogan et al., 2024](https://arxiv.org/abs/2409.00608)) targets generic JSON tool calls; I narrowed scope instead to the MCP schema and treated abstention as a trained output rather than a fallback. Fine-tuning raised own-test-split call accuracy from 523/929 to 698/929 and abstention accuracy from 9/31 to 22/31. |
|
|
| ## 2. Data |
|
|
| I pulled positive training rows from [Salesforce/xlam-function-calling-60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) (60,000 verified examples across 3,673 APIs and 21 domains), sampling 9,300 of them (`random_state=5002`) and reformatting xlam's inline tool schema into MCP-style `inputSchema` JSON, with each `answers[0]` reshaped into a single `{"name":..., "arguments":...}` call. I self-generated the negative (no-applicable-tool) examples, but only landed 300 cleaned records in `negatives_data.json` instead of the roughly 4,000 I'd planned - the generator model had to be swapped from a 70B to a 14B model after a disk-quota issue, which cut the negative target "to fit a hard deadline." That left about a 97/3 positive/negative split instead of the planned 70/30. I shuffled the combined set (`frac=1, random_state=5002`) and split it 80% train / 10% eval / 10% test, producing `train_split.json`, `eval_split.json`, and a 960-record `test_split.json`; the actual training run further subsampled to 2,500 train rows and 300 eval rows (`random_state=5002`) for time. |
|
|
| ## 3. Methodology |
|
|
| I chose [LoRA](https://arxiv.org/abs/2106.09685) for compute efficiency on a single A100, applying adapters to `q_proj` and `v_proj` (`bias="none"`, `task_type="CAUSAL_LM"`) across three hyperparameter combinations, each trained for 1 epoch at `learning_rate=1e-4` with batch size 1 (train and eval), 2048-token max sequence length, and evaluation/checkpointing every 1000 steps with `load_best_model_at_end=True`. The three combos I tested were `r=64, alpha=64, dropout=0.05` (combo1_high_capacity), `r=16, alpha=16, dropout=0.10` (combo2_low_capacity), and `r=32, alpha=64, dropout=0.15` (combo3_mid_capacity). combo1_high_capacity won on final eval loss (0.2866 vs. 0.2868 for combo3 and 0.3099 for combo2), so I saved its adapter and used it for all reported evaluation results. |
|
|
| ## 4. Evaluation |
|
|
| I used four benchmarks to cover different failure modes. [BFCL](https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard) scored tool-call correctness and abstention on a mix of simple and irrelevance examples, [Nexus](https://huggingface.co/datasets/Nexusflow/NexusRaven_API_evaluation) scored single-turn function calling against real-world API definitions, [MMLU](https://huggingface.co/datasets/cais/mmlu) served as a catastrophic-forgetting check with a plain multiple-choice prompt and no scaffold, and `own_test_split` scored the model on my own held-out MCP-formatted test rows. I don't have comparison-model benchmarks in this same suite - I ran a smaller in-context comparison of `Qwen2.5-1.5B-Instruct`, `Llama-3.1-8B-Instruct`, and `Qwen2.5-14B-Instruct` in CheckIn3, but on only 2 fixed examples with call-accuracy scoring, so it doesn't map onto this table. Fine-tuning produced a large gain on own_test_split abstention accuracy, a sizable gain on Nexus, a smaller gain on BFCL, and a small drop on MMLU. |
|
|
| | Benchmark | This Model (posttrain) | Base Model (pretrain) | Comparison Model 1 | Comparison Model 2 | |
| |---|---|---|---|---| |
| | [BFCL](https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard) | call_acc 67/100, abstention_acc 72/100, false_call 28/100 | call_acc 58/100, abstention_acc 44/100, false_call 22/100 | TODO -- no comparison model benchmark data found in source files | TODO -- no comparison model benchmark data found in source files | |
| | [Nexus](https://huggingface.co/datasets/Nexusflow/NexusRaven_API_evaluation) | call_acc 142/200 | call_acc 43/200 | TODO -- no comparison model benchmark data found in source files | TODO -- no comparison model benchmark data found in source files | |
| | [MMLU](https://huggingface.co/datasets/cais/mmlu) | acc 107/500 (21.4%) | acc 121/500 (24.2%) | TODO -- no comparison model benchmark data found in source files | TODO -- no comparison model benchmark data found in source files | |
| | own_test_split | call_acc 698/929, abstention_acc 22/31 | call_acc 523/929, abstention_acc 9/31 | TODO -- no comparison model benchmark data found in source files | TODO -- no comparison model benchmark data found in source files | |
|
|
| ## 5. Usage and Intended Uses |
|
|
| This model handles single-step MCP tool-call generation only; it isn't built for multi-step orchestration or planning loops. Load it as a PEFT/LoRA adapter on top of the Qwen3-8B-Base checkpoint, using the snippet below. |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| base_model_id = "Qwen/Qwen3-8B-Base" |
| adapter_repo_id = "tool-concierge" # replace with your-username/tool-concierge |
| |
| tokenizer = AutoTokenizer.from_pretrained(base_model_id) |
| base_model = AutoModelForCausalLM.from_pretrained(base_model_id) |
| model = PeftModel.from_pretrained(base_model, adapter_repo_id) |
| ``` |
|
|
| ## 6. Prompt Format |
|
|
| I used the fixed "Least-to-Most Prompting" scaffold below ([Zhou et al., 2022](https://arxiv.org/abs/2205.10625)) for every training row and eval prompt, except MMLU, which used a plain multiple-choice prompt instead. |
|
|
| ``` |
| SCAFFOLD_TEMPLATE = """\ |
| You generate a single MCP tool call for a user request. Work through the steps in order, using the answer to each step in the next one. |
| Step 1. Restate the user's goal in one sentence. |
| Step 2. List the available tools. For each tool, check its inputSchema and decide whether it can satisfy the goal. |
| Step 3. Pick the one tool that fits, or decide that none apply. |
| Step 4. If a tool fits, read its inputSchema and pull each argument value from the request, matching the schema's types and including every required field. If no tool fits, prepare an abstention. |
| Step 5. Output the final answer as a single JSON object and nothing else. For a tool call: {{"name": <tool>, "arguments": {{...}}}}. If no tool applies: {{"name": "no_applicable_tool", "arguments": {{"reason": <short reason>}}}}. |
| |
| User request: |
| {query} |
| |
| Available MCP tools: |
| {tools_json}""" |
| ``` |
|
|
| ## 7. Expected Output Format |
|
|
| The model's final line is a single JSON object with a tool name and typed arguments. Below is a real posttrain example from `posttrain_samples.json` (`own_test_split[0]`), shown as-is, including its recorded `correct: false` score: |
|
|
| ```json |
| {"name": "generate_random_string", "arguments": {"length": 10, "uppercase": true, "lowercase": true, "digits": true}} |
| ``` |
|
|
| ## 8. Limitations |
|
|
| MMLU accuracy dropped from 121/500 (24.2%) to 107/500 (21.4%) after fine-tuning. Both numbers sit close to the 25% chance rate for four-option questions, so the signal is weak, though the drop is directionally consistent with LoRA's catastrophic-forgetting risk. BFCL's false-call rate rose from 22/100 to 28/100 even as abstention accuracy improved to 72/100: pretrain only resolved 66/100 irrelevance examples to a parseable abstain-or-false-call output at all, versus all 100 posttrain, so I read this as an output-format fix rather than better abstention judgment. own_test_split abstention accuracy improved from 9/31 to 22/31, but 9 of 31 cases were still wrong. My training corpus ended up at roughly 9,300 positive to 300 negative rows instead of the ~70/30 mix I'd planned, the split ended at 80/10/10 instead of the planned 80/20, and I subsampled the final run to 2,500 train rows and 300 eval rows - all deadline-driven. I only benchmarked the winning combo (combo1) post-training, so I couldn't compare the MMLU drop across combos. The Nexus call-accuracy jump (43/200 to 142/200) was much larger than BFCL's (58/100 to 67/100); this could be specific to Nexus's toolset format or parser rather than a uniform training effect. A smaller CheckIn3 pilot BFCL run (n=40) produced different pretrain numbers (19/40 call_acc, 22/40 abstention_acc) than the n=100 run reported here - I'm chalking that up to sample-size variance. These results only test the MCP schema framing. I didn't run a plain-function-calling ablation to compare against it, which matters given prior work questioning whether MCP reliably outperforms plain function-calling ([Luo et al., 2025](https://arxiv.org/abs/2504.11094)). |
|
|
| ## Citations |
|
|
| Anthropic. (2024, November 25). *Introducing the Model Context Protocol*. https://www.anthropic.com/news/model-context-protocol |
|
|
| Erdogan, L. E., Lee, N., Jha, S., Kim, S., Tabrizi, R., Moon, S., Hooper, C., Anumanchipalli, G., Keutzer, K., & Gholami, A. (2024). *TinyAgent: Function calling at the edge*. arXiv. https://arxiv.org/abs/2409.00608 |
|
|
| Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). *LoRA: Low-rank adaptation of large language models*. arXiv. https://arxiv.org/abs/2106.09685 |
|
|
| Luo, Z., Shi, X., Lin, X., & Gao, J. (2025). *Evaluation report on MCP servers*. arXiv. https://arxiv.org/abs/2504.11094 |
|
|
| Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). *Gorilla: Large language model connected with massive APIs*. arXiv. https://arxiv.org/abs/2305.15334 |
|
|
| Qwen Team. (2024). *Qwen3-8B-Base* [Model]. Hugging Face. https://huggingface.co/Qwen/Qwen3-8B-Base |
|
|
| Salesforce. (n.d.). *xlam-function-calling-60k* [Dataset]. Hugging Face. https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k |
|
|
| Zhou, D., et al. (2022). *Least-to-most prompting enables complex reasoning in large language models*. arXiv. https://arxiv.org/abs/2205.10625 |
|
|
