qwen2.5-1.5b-function-calling-lora

A LoRA adapter fine-tuning Qwen/Qwen2.5-1.5B-Instruct for structured function-calling / tool-call generation. Given a set of available tools and a natural-language query, the model emits a JSON list of function calls.

Results

Evaluated on a held-out test set of 499 examples with greedy decoding, using a BFCL-style AST match (strict function-name + argument comparison). No LLM judge.

Metric	Base	Fine-tuned	Δ
JSON validity	74.5%	99.4%	+24.9%
Function-name accuracy	43.5%	98.4%	+54.9%
Exact-match (AST)	31.1%	78.8%	+47.7%

Intended prompt format

The adapter was trained on one specific format — the tool schemas go in the system message and the model replies with a raw JSON list. Use this exact shape for best results:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import json

base  = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "Arunark/qwen2.5-1.5b-function-calling-lora")
tok   = AutoTokenizer.from_pretrained("Arunark/qwen2.5-1.5b-function-calling-lora")

SYS = ('You are a function-calling assistant. Given the available tools, respond with '
       'ONLY a JSON list of calls, each {"name": <tool>, "arguments": {<args>}}. '
       'If nothing applies, respond with []. Tools:\n')

tools = [{"name": "get_weather", "parameters": {"city": {"type": "string"}}}]
messages = [{"role": "system", "content": SYS + json.dumps(tools)},
            {"role": "user",   "content": "What is the weather in Paris?"}]

prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
enc = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Training

• Method: QLoRA (4-bit nf4), LoRA r=16, α=32, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Data: 4999 examples from Salesforce/xlam-function-calling-60k, 3 epochs, effective batch 16, lr 0.0002, max_length 1024
• Hardware: single GPU, ~35 min (fits free-tier Colab with T4/QLoRA)

Error analysis

Of 106/499 exact-match misses (21.2%), the breakdown by failure type:

• wrong_arg_values: 63 (59% of misses)
• wrong_arg_keys: 33 (31% of misses)
• wrong_function_name: 4 (4% of misses)
• invalid_json: 3 (3% of misses)
• other: 2 (2% of misses)
• wrong_call_count: 1 (1% of misses)

Fine-tuning eliminated format and function-selection errors (JSON validity 99.4%, name accuracy 98.4%). Residual 21% error is 90% argument-level: wrong values (unknown API enum codes, e.g. category '4093' vs '6017', 'uk' vs ISO 'gb') and wrong/missing keys. Type-coercion artifacts are negligible (+0.2%), so strict exact-match is a faithful metric. Type-insensitive scoring changes the result by only 0.2%, so the strict metric is faithful.

Limitations

• Fine-tuned on a research-only dataset; intended for research/demonstration.
• Residual errors are almost entirely argument-level (wrong API values or keys), not format or tool-selection.
• Small (1.5B) model; not a substitute for larger tool-calling models in production.

Attribution

Base model: Qwen/Qwen2.5-1.5B-Instruct (Apache-2.0). Training data: Salesforce/xlam-function-calling-60k (research/non-commercial). Adapter released under CC-BY-NC-4.0 to respect the dataset's terms.