---
license: cc-by-nc-4.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
tags: [function-calling, tool-use, lora, qlora, peft]
datasets: [Salesforce/xlam-function-calling-60k]
language: [en]
---

# qwen2.5-1.5b-function-calling-lora

A LoRA adapter fine-tuning **Qwen/Qwen2.5-1.5B-Instruct** for structured function-calling / tool-call generation. Given a set of available tools and a natural-language query, the model emits a JSON list of function calls.

## Results

Evaluated on a held-out test set of 499 examples with greedy decoding, using a BFCL-style AST match (strict function-name + argument comparison). No LLM judge.

| Metric | Base | Fine-tuned | Δ |
|---|---|---|---|
| JSON validity | 74.5% | 99.4% | +24.9% |
| Function-name accuracy | 43.5% | 98.4% | +54.9% |
| **Exact-match (AST)** | **31.1%** | **78.8%** | **+47.7%** |

## Intended prompt format

The adapter was trained on one specific format — the tool schemas go in the system message and the model replies with a raw JSON list. Use this exact shape for best results:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import json

base  = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct", device_map="auto")
model = PeftModel.from_pretrained(base, "Arunark/qwen2.5-1.5b-function-calling-lora")
tok   = AutoTokenizer.from_pretrained("Arunark/qwen2.5-1.5b-function-calling-lora")

SYS = ('You are a function-calling assistant. Given the available tools, respond with '
       'ONLY a JSON list of calls, each {"name": <tool>, "arguments": {<args>}}. '
       'If nothing applies, respond with []. Tools:\n')

tools = [{"name": "get_weather", "parameters": {"city": {"type": "string"}}}]
messages = [{"role": "system", "content": SYS + json.dumps(tools)},
            {"role": "user",   "content": "What is the weather in Paris?"}]

prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
enc = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Training

- **Method:** QLoRA (4-bit nf4), LoRA r=16, α=32, dropout=0.05
- **Target modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- **Data:** 4999 examples from `Salesforce/xlam-function-calling-60k`, 3 epochs, effective batch 16, lr 0.0002, max_length 1024
- **Hardware:** single GPU, ~35 min (fits free-tier Colab with T4/QLoRA)

## Error analysis

Of 106/499 exact-match misses (21.2%), the breakdown by failure type:

- `wrong_arg_values`: 63 (59% of misses)
- `wrong_arg_keys`: 33 (31% of misses)
- `wrong_function_name`: 4 (4% of misses)
- `invalid_json`: 3 (3% of misses)
- `other`: 2 (2% of misses)
- `wrong_call_count`: 1 (1% of misses)

Fine-tuning eliminated format and function-selection errors (JSON validity 99.4%, name accuracy 98.4%). Residual 21% error is 90% argument-level: wrong values (unknown API enum codes, e.g. category '4093' vs '6017', 'uk' vs ISO 'gb') and wrong/missing keys. Type-coercion artifacts are negligible (+0.2%), so strict exact-match is a faithful metric. Type-insensitive scoring changes the result by only 0.2%, so the strict metric is faithful.

## Limitations

- Fine-tuned on a research-only dataset; intended for research/demonstration.
- Residual errors are almost entirely argument-level (wrong API values or keys), not format or tool-selection.
- Small (1.5B) model; not a substitute for larger tool-calling models in production.

## Attribution

Base model: `Qwen/Qwen2.5-1.5B-Instruct` (Apache-2.0). Training data: `Salesforce/xlam-function-calling-60k` (research/non-commercial). Adapter released under CC-BY-NC-4.0 to respect the dataset's terms.