README.md

---
library_name: transformers
tags: []
---

language:
- ar
tags:
- function-calling
- tool-use
- arabic
- instruction-tuning
- gemma
- transformers
license: apache-2.0
base_model: google/functiongemma-270m-it
---

# FunctionGemma-270M Arabic Tool Use

This model is a finetuned version of **`google/functiongemma-270m-it`** for Arabic **tool use / function calling** across multiple dialects and domains.

It is trained to produce **exactly one tool call** when a tool is required, using **FunctionGemma-native tool formatting** (special function-call tokens) and structured JSON arguments.

## Base model
- `google/functiongemma-270m-it`

## Dataset
- `metga97/arabic-tooluse-functiongemma-v1`

## What the model outputs

When a tool is required, generation should include a FunctionGemma tool call pattern such as:

- `<start_function_call>call:TOOL_NAME{ ...json args... }<end_function_call>`

For non-tool requests, it returns a short Arabic reply.

## Evaluation (by slang / dialect)

Evaluated on the test split of `metga97/arabic-tooluse-functiongemma-v1`.

### Overall
- Parsed OK rate: **0.891**
- Tool name accuracy: **0.9921**
- Strict EM: **0.6564**
- Key-F1 (avg): **0.9925**
- Missed-call rate: **0.0064**
- False-call rate (negatives): **0.0**

### Strict EM by slang / dialect
- **Egyptian**: 0.6791 (denom_calls: 1069)
- **Gulf**: 0.6237 (denom_calls: 1172)
- **Levantine**: 0.6558 (denom_calls: 706)
- **MSA**: 0.6804 (denom_calls: 1408)
- **Maghrebi**: 0.5455 (denom_calls: 176)

### Strict EM by domain
- banking_finance: 0.6255 (denom_calls: 542)
- ecommerce: 0.64 (denom_calls: 550)
- government_services: 0.7651 (denom_calls: 613)
- healthcare: 0.5754 (denom_calls: 577)
- islamic_services: 0.7119 (denom_calls: 597)
- travel: 0.6028 (denom_calls: 564)
- utilities: 0.4652 (denom_calls: 561)
- weather: 0.8653 (denom_calls: 527)

## Inference (important)

### 1) Use left padding for decoder-only generation
Set:
- `tokenizer.padding_side = "left"`
- `tokenizer.pad_token = tokenizer.eos_token` (if missing)

### 2) Pass tools via `apply_chat_template(..., tools=tools_list)`
This is critical for FunctionGemma-style function calling.

Example outline:
1. Select a tool subset for the request (domain pack + deterministic sampling).
2. Build prompt with `apply_chat_template` including `tools=tools_list`.
3. `generate()` deterministically (`do_sample=False`, `temperature=0.0`).
4. Parse tool call tokens and arguments.

## Known limitations / improvement ideas

- Some outputs may translate slot values into English (e.g., “Abu Dhabi”, “ID renewal”).
  - Mitigations: stronger developer prompt constraints, post-processing, adding explicit anti-translation supervision, and/or filtering/rebalancing training examples where values are English.
- Parsed OK < 1.0: you can improve formatting consistency with:
  - longer training
  - slightly stronger prompt
  - adding more negative/no-tool examples with explicit non-tool responses