---
license: gemma
base_model: unsloth/functiongemma-270m-it
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- function-calling
- tool-use
- mobile-automation
- android
- gemma
- unsloth
- agent
---

# FunctionGemma-270M — Mobile Actions

A fine-tune of [`unsloth/functiongemma-270m-it`](https://huggingface.co/unsloth/functiongemma-270m-it)
that turns a natural-language instruction into a **single, validated mobile-action tool call**.
It targets two uses:

- **Mobile test automation** — drive an Appium / UiAutomator / Espresso / XCUITest layer from plain-English steps.
- **Agentic workflows** — let an on-device agent pick the next action as a structured, executable call.

At 270M parameters it runs on CPU, a modest GPU, or fully on-device (LiteRT-LM / Google AI Edge).

## Repository layout

| Path | Contents |
| --- | --- |
| `merged_16bit/` | Standalone 16-bit model (load this for inference) |
| `lora/` | LoRA adapter only (apply on top of the base model) |

## Output format

The model emits FunctionGemma's call DSL, e.g.:

```
<start_function_call>call:tap{selector:<escape>login_button<escape>}<end_function_call>
```

Parse it with:

```python
import re
CALL = re.compile(r"<start_function_call>call:(\w+)\{(.*?)\}<end_function_call>", re.DOTALL)
ARG  = re.compile(r"(\w+):(?:<escape>(.*?)<escape>|([^,}]*))")

def parse(text):
    out = []
    for name, raw in CALL.findall(text):
        args = {k: (e or p).strip() for k, e, p in ARG.findall(raw)}
        out.append({"name": name, "arguments": args})
    return out
```

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "ahmadw/functiongemma-270m-mobile"
tok = AutoTokenizer.from_pretrained(REPO, subfolder="merged_16bit")
model = AutoModelForCausalLM.from_pretrained(REPO, subfolder="merged_16bit").eval()

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Tap the login button."}],
    tools=TOOLS,                 # the schema below
    tokenize=False, add_generation_prompt=True,
).removeprefix("<bos>")

inputs = tok(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# -> <start_function_call>call:tap{selector:<escape>login_button<escape>}<end_function_call>
```

You **must** pass the `tools` schema (below) so the chat template declares the available actions.

### Reliable tool calls (recommended)

For production, constrain decoding so the model can only emit a syntactically valid call naming one
of your tools (prevents hallucinated names). Pair the model with
[`xgrammar`](https://pypi.org/project/xgrammar/) — build an EBNF grammar whose function name is
restricted to your tool list, pass it as a `LogitsProcessor` to `generate(...)`, and snap any
out-of-schema name to the nearest valid tool as a dependency-free fallback.

### Wiring into an automation driver

The model decides *what*; your driver decides *how*. Map each tool name to a concrete action:

```python
HANDLERS = {
    "launch_app": lambda d, app_name:       d.activate_app(app_name),
    "tap":        lambda d, selector:       d.find(selector).click(),
    "type_text":  lambda d, selector, text: d.find(selector).send_keys(text),
    "swipe":      lambda d, direction:      d.swipe(direction),
    "wait":       lambda d, duration_ms:    d.sleep(duration_ms / 1000),
    # ... one entry per tool ...
}
```

> Selectors come from the live UI. The model infers a *reasonable* id (e.g. `login_button`) from
> the wording; resolve it against the real accessibility tree, or pass the available element ids in
> the instruction so the model can pick an exact match.

## Supported actions (38 tools)

| Category | Tools |
| --- | --- |
| App lifecycle | `launch_app`, `close_app` |
| Touch | `tap`, `double_tap`, `long_press`, `tap_coordinates`, `drag`, `swipe`, `pinch`, `scroll_to`, `pull_to_refresh` |
| Text input | `type_text`, `clear_text`, `paste_text`, `copy_text`, `dismiss_keyboard` |
| Controls | `toggle_switch`, `set_checkbox`, `select_option`, `set_slider`, `set_date`, `set_time`, `set_volume` |
| Navigation | `navigate_back`, `go_home`, `open_app_switcher`, `open_notifications`, `open_quick_settings`, `press_keyboard_key`, `rotate_screen` |
| Synchronization | `wait`, `wait_for_element` |
| Assertions | `assert_element_visible`, `assert_element_not_visible`, `assert_text_equals`, `assert_text_contains` |
| Utility | `take_screenshot`, `open_url` |

<details>
<summary>Full tool schema (copy into <code>TOOLS</code>)</summary>

```json
[
  {
    "type": "function",
    "function": {
      "name": "launch_app",
      "description": "Launch a mobile application by its visible name.",
      "parameters": {
        "type": "object",
        "properties": {
          "app_name": {
            "type": "string",
            "description": "Visible name of the app to open."
          }
        },
        "required": [
          "app_name"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "close_app",
      "description": "Close or force stop a running application.",
      "parameters": {
        "type": "object",
        "properties": {
          "app_name": {
            "type": "string",
            "description": "Visible name of the app to close."
          }
        },
        "required": [
          "app_name"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "tap",
      "description": "Tap a single UI element once by its selector. Use tap_coordinates for raw screen pixels.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id or label of the element."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "double_tap",
      "description": "Double tap a UI element.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id or label of the element."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "long_press",
      "description": "Press and hold a UI element.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id or label of the element."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "tap_coordinates",
      "description": "Tap the screen at absolute pixel x/y coordinates, with no element selector.",
      "parameters": {
        "type": "object",
        "properties": {
          "x": {
            "type": "integer",
            "description": "Horizontal pixel coordinate."
          },
          "y": {
            "type": "integer",
            "description": "Vertical pixel coordinate."
          }
        },
        "required": [
          "x",
          "y"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "drag",
      "description": "Drag one element onto another.",
      "parameters": {
        "type": "object",
        "properties": {
          "source": {
            "type": "string",
            "description": "Logical id of the element to drag."
          },
          "target": {
            "type": "string",
            "description": "Logical id of the drop target."
          }
        },
        "required": [
          "source",
          "target"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "swipe",
      "description": "Swipe across the screen in a direction.",
      "parameters": {
        "type": "object",
        "properties": {
          "direction": {
            "type": "string",
            "description": "One of: up, down, left, right."
          }
        },
        "required": [
          "direction"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "pinch",
      "description": "Pinch with two fingers to zoom in or out on an element. Does not rotate the screen.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the element to zoom."
          },
          "direction": {
            "type": "string",
            "description": "Zoom direction: in or out."
          }
        },
        "required": [
          "selector",
          "direction"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "scroll_to",
      "description": "Scroll until an element becomes visible.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the target element."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "pull_to_refresh",
      "description": "Pull down from the top to refresh the screen.",
      "parameters": {
        "type": "object",
        "properties": {},
        "required": []
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "type_text",
      "description": "Type text into an input field.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the input field."
          },
          "text": {
            "type": "string",
            "description": "Text to enter into the field."
          }
        },
        "required": [
          "selector",
          "text"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "clear_text",
      "description": "Clear the contents of an input field.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the input field."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "paste_text",
      "description": "Paste the clipboard contents into an input field.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the input field."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "copy_text",
      "description": "Copy the text of an element to the clipboard.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the element to copy from."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "dismiss_keyboard",
      "description": "Hide the on-screen keyboard.",
      "parameters": {
        "type": "object",
        "properties": {},
        "required": []
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "toggle_switch",
      "description": "Set a toggle or switch to a desired state.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the switch."
          },
          "state": {
            "type": "string",
            "description": "Desired state: on or off."
          }
        },
        "required": [
          "selector",
          "state"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "set_checkbox",
      "description": "Check or uncheck a checkbox.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the checkbox."
          },
          "state": {
            "type": "string",
            "description": "Desired state: checked or unchecked."
          }
        },
        "required": [
          "selector",
          "state"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "select_option",
      "description": "Select an option from a dropdown or picker.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the dropdown."
          },
          "option": {
            "type": "string",
            "description": "Visible option to select."
          }
        },
        "required": [
          "selector",
          "option"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "set_slider",
      "description": "Set a slider to a numeric value.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the slider."
          },
          "value": {
            "type": "integer",
            "description": "Target value for the slider."
          }
        },
        "required": [
          "selector",
          "value"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "set_date",
      "description": "Set a date picker to a specific date.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the date field."
          },
          "date": {
            "type": "string",
            "description": "Date in YYYY-MM-DD format."
          }
        },
        "required": [
          "selector",
          "date"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "set_time",
      "description": "Set a time picker to a specific time.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the time field."
          },
          "time": {
            "type": "string",
            "description": "Time in HH:MM 24-hour format."
          }
        },
        "required": [
          "selector",
          "time"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "navigate_back",
      "description": "Go back one step to the previous screen. Does not return to the home screen.",
      "parameters": {
        "type": "object",
        "properties": {},
        "required": []
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "go_home",
      "description": "Return to the device home screen (launcher). This is not a single back step.",
      "parameters": {
        "type": "object",
        "properties": {},
        "required": []
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "open_app_switcher",
      "description": "Open the recent apps or app switcher.",
      "parameters": {
        "type": "object",
        "properties": {},
        "required": []
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "press_keyboard_key",
      "description": "Press a soft keyboard action key.",
      "parameters": {
        "type": "object",
        "properties": {
          "key": {
            "type": "string",
            "description": "One of: enter, search, done, next, go."
          }
        },
        "required": [
          "key"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "open_notifications",
      "description": "Open the notifications panel.",
      "parameters": {
        "type": "object",
        "properties": {},
        "required": []
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "open_quick_settings",
      "description": "Open the quick toggles or control panel.",
      "parameters": {
        "type": "object",
        "properties": {},
        "required": []
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "rotate_screen",
      "description": "Rotate the screen orientation between portrait and landscape. Does not zoom.",
      "parameters": {
        "type": "object",
        "properties": {
          "orientation": {
            "type": "string",
            "description": "Orientation: portrait or landscape."
          }
        },
        "required": [
          "orientation"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "set_volume",
      "description": "Set the media volume to a level.",
      "parameters": {
        "type": "object",
        "properties": {
          "level": {
            "type": "integer",
            "description": "Volume level from 0 to 100."
          }
        },
        "required": [
          "level"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "wait_for_element",
      "description": "Wait until a specific element appears, up to a timeout. Requires an element selector.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the element."
          },
          "timeout_ms": {
            "type": "integer",
            "description": "Maximum wait time in milliseconds."
          }
        },
        "required": [
          "selector",
          "timeout_ms"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "wait",
      "description": "Pause execution for a fixed duration in milliseconds. Does not wait for any element.",
      "parameters": {
        "type": "object",
        "properties": {
          "duration_ms": {
            "type": "integer",
            "description": "Duration to wait in milliseconds."
          }
        },
        "required": [
          "duration_ms"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "assert_element_visible",
      "description": "Assert that an element is visible on screen.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the element."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "assert_element_not_visible",
      "description": "Assert that an element is not visible on screen.",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the element."
          }
        },
        "required": [
          "selector"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "assert_text_equals",
      "description": "Assert that an element's text exactly equals an expected value (full match).",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the element."
          },
          "expected_text": {
            "type": "string",
            "description": "Expected text content."
          }
        },
        "required": [
          "selector",
          "expected_text"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "assert_text_contains",
      "description": "Assert that an element's text contains a substring (partial match).",
      "parameters": {
        "type": "object",
        "properties": {
          "selector": {
            "type": "string",
            "description": "Logical id of the element."
          },
          "text": {
            "type": "string",
            "description": "Substring expected within the element text."
          }
        },
        "required": [
          "selector",
          "text"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "take_screenshot",
      "description": "Capture a screenshot of the current screen.",
      "parameters": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "description": "File name for the screenshot."
          }
        },
        "required": [
          "name"
        ]
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "open_url",
      "description": "Open a URL in the default browser.",
      "parameters": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string",
            "description": "Fully qualified URL to open."
          }
        },
        "required": [
          "url"
        ]
      }
    }
  }
]
```

</details>

## Evaluation

Greedy + grammar-constrained decoding, on held-out paraphrase sets the model never saw in training:

| Set | Tool selection | Exact match (name + args) |
| --- | --- | --- |
| Wild (colloquial paraphrases) | 97.7% | 90.7% |
| Stress (high-variance, 64 cases) | 93.8% | 92.2% |

Hallucinated tool names: 0. Remaining misses cluster on indirect phrasings with no action keyword
(e.g. "make the picture smaller" -> `pinch` out). Accuracy is highest when the instruction names the action.

## Training

- Base: `unsloth/functiongemma-270m-it`
- Method: 16-bit LoRA (r=16, alpha=16; q/k/v/o + gate/up/down) via Unsloth, on a free Tesla T4
- LR 2e-4, linear schedule, `adamw_8bit`, 3 epochs, response-only loss masking
- Synthetic instruction -> tool-call data with contrastive confusable pairs and broad phrasing variety

## License

Governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). Inherits the base model license.