| --- |
| license: gemma |
| base_model: unsloth/functiongemma-270m-it |
| library_name: transformers |
| pipeline_tag: text-generation |
| language: |
| - en |
| tags: |
| - function-calling |
| - tool-use |
| - mobile-automation |
| - android |
| - gemma |
| - unsloth |
| - agent |
| --- |
| |
| # FunctionGemma-270M — Mobile Actions |
|
|
| A fine-tune of [`unsloth/functiongemma-270m-it`](https://huggingface.co/unsloth/functiongemma-270m-it) |
| that turns a natural-language instruction into a **single, validated mobile-action tool call**. |
| It targets two uses: |
|
|
| - **Mobile test automation** — drive an Appium / UiAutomator / Espresso / XCUITest layer from plain-English steps. |
| - **Agentic workflows** — let an on-device agent pick the next action as a structured, executable call. |
|
|
| At 270M parameters it runs on CPU, a modest GPU, or fully on-device (LiteRT-LM / Google AI Edge). |
|
|
| ## Repository layout |
|
|
| | Path | Contents | |
| | --- | --- | |
| | `merged_16bit/` | Standalone 16-bit model (load this for inference) | |
| | `lora/` | LoRA adapter only (apply on top of the base model) | |
|
|
| ## Output format |
|
|
| The model emits FunctionGemma's call DSL, e.g.: |
|
|
| ``` |
| <start_function_call>call:tap{selector:<escape>login_button<escape>}<end_function_call> |
| ``` |
|
|
| Parse it with: |
|
|
| ```python |
| import re |
| CALL = re.compile(r"<start_function_call>call:(\w+)\{(.*?)\}<end_function_call>", re.DOTALL) |
| ARG = re.compile(r"(\w+):(?:<escape>(.*?)<escape>|([^,}]*))") |
| |
| def parse(text): |
| out = [] |
| for name, raw in CALL.findall(text): |
| args = {k: (e or p).strip() for k, e, p in ARG.findall(raw)} |
| out.append({"name": name, "arguments": args}) |
| return out |
| ``` |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| REPO = "ahmadw/functiongemma-270m-mobile" |
| tok = AutoTokenizer.from_pretrained(REPO, subfolder="merged_16bit") |
| model = AutoModelForCausalLM.from_pretrained(REPO, subfolder="merged_16bit").eval() |
| |
| prompt = tok.apply_chat_template( |
| [{"role": "user", "content": "Tap the login button."}], |
| tools=TOOLS, # the schema below |
| tokenize=False, add_generation_prompt=True, |
| ).removeprefix("<bos>") |
| |
| inputs = tok(prompt, return_tensors="pt") |
| with torch.no_grad(): |
| out = model.generate(**inputs, max_new_tokens=128, do_sample=False) |
| print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| # -> <start_function_call>call:tap{selector:<escape>login_button<escape>}<end_function_call> |
| ``` |
|
|
| You **must** pass the `tools` schema (below) so the chat template declares the available actions. |
|
|
| ### Reliable tool calls (recommended) |
|
|
| For production, constrain decoding so the model can only emit a syntactically valid call naming one |
| of your tools (prevents hallucinated names). Pair the model with |
| [`xgrammar`](https://pypi.org/project/xgrammar/) — build an EBNF grammar whose function name is |
| restricted to your tool list, pass it as a `LogitsProcessor` to `generate(...)`, and snap any |
| out-of-schema name to the nearest valid tool as a dependency-free fallback. |
|
|
| ### Wiring into an automation driver |
|
|
| The model decides *what*; your driver decides *how*. Map each tool name to a concrete action: |
|
|
| ```python |
| HANDLERS = { |
| "launch_app": lambda d, app_name: d.activate_app(app_name), |
| "tap": lambda d, selector: d.find(selector).click(), |
| "type_text": lambda d, selector, text: d.find(selector).send_keys(text), |
| "swipe": lambda d, direction: d.swipe(direction), |
| "wait": lambda d, duration_ms: d.sleep(duration_ms / 1000), |
| # ... one entry per tool ... |
| } |
| ``` |
|
|
| > Selectors come from the live UI. The model infers a *reasonable* id (e.g. `login_button`) from |
| > the wording; resolve it against the real accessibility tree, or pass the available element ids in |
| > the instruction so the model can pick an exact match. |
| |
| ## Supported actions (38 tools) |
| |
| | Category | Tools | |
| | --- | --- | |
| | App lifecycle | `launch_app`, `close_app` | |
| | Touch | `tap`, `double_tap`, `long_press`, `tap_coordinates`, `drag`, `swipe`, `pinch`, `scroll_to`, `pull_to_refresh` | |
| | Text input | `type_text`, `clear_text`, `paste_text`, `copy_text`, `dismiss_keyboard` | |
| | Controls | `toggle_switch`, `set_checkbox`, `select_option`, `set_slider`, `set_date`, `set_time`, `set_volume` | |
| | Navigation | `navigate_back`, `go_home`, `open_app_switcher`, `open_notifications`, `open_quick_settings`, `press_keyboard_key`, `rotate_screen` | |
| | Synchronization | `wait`, `wait_for_element` | |
| | Assertions | `assert_element_visible`, `assert_element_not_visible`, `assert_text_equals`, `assert_text_contains` | |
| | Utility | `take_screenshot`, `open_url` | |
|
|
| <details> |
| <summary>Full tool schema (copy into <code>TOOLS</code>)</summary> |
|
|
| ```json |
| [ |
| { |
| "type": "function", |
| "function": { |
| "name": "launch_app", |
| "description": "Launch a mobile application by its visible name.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "app_name": { |
| "type": "string", |
| "description": "Visible name of the app to open." |
| } |
| }, |
| "required": [ |
| "app_name" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "close_app", |
| "description": "Close or force stop a running application.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "app_name": { |
| "type": "string", |
| "description": "Visible name of the app to close." |
| } |
| }, |
| "required": [ |
| "app_name" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "tap", |
| "description": "Tap a single UI element once by its selector. Use tap_coordinates for raw screen pixels.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id or label of the element." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "double_tap", |
| "description": "Double tap a UI element.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id or label of the element." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "long_press", |
| "description": "Press and hold a UI element.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id or label of the element." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "tap_coordinates", |
| "description": "Tap the screen at absolute pixel x/y coordinates, with no element selector.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "x": { |
| "type": "integer", |
| "description": "Horizontal pixel coordinate." |
| }, |
| "y": { |
| "type": "integer", |
| "description": "Vertical pixel coordinate." |
| } |
| }, |
| "required": [ |
| "x", |
| "y" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "drag", |
| "description": "Drag one element onto another.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "source": { |
| "type": "string", |
| "description": "Logical id of the element to drag." |
| }, |
| "target": { |
| "type": "string", |
| "description": "Logical id of the drop target." |
| } |
| }, |
| "required": [ |
| "source", |
| "target" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "swipe", |
| "description": "Swipe across the screen in a direction.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "direction": { |
| "type": "string", |
| "description": "One of: up, down, left, right." |
| } |
| }, |
| "required": [ |
| "direction" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "pinch", |
| "description": "Pinch with two fingers to zoom in or out on an element. Does not rotate the screen.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the element to zoom." |
| }, |
| "direction": { |
| "type": "string", |
| "description": "Zoom direction: in or out." |
| } |
| }, |
| "required": [ |
| "selector", |
| "direction" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "scroll_to", |
| "description": "Scroll until an element becomes visible.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the target element." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "pull_to_refresh", |
| "description": "Pull down from the top to refresh the screen.", |
| "parameters": { |
| "type": "object", |
| "properties": {}, |
| "required": [] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "type_text", |
| "description": "Type text into an input field.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the input field." |
| }, |
| "text": { |
| "type": "string", |
| "description": "Text to enter into the field." |
| } |
| }, |
| "required": [ |
| "selector", |
| "text" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "clear_text", |
| "description": "Clear the contents of an input field.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the input field." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "paste_text", |
| "description": "Paste the clipboard contents into an input field.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the input field." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "copy_text", |
| "description": "Copy the text of an element to the clipboard.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the element to copy from." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "dismiss_keyboard", |
| "description": "Hide the on-screen keyboard.", |
| "parameters": { |
| "type": "object", |
| "properties": {}, |
| "required": [] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "toggle_switch", |
| "description": "Set a toggle or switch to a desired state.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the switch." |
| }, |
| "state": { |
| "type": "string", |
| "description": "Desired state: on or off." |
| } |
| }, |
| "required": [ |
| "selector", |
| "state" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "set_checkbox", |
| "description": "Check or uncheck a checkbox.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the checkbox." |
| }, |
| "state": { |
| "type": "string", |
| "description": "Desired state: checked or unchecked." |
| } |
| }, |
| "required": [ |
| "selector", |
| "state" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "select_option", |
| "description": "Select an option from a dropdown or picker.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the dropdown." |
| }, |
| "option": { |
| "type": "string", |
| "description": "Visible option to select." |
| } |
| }, |
| "required": [ |
| "selector", |
| "option" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "set_slider", |
| "description": "Set a slider to a numeric value.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the slider." |
| }, |
| "value": { |
| "type": "integer", |
| "description": "Target value for the slider." |
| } |
| }, |
| "required": [ |
| "selector", |
| "value" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "set_date", |
| "description": "Set a date picker to a specific date.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the date field." |
| }, |
| "date": { |
| "type": "string", |
| "description": "Date in YYYY-MM-DD format." |
| } |
| }, |
| "required": [ |
| "selector", |
| "date" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "set_time", |
| "description": "Set a time picker to a specific time.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the time field." |
| }, |
| "time": { |
| "type": "string", |
| "description": "Time in HH:MM 24-hour format." |
| } |
| }, |
| "required": [ |
| "selector", |
| "time" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "navigate_back", |
| "description": "Go back one step to the previous screen. Does not return to the home screen.", |
| "parameters": { |
| "type": "object", |
| "properties": {}, |
| "required": [] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "go_home", |
| "description": "Return to the device home screen (launcher). This is not a single back step.", |
| "parameters": { |
| "type": "object", |
| "properties": {}, |
| "required": [] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "open_app_switcher", |
| "description": "Open the recent apps or app switcher.", |
| "parameters": { |
| "type": "object", |
| "properties": {}, |
| "required": [] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "press_keyboard_key", |
| "description": "Press a soft keyboard action key.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "key": { |
| "type": "string", |
| "description": "One of: enter, search, done, next, go." |
| } |
| }, |
| "required": [ |
| "key" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "open_notifications", |
| "description": "Open the notifications panel.", |
| "parameters": { |
| "type": "object", |
| "properties": {}, |
| "required": [] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "open_quick_settings", |
| "description": "Open the quick toggles or control panel.", |
| "parameters": { |
| "type": "object", |
| "properties": {}, |
| "required": [] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "rotate_screen", |
| "description": "Rotate the screen orientation between portrait and landscape. Does not zoom.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "orientation": { |
| "type": "string", |
| "description": "Orientation: portrait or landscape." |
| } |
| }, |
| "required": [ |
| "orientation" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "set_volume", |
| "description": "Set the media volume to a level.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "level": { |
| "type": "integer", |
| "description": "Volume level from 0 to 100." |
| } |
| }, |
| "required": [ |
| "level" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "wait_for_element", |
| "description": "Wait until a specific element appears, up to a timeout. Requires an element selector.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the element." |
| }, |
| "timeout_ms": { |
| "type": "integer", |
| "description": "Maximum wait time in milliseconds." |
| } |
| }, |
| "required": [ |
| "selector", |
| "timeout_ms" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "wait", |
| "description": "Pause execution for a fixed duration in milliseconds. Does not wait for any element.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "duration_ms": { |
| "type": "integer", |
| "description": "Duration to wait in milliseconds." |
| } |
| }, |
| "required": [ |
| "duration_ms" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "assert_element_visible", |
| "description": "Assert that an element is visible on screen.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the element." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "assert_element_not_visible", |
| "description": "Assert that an element is not visible on screen.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the element." |
| } |
| }, |
| "required": [ |
| "selector" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "assert_text_equals", |
| "description": "Assert that an element's text exactly equals an expected value (full match).", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the element." |
| }, |
| "expected_text": { |
| "type": "string", |
| "description": "Expected text content." |
| } |
| }, |
| "required": [ |
| "selector", |
| "expected_text" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "assert_text_contains", |
| "description": "Assert that an element's text contains a substring (partial match).", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "selector": { |
| "type": "string", |
| "description": "Logical id of the element." |
| }, |
| "text": { |
| "type": "string", |
| "description": "Substring expected within the element text." |
| } |
| }, |
| "required": [ |
| "selector", |
| "text" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "take_screenshot", |
| "description": "Capture a screenshot of the current screen.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "name": { |
| "type": "string", |
| "description": "File name for the screenshot." |
| } |
| }, |
| "required": [ |
| "name" |
| ] |
| } |
| } |
| }, |
| { |
| "type": "function", |
| "function": { |
| "name": "open_url", |
| "description": "Open a URL in the default browser.", |
| "parameters": { |
| "type": "object", |
| "properties": { |
| "url": { |
| "type": "string", |
| "description": "Fully qualified URL to open." |
| } |
| }, |
| "required": [ |
| "url" |
| ] |
| } |
| } |
| } |
| ] |
| ``` |
|
|
| </details> |
|
|
| ## Evaluation |
|
|
| Greedy + grammar-constrained decoding, on held-out paraphrase sets the model never saw in training: |
|
|
| | Set | Tool selection | Exact match (name + args) | |
| | --- | --- | --- | |
| | Wild (colloquial paraphrases) | 97.7% | 90.7% | |
| | Stress (high-variance, 64 cases) | 93.8% | 92.2% | |
|
|
| Hallucinated tool names: 0. Remaining misses cluster on indirect phrasings with no action keyword |
| (e.g. "make the picture smaller" -> `pinch` out). Accuracy is highest when the instruction names the action. |
|
|
| ## Training |
|
|
| - Base: `unsloth/functiongemma-270m-it` |
| - Method: 16-bit LoRA (r=16, alpha=16; q/k/v/o + gate/up/down) via Unsloth, on a free Tesla T4 |
| - LR 2e-4, linear schedule, `adamw_8bit`, 3 epochs, response-only loss masking |
| - Synthetic instruction -> tool-call data with contrastive confusable pairs and broad phrasing variety |
|
|
| ## License |
|
|
| Governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). Inherits the base model license. |
|
|
