README.md

---
library_name: transformers
license: gemma
base_model: google/gemma-3n-E4B-it
tags:
- llama-factory
- lora
- generated_from_trainer
- android-control
- ui-automation
- vision-language-model
datasets:
- OfficerChul/Android-Control-84k
model-index:
- name: gemma-3n-e4b-android-control
  results: []

---

# Gemma-3n-E4B-it Android Control LoRA Fine-tuned Model

## Model Overview
This model is a fine-tuned version of Google's `gemma-3n-E4B-it` base model with LoRA adaptation for Android UI control tasks.

## Training Data
- **Dataset**: [OfficerChul/Android-Control-84k](https://huggingface.co/datasets/OfficerChul/Android-Control-84k)
- **Data Format**: Mobile UI screenshots paired with user instructions to perform appropriate actions (click, scroll, input, etc.)

### Training Data Format Example
```json
{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant that can identify what action to perform on mobile UI Screenshot given the user instruction."
    },
    {
      "role": "user",
      "content": "<image>Click on the Recording 2"
    },
    {
      "role": "assistant",
      "content": "{\"action_type\": \"click\", \"x\": 561, \"y\": 535}"
    }
  ],
  "images": ["and_ctrl/out_episode_18557_step_001.png"]
}
```

## Training Method
LoRA fine-tuning performed using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) framework

### 1. Training Configuration (`gemma3n-e4b-it.yaml`)
- **Base Model**: `google/gemma-3n-E4B-it`
- **Training Method**: LoRA (Low-Rank Adaptation)
- **LoRA Configuration**:
  - Rank: 32
  - Target modules: `q_proj, k_proj, v_proj, o_proj`
- **Training Parameters**:
  - Batch size: 4 (gradient accumulation: 48)
  - Learning rate: 2e-5
  - Epochs: 5
  - LR scheduler: Cosine
  - Optimizer: AdamW (fused)
  - Precision: bf16
- **Additional Settings**:
  - Gradient checkpointing enabled
  - Vision tower, multi-modal projector, and language model all trainable
  - DeepSpeed ZeRO-2 utilized

### 2. Model Merging (`gemma3n-e4b-it_lora_sft_merge.yaml`)
Merged trained LoRA adapter with base model:
- **Base Model**: `google/gemma-3n-E4B-it`

## Training Results
| Training Loss | Epoch  | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 0.2226        | 2.4288 | 500  | 0.2229          |
| 0.1658        | 4.8577 | 1000 | 0.2125          |

Final validation loss: **0.2124**

## Supported Action Types
- `click`: Click on specific coordinates
- `long_press`: Long press action
- `scroll`: Scroll (up/down)
- `input_text`: Text input
- `navigate_back`: Navigate back
- `navigate_home`: Navigate to home screen
- `open_app`: Open application
- `wait`: Wait action

## Usage
The merged model can be directly loaded using the Hugging Face Transformers library.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "OfficerChul/gemma-3n-E4B-it-Android-Control-84k"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True
)
```

## Evaluation Results

Performance comparison on Android UI control benchmark:

| Model | Action Type Accuracy | Click L2 Distance | Input Text Match | Scroll Direction Match | Avg. Episode Accuracy | Malformed JSON | Execution Time (s) | Inference Time (s) |
|-------|---------------------|-------------------|------------------|----------------------|---------------------|----------------|-------------------|-------------------|
| Qwen/Qwen3-VL-30B-A3B-Instruct | 0.9090 | 705.72 (n=812) | 0.8264 (n=121) | 0.3226 (n=248) | 0.9063 | 110 (5.5%) | 1101.43 | 485.12 |
| Qwen/Qwen2.5-VL-7B-Instruct | 0.6125 | 59.89 (n=544) | 0.8197 (n=61) | 0.3243 (n=111) | 0.6163 | 499 (24.9%) | 720.88 | 580.92 |
| Qwen/Qwen2.5-VL-3B-Instruct | 0.6645 | 88.21 (n=165) | 0.7889 (n=90) | 0.3519 (n=108) | 0.6615 | 440 (22.0%) | 676.76 | 536.27 |
| OfficerChul/Qwen2.5-VL-7B-Instruct-Android-Control-5a | 0.9970 | 427.30 (n=1466) | 0.9434 (n=159) | 0.9775 (n=267) | 0.9974 | 0 (0.0%) | 1086.97 | 581.82 |
| OfficerChul/Qwen2.5-VL-3B-Instruct-Android-Control-5a | 0.9965 | 446.54 (n=1467) | 0.9363 (n=157) | 0.9738 (n=267) | 0.9976 | 1 (0.1%) | 672.88 | 530.95 |
| OfficerChul/InfiGUI-G1-7B-Android-Control-5a | 0.9970 | 466.24 (n=1466) | 0.9434 (n=159) | 0.9775 (n=267) | 0.9968 | 1 (0.1%) | 897.58 | 552.23 |
| OfficerChul/InfiGUI-G1-3B-Android-Control-5a | 0.9980 | 449.73 (n=1467) | 0.9625 (n=160) | 0.9625 (n=267) | 0.9983 | 0 (0.0%) | 722.63 | 529.57 |
| InfiX-ai/InfiGUI-G1-7B | 0.6715 | 82.21 (n=821) | 0.8000 (n=70) | 0.2268 (n=194) | 0.6763 | 457 (22.9%) | 698.77 | 557.50 |
| InfiX-ai/InfiGUI-G1-3B | 0.8745 | 102.39 (n=1020) | 0.7700 (n=100) | 0.2299 (n=174) | 0.8910 | 78 (3.9%) | 702.93 | 559.65 |
| OfficerChul/gemma-3n-E2B-it-Android-Control-84k | 0.5819 | 985.82 (n=123) | 0.8596 (n=114) | 0.2159 (n=88) | 0.5781 | 0 (0.0%) | 322.95 | 159.23 |
| **OfficerChul/gemma-3n-E4B-it-Android-Control-84k** | **0.5088** | **878.66 (n=124)** | **0.8763 (n=97)** | **0.3689 (n=103)** | **0.5121** | **0 (0.0%)** | **363.23** | **177.11** |

## License
Follows the license terms of the Google Gemma model.

## Notes
- This model was developed for research purposes in mobile UI automation and accessibility enhancement
- Proper validation is required when using in production environments

## Framework Versions
- PEFT 0.17.1
- Transformers 4.57.0
- PyTorch 2.8.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.1