UI-Mate-democua-27B

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Show the workflow once. Let the agent adapt it to the task at hand.

Tencent HY Frontier

Project Page · GitHub · arXiv

Overview

UI-Mate-democua-27B is the demonstration-guided checkpoint of UI-Mate. It observes live screenshots, reasons over the visible state, and produces structured keyboard and mouse actions, and it can additionally take one recorded workflow as input and carry that procedure over to a new task.

It starts from the general reinforcement-learning checkpoint and is further trained with supervised fine-tuning on a mixture of general computer-use data and demonstration-augmented data. Keeping general data in the mixture is what preserves instruction-only competence: the model still runs a task from an instruction alone, and gains the ability to use a demonstration when one is supplied.

A demonstration is treated as guidance rather than a fixed action script. Recorded coordinates are never replayed, and the model re-plans from the live interface whenever the content, layout, or application state differs.

Model Details

• Parameters: 27B
• Base model: Qwen3.6-27B
• Input: task instruction, screenshots, interaction history, and an optional demonstration workflow
• Output: reasoning, a concise action description, and structured computer-use tool calls
• Action space: mouse, keyboard, scrolling, waiting, user interaction, subtask completion, and task completion
• Training: supervised fine-tuning, online reinforcement learning in executable GUI environments, then supervised fine-tuning on mixed general and demonstration-augmented computer-use data
• License: Apache-2.0

UI-Mate is an agent checkpoint rather than a standalone visual-chat model. We recommend using the official prompt, response parser, and interaction harness from the UI-Mate repository.

Checkpoint	Intended use
UI-Mate-27B	General computer use at 27B
UI-Mate-9B	General computer use at 9B
UI-Mate-democua-27B	Demonstration-guided computer use

Highlights

• One-shot procedural learning from a single demonstration.
• Instruction-only execution retained through mixed general and demonstration training data.
• Subtask-level guidance rather than whole-workflow injection.
• Live-screen grounding instead of coordinate replay.
• Trained on incomplete workflows, so milestones must be connected by reading the screen.
• Structured actions compatible with pyautogui, served through an OpenAI-compatible interface.

Demonstration-Guided Execution

A demonstration is a recorded successful execution: every keyboard and pointer action, with screenshots immediately before and after each one. It may be recorded by a human, or taken from a successful rollout of a stronger GUI agent. The raw trace is then:

1. normalized into a consistent action-and-frame representation;
2. annotated by a vision-language model along four axes — screen state, intent, the action taken, and how its target was located visually;
3. segmented into named subtasks, each with a short goal and an explicitly checkable completion criterion; and
4. supplied at inference time as a compact view of the active subtask.

Three blocks are placed before the instruction: <workflow_progress> listing all subtasks as done, current, or upcoming; <current_subtask> carrying the sub-instruction and its completion criterion; and <current_subtask_action_list> holding the ordered steps of that subtask only. The tool schema gains a subtask_complete action, and the model reports it to advance the pointer.

Training deliberately withholds part of this guidance. The full trajectory remains the supervision target, but the workflow shown to the model keeps only key actions — focus clicks, scrolling, and popup dismissal are omitted — so the model cannot reduce the task to copying the next line and must infer the missing steps from the screenshot. The training mixture also covers three guidance-to-screen relationships: full alignment, partial misalignment where the model corrects mismatched steps from the screenshot, and irrelevance where it ignores the workflow entirely. Full alignment stays the majority case so that guidance remains useful.

At inference the complete action sequence of the current subtask is passed instead, with no key-action extraction. Once the model treats the demonstration as a fallible reference and arbitrates in favour of the screenshot, fuller guidance is simply more informative, and dropping the extraction step removes one model call and a source of silent corruption.

Evaluation

Paired evaluation in the self-demo setting, where each target is paired with a successful rollout of that same task from a stronger agent. Task instructions, initial environment states, interaction budgets, and evaluators are identical across the two conditions; only the demonstration differs. Episodes are allowed up to 1,000 interaction steps.

Evaluation set · metric	Instruction only	+ one demonstration	Change
OSWorkerBench-Subset (33) · strict success	17.17	35.35	+18.18 pp
OSWorkerBench-Subset (33) · progress	67.85	81.14	+13.29 pp
OSWorld-Subset (30) · progress	40.27	65.75	+25.48 pp
GameDev (10) · average score	76.76	81.15	+4.39 pp

Results are averaged over three runs per target on OSWorkerBench-Subset and five runs elsewhere.

OSWorkerBench-Subset (33 tasks) contains multi-application office tasks requiring coordinated interaction across three to five applications, with repeated subtask patterns and branching execution paths. Demonstrations improve 28 of the 33 tasks, and the number of tasks scoring perfectly rises from one to five. Average trajectory length also rises, from 173.3 to 216.0 steps: unguided runs often terminate after completing only part of the requested workflow, and the demonstration surfaces the remaining branches and repeated operations, producing longer but more complete trajectories.

OSWorld-Subset (30 tasks) consists of feasible tasks that the instruction-only model fails but a stronger reference agent can solve, which isolates procedural knowledge unlikely to be discovered from the instruction alone. Performance improves on 18 tasks and is unchanged on eight. Four tasks that score zero without guidance — chrome-02, chrome-03, multi-02, and os-01 — are solved perfectly in every demonstration-conditioned run.

GameDev (10 tasks) is a curated set of exceptionally long-horizon tasks, each requiring more than 200 human actions on average, centered on eight Godot tasks that together cover end-to-end 2D game development from scratch. The largest gains fall on godot-04 (+18.89), godot-07 (+17.89), and qgis-01 (+11.25), all of which require long, structured sequences of fine-grained operations; the three tasks already solved perfectly are unchanged. Here demonstrations shorten the average trajectory, from 303.6 to 253.1 steps, by removing exploratory detours.

Quick Start

1. Serve the model with vLLM

vLLM automatically downloads the checkpoint from Hugging Face Hub on first launch.

pip install -U vllm openai pillow

vllm serve tencent/UI-Mate-democua-27B \
    --trust-remote-code \
    --served-model-name UI_Mate \
    --port 8000 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.85 \
    --mm-encoder-tp-mode data \
    --chat-template-content-format openai \
    --limit-mm-per-prompt '{"image":6,"video":0}'

The default agent retains five screenshots in context. The server must therefore admit at least six images because the newest screenshot arrives before the oldest one is collapsed.

Confirm that the endpoint exposes the expected model name:

curl -s http://127.0.0.1:8000/v1/models

2. Guide a run with a demonstration

The UI-Mate repository on GitHub ships a real demonstration under resources/example_demonstration/:

File	What it is
task.json	An OSWorld task for configuring a chroot-restricted SSH user — instruction, setup, and evaluator.
trajectory_captioned.json	A run of that task scored 1.0, distilled into 12 subtasks over 42 steps. Same task, so this is the self-demo setting.

git clone https://github.com/Tencent/UI-Mate.git
cd UI-Mate

python examples/run_agent.py \
    --demo resources/example_demonstration/trajectory_captioned.json \
    --image resources/example_single_step/os_install_spotify.png \
    --instruction 'Please create an SSH user named "charles" with password "Ex@mpleP@55w0rd!" on Ubuntu who is only allowed to access the folder "/home/test1".' \
    --base-url http://127.0.0.1:8000/v1

Drop --demo and the same command runs in the instruction-only mode.

3. Use the Python agent

Pass demo= a demonstration file, or a directory holding a single trajectory_captioned*.json:

from agents.ui_mate_agent import UIMateAgent

agent = UIMateAgent(
    base_url="http://127.0.0.1:8000/v1",
    model="UI_Mate",
    demo="resources/example_demonstration/trajectory_captioned.json",
)
agent.reset()

with open("screen.png", "rb") as f:
    response, actions = agent.predict(instruction, {"screenshot": f.read()})

print(response)
print(actions)

A step where the model only reports progress comes back as WAIT. A premature finished on a non-final subtask advances the workflow instead of ending the episode, so completing one subtask cannot end the task early.

For long episodes the harness controls context growth by folding the oldest interaction steps into a compact progress note before the request exceeds the context window, keeping recent steps verbatim and the most recent screenshots intact.

Intended Use and Limitations

UI-Mate-democua-27B is intended for research and development of screenshot-based GUI agents in controlled desktop environments.

Its behavior can be affected by application versions, screen layouts, display scaling, latency, and unexpected UI state. Benchmark performance does not guarantee reliable execution in arbitrary environments, and the model requires an external runtime to execute its predicted actions.

Safety

Computer-use agents can make mistakes, encounter prompt injection, or trigger consequential actions.

• Prefer isolated or disposable environments.
• Avoid unattended, high-stakes, or destructive workflows.
• Require human confirmation before sensitive operations.
• Monitor the interaction trajectory and verify the resulting application state.
• Do not treat a model-reported success as proof that the intended outcome was achieved.

License

UI-Mate is released under the Apache License 2.0. Third-party components remain subject to their respective licenses. See the repository LICENSE for details.

Citation

@article{uimate2026,
  title         = {UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations},
  author        = {Tencent HY Frontier Team},
  journal       = {arXiv preprint arXiv:2608.15930},
  year          = {2026},
}