Aligned screen reader for MobileGym (LoRA on Qwen3-VL-4B-Instruct)

This adapter comes with the paper Practice in Text, Act in Pixels: Correcting Where Perception Fails (arXiv:2611.22722) and its code at https://github.com/Carlxuzhl/practice-in-text.

GUI agents can practise cheaply on a text version of an interface and act on real screenshots through a screen reader that turns each screenshot into the same kind of list. This LoRA aligns the reader Qwen3-VL-4B-Instruct to that list: for a MobileGym screenshot it writes the screen's text view, one line per visible element in the layout order (top to bottom, left to right within a row), in the line format the code's parser reads. Policies trained only in text can then act on screenshots without being trained on pixels.

Results (from the paper)

  • Element error rate (missed elements, wrong text or wrong state) on 1,007 held-back MobileGym screenshots: 25.62% with the original reader, 1.56% with this adapter.
  • A policy that practised only in text, acting through this reader, completes 80.9% of the confirmation tasks, against 55.6% for a policy trained directly on the same budget of about 2,000 screenshots through the original reader.

How to use it

Use it with the code repository, which holds the prompt, the parser and the settings it was trained for:

python -m reader.merge_lora --base <Qwen3-VL-4B-Instruct directory> --adapter <this adapter's directory> --out <merged directory>
bash scripts/with_servers.sh --gpu 0 --no-policy --reader <merged directory> -- <command>

The reader must be called exactly as the original one: the prompt of reader/reader.py (in Chinese, the apps' language), greedy decoding, repetition penalty 1.05, screenshots at half size (configs/reader.json).

Training

  • LoRA rank 16, alpha 32, dropout 0.05, on the q/k/v/o/gate/up/down projections of the language model; the vision encoder and its merger are frozen.
  • 2,004 calibration screenshots of 29 MobileGym templates, each labelled with its true text view (2,003 used; one was longer than 10,240 tokens). Learning rate 1e-4, 2 epochs, one example per step with 16 accumulated, cosine schedule with 3% warmup, last checkpoint; 251 updates, 93 minutes on one RTX A6000. Details in training_summary.json.

Scope and limits

  • Aligned on MobileGym only (its 29 practised templates, simulated Chinese-language apps). Expect little benefit on other apps or platforms without aligning again on their screenshots.
  • It is tied to the prompt, the output format and the screenshot scale above.

Licence

CC BY-NC 4.0: the adapter was trained on screenshots and labels generated from MobileGym, whose data is under CC BY-NC 4.0 (academic, non-commercial use). The base model's own licence applies to the base weights (see the Qwen3-VL-4B-Instruct model card). The code repository is under Apache-2.0.

Citation

@article{xu2026practice,
  title   = {Practice in Text, Act in Pixels: Correcting Where Perception Fails},
  author  = {Xu, Zhongling and Wang, Hairong},
  journal = {arXiv preprint arXiv:2611.22722},
  year    = {2026}
}
Downloads last month
25
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for xuzhl/practice-in-text-aligned-reader

Adapter
(255)
this model

Paper for xuzhl/practice-in-text-aligned-reader