Upload Qwen-Click-DiT model

README.md

@@ -22,24 +22,69 @@ This model predicts click coordinates given a screenshot and natural language in
 - **Qwen2.5-VL-3B** as a frozen vision-language backbone
 - **DiT (Diffusion Transformer)** action head using flow matching
 
-## Usage
+## Quick Start
+
+### Installation
+
+```bash
+pip install torch transformers accelerate qwen-vl-utils pillow
+git clone https://github.com/HusseinLezzaik/Qwen-Click-DiT.git
+cd Qwen-Click-DiT
+```
+
+### Inference
 
 ```python
+import torch
+from PIL import Image
 from transformers import AutoProcessor, AutoConfig
+from qwen_vl_utils import process_vision_info
+
+# Clone the repo first to get the model class
 from src.model import Qwen2_5_VLForClickPrediction
 
 # Load model
-config = AutoConfig.from_pretrained("TESS-Computer/qwen-click-dit")
-model = Qwen2_5_VLForClickPrediction.from_pretrained("TESS-Computer/qwen-click-dit")
-processor = AutoProcessor.from_pretrained("TESS-Computer/qwen-click-dit")
-
-# Predict
-from PIL import Image
-image = Image.open("screenshot.png")
-# ... process and predict
+model_id = "TESS-Computer/qwen-click-dit"
+config = AutoConfig.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
+config.dit_hidden_size = 512
+config.dit_num_layers = 6
+config.dit_num_heads = 8
+config.dit_dropout = 0.1
+config.num_inference_steps = 16
+
+model = Qwen2_5_VLForClickPrediction.from_pretrained(
+    model_id, config=config, torch_dtype=torch.bfloat16
+)
+model = model.to("cuda").eval()
+processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")
+
+# Prepare input
+image = Image.open("screenshot.png").convert("RGB")
+prompt = "Click on the search button"
+
+messages = [{
+    "role": "user",
+    "content": [
+        {"type": "image", "image": image, "min_pixels": 200704, "max_pixels": 401408},
+        {"type": "text", "text": prompt},
+    ],
+}]
+
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)
+inputs = processor(text=text, images=image_inputs, videos=video_inputs, return_tensors="pt", **video_kwargs)
+inputs = {k: v.to("cuda") if torch.is_tensor(v) else v for k, v in inputs.items()}
+
+# Predict click coordinates
+with torch.no_grad():
+    click_xy = model.predict(**inputs)
+
+x, y = click_xy[0].cpu().tolist()
+print(f"Normalized: ({x:.4f}, {y:.4f})")
+print(f"Pixels: ({int(x * image.width)}, {int(y * image.height)})")
 ```
 
-See [GitHub repo](https://github.com/TESS-Computer/qwen-click-dit) for full inference code.
+See [GitHub repo](https://github.com/HusseinLezzaik/Qwen-Click-DiT) for more examples.
 
 ## Metrics
 
@@ -69,11 +114,11 @@ See [GitHub repo](https://github.com/TESS-Computer/qwen-click-dit) for full infe
 ## Citation
 
 ```bibtex
-@misc{lezzaik2025qwenclickdit,
+@misc{lezzaik2026qwenclickdit,
   title = {Qwen-Click-DiT: Vision-Language Model with Diffusion Transformer for GUI Click Prediction},
   author = {Lezzaik, Hussein},
-  year = {2025},
-  howpublished = {\url{https://github.com/HusseinLezzaik/qwen-click-dit}},
+  year = {2026},
+  howpublished = {\url{https://github.com/HusseinLezzaik/Qwen-Click-DiT}},
 }
 ```