Fix model card: base model, title, and image-text-to-text quick start

#3
by SamJoshua - opened
Files changed (1) hide show
  1. README.md +44 -15
README.md CHANGED
@@ -1,34 +1,65 @@
1
  ---
 
2
  datasets: smolagents/aguvis-stage-2
3
  library_name: transformers
 
4
  tags:
5
  - generated_from_trainer
6
  - open-r1
7
  - vision-language
8
  - vlm
9
- licence: license
 
 
10
  ---
11
 
12
- # Model Card for None
13
 
14
- This model is a fine-tuned version of [None](https://huggingface.co/None) on the [smolagents/aguvis-stage-2](https://huggingface.co/datasets/smolagents/aguvis-stage-2) dataset.
15
- It has been trained using [TRL](https://github.com/huggingface/trl).
 
 
 
 
16
 
17
  ## Quick start
18
 
 
 
19
  ```python
20
- from transformers import pipeline
 
 
 
 
 
 
 
21
 
22
- question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
- generator = pipeline("text-generation", model="None", device="cuda")
24
- output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
- print(output["generated_text"])
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  ```
27
 
28
- ## Training procedure
29
 
30
- [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/huggingface/smolagents/runs/9jig6y4c)
31
 
 
32
 
33
  This model was trained with SFT.
34
 
@@ -42,10 +73,8 @@ This model was trained with SFT.
42
 
43
  ## Citations
44
 
45
-
46
-
47
  Cite TRL as:
48
-
49
  ```bibtex
50
  @misc{vonwerra2022trl,
51
  title = {{TRL: Transformer Reinforcement Learning}},
@@ -55,4 +84,4 @@ Cite TRL as:
55
  publisher = {GitHub},
56
  howpublished = {\url{https://github.com/huggingface/trl}}
57
  }
58
- ```
 
1
  ---
2
+ base_model: HuggingFaceTB/SmolVLM2-2.2B-Instruct
3
  datasets: smolagents/aguvis-stage-2
4
  library_name: transformers
5
+ pipeline_tag: image-text-to-text
6
  tags:
7
  - generated_from_trainer
8
  - open-r1
9
  - vision-language
10
  - vlm
11
+ - gui-agent
12
+ - computer-use
13
+ - smolagents
14
  ---
15
 
16
+ # SmolVLM2-2.2B-Instruct-Agentic-GUI
17
 
18
+ A GUI agent model for computer use, produced by the [Smol2Operator](https://huggingface.co/blog/smol2operator) recipe.
19
+ It starts from [HuggingFaceTB/SmolVLM2-2.2B-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct): phase 1 adds GUI grounding, and phase 2 (this checkpoint) adds agentic reasoning via SFT on [smolagents/aguvis-stage-2](https://huggingface.co/datasets/smolagents/aguvis-stage-2).
20
+ It was trained with [TRL](https://github.com/huggingface/trl).
21
+
22
+ - Blog post: [Smol2Operator: Post-Training GUI Agents for Computer Use](https://huggingface.co/blog/smol2operator)
23
+ - Training code: [huggingface/smol2operator](https://github.com/huggingface/smol2operator)
24
 
25
  ## Quick start
26
 
27
+ This is an image-text-to-text model: give it a screenshot plus an instruction.
28
+
29
  ```python
30
+ import torch
31
+ from transformers import AutoProcessor, AutoModelForImageTextToText
32
+
33
+ model_id = "smolagents/SmolVLM2-2.2B-Instruct-Agentic-GUI"
34
+ processor = AutoProcessor.from_pretrained(model_id)
35
+ model = AutoModelForImageTextToText.from_pretrained(
36
+ model_id, torch_dtype=torch.bfloat16, device_map="auto"
37
+ )
38
 
39
+ messages = [
40
+ {
41
+ "role": "user",
42
+ "content": [
43
+ {"type": "image", "url": "screenshot.png"},
44
+ {"type": "text", "text": "Click on the search bar."},
45
+ ],
46
+ }
47
+ ]
48
+
49
+ inputs = processor.apply_chat_template(
50
+ messages, add_generation_prompt=True, tokenize=True,
51
+ return_dict=True, return_tensors="pt",
52
+ ).to(model.device, dtype=torch.bfloat16)
53
+
54
+ output = model.generate(**inputs, max_new_tokens=128)
55
+ print(processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
56
  ```
57
 
58
+ For the exact system prompt and action format used in training, see the [Smol2Operator blog post](https://huggingface.co/blog/smol2operator) and the [recipe notebook](https://github.com/huggingface/smol2operator/blob/main/recipe.ipynb).
59
 
60
+ ## Training procedure
61
 
62
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/huggingface/smolagents/runs/9jig6y4c)
63
 
64
  This model was trained with SFT.
65
 
 
73
 
74
  ## Citations
75
 
 
 
76
  Cite TRL as:
77
+
78
  ```bibtex
79
  @misc{vonwerra2022trl,
80
  title = {{TRL: Transformer Reinforcement Learning}},
 
84
  publisher = {GitHub},
85
  howpublished = {\url{https://github.com/huggingface/trl}}
86
  }
87
+ ```