Yinpei commited on
Commit
9e2347e
·
1 Parent(s): f67c2e1

update task hint

Browse files
Files changed (2) hide show
  1. gradio-web/config.py +5 -2
  2. gradio-web/note_content.py +97 -102
gradio-web/config.py CHANGED
@@ -16,7 +16,7 @@ RIGHT_TOP_ACTION_SCALE = 2
16
  RIGHT_TOP_LOG_SCALE = 1
17
 
18
  # 全局界面字号(不作用于页面主标题)
19
- UI_GLOBAL_FONT_SIZE = "25px"
20
 
21
  # Session超时配置
22
  SESSION_TIMEOUT = 300 # Session超时时间(秒),如果30秒内没有execute_step操作,将自动回收session
@@ -29,14 +29,17 @@ TASK_NAME_LIST = [
29
  "StopCube",
30
  "PickXtimes",
31
  "SwingXtimes",
 
32
  "ButtonUnmask",
33
  "VideoUnmask",
34
  "VideoUnmaskSwap",
35
  "ButtonUnmaskSwap",
 
36
  "PickHighlight",
37
  "VideoRepick",
38
  "VideoPlaceButton",
39
  "VideoPlaceOrder",
 
40
  "MoveCube",
41
  "InsertPeg",
42
  "PatternLock",
@@ -59,7 +62,7 @@ DEMO_VIDEO_ENV_IDS = [
59
 
60
  UI_TEXT = {
61
  "log": {
62
- "action_selection_prompt": "Please select the action in the left 👈,\nActions with 🎯 need to select a point on the image as input",
63
  "point_selection_prompt": "Current action needs location input, please click on the image to select key pixel",
64
  "demo_video_prompt": 'Press "Watch Video Input 🎬" to watch a video\nNote: you can only watch the video once',
65
  "session_error": "Session Error",
 
16
  RIGHT_TOP_LOG_SCALE = 1
17
 
18
  # 全局界面字号(不作用于页面主标题)
19
+ UI_GLOBAL_FONT_SIZE = "24px"
20
 
21
  # Session超时配置
22
  SESSION_TIMEOUT = 300 # Session超时时间(秒),如果30秒内没有execute_step操作,将自动回收session
 
29
  "StopCube",
30
  "PickXtimes",
31
  "SwingXtimes",
32
+
33
  "ButtonUnmask",
34
  "VideoUnmask",
35
  "VideoUnmaskSwap",
36
  "ButtonUnmaskSwap",
37
+
38
  "PickHighlight",
39
  "VideoRepick",
40
  "VideoPlaceButton",
41
  "VideoPlaceOrder",
42
+
43
  "MoveCube",
44
  "InsertPeg",
45
  "PatternLock",
 
62
 
63
  UI_TEXT = {
64
  "log": {
65
+ "action_selection_prompt": "Please select the action.\nActions with 🎯 need to select a point on the image as input",
66
  "point_selection_prompt": "Current action needs location input, please click on the image to select key pixel",
67
  "demo_video_prompt": 'Press "Watch Video Input 🎬" to watch a video\nNote: you can only watch the video once',
68
  "session_error": "Session Error",
gradio-web/note_content.py CHANGED
@@ -36,156 +36,151 @@ def get_task_hint(env_id):
36
  # Order follows solve_3.5_parallel_multi_loop_v4.py DEFAULT_ENVS list
37
  hints = {
38
  "PickXtimes": """\
39
- Suppose the task goal is to pick up red cubes for two times, a typical action sequence could be:
40
- 1. Pick up the cube (use mouse click to select the cube with the correct color)
41
- 2. Place the cube onto the target.
42
- 3. Pick up the cube (use mouse click to select the cube with the correct color)
43
- 4. Place the cube onto the target.
44
  5. Press the button to stop.
45
- Select "Ground Truth Action" if you need help, and "Execute" it
 
46
  """,
47
 
48
  "StopCube": """\
49
- Suppose the task goal is to stop the cube on the target for three times, a typical action sequence could be:
50
- 1. Move to the top of the button to prepare
51
- 2. Remain static (it will execute for a fixed time duration, you need to count the times the cube has passed the target)
52
- 3. Remain static
53
- 4. Remain static
54
- 5. Remain static (Suppose you feel the cube is about to reach the target for the expected number of times, you should press the button to stop the cube directly)
55
- 6. Press the button to stop.
56
- Select "Ground Truth Action" if you need help, and "Execute" it
57
  """,
58
 
59
  "SwingXtimes": """\
60
- Suppose the task goal is to swing the back and forth for two times, a typical action sequence could be:
61
- 1. Pick up the cube (use mouse click to select the cube with the correct color)
62
- 2. Move to the top of the target (use mouse click to select the right-side target)
63
- 3. Move to the top of the target (use mouse click to select the left-side target)
64
- 4. Move to the top of the target (use mouse click to select the right-side target)
65
- 5. Move to the top of the target (use mouse click to select the left-side target)
66
- 6. Put the cube onto the table
67
- 7. Press the button to stop.
68
- Spatial directions (e.g., left, right) follow the robot base coordinate frame
69
- Select "Ground Truth Action" if you need help, and "Execute" it
70
  """,
71
 
72
  "BinFill": """\
73
- Suppose the task goal is to pick two red cubes in the bin, a typical action sequence could be:
74
- 1. Pick up the cube (use mouse click to select the cube with the correct color)
75
- 2. Put it into the bin.
76
- 3. Pick up the cube (use mouse click to select the cube with the correct color)
77
- 4. Put it into the bin.
78
- 5. Press the button to stop.
79
- Select "Ground Truth Action" if you need help, and "Execute" it
80
  """,
81
 
82
  "VideoUnmaskSwap": """\
83
- Watch the video carefully. Cubes will be hidden by containers, and you need to memorize the color of the cube inside each one.
84
- You need to track the containers since they swap positions!
85
- A typical action sequence could be:
86
- 1. Pick up the container (use mouse click to select the container)
87
- 2. Drop the container down.
88
- Pick up another container if the task goal is to find two containers.
89
- Select "Ground Truth Action" if you need help, and "Execute" it
90
  """,
91
 
92
  "VideoUnmask": """\
93
- Watch the video carefully. Cubes will be hidden by containers, and you need to memorize the color of the cube inside each one.
94
- A typical action sequence could be:
95
- 1. Pick up the container (use mouse click to select the container)
96
- 2. Drop the container down.
97
- Pick up another container if the task goal is to find two containers.
98
- Select "Ground Truth Action" if you need help, and "Execute" it
99
  """,
100
 
101
  "ButtonUnmaskSwap": """\
102
- Press the buttons sequentially. While pressing the buttons, the cubes will be hidden inside the containers, and you need to memorize the color of the cube inside each one.
103
- You need to track the containers since they swap positions!
104
- A typical action sequence could be:
105
- 1. Press the first button.
106
- 2. Press the second button.
107
- 3. Pick up the container (use mouse click to select the container)
108
- 4. Drop the container down.
109
- Pick up another container if the task goal is to find two containers.
110
- Select "Ground Truth Action" if you need help, and "Execute" it
111
  """,
112
 
113
  "ButtonUnmask": """\
114
- Press the buttons sequentially. While pressing the buttons, the cubes will be hidden inside the containers, and you need to memorize the color of the cube inside each one.
115
- A typical action sequence could be:
116
  1. Press the button.
117
- 2. Pick up the container (use mouse click to select the container)
118
- 3. Drop the container down.
119
- Pick up another container if the task goal is to find two containers.
120
- Select "Ground Truth Action" if you need help, and "Execute" it
121
  """,
122
 
123
  "VideoRepick": """\
124
- Remember the cube that has been picked up before, and then pick it up again. The cubes might be swapped positions.
125
- A typical action sequence could be:
126
- 1. Pick up the cube (use mouse click to select the correct cube with the correct color)
127
- 2. Put the cube down on the table.
128
- (Repeat 1 and 2 for the expected number of times)
129
- 3. Press the button to stop.
130
- Select "Ground Truth Action" if you need help, and "Execute" it
 
131
  """,
132
 
133
  "VideoPlaceButton": """\
134
- The video shows a robot placing a cube on different targets and pressing the button in a sequence. The targets may change positions.
135
- A typical action sequence could be:
136
- 1. Pick up the cube (use mouse click to select the correct cube with the correct color)
137
- 2. Put the cube down on the target (use mouse click to select the target)
138
- Select "Ground Truth Action" if you need help, and "Execute" it
 
139
  """,
140
 
141
  "VideoPlaceOrder": """\
142
- The video shows a robot placing a cube on different targets and pressing the button in a sequence. The targets may change positions.
143
- A typical action sequence could be:
144
- 1. Pick up the cube (use mouse click to select the correct cube with the correct color)
145
- 2. Put the cube down on the target (use mouse click to select the target)
146
- Select "Ground Truth Action" if you need help, and "Execute" it
 
147
  """,
148
 
149
  "PickHighlight": """\
150
- While the robot is pressing the button, some cubes will be highlighted with white discs on the table. Remember them.
151
- A typical action sequence could be:
152
  1. Press the button.
153
- 2. Pick up the cube (use mouse click to select the correct cube with the correct color)
154
- 3. Put the cube down on the table.
155
- (Repeat 2 and 3 for with the rest of highlighted cubes)
156
- Select "Ground Truth Action" if you need help, and "Execute" it
157
  """,
158
 
159
  "InsertPeg": """\
160
- The video shows a robot picking up and inserting a peg into a hole.
161
- The peg consists of two parts with different colors; you need to pick up the correct part of the peg and insert it into the hole from the correct side.
162
- A typical action sequence could be:
163
- 1. Pick up the peg (use mouse click to select the correct peg and the correct part of the peg)
164
- 2. Insert the peg into the hole on the left side
165
- Spatial directions (e.g., left, right) follow the robot base coordinate frame.
166
- Select "Ground Truth Action" if you need help, and "Execute" it
167
  """,
168
 
169
  "MoveCube": """\
170
- The video shows a robot moving a cube to a target using different methods.
171
- The robot might (1) pick up and place the cube, (2) push it with the gripper, or (3) hook it using a peg.
172
- Remember the way the robot moves the cube and choose the correct action to execute.
173
- Select "Ground Truth Action" if you need help, and "Execute" it
174
  """,
175
 
176
  "PatternLock": """\
177
- The video shows a robot tracing a pattern with a stick.
178
- Remember the movements and reproduce them by choosing correct actions.
179
- Spatial directions (e.g., left, right, forward, backward) follow the robot base coordinate frame.
180
- Select "Ground Truth Action" if you need help, and "Execute" it
181
  """,
182
 
183
  "RouteStick": """\
184
- The video shows a robot navigating from one target to another by circling around a stick.
185
- The movement can be clockwise or counter-clockwise, and the stick may be on the left or right side.
186
- Remember the sequence of actions and choose the correct action to execute.
187
- Spatial directions (e.g., left, right) follow the robot base coordinate frame.
188
- Select "Ground Truth Action" if you need help, and "Execute" it
189
  """,
190
 
191
  }
 
36
  # Order follows solve_3.5_parallel_multi_loop_v4.py DEFAULT_ENVS list
37
  hints = {
38
  "PickXtimes": """\
39
+ To pick up red cubes twice, a typical sequence:
40
+ 1. Pick up the cube (click to select the correct color)
41
+ 2. Place it on the target.
42
+ 3. Pick up the cube (click to select the correct color)
43
+ 4. Place it on the target.
44
  5. Press the button to stop.
45
+
46
+ Select "Ground Truth Action" if you need help, then "Execute" it.
47
  """,
48
 
49
  "StopCube": """\
50
+ To stop the cube on the target three times, a typical sequence:
51
+ 1. Move above the button to prepare.
52
+ 2. Remain static for a fixed duration (count how many times the cube passes the target, may select "remain static" multiple times).
53
+ 3. When the cube is about to reach the target for the last time, press the button to stop.
54
+
55
+ Select "Ground Truth Action" if you need help, then "Execute" it.
 
 
56
  """,
57
 
58
  "SwingXtimes": """\
59
+ To swing right-to-left twice, a typical sequence:
60
+ 1. Pick up the cube (click to select the correct color).
61
+ 2. Move above the right target, then the left (click to select each).
62
+ 3. Repeat: right, then left.
63
+ 4. Put the cube on the table and press the button to stop.
64
+
65
+ Spatial directions (left, right) follow the robot base frame.
66
+ Select "Ground Truth Action" if you need help, then "Execute" it.
 
 
67
  """,
68
 
69
  "BinFill": """\
70
+ To pick two red cubes and put them in the bin, a typical sequence:
71
+ 1. Pick up a red cube (click to select), then put it in the bin.
72
+ 2. Pick up another red cube (click to select), then put it in the bin.
73
+ 3. Press the button to stop.
74
+
75
+ Select "Ground Truth Action" if you need help, then "Execute" it.
 
76
  """,
77
 
78
  "VideoUnmaskSwap": """\
79
+ Watch the video: cubes are hidden by containers. Memorize each cube's color. Track the swap of containers.
80
+ Typical sequence:
81
+ 1. Pick up a container (click to select), then drop it.
82
+ Repeat for a second container if the goal is to find two cubes.
83
+
84
+ Select "Ground Truth Action" if you need help, then "Execute" it.
 
85
  """,
86
 
87
  "VideoUnmask": """\
88
+ Watch the video: cubes are hidden by containers. Memorize each cube's color.
89
+ Typical sequence:
90
+ 1. Pick up a container (click to select), then drop it.
91
+ Repeat for a second container if the goal is to find two cubes.
92
+
93
+ Select "Ground Truth Action" if you need help, then "Execute" it.
94
  """,
95
 
96
  "ButtonUnmaskSwap": """\
97
+ Press the buttons. While doing so, cubes are hidden in containers. Memorize each cube's color. Track the swap of containers.
98
+ Typical sequence:
99
+ 1. Press the first button, then the second.
100
+ 2. Pick up a container (click to select), then drop it.
101
+ Repeat for a second container if the goal is to find two cubes.
102
+
103
+ Select "Ground Truth Action" if you need help, then "Execute" it.
 
 
104
  """,
105
 
106
  "ButtonUnmask": """\
107
+ Press the buttons in order. Cubes are hidden in containersmemorize each cube's color.
108
+ Typical sequence:
109
  1. Press the button.
110
+ 2. Pick up a container (click to select), then drop it.
111
+ Repeat for a second container if the goal is to find two cubes.
112
+
113
+ Select "Ground Truth Action" if you need help, then "Execute" it.
114
  """,
115
 
116
  "VideoRepick": """\
117
+ Remember which cube was picked in the video, then pick it again. Cube positions may be swapped.
118
+ Typical sequence:
119
+ 1. Pick up the correct cube (click to select by color)
120
+ 2. Put it on the table.
121
+ 3. Repeat step 1 for the required number of times.
122
+ 4. Press the button to stop.
123
+
124
+ Select "Ground Truth Action" if you need help, then "Execute" it.
125
  """,
126
 
127
  "VideoPlaceButton": """\
128
+ The video shows a robot placing a cube on different targets and pressing the button in sequence. Targets may change positions.
129
+ Typical sequence:
130
+ 1. Pick up the correct cube (click to select)
131
+ 2. Place it on the target (click to select target).
132
+
133
+ Select "Ground Truth Action" if you need help, then "Execute" it.
134
  """,
135
 
136
  "VideoPlaceOrder": """\
137
+ The video shows a robot placing a cube on different targets and pressing the button in sequence. Targets may change positions.
138
+ Typical sequence:
139
+ 1. Pick up the correct cube (click to select)
140
+ 2. Place it on the target (click to select target).
141
+
142
+ Select "Ground Truth Action" if you need help, then "Execute" it.
143
  """,
144
 
145
  "PickHighlight": """\
146
+ While the robot presses the button, some cubes are highlighted with white discs—remember them.
147
+ Typical sequence:
148
  1. Press the button.
149
+ 2. Pick up each highlighted cube (click to select), put it on the table. Repeat for all highlighted cubes.
150
+
151
+ Select "Ground Truth Action" if you need help, then "Execute" it.
 
152
  """,
153
 
154
  "InsertPeg": """\
155
+ The video shows a robot inserting a peg into a hole. The peg has two colored parts—pick the correct part and insert from the correct side.
156
+ Typical sequence:
157
+ 1. Pick up the peg (click to select correct peg and part).
158
+ 2. Insert it into the hole on the left.
159
+
160
+ Spatial directions (left, right) follow the robot base frame.
161
+ Select "Ground Truth Action" if you need help, then "Execute" it.
162
  """,
163
 
164
  "MoveCube": """\
165
+ The video shows a robot moving a cube to a target by (1) pick-and-place, (2) pushing with the gripper, or (3) hooking with a peg.
166
+ Remember which method was used and reproduce it.
167
+
168
+ Select "Ground Truth Action" if you need help, then "Execute" it.
169
  """,
170
 
171
  "PatternLock": """\
172
+ The video shows a robot tracing a pattern with a stick. Remember the movements and reproduce them.
173
+
174
+ Spatial directions (left, right, forward, backward) follow the robot base frame.
175
+ Select "Ground Truth Action" if you need help, then "Execute" it.
176
  """,
177
 
178
  "RouteStick": """\
179
+ The video shows a robot moving between targets by circling around a stick (clockwise or counter-clockwise; move left or right around the stick).
180
+ Remember the action sequence and reproduce it.
181
+
182
+ Spatial directions (left, right) follow the robot base frame.
183
+ Select "Ground Truth Action" if you need help, then "Execute" it.
184
  """,
185
 
186
  }