futurefantasy commited on
Commit
82c454e
·
verified ·
1 Parent(s): 77f673b

Simplify quick start to prompt plus video path only

Browse files
Files changed (3) hide show
  1. README.md +1 -16
  2. quick_start/README.md +9 -33
  3. quick_start/run_example.py +12 -77
README.md CHANGED
@@ -60,21 +60,6 @@ python quick_start/render_prediction_video.py \
60
  --input-jsonl quick_start/outputs/video.jsonl
61
  ```
62
 
63
- ## Reproduce Bundled Examples
64
-
65
- You can reproduce the bundled reference outputs directly from the release root.
66
-
67
- Single example:
68
-
69
- ```bash
70
- python quick_start/run_example.py \
71
- --model-path /path/to/VLAC2-Qwen3VL-30B-A3B-Progress \
72
- --metadata-path examples/example_01/metadata.json \
73
- --output-jsonl quick_start/outputs/example_01.jsonl
74
-
75
- python quick_start/render_prediction_video.py \
76
- --input-jsonl quick_start/outputs/example_01.jsonl \
77
- --output-video quick_start/outputs/example_01_pred_progress.mp4
78
- ```
79
 
80
  More detailed usage is provided in `quick_start/README.md`.
 
60
  --input-jsonl quick_start/outputs/video.jsonl
61
  ```
62
 
63
+ Bundled videos under `examples/` can be used with the same interface by passing the video path and prompt text directly.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64
 
65
  More detailed usage is provided in `quick_start/README.md`.
quick_start/README.md CHANGED
@@ -25,6 +25,14 @@ python quick_start/run_example.py \
25
  --task-instruction "把洋葱放进快递箱里。"
26
  ```
27
 
 
 
 
 
 
 
 
 
28
  If you have a more detailed task plan, pass it directly as text:
29
 
30
  ```bash
@@ -51,38 +59,6 @@ python quick_start/run_example.py \
51
  --task-instruction "把洋葱放进快递箱里。"
52
  ```
53
 
54
- ## Use Bundled Examples
55
-
56
- The bundled examples already provide `metadata.json` files with task information, so you can reproduce the reference outputs directly from the release root.
57
-
58
- Single example:
59
-
60
- ```bash
61
- python quick_start/run_example.py \
62
- --model-path /path/to/VLAC2-Qwen3VL-30B-A3B-Progress \
63
- --metadata-path examples/example_01/metadata.json \
64
- --output-jsonl quick_start/outputs/example_01.jsonl
65
-
66
- python quick_start/render_prediction_video.py \
67
- --input-jsonl quick_start/outputs/example_01.jsonl \
68
- --output-video quick_start/outputs/example_01_pred_progress.mp4
69
- ```
70
-
71
- All three bundled examples:
72
-
73
- ```bash
74
- for ex in example_01 example_02 example_03; do
75
- python quick_start/run_example.py \
76
- --model-path /path/to/VLAC2-Qwen3VL-30B-A3B-Progress \
77
- --metadata-path "examples/${ex}/metadata.json" \
78
- --output-jsonl "quick_start/outputs/${ex}.jsonl"
79
-
80
- python quick_start/render_prediction_video.py \
81
- --input-jsonl "quick_start/outputs/${ex}.jsonl" \
82
- --output-video "quick_start/outputs/${ex}_pred_progress.mp4"
83
- done
84
- ```
85
-
86
  ## Output
87
 
88
  By default the inference script writes:
@@ -92,7 +68,7 @@ By default the inference script writes:
92
  Each output file contains one JSONL row with:
93
 
94
  - input video path
95
- - optional metadata fields when `--metadata-path` is used
96
  - sampled frame indices and timestamps
97
  - decoded video statistics
98
  - prompt text
 
25
  --task-instruction "把洋葱放进快递箱里。"
26
  ```
27
 
28
+ Bundled videos under `examples/` are used the same way:
29
+
30
+ ```bash
31
+ python quick_start/run_example.py \
32
+ --video-path examples/example_01/episode.mp4 \
33
+ --prompt $'任务描述和具体规划: 抓取紫色方块使其从方形洞口落入积木桶中\n爪夹开始移动:0%\n爪夹接近紫色方块:20%\n爪夹抓紧紫色方块:40%\n爪夹抓紧紫色方块接近方形洞口:60%\n爪夹将紫色方块对准方形洞口:80%\n紫色方块从方形洞口落入积木桶中,爪夹移开:100%\n\n请根据任务描述和具体规划,找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求:每个关键点一行,格式为:\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出,不要输出额外说明。'
34
+ ```
35
+
36
  If you have a more detailed task plan, pass it directly as text:
37
 
38
  ```bash
 
59
  --task-instruction "把洋葱放进快递箱里。"
60
  ```
61
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  ## Output
63
 
64
  By default the inference script writes:
 
68
  Each output file contains one JSONL row with:
69
 
70
  - input video path
71
+ - optional task instruction and task plan text
72
  - sampled frame indices and timestamps
73
  - decoded video statistics
74
  - prompt text
quick_start/run_example.py CHANGED
@@ -7,7 +7,6 @@ import os
7
  import re
8
  import tempfile
9
  from pathlib import Path
10
- from typing import Any
11
 
12
  import numpy as np
13
  from PIL import Image
@@ -38,14 +37,8 @@ def parse_args() -> argparse.Namespace:
38
  parser.add_argument(
39
  "--video-path",
40
  type=Path,
41
- default=None,
42
- help="Path to the input video. Optional when --metadata-path is provided and metadata contains video_path.",
43
- )
44
- parser.add_argument(
45
- "--metadata-path",
46
- type=Path,
47
- default=None,
48
- help="Optional metadata JSON path. If provided, task fields and relative video_path can be inferred from it.",
49
  )
50
  parser.add_argument(
51
  "--task-instruction",
@@ -86,29 +79,6 @@ def parse_args() -> argparse.Namespace:
86
  return parser.parse_args()
87
 
88
 
89
- def load_metadata(metadata_path: Path | None) -> tuple[Path | None, dict[str, Any] | None]:
90
- if metadata_path is None:
91
- return None, None
92
- resolved = metadata_path.resolve()
93
- if not resolved.exists():
94
- raise SystemExit(f"Missing metadata JSON: {resolved}")
95
- payload = json.loads(resolved.read_text(encoding="utf-8"))
96
- if not isinstance(payload, dict):
97
- raise SystemExit(f"Metadata JSON must contain an object: {resolved}")
98
- return resolved, payload
99
-
100
-
101
- def extract_plan_text(task_instruction: str, task_description: str) -> str:
102
- lines = [line.strip() for line in str(task_description or "").splitlines() if line.strip()]
103
- if not lines:
104
- return ""
105
- first_line = lines[0].rstrip("::")
106
- normalized_instruction = str(task_instruction or "").strip().rstrip("::")
107
- if normalized_instruction and first_line == normalized_instruction:
108
- lines = lines[1:]
109
- return "\n".join(lines).strip()
110
-
111
-
112
  def build_chunk_all_prompt(task_instruction: str, task_plan: str | None) -> str:
113
  task_instruction = task_instruction.strip()
114
  task_plan = (task_plan or "").strip()
@@ -122,54 +92,26 @@ def build_chunk_all_prompt(task_instruction: str, task_plan: str | None) -> str:
122
  )
123
 
124
 
125
- def resolve_prompt(args: argparse.Namespace, metadata: dict[str, Any] | None) -> tuple[str, str, str | None, str | None]:
126
  prompt = args.prompt.strip() if args.prompt else None
127
  task_plan = args.task_plan.strip() if args.task_plan else None
128
-
129
- metadata_task_instruction = None
130
- metadata_task_plan = None
131
- if metadata is not None:
132
- metadata_task_instruction = str(metadata.get("task_instruction") or "").strip() or None
133
- metadata_task_description = str(metadata.get("task_description") or "").strip()
134
- if metadata_task_instruction and metadata_task_description:
135
- metadata_task_plan = extract_plan_text(metadata_task_instruction, metadata_task_description) or None
136
-
137
- task_instruction = args.task_instruction.strip() if args.task_instruction else metadata_task_instruction
138
- if task_plan is None:
139
- task_plan = metadata_task_plan
140
 
141
  if prompt:
142
  return prompt, "raw_prompt", task_instruction, task_plan
143
 
144
  if not task_instruction:
145
  raise SystemExit(
146
- "Either provide --prompt, or provide --task-instruction, "
147
- "or provide --metadata-path with task_instruction."
148
  )
149
 
150
  return build_chunk_all_prompt(task_instruction, task_plan), "task_instruction_plus_plan", task_instruction, task_plan
151
 
152
 
153
- def resolve_video_path(
154
- video_path_arg: Path | None,
155
- metadata: dict[str, Any] | None,
156
- ) -> Path:
157
- if video_path_arg is not None:
158
- video_path = video_path_arg.resolve()
159
- if not video_path.exists():
160
- raise SystemExit(f"Missing video: {video_path}")
161
- return video_path
162
-
163
- if metadata is None or not metadata.get("video_path"):
164
- raise SystemExit("Either provide --video-path, or provide --metadata-path with video_path.")
165
-
166
- metadata_video_path = Path(str(metadata["video_path"]))
167
- if metadata_video_path.is_absolute():
168
- video_path = metadata_video_path
169
- else:
170
- video_path = (MODEL_ROOT / metadata_video_path).resolve()
171
  if not video_path.exists():
172
- raise SystemExit(f"Missing video resolved from metadata: {video_path}")
173
  return video_path
174
 
175
 
@@ -333,13 +275,12 @@ def parse_point_blocks(text: str) -> tuple[list[float], list[float]]:
333
 
334
  def main() -> int:
335
  args = parse_args()
336
- metadata_path, metadata = load_metadata(args.metadata_path)
337
  model_path = args.model_path.resolve()
338
  if not model_path.exists():
339
  raise SystemExit(f"Missing model path: {model_path}")
340
 
341
- prompt, prompt_source, task_instruction, task_plan = resolve_prompt(args, metadata)
342
- video_path = resolve_video_path(args.video_path, metadata)
343
  video_path_for_output = maybe_relativize_to_model_root(video_path)
344
  output_path = args.output_jsonl or (THIS_DIR / "outputs" / f"{video_path.stem}.jsonl")
345
  output_path.parent.mkdir(parents=True, exist_ok=True)
@@ -377,17 +318,11 @@ def main() -> int:
377
 
378
  pred_point_times_sec, pred_point_progress = parse_point_blocks(response)
379
 
380
- output_row: dict[str, Any] = {}
381
- if metadata is not None:
382
- for key in ("example_id", "bucket", "global_episode_id", "task_instruction", "task_description"):
383
- if key in metadata:
384
- output_row[key] = metadata.get(key)
385
- if "task_instruction" not in output_row and task_instruction is not None:
386
  output_row["task_instruction"] = task_instruction
387
  if task_plan:
388
  output_row["task_plan"] = task_plan
389
- if metadata_path is not None:
390
- output_row["metadata_path"] = maybe_relativize_to_model_root(metadata_path)
391
 
392
  output_row.update(
393
  {
 
7
  import re
8
  import tempfile
9
  from pathlib import Path
 
10
 
11
  import numpy as np
12
  from PIL import Image
 
37
  parser.add_argument(
38
  "--video-path",
39
  type=Path,
40
+ required=True,
41
+ help="Path to the input video.",
 
 
 
 
 
 
42
  )
43
  parser.add_argument(
44
  "--task-instruction",
 
79
  return parser.parse_args()
80
 
81
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  def build_chunk_all_prompt(task_instruction: str, task_plan: str | None) -> str:
83
  task_instruction = task_instruction.strip()
84
  task_plan = (task_plan or "").strip()
 
92
  )
93
 
94
 
95
+ def resolve_prompt(args: argparse.Namespace) -> tuple[str, str, str | None, str | None]:
96
  prompt = args.prompt.strip() if args.prompt else None
97
  task_plan = args.task_plan.strip() if args.task_plan else None
98
+ task_instruction = args.task_instruction.strip() if args.task_instruction else None
 
 
 
 
 
 
 
 
 
 
 
99
 
100
  if prompt:
101
  return prompt, "raw_prompt", task_instruction, task_plan
102
 
103
  if not task_instruction:
104
  raise SystemExit(
105
+ "Either provide --prompt, or provide --task-instruction with optional --task-plan."
 
106
  )
107
 
108
  return build_chunk_all_prompt(task_instruction, task_plan), "task_instruction_plus_plan", task_instruction, task_plan
109
 
110
 
111
+ def resolve_video_path(video_path_arg: Path) -> Path:
112
+ video_path = video_path_arg.resolve()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
  if not video_path.exists():
114
+ raise SystemExit(f"Missing video: {video_path}")
115
  return video_path
116
 
117
 
 
275
 
276
  def main() -> int:
277
  args = parse_args()
 
278
  model_path = args.model_path.resolve()
279
  if not model_path.exists():
280
  raise SystemExit(f"Missing model path: {model_path}")
281
 
282
+ prompt, prompt_source, task_instruction, task_plan = resolve_prompt(args)
283
+ video_path = resolve_video_path(args.video_path)
284
  video_path_for_output = maybe_relativize_to_model_root(video_path)
285
  output_path = args.output_jsonl or (THIS_DIR / "outputs" / f"{video_path.stem}.jsonl")
286
  output_path.parent.mkdir(parents=True, exist_ok=True)
 
318
 
319
  pred_point_times_sec, pred_point_progress = parse_point_blocks(response)
320
 
321
+ output_row: dict[str, object] = {}
322
+ if task_instruction is not None:
 
 
 
 
323
  output_row["task_instruction"] = task_instruction
324
  if task_plan:
325
  output_row["task_plan"] = task_plan
 
 
326
 
327
  output_row.update(
328
  {