futurefantasy commited on
Commit
81cb25c
verified
1 Parent(s): dcbfddc

Update VLAC-Cut model card links and quick start

Browse files
Files changed (1) hide show
  1. README.md +11 -26
README.md CHANGED
@@ -16,7 +16,7 @@ license: other
16
 
17
  **Video-based task progress estimation model for robot manipulation.**
18
 
19
- [Model](https://huggingface.co/InternRobotics/VLAC-Cut) 路 [Code](https://github.com/InternRobotics/VLAC-Cut) 路 [Paper](<PAPER_URL>) 路 [VPB Benchmark](https://huggingface.co/datasets/InternRobotics/VLAC-Cut-Benchmark)
20
 
21
  VLAC-Cut predicts how a robot task progresses over time from a natural-language instruction and an execution video. Instead of assuming that progress always increases with time, the model captures non-monotonic execution patterns including advancement, stagnation, regression, and recovery.
22
 
@@ -24,10 +24,9 @@ The model is designed for long-horizon robot manipulation videos containing fail
24
 
25
  ## Highlights
26
 
27
- - **Video-level progress understanding:** reasons over robot videos rather than isolated images or image pairs, and cuts the video into good segments and bad segments.
28
  - **Non-monotonic process modeling:** recognizes advancement, stagnation, regression, and recovery.
29
- - **General understanding capability:** General understanding capability, enabling zero-shot generalization across tasks, scenes, and viewpoints.
30
- - **Coarse/Fine-grained flexible:** The ability to adjust the frequency of video understanding, achieving progress analysis from coarse-grained to fine-grained.
31
 
32
  ## Model at a Glance
33
 
@@ -36,13 +35,13 @@ The model is designed for long-horizon robot manipulation videos containing fail
36
  | Base model | `Qwen/Qwen3-VL-30B-A3B-Instruct` |
37
  | Input | Task instruction, optional task plan, and sampled video frames |
38
  | Output | Timestamped task progress |
39
- | flexiable video sampling rate | `2 Hz~20Hz`(Default: `2.0`) |
40
 
41
  ## Model Design
42
 
43
  VLAC-Cut is fine-tuned from Qwen3-VL-30B-A3B-Instruct and retains its original multimodal generation interface. The model receives a task instruction, an optional task plan, and sampled video frames, then generates task progress at different timestamps as text.
44
 
45
- The training data also includes supervision for robot behavior description, failure analysis, and correction planning, helping the model understand realistic robot execution processes.
46
 
47
  This repository contains the model checkpoint and loading assets only. Inference scripts, examples, and VPB evaluation code are maintained in the GitHub repository.
48
 
@@ -67,15 +66,15 @@ model.eval()
67
  Run progress inference on a local video using the GitHub code:
68
 
69
  ```bash
70
- git clone https://github.com/InternRobotics/VLAC-Cut
71
- cd VLAC-Cut
72
 
73
  python scripts/run_example.py \
74
  --model-path InternRobotics/VLAC-Cut \
75
- --video-path examples/example_01/episode.mp4 \
76
  --task-instruction "<natural-language task instruction>" \
77
  --task-plan $'<optional step-by-step task plan>' \
78
- --output-jsonl outputs/example_01.jsonl
79
  ```
80
 
81
  Main arguments:
@@ -100,24 +99,10 @@ Render a prediction preview video:
100
 
101
  ```bash
102
  python scripts/utils/render_prediction_video.py \
103
- --input-jsonl outputs/example_01.jsonl \
104
- --output-video outputs/example_01_pred_progress.mp4
105
  ```
106
 
107
- ## Evaluation on VPB
108
-
109
- VLAC-Cut is evaluated on the Video Progress Benchmark. The benchmark data are released at `InternRobotics/VLAC-Cut-Benchmark`, and the evaluator is provided in the GitHub repository:
110
-
111
- ```bash
112
- python scripts/evaluate_vpb_predictions.py \
113
- --benchmark-root /path/to/VLAC-Cut-Benchmark/splits \
114
- --predictions predictions/vlac_cut_predictions.jsonl \
115
- --out-json reports/vlac_cut_vpb_eval.json \
116
- --out-md reports/vlac_cut_vpb_eval.md
117
- ```
118
-
119
- Report final VPB numbers here before public release.
120
-
121
  ## Citation
122
 
123
  Please replace the placeholder below with the final paper BibTeX before public release.
 
16
 
17
  **Video-based task progress estimation model for robot manipulation.**
18
 
19
+ [Paper](https://arxiv.org/abs/2607.09776) 路 [Code](https://github.com/InternRobotics/VLAC-cut) 路 [Model](https://huggingface.co/InternRobotics/VLAC-Cut) 路 [Benchmark](https://huggingface.co/datasets/InternRobotics/VLAC-Cut-Benchmark)
20
 
21
  VLAC-Cut predicts how a robot task progresses over time from a natural-language instruction and an execution video. Instead of assuming that progress always increases with time, the model captures non-monotonic execution patterns including advancement, stagnation, regression, and recovery.
22
 
 
24
 
25
  ## Highlights
26
 
27
+ - **Video-level progress understanding:** reasons over temporally ordered robot videos rather than isolated images or image pairs.
28
  - **Non-monotonic process modeling:** recognizes advancement, stagnation, regression, and recovery.
29
+ - **Generative output:** produces timestamped progress predictions through the standard autoregressive language-model interface, without a task-specific regression head.
 
30
 
31
  ## Model at a Glance
32
 
 
35
  | Base model | `Qwen/Qwen3-VL-30B-A3B-Instruct` |
36
  | Input | Task instruction, optional task plan, and sampled video frames |
37
  | Output | Timestamped task progress |
38
+ | Default video sampling rate | `2 Hz` |
39
 
40
  ## Model Design
41
 
42
  VLAC-Cut is fine-tuned from Qwen3-VL-30B-A3B-Instruct and retains its original multimodal generation interface. The model receives a task instruction, an optional task plan, and sampled video frames, then generates task progress at different timestamps as text.
43
 
44
+ The training data also include supervision for robot behavior description, failure analysis, and correction planning, helping the model understand realistic robot execution processes.
45
 
46
  This repository contains the model checkpoint and loading assets only. Inference scripts, examples, and VPB evaluation code are maintained in the GitHub repository.
47
 
 
66
  Run progress inference on a local video using the GitHub code:
67
 
68
  ```bash
69
+ git clone https://github.com/InternRobotics/VLAC-cut
70
+ cd VLAC-cut
71
 
72
  python scripts/run_example.py \
73
  --model-path InternRobotics/VLAC-Cut \
74
+ --video-path <path-to-video.mp4> \
75
  --task-instruction "<natural-language task instruction>" \
76
  --task-plan $'<optional step-by-step task plan>' \
77
+ --output-jsonl <path-to-output.jsonl>
78
  ```
79
 
80
  Main arguments:
 
99
 
100
  ```bash
101
  python scripts/utils/render_prediction_video.py \
102
+ --input-jsonl <path-to-output.jsonl> \
103
+ --output-video <path-to-preview.mp4>
104
  ```
105
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
106
  ## Citation
107
 
108
  Please replace the placeholder below with the final paper BibTeX before public release.