Update model card and repository split

Keep model assets on the Hub and move examples/quick-start code to the GitHub repository.

LICENSE

@@ -0,0 +1,3 @@
+TODO: replace this file with the final VLAC-Cut model license before public release.
+
+Do not publish this model repository as open source until the license text has been finalized.

NOTICE

@@ -0,0 +1,3 @@
+VLAC-Cut model release
+
+TODO: replace this file with the final model notice, base-model attribution, and third-party notices before public release.

README.md

@@ -1,26 +1,51 @@
 ---
 library_name: transformers
 pipeline_tag: image-text-to-text
+base_model: Qwen/Qwen3-VL-30B-A3B-Instruct
 tags:
   - robotics
+  - embodied-ai
   - video-understanding
-  - multimodal
   - progress-estimation
+  - reward-modeling
+  - qwen3-vl
+license: other
 ---
 
 # VLAC-Cut
 
-VLAC-Cut is a video-language progress estimation model for robotic manipulation. Given a task description and a video, it predicts time-progress keypoints that can be aligned into a progress curve.
+**Video-based task progress estimation model for robot manipulation.**
 
-This release contains the inference-ready model weights, tokenizer and processor files, three example videos, reference prediction outputs, and minimal local inference utilities.
+[Model](https://huggingface.co/InternRobotics/VLAC-Cut) · [Code](https://github.com/InternRobotics/VLAC-Cut) · [Paper](<PAPER_URL>) · [VPB Benchmark](https://huggingface.co/datasets/InternRobotics/VLAC-Cut-Benchmark)
+
+VLAC-Cut predicts how a robot task progresses over time from a natural-language instruction and an execution video. Instead of assuming that progress always increases with time, the model captures non-monotonic execution patterns including advancement, stagnation, regression, and recovery.
+
+The model is designed for long-horizon robot manipulation videos containing failed grasps, object drops, incorrect interactions, off-target placements, and corrective behavior.
 
 ## Highlights
 
-- Progress estimation for long-horizon manipulation videos
-- Rollback and regression recognition in non-expert trajectories
-- Example videos from both expert and non-expert trajectories
+- **Video-level progress understanding:** reasons over temporally ordered robot videos rather than isolated images or image pairs.
+- **Non-monotonic process modeling:** recognizes advancement, stagnation, regression, and recovery.
+- **Generative output:** produces timestamped progress predictions through the standard autoregressive language-model interface, without a task-specific regression head.
+
+## Model at a Glance
+
+| Property | Description |
+|---|---|
+| Base model | `Qwen/Qwen3-VL-30B-A3B-Instruct` |
+| Input | Task instruction, optional task plan, and sampled video frames |
+| Output | Timestamped task progress |
+| Default video sampling rate | `2 Hz` |
+
+## Model Design
+
+VLAC-Cut is fine-tuned from Qwen3-VL-30B-A3B-Instruct and retains its original multimodal generation interface. The model receives a task instruction, an optional task plan, and sampled video frames, then generates task progress at different timestamps as text.
 
-## Use with Transformers
+The training data also include supervision for robot behavior description, failure analysis, and correction planning, helping the model understand realistic robot execution processes.
+
+This repository contains the model checkpoint and loading assets only. Inference scripts, examples, and VPB evaluation code are maintained in the GitHub repository.
+
+## Load with Transformers
 
 ```python
 from transformers import AutoModelForImageTextToText, AutoProcessor
@@ -30,39 +55,76 @@ model_id = "InternRobotics/VLAC-Cut"
 processor = AutoProcessor.from_pretrained(model_id)
 model = AutoModelForImageTextToText.from_pretrained(
     model_id,
-    torch_dtype="auto",
+    dtype="auto",
     device_map="auto",
 )
+model.eval()
 ```
 
 ## Quick Start
 
-Run progress inference on an arbitrary video:
+Run progress inference on a local video using the GitHub code:
 
 ```bash
-python quick_start/run_example.py \
-  --model-path /path/to/VLAC-Cut \
-  --video-path /path/to/video.mp4 \
+git clone https://github.com/InternRobotics/VLAC-Cut
+cd VLAC-Cut
+
+python scripts/run_example.py \
+  --model-path InternRobotics/VLAC-Cut \
+  --video-path examples/example_01/episode.mp4 \
   --task-instruction "<natural-language task instruction>" \
   --task-plan $'<optional step-by-step task plan>' \
-  --output-jsonl quick_start/outputs/video.jsonl
+  --output-jsonl outputs/example_01.jsonl
+```
+
+Main arguments:
+
+- `--model-path`: Hugging Face model id or local checkpoint path.
+- `--video-path`: input video path.
+- `--task-instruction`: natural-language task description.
+- `--task-plan`: optional step-by-step task plan.
+- `--sample-hz`: video sampling rate. Default: `2.0`.
+- `--prompt`: optional full-prompt override.
+- `--output-jsonl`: optional output path; if omitted, the model response is printed only.
+
+Example response:
+
+```text
+时间: 0.0s, 进度: 0%
+时间: 1.0s, 进度: 25%
+时间: 2.0s, 进度: 40%
+```
+
+Render a prediction preview video:
+
+```bash
+python scripts/utils/render_prediction_video.py \
+  --input-jsonl outputs/example_01.jsonl \
+  --output-video outputs/example_01_pred_progress.mp4
 ```
 
-- `--video-path`: required input video.
-- `--task-instruction`: task description, unless `--prompt` is used instead.
-- `--task-plan`: optional step-by-step plan text for the default `chunk_all` prompt.
-- `--model-path`: optional. Defaults to the parent directory of `quick_start/`.
-- `--output-jsonl`: optional. If omitted, the script prints the prediction only and does not save a JSONL file.
-- `--prompt`: optional full prompt override.
-- `--sample-hz`: optional inference control. Default: `2.0`.
+## Evaluation on VPB
 
-Render a prediction-only preview video from the JSONL output:
+VLAC-Cut is evaluated on the Video Progress Benchmark. The benchmark data are released at `InternRobotics/VLAC-Cut-Benchmark`, and the evaluator is provided in the GitHub repository:
 
 ```bash
-python quick_start/render_prediction_video.py \
-  --input-jsonl quick_start/outputs/video.jsonl
+python scripts/evaluate_vpb_predictions.py \
+  --benchmark-root /path/to/VLAC-Cut-Benchmark/splits \
+  --predictions predictions/vlac_cut_predictions.jsonl \
+  --out-json reports/vlac_cut_vpb_eval.json \
+  --out-md reports/vlac_cut_vpb_eval.md
+```
+
+Report final VPB numbers here before public release.
+
+## Citation
+
+Please replace the placeholder below with the final paper BibTeX before public release.
+
+```bibtex
+
 ```
 
-Videos under `examples/` can be used with the same interface by passing the video path and prompt text directly.
+## License
 
-More detailed usage is provided in `quick_start/README.md`.
+See the license files distributed with this checkpoint. The base model and third-party training data remain subject to their original licenses and terms of use.

examples/example_01/episode.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:107c36cd11846dc0d622a2168fb0d2129d912dda5301c6748e0a1417ff5cde28
-size 1477548

examples/example_02/episode.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:52c1494005556db7242d49982cc0ee2d2be31962e0ce9dd59a590f96b5462ec5
-size 1908478

examples/example_03/episode.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:9221f9782d815175fd851555d78e3950d4bd42ff362569b5d66da7be8f7df724
-size 1047570

examples/reference_outputs/example_01.jsonl

@@ -1 +0,0 @@
-{"task_instruction": "抓取紫色方块使其从方形洞口落入积木桶中", "task_plan": "爪夹开始移动：0%\n爪夹接近紫色方块：20%\n爪夹抓紧紫色方块：40%\n爪夹抓紧紫色方块接近方形洞口：60%\n爪夹将紫色方块对准方形洞口：80%\n紫色方块从方形洞口落入积木桶中，爪夹移开：100%", "video_path": "examples/example_01/episode.mp4", "prompt_source": "task_instruction_plus_plan", "prompt_variant": "chunk_all", "input_sample_hz": 2.0, "input_frame_indices": [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225, 240, 255], "input_timestamps_sec": [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5], "input_frame_count": 18, "decoded_video_stats": {"decoded_frame_count": 258.0, "decoded_avg_fps": 30.0, "decoded_duration_sec": 8.566667}, "prompt": "任务描述和具体规划: 抓取紫色方块使其从方形洞口落入积木桶中\n爪夹开始移动：0%\n爪夹接近紫色方块：20%\n爪夹抓紧紫色方块：40%\n爪夹抓紧紫色方块接近方形洞口：60%\n爪夹将紫色方块对准方形洞口：80%\n紫色方块从方形洞口落入积木桶中，爪夹移开：100%\n\n请根据任务描述和具体规划，找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求：每个关键点一行，格式为：\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出，不要输出额外说明。", "response": "时间: 1.3s, 进度: 0%\n时间: 2.7s, 进度: 10%\n时间: 3.7s, 进度: 15%\n时间: 5.5s, 进度: 10%\n时间: 6.3s, 进度: -15%\n时间: 7.1s, 进度: -25%\n时间: 8.0s, 进度: -15%", "pred_curve_parse_ok": true, "pred_curve_point_times_sec": [1.3, 2.7, 3.7, 5.5, 6.3, 7.1, 8.0], "pred_curve_point_progress": [0.0, 10.0, 15.0, 10.0, -15.0, -25.0, -15.0], "preview_video_path": "examples/reference_outputs/example_01_pred_progress.mp4"}

examples/reference_outputs/example_01_pred_progress.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:3ea6fb101f9823664cd7531ce5297f9490c40e6de7e5c1bb584b9d6fa5223301
-size 2043716

examples/reference_outputs/example_02.jsonl

@@ -1 +0,0 @@
-{"task_instruction": "把洋葱放进快递箱里。", "task_plan": "开始移动:0%\n爪夹靠近洋葱:20%\n爪夹抓取洋葱:40%\n爪夹抓住洋葱接近快递箱:60%\n爪夹将洋葱放入快递箱:80%\n爪夹松开,洋葱落入快递箱:100%", "video_path": "examples/example_02/episode.mp4", "prompt_source": "task_instruction_plus_plan", "prompt_variant": "chunk_all", "input_sample_hz": 2.0, "input_frame_indices": [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225, 240, 255, 270, 285], "input_timestamps_sec": [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 9.5], "input_frame_count": 20, "decoded_video_stats": {"decoded_frame_count": 289.0, "decoded_avg_fps": 30.0, "decoded_duration_sec": 9.6}, "prompt": "任务描述和具体规划: 把洋葱放进快递箱里。\n开始移动:0%\n爪夹靠近洋葱:20%\n爪夹抓取洋葱:40%\n爪夹抓住洋葱接近快递箱:60%\n爪夹将洋葱放入快递箱:80%\n爪夹松开,洋葱落入快递箱:100%\n\n请根据任务描述和具体规划，找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求：每个关键点一行，格式为：\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出，不要输出额外说明。", "response": "时间: 0.0s, 进度: 0%\n时间: 2.3s, 进度: 5%\n时间: 3.3s, 进度: 8%\n时间: 4.3s, 进度: 10%\n时间: 5.1s, 进度: -5%\n时间: 6.1s, 进度: -8%\n时间: 7.1s, 进度: -2%\n时间: 8.3s, 进度: 2%\n时间: 9.5s, 进度: 4%", "pred_curve_parse_ok": true, "pred_curve_point_times_sec": [0.0, 2.3, 3.3, 4.3, 5.1, 6.1, 7.1, 8.3, 9.5], "pred_curve_point_progress": [0.0, 5.0, 8.0, 10.0, -5.0, -8.0, -2.0, 2.0, 4.0], "preview_video_path": "examples/reference_outputs/example_02_pred_progress.mp4"}

examples/reference_outputs/example_02_pred_progress.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:691fe25a519518ac2c24ba86c71ab7714d41f5616482cc0f079b874f897f679f
-size 2546146

examples/reference_outputs/example_03.jsonl

@@ -1 +0,0 @@
-{"task_instruction": "将三角烧杯放在三脚架上。", "task_plan": "爪夹准备移动：0%\n爪夹开始移动：10%\n爪夹靠近三角烧杯：20%\n爪夹抓取三角烧杯：40%\n爪夹合拢并靠近三脚架：60%\n爪夹移动到三脚架的正上方：80%\n爪夹松开，三角烧杯落在三脚架上：100%", "video_path": "examples/example_03/episode.mp4", "prompt_source": "task_instruction_plus_plan", "prompt_variant": "chunk_all", "input_sample_hz": 2.0, "input_frame_indices": [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165], "input_timestamps_sec": [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5], "input_frame_count": 12, "decoded_video_stats": {"decoded_frame_count": 172.0, "decoded_avg_fps": 30.0, "decoded_duration_sec": 5.7}, "prompt": "任务描述和具体规划: 将三角烧杯放在三脚架上。\n爪夹准备移动：0%\n爪夹开始移动：10%\n爪夹靠近三角烧杯：20%\n爪夹抓取三角烧杯：40%\n爪夹合拢并靠近三脚架：60%\n爪夹移动到三脚架的正上方：80%\n爪夹松开，三角烧杯落在三脚架上：100%\n\n请根据任务描述和具体规划，找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求：每个关键点一行，格式为：\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出，不要输出额外说明。", "response": "时间: 0.1s, 进度: 0%\n时间: 0.6s, 进度: 10%\n时间: 1.1s, 进度: 20%\n时间: 1.6s, 进度: 30%\n时间: 2.3s, 进度: 40%\n时间: 3.4s, 进度: 60%\n时间: 4.1s, 进度: 80%\n时间: 5.3s, 进度: 100%", "pred_curve_parse_ok": true, "pred_curve_point_times_sec": [0.1, 0.6, 1.1, 1.6, 2.3, 3.4, 4.1, 5.3], "pred_curve_point_progress": [0.0, 10.0, 20.0, 30.0, 40.0, 60.0, 80.0, 100.0], "preview_video_path": "examples/reference_outputs/example_03_pred_progress.mp4"}

examples/reference_outputs/example_03_pred_progress.mp4

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:211dfef38fd8e374f20b4bcf83a607fd6e3461fafb53a8228386995bbab118fe
-size 1436259

quick_start/README.md

@@ -1,93 +0,0 @@
-# Quick Start
-
-This quick start has two steps:
-
-1. Run progress inference on a video. Optionally save one JSONL output.
-2. Render a prediction-only preview video from a saved JSONL output.
-
-## Requirements
-
-- `python >= 3.10`
-- `torch`
-- `transformers >= 4.57`
-- `ms-swift`
-- `decord`
-- `numpy`
-- `Pillow`
-- `opencv-python` for preview video rendering
-- at least `1` GPU with enough memory for this `30B` checkpoint
-
-## Usage
-
-Run inference:
-
-```bash
-python quick_start/run_example.py \
-  --model-path /path/to/VLAC-Cut \
-  --video-path /path/to/video.mp4 \
-  --task-instruction "<natural-language task instruction>" \
-  --task-plan $'<optional step-by-step task plan>' \
-  --output-jsonl quick_start/outputs/video.jsonl
-```
-
-Important arguments:
-
-- `--video-path`: required. Input video path.
-- `--task-instruction`: optional if `--prompt` is provided. Natural-language task description.
-- `--task-plan`: optional. Multi-line plan text appended after `--task-instruction` when building the default `chunk_all` prompt.
-- `--model-path`: optional. Defaults to the parent directory of `quick_start/`.
-- `--output-jsonl`: optional. If omitted, the script prints the prediction only and does not save a JSONL file.
-- `--sample-hz`: optional. Video sampling rate for inference. Default: `2.0`.
-- `--prompt`: optional full prompt override. If set, `--task-instruction` and `--task-plan` are ignored.
-
-Render a preview video:
-
-```bash
-python quick_start/render_prediction_video.py \
-  --input-jsonl quick_start/outputs/video.jsonl \
-  --output-video quick_start/outputs/video_pred_progress.mp4
-```
-
-## Example Commands
-
-`example_01`
-
-```bash
-python quick_start/run_example.py \
-  --video-path examples/example_01/episode.mp4 \
-  --task-instruction "抓取紫色方块使其从方形洞口落入积木桶中" \
-  --task-plan $'爪夹开始移动：0%\n爪夹接近紫色方块：20%\n爪夹抓紧紫色方块：40%\n爪夹抓紧紫色方块接近方形洞口：60%\n爪夹将紫色方块对准方形洞口：80%\n紫色方块从方形洞口落入积木桶中，爪夹移开：100%' \
-  --output-jsonl quick_start/outputs/example_01.jsonl
-
-python quick_start/render_prediction_video.py \
-  --input-jsonl quick_start/outputs/example_01.jsonl \
-  --output-video quick_start/outputs/example_01_pred_progress.mp4
-```
-
-`example_02`
-
-```bash
-python quick_start/run_example.py \
-  --video-path examples/example_02/episode.mp4 \
-  --task-instruction "把洋葱放进快递箱里。" \
-  --task-plan $'开始移动:0%\n爪夹靠近洋葱:20%\n爪夹抓取洋葱:40%\n爪夹抓住洋葱接近快递箱:60%\n爪夹将洋葱放入快递箱:80%\n爪夹松开,洋葱落入快递箱:100%' \
-  --output-jsonl quick_start/outputs/example_02.jsonl
-
-python quick_start/render_prediction_video.py \
-  --input-jsonl quick_start/outputs/example_02.jsonl \
-  --output-video quick_start/outputs/example_02_pred_progress.mp4
-```
-
-`example_03`
-
-```bash
-python quick_start/run_example.py \
-  --video-path examples/example_03/episode.mp4 \
-  --task-instruction "将三角烧杯放在三脚架上。" \
-  --task-plan $'爪夹准备移动：0%\n爪夹开始移动：10%\n爪夹靠近三角烧杯：20%\n爪夹抓取三角烧杯：40%\n爪夹合拢并靠近三脚架：60%\n爪夹移动到三脚架的正上方：80%\n爪夹松开，三角烧杯落在三脚架上：100%' \
-  --output-jsonl quick_start/outputs/example_03.jsonl
-
-python quick_start/render_prediction_video.py \
-  --input-jsonl quick_start/outputs/example_03.jsonl \
-  --output-video quick_start/outputs/example_03_pred_progress.mp4
-```

quick_start/render_prediction_video.py

@@ -1,243 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import json
-import math
-from pathlib import Path
-
-import cv2
-import numpy as np
-
-THIS_DIR = Path(__file__).resolve().parent
-MODEL_ROOT = THIS_DIR.parent
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Render a predicted-progress preview video from a quick_start JSONL output.",
-    )
-    parser.add_argument(
-        "--input-jsonl",
-        type=Path,
-        required=True,
-        help="Path to the quick_start JSONL output.",
-    )
-    parser.add_argument(
-        "--video-path",
-        type=Path,
-        default=None,
-        help="Optional input video override. Defaults to the video_path stored in the JSONL row.",
-    )
-    parser.add_argument(
-        "--output-video",
-        type=Path,
-        default=None,
-        help="Optional output path. Defaults to <input_jsonl_stem>_pred_progress.mp4",
-    )
-    parser.add_argument(
-        "--panel-height",
-        type=int,
-        default=260,
-        help="Height in pixels for the lower progress panel.",
-    )
-    return parser.parse_args()
-
-
-def load_row(path: Path) -> dict:
-    lines = path.read_text(encoding="utf-8").splitlines()
-    if not lines:
-        raise SystemExit(f"Empty JSONL file: {path}")
-    return json.loads(lines[0])
-
-
-def resolve_video_path(path_arg: Path | None, row: dict) -> Path:
-    if path_arg is not None:
-        path = path_arg.resolve()
-        if not path.exists():
-            raise SystemExit(f"Missing video: {path}")
-        return path
-
-    row_video_path = row.get("video_path")
-    if not row_video_path:
-        raise SystemExit("Missing video_path in JSONL row. Pass --video-path explicitly.")
-    candidate = Path(str(row_video_path))
-    if candidate.is_absolute():
-        path = candidate
-    else:
-        path = (MODEL_ROOT / candidate).resolve()
-    if not path.exists():
-        raise SystemExit(f"Missing video resolved from JSONL row: {path}")
-    return path
-
-
-def render_preview(
-    *,
-    row: dict,
-    video_path: Path,
-    output_path: Path,
-    panel_height: int,
-) -> None:
-    pred_times = [float(x) for x in row.get("pred_curve_point_times_sec", [])]
-    pred_vals = [float(x) for x in row.get("pred_curve_point_progress", [])]
-    if not pred_times or not pred_vals:
-        raise SystemExit("No predicted progress points found in JSONL row.")
-
-    cap = cv2.VideoCapture(str(video_path))
-    if not cap.isOpened():
-        raise SystemExit(f"Failed to open video: {video_path}")
-
-    fps = float(cap.get(cv2.CAP_PROP_FPS))
-    frame_count = int(round(cap.get(cv2.CAP_PROP_FRAME_COUNT)))
-    frame_w = int(round(cap.get(cv2.CAP_PROP_FRAME_WIDTH)))
-    frame_h = int(round(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)))
-    if fps <= 0 or frame_count <= 0 or frame_w <= 0 or frame_h <= 0:
-        cap.release()
-        raise SystemExit(
-            f"Invalid decoded video stats from {video_path}: fps={fps}, frames={frame_count}, "
-            f"width={frame_w}, height={frame_h}"
-        )
-
-    total_h = frame_h + panel_height
-    writer = cv2.VideoWriter(str(output_path), cv2.VideoWriter_fourcc(*"mp4v"), fps, (frame_w, total_h))
-    if not writer.isOpened():
-        cap.release()
-        raise SystemExit(f"Failed to open output video writer: {output_path}")
-
-    duration = max(0.0, (frame_count - 1) / fps)
-    dense_t = np.arange(frame_count, dtype=np.float32) / fps
-    dense_v = np.interp(
-        dense_t,
-        np.array(pred_times, dtype=np.float32),
-        np.array(pred_vals, dtype=np.float32),
-        left=float(pred_vals[0]),
-        right=float(pred_vals[-1]),
-    )
-
-    y_min = min(-40.0, math.floor(min(pred_vals) / 10.0) * 10.0 - 10.0)
-    y_max = max(100.0, math.ceil(max(pred_vals) / 10.0) * 10.0 + 10.0)
-    left, right, top, bottom = 78, 24, 28, 52
-    chart_x0, chart_x1 = left, frame_w - right
-    chart_y0, chart_y1 = frame_h + top, total_h - bottom
-    tick_vals = np.linspace(y_min, y_max, num=6)
-    title = str(row.get("example_id") or video_path.stem)
-
-    def map_x(t: float) -> int:
-        if duration <= 1e-9:
-            return chart_x0
-        return int(round(chart_x0 + (chart_x1 - chart_x0) * (t / duration)))
-
-    def map_y(v: float) -> int:
-        ratio = (v - y_min) / max(1e-9, (y_max - y_min))
-        return int(round(chart_y1 - ratio * (chart_y1 - chart_y0)))
-
-    try:
-        idx = 0
-        while True:
-            ok, frame = cap.read()
-            if not ok:
-                break
-            if idx >= frame_count:
-                break
-
-            canvas = np.full((total_h, frame_w, 3), 255, dtype=np.uint8)
-            canvas[:frame_h, :, :] = frame
-
-            cv2.putText(
-                canvas,
-                f"{title} predicted progress so far",
-                (18, frame_h + 22),
-                cv2.FONT_HERSHEY_SIMPLEX,
-                0.65,
-                (20, 20, 20),
-                2,
-                cv2.LINE_AA,
-            )
-            cv2.rectangle(canvas, (chart_x0, chart_y0), (chart_x1, chart_y1), (210, 210, 210), 1)
-
-            for tick in tick_vals:
-                y = map_y(float(tick))
-                cv2.line(canvas, (chart_x0, y), (chart_x1, y), (238, 238, 238), 1)
-                cv2.putText(
-                    canvas,
-                    f"{int(round(tick))}",
-                    (12, y + 5),
-                    cv2.FONT_HERSHEY_SIMPLEX,
-                    0.44,
-                    (90, 90, 90),
-                    1,
-                    cv2.LINE_AA,
-                )
-
-            for frac in np.linspace(0.0, 1.0, num=5):
-                t_mark = duration * frac
-                x = map_x(float(t_mark))
-                cv2.line(canvas, (x, chart_y0), (x, chart_y1), (245, 245, 245), 1)
-                cv2.putText(
-                    canvas,
-                    f"{t_mark:.1f}s",
-                    (x - 16, total_h - 16),
-                    cv2.FONT_HERSHEY_SIMPLEX,
-                    0.42,
-                    (100, 100, 100),
-                    1,
-                    cv2.LINE_AA,
-                )
-
-            current_t = float(dense_t[idx])
-            current_v = float(dense_v[idx])
-            pts = [[map_x(float(dense_t[j])), map_y(float(dense_v[j]))] for j in range(idx + 1)]
-            if len(pts) >= 2:
-                cv2.polylines(canvas, [np.array(pts, dtype=np.int32)], False, (0, 140, 255), 3, cv2.LINE_AA)
-            elif len(pts) == 1:
-                cv2.circle(canvas, tuple(pts[0]), 3, (0, 140, 255), -1, cv2.LINE_AA)
-
-            for t, v in zip(pred_times, pred_vals):
-                if t <= current_t + 1e-6:
-                    cv2.circle(canvas, (map_x(t), map_y(v)), 5, (20, 20, 20), -1, cv2.LINE_AA)
-
-            cx, cy = map_x(current_t), map_y(current_v)
-            cv2.line(canvas, (cx, chart_y0), (cx, chart_y1), (120, 120, 120), 1)
-            cv2.circle(canvas, (cx, cy), 6, (0, 90, 220), -1, cv2.LINE_AA)
-            cv2.putText(
-                canvas,
-                f"time={current_t:.2f}s  pred={current_v:.1f}%",
-                (frame_w - 340, frame_h + 22),
-                cv2.FONT_HERSHEY_SIMPLEX,
-                0.55,
-                (30, 30, 30),
-                2,
-                cv2.LINE_AA,
-            )
-
-            writer.write(canvas)
-            idx += 1
-    finally:
-        cap.release()
-        writer.release()
-
-
-def main() -> int:
-    args = parse_args()
-    input_jsonl = args.input_jsonl.resolve()
-    if not input_jsonl.exists():
-        raise SystemExit(f"Missing JSONL: {input_jsonl}")
-    row = load_row(input_jsonl)
-    video_path = resolve_video_path(args.video_path, row)
-    output_path = args.output_video or input_jsonl.with_name(f"{input_jsonl.stem}_pred_progress.mp4")
-    output_path.parent.mkdir(parents=True, exist_ok=True)
-
-    render_preview(
-        row=row,
-        video_path=video_path,
-        output_path=output_path,
-        panel_height=args.panel_height,
-    )
-    print(f"input_jsonl={input_jsonl}")
-    print(f"video_path={video_path}")
-    print(f"output_video={output_path.resolve()}")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

quick_start/run_example.py

@@ -1,360 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import re
-import tempfile
-from pathlib import Path
-
-import numpy as np
-from PIL import Image
-
-THIS_DIR = Path(__file__).resolve().parent
-MODEL_ROOT = THIS_DIR.parent
-DEFAULT_SAMPLE_HZ = 2.0
-
-SIGNED_NUM = r"([+-]?[0-9]+(?:\.[0-9]+)?)"
-POINT_TIME_RE = re.compile(r"(?:Time|时间)[:：]?\s*([0-9]+(?:\.[0-9]+)?)\s*s?", re.IGNORECASE)
-POINT_PROGRESS_LINE_RE = re.compile(rf"(?im)^\s*(?:Progress|进度)[:：]?\s*{SIGNED_NUM}\s*%")
-INLINE_POINT_RE = re.compile(
-    rf"(?:Time|时间)[:：]?\s*([0-9]+(?:\.[0-9]+)?)\s*s?\s*[,，]?\s*(?:Progress|进度)[:：]?\s*{SIGNED_NUM}\s*%",
-    re.IGNORECASE,
-)
-
-
-def parse_args() -> argparse.Namespace:
-    parser = argparse.ArgumentParser(
-        description="Run VLAC progress inference on an arbitrary input video.",
-    )
-    parser.add_argument(
-        "--model-path",
-        type=Path,
-        default=MODEL_ROOT,
-        help="Path to the released VLAC model directory.",
-    )
-    parser.add_argument(
-        "--video-path",
-        type=Path,
-        required=True,
-        help="Path to the input video.",
-    )
-    parser.add_argument(
-        "--task-instruction",
-        type=str,
-        default=None,
-        help="Task instruction used to build the default chunk_all prompt.",
-    )
-    parser.add_argument(
-        "--task-plan",
-        type=str,
-        default=None,
-        help="Optional task plan text appended after the task instruction.",
-    )
-    parser.add_argument(
-        "--prompt",
-        type=str,
-        default=None,
-        help="Optional full prompt override. If set, task instruction and plan are ignored.",
-    )
-    parser.add_argument(
-        "--sample-hz",
-        type=float,
-        default=DEFAULT_SAMPLE_HZ,
-        help="Video sampling rate in Hz. Defaults to 2.0.",
-    )
-    parser.add_argument(
-        "--output-jsonl",
-        type=Path,
-        default=None,
-        help="Optional output path. If omitted, print the response only and do not save a JSONL file.",
-    )
-    parser.add_argument(
-        "--max-new-tokens",
-        type=int,
-        default=1024,
-        help="Generation cap for the response.",
-    )
-    return parser.parse_args()
-
-
-def build_chunk_all_prompt(task_instruction: str, task_plan: str | None) -> str:
-    task_instruction = task_instruction.strip()
-    task_plan = (task_plan or "").strip()
-    task_and_plan = task_instruction if not task_plan else f"{task_instruction}\n{task_plan}"
-    return (
-        f"任务描述和具体规划: {task_and_plan}\n\n"
-        "请根据任务描述和具体规划，找到并逐点生成视频中的关键动作点和相应的进度标注。"
-        "输出格式要求：每个关键点一行，格式为：\n"
-        "时间: X.Xs, 进度: Y%\n\n"
-        "请严格按照上述格式输出，不要输出额外说明。"
-    )
-
-
-def resolve_prompt(args: argparse.Namespace) -> tuple[str, str, str | None, str | None]:
-    prompt = args.prompt.strip() if args.prompt else None
-    task_plan = args.task_plan.strip() if args.task_plan else None
-    task_instruction = args.task_instruction.strip() if args.task_instruction else None
-
-    if prompt:
-        return prompt, "raw_prompt", task_instruction, task_plan
-
-    if not task_instruction:
-        raise SystemExit(
-            "Either provide --prompt, or provide --task-instruction with optional --task-plan."
-        )
-
-    return build_chunk_all_prompt(task_instruction, task_plan), "task_instruction_plus_plan", task_instruction, task_plan
-
-
-def resolve_video_path(video_path_arg: Path) -> Path:
-    video_path = video_path_arg.resolve()
-    if not video_path.exists():
-        raise SystemExit(f"Missing video: {video_path}")
-    return video_path
-
-
-def maybe_relativize_to_model_root(path: Path) -> str:
-    try:
-        return str(path.resolve().relative_to(MODEL_ROOT.resolve()))
-    except ValueError:
-        return str(path.resolve())
-
-
-def compute_sampled_indices(num_frames: int, fps: float, sample_hz: float) -> tuple[list[int], list[float]]:
-    if num_frames <= 0:
-        raise SystemExit(f"Invalid decoded frame count: {num_frames}")
-    if fps <= 0:
-        raise SystemExit(f"Invalid decoded fps: {fps}")
-    if sample_hz <= 0:
-        raise SystemExit(f"sample_hz must be positive, got {sample_hz}")
-
-    frame_ids = list(range(num_frames))
-    eligible_arr = np.array(frame_ids, dtype=float)
-    duration_sec = max(0.0, (num_frames - 1) / fps)
-    step_sec = 1.0 / sample_hz
-
-    target_times: list[float] = []
-    current = 0.0
-    eps = 1e-9
-    while current <= duration_sec + eps:
-        target_times.append(round(current, 6))
-        current += step_sec
-    if not target_times:
-        target_times = [0.0]
-
-    sampled_indices: list[int] = []
-    sampled_timestamps_sec: list[float] = []
-    seen = set()
-    for target_time in target_times:
-        target_frame = target_time * fps
-        pos = int(np.argmin(np.abs(eligible_arr - target_frame)))
-        idx = int(eligible_arr[pos])
-        if idx in seen:
-            continue
-        seen.add(idx)
-        sampled_indices.append(idx)
-        sampled_timestamps_sec.append(round(idx / fps, 6))
-    return sampled_indices, sampled_timestamps_sec
-
-
-def extract_sampled_frame_paths(
-    video_path: Path,
-    sample_hz: float,
-    *,
-    temp_dir: Path,
-) -> tuple[list[str], dict[str, float], list[int], list[float]]:
-    try:
-        from decord import VideoReader, cpu
-    except ImportError as exc:
-        raise SystemExit("Missing dependency: decord is required for video sampling in quick_start.") from exc
-
-    vr = VideoReader(str(video_path), ctx=cpu(0), num_threads=1)
-    actual_frame_count = len(vr)
-    actual_fps = float(vr.get_avg_fps())
-    sampled_indices, sampled_timestamps_sec = compute_sampled_indices(actual_frame_count, actual_fps, sample_hz)
-    if not sampled_indices:
-        raise SystemExit("No sampled frame indices were generated.")
-    if sampled_indices[-1] >= actual_frame_count:
-        raise SystemExit(
-            f"Decoded video is shorter than expected. Need frame index {sampled_indices[-1]}, "
-            f"got {actual_frame_count} frames."
-        )
-
-    batch = vr.get_batch(sampled_indices).asnumpy()
-    frame_paths: list[str] = []
-    for idx, frame in zip(sampled_indices, batch, strict=True):
-        frame_path = temp_dir / f"frame_{idx:06d}.jpg"
-        Image.fromarray(frame).save(frame_path, quality=95)
-        frame_paths.append(str(frame_path.resolve()))
-
-    video_stats = {
-        "decoded_frame_count": float(actual_frame_count),
-        "decoded_avg_fps": actual_fps,
-        "decoded_duration_sec": round(max(0.0, (actual_frame_count - 1) / actual_fps), 6),
-    }
-    return frame_paths, video_stats, sampled_indices, sampled_timestamps_sec
-
-
-def load_swift_runtime():
-    os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
-    os.environ.setdefault("IMAGE_MAX_TOKEN_NUM", "256")
-    os.environ.setdefault("VIDEO_MAX_TOKEN_NUM", "256")
-    os.environ.setdefault("VIDEO_MIN_TOKEN_NUM", "4")
-    os.environ.setdefault("QWEN_VL_UTILS_MAX_FRAME_LIST", "0")
-
-    try:
-        import torch  # noqa: F401
-    except ImportError as exc:
-        raise SystemExit("Missing dependency: torch is required for quick_start inference.") from exc
-
-    try:
-        from swift.llm import InferRequest, PtEngine, RequestConfig
-    except ImportError:
-        from swift.infer_engine import InferRequest, RequestConfig, TransformersEngine as PtEngine  # type: ignore
-    return InferRequest, PtEngine, RequestConfig
-
-
-def strip_code_fence(text: str) -> str:
-    cleaned = str(text or "").strip()
-    if cleaned.startswith("```") and cleaned.endswith("```"):
-        lines = cleaned.splitlines()
-        if len(lines) >= 3:
-            return "\n".join(lines[1:-1]).strip()
-    return cleaned
-
-
-def dedupe_sorted_points(times: list[float], values: list[float]) -> tuple[list[float], list[float]]:
-    if not times:
-        return [], []
-    pairs = sorted(zip(times, values), key=lambda item: (item[0], item[1]))
-    out_times: list[float] = []
-    out_values: list[float] = []
-    cur_time = pairs[0][0]
-    bucket: list[float] = []
-    for time_val, progress_val in pairs:
-        if abs(time_val - cur_time) > 1e-9:
-            out_times.append(float(cur_time))
-            out_values.append(float(sum(bucket) / len(bucket)))
-            cur_time = time_val
-            bucket = [float(progress_val)]
-        else:
-            bucket.append(float(progress_val))
-    out_times.append(float(cur_time))
-    out_values.append(float(sum(bucket) / len(bucket)))
-    return out_times, out_values
-
-
-def parse_point_blocks(text: str) -> tuple[list[float], list[float]]:
-    cleaned = strip_code_fence(text)
-    if not cleaned:
-        return [], []
-
-    inline_matches = INLINE_POINT_RE.findall(cleaned)
-    if inline_matches:
-        return dedupe_sorted_points(
-            [float(time_val) for time_val, _ in inline_matches],
-            [float(progress_val) for _, progress_val in inline_matches],
-        )
-
-    blocks = re.split(r"(?=(?:Time|时间)[:：]?\s*[0-9])", cleaned, flags=re.IGNORECASE)
-    times: list[float] = []
-    values: list[float] = []
-    for block in blocks:
-        block = block.strip()
-        if not block:
-            continue
-        time_match = POINT_TIME_RE.search(block)
-        progress_match = POINT_PROGRESS_LINE_RE.search(block)
-        if time_match and progress_match:
-            times.append(float(time_match.group(1)))
-            values.append(float(progress_match.group(1)))
-    return dedupe_sorted_points(times, values)
-
-
-def main() -> int:
-    args = parse_args()
-    model_path = args.model_path.resolve()
-    if not model_path.exists():
-        raise SystemExit(f"Missing model path: {model_path}")
-
-    prompt, prompt_source, task_instruction, task_plan = resolve_prompt(args)
-    video_path = resolve_video_path(args.video_path)
-    video_path_for_output = maybe_relativize_to_model_root(video_path)
-    output_path = args.output_jsonl.resolve() if args.output_jsonl is not None else None
-    if output_path is not None:
-        output_path.parent.mkdir(parents=True, exist_ok=True)
-
-    InferRequest, PtEngine, RequestConfig = load_swift_runtime()
-    engine = PtEngine(
-        str(model_path),
-        model_type="qwen3_moe_vl",
-        max_batch_size=1,
-        device_map="auto",
-    )
-
-    with tempfile.TemporaryDirectory(prefix="vlac_quick_start_frames_") as temp_dir_str:
-        temp_dir = Path(temp_dir_str)
-        frame_paths, video_stats, sampled_indices, sampled_timestamps_sec = extract_sampled_frame_paths(
-            video_path,
-            args.sample_hz,
-            temp_dir=temp_dir,
-        )
-        request = InferRequest(
-            messages=[
-                {
-                    "role": "user",
-                    "content": [
-                        {"type": "video", "video": frame_paths},
-                        {"type": "text", "text": prompt},
-                    ],
-                }
-            ]
-        )
-        response = engine.infer(
-            [request],
-            RequestConfig(max_tokens=args.max_new_tokens, temperature=0.0, top_k=1, top_p=1.0),
-        )[0].choices[0].message.content
-
-    pred_point_times_sec, pred_point_progress = parse_point_blocks(response)
-
-    output_row: dict[str, object] = {}
-    if task_instruction is not None:
-        output_row["task_instruction"] = task_instruction
-    if task_plan:
-        output_row["task_plan"] = task_plan
-
-    output_row.update(
-        {
-            "video_path": video_path_for_output,
-            "prompt_source": prompt_source,
-            "prompt_variant": "chunk_all" if prompt_source != "raw_prompt" else "custom",
-            "input_sample_hz": float(args.sample_hz),
-            "input_frame_indices": sampled_indices,
-            "input_timestamps_sec": sampled_timestamps_sec,
-            "input_frame_count": len(sampled_indices),
-            "decoded_video_stats": video_stats,
-            "prompt": prompt,
-            "response": response,
-            "pred_curve_parse_ok": bool(pred_point_progress),
-            "pred_curve_point_times_sec": pred_point_times_sec,
-            "pred_curve_point_progress": pred_point_progress,
-        }
-    )
-    if output_path is not None:
-        output_path.write_text(json.dumps(output_row, ensure_ascii=False) + "\n", encoding="utf-8")
-
-    if output_path is not None:
-        print(f"video_path={video_path}")
-        print(f"output_jsonl={output_path}")
-        print(f"input_sample_hz={float(args.sample_hz):.4f}")
-        print(f"input_frame_count={len(sampled_indices)}")
-        print(f"pred_keypoint_count={len(pred_point_progress)}")
-        print()
-    print(response)
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())