Instructions to use InternRobotics/VLAC-Cut with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use InternRobotics/VLAC-Cut with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="InternRobotics/VLAC-Cut")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("InternRobotics/VLAC-Cut")
model = AutoModelForMultimodalLM.from_pretrained("InternRobotics/VLAC-Cut", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use InternRobotics/VLAC-Cut with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "InternRobotics/VLAC-Cut"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternRobotics/VLAC-Cut",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/InternRobotics/VLAC-Cut

SGLang

How to use InternRobotics/VLAC-Cut with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "InternRobotics/VLAC-Cut" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternRobotics/VLAC-Cut",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "InternRobotics/VLAC-Cut" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternRobotics/VLAC-Cut",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use InternRobotics/VLAC-Cut with Docker Model Runner:
```
docker model run hf.co/InternRobotics/VLAC-Cut
```

futurefantasy commited on Jun 23

Commit

6f70d65

verified ·

1 Parent(s): 82c454e

Simplify quick start README and add explicit example commands

Browse files

Files changed (1) hide show

quick_start/README.md +41 -43

quick_start/README.md CHANGED Viewed

@@ -1,7 +1,9 @@
 # Quick Start
-This quick start runs VLAC progress inference on an arbitrary input video.
-It samples the input video at `2 Hz` by default, sends the sampled frames and prompt to the model, writes one JSONL row with the raw response and parsed predicted progress points, and can optionally render a predicted-progress preview video with a separate helper script.
 ## Requirements
@@ -15,9 +17,9 @@ It samples the input video at `2 Hz` by default, sends the sampled frames and pr
 - `opencv-python` for preview video rendering
 - at least `1` GPU with enough memory for this `30B` checkpoint
-## Generic Inference
-Run on an arbitrary input video with a task instruction:
 ```bash
 python quick_start/run_example.py \
@@ -25,60 +27,56 @@ python quick_start/run_example.py \
   --task-instruction "把洋葱放进快递箱里。"
 ```
-Bundled videos under `examples/` are used the same way:
 ```bash
-python quick_start/run_example.py \
-  --video-path examples/example_01/episode.mp4 \
-  --prompt $'任务描述和具体规划: 抓取紫色方块使其从方形洞口落入积木桶中\n爪夹开始移动：0%\n爪夹接近紫色方块：20%\n爪夹抓紧紫色方块：40%\n爪夹抓紧紫色方块接近方形洞口：60%\n爪夹将紫色方块对准方形洞口：80%\n紫色方块从方形洞口落入积木桶中，爪夹移开：100%\n\n请根据任务描述和具体规划，找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求：每个关键点一行，格式为：\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出，不要输出额外说明。'
 ```
-If you have a more detailed task plan, pass it directly as text:
-```bash
-python quick_start/run_example.py \
-  --video-path /path/to/video.mp4 \
-  --task-instruction "将三角烧杯放在三脚架上。" \
-  --task-plan $'爪夹准备移动：0%\n爪夹开始移动：10%\n爪夹靠近三角烧杯：20%\n爪夹抓取三角烧杯：40%\n爪夹合拢并靠近三脚架：60%\n爪夹移动到三脚架的正上方：80%\n爪夹松开，三角烧杯落在三脚架上：100%'
-```
-If you want full control over the prompt, pass it directly as text:
 ```bash
 python quick_start/run_example.py \
-  --video-path /path/to/video.mp4 \
-  --prompt $'任务描述和具体规划: 把洋葱放进快递箱里。\n开始移动:0%\n爪夹靠近洋葱:20%\n爪夹抓取洋葱:40%\n爪夹抓住洋葱接近快递箱:60%\n爪夹将洋葱放入快递箱:80%\n爪夹松开,洋葱落入快递箱:100%\n\n请根据任务描述和具体规划，找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求：每个关键点一行，格式为：\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出，不要输出额外说明。'
 ```
-If the model directory is not the parent of `quick_start/`, pass it explicitly:
 ```bash
 python quick_start/run_example.py \
-  --model-path /path/to/VLAC2-Qwen3VL-30B-A3B-Progress \
-  --video-path /path/to/video.mp4 \
-  --task-instruction "把洋葱放进快递箱里。"
 ```
-## Output
-By default the inference script writes:
-- `quick_start/outputs/<video_stem>.jsonl`
-Each output file contains one JSONL row with:
-- input video path
-- optional task instruction and task plan text
-- sampled frame indices and timestamps
-- decoded video statistics
-- prompt text
-- raw model response
-- parsed predicted key points: `时间 / 进度`
-The preview video helper writes:
-- `<jsonl_stem>_pred_progress.mp4`
-The preview video shows the source video on top and predicted progress-so-far on the bottom.
-No ground truth, alignment, metrics, or comparison plots are produced by the inference script or preview renderer.

 # Quick Start
+This quick start has two steps:
+1. Run progress inference on a video and write one JSONL output.
+2. Render a prediction-only preview video from that JSONL output.
 ## Requirements
 - `opencv-python` for preview video rendering
 - at least `1` GPU with enough memory for this `30B` checkpoint
+## Usage
+Run inference:
 ```bash
 python quick_start/run_example.py \
   --task-instruction "把洋葱放进快递箱里。"
 ```
+Render a preview video:
 ```bash
+python quick_start/render_prediction_video.py \
+  --input-jsonl quick_start/outputs/video.jsonl \
+  --output-video quick_start/outputs/video_pred_progress.mp4
 ```
+If the model directory is not the parent of `quick_start/`, pass `--model-path`.
+## Bundled Examples
+`example_01`
 ```bash
 python quick_start/run_example.py \
+  --video-path examples/example_01/episode.mp4 \
+  --task-instruction "抓取紫色方块使其从方形洞口落入积木桶中" \
+  --task-plan $'爪夹开始移动：0%\n爪夹接近紫色方块：20%\n爪夹抓紧紫色方块：40%\n爪夹抓紧紫色方块接近方形洞口：60%\n爪夹将紫色方块对准方形洞口：80%\n紫色方块从方形洞口落入积木桶中，爪夹移开：100%' \
+  --output-jsonl quick_start/outputs/example_01.jsonl
+python quick_start/render_prediction_video.py \
+  --input-jsonl quick_start/outputs/example_01.jsonl \
+  --output-video quick_start/outputs/example_01_pred_progress.mp4
 ```
+`example_02`
 ```bash
 python quick_start/run_example.py \
+  --video-path examples/example_02/episode.mp4 \
+  --task-instruction "把洋葱放进快递箱里。" \
+  --task-plan $'开始移动:0%\n爪夹靠近洋葱:20%\n爪夹抓取洋葱:40%\n爪夹抓住洋葱接近快递箱:60%\n爪夹将洋葱放入快递箱:80%\n爪夹松开,洋葱落入快递箱:100%' \
+  --output-jsonl quick_start/outputs/example_02.jsonl
+python quick_start/render_prediction_video.py \
+  --input-jsonl quick_start/outputs/example_02.jsonl \
+  --output-video quick_start/outputs/example_02_pred_progress.mp4
 ```
+`example_03`
+```bash
+python quick_start/run_example.py \
+  --video-path examples/example_03/episode.mp4 \
+  --task-instruction "将三角烧杯放在三脚架上。" \
+  --task-plan $'爪夹准备移动：0%\n爪夹开始移动：10%\n爪夹靠近三角烧杯：20%\n爪夹抓取三角烧杯：40%\n爪夹合拢并靠近三脚架：60%\n爪夹移动到三脚架的正上方：80%\n爪夹松开，三角烧杯落在三脚架上：100%' \
+  --output-jsonl quick_start/outputs/example_03.jsonl
+python quick_start/render_prediction_video.py \
+  --input-jsonl quick_start/outputs/example_03.jsonl \
+  --output-video quick_start/outputs/example_03_pred_progress.mp4
+```