Instructions to use InternRobotics/VLAC-Cut with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use InternRobotics/VLAC-Cut with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="InternRobotics/VLAC-Cut")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("InternRobotics/VLAC-Cut")
model = AutoModelForMultimodalLM.from_pretrained("InternRobotics/VLAC-Cut", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use InternRobotics/VLAC-Cut with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "InternRobotics/VLAC-Cut"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternRobotics/VLAC-Cut",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/InternRobotics/VLAC-Cut

SGLang

How to use InternRobotics/VLAC-Cut with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "InternRobotics/VLAC-Cut" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternRobotics/VLAC-Cut",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "InternRobotics/VLAC-Cut" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "InternRobotics/VLAC-Cut",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use InternRobotics/VLAC-Cut with Docker Model Runner:
```
docker model run hf.co/InternRobotics/VLAC-Cut
```

futurefantasy commited on about 1 month ago

Commit

62a16e0

verified ·

1 Parent(s): 9807b39

Upload examples/reference_outputs/example_02.jsonl with huggingface_hub

Browse files

Files changed (1) hide show

examples/reference_outputs/example_02.jsonl +1 -0

examples/reference_outputs/example_02.jsonl ADDED Viewed

	@@ -0,0 +1 @@

+ {"example_id": "example_02", "bucket": "test_nonexpert_unseen", "global_episode_id": "ARX-data/human-data-1204-2cameras/20251128-183101-bad-onion-express-box/videos/chunk-000/observation.images.front/episode_000000", "task_instruction": "把洋葱放进快递箱里。", "task_description": "把洋葱放进快递箱里。\n开始移动:0%\n爪夹靠近洋葱:20%\n爪夹抓取洋葱:40%\n爪夹抓住洋葱接近快递箱:60%\n爪夹将洋葱放入快递箱:80%\n爪夹松开,洋葱落入快递箱:100%", "video_path": "examples/example_02/episode.mp4", "prompt_variant": "chunk_all", "input_sample_hz": 2.0, "input_frame_indices_2hz": [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225, 240, 255, 270, 285], "input_timestamps_sec_2hz": [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 9.5], "input_frame_count_2hz": 20, "decoded_video_stats": {"decoded_frame_count": 289.0, "decoded_avg_fps": 30.0}, "prompt": "任务描述和具体规划: 把洋葱放进快递箱里。\n开始移动:0%\n爪夹靠近洋葱:20%\n爪夹抓取洋葱:40%\n爪夹抓住洋葱接近快递箱:60%\n爪夹将洋葱放入快递箱:80%\n爪夹松开,洋葱落入快递箱:100%\n\n请根据任务描述和具体规划，找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求：每个关键点一行，格式为：\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出，不要输出额外说明。", "response": "时间: 0.0s, 进度: 0%\n时间: 2.3s, 进度: 5%\n时间: 3.3s, 进度: 8%\n时间: 4.3s, 进度: 10%\n时间: 5.1s, 进度: -5%\n时间: 6.1s, 进度: -8%\n时间: 7.1s, 进度: -2%\n时间: 8.3s, 进度: 2%\n时间: 9.5s, 进度: 4%", "benchmark_progress_type": "dense_kinematic_progress", "benchmark_progress_source": {"bucket": "test_nonexpert_unseen", "global_episode_id": "ARX-data/human-data-1204-2cameras/20251128-183101-bad-onion-express-box/videos/chunk-000/observation.images.front/episode_000000"}, "benchmark_semantic_anchors": [{"frame": 0, "progress": 0.0}, {"frame": 70, "progress": 10.0}, {"frame": 130, "progress": 18.0}, {"frame": 162, "progress": 5.0}, {"frame": 240, "progress": -5.0}, {"frame": 272, "progress": -10.0}], "gt_dense_progress": [0.0, 0.1, 0.3, 0.4, 0.6, 0.7, 0.9, 1.0, 1.1, 1.3, 1.4, 1.6, 1.7, 1.9, 2.0, 2.1, 2.3, 2.4, 2.6, 2.7, 2.9, 3.0, 3.1, 3.3, 3.4, 3.6, 3.7, 3.9, 4.0, 4.1, 4.3, 4.4, 4.6, 4.7, 4.9, 5.0, 5.1, 5.3, 5.4, 5.6, 5.7, 5.9, 6.0, 6.1, 6.3, 6.4, 6.6, 6.7, 6.9, 7.0, 7.1, 7.3, 7.4, 7.6, 7.7, 7.9, 8.0, 8.1, 8.3, 8.4, 8.6, 8.7, 8.9, 9.0, 9.1, 9.3, 9.4, 9.6, 9.7, 9.9, 10.0, 10.1, 10.3, 10.4, 10.5, 10.7, 10.8, 10.9, 11.1, 11.2, 11.3, 11.5, 11.6, 11.7, 11.9, 12.0, 12.1, 12.3, 12.4, 12.5, 12.7, 12.8, 12.9, 13.1, 13.2, 13.3, 13.5, 13.6, 13.7, 13.9, 14.0, 14.1, 14.3, 14.4, 14.5, 14.7, 14.8, 14.9, 15.1, 15.2, 15.3, 15.5, 15.6, 15.7, 15.9, 16.0, 16.1, 16.3, 16.4, 16.5, 16.7, 16.8, 16.9, 17.1, 17.2, 17.3, 17.5, 17.6, 17.7, 17.9, 18.0, 17.6, 17.2, 16.8, 16.4, 16.0, 15.6, 15.2, 14.8, 14.3, 13.9, 13.5, 13.1, 12.7, 12.3, 11.9, 11.5, 11.1, 10.7, 10.3, 9.9, 9.5, 9.1, 8.7, 8.2, 7.8, 7.4, 7.0, 6.6, 6.2, 5.8, 5.4, 5.0, 4.9, 4.7, 4.6, 4.5, 4.4, 4.2, 4.1, 4.0, 3.8, 3.7, 3.6, 3.5, 3.3, 3.2, 3.1, 2.9, 2.8, 2.7, 2.6, 2.4, 2.3, 2.2, 2.1, 1.9, 1.8, 1.7, 1.5, 1.4, 1.3, 1.2, 1.0, 0.9, 0.8, 0.6, 0.5, 0.4, 0.3, 0.1, 0.0, -0.1, -0.3, -0.4, -0.5, -0.6, -0.8, -0.9, -1.0, -1.2, -1.3, -1.4, -1.5, -1.7, -1.8, -1.9, -2.1, -2.2, -2.3, -2.4, -2.6, -2.7, -2.8, -2.9, -3.1, -3.2, -3.3, -3.5, -3.6, -3.7, -3.8, -4.0, -4.1, -4.2, -4.4, -4.5, -4.6, -4.7, -4.9, -5.0, -5.2, -5.3, -5.5, -5.6, -5.8, -5.9, -6.1, -6.2, -6.4, -6.6, -6.7, -6.9, -7.0, -7.2, -7.3, -7.5, -7.7, -7.8, -8.0, -8.1, -8.3, -8.4, -8.6, -8.8, -8.9, -9.1, -9.2, -9.4, -9.5, -9.7, -9.8, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0, -10.0], "gt_dense_timestamps_sec": [0.0, 0.033333, 0.066667, 0.1, 0.133333, 0.166667, 0.2, 0.233333, 0.266667, 0.3, 0.333333, 0.366667, 0.4, 0.433333, 0.466667, 0.5, 0.533333, 0.566667, 0.6, 0.633333, 0.666667, 0.7, 0.733333, 0.766667, 0.8, 0.833333, 0.866667, 0.9, 0.933333, 0.966667, 1.0, 1.033333, 1.066667, 1.1, 1.133333, 1.166667, 1.2, 1.233333, 1.266667, 1.3, 1.333333, 1.366667, 1.4, 1.433333, 1.466667, 1.5, 1.533333, 1.566667, 1.6, 1.633333, 1.666667, 1.7, 1.733333, 1.766667, 1.8, 1.833333, 1.866667, 1.9, 1.933333, 1.966667, 2.0, 2.033333, 2.066667, 2.1, 2.133333, 2.166667, 2.2, 2.233333, 2.266667, 2.3, 2.333333, 2.366667, 2.4, 2.433333, 2.466667, 2.5, 2.533333, 2.566667, 2.6, 2.633333, 2.666667, 2.7, 2.733333, 2.766667, 2.8, 2.833333, 2.866667, 2.9, 2.933333, 2.966667, 3.0, 3.033333, 3.066667, 3.1, 3.133333, 3.166667, 3.2, 3.233333, 3.266667, 3.3, 3.333333, 3.366667, 3.4, 3.433333, 3.466667, 3.5, 3.533333, 3.566667, 3.6, 3.633333, 3.666667, 3.7, 3.733333, 3.766667, 3.8, 3.833333, 3.866667, 3.9, 3.933333, 3.966667, 4.0, 4.033333, 4.066667, 4.1, 4.133333, 4.166667, 4.2, 4.233333, 4.266667, 4.3, 4.333333, 4.366667, 4.4, 4.433333, 4.466667, 4.5, 4.533333, 4.566667, 4.6, 4.633333, 4.666667, 4.7, 4.733333, 4.766667, 4.8, 4.833333, 4.866667, 4.9, 4.933333, 4.966667, 5.0, 5.033333, 5.066667, 5.1, 5.133333, 5.166667, 5.2, 5.233333, 5.266667, 5.3, 5.333333, 5.366667, 5.4, 5.433333, 5.466667, 5.5, 5.533333, 5.566667, 5.6, 5.633333, 5.666667, 5.7, 5.733333, 5.766667, 5.8, 5.833333, 5.866667, 5.9, 5.933333, 5.966667, 6.0, 6.033333, 6.066667, 6.1, 6.133333, 6.166667, 6.2, 6.233333, 6.266667, 6.3, 6.333333, 6.366667, 6.4, 6.433333, 6.466667, 6.5, 6.533333, 6.566667, 6.6, 6.633333, 6.666667, 6.7, 6.733333, 6.766667, 6.8, 6.833333, 6.866667, 6.9, 6.933333, 6.966667, 7.0, 7.033333, 7.066667, 7.1, 7.133333, 7.166667, 7.2, 7.233333, 7.266667, 7.3, 7.333333, 7.366667, 7.4, 7.433333, 7.466667, 7.5, 7.533333, 7.566667, 7.6, 7.633333, 7.666667, 7.7, 7.733333, 7.766667, 7.8, 7.833333, 7.866667, 7.9, 7.933333, 7.966667, 8.0, 8.033333, 8.066667, 8.1, 8.133333, 8.166667, 8.2, 8.233333, 8.266667, 8.3, 8.333333, 8.366667, 8.4, 8.433333, 8.466667, 8.5, 8.533333, 8.566667, 8.6, 8.633333, 8.666667, 8.7, 8.733333, 8.766667, 8.8, 8.833333, 8.866667, 8.9, 8.933333, 8.966667, 9.0, 9.033333, 9.066667, 9.1, 9.133333, 9.166667, 9.2, 9.233333, 9.266667, 9.3, 9.333333, 9.366667, 9.4, 9.433333, 9.466667, 9.5, 9.533333, 9.566667, 9.6], "pred_curve_parse_ok": true, "pred_curve_point_times_sec": [0.0, 2.3, 3.3, 4.3, 5.1, 6.1, 7.1, 8.3, 9.5], "pred_curve_point_progress": [0.0, 5.0, 8.0, 10.0, -5.0, -8.0, -2.0, 2.0, 4.0], "pred_dense_progress_aligned_to_gt": [0.0, 0.07246304347826088, 0.14492826086956526, 0.21739130434782614, 0.289854347826087, 0.3623195652173914, 0.4347826086956523, 0.5072456521739132, 0.5797108695652174, 0.6521739130434784, 0.7246369565217392, 0.7971021739130436, 0.8695652173913045, 0.9420282608695654, 1.0144934782608697, 1.0869565217391306, 1.1594195652173913, 1.2318847826086958, 1.3043478260869568, 1.3768108695652177, 1.449276086956522, 1.5217391304347827, 1.5942021739130436, 1.6666673913043482, 1.739130434782609, 1.8115934782608698, 1.884058695652174, 1.956521739130435, 2.028984782608696, 2.1014500000000003, 2.173913043478261, 2.246376086956522, 2.3188413043478264, 2.3913043478260874, 2.463767391304348, 2.536232608695652, 2.6086956521739135, 2.6811586956521745, 2.7536239130434788, 2.8260869565217397, 2.8985500000000006, 2.971015217391305, 3.0434782608695654, 3.1159413043478263, 3.1884065217391306, 3.2608695652173916, 3.333332608695653, 3.4057978260869572, 3.478260869565218, 3.5507239130434787, 3.623189130434783, 3.695652173913044, 3.768115217391305, 3.840580434782609, 3.91304347826087, 3.9855065217391314, 4.057971739130435, 4.130434782608696, 4.202897826086957, 4.2753630434782615, 4.347826086956522, 4.420289130434783, 4.492754347826088, 4.565217391304349, 4.6376804347826095, 4.710145652173914, 4.782608695652175, 4.855071739130436, 4.92753695652174, 5.0, 5.099999, 5.200001, 5.300000000000001, 5.399999000000001, 5.500001000000001, 5.6000000000000005, 5.699999, 5.800001, 5.9, 5.999999000000001, 6.100001000000001, 6.200000000000001, 6.299999000000001, 6.4000010000000005, 6.5, 6.599999, 6.700001, 6.800000000000001, 6.899999000000001, 7.000001000000001, 7.1000000000000005, 7.199999, 7.300001, 7.4, 7.499999000000001, 7.600001000000001, 7.700000000000001, 7.799999000000001, 7.9000010000000005, 8.0, 8.066666000000001, 8.133334000000001, 8.2, 8.266666, 8.333334, 8.4, 8.466666, 8.533334, 8.600000000000001, 8.666666, 8.733334, 8.8, 8.866666, 8.933334, 9.0, 9.066666000000001, 9.133334000000001, 9.2, 9.266666, 9.333334, 9.4, 9.466666, 9.533334, 9.6, 9.666666000000001, 9.733334000000001, 9.8, 9.866666, 9.933334, 10.0, 9.375006250000004, 8.749993750000003, 8.12499999999999, 7.500006249999993, 6.874993749999993, 6.2499999999999964, 5.625006249999999, 4.999993749999999, 4.375000000000003, 3.7500062499999887, 3.124993749999989, 2.499999999999992, 1.8750062499999949, 1.2499937499999945, 0.6249999999999982, 6.250000001983835e-06, -0.6250062499999984, -1.2500000000000124, -1.8749937500000087, -2.500006250000009, -3.1250000000000053, -3.7499937500000033, -4.375006250000002, -5.0, -5.099999000000002, -5.200001000000002, -5.300000000000002, -5.399999000000001, -5.500001000000001, -5.6000000000000005, -5.699999, -5.800001, -5.900000000000002, -5.999999000000002, -6.100001000000002, -6.200000000000001, -6.299999000000001, -6.4000010000000005, -6.5, -6.599999000000002, -6.700001000000002, -6.800000000000002, -6.899999000000001, -7.000001000000001, -7.1000000000000005, -7.199999, -7.300001, -7.400000000000002, -7.499999000000002, -7.600001000000002, -7.700000000000001, -7.799999000000001, -7.9000010000000005, -8.0, -7.800001999999996, -7.599997999999996, -7.399999999999997, -7.200001999999998, -6.999997999999998, -6.799999999999999, -6.600002, -6.399998, -6.199999999999996, -6.000001999999997, -5.799997999999997, -5.599999999999998, -5.400001999999999, -5.199997999999999, -5.0, -4.800001999999996, -4.599997999999996, -4.399999999999997, -4.200001999999998, -3.999997999999998, -3.799999999999999, -3.600002, -3.399998, -3.1999999999999957, -3.0000019999999967, -2.799997999999997, -2.599999999999998, -2.400001999999999, -2.199997999999999, -2.0, -1.8888899999999977, -1.7777766666666646, -1.6666666666666652, -1.5555566666666658, -1.4444433333333326, -1.3333333333333335, -1.2222233333333339, -1.111110000000001, -0.9999999999999984, -0.8888899999999991, -0.7777766666666661, -0.6666666666666667, -0.5555566666666674, -0.4444433333333342, -0.3333333333333348, -0.22222333333333255, -0.11110999999999938, 0.0, 0.11110999999999915, 0.22222333333333255, 0.3333333333333317, 0.4444433333333313, 0.5555566666666643, 0.6666666666666665, 0.7777766666666661, 0.8888899999999991, 0.9999999999999987, 1.111110000000001, 1.2222233333333339, 1.3333333333333304, 1.4444433333333326, 1.555556666666666, 1.666666666666662, 1.7777766666666643, 1.8888899999999977, 2.0, 2.0555549999999982, 2.1111116666666647, 2.166666666666666, 2.2222216666666643, 2.2777783333333312, 2.333333333333332, 2.3888883333333335, 2.4444450000000004, 2.4999999999999987, 2.555555, 2.6111116666666665, 2.6666666666666647, 2.722221666666666, 2.7777783333333326, 2.833333333333334, 2.888888333333332, 2.9444449999999986, 3.0, 3.0555549999999982, 3.111111666666665, 3.166666666666666, 3.2222216666666674, 3.277778333333334, 3.333333333333332, 3.3888883333333335, 3.444445, 3.4999999999999982, 3.555555, 3.6111116666666665, 3.666666666666668, 3.722221666666666, 3.7777783333333326, 3.833333333333334, 3.888888333333332, 3.9444449999999986, 4.0, 4.0, 4.0, 4.0], "curve_compare_metrics": {"point_count": 289, "mae": 6.6085736332179925, "rmse": 7.729949344246744, "pearson": 0.43327740758574307, "spearman": 0.5164980259299339}, "curve_compare_plot": "examples/reference_outputs/example_02_gt_vs_pred.png"}