Image-Text-to-Text
Transformers
Safetensors
qwen3_vl_moe
robotics
embodied-ai
video-understanding
progress-estimation
reward-modeling
qwen3-vl
conversational
Instructions to use InternRobotics/VLAC-Cut with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use InternRobotics/VLAC-Cut with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="InternRobotics/VLAC-Cut") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("InternRobotics/VLAC-Cut") model = AutoModelForMultimodalLM.from_pretrained("InternRobotics/VLAC-Cut", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use InternRobotics/VLAC-Cut with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "InternRobotics/VLAC-Cut" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "InternRobotics/VLAC-Cut", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/InternRobotics/VLAC-Cut
- SGLang
How to use InternRobotics/VLAC-Cut with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "InternRobotics/VLAC-Cut" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "InternRobotics/VLAC-Cut", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "InternRobotics/VLAC-Cut" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "InternRobotics/VLAC-Cut", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use InternRobotics/VLAC-Cut with Docker Model Runner:
docker model run hf.co/InternRobotics/VLAC-Cut
Upload examples/reference_outputs/example_01.jsonl with huggingface_hub
Browse files
examples/reference_outputs/example_01.jsonl
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"example_id": "example_01", "bucket": "test_nonexpert_seen", "global_episode_id": "ARX-data/human-data-1204-2cameras/20251130-162433-bad-purple cube-shaped building block/videos/chunk-000/observation.images.front/episode_000007", "task_instruction": "抓取紫色方块使其从方形洞口落入积木桶中", "task_description": "抓取紫色方块使其从方形洞口落入积木桶中:\n爪夹开始移动:0%\n爪夹接近紫色方块:20%\n爪夹抓紧紫色方块:40%\n爪夹抓紧紫色方块接近方形洞口:60%\n爪夹将紫色方块对准方形洞口:80%\n紫色方块从方形洞口落入积木桶中,爪夹移开:100%", "video_path": "examples/example_01/episode.mp4", "prompt_variant": "chunk_all", "input_sample_hz": 2.0, "input_frame_indices_2hz": [0, 15, 30, 45, 60, 75, 90, 105, 120, 135, 150, 165, 180, 195, 210, 225, 240, 255], "input_timestamps_sec_2hz": [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5], "input_frame_count_2hz": 18, "decoded_video_stats": {"decoded_frame_count": 258.0, "decoded_avg_fps": 30.0}, "prompt": "任务描述和具体规划: 抓取紫色方块使其从方形洞口落入积木桶中\n爪夹开始移动:0%\n爪夹接近紫色方块:20%\n爪夹抓紧紫色方块:40%\n爪夹抓紧紫色方块接近方形洞口:60%\n爪夹将紫色方块对准方形洞口:80%\n紫色方块从方形洞口落入积木桶中,爪夹移开:100%\n\n请根据任务描述和具体规划,找到并逐点生成视频中的关键动作点和相应的进度标注。输出格式要求:每个关键点一行,格式为:\n时间: X.Xs, 进度: Y%\n\n请严格按照上述格式输出,不要输出额外说明。", "response": "时间: 1.3s, 进度: 0%\n时间: 2.7s, 进度: 10%\n时间: 3.7s, 进度: 15%\n时间: 5.5s, 进度: 10%\n时间: 6.3s, 进度: -15%\n时间: 7.1s, 进度: -25%\n时间: 8.0s, 进度: -15%", "benchmark_progress_type": "dense_kinematic_progress", "benchmark_progress_source": {"bucket": "test_nonexpert_seen", "global_episode_id": "ARX-data/human-data-1204-2cameras/20251130-162433-bad-purple cube-shaped building block/videos/chunk-000/observation.images.front/episode_000007"}, "benchmark_semantic_anchors": [{"frame": 49, "progress": 0.0}, {"frame": 74, "progress": 5.0}, {"frame": 87, "progress": 10.0}, {"frame": 134, "progress": 15.0}, {"frame": 173, "progress": 13.0}, {"frame": 186, "progress": -15.0}, {"frame": 220, "progress": -30.0}, {"frame": 239, "progress": -20.0}, {"frame": 257, "progress": -10.0}], "gt_dense_progress": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0, 2.2, 2.4, 2.6, 2.8, 3.0, 3.2, 3.4, 3.6, 3.8, 4.0, 4.2, 4.4, 4.6, 4.8, 5.0, 5.4, 5.8, 6.2, 6.5, 6.9, 7.3, 7.7, 8.1, 8.5, 8.8, 9.2, 9.6, 10.0, 10.1, 10.2, 10.3, 10.4, 10.5, 10.6, 10.7, 10.9, 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8, 11.9, 12.0, 12.1, 12.2, 12.3, 12.4, 12.6, 12.7, 12.8, 12.9, 13.0, 13.1, 13.2, 13.3, 13.4, 13.5, 13.6, 13.7, 13.8, 13.9, 14.0, 14.1, 14.3, 14.4, 14.5, 14.6, 14.7, 14.8, 14.9, 15.0, 14.9, 14.9, 14.8, 14.8, 14.7, 14.7, 14.6, 14.6, 14.5, 14.5, 14.4, 14.4, 14.3, 14.3, 14.2, 14.2, 14.1, 14.1, 14.0, 14.0, 13.9, 13.9, 13.8, 13.8, 13.7, 13.7, 13.6, 13.6, 13.5, 13.5, 13.4, 13.4, 13.3, 13.3, 13.2, 13.2, 13.1, 13.1, 13.0, 10.8, 8.7, 6.5, 4.4, 2.2, 0.1, -2.1, -4.2, -6.4, -8.5, -10.7, -12.8, -15.0, -15.4, -15.9, -16.3, -16.8, -17.2, -17.6, -18.1, -18.5, -19.0, -19.4, -19.9, -20.3, -20.7, -21.2, -21.6, -22.1, -22.5, -22.9, -23.4, -23.8, -24.3, -24.7, -25.1, -25.6, -26.0, -26.5, -26.9, -27.4, -27.8, -28.2, -28.7, -29.1, -29.6, -30.0, -29.5, -28.9, -28.4, -27.9, -27.4, -26.8, -26.3, -25.8, -25.3, -24.7, -24.2, -23.7, -23.2, -22.6, -22.1, -21.6, -21.1, -20.5, -20.0, -19.4, -18.9, -18.3, -17.8, -17.2, -16.7, -16.1, -15.6, -15.0, -14.4, -13.9, -13.3, -12.8, -12.2, -11.7, -11.1, -10.6, -10.0], "gt_dense_timestamps_sec": [0.0, 0.033333, 0.066667, 0.1, 0.133333, 0.166667, 0.2, 0.233333, 0.266667, 0.3, 0.333333, 0.366667, 0.4, 0.433333, 0.466667, 0.5, 0.533333, 0.566667, 0.6, 0.633333, 0.666667, 0.7, 0.733333, 0.766667, 0.8, 0.833333, 0.866667, 0.9, 0.933333, 0.966667, 1.0, 1.033333, 1.066667, 1.1, 1.133333, 1.166667, 1.2, 1.233333, 1.266667, 1.3, 1.333333, 1.366667, 1.4, 1.433333, 1.466667, 1.5, 1.533333, 1.566667, 1.6, 1.633333, 1.666667, 1.7, 1.733333, 1.766667, 1.8, 1.833333, 1.866667, 1.9, 1.933333, 1.966667, 2.0, 2.033333, 2.066667, 2.1, 2.133333, 2.166667, 2.2, 2.233333, 2.266667, 2.3, 2.333333, 2.366667, 2.4, 2.433333, 2.466667, 2.5, 2.533333, 2.566667, 2.6, 2.633333, 2.666667, 2.7, 2.733333, 2.766667, 2.8, 2.833333, 2.866667, 2.9, 2.933333, 2.966667, 3.0, 3.033333, 3.066667, 3.1, 3.133333, 3.166667, 3.2, 3.233333, 3.266667, 3.3, 3.333333, 3.366667, 3.4, 3.433333, 3.466667, 3.5, 3.533333, 3.566667, 3.6, 3.633333, 3.666667, 3.7, 3.733333, 3.766667, 3.8, 3.833333, 3.866667, 3.9, 3.933333, 3.966667, 4.0, 4.033333, 4.066667, 4.1, 4.133333, 4.166667, 4.2, 4.233333, 4.266667, 4.3, 4.333333, 4.366667, 4.4, 4.433333, 4.466667, 4.5, 4.533333, 4.566667, 4.6, 4.633333, 4.666667, 4.7, 4.733333, 4.766667, 4.8, 4.833333, 4.866667, 4.9, 4.933333, 4.966667, 5.0, 5.033333, 5.066667, 5.1, 5.133333, 5.166667, 5.2, 5.233333, 5.266667, 5.3, 5.333333, 5.366667, 5.4, 5.433333, 5.466667, 5.5, 5.533333, 5.566667, 5.6, 5.633333, 5.666667, 5.7, 5.733333, 5.766667, 5.8, 5.833333, 5.866667, 5.9, 5.933333, 5.966667, 6.0, 6.033333, 6.066667, 6.1, 6.133333, 6.166667, 6.2, 6.233333, 6.266667, 6.3, 6.333333, 6.366667, 6.4, 6.433333, 6.466667, 6.5, 6.533333, 6.566667, 6.6, 6.633333, 6.666667, 6.7, 6.733333, 6.766667, 6.8, 6.833333, 6.866667, 6.9, 6.933333, 6.966667, 7.0, 7.033333, 7.066667, 7.1, 7.133333, 7.166667, 7.2, 7.233333, 7.266667, 7.3, 7.333333, 7.366667, 7.4, 7.433333, 7.466667, 7.5, 7.533333, 7.566667, 7.6, 7.633333, 7.666667, 7.7, 7.733333, 7.766667, 7.8, 7.833333, 7.866667, 7.9, 7.933333, 7.966667, 8.0, 8.033333, 8.066667, 8.1, 8.133333, 8.166667, 8.2, 8.233333, 8.266667, 8.3, 8.333333, 8.366667, 8.4, 8.433333, 8.466667, 8.5, 8.533333, 8.566667], "pred_curve_parse_ok": true, "pred_curve_point_times_sec": [1.3, 2.7, 3.7, 5.5, 6.3, 7.1, 8.0], "pred_curve_point_progress": [0.0, 10.0, 15.0, 10.0, -15.0, -25.0, -15.0], "pred_dense_progress_aligned_to_gt": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.23809285714285752, 0.47619285714285736, 0.7142857142857133, 0.9523785714285709, 1.1904785714285706, 1.4285714285714282, 1.6666642857142857, 1.9047642857142855, 2.1428571428571432, 2.380949999999999, 2.6190499999999988, 2.8571428571428563, 3.095235714285714, 3.3333357142857136, 3.571428571428571, 3.8095214285714287, 4.0476214285714285, 4.285714285714285, 4.523807142857142, 4.761907142857142, 4.999999999999999, 5.238092857142855, 5.476192857142855, 5.714285714285714, 5.95237857142857, 6.19047857142857, 6.428571428571429, 6.666664285714285, 6.904764285714285, 7.142857142857141, 7.3809499999999995, 7.61905, 7.857142857142856, 8.095235714285714, 8.333335714285715, 8.571428571428571, 8.809521428571426, 9.047621428571427, 9.285714285714285, 9.523807142857141, 9.761907142857142, 10.0, 10.166664999999998, 10.333334999999998, 10.499999999999998, 10.666665, 10.833335, 10.999999999999998, 11.166665, 11.333335, 11.5, 11.666664999999998, 11.833334999999998, 12.0, 12.166664999999998, 12.333334999999998, 12.5, 12.666664999999998, 12.833334999999998, 12.999999999999998, 13.166665, 13.333335, 13.499999999999998, 13.666665, 13.833335, 14.0, 14.166664999999998, 14.333334999999998, 14.5, 14.666664999999998, 14.833334999999998, 15.0, 14.907408333333334, 14.814813888888889, 14.722222222222223, 14.629630555555556, 14.537036111111112, 14.444444444444445, 14.351852777777777, 14.259258333333333, 14.166666666666668, 14.074075, 13.981480555555557, 13.88888888888889, 13.796297222222222, 13.703702777777778, 13.61111111111111, 13.518519444444445, 13.425925, 13.333333333333334, 13.240741666666668, 13.148147222222224, 13.055555555555555, 12.962963888888888, 12.870369444444444, 12.777777777777779, 12.685186111111111, 12.592591666666667, 12.500000000000002, 12.407408333333333, 12.314813888888889, 12.222222222222221, 12.129630555555556, 12.037036111111112, 11.944444444444445, 11.85185277777778, 11.759258333333335, 11.666666666666666, 11.574075, 11.481480555555555, 11.38888888888889, 11.296297222222222, 11.203702777777778, 11.111111111111112, 11.018519444444443, 10.925925, 10.833333333333332, 10.740741666666667, 10.648147222222223, 10.555555555555557, 10.46296388888889, 10.370369444444446, 10.277777777777777, 10.185186111111111, 10.092591666666667, 10.0, 8.958343750000004, 7.916656250000005, 6.875000000000011, 5.833343749999988, 4.791656249999988, 3.749999999999993, 2.7083437499999983, 1.666656249999999, 0.6250000000000036, -0.41665624999999196, -1.4583437499999903, -2.500000000000014, -3.5416562500000097, -4.583343750000008, -5.6250000000000036, -6.666656249999999, -7.708343749999997, -8.749999999999993, -9.791656250000017, -10.833343750000015, -11.87500000000001, -12.916656250000006, -13.958343750000004, -15.0, -15.416662499999997, -15.833337499999997, -16.250000000000007, -16.666662500000005, -17.083337500000006, -17.500000000000004, -17.9166625, -18.3333375, -18.75, -19.166662500000008, -19.583337500000006, -20.000000000000007, -20.416662500000005, -20.833337500000003, -21.25, -21.6666625, -22.0833375, -22.500000000000007, -22.916662500000008, -23.333337500000006, -23.750000000000007, -24.1666625, -24.583337500000003, -25.0, -24.629633333333324, -24.259255555555548, -23.888888888888882, -23.51852222222222, -23.14814444444444, -22.77777777777778, -22.407411111111113, -22.037033333333333, -21.66666666666666, -21.296299999999995, -20.92592222222222, -20.555555555555554, -20.185188888888888, -19.814811111111112, -19.444444444444446, -19.074077777777774, -18.703699999999994, -18.33333333333333, -17.962966666666667, -17.592588888888887, -17.22222222222222, -16.85185555555556, -16.481477777777783, -16.111111111111107, -15.740744444444442, -15.370366666666664, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0, -15.0], "curve_compare_metrics": {"point_count": 258, "mae": 2.4426571920757962, "rmse": 3.2500268666664343, "pearson": 0.977103530223041, "spearman": 0.9351348813030206}, "curve_compare_plot": "examples/reference_outputs/example_01_gt_vs_pred.png"}
|