Deploying Open Source Vision Language Models (VLM) on Jetson

Vision-Language Models (VLMs) mark a significant leap in AI by blending visual perception with semantic reasoning. Moving beyond traditional models constrained by fixed labels, VLMs utilize a joint embedding space to interpret and discuss complex, open-ended environments using natural language.

The rapid evolution of reasoning accuracy and efficiency has made these models ideal for edge devices. The NVIDIA Jetson family, ranging from the high-performance AGX Thor and AGX Orin to the compact Orin Nano Super is purpose-built to drive accelerated applications for physical AI and robotics, providing the optimized runtime necessary for leading open source models.

In this tutorial, we will demonstrate how to deploy the NVIDIA Cosmos Reason 2B model across the Jetson lineup using the vLLM framework. We will also guide you through connecting this model to the Live VLM WebUI, enabling a real-time, webcam-based interface for interactive physical AI.

Prerequisites

Supported Devices:

• Jetson AGX Thor Developer Kit
• Jetson AGX Orin (64GB / 32GB)
• Jetson Orin Super Nano

JetPack Version:

• JetPack 6 (L4T r36.x) — for Orin devices
• JetPack 7 (L4T r38.x) — for Thor

Storage: NVMe SSD required

• ~5 GB for the FP8 model weights
• ~8 GB for the vLLM container image

Accounts:

• Create NVIDIA NGC account(free) to download both the model and vLLM contanier

---

Overview

The workflow is the same for both devices:

1. Download the FP8 model checkpoint via NGC CLI
2. Pull the vLLM Docker image for your device
3. Launch the container with the model mounted as a volume
4. Connect Live VLM WebUI to the vLLM endpoint

---

Step 1: Install the NGC CLI

The NGC CLI lets you download model checkpoints from the NVIDIA NGC Catalog.

Download and install

mkdir -p ~/Projects/CosmosReason
cd ~/Projects/CosmosReason

# Download the NGC CLI for ARM64
# Get the latest installer URL from: https://org.ngc.nvidia.com/setup/installers/cli
wget -O ngccli_arm64.zip https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/4.13.0/files/ngccli_arm64.zip
unzip ngccli_arm64.zip
chmod u+x ngc-cli/ngc

# Add to PATH
export PATH="$PATH:$(pwd)/ngc-cli"

Configure the CLI

ngc config set

You will be prompted for:

• API Key — generate one at NGC API Key setup
• CLI output format — choose json or ascii
• org — press Enter to accept the default

---

Step 2: Download the Model

Download the FP8 quantized checkpoint. This is used on all Jetson devices:

cd ~/Projects/CosmosReason
ngc registry model download-version "nim/nvidia/cosmos-reason2-2b:1208-fp8-static-kv8"

This creates a directory called cosmos-reason2-2b_v1208-fp8-static-kv8/ containing the model weights. Note the full path — you will mount it into the Docker container as a volume.

---

Step 3: Pull the vLLM Docker Image

For Jetson AGX Thor

docker pull nvcr.io/nvidia/vllm:26.01-py3

For Jetson AGX Orin / Orin Super Nano

docker pull ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04

---

Step 4: Serve Cosmos Reason 2B with vLLM

Option A: Jetson AGX Thor

Thor has ample GPU memory and can run the model with a generous context length.

Set the path to your downloaded model and free cached memory on the host:

MODEL_PATH="$HOME/Projects/CosmosReason/cosmos-reason2-2b_v1208-fp8-static-kv8"
sudo sysctl -w vm.drop_caches=3

Launch the container with the model mounted:

docker run --rm -it \
  --runtime nvidia \
  --network host \
  --ipc host \
  -v "$MODEL_PATH:/models/cosmos-reason2-2b:ro" \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
  nvcr.io/nvidia/vllm:26.01-py3 \
  bash

Inside the container, activate the environment and serve the model:

vllm serve /models/cosmos-reason2-2b \
  --max-model-len 8192 \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.8

Note: The --reasoning-parser qwen3 flag enables chain-of-thought reasoning extraction. The --media-io-kwargs flag configures video frame handling.

Wait until you see:

INFO:     Uvicorn running on http://0.0.0.0:8000

Option B: Jetson AGX Orin

AGX Orin has enough memory to run the model with the same generous parameters as Thor.

Set the path to your downloaded model and free cached memory on the host:

MODEL_PATH="$HOME/Projects/CosmosReason/cosmos-reason2-2b_v1208-fp8-static-kv8"
sudo sysctl -w vm.drop_caches=3

1. Launch the container:

docker run --rm -it \
  --runtime nvidia \
  --network host \
  -v "$MODEL_PATH:/models/cosmos-reason2-2b:ro" \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
  ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04 \
  bash

2. Inside the container, activate the environment and serve:

cd /opt/
source venv/bin/activate

vllm serve /models/cosmos-reason2-2b \
  --max-model-len 8192 \
  --media-io-kwargs '{"video": {"num_frames": -1}}' \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.8

Wait until you see:

INFO:     Uvicorn running on http://0.0.0.0:8000

Option C: Jetson Orin Super Nano (memory-constrained)

The Orin Super Nano has significantly less RAM, so we need aggressive memory optimization flags.

Set the path to your downloaded model and free cached memory on the host:

MODEL_PATH="$HOME/Projects/CosmosReason/cosmos-reason2-2b_v1208-fp8-static-kv8"
sudo sysctl -w vm.drop_caches=3

1. Launch the container:

docker run --rm -it \
  --runtime nvidia \
  --network host \
  -v "$MODEL_PATH:/models/cosmos-reason2-2b:ro" \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
  ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04 \
  bash

2. Inside the container, activate the environment and serve:

cd /opt/
source venv/bin/activate

vllm serve /models/cosmos-reason2-2b \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --enforce-eager \
  --max-model-len 256 \
  --max-num-batched-tokens 256 \
  --gpu-memory-utilization 0.65 \
  --max-num-seqs 1 \
  --enable-chunked-prefill \
  --limit-mm-per-prompt '{"image":1,"video":1}' \
  --mm-processor-kwargs '{"num_frames":2,"max_pixels":150528}'

Key flags explained (Orin Super Nano only):

Wait until you see the server is ready:

INFO:     Uvicorn running on http://0.0.0.0:8000

Verify the server is running

From another terminal on the Jetson:

curl http://localhost:8000/v1/models

You should see the model listed in the response.

---

Step 5: Test with a Quick API Call

Before connecting the WebUI, verify the model responds correctly:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models/cosmos-reason2-2b",
    "messages": [
      {
        "role": "user",
        "content": "What capabilities do you have?"
      }
    ],
    "max_tokens": 128
  }' | python3 -m json.tool

Tip: The model name used in the API request must match what vLLM reports. Verify with curl http://localhost:8000/v1/models.

---

Step 6: Connect to Live VLM WebUI

Live VLM WebUI provides a real-time webcam-to-VLM interface. With vLLM serving Cosmos Reason 2B, you can stream your webcam and get live AI analysis with reasoning.

Install Live VLM WebUI

The easiest method is pip (Open another terminal):

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
cd ~/Projects/CosmosReason
uv venv .live-vlm --python 3.12
source .live-vlm/bin/activate
uv pip install live-vlm-webui
live-vlm-webui

Or use Docker:

git clone https://github.com/nvidia-ai-iot/live-vlm-webui.git
cd live-vlm-webui
./scripts/start_container.sh

Configure the WebUI

1. Open https://localhost:8090 in your browser
2. Accept the self-signed certificate (click Advanced → Proceed)
3. In the VLM API Configuration section on the left sidebar:
   • Set API Base URL to http://localhost:8000/v1
   • Click the Refresh button to detect the model
   • Select the Cosmos Reason 2B model from the dropdown
4. Select your camera and click Start

The WebUI will now stream your webcam frames to Cosmos Reason 2B and display the model’s analysis in real-time.

Recommended WebUI settings for Orin

Since Orin runs with a shorter context length, adjust these settings in the WebUI:

• Max Tokens: Set to 100–150 (shorter responses complete faster)
• Frame Processing Interval: Set to 60+ (gives the model time between frames)

---

Troubleshooting

Out of memory on Orin

Problem: vLLM crashes with CUDA out-of-memory errors.

Solution:

1. Free system memory before starting:

   sudo sysctl -w vm.drop_caches=3
   

2. Lower --gpu-memory-utilization (try 0.55 or 0.50)

3. Reduce --max-model-len further (try 128)

4. Make sure no other GPU-intensive processes are running

Model not found in WebUI

Problem: The model doesn’t appear in the Live VLM WebUI dropdown.

Solution:

1. Verify vLLM is running: curl http://localhost:8000/v1/models
2. Make sure the WebUI API Base URL is set to http://localhost:8000/v1 (not https)
3. If vLLM and WebUI are in separate containers, use http://<jetson-ip>:8000/v1 instead of localhost

Slow inference on Orin

Problem: Each response takes a very long time.

Solution:

• This is expected with the memory-constrained configuration. Cosmos Reason 2B FP8 on Orin prioritizes fitting in memory over speed
• Reduce max_tokens in the WebUI to get shorter, faster responses
• Increase the frame interval so the model isn’t constantly processing new frames

vLLM fails to load model

Problem: vLLM reports that the model path doesn’t exist or can’t be loaded.

Solution:

• Verify the NGC download completed successfully: ls ~/Projects/CosmosReason/cosmos-reason2-2b_v1208-fp8-static-kv8/
• Make sure the volume mount path is correct in your docker run command
• Check that the model directory is mounted as read-only (:ro) and the path inside the container matches what you pass to vllm serve

---

Summary

In this tutorial, we showcased how to deploy NVIDIA Cosmos Reason 2B model on Jetson family of devices using vLLM.

The combination of Cosmos Reason 2B’s chain-of-thought capabilities with Live VLM WebUI’s real-time streaming makes it ideal to prototype and evaluate vision AI applications at the edge.

---

Additional Resources

• Cosmos Reason 2B on NVIDIA Build: https://huggingface.co/nvidia/Cosmos-Reason2-2B
• NGC Model Catalog: https://catalog.ngc.nvidia.com/
• Live VLM WebUI: https://github.com/NVIDIA-AI-IOT/live-vlm-webui
• vLLM container for Jetson Thor: https://ghcr.io/nvidia-ai-iot/vllm:r36.4-tegra-aarch64-cu126-22.04
• vLLM container for Jetson AGX Orin, and Orin Super Nano: https://nvcr.io/nvidia/vllm:26.01-py3
• NGC CLI Installers: https://org.ngc.nvidia.com/setup/installers/cli
• Open Models supported on Jetson: https://www.jetson-ai-lab.com/models/
• Getting started with Jetson: https://www.jetson-ai-lab.com/tutorials/