Hierarchical System ROS2

0- Preface

What we have

The paths and names of the main logic (Python) folders and files are as follows:

src/
   └── g0_vlm_node/
        └── g0_vlm_node
            ├── utils/                  # Stores functions related to Gemini API processing
            └── vlm_main.py             # Core logic for VLM service provision

Note: In the above package:
- vlm_main.py

Development Log

VLM
1. Format the String so that the JSON string sent by EHI is converted into a structured string.
2. Support the cache switch for receiving repeated instruction from EHI.
3. Support parameterized startup, using --use-qwen and --no-use-qwen to control model usage, with Gemini as the default.

1- Install

Install Python dependencies

Refer to https://github.com/whitbrunn/G0

Compile the workspace

Clone the src/ folder to the local workspace under TO/YOUR/WORKSPACE/, then run:

cd TO/YOUR/WORKSPACE/
colcon build --symlink-install --cmake-args -DPython3_ROOT_DIR=$CONDA_PREFIX

Note:

Use ros2 pkg list | grep PACK_NAME to check if the following ROS packages exist:

g0_vlm_node

2- Usage

Set your VLM API key

export VLM_API_KEY=<YOUR_GEMINI_API_KEY> 
export VLM_API_KEY_QWEN=<YOUR_QWEN_API_KEY>

Start the VLM Node

1.1 First configure the proxy according to the environment (necessary for Gemini, if using the qwen version, skip to 1.3)

export https_proxy=http://127.0.0.1:<PORT>
export http_proxy=http://127.0.0.1:<PORT>
export all_proxy=http://127.0.0.1:<PORT>

1.2 Verify if the external network is accessible

curl -I www.google.com

Expected output (partial):

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: private
Connection: keep-alive

1.3 After confirming the above step is OK, start the VLM node

ros2 run g0_vlm_node vlm_main

*If using the qwen model inference:

unset http_proxy
unset https_proxy
unset all_proxy
ros2 run g0_vlm_node vlm_main -- --use-qwen

3- What you expect

VLM receives a Send request output, e.g.,

2025-11-05 07:40:33.230 | INFO     | g0_vlm_node.vlm_main:vlm_processor1:153 - One hp successfully processed: 将咖啡罐用右手放到托盘上 -> [Low]: Pick up the coffee can with the right hand and place it on the tray.!

VLM receives a confirm request, e.g.,

2025-11-05 07:40:47.641 | INFO     | g0_vlm_node.vlm_main:vlm_processor2:169 - One hp_ successfully sent to VLA: [Low]: Pick up the coffee can with the right hand and place it on the tray.!