README.md

metadata

title: Reachy Mini - Camping AI
emoji: 🔥
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 4.44.0
app_file: app.py
pinned: false
models:
  - openbmb/MiniCPM-V-4.6
  - openbmb/MiniCPM5-1B
  - openai/whisper-large-v3-turbo
  - 2noise/ChatTTS
tags:
  - robotics
  - voice-control
  - reachy-mini
  - camping-assistant
  - track:backyard
  - sponsor:openbmb
  - sponsor:openai
  - achievement:offgrid
  - badge-tiny-titan
  - award-best-agent
  - award-best-demo

🏕️ Camping AI: Reachy Mini Smart Camping Voice & Guard Control Platform

This project is a smart camping assistant system custom-tailored for the Reachy Mini robot. It integrates multimodal vision recognition (MiniCPM-V), smart voice dialogue (Whisper / MiniCPM / ChatTTS), security patrol guarding, and GPS location reporting, complete with an intuitive and aesthetic Gradio web control interface.

Demo & links

• ▶️ Demo video — https://youtu.be/M9ghm4zdzww
• 💬 Launch post (X) — https://x.com/ivan_lee1007/status/2066642941964382668

---

⚠️ Important Hardware & System Constraints

Before running this project, ensure that your environment meets the following hardware and system requirements:

1. RAM Constraint: The computer or server running/hosting the local models (LLM/VLM/TTS) should have more than 16 GB of RAM. A dedicated GPU with CUDA support is highly recommended for model acceleration.
2. GPS Device Constraint: A GPS-enabled smartphone is required (either sharing location with the host Mac or using specific Apple Shortcuts/integrations to send coordinates) to enable the location reporting feature.
3. Robot Body: A Reachy Mini robot with a stable network connection.

---

📖 Project Introduction

This project bridges AI technology and physical robotics. Using Reachy Mini's rich body language (head movements, antenna wiggling) and sensory modules (head camera, microphone, speaker), it provides four core experiences:

• Smart Voice Dialogue: Understands user questions and responds using speech synthesis coupled with expressive body gestures.
• Camping Guard Patrol: Conducts scheduled or manually triggered patrols, leveraging multimodal vision to detect potential hazards or wild animals (such as snakes) nearby.
• Wild Species Identification: Captures photos of objects or plants in front of the robot and details them out loud.
• Real-Time Location Report: Integrates phone GPS signals with macOS shortcuts to calculate current latitude/longitude and greets the user.

---

🛠️ Installation Guide

1. System Dependencies

On macOS, certain system utilities are required by the python packages:

# Install ffmpeg for audio format conversion (WAV/MP3 conversion)
brew install ffmpeg

# Install portaudio for microphone recording and voice monitoring (PyAudio dependency)
brew install portaudio

2. Virtual Environment & Python Packages

Python 3.8+ is recommended:

# Create a virtual environment
python3 -m venv .venv

# Activate the virtual environment (macOS / Linux)
source .venv/bin/activate

# Upgrade pip
pip install --upgrade pip

# Install dependencies
pip install -r requirements.txt

The reachy_mini package is the official robot SDK. If it is not publicly hosted on PyPI, please follow Reachy Mini's official setup instructions to install it manually. The ChatTTS package can also be installed manually via pip install ChatTTS.

3. Configure Environment Variables (.env)

Copy .env.example to .env:

cp .env.example .env

Edit .env to configure your settings:

# IP address of the local or remote model server
server_ip=10.112.5.79   # Update to your actual model server IP

---

🧠 Local Model Server Configuration (llm_server)

The local model calculations (LLM/VLM/STT/TTS) rely on services hosted under llm_server. The respective model servers must be running for the main application to function properly.

These model servers are typically launched on a machine equipped with GPU acceleration (refer to llm_server/README.md):

1. MiniCPM-V-4.6 Multimodal Vision Service (Port 4000)

Used for image analysis in Species Identification and Guardian Mode.

transformers serve openbmb/MiniCPM-V-4.6 --port 4000 --host 0.0.0.0 --trust-remote-code

2. MiniCPM5-1B Language Model Service (Port 4001)

Used for text generation in Smart Dialogue and Location Integration.

transformers serve openbmb/MiniCPM5-1B --port 4001 --host 0.0.0.0

3. Whisper Speech-to-Text (ASR) Service (Port 4002)

Used to transcribe recorded voice commands.

# Start STT server
python llm_server/stt_server.py

4. ChatTTS Text-to-Speech (TTS) Service (Port 4003)

Used to synthesize robot voice responses.

# Start TTS server
python llm_server/tts_server.py

---

🚀 Starting Services & Function Introduction

Launch the Gradio Web Application

Run the following command in the project root:

python app.py

This will start the web control panel interface (accessible at http://127.0.0.1:7860 by default).

---

Core Web Control Panel Features & Purpose

1. 🔌 Connection & Settings

• Purpose: Connect the web application to the Reachy Mini robot.
• Features:
  • Specify the robot's Host IP (e.g., localhost or its local network IP).
  • Configure the connection mode and video backend (WebRTC or default stream).
  • Toggle SSH Tunnel Mode to resolve remote signaling issues.

2. 📷 Live Stream Viewer

• Purpose: Real-time monitoring of Reachy Mini's head camera.
• Features:
  • Real-time video feedback from Reachy Mini's head camera (converted to RGB format for web display).
  • Capture and save snapshot frames to the record/ folder via the Capture Frame & Save button.

3. 🎙️ Camp AI Voice Control System

You can interact using the Voice Monitoring Mode (continuous microphone listening) or manually click buttons in Manual Mode:

Feature Name (Manual Mode Button)	Triggered Script	Core Purpose & Workflow
📍 Location Report	location.py	Get GPS Coordinates & Greet User: 1. Triggers macOS shortcuts to fetch GPS latitude/longitude. 2. Combines coordinates and sends them to MiniCPM5-1B to generate a friendly Chinese response. 3. Converts response to audio via ChatTTS and plays it while the robot nods.
💬 Voice Dialogue	ask.py	Bidirectional Conversational AI: 1. Record your query (manually or via voice activation). 2. Transcribe audio via Whisper and generate replies via MiniCPM5-1B. 3. Speak the answer via ChatTTS while the robot plays interactive head/antenna gestures.
🛡️ Guardian Mode	guard.py	Security Guard Patrol & Danger Detection: 1. Reachy Mini turns its head (0°, 60°, -60°) to capture snapshots. 2. Evaluates the images via MiniCPM-V-4.6 to detect wild animals (e.g., snakes). 3. If danger is found (danger=1), it plays a warning audio (warning_song.mp3) and wiggles its antennas violently.
🔍 Species Identification	look_that.py	Object & Species Identification: 1. Robot moves up, opens its antennas, turns its body 30°, and takes a snapshot. 2. Image is evaluated by MiniCPM-V-4.6 to describe the object. 3. Plays description via ChatTTS and wiggles its antennas in a cute interaction before returning to base.

Voice-Activated Commands (Start Camp AI): When you click Start Camp AI, the system continuously monitors the microphone input. When any of the following English phrases are recognized, it triggers the corresponding script automatically:

• "what is this" or "what's this" ➡️ Species Identification (look_that.py)
• "suggestion" or "suggestions" ➡️ Voice Dialogue (ask.py)
• "guard mode", "god mode", or "grad mode" ➡️ Guardian Mode (guard.py)
• "location" ➡️ Location Report (location.py)