README.md

---
title: Reachy Mini - Camping AI
emoji: 🔥
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 4.44.0
app_file: app.py
pinned: false
models:
  - openbmb/MiniCPM-V-4.6
  - openbmb/MiniCPM5-1B
  - openai/whisper-large-v3-turbo
  - 2noise/ChatTTS
tags:
  - robotics
  - voice-control
  - reachy-mini
  - camping-assistant
  - track:backyard
  - sponsor:openbmb
  - sponsor:openai
  - achievement:offgrid
  - badge-tiny-titan
  - award-best-agent
  - award-best-demo
---

## 🏕️ Camping AI: Reachy Mini Smart Camping Voice & Guard Control Platform

This project is a smart camping assistant system custom-tailored for the **Reachy Mini robot**. It integrates multimodal vision recognition (MiniCPM-V), smart voice dialogue (Whisper / MiniCPM / ChatTTS), security patrol guarding, and GPS location reporting, complete with an intuitive and aesthetic Gradio web control interface.


## Demo & links

- ▶️ **Demo video** — https://youtu.be/M9ghm4zdzww
- 💬 **Launch post (X)** — https://x.com/ivan_lee1007/status/2066642941964382668

---

## ⚠️ Important Hardware & System Constraints

Before running this project, ensure that your environment meets the following hardware and system requirements:

1. **RAM Constraint**: The computer or server running/hosting the local models (LLM/VLM/TTS) should have **more than 16 GB of RAM**. A dedicated GPU with CUDA support is highly recommended for model acceleration.
2. **GPS Device Constraint**: A **GPS-enabled smartphone** is required (either sharing location with the host Mac or using specific Apple Shortcuts/integrations to send coordinates) to enable the location reporting feature.
3. **Robot Body**: A **Reachy Mini** robot with a stable network connection.

---

## 📖 Project Introduction

This project bridges AI technology and physical robotics. Using Reachy Mini's rich body language (head movements, antenna wiggling) and sensory modules (head camera, microphone, speaker), it provides four core experiences:
- **Smart Voice Dialogue**: Understands user questions and responds using speech synthesis coupled with expressive body gestures.
- **Camping Guard Patrol**: Conducts scheduled or manually triggered patrols, leveraging multimodal vision to detect potential hazards or wild animals (such as snakes) nearby.
- **Wild Species Identification**: Captures photos of objects or plants in front of the robot and details them out loud.
- **Real-Time Location Report**: Integrates phone GPS signals with macOS shortcuts to calculate current latitude/longitude and greets the user.

---

## 🛠️ Installation Guide

### 1. System Dependencies
On macOS, certain system utilities are required by the python packages:
```bash
# Install ffmpeg for audio format conversion (WAV/MP3 conversion)
brew install ffmpeg

# Install portaudio for microphone recording and voice monitoring (PyAudio dependency)
brew install portaudio
```

### 2. Virtual Environment & Python Packages
Python 3.8+ is recommended:
```bash
# Create a virtual environment
python3 -m venv .venv

# Activate the virtual environment (macOS / Linux)
source .venv/bin/activate

# Upgrade pip
pip install --upgrade pip

# Install dependencies
pip install -r requirements.txt
```
> [!NOTE]
> The `reachy_mini` package is the official robot SDK. If it is not publicly hosted on PyPI, please follow Reachy Mini's official setup instructions to install it manually.
> The `ChatTTS` package can also be installed manually via `pip install ChatTTS`.

### 3. Configure Environment Variables (`.env`)
Copy `.env.example` to `.env`:
```bash
cp .env.example .env
```
Edit `.env` to configure your settings:
```env
# IP address of the local or remote model server
server_ip=10.112.5.79   # Update to your actual model server IP
```

---

## 🧠 Local Model Server Configuration (`llm_server`)

The local model calculations (LLM/VLM/STT/TTS) rely on services hosted under `llm_server`. **The respective model servers must be running for the main application to function properly.**

These model servers are typically launched on a machine equipped with GPU acceleration (refer to [llm_server/README.md](file:///Users/ivan.lee/dev/camping_ai/llm_server/README.md)):

### 1. MiniCPM-V-4.6 Multimodal Vision Service (Port 4000)
Used for image analysis in **Species Identification** and **Guardian Mode**.
```bash
transformers serve openbmb/MiniCPM-V-4.6 --port 4000 --host 0.0.0.0 --trust-remote-code
```

### 2. MiniCPM5-1B Language Model Service (Port 4001)
Used for text generation in **Smart Dialogue** and **Location Integration**.
```bash
transformers serve openbmb/MiniCPM5-1B --port 4001 --host 0.0.0.0
```

### 3. Whisper Speech-to-Text (ASR) Service (Port 4002)
Used to transcribe recorded voice commands.
```bash
# Start STT server
python llm_server/stt_server.py
```

### 4. ChatTTS Text-to-Speech (TTS) Service (Port 4003)
Used to synthesize robot voice responses.
```bash
# Start TTS server
python llm_server/tts_server.py
```

---

## 🚀 Starting Services & Function Introduction

### Launch the Gradio Web Application
Run the following command in the project root:
```bash
python app.py
```
This will start the web control panel interface (accessible at `http://127.0.0.1:7860` by default).

---

### Core Web Control Panel Features & Purpose

#### 1. 🔌 Connection & Settings
- **Purpose**: Connect the web application to the Reachy Mini robot.
- **Features**:
  - Specify the robot's **Host IP** (e.g., `localhost` or its local network IP).
  - Configure the connection mode and video backend (WebRTC or default stream).
  - Toggle **SSH Tunnel Mode** to resolve remote signaling issues.

#### 2. 📷 Live Stream Viewer
- **Purpose**: Real-time monitoring of Reachy Mini's head camera.
- **Features**:
  - Real-time video feedback from Reachy Mini's head camera (converted to RGB format for web display).
  - Capture and save snapshot frames to the `record/` folder via the **Capture Frame & Save** button.

#### 3. 🎙️ Camp AI Voice Control System
You can interact using the **Voice Monitoring Mode** (continuous microphone listening) or manually click buttons in **Manual Mode**:

| Feature Name (Manual Mode Button) | Triggered Script | Core Purpose & Workflow |
| :--- | :--- | :--- |
| **📍 Location Report** | [location.py](file:///Users/ivan.lee/dev/camping_ai/location.py) | **Get GPS Coordinates & Greet User:**<br>1. Triggers macOS shortcuts to fetch GPS latitude/longitude.<br>2. Combines coordinates and sends them to **MiniCPM5-1B** to generate a friendly Chinese response.<br>3. Converts response to audio via **ChatTTS** and plays it while the robot nods. |
| **💬 Voice Dialogue** | [ask.py](file:///Users/ivan.lee/dev/camping_ai/ask.py) | **Bidirectional Conversational AI:**<br>1. Record your query (manually or via voice activation).<br>2. Transcribe audio via **Whisper** and generate replies via **MiniCPM5-1B**.<br>3. Speak the answer via **ChatTTS** while the robot plays interactive head/antenna gestures. |
| **🛡️ Guardian Mode** | [guard.py](file:///Users/ivan.lee/dev/camping_ai/guard.py) | **Security Guard Patrol & Danger Detection:**<br>1. Reachy Mini turns its head (0°, 60°, -60°) to capture snapshots.<br>2. Evaluates the images via **MiniCPM-V-4.6** to detect wild animals (e.g., snakes).<br>3. If danger is found (`danger=1`), it plays a warning audio (`warning_song.mp3`) and wiggles its antennas violently. |
| **🔍 Species Identification** | [look_that.py](file:///Users/ivan.lee/dev/camping_ai/look_that.py) | **Object & Species Identification:**<br>1. Robot moves up, opens its antennas, turns its body 30°, and takes a snapshot.<br>2. Image is evaluated by **MiniCPM-V-4.6** to describe the object.<br>3. Plays description via **ChatTTS** and wiggles its antennas in a cute interaction before returning to base. |

> [!TIP]
> **Voice-Activated Commands (Start Camp AI)**:
> When you click **Start Camp AI**, the system continuously monitors the microphone input. When any of the following English phrases are recognized, it triggers the corresponding script automatically:
> - `"what is this"` or `"what's this"` ➡️ **Species Identification (`look_that.py`)**
> - `"suggestion"` or `"suggestions"` ➡️ **Voice Dialogue (`ask.py`)**
> - `"guard mode"`, `"god mode"`, or `"grad mode"` ➡️ **Guardian Mode (`guard.py`)**
> - `"location"` ➡️ **Location Report (`location.py`)**