| --- |
| title: Reachy Mini - Camping AI |
| emoji: 🔥 |
| colorFrom: blue |
| colorTo: green |
| sdk: gradio |
| sdk_version: 4.44.0 |
| app_file: app.py |
| pinned: false |
| models: |
| - openbmb/MiniCPM-V-4.6 |
| - openbmb/MiniCPM5-1B |
| - openai/whisper-large-v3-turbo |
| - 2noise/ChatTTS |
| tags: |
| - robotics |
| - voice-control |
| - reachy-mini |
| - camping-assistant |
| - track:backyard |
| - sponsor:openbmb |
| - sponsor:openai |
| - achievement:offgrid |
| - badge-tiny-titan |
| - award-best-agent |
| - award-best-demo |
| --- |
| |
| ## 🏕️ Camping AI: Reachy Mini Smart Camping Voice & Guard Control Platform |
|
|
| This project is a smart camping assistant system custom-tailored for the **Reachy Mini robot**. It integrates multimodal vision recognition (MiniCPM-V), smart voice dialogue (Whisper / MiniCPM / ChatTTS), security patrol guarding, and GPS location reporting, complete with an intuitive and aesthetic Gradio web control interface. |
|
|
|
|
| ## Demo & links |
|
|
| - ▶️ **Demo video** — https://youtu.be/M9ghm4zdzww |
| - 💬 **Launch post (X)** — https://x.com/ivan_lee1007/status/2066642941964382668 |
| |
| --- |
| |
| ## ⚠️ Important Hardware & System Constraints |
| |
| Before running this project, ensure that your environment meets the following hardware and system requirements: |
| |
| 1. **RAM Constraint**: The computer or server running/hosting the local models (LLM/VLM/TTS) should have **more than 16 GB of RAM**. A dedicated GPU with CUDA support is highly recommended for model acceleration. |
| 2. **GPS Device Constraint**: A **GPS-enabled smartphone** is required (either sharing location with the host Mac or using specific Apple Shortcuts/integrations to send coordinates) to enable the location reporting feature. |
| 3. **Robot Body**: A **Reachy Mini** robot with a stable network connection. |
| |
| --- |
| |
| ## 📖 Project Introduction |
| |
| This project bridges AI technology and physical robotics. Using Reachy Mini's rich body language (head movements, antenna wiggling) and sensory modules (head camera, microphone, speaker), it provides four core experiences: |
| - **Smart Voice Dialogue**: Understands user questions and responds using speech synthesis coupled with expressive body gestures. |
| - **Camping Guard Patrol**: Conducts scheduled or manually triggered patrols, leveraging multimodal vision to detect potential hazards or wild animals (such as snakes) nearby. |
| - **Wild Species Identification**: Captures photos of objects or plants in front of the robot and details them out loud. |
| - **Real-Time Location Report**: Integrates phone GPS signals with macOS shortcuts to calculate current latitude/longitude and greets the user. |
| |
| --- |
| |
| ## 🛠️ Installation Guide |
| |
| ### 1. System Dependencies |
| On macOS, certain system utilities are required by the python packages: |
| ```bash |
| # Install ffmpeg for audio format conversion (WAV/MP3 conversion) |
| brew install ffmpeg |
| |
| # Install portaudio for microphone recording and voice monitoring (PyAudio dependency) |
| brew install portaudio |
| ``` |
| |
| ### 2. Virtual Environment & Python Packages |
| Python 3.8+ is recommended: |
| ```bash |
| # Create a virtual environment |
| python3 -m venv .venv |
| |
| # Activate the virtual environment (macOS / Linux) |
| source .venv/bin/activate |
| |
| # Upgrade pip |
| pip install --upgrade pip |
| |
| # Install dependencies |
| pip install -r requirements.txt |
| ``` |
| > [!NOTE] |
| > The `reachy_mini` package is the official robot SDK. If it is not publicly hosted on PyPI, please follow Reachy Mini's official setup instructions to install it manually. |
| > The `ChatTTS` package can also be installed manually via `pip install ChatTTS`. |
|
|
| ### 3. Configure Environment Variables (`.env`) |
| Copy `.env.example` to `.env`: |
| ```bash |
| cp .env.example .env |
| ``` |
| Edit `.env` to configure your settings: |
| ```env |
| # IP address of the local or remote model server |
| server_ip=10.112.5.79 # Update to your actual model server IP |
| ``` |
|
|
| --- |
|
|
| ## 🧠 Local Model Server Configuration (`llm_server`) |
| |
| The local model calculations (LLM/VLM/STT/TTS) rely on services hosted under `llm_server`. **The respective model servers must be running for the main application to function properly.** |
|
|
| These model servers are typically launched on a machine equipped with GPU acceleration (refer to [llm_server/README.md](file:///Users/ivan.lee/dev/camping_ai/llm_server/README.md)): |
|
|
| ### 1. MiniCPM-V-4.6 Multimodal Vision Service (Port 4000) |
| Used for image analysis in **Species Identification** and **Guardian Mode**. |
| ```bash |
| transformers serve openbmb/MiniCPM-V-4.6 --port 4000 --host 0.0.0.0 --trust-remote-code |
| ``` |
|
|
| ### 2. MiniCPM5-1B Language Model Service (Port 4001) |
| Used for text generation in **Smart Dialogue** and **Location Integration**. |
| ```bash |
| transformers serve openbmb/MiniCPM5-1B --port 4001 --host 0.0.0.0 |
| ``` |
|
|
| ### 3. Whisper Speech-to-Text (ASR) Service (Port 4002) |
| Used to transcribe recorded voice commands. |
| ```bash |
| # Start STT server |
| python llm_server/stt_server.py |
| ``` |
|
|
| ### 4. ChatTTS Text-to-Speech (TTS) Service (Port 4003) |
| Used to synthesize robot voice responses. |
| ```bash |
| # Start TTS server |
| python llm_server/tts_server.py |
| ``` |
|
|
| --- |
|
|
| ## 🚀 Starting Services & Function Introduction |
|
|
| ### Launch the Gradio Web Application |
| Run the following command in the project root: |
| ```bash |
| python app.py |
| ``` |
| This will start the web control panel interface (accessible at `http://127.0.0.1:7860` by default). |
|
|
| --- |
|
|
| ### Core Web Control Panel Features & Purpose |
|
|
| #### 1. 🔌 Connection & Settings |
| - **Purpose**: Connect the web application to the Reachy Mini robot. |
| - **Features**: |
| - Specify the robot's **Host IP** (e.g., `localhost` or its local network IP). |
| - Configure the connection mode and video backend (WebRTC or default stream). |
| - Toggle **SSH Tunnel Mode** to resolve remote signaling issues. |
|
|
| #### 2. 📷 Live Stream Viewer |
| - **Purpose**: Real-time monitoring of Reachy Mini's head camera. |
| - **Features**: |
| - Real-time video feedback from Reachy Mini's head camera (converted to RGB format for web display). |
| - Capture and save snapshot frames to the `record/` folder via the **Capture Frame & Save** button. |
|
|
| #### 3. 🎙️ Camp AI Voice Control System |
| You can interact using the **Voice Monitoring Mode** (continuous microphone listening) or manually click buttons in **Manual Mode**: |
|
|
| | Feature Name (Manual Mode Button) | Triggered Script | Core Purpose & Workflow | |
| | :--- | :--- | :--- | |
| | **📍 Location Report** | [location.py](file:///Users/ivan.lee/dev/camping_ai/location.py) | **Get GPS Coordinates & Greet User:**<br>1. Triggers macOS shortcuts to fetch GPS latitude/longitude.<br>2. Combines coordinates and sends them to **MiniCPM5-1B** to generate a friendly Chinese response.<br>3. Converts response to audio via **ChatTTS** and plays it while the robot nods. | |
| | **💬 Voice Dialogue** | [ask.py](file:///Users/ivan.lee/dev/camping_ai/ask.py) | **Bidirectional Conversational AI:**<br>1. Record your query (manually or via voice activation).<br>2. Transcribe audio via **Whisper** and generate replies via **MiniCPM5-1B**.<br>3. Speak the answer via **ChatTTS** while the robot plays interactive head/antenna gestures. | |
| | **🛡️ Guardian Mode** | [guard.py](file:///Users/ivan.lee/dev/camping_ai/guard.py) | **Security Guard Patrol & Danger Detection:**<br>1. Reachy Mini turns its head (0°, 60°, -60°) to capture snapshots.<br>2. Evaluates the images via **MiniCPM-V-4.6** to detect wild animals (e.g., snakes).<br>3. If danger is found (`danger=1`), it plays a warning audio (`warning_song.mp3`) and wiggles its antennas violently. | |
| | **🔍 Species Identification** | [look_that.py](file:///Users/ivan.lee/dev/camping_ai/look_that.py) | **Object & Species Identification:**<br>1. Robot moves up, opens its antennas, turns its body 30°, and takes a snapshot.<br>2. Image is evaluated by **MiniCPM-V-4.6** to describe the object.<br>3. Plays description via **ChatTTS** and wiggles its antennas in a cute interaction before returning to base. | |
|
|
| > [!TIP] |
| > **Voice-Activated Commands (Start Camp AI)**: |
| > When you click **Start Camp AI**, the system continuously monitors the microphone input. When any of the following English phrases are recognized, it triggers the corresponding script automatically: |
| > - `"what is this"` or `"what's this"` ➡️ **Species Identification (`look_that.py`)** |
| > - `"suggestion"` or `"suggestions"` ➡️ **Voice Dialogue (`ask.py`)** |
| > - `"guard mode"`, `"god mode"`, or `"grad mode"` ➡️ **Guardian Mode (`guard.py`)** |
| > - `"location"` ➡️ **Location Report (`location.py`)** |
| |
