README.md

---
title: Reachy Mini Simple Visual Novel
emoji: 🤖
colorFrom: blue
colorTo: purple
sdk: gradio
pinned: false
app: app.py
python_version: 3.12
---
## Visual Novel Demo
- Install `uv`
- Run `uv run python app.py`.

### Features
- Register characters with sprite URLs or inline SVG data-URIs (see `create_sprite_data_url`).
- Toggle simple idle animation per character (set `animated=True`) or point to GIF/WebP assets for full animation.
- Change backgrounds between scenes.
- Show, hide, and move characters between left/center/right anchors.
- Display narration or speaker dialogue in a speech-bubble overlay anchored to the scene.
- Navigate forward/backward through the story timeline.
- Opt-in webcam overlay per scene: call `builder.set_camera(True|False)` to show or hide the FastRTC stream alongside your story.
- Voice sandbox: record or upload microphone audio, forward it to a placeholder AI companion, and hear a synthetic confirmation tone back.
- Dynamixel control (Web Serial): connect over serial to XL330 servos and send goal positions directly from the browser—no Python SDK needed.
- **Reachy Mini robot control (WebSocket)**: connect to a Reachy Mini robot server and send real-time pose commands for head position, orientation, body yaw, and antennas.
- Per-scene toggles: show/hide camera, voice, motor, and robot controls with `set_camera`, `set_voice`, `set_motors`, and `set_robot`.

#### Customizing sprites
- Replace the SVG data-URIs in `build_sample_story()` with your own URLs (PNG/GIF/WebP).
- For animated sprites, provide an animated GIF/WebP URL and set `animated=True` to also enable the floaty idle motion.
- If you need frame-based animation control, extend `CharacterDefinition` with additional fields (e.g., `animation_frames`) and update `render_scene()` accordingly.

#### Camera widget
- Grant permission when prompted; the browser's default camera is streamed with FastRTC (`WebRTC` component).
- Scenes control whether the webcam appears. If a scene doesn't request it, you'll see a friendly notice instead of the stream.
- Browsers typically require HTTPS (or `http://localhost`) plus user permission before the stream can start; if the feed doesn’t appear, reload after granting access.

#### Voice sandbox
- Scenes decide whether voice capture shows up. Call `builder.set_voice(True|False)` per scene; when disabled, the audio UI hides completely.
- Use the **Voice & Audio Agent** accordion (when visible) to record or upload a clip; hit **Send to voice agent** to hand it to the (placeholder) AI hook.
- The app echoes your recording for playback and emits a synthetic tone to represent an AI voice. Replace `process_voice_interaction()` in `main.py` with real ASR/LLM/TTS calls to integrate your model stack.
- Default prompt text gives the agent scene context; edit it freely.

#### Dynamixel XL330 control
- The control panel lives entirely in the browser using the Web Serial API (Chrome/Edge on desktop). When prompted, select the USB/serial adapter attached to your Dynamixel bus.
- Choose baud, motor ID, and goal angle in the **Dynamixel XL330 Control** panel; click **Connect serial** (triggers the browser port picker) then **Send goal**. Use **Torque on/off** to toggle torque.
- Commands write Protocol 2.0 registers: torque enable (64) and goal position (116, 4 bytes). Angles 0–360° map to 0–4095 ticks.
- Frontend code lives in `web/dxl_webserial.js` and is loaded via `file=web/dxl_webserial.js`, mirroring the structure of `feetech.js`.

#### Reachy Mini robot control
- Connect to a Reachy Mini robot via WebSocket for real-time pose control during story scenes.
- **Requirements**: A running Reachy Mini server at `localhost:8000` with WebSocket endpoint `/api/move/ws/set_target`.
- The connection status is shown in the robot control panel with a color-coded indicator (🔴 disconnected / 🟢 connected).
- **Enable in scenes**: Call `builder.set_robot(True)` to show the robot control widget for specific scenes.
- **Send poses from story**: Use `builder.send_robot_pose()` to command the robot when a scene is displayed:
  ```python
  builder.send_robot_pose(
      head_x=0.0, head_y=0.0, head_z=0.02,  # Head position in meters
      head_roll=0.0, head_pitch=-0.1, head_yaw=0.0,  # Head orientation in radians
      body_yaw=0.0,  # Body rotation in radians
      antenna_left=-0.2, antenna_right=0.2  # Antenna positions in radians
  )
  ```
- WebSocket automatically connects when the widget becomes visible and reconnects if disconnected.
- Poses are sent automatically when navigating to scenes with robot commands (similar to motor commands and audio).

Edit `main.py` to customize `build_sample_story()` or create your own builder logic with `VisualNovelBuilder`.

### Using Custom Assets

Place your files in the `assets/` directory:
- `assets/backgrounds/` - Background images (1200x800 recommended)
- `assets/sprites/` - Character sprites (400x800 recommended, PNG with transparency)
- `assets/audio/` - Audio files (WAV, MP3, etc.)

Then use the helper functions in your story:

```python
from engine import background_asset, sprite_asset, audio_asset

builder.set_background(background_asset("my_background.png"), label="My Scene")

builder.set_characters([
    CharacterDefinition(
        name="Hero",
        image_url=sprite_asset("hero.png"),
        animated=False
    ),
])

# Play audio when scene is displayed
builder.play_sound(audio_asset("my_sound.wav"))
```