docs/dev-guide.md

Show Me — Developer Guide

Quick Start

cd show-me

# Install JS dependencies
npm install

# Start the simulated robot (needs Python venv with reachy-mini)
.venv/bin/python launch_sim_webrtc.py

# In another terminal, start the dev server
npm run dev

Open http://localhost:5174 — click Connect to see SMPTE test bars from the sim.

To connect to a real robot instead, change ROBOT_HOST in src/App.svelte to "reachy-mini.local".

Project Structure

show-me/
├── src/
│   ├── main.js                          # Mounts Svelte app
│   ├── App.svelte                       # Root layout, creates robot store
│   ├── lib/
│   │   ├── robot/
│   │   │   ├── connection.js            # ReachyMini class (pure JS, no Svelte)
│   │   │   └── commands.js              # High-level command helpers
│   │   └── stores/
│   │       └── robot.svelte.js          # Svelte 5 runes reactive adapter
│   ├── components/
│   │   ├── VideoFeed.svelte             # <video> element bound to stream
│   │   └── Controls.svelte              # Connect, Wake Up, Sleep, State viewer
│   └── styles/
│       └── global.css                   # Dark theme, full-viewport reset
├── vendor/
│   └── gstwebrtc-api-3.0.0.tgz         # GStreamer WebRTC JS lib (built from source)
├── patches/
│   └── fix_webrtc_asyncio.py            # Fix for robot daemon async commands
├── docs/                                # Documentation
├── launch_sim_webrtc.py                 # Desktop simulation launcher
├── index.html                           # Entry point
├── package.json
├── vite.config.js
└── svelte.config.js

Architecture Layers

┌─────────────────────────────────────────────────────┐
│  Svelte Components (VideoFeed, Controls, App)       │  UI — reads from store
├─────────────────────────────────────────────────────┤
│  robot.svelte.js — Svelte 5 reactive adapter        │  Glue — $state() runes
├─────────────────────────────────────────────────────┤
│  connection.js + commands.js — pure JS              │  Robot — framework-agnostic
├─────────────────────────────────────────────────────┤
│  gstwebrtc-api v3 — WebRTC signaling + media        │  Transport
└─────────────────────────────────────────────────────┘

The robot layer (src/lib/robot/) has zero Svelte imports. It extends EventTarget and emits standard events. This is intentional — it's designed for future extraction into an npm package (reachy-mini-js).

The store layer (src/lib/stores/) is a thin adapter that wires robot events to Svelte 5 $state() runes.

Key Technical Details

gstwebrtc-api v3

Built from gst-plugins-rs net/webrtc/gstwebrtc-api/. Stored as vendor/gstwebrtc-api-3.0.0.tgz.

v3 API patterns (different from v2):

// Discovery — callback-based, NOT addEventListener
api.registerPeerListener({
  producerAdded(producer) { /* producer.id, producer.meta */ },
  producerRemoved(producer) { },
});

// Connection listener
api.registerConnectionListener({
  connected(clientId) { },
  disconnected() { },
});

// Consumer session
const session = api.createConsumerSession(producerId);
session.addEventListener("streamsChanged", () => { /* session.streams */ });
session.addEventListener("stateChanged", () => { /* session.state: 0-3 */ });
session.connect();

// Session states: 0=idle, 1=connecting, 2=streaming, 3=closed

Data channel caveat: The library only handles channels named "control" internally. The robot creates a channel named "data" — we intercept it via the underlying RTCPeerConnection:

session.addEventListener("rtcPeerConnectionChanged", () => {
  const pc = session.rtcPeerConnection;
  pc.addEventListener("datachannel", (event) => {
    if (event.channel.label === "data") { /* use it */ }
  });
});

Data Channel Protocol

Commands are JSON objects with a single top-level key. Responses include "status", "error", or "command" keys.

Command	Example	Async?
get_state	{ "get_state": true }	No
set_target	{ "set_target": [[4x4 matrix]] }	No
set_antennas	{ "set_antennas": [right, left] }	No
set_body_yaw	{ "set_body_yaw": 0.5 }	No
goto_target	{ "goto_target": { head, antennas, duration, body_yaw } }	Yes
wake_up	{ "wake_up": true }	Yes
goto_sleep	{ "goto_sleep": true }	Yes
play_sound	{ "play_sound": "file.wav" }	No
set_motor_mode	{ "set_motor_mode": "enabled" }	No
get_motor_mode	{ "get_motor_mode": true }	No

Async commands respond only when the motion/animation completes (up to 30s).

All angles are in radians. Head poses are 4x4 homogeneous matrices (row-major).

State poll vs command response disambiguation: State poll responses only have a "state" key. Command responses have "status", "error", or "command" keys. The message handler in connection.js uses this to avoid state polls accidentally resolving command promises (FIFO queue).

Robot Daemon Bug: GStreamer Thread vs asyncio

The daemon's data channel callback fires on a GStreamer thread, not the asyncio event loop. Synchronous commands work fine, but async ones (wake_up, goto_sleep, goto_target) call asyncio.create_task() which fails with "no running event loop".

Fix: patches/fix_webrtc_asyncio.py — creates a dedicated asyncio.new_event_loop() on a background thread and dispatches all data channel messages there via call_soon_threadsafe().

This patch is applied automatically by launch_sim_webrtc.py for local simulation. For a real robot, copy the patch and apply it before the daemon starts (see instructions in the patch file).

The bug exists in reachy-mini v1.3.1. The command handler is in reachy_mini/daemon/backend/abstract.py → _process_webrtc_command() (line ~874).

Simulation

launch_sim_webrtc.py monkey-patches the daemon to replace hardware-specific GStreamer elements:

Original (RPi)	Patched (desktop)	Purpose
libcamerasrc	videotestsrc (SMPTE bars)	Camera
v4l2h264enc	handled by webrtcsink	Encoding
alsasrc	audiotestsrc (silence)	Microphone
alsasink	skipped	Speaker

Requires: Python 3.10+, GStreamer with Rust plugins (brew install gstreamer on macOS), and reachy-mini[gstreamer] package.

The sim exposes:

• http://localhost:8000 — REST API + dashboard
• ws://localhost:8443 — WebRTC signaling

Build

npm run build    # → dist/ (static output, ~135KB JS + 2KB CSS)
npm run preview  # preview the built app

What's Done (V0)

• Svelte 5 + Vite 7 scaffold
• gstwebrtc-api v3 integration (built from source)
• WebRTC connection with producer discovery
• Video streaming display (full-viewport, mobile-first)
• Data channel for robot commands
• State polling (500ms) with reactive updates
• Basic controls: Connect, Disconnect, Wake Up, Sleep
• Robot state JSON viewer
• Desktop simulation support
• Daemon asyncio bug fix

What's Next (V1+)

Per the brainstorm (docs/brainstorm.md), the roadmap includes:

• AI models — STT (Whisper via transformers.js), LLM (SmolLM2), TTS (Web Speech API / Kokoro), Vision (YOLO)
• Tools system — LLM-callable tools (src/lib/tools/) for object detection, web search, robot control
• "Show Me" interaction — the core UX where the robot directs what's on screen
• Canvas overlay — AR annotations on the video feed (<canvas> over <video>)
• Web content display — embedded videos, images, cards
• Expressive behavior — head tracking, gestures, conversational feedback
• HF Spaces deployment — static SDK, auto-build on push
• reachy-mini-js npm package — extract src/lib/robot/ into standalone package