docs/dev-guide.md

# Show Me — Developer Guide

## Quick Start

```bash
cd show-me

# Install JS dependencies
npm install

# Start the simulated robot (needs Python venv with reachy-mini)
.venv/bin/python launch_sim_webrtc.py

# In another terminal, start the dev server
npm run dev
```

Open `http://localhost:5174` — click Connect to see SMPTE test bars from the sim.

To connect to a real robot instead, change `ROBOT_HOST` in `src/App.svelte` to `"reachy-mini.local"`.

## Project Structure

```
show-me/
├── src/
│   ├── main.js                          # Mounts Svelte app
│   ├── App.svelte                       # Root layout, creates robot store
│   ├── lib/
│   │   ├── robot/
│   │   │   ├── connection.js            # ReachyMini class (pure JS, no Svelte)
│   │   │   └── commands.js              # High-level command helpers
│   │   └── stores/
│   │       └── robot.svelte.js          # Svelte 5 runes reactive adapter
│   ├── components/
│   │   ├── VideoFeed.svelte             # <video> element bound to stream
│   │   └── Controls.svelte              # Connect, Wake Up, Sleep, State viewer
│   └── styles/
│       └── global.css                   # Dark theme, full-viewport reset
├── vendor/
│   └── gstwebrtc-api-3.0.0.tgz         # GStreamer WebRTC JS lib (built from source)
├── patches/
│   └── fix_webrtc_asyncio.py            # Fix for robot daemon async commands
├── docs/                                # Documentation
├── launch_sim_webrtc.py                 # Desktop simulation launcher
├── index.html                           # Entry point
├── package.json
├── vite.config.js
└── svelte.config.js
```

### Architecture Layers

```
┌─────────────────────────────────────────────────────┐
│  Svelte Components (VideoFeed, Controls, App)       │  UI — reads from store
├─────────────────────────────────────────────────────┤
│  robot.svelte.js — Svelte 5 reactive adapter        │  Glue — $state() runes
├─────────────────────────────────────────────────────┤
│  connection.js + commands.js — pure JS              │  Robot — framework-agnostic
├─────────────────────────────────────────────────────┤
│  gstwebrtc-api v3 — WebRTC signaling + media        │  Transport
└─────────────────────────────────────────────────────┘
```

The robot layer (`src/lib/robot/`) has **zero** Svelte imports. It extends `EventTarget` and emits standard events. This is intentional — it's designed for future extraction into an `npm` package (`reachy-mini-js`).

The store layer (`src/lib/stores/`) is a thin adapter that wires robot events to Svelte 5 `$state()` runes.

## Key Technical Details

### gstwebrtc-api v3

Built from [gst-plugins-rs](https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs) `net/webrtc/gstwebrtc-api/`. Stored as `vendor/gstwebrtc-api-3.0.0.tgz`.

**v3 API patterns (different from v2):**

```javascript
// Discovery — callback-based, NOT addEventListener
api.registerPeerListener({
  producerAdded(producer) { /* producer.id, producer.meta */ },
  producerRemoved(producer) { },
});

// Connection listener
api.registerConnectionListener({
  connected(clientId) { },
  disconnected() { },
});

// Consumer session
const session = api.createConsumerSession(producerId);
session.addEventListener("streamsChanged", () => { /* session.streams */ });
session.addEventListener("stateChanged", () => { /* session.state: 0-3 */ });
session.connect();

// Session states: 0=idle, 1=connecting, 2=streaming, 3=closed
```

**Data channel caveat:** The library only handles channels named `"control"` internally. The robot creates a channel named `"data"` — we intercept it via the underlying `RTCPeerConnection`:

```javascript
session.addEventListener("rtcPeerConnectionChanged", () => {
  const pc = session.rtcPeerConnection;
  pc.addEventListener("datachannel", (event) => {
    if (event.channel.label === "data") { /* use it */ }
  });
});
```

### Data Channel Protocol

Commands are JSON objects with a single top-level key. Responses include `"status"`, `"error"`, or `"command"` keys.

| Command | Example | Async? |
|---|---|---|
| `get_state` | `{ "get_state": true }` | No |
| `set_target` | `{ "set_target": [[4x4 matrix]] }` | No |
| `set_antennas` | `{ "set_antennas": [right, left] }` | No |
| `set_body_yaw` | `{ "set_body_yaw": 0.5 }` | No |
| `goto_target` | `{ "goto_target": { head, antennas, duration, body_yaw } }` | Yes |
| `wake_up` | `{ "wake_up": true }` | Yes |
| `goto_sleep` | `{ "goto_sleep": true }` | Yes |
| `play_sound` | `{ "play_sound": "file.wav" }` | No |
| `set_motor_mode` | `{ "set_motor_mode": "enabled" }` | No |
| `get_motor_mode` | `{ "get_motor_mode": true }` | No |

Async commands respond only when the motion/animation completes (up to 30s).

All angles are in **radians**. Head poses are **4x4 homogeneous matrices** (row-major).

**State poll vs command response disambiguation:** State poll responses only have a `"state"` key. Command responses have `"status"`, `"error"`, or `"command"` keys. The message handler in `connection.js` uses this to avoid state polls accidentally resolving command promises (FIFO queue).

### Robot Daemon Bug: GStreamer Thread vs asyncio

The daemon's data channel callback fires on a GStreamer thread, not the asyncio event loop. Synchronous commands work fine, but async ones (`wake_up`, `goto_sleep`, `goto_target`) call `asyncio.create_task()` which fails with "no running event loop".

**Fix:** `patches/fix_webrtc_asyncio.py` — creates a dedicated `asyncio.new_event_loop()` on a background thread and dispatches all data channel messages there via `call_soon_threadsafe()`.

This patch is applied automatically by `launch_sim_webrtc.py` for local simulation. For a real robot, copy the patch and apply it before the daemon starts (see instructions in the patch file).

The bug exists in reachy-mini v1.3.1. The command handler is in `reachy_mini/daemon/backend/abstract.py` → `_process_webrtc_command()` (line ~874).

### Simulation

`launch_sim_webrtc.py` monkey-patches the daemon to replace hardware-specific GStreamer elements:

| Original (RPi) | Patched (desktop) | Purpose |
|---|---|---|
| `libcamerasrc` | `videotestsrc` (SMPTE bars) | Camera |
| `v4l2h264enc` | handled by `webrtcsink` | Encoding |
| `alsasrc` | `audiotestsrc` (silence) | Microphone |
| `alsasink` | skipped | Speaker |

Requires: Python 3.10+, GStreamer with Rust plugins (`brew install gstreamer` on macOS), and `reachy-mini[gstreamer]` package.

The sim exposes:
- `http://localhost:8000` — REST API + dashboard
- `ws://localhost:8443` — WebRTC signaling

## Build

```bash
npm run build    # → dist/ (static output, ~135KB JS + 2KB CSS)
npm run preview  # preview the built app
```

## What's Done (V0)

- Svelte 5 + Vite 7 scaffold
- gstwebrtc-api v3 integration (built from source)
- WebRTC connection with producer discovery
- Video streaming display (full-viewport, mobile-first)
- Data channel for robot commands
- State polling (500ms) with reactive updates
- Basic controls: Connect, Disconnect, Wake Up, Sleep
- Robot state JSON viewer
- Desktop simulation support
- Daemon asyncio bug fix

## What's Next (V1+)

Per the brainstorm (`docs/brainstorm.md`), the roadmap includes:

- **AI models** — STT (Whisper via transformers.js), LLM (SmolLM2), TTS (Web Speech API / Kokoro), Vision (YOLO)
- **Tools system** — LLM-callable tools (`src/lib/tools/`) for object detection, web search, robot control
- **"Show Me" interaction** — the core UX where the robot directs what's on screen
- **Canvas overlay** — AR annotations on the video feed (`<canvas>` over `<video>`)
- **Web content display** — embedded videos, images, cards
- **Expressive behavior** — head tracking, gestures, conversational feedback
- **HF Spaces deployment** — static SDK, auto-build on push
- **`reachy-mini-js` npm package** — extract `src/lib/robot/` into standalone package