ARCHITECTURE.md

Marionette Architecture

This document explains the design of the Marionette app for developers who want to understand, modify, or debug it.

What Marionette Does

Marionette records and plays back head movements (+ optional audio) for the Reachy Mini robot. A user moves the robot's head by hand while the app records the pose at 100 Hz, then replays it as an animated movement. Audio can be recorded from the mic, uploaded as a file, downloaded from YouTube, or picked from the robot's filesystem.

System Overview

┌─────────────────┐         HTTP (polling)        ┌──────────────────┐
│   Browser (UI)  │ ◄──────────────────────────── │  FastAPI Server  │
│   main.js       │ ──────────────────────────── ►│  (Uvicorn)       │
│   index.html    │     POST /api/record          │                  │
│   style.css     │     POST /api/play            │  marionette/     │
└─────────────────┘     GET  /api/state           │  ├── app.py      │
                                                  │  ├── routes.py   │
                                                  │  ├── recording.py│
                                                  │  ├── datasets.py │
                                                  │  ├── audio.py    │
                                                  │  ├── state.py    │
                                                  │  └── models.py   │
                                                  └────────┬─────────┘
                                                           │
                                                  ┌────────▼─────────┐
                                                  │   Reachy Mini    │
                                                  │   (Robot SDK)    │
                                                  └──────────────────┘

Two-Machine vs Single-Machine

• Reachy Mini: Backend runs on the robot (Linux ARM), browser runs on a laptop. Connected via WiFi.
• Reachy Mini Light: Backend and browser run on the same laptop. The robot connects via USB.

Module Map

Module	Purpose	Key classes/functions
models.py	Data types, constants, Pydantic models	RecordingMetadata, RecordingRequest, DatasetEntry, all *Payload classes
audio.py	Stateless audio functions	play_wav_chunked(), preload_wav(), play_preloaded_wav()
state.py	Thread-safe state read/write	StateMixin._serialize_state(), _set_state(), _set_idle_state()
recording.py	Motion capture + playback	RecordingMixin._capture_motion(), _perform_recording(), _perform_playback()
datasets.py	Dataset filesystem + HF sync	DatasetMixin._load_dataset_registry(), _sync_dataset(), _check_hf_login()
routes.py	HTTP endpoint definitions	register_routes() — all FastAPI route closures
app.py	Main class, run loop, robot helpers	Marionette, create_app()
main.py	Re-export hub	Imports and re-exports everything for backward compatibility
motion_models.py	Lead compensation model	MotionModelRegistry, shifts commands forward to counter mechanical lag

Threading Model

The app has three types of threads:

1. Uvicorn thread — Runs the FastAPI HTTP server. Handles all API requests. This is the thread that calls route handlers in routes.py.
2. Main robot thread — Runs Marionette.run(). Polls for pending jobs (recording or playback) in a 50ms loop. Executes recording/playback synchronously.
3. Audio threads — Spawned as daemon threads during playback. Push audio chunks to the robot's GStreamer pipeline.

Thread Safety

All shared state is protected by _state_lock (a threading.Lock):

• The HTTP thread writes: _pending_recording, _pending_playback, _mode
• The robot thread reads and clears: _pending_recording, _pending_playback
• Both threads read: _mode, _recordings, _datasets

The lock is held briefly — never during I/O or network calls.

Cancel Events

• _recording_cancel_event — Set by the HTTP thread (POST /api/record/stop), checked by the robot thread's capture loop.
• _playback_cancel_event — Set by the HTTP thread (POST /api/play/stop), checked by the robot thread's playback loop.

State Machine

          POST /api/record
  idle ──────────────────── queued
   ▲                          │
   │                     (robot thread picks up)
   │                          ▼
   │                      countdown (3s)
   │                          │
   │                          ▼
   └──────────────────── recording
                              │
                         (duration elapsed or stop)
                              ▼
                            idle

          POST /api/play
  idle ──────────────────── queued
   ▲                          │
   │                     (robot thread picks up)
   │                          ▼
   └──────────────────── playing
                              │
                         (move ends or stop)
                              ▼
                            idle

Additional states: starting_up (during boot animation), error (transient, returns to idle).

Frontend Architecture

The frontend is vanilla JavaScript (no framework). Key patterns:

Polling Loop

The browser polls GET /api/state at regular intervals:

• 1500ms when idle (nothing happening)
• 200ms when active (recording, playing, countdown)

Each poll returns the full app state. The frontend updates the UI accordingly.

Dirty Flag Pattern (Flickering Fix)

The moves list and dataset dropdown are expensive to rebuild (full DOM replacement). Without optimization, they flicker on every poll. The fix:

• movesListDirty and datasetsDirty flags start as true
• renderMoves() / updateDatasetUI() only run when the flag is true
• Flags are set true after user actions (record, delete, switch dataset, etc.)
• During idle polling, only lightweight updates happen (mode badge, play/stop buttons)

NTP-Style Clock Sync

The backend sends server_time with each state response. The frontend computes clockOffset = serverTime - localTime and uses it to accurately display countdown timers and recording progress bars, even when the browser and robot are on different machines with unsynchronized clocks.

Phase Overlay

During countdown and recording, a full-screen overlay shows progress. This uses requestAnimationFrame for smooth 60fps animation, independent of the polling interval.

Dataset Layout

~/reachy_mini_datasets/           # dataset root (configurable)
├── local_dataset/                # default dataset
│   └── data/                     # all recordings live here
│       ├── happy-dance.json      # motion trajectory
│       └── happy-dance.wav       # optional audio
├── my-custom-set/
│   └── data/
│       └── ...
└── user-community-set/           # downloaded from HF
    └── data/
        └── ...

The dataset_registry.json file (next to the app) tracks which datasets exist, which is active, and per-dataset metadata (uploaded move IDs, origin, etc.).

Audio Playback Pipeline

Audio is played through the Reachy Mini's GStreamer pipeline using chunk-based pushing:

1. Preload: Read WAV file, resample to output rate if needed
2. Prime pipeline: Call start_playing() to initialize GStreamer
3. Wait for sync: Audio thread waits for start_signal (set when first motion command is sent)
4. Push chunks: Feed 20ms audio chunks at ~1.25x real-time
5. Drain buffer: Wait for remaining audio to play out
6. Cleanup: Call stop_playing() with timeout (GStreamer can hang)

This push-based approach allows stopping audio at any time, unlike play_sound() which creates an uninterruptible pipeline.

Lead Compensation

Mechanical lag means the robot's actual motion trails the commanded trajectory. The lead compensation model (in motion_models.py) shifts commands forward in time so the actual motion matches the original recording. Parameters (head lead, antenna lead) are tunable in the settings UI.

Testing

Tests live in tests/test_api.py and use FastAPI's TestClient for synchronous HTTP testing without a real robot. The conftest.py creates a Marionette instance with a temporary dataset directory.

Key test patterns:

• HF functions are monkeypatched on marionette.datasets (the canonical location)
• Audio functions are monkeypatched on marionette.recording (where they're imported)
• Recording/playback tests use fake ReachyMini objects with stub methods