"""Grabette fleet — operator dashboard + command broker (Docker HF Space). A free-tier-friendly Docker Space. It is self-contained (no `grabette` import) so it deploys standalone to HF. Responsibilities: * Operator UI + login via HF Spaces native OAuth (`hf_oauth: true`). * Command broker: an in-memory, per-owner device registry + command queue. * Device auth: devices call with `Authorization: Bearer `; we resolve the owner via `whoami` (cached) and group devices by HF identity. Transport is short-polling (devices GET /api/devices/poll every couple seconds). That polling traffic doubles as the keep-alive that stops a free Space sleeping (sleep is timed from the last request). State is in-memory, so a restart drops it — devices simply re-register on their next poll; durable data lives in the device's own HF datasets, not here. Designed to be duplicated per user: one Space = one owner = one fleet. """ from __future__ import annotations import asyncio import logging import os import time import uuid from dataclasses import dataclass, field from datetime import datetime, timedelta, timezone from typing import Any, Optional from fastapi import Depends, FastAPI, Header, HTTPException, Request from fastapi.responses import HTMLResponse, RedirectResponse logger = logging.getLogger("grabette-fleet") from huggingface_hub import attach_huggingface_oauth, parse_huggingface_oauth, whoami from pydantic import BaseModel # A device is "online" if the fleet heard from it within this window. Liveness # no longer rides on the command poll (long-polling holds that open ~LONG_POLL_S, # and muxing runs off the poll loop) but on a dedicated lightweight HEARTBEAT the # device sends every DEVICE_HEARTBEAT_S. So this only needs to cover a missed # heartbeat + jitter (~3× the interval) — small, for fast disconnect detection. DEVICE_HEARTBEAT_S = 5.0 # must match the device's relay heartbeat cadence ONLINE_WINDOW = 15.0 # ~3 missed heartbeats → offline within ~15s # Only discard a device's queued commands once it's been gone THIS long — far # beyond ONLINE_WINDOW and any transient blip. Clearing on the mere online # window (15s) would nuke an in-flight start/stop the instant a device flapped, # breaking group sync; a device that's briefly "offline" but still holding a # long-poll receives its queued command the moment it's enqueued regardless. STALE_QUEUE_S = 120.0 WHOAMI_TTL = 300.0 # Long-polling: the poll endpoint holds the connection open up to this many # seconds when the device's queue is empty, returning the instant a command is # enqueued (see _enqueue). This cuts command-delivery latency from ~1 poll # interval to a network round-trip — notably a fanned-out group STOP reaches a # peer in ~ms, so grouped episodes end within ~ms of each other instead of ~1s. # MUST stay below BOTH the HF Space reverse-proxy idle timeout AND ONLINE_WINDOW # (a held poll only refreshes last_seen when it returns, so a device mid-hold # must not age out as offline). Set to 0 to DISABLE → classic short-polling: the # poll returns immediately and the client throttles to its poll_interval. The # relay client auto-detects which mode the server is in, so flipping this needs # no device change — the escape hatch if the Space's proxy/sleep misbehaves. LONG_POLL_S = 25.0 # LeRobot dataset generation: the fleet gathers the selected tasks' episodes, # has each device push its OWN streams to a shared raw dataset (by role), then # triggers a processing Space that converts raw → LeRobot (mono or bimanual, # per the device set). Overridable so the test Space can be targeted. LEROBOT_SPACE_URL = os.environ.get( "GRABETTE_y.LEROBOT_SPACE_URL", "https://pollen-robotics-grabette-slam-bimanual.hf.space" ).rstrip("/") # Lead time before a group's synchronized start actually fires. The device # warms its hardware DURING this lead, then waits out the shared T0 on its own # NTP-disciplined clock. The lead must cover (poll delivery + hardware warmup) # so warmup finishes BEFORE T0 — otherwise the variable warmup leaks into the # start and the devices drift apart. Warmup is only long when the OAK-D is cold # (multi-second cold boot); once it has recorded recently it stays warm (device # keepalive), so we use two leads: # Must cover (poll delivery to the peer ~2.5s + OAK-D cold boot). Measured: a # peer overran a 12s lead by ~0.7s (cold boot ≈10.2s, but it only had ~9.5s # after poll delivery), so 15s gives ~2.3s margin over the observed cold boot. GROUP_START_LEAD_COLD_S = 15.0 # OAK-D likely asleep → cover poll delivery + cold boot # Even when warm, the lead must exceed the WORST-CASE command delivery to a # peer: a device polls only every ~2.5s AND its relay is blocked while it muxes # the previous episode's stop (several seconds), so a too-short warm lead makes # the peer receive its start_capture after T0 (→ started late, best-effort — it # no longer drops the episode, but it's desynced). 3s covers the common case; # the real cure for back-to-back on-time starts is non-blocking relay muxing. GROUP_START_LEAD_WARM_S = 1.0 # Fleet can't see the OAK-D power state directly; it infers "warm" from the time # since the session's last recording stop. This window MUST stay safely below # the device's OAK-D keepalive (GRABETTE oakd_keepalive_s, default 30s) so that # whenever fleet says "warm" the OAK-D is DEFINITELY still powered — a false # "warm" would desync the start, a false "cold" only costs an unnecessarily long # lead. Fleet's stop timestamp is the dispatch time, earlier than the device's # actual keepalive countdown start, which makes the estimate extra conservative. OAK_WARM_WINDOW_S = 25.0 # Lead for a lead-based SYNCHRONIZED stop (shared future T_stop fanned to every # member so they end together). CURRENTLY UNUSED: stops are dispatched # immediately (see _dispatch_episode_stop) so a button press feels instant and # peers trail only by the ~1s poll delivery. Kept here (with the device-side # CaptureScheduler.schedule_stop path) so lead-based sync can be re-enabled # without re-plumbing — e.g. if delivery latency ever grew. To switch back: # have _dispatch_episode_stop send args={"stop_at_utc": now + GROUP_STOP_LEAD_S}. GROUP_STOP_LEAD_S = 3.0 # --- device identity (Bearer token -> HF username, cached) ------------------- _whoami_cache: dict[str, tuple[str, float]] = {} # Derived namespace list per owner (username + orgs) for the dataset owner # dropdown — NOT the raw token. Device tokens list orgs reliably (unlike the # short-lived operator OAuth token), so we compute the list from a device token # when one passes through, cache the RESULT, and discard the token. In a shared # Space this avoids holding every user's write-capable token in memory. # owner -> (expiry_epoch, [namespaces]). _namespaces_cache: dict[str, tuple[float, list[str]]] = {} _NAMESPACES_TTL = 600.0 def _cached_user(token: str) -> Optional[str]: """Fast path: no I/O. Returns the cached username if still fresh.""" hit = _whoami_cache.get(token) if hit and hit[1] > time.time(): return hit[0] return None def _whoami_blocking(token: str) -> str: """Slow path: hits the HF API. Synchronous — the huggingface_hub client has no async variant, so this must always be run off the event loop thread (see verify_user) or a single cache-miss stalls every other request the whole broker is serving (all owners' polling, dispatch, and the synchronized-start scheduling), since this process runs a single worker with all state in memory. """ try: info = whoami(token=token) except Exception as e: # noqa: BLE001 raise ValueError(f"Invalid HF token: {e}") from e name = (info or {}).get("name", "") if not name: raise ValueError("Could not resolve HF username") _whoami_cache[token] = (name, time.time() + WHOAMI_TTL) return name async def verify_user(token: str) -> str: cached = _cached_user(token) if cached is not None: return cached return await asyncio.to_thread(_whoami_blocking, token) async def device_auth(authorization: Optional[str] = Header(None)) -> tuple[str, str]: """FastAPI dep: resolve (owner, token) from a device's Bearer HF token. Returns the raw token too — handlers use it transiently within the request (e.g. dispatching a device's own Space call). It is deliberately NOT retained anywhere on the fleet: in a shared Space that would concentrate every user's write-capable HF token in one process. Owner identity is resolved via a cached whoami; the operator's namespace list is derived + cached separately (see _touch_namespaces) so no raw token needs to be kept. """ if not authorization or not authorization.startswith("Bearer "): raise HTTPException(401, "Missing 'Authorization: Bearer '") token = authorization[len("Bearer ") :].strip() try: owner = await verify_user(token) except ValueError as e: raise HTTPException(401, str(e)) from e return owner, token async def device_owner(authorization: Optional[str] = Header(None)) -> str: """FastAPI dep: resolve just the device's owner from its Bearer HF token.""" owner, _token = await device_auth(authorization) return owner def _cache_namespaces_blocking(owner: str, token: str) -> None: """Blocking: resolve the owner's pushable namespaces (username + orgs) from a device token and cache the RESULT — never the token itself.""" names = [owner] try: info = whoami(token=token) names += [o["name"] for o in (info.get("orgs") or []) if o.get("name")] except Exception: logger.debug("namespaces whoami failed for %s", owner, exc_info=True) _namespaces_cache[owner] = (time.time() + _NAMESPACES_TTL, list(dict.fromkeys(names))) def _touch_namespaces(owner: str, token: Optional[str]) -> None: """TTL-gated, fire-and-forget: refresh the owner's namespace list from a passing device token, so the dataset-owner dropdown works WITHOUT the fleet keeping the token. Marks the cache fresh up front to avoid a refresh stampede.""" if not token: return ent = _namespaces_cache.get(owner) if ent and ent[0] > time.time(): return # still fresh _namespaces_cache[owner] = (time.time() + _NAMESPACES_TTL, ent[1] if ent else [owner]) t = asyncio.create_task(asyncio.to_thread(_cache_namespaces_blocking, owner, token)) _bg_tasks.add(t) t.add_done_callback(_bg_tasks.discard) def operator_auth(request: Request) -> tuple[str, Optional[str]]: """Resolve (owner, oauth_access_token) for the logged-in operator, or 401. The OAuth token identifies the operator (their namespace). The fleet keeps no durable state of its own — the task/episode view is aggregated live from connected devices' reports (see _reported_tasks).""" info = parse_huggingface_oauth(request) if info is None: raise HTTPException(401, "Not logged in") owner = info.user_info.preferred_username or info.user_info.name return owner, getattr(info, "access_token", None) def operator_name(request: Request) -> str: """Resolve the logged-in operator's HF username, or 401.""" return operator_auth(request)[0] # --- in-memory fleet state --------------------------------------------------- @dataclass class Command: id: str type: str args: dict[str, Any] status: str = "pending" # pending | sent | done result: Optional[dict[str, Any]] = None created_at: float = field(default_factory=time.time) done_at: Optional[float] = None @dataclass class Device: device_id: str name: str capabilities: list[str] hand: str = "" # "left" or "right", reported by the device on register ip: str = "" # device LAN IPv4, reported by the device on register battery: Optional[float] = None # last-reported battery %, sent via heartbeat # Device self-reported activity, sent on the heartbeat: # "" (device not updated / unknown) | idle | capturing | uploading | processing. # Empty falls back to fleet-side inference (see _device_activity). reported_status: str = "" # This device's recorded tasks, reported on register (see TaskManager.report_tasks). # The device is the durable source of truth; the fleet aggregates these across # connected devices (phase 2). Stored but not yet consumed in phase 1. tasks: list[dict] = field(default_factory=list) # Bumped each register (when tasks are (re)reported). Feeds the aggregation # cache signature so it recomputes only when a device's report changed. report_rev: int = 0 last_seen: float = field(default_factory=time.time) queue: list[Command] = field(default_factory=list) history: list[Command] = field(default_factory=list) pending_delete: bool = False # Set whenever a command is enqueued (see _enqueue) to release a long-poll # holding on this device. Excluded from repr/eq — it's live runtime state, # never serialized (devices aren't persisted). asyncio.Event() binds to a # loop lazily (on first await), so constructing it off-loop here is fine. wakeup: asyncio.Event = field(default_factory=asyncio.Event, repr=False, compare=False) @property def online(self) -> bool: return (time.time() - self.last_seen) < ONLINE_WINDOW FLEET: dict[str, dict[str, Device]] = {} # owner -> device_id -> Device def _fleet_of(owner: str) -> dict[str, Device]: return FLEET.setdefault(owner, {}) def kind_of(dev: Device) -> str: """Mirrors the frontend's kindOf(): classify a device as grabette/gripette/casquette.""" s = f"{dev.name} {dev.device_id} {' '.join(dev.capabilities)}".lower() if "gripette" in s: return "gripette" if "casquette" in s: return "casquette" return "grabette" def _device_slot(dev: Device) -> Optional[str]: """The task-signature role this device fills: 'left'/'right' for a handed grabette, 'casquette' for a casquette, else None (not recordable).""" k = kind_of(dev) if k == "casquette": return "casquette" if k == "grabette" and dev.hand in ("left", "right"): return dev.hand return None # --- tasks (registry, source of truth for task names sent to devices) -------- # A task is a *type of action*. It carries only the device *roles* it expects # (its "signature"), never concrete device ids — those live on the group. The # task NAME is the stable join key devices resolve locally via # get_or_create_task, so it must come from this single registry. VALID_SLOTS = ("left", "right", "casquette") @dataclass class Task: id: str name: str description: str = "" # Expected device roles, a subset of VALID_SLOTS (e.g. ["left","right"] for # a bimanual task, ["right"] for a single right-hand one). Empty = no # constraint. Used to validate the group assigned to the task and to tell # the dataset builder which roles produce data. device_signature: list[str] = field(default_factory=list) created_at: float = field(default_factory=time.time) TASKS: dict[str, dict[str, Task]] = {} # owner -> task_id -> Task def _tasks_of(owner: str) -> dict[str, Task]: return TASKS.setdefault(owner, {}) # --- task/episode aggregation from device reports --------------------------- # The devices are the durable source of truth for tasks and for who recorded each # episode. On connect each device reports its tasks (see TaskManager.report_tasks); # the fleet merges the reports from currently-connected devices to reconstruct the # task/episode view — so ANY operator sees the existing tasks and can generate a # dataset from them, regardless of which HF account did the original acquisition. # Task names to hide briefly after an edit(rename)/delete, until the devices have # processed the command and re-reported without them. Bridges the window where a # stale report would otherwise resurrect a just-deleted/renamed task. owner -> # {name: expiry_epoch}. TASK_SUPPRESSED: dict[str, dict[str, float]] = {} _SUPPRESS_S = 20.0 # Grace after a session stops before its episodes can be considered "orphaned": # covers the window where one member has registered + re-reported the episode but # the peer hasn't yet, so an unfinished recording isn't mistaken for a lost pair. ORPHAN_GRACE_S = 30.0 # Orphaned-episode reconciliation is EVENT-DRIVEN: recomputed when a device # registers (connects / re-reports its tasks) — the only moments orphans change — # and cached here, so the operator UI just reads the cache instead of forcing a # recompute on every poll. owner -> list of orphan-episode dicts. ORPHANS_PENDING: dict[str, list[dict]] = {} def _suppress_task_name(owner: str, name: str) -> None: TASK_SUPPRESSED.setdefault(owner, {})[name] = time.time() + _SUPPRESS_S def _suppressed_names(owner: str) -> set[str]: now = time.time() supp = TASK_SUPPRESSED.get(owner) if not supp: return set() # Prune expired so a name can come back if it's ever legitimately re-created. for n in [n for n, exp in supp.items() if exp <= now]: del supp[n] return set(supp) def _devices_reporting_task(owner: str, name: str) -> list["Device"]: """Online devices whose latest report includes a task with this name.""" out = [] for dev in _fleet_of(owner).values(): if dev.online and any((t.get("name") == name) for t in (dev.tasks or [])): out.append(dev) return out def _task_episode_items(t: dict): """Yield (episode_id, members) for a reported task, supporting the compact 'groups' format (episodes grouped by shared membership) and the legacy per-episode 'episodes' format (older device firmware).""" if "groups" in t: for grp in t.get("groups", []): members = grp.get("members") or {} for eid in grp.get("episode_ids", []): if eid: yield eid, members else: for ep in t.get("episodes", []): eid = ep.get("episode_id") if eid: yield eid, (ep.get("members") or {}) # Aggregation is memoized per owner: recomputing it (O(all reported episodes)) # on every dashboard poll doesn't scale. A cheap signature over the owner's # devices (online status + a per-device report revision + suppressed names) # tells us when anything that feeds the aggregation actually changed; otherwise # we return the cached result. owner -> (signature, aggregation). _REPORTED_CACHE: dict[str, tuple] = {} def _reported_sig(owner: str): devs = _fleet_of(owner) return (tuple(sorted((d.device_id, d.online, d.report_rev) for d in devs.values())), tuple(sorted(_suppressed_names(owner)))) def _compute_reported_tasks(owner: str) -> dict[str, dict]: agg: dict[str, dict] = {} suppressed = _suppressed_names(owner) # just-edited/deleted names, hidden briefly for dev in _fleet_of(owner).values(): if not dev.online: continue for t in dev.tasks or []: name = t.get("name") if not name or name in suppressed: continue entry = agg.setdefault(name, {"description": "", "device_signature": [], "episodes": {}}) if t.get("description") and not entry["description"]: entry["description"] = t["description"] if t.get("device_signature") and not entry["device_signature"]: entry["device_signature"] = list(t["device_signature"]) for eid, ep_members in _task_episode_items(t): members = entry["episodes"].setdefault(eid, {}) for role, who in ep_members.items(): members.setdefault(role, who) return agg def _reported_tasks(owner: str) -> dict[str, dict]: """Merge online devices' reported tasks, keyed by task name (memoized). The SAME episode is reported by every member device (same episode_id + members), so we dedup by episode_id and union members. Returns {name: {"description", "device_signature", "episodes": {episode_id: {role: {device_id, name}}}}}. The result is cached and must be treated as READ-ONLY by callers.""" sig = _reported_sig(owner) cached = _REPORTED_CACHE.get(owner) if cached is not None and cached[0] == sig: return cached[1] agg = _compute_reported_tasks(owner) _REPORTED_CACHE[owner] = (sig, agg) return agg def _episode_started_at(episode_id: str) -> float: """Best-effort epoch seconds from an episode id (UTC 'YYYYmmdd_HHMMSS').""" try: return datetime.strptime(episode_id, "%Y%m%d_%H%M%S").replace(tzinfo=timezone.utc).timestamp() except ValueError: return 0.0 def _episodes_from_entry(entry: Optional[dict]) -> list[dict]: """Normalise a _reported_tasks entry's episodes into a sorted list carrying both members (role → {device_id, name}) and a plain roles map (role → id).""" if not entry: return [] out = [] for eid, members in entry["episodes"].items(): out.append({ "episode_id": eid, "members": members, "roles": {role: who.get("device_id") for role, who in members.items()}, "started_at": _episode_started_at(eid), }) out.sort(key=lambda e: e["episode_id"]) return out def _reconcile_tasks(owner: str) -> None: """Ensure a Task exists locally for every task reported by connected devices, so a freshly-connected operator's registry is populated from the devices (the source of truth). Purely in-memory: the fleet persists nothing, so this is rebuilt from the reports on every restart.""" tasks = _tasks_of(owner) by_name = {t.name: t for t in tasks.values()} for name, entry in _reported_tasks(owner).items(): t = by_name.get(name) if t is None: tid = uuid.uuid4().hex[:8] tasks[tid] = Task(id=tid, name=name, description=entry["description"], device_signature=list(entry["device_signature"])) else: # Backfill from the authoritative device report where we're missing it. if not t.device_signature and entry["device_signature"]: t.device_signature = list(entry["device_signature"]) if not t.description and entry["description"]: t.description = entry["description"] def _orphan_episodes(owner: str) -> list[dict]: """Episodes that lost a pair — detected purely from the live reports, no tombstone. An episode is orphaned when one of its member devices is ONLINE but no longer reports it (it deleted the episode / task), while another online member still holds it. This is what surfaces after a multi-device task was deleted with a peer offline: on reconnect, the peer's copies are seen as orphaned (their now-online partner deleted them) and cleanup is proposed. Episodes whose missing member is OFFLINE are NOT flagged — we can't tell if that device still has them. Just-deleted (suppressed) task names are skipped so a normal both-online delete doesn't flash transient orphans mid-teardown. Returns one entry per orphaned episode.""" fleet = _fleet_of(owner) suppressed = _suppressed_names(owner) online = {d.device_id: d for d in fleet.values() if d.online} # Episodes still "in flux" — being recorded now, or just stopped — must NOT be # flagged: one member registers + re-reports before the other, so for a few # seconds it exists on one device and not (yet) the peer, which looks orphaned # but is just an unfinished recording. Skip any episode in an open session, or # in a session stopped within the grace window (covers the re-report lag). now = time.time() in_flux: set[str] = set() for s in _sessions_of(owner).values(): if s.status == "open" or (s.last_stop_at is not None and now - s.last_stop_at < ORPHAN_GRACE_S): for ep in s.episodes: eid = ep.get("episode_id") if eid: in_flux.add(eid) dev_eps: dict[str, set[str]] = {} # device_id -> episode ids it reports ep_info: dict[str, dict] = {} # episode_id -> {members, task} for dev in online.values(): eids: set[str] = set() for t in dev.tasks or []: name = t.get("name") if not name or name in suppressed: continue for eid, ep_members in _task_episode_items(t): eids.add(eid) ep_info.setdefault(eid, {"members": ep_members, "task": name}) dev_eps[dev.device_id] = eids def _name(mid: str, fallback: str) -> str: return online[mid].name if mid in online else (fallback or mid) out = [] for eid, info in ep_info.items(): if eid in in_flux: continue # being recorded / just stopped — not a real orphan member_ids = {w.get("device_id"): w.get("name") for w in info["members"].values() if w.get("device_id")} deleters = [(m, _name(m, n)) for m, n in member_ids.items() if m in online and eid not in dev_eps.get(m, set())] holders = [(m, _name(m, n)) for m, n in member_ids.items() if m in online and eid in dev_eps.get(m, set())] if deleters and holders: # a peer deleted it, but someone online still has it out.append({ "episode_id": eid, "task": info["task"], "started_at": _episode_started_at(eid), "holders": [{"device_id": m, "name": n} for m, n in holders], "deleted_by": [{"device_id": m, "name": n} for m, n in deleters], }) return out def _task_name(owner: str, task_id: str) -> str: t = _tasks_of(owner).get(task_id) return t.name if t else "" @dataclass class Group: id: str name: str left: str = "" # device_id of the left-hand grabette in this group right: str = "" # device_id of the right-hand grabette in this group casquette: str = "" # device_id of the casquette in this group created_at: float = field(default_factory=time.time) # (No task here — the task is chosen when a session is launched, not stored # on the group. See Session.) GROUPS: dict[str, dict[str, Group]] = {} # owner -> group_id -> Group def _groups_of(owner: str) -> dict[str, Group]: return GROUPS.setdefault(owner, {}) # --- sessions (fleet-only: the recording-run + upload manifest) -------------- # A session is a run of one or more episodes recorded together for ONE task. # It carries its own role→device membership (a "group of one" is just a # single-entry map — no Group object is materialised) and the per-episode # manifest: which physical device held which role for each episode. That # manifest is what lets the dataset builder regroup an episode's data even # when the device filling a role changes between episodes. @dataclass class Session: id: str task_id: str members: dict[str, str] # role -> device_id, captured at launch status: str = "open" # open | closed # Optimistic recording indicator: True between an episode start and its # stop. Maintained on every start/stop path (operator + physical button). # Not a hardware truth (real per-device state is reconciled in phase 3) — # enough to drive the "● recording" indicator in the UI. recording: bool = False # Epoch time of the last episode stop dispatched for this session. Drives # the warm/cold lead decision: within OAK_WARM_WINDOW_S the OAK-D is still # powered (device keepalive), so the next episode can use the short lead. # None until the first episode stops → first episode uses the cold lead. last_stop_at: Optional[float] = None # "stopping" phase, reconciled against reality: set True when an episode stop # is dispatched, and cleared when every dispatched stop_capture command has # reported its result (i.e. the devices actually finished tearing down + mux) # — so the UI's "stopping" indicator ends when the devices really stop, not # after a fixed guess. pending_stop_cmds holds the command ids still awaited. stopping: bool = False pending_stop_cmds: set[str] = field(default_factory=set) # Each entry: {"episode_id": str, "roles": {role: device_id}, "started_at": float} episodes: list[dict[str, Any]] = field(default_factory=list) started_at: float = field(default_factory=time.time) SESSIONS: dict[str, dict[str, Session]] = {} # owner -> session_id -> Session @dataclass class DatasetJob: """A LeRobot dataset build. Transient runtime state (not persisted): the fleet dispatches per-device uploads, waits for them, then (step 3) triggers the processing Space and tracks it to completion.""" id: str task_ids: list[str] roles: list[str] # the shared device signature of the selected tasks raw_repo: str # {owner}/grabette-raw- — deleted after processing target_repo: str # {owner}/ — the resulting LeRobot dataset status: str = "uploading" # uploading | raw_ready | processing | done | error message: str = "" # 0..1 fraction for the determinate upload phase; None once processing starts # (the Space conversion runs opaquely behind one blocking device call, so the # UI shows an indeterminate bar there). 1.0 when done. progress: Optional[float] = 0.0 result_url: Optional[str] = None error: Optional[str] = None upload_cmds: dict[str, str] = field(default_factory=dict) # device_id -> command id created_at: float = field(default_factory=time.time) DATASET_JOBS: dict[str, dict[str, DatasetJob]] = {} # owner -> job_id -> job def _dataset_jobs_of(owner: str) -> dict[str, DatasetJob]: return DATASET_JOBS.setdefault(owner, {}) def _sessions_of(owner: str) -> dict[str, Session]: return SESSIONS.setdefault(owner, {}) def _open_session_for_device(owner: str, device_id: str) -> Optional[Session]: for s in _sessions_of(owner).values(): if s.status == "open" and device_id in s.members.values(): return s return None # --- device activity: what a device is doing, for the operator UI + the recording # gates. Prefer what the fleet can INFER on its own from the work it dispatched (an # in-flight upload/process command in the device's queue); then the device's own # self-reported status (which also covers dashboard-initiated work the fleet never # sees); then capture inferred from open-session membership. Pure in-memory reads, # no extra I/O, so it stays cheap at fleet scale. _RECORDING_BLOCKERS = ("uploading", "processing") # dataset work that must not overlap a recording def _device_activity(owner: str, dev: Device) -> str: """One of: idle | capturing | uploading | processing.""" # Dataset work the fleet ITSELF dispatched — authoritative, and the device may # not self-report it (relay commands don't create a local job). A command is # removed from the queue on result, so anything left here is live. types = {c.type for c in dev.queue} if "process_dataset" in types: return "processing" if "upload_episodes" in types: return "uploading" # Device self-reported activity — covers local dashboard work (SLAM push / # episode upload) and live capture that the fleet can't see on its own. st = (dev.reported_status or "").strip().lower() if st in ("capturing", "uploading", "processing"): return st # Capture inferred from open-session membership (fallback if not reported). if _open_session_for_device(owner, dev.device_id) is not None: return "capturing" return "idle" def _busy_recording_blockers(owner: str, device_ids) -> list[str]: """Subset of device_ids currently tied up by dataset work (upload/convert), which must be free before they can (re)start a recording.""" fleet = _fleet_of(owner) return [did for did in device_ids if (d := fleet.get(did)) is not None and _device_activity(owner, d) in _RECORDING_BLOCKERS] def _remove_from_groups(owner: str, device_id: str) -> None: for g in _groups_of(owner).values(): if g.left == device_id: g.left = "" if g.right == device_id: g.right = "" if g.casquette == device_id: g.casquette = "" def _validate_task_signature(owner: str, task_id: str, slots: set[str]) -> None: """The devices launched for a task must fill exactly the task's required roles — exact match (not just superset) so the dataset builder knows precisely which roles produce data. No-op when the task has no signature.""" t = _tasks_of(owner).get(task_id) if t is None: raise HTTPException(404, f"Task {task_id} not found") if t.device_signature and set(t.device_signature) != slots: raise HTTPException(400, { "message": "selected devices don't match the task's required devices", "task": t.name, "required": sorted(t.device_signature), "got": sorted(slots), }) # --- persistence: none ------------------------------------------------------- # The fleet keeps NO durable state of its own. Devices are the source of truth: # each re-registers on connect and reports its tasks + per-episode membership # (see TaskManager.report_tasks), which the fleet aggregates in memory (see # _reported_tasks / _reconcile_tasks). This means ANY operator sees the existing # tasks and can build a dataset from them, regardless of which HF account did the # acquisition — and nothing is ever written to an HF namespace on the fleet's # behalf. Groups and open sessions are runtime-only and reset on a Space restart. # Strong refs to fire-and-forget background tasks (e.g. dataset jobs). asyncio # only weakly references tasks, so without this the GC can cancel one mid-run. _bg_tasks: set[asyncio.Task] = set() def _enqueue(dev: "Device", cmd: Command) -> None: """Queue a command for a device AND wake any long-poll holding on it, so the command is delivered on the next network round-trip instead of the next poll interval. Centralized on purpose: a bare dev.queue.append() would silently strand the command for up to LONG_POLL_S until the hold times out.""" dev.queue.append(cmd) dev.wakeup.set() async def _operator_loaded(request: Request) -> str: """Resolve the authenticated operator (namespace). Kept as the single dependency every operator-facing handler uses; there is no snapshot to load anymore — the task/episode view is aggregated live from device reports.""" owner, _token = operator_auth(request) return owner # --- app + OAuth ------------------------------------------------------------- app = FastAPI(title="Grabette fleet") attach_huggingface_oauth(app) # adds /oauth/huggingface/{login,logout,callback} @app.get("/healthz") async def healthz() -> dict[str, str]: """Public, auth-free liveness ping. A device hits this to WAKE the (free-tier, sleep-when-idle) Space and confirm it's up before starting OAuth — so the OAuth callback, which HF routes through this Space, lands on a running relay even when no operator has a fleet dashboard tab open to keep it warm.""" return {"status": "ok"} # === device-facing API (Bearer auth) ======================================== class RegisterReq(BaseModel): device_id: str name: str = "" capabilities: list[str] = [] hand: str = "" # "left" or "right" (from GRABETTE_HAND on the device) ip: str = "" # device LAN IPv4, best-effort tasks: list[dict] = [] # this device's recorded tasks (source of truth for aggregation) class ResultReq(BaseModel): device_id: str command_id: str result: dict[str, Any] = {} @app.post("/api/devices/register") async def register(req: RegisterReq, auth: tuple[str, str] = Depends(device_auth)) -> dict[str, Any]: owner, token = auth _touch_namespaces(owner, token) # refresh the owner's org list (token not kept) fleet = _fleet_of(owner) dev = fleet.get(req.device_id) if dev is None: dev = Device(req.device_id, req.name or req.device_id, req.capabilities, hand=req.hand, ip=req.ip, tasks=req.tasks) fleet[req.device_id] = dev else: dev.name = req.name or dev.name dev.capabilities = req.capabilities or dev.capabilities dev.hand = req.hand or dev.hand dev.ip = req.ip or dev.ip # Reported every register: the device is authoritative, so replace wholesale. dev.tasks = req.tasks dev.report_rev += 1 # invalidates the aggregation cache for this owner dev.last_seen = time.time() # A device just (re)reported its tasks — the moment orphaned episodes can # appear/change. Recompute the reconciliation snapshot now (event-driven) # rather than on every operator poll. ORPHANS_PENDING[owner] = _orphan_episodes(owner) return {"status": "ok", "pending": len(dev.queue)} @app.post("/api/devices/heartbeat") async def heartbeat(device_id: str, battery: Optional[float] = None, status: Optional[str] = None, auth: tuple[str, str] = Depends(device_auth)) -> dict[str, str]: """Lightweight liveness ping, sent every DEVICE_HEARTBEAT_S independently of the (long-held) command poll — this is what keeps last_seen fresh and lets the fleet detect a disconnect within ONLINE_WINDOW. Deliberately does no HF I/O so it stays cheap at a few-second cadence. Also carries the device's battery % and its self-reported activity, so the fleet can show device state without polling.""" owner, _token = auth dev = _fleet_of(owner).get(device_id) if dev is None: raise HTTPException(404, "Device not registered") dev.last_seen = time.time() if battery is not None: dev.battery = battery if status is not None: dev.reported_status = status return {"status": "ok"} @app.get("/api/devices/poll") async def poll(device_id: str, auth: tuple[str, str] = Depends(device_auth)) -> dict[str, Any]: owner, token = auth dev = _fleet_of(owner).get(device_id) if dev is None: raise HTTPException(404, "Device not registered") _touch_namespaces(owner, token) # keeps the org list fresh (TTL-gated; token not kept) dev.last_seen = time.time() # Long-poll: hold the connection open until a command is enqueued (which # sets dev.wakeup) or LONG_POLL_S elapses, so a command is delivered on the # next round-trip instead of the next poll interval. Clear the event FIRST, # then re-check the queue: an enqueue landing in that gap sets the event, so # the wait() returns immediately rather than stranding the command. When # LONG_POLL_S<=0 this whole block is skipped → short-poll (return at once). if LONG_POLL_S > 0: dev.wakeup.clear() if not any(c.status == "pending" for c in dev.queue): try: await asyncio.wait_for(dev.wakeup.wait(), timeout=LONG_POLL_S) except asyncio.TimeoutError: pass # NB: do NOT refresh last_seen here — a held poll that times out on a # DEAD device would otherwise look like fresh contact and delay # offline detection. last_seen reflects the poll's ARRIVAL (above); # liveness is kept fresh by the separate lightweight heartbeat. pending = [c for c in dev.queue if c.status == "pending"] for c in pending: c.status = "sent" return {"commands": [{"id": c.id, "type": c.type, "args": c.args} for c in pending]} @app.post("/api/devices/result") async def result(req: ResultReq, auth: tuple[str, str] = Depends(device_auth)) -> dict[str, str]: owner, _token = auth dev = _fleet_of(owner).get(req.device_id) if dev is None: raise HTTPException(404, "Device not registered") for c in dev.queue: if c.id == req.command_id: c.status = "done" c.result = req.result c.done_at = time.time() dev.queue.remove(c) dev.history.insert(0, c) del dev.history[20:] if c.type == "logout" and dev.pending_delete: fleet = _fleet_of(owner) fleet.pop(dev.device_id, None) _remove_from_groups(owner, dev.device_id) break # A stop_capture result means that device actually finished tearing down its # capture. End the session's "stopping" phase once EVERY dispatched stop has # reported — so the UI leaves "stopping" exactly when the devices really stop. for s in _sessions_of(owner).values(): if req.command_id in s.pending_stop_cmds: s.pending_stop_cmds.discard(req.command_id) if not s.pending_stop_cmds: s.stopping = False break return {"status": "ok"} # === operator-facing API (session OAuth) ===================================== class DispatchReq(BaseModel): device_id: str type: str args: dict[str, Any] = {} @app.get("/api/fleet/me") async def me(request: Request) -> dict[str, Any]: info = parse_huggingface_oauth(request) if info is None: return {"logged_in": False} return {"logged_in": True, "username": info.user_info.preferred_username or info.user_info.name} @app.get("/api/fleet/namespaces") async def namespaces(request: Request) -> dict[str, Any]: """The namespaces the operator can push a dataset to: their own username + the orgs they belong to. Used to populate the dataset owner dropdown. Served from the cache built when a device token passed through (device tokens list orgs reliably; see _touch_namespaces) — so the fleet keeps no raw token. Cold start (no device seen yet for this owner): best-effort via the operator's OAuth token, which lists orgs poorly, falling back to just the username.""" owner, oauth_token = operator_auth(request) ent = _namespaces_cache.get(owner) if ent and ent[1]: return {"namespaces": ent[1], "default": owner} names = [owner] if oauth_token: try: info = await asyncio.to_thread(whoami, oauth_token) names += [o["name"] for o in (info.get("orgs") or []) if o.get("name")] except Exception: logger.warning("whoami for namespaces failed for %s", owner, exc_info=True) seen: set[str] = set() uniq = [n for n in names if not (n in seen or seen.add(n))] return {"namespaces": uniq, "default": owner} @app.get("/api/fleet/devices") async def list_devices(request: Request) -> dict[str, Any]: owner = operator_name(request) devices = list(_fleet_of(owner).values()) now = time.time() for d in devices: # Only drop stale commands for a LONG-gone device — never on the mere # 15s online window (would nuke an in-flight group start/stop). if now - d.last_seen > STALE_QUEUE_S: d.queue.clear() return { "owner": owner, "devices": [ { "device_id": d.device_id, "name": d.name, "online": d.online, "capabilities": d.capabilities, "hand": d.hand, "ip": d.ip, "battery": d.battery, "activity": _device_activity(owner, d), "pending": len(d.queue), "history": [ {"type": c.type, "args": c.args, "result": c.result, "ts": c.done_at or c.created_at} for c in d.history[:5] ], } for d in devices ], } @app.delete("/api/fleet/devices/{device_id}") async def remove_device(device_id: str, request: Request) -> dict[str, str]: owner = await _operator_loaded(request) fleet = _fleet_of(owner) if device_id not in fleet: raise HTTPException(404, "Device not found") del fleet[device_id] _remove_from_groups(owner, device_id) return {"status": "ok"} @app.post("/api/fleet/devices/{device_id}/remove") async def remove_device_graceful(device_id: str, request: Request) -> dict[str, str]: """Remove a device: dispatch logout if online (then delete on result), delete immediately if offline.""" owner = await _operator_loaded(request) fleet = _fleet_of(owner) dev = fleet.get(device_id) if dev is None: raise HTTPException(404, "Device not found") if dev.online: dev.pending_delete = True cmd = Command(id=uuid.uuid4().hex[:12], type="logout", args={}) _enqueue(dev, cmd) else: fleet.pop(device_id, None) _remove_from_groups(owner, device_id) return {"status": "ok"} # === tasks (operator-facing, session OAuth) ================================== class TaskReq(BaseModel): name: str = "" description: str = "" device_signature: list[str] = [] def _task_dict(t: Task) -> dict[str, Any]: return {"id": t.id, "name": t.name, "description": t.description, "device_signature": t.device_signature} @app.get("/api/fleet/tasks") async def list_tasks(request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) _reconcile_tasks(owner) # surface tasks reported by connected devices agg = _reported_tasks(owner) out = [] for t in _tasks_of(owner).values(): d = _task_dict(t) eps = _episodes_from_entry(agg.get(t.name)) d["episodes"] = eps d["episode_count"] = len(eps) out.append(d) return {"tasks": out} def _clean_signature(sig: list[str]) -> list[str]: bad = [s for s in sig if s not in VALID_SLOTS] if bad: raise HTTPException(400, f"Invalid device_signature entries: {bad}; allowed: {list(VALID_SLOTS)}") # de-dup, canonical order cleaned = [s for s in VALID_SLOTS if s in sig] if not cleaned: raise HTTPException(400, "A task must require at least one device") return cleaned @app.post("/api/fleet/tasks") async def create_task(req: TaskReq, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) if not req.name.strip(): raise HTTPException(400, "Task needs a name") tasks = _tasks_of(owner) if any(t.name == req.name.strip() for t in tasks.values()): raise HTTPException(400, "A task with this name already exists") tid = uuid.uuid4().hex[:8] tasks[tid] = Task(id=tid, name=req.name.strip(), description=req.description.strip(), device_signature=_clean_signature(req.device_signature)) return {"status": "ok", "id": tid, "task": _task_dict(tasks[tid])} @app.put("/api/fleet/tasks/{task_id}") async def update_task(task_id: str, req: TaskReq, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) _reconcile_tasks(owner) t = _tasks_of(owner).get(task_id) if t is None: raise HTTPException(404, "Task not found") old_name = t.name new_name = req.name.strip() or old_name new_desc = req.description.strip() new_sig = _clean_signature(req.device_signature) # The device signature is locked ONCE it's known and episodes exist — the # recordings were made with that device set, so changing it would misdescribe # them. But an old task with episodes and NO recorded signature can be set # once (to backfill it); after that it's known → locked. has_episodes = bool(_episodes_from_entry(_reported_tasks(owner).get(old_name))) sig_locked = has_episodes and bool(t.device_signature) if sig_locked and sorted(new_sig) != sorted(t.device_signature): raise HTTPException(409, {"message": "cannot change the required devices of a task " "that already has recorded episodes"}) if new_name != old_name and any(o.name == new_name for o in _tasks_of(owner).values()): raise HTTPException(400, "A task with this name already exists") sig_changed = (not sig_locked) and sorted(new_sig) != sorted(t.device_signature) # Apply on the fleet, then propagate to every device that recorded this task # (the source of truth) so the change survives the next report reconcile. t.name = new_name t.description = new_desc if not sig_locked: t.device_signature = new_sig for dev in _devices_reporting_task(owner, old_name): args = {"name": old_name, "new_name": new_name if new_name != old_name else None, "description": new_desc} if sig_changed: # backfilling an old task's signature → devices must store it args["device_signature"] = new_sig _enqueue(dev, Command(id=uuid.uuid4().hex[:12], type="edit_task", args=args)) if new_name != old_name: _suppress_task_name(owner, old_name) # hide stale old-name reports until re-report return {"status": "ok", "task": _task_dict(t)} @app.delete("/api/fleet/tasks/{task_id}") async def delete_task(task_id: str, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) _reconcile_tasks(owner) tasks = _tasks_of(owner) t = tasks.get(task_id) if t is None: raise HTTPException(404, "Task not found") if any(s.status == "open" and s.task_id == task_id for s in _sessions_of(owner).values()): raise HTTPException(409, "Cannot delete a task with an open session") name = t.name # Delete on every ONLINE device that has this task — the task AND its recorded # episodes go (the device is the source of truth; removing here alone would let # the next report resurrect it). Offline members keep their copy for now; they # are reconciled when they reconnect (see orphan detection, phase 3). dispatched = [] for dev in _devices_reporting_task(owner, name): _enqueue(dev, Command(id=uuid.uuid4().hex[:12], type="delete_task", args={"name": name})) dispatched.append(dev.device_id) del tasks[task_id] _suppress_task_name(owner, name) # hide it until the devices re-report without it return {"status": "ok", "dispatched": dispatched} class OrphanCleanupReq(BaseModel): episode_ids: list[str] = [] @app.get("/api/fleet/orphans") async def list_orphans(request: Request) -> dict[str, Any]: """Orphaned episodes (a peer deleted them but a connected device still holds them), grouped by task for the reconciliation banner. Returns the cached snapshot computed on the last device register — not a per-poll recompute.""" owner = await _operator_loaded(request) reported = _reported_tasks(owner) groups: dict[str, dict] = {} for o in ORPHANS_PENDING.get(owner, []): g = groups.setdefault(o["task"], {"task": o["task"], "episode_ids": [], "holders": {}, "deleted_by": {}}) g["episode_ids"].append(o["episode_id"]) for h in o["holders"]: g["holders"][h["device_id"]] = h["name"] for d in o["deleted_by"]: g["deleted_by"][d["device_id"]] = d["name"] out = [] for g in groups.values(): # If EVERY remaining episode of the task is orphaned, the whole task was # deleted on the peer(s) — say so, rather than just "episodes lost pairs". entry = reported.get(g["task"]) total = len(entry["episodes"]) if entry else len(g["episode_ids"]) out.append({"task": g["task"], "episode_ids": g["episode_ids"], "count": len(g["episode_ids"]), "whole_task": len(g["episode_ids"]) >= total, "holders": list(g["holders"].values()), "deleted_by": list(g["deleted_by"].values())}) return {"groups": out} @app.post("/api/fleet/orphans/cleanup") async def cleanup_orphans(req: OrphanCleanupReq, request: Request) -> dict[str, Any]: """Delete the given orphaned episodes on the devices still holding them. Recomputes orphans so only genuinely-orphaned episodes are touched.""" owner = await _operator_loaded(request) # Recompute live here (accuracy matters for a destructive action). orphans = {o["episode_id"]: o for o in _orphan_episodes(owner)} fleet = _fleet_of(owner) dispatched = 0 cleaned = set() for eid in req.episode_ids: o = orphans.get(eid) if not o: continue for h in o["holders"]: dev = fleet.get(h["device_id"]) if dev is not None and dev.online: _enqueue(dev, Command(id=uuid.uuid4().hex[:12], type="delete_episode", args={"episode_id": eid})) dispatched += 1 cleaned.add(eid) # Drop the cleaned episodes from the cached snapshot so the banner clears at # once; the holders' re-register will confirm on the next report. if cleaned and owner in ORPHANS_PENDING: ORPHANS_PENDING[owner] = [o for o in ORPHANS_PENDING[owner] if o["episode_id"] not in cleaned] return {"status": "ok", "dispatched": dispatched} # === device groups (operator-facing, session OAuth) ========================== class GroupReq(BaseModel): name: str = "" left: str = "" right: str = "" casquette: str = "" def _validate_group_members(owner: str, req: GroupReq, group_id: Optional[str] = None) -> None: fleet = _fleet_of(owner) groups = _groups_of(owner) slots = {"left": req.left, "right": req.right, "casquette": req.casquette} if not any(slots.values()): raise HTTPException(400, "Group needs at least one device") for slot, device_id in slots.items(): if not device_id: continue dev = fleet.get(device_id) if dev is None: raise HTTPException(404, f"Device {device_id} not found") if slot in ("left", "right"): if kind_of(dev) != "grabette" or dev.hand != slot: raise HTTPException(400, f"Device {device_id} is not a {slot}-hand grabette") elif kind_of(dev) != "casquette": raise HTTPException(400, f"Device {device_id} is not a casquette") for gid, g in groups.items(): if gid == group_id: continue if device_id in (g.left, g.right, g.casquette): raise HTTPException(400, f"Device {device_id} is already in another group") @app.get("/api/fleet/groups") async def list_groups(request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) groups = list(_groups_of(owner).values()) return { "groups": [ {"id": g.id, "name": g.name, "left": g.left, "right": g.right, "casquette": g.casquette} for g in groups ] } @app.post("/api/fleet/groups") async def create_group(req: GroupReq, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) _validate_group_members(owner, req) groups = _groups_of(owner) gid = uuid.uuid4().hex[:8] groups[gid] = Group( id=gid, name=req.name.strip() or f"Group {len(groups) + 1}", left=req.left, right=req.right, casquette=req.casquette, ) return {"status": "ok", "id": gid} @app.put("/api/fleet/groups/{group_id}") async def update_group(group_id: str, req: GroupReq, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) groups = _groups_of(owner) g = groups.get(group_id) if g is None: raise HTTPException(404, "Group not found") _validate_group_members(owner, req, group_id=group_id) g.name = req.name.strip() or g.name g.left, g.right, g.casquette = req.left, req.right, req.casquette return {"status": "ok"} @app.delete("/api/fleet/groups/{group_id}") async def delete_group(group_id: str, request: Request) -> dict[str, str]: owner = await _operator_loaded(request) groups = _groups_of(owner) if group_id not in groups: raise HTTPException(404, "Group not found") del groups[group_id] return {"status": "ok"} def _group_members(g: Group) -> list[str]: return [d for d in (g.left, g.right, g.casquette) if d] def _lead_for(session: Session) -> float: """Pick the start lead for the next episode: short if the OAK-D is still warm (a recent episode stopped within OAK_WARM_WINDOW_S), long otherwise (first episode of the session, or the OAK-D has since gone to sleep) so the cold boot finishes before T0. Conservative: unknown/old → cold.""" if session.last_stop_at is not None and (time.time() - session.last_stop_at) < OAK_WARM_WINDOW_S: return GROUP_START_LEAD_WARM_S return GROUP_START_LEAD_COLD_S def _named_members(owner: str, members: dict[str, str]) -> dict[str, dict[str, str]]: """role → {device_id, name}. Sent to devices so each records who its peers were (by stable device_id + display name), letting a device name an offline peer later — the device is the durable source of truth for episode membership.""" fleet = _fleet_of(owner) return { r: {"device_id": d, "name": (fleet[d].name if d in fleet else d)} for r, d in members.items() } def _schedule_episode_start( owner: str, task_name: str, members: dict[str, str], lead_s: float, exclude_device_id: Optional[str] = None, signature: Optional[list[str]] = None, ) -> str: """Enqueue a synchronized start_capture to every member device (except exclude_device_id, which self-schedules from the returned T0). Raises 409 if any member is offline. Returns the shared start_at_utc (ISO). Each command carries the episode's full membership (role → device_id+name) and the task's device signature so every device persists who recorded with it — the fleet no longer needs its own HF-backed record of this.""" fleet = _fleet_of(owner) device_ids = list(members.values()) offline = [d for d in device_ids if d not in fleet or not fleet[d].online] if offline: raise HTTPException(409, {"message": "one or more devices are offline", "offline": offline}) named = _named_members(owner, members) target_iso = (datetime.now(timezone.utc) + timedelta(seconds=lead_s)).isoformat() for device_id in device_ids: if device_id == exclude_device_id: continue cmd = Command( id=uuid.uuid4().hex[:12], type="start_capture", args={"task_name": task_name, "start_at_utc": target_iso, "members": named, "signature": signature or []}, ) _enqueue(fleet[device_id], cmd) return target_iso def _dispatch_episode_stop(owner: str, members: dict[str, str], exclude_device_id: Optional[str] = None) -> list[str]: """Fan out an IMMEDIATE stop to every member device except exclude_device_id (the acting device, which has already stopped locally). Each peer stops as soon as it receives the command, i.e. within ~1 poll interval — the pressed device is instant and peers trail only by the short delivery latency, no scheduled lead. Chaining fast stays safe because the next episode's start still uses a lead (GROUP_START_LEAD_*) that the peer's stop+mux fits inside. Returns the dispatched stop command ids so the caller can wait for their results (see Session.pending_stop_cmds) and end the "stopping" phase exactly when the devices report done, rather than after a fixed guess. (A synchronized, lead-based stop — shared future T_stop via GROUP_STOP_LEAD_S + the device-side CaptureScheduler.schedule_stop path — remains wired up but unused here; it'd matter only if delivery latency grew. With long-polling delivery drops to a network round-trip, making even this spread negligible.)""" fleet = _fleet_of(owner) ids: list[str] = [] for device_id in members.values(): if device_id == exclude_device_id: continue dev = fleet.get(device_id) if dev is not None: cmd = Command(id=uuid.uuid4().hex[:12], type="stop_capture", args={}) _enqueue(dev, cmd) ids.append(cmd.id) return ids def _begin_episode_stop(s: "Session", stop_cmd_ids: list[str]) -> None: """Mark a session as no longer recording and enter the 'stopping' phase, awaiting the dispatched stops' results (see Session.stopping). Only claims 'stopping' when there are commands to wait on — a stop with no tracked device (e.g. a solo device that stopped locally) goes straight to idle.""" s.recording = False s.last_stop_at = time.time() # also drives the warm/cold lead for the next episode s.pending_stop_cmds = set(stop_cmd_ids) s.stopping = bool(stop_cmd_ids) def _episode_id_for_target(target_iso: str) -> str: """Mirror the device's episode_id_for(T0): same UTC second → same id, so the manifest entry matches the folder every device actually creates.""" return datetime.fromisoformat(target_iso).astimezone(timezone.utc).strftime("%Y%m%d_%H%M%S") # === sessions (operator-facing, session OAuth) =============================== # The recording unit. Launch = pick a task + a target (a group, or a single # device treated as a group of one), validate the target's roles match the # task's required devices, then record one or more synchronized episodes into # the session's manifest, and close it. class SessionLaunchReq(BaseModel): task_id: str # A device id per role, picked from the fleet at launch time. The device # selection IS the grouping — no Group object is created (the session owns # its members map). left: str = "" right: str = "" casquette: str = "" def _resolve_members(owner: str, req: SessionLaunchReq) -> dict[str, str]: """Build the session's role→device map from the per-role device ids, checking each device actually fills the role it was placed in.""" fleet = _fleet_of(owner) members: dict[str, str] = {} for role, device_id in (("left", req.left), ("right", req.right), ("casquette", req.casquette)): if not device_id: continue dev = fleet.get(device_id) if dev is None: raise HTTPException(404, f"Device {device_id} not found") if _device_slot(dev) != role: raise HTTPException(400, f"Device {device_id} cannot fill the {role} role") if not dev.online: # A session can't be launched with an offline device — it couldn't # receive the synchronized start and would fail at the first episode. raise HTTPException(409, {"message": f"{dev.name} is offline", "offline": [device_id]}) members[role] = device_id if not members: raise HTTPException(400, "Select at least one device") return members # Safety cap: if a device never reports its stop result (e.g. it dropped offline # mid-stop), don't leave the "stopping" phase stuck — expose it as ended after this. STOP_PHASE_MAX_S = 30.0 def _session_dict(owner: str, s: Session) -> dict[str, Any]: fleet = _fleet_of(owner) def dname(d): dev = fleet.get(d) return dev.name if dev else d # "stopping" ends when every dispatched stop reported (s.stopping cleared); # the time cap is only a fallback for a device that never reports. stopping = s.stopping and s.last_stop_at is not None and (time.time() - s.last_stop_at) < STOP_PHASE_MAX_S return { "id": s.id, "status": s.status, "task_id": s.task_id, "task_name": _task_name(owner, s.task_id), "recording": s.recording, "stopping": stopping, # accurate "stopping" phase — ends when the devices really stop "started_at": s.started_at, "members": {r: {"device_id": d, "name": dname(d)} for r, d in s.members.items()}, "episode_count": len(s.episodes), "episodes": s.episodes, } @app.get("/api/fleet/sessions") async def list_sessions(request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) return {"sessions": [_session_dict(owner, s) for s in _sessions_of(owner).values()]} @app.post("/api/fleet/sessions") async def launch_session(req: SessionLaunchReq, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) task = _tasks_of(owner).get(req.task_id) if task is None: raise HTTPException(404, "Task not found") members = _resolve_members(owner, req) # The target must provide EXACTLY the roles the task requires. _validate_task_signature(owner, req.task_id, set(members.keys())) # A single open session at a time (fleet-wide). Launching is done from the # task list; the running one must be closed before another can start. if any(s.status == "open" for s in _sessions_of(owner).values()): raise HTTPException(409, {"message": "a session is already running; close it first"}) # A device tied up by a dataset upload/conversion can't also record. busy = _busy_recording_blockers(owner, members.values()) if busy: fleet = _fleet_of(owner) names = ", ".join(fleet[d].name for d in busy if d in fleet) or "a device" raise HTTPException(409, {"message": f"{names} is busy processing a dataset — wait for it to finish", "devices": busy}) sid = uuid.uuid4().hex[:8] _sessions_of(owner)[sid] = Session(id=sid, task_id=req.task_id, members=members) # No warm-up on launch: keeping the OAK-D on until the first episode would # drain the battery if the operator waits. Instead the first episode uses # the COLD lead (long enough for the scheduler to warm the OAK-D during the # lead), and after that the device keepalive keeps it warm between episodes. return {"status": "ok", "id": sid, "session": _session_dict(owner, _sessions_of(owner)[sid])} def _get_open_session(owner: str, session_id: str) -> Session: s = _sessions_of(owner).get(session_id) if s is None: raise HTTPException(404, "Session not found") if s.status != "open": raise HTTPException(409, "Session is closed") return s @app.post("/api/fleet/sessions/{session_id}/episode/start") async def session_episode_start(session_id: str, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) s = _get_open_session(owner, session_id) # A member pulled into a dataset upload/conversion can't start a new episode. busy = _busy_recording_blockers(owner, s.members.values()) if busy: fleet = _fleet_of(owner) names = ", ".join(fleet[d].name for d in busy if d in fleet) or "a device" raise HTTPException(409, {"message": f"{names} is busy processing a dataset — wait for it to finish", "devices": busy}) _task = _tasks_of(owner).get(s.task_id) target_iso = _schedule_episode_start( owner, _task_name(owner, s.task_id), s.members, _lead_for(s), signature=_task.device_signature if _task else [], ) episode_id = _episode_id_for_target(target_iso) # Record the manifest entry up front from the deterministic id; per-device # success is reconciled later via /api/devices/result (phase 3). start_at_utc # is the shared T0 — the UI shows "initializing" until then, "recording" after. s.episodes.append({"episode_id": episode_id, "roles": dict(s.members), "started_at": datetime.now(timezone.utc).isoformat(), "start_at_utc": target_iso}) s.recording = True return {"status": "scheduled", "scheduled_start_utc": target_iso, "episode_id": episode_id, "devices": list(s.members.values())} @app.post("/api/fleet/sessions/{session_id}/episode/stop") async def session_episode_stop(session_id: str, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) s = _get_open_session(owner, session_id) ids = _dispatch_episode_stop(owner, s.members) _begin_episode_stop(s, ids) return {"status": "ok", "devices": list(s.members.values())} @app.post("/api/fleet/sessions/{session_id}/episode/delete-last") async def session_delete_last_episode(session_id: str, request: Request) -> dict[str, Any]: """Drop the session's most recent episode from the manifest and tell each device that recorded it to delete its local files. Refused while recording (the last manifest entry is then the in-progress episode).""" owner = await _operator_loaded(request) s = _get_open_session(owner, session_id) if s.recording: raise HTTPException(409, {"message": "stop the current episode before deleting"}) if not s.episodes: raise HTTPException(409, {"message": "no episode to delete"}) ep = s.episodes.pop() # most recent episode_id = ep["episode_id"] fleet = _fleet_of(owner) devices = list((ep.get("roles") or {}).values()) for dev_id in devices: dev = fleet.get(dev_id) if dev is not None: _enqueue(dev, Command(id=uuid.uuid4().hex[:12], type="delete_episode", args={"episode_id": episode_id})) return {"status": "ok", "episode_id": episode_id, "devices": devices} @app.post("/api/fleet/sessions/{session_id}/close") async def close_session(session_id: str, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) s = _sessions_of(owner).get(session_id) if s is None: raise HTTPException(404, "Session not found") # Best-effort: make sure nothing is left recording on the members. _dispatch_episode_stop(owner, s.members) if not s.episodes: # A session that recorded nothing isn't worth keeping — drop it so it # never clutters the task's history. del _sessions_of(owner)[session_id] return {"status": "ok", "discarded": True} s.recording = False s.last_stop_at = time.time() s.status = "closed" return {"status": "ok"} @app.delete("/api/fleet/sessions/{session_id}") async def delete_session(session_id: str, request: Request) -> dict[str, str]: owner = await _operator_loaded(request) if session_id not in _sessions_of(owner): raise HTTPException(404, "Session not found") del _sessions_of(owner)[session_id] return {"status": "ok"} # === LeRobot dataset generation (operator OAuth) ============================ # Selected tasks (all sharing one device signature) → each involved device # pushes its own streams to a shared raw dataset (by role) → a processing Space # converts raw → LeRobot. The fleet only orchestrates; devices upload with their # own write tokens (the fleet can't reach them, and needs no token for uploads). class DatasetReq(BaseModel): task_ids: list[str] name: str = "" # target dataset name; bare name → {owner}/{name} private: bool = False # create the resulting LeRobot dataset as private # If True, include only episodes whose EVERY role-device is currently online # (skip episodes that reference an offline device) instead of requiring the # whole set of devices to be up. only_available: bool = False # "Use only" advanced option: a subset of roles (e.g. ["left"] or # ["left","casquette"]). When set, ANY task whose signature is a superset is # eligible, and only these roles' devices are uploaded → build a dataset from # just this sub-combination. Empty → use each selected task's full signature. roles: list[str] = [] def _command_status(dev: Device, cmd_id: str) -> Optional[Command]: """Find a dispatched command by id, whether still queued or completed (results move a command from queue to history).""" for c in list(dev.queue) + list(dev.history): if c.id == cmd_id: return c return None def _resolve_dataset_plan(owner: str, task_ids: list[str], only_available: bool = False, roles_override: Optional[list[str]] = None) -> tuple[list[str], dict[str, dict]]: """Validate the selection and build the per-device upload plan from the device reports. Returns (roles, plan) where plan maps device_id -> {"role": role, "episode_ids": set[str]}. roles_override ("Use only"): a subset of roles. When given, every selected task must merely CONTAIN these roles (superset), and only these roles' devices are uploaded — so you can build, say, a left-only dataset from bimanual tasks. When absent, the selected tasks must share one signature and all its roles are used. only_available: skip episodes that reference a device which is currently offline — the resulting dataset covers only what can be fully uploaded now.""" _reconcile_tasks(owner) tasks = _tasks_of(owner) sel = [tasks[t] for t in task_ids if t in tasks] if not sel: raise HTTPException(404, "no matching tasks") override = [r for r in (roles_override or []) if r in VALID_SLOTS] if override: oset = set(override) bad = [t.name for t in sel if not oset.issubset(set(t.device_signature))] if bad: raise HTTPException(409, {"message": "some selected tasks don't have the chosen devices", "tasks": bad}) roles = sorted(oset) else: sigs = {tuple(sorted(t.device_signature)) for t in sel} if len(sigs) != 1: raise HTTPException(409, {"message": "selected tasks must share the same device signature"}) roles = sorted(sel[0].device_signature) agg = _reported_tasks(owner) fleet = _fleet_of(owner) def _online(d): return d in fleet and fleet[d].online # Episodes come from the devices' reports (the source of truth), matched by # task name. members' device_id per role tells us which device to ask to # upload that role's data for each episode. With an override, restrict to # just those roles so only the requested devices upload. role_set = set(roles) plan: dict[str, dict] = {} for t in sel: for ep in _episodes_from_entry(agg.get(t.name)): roles_map = {r: d for r, d in ep["roles"].items() if r in role_set} if not roles_map: continue # this episode lacks the requested roles if only_available and not all(_online(d) for d in roles_map.values()): continue # a needed device is offline → episode unusable for role, dev_id in roles_map.items(): if not dev_id: continue p = plan.setdefault(dev_id, {"role": role, "episode_ids": set()}) p["episode_ids"].add(ep["episode_id"]) if not plan: msg = ("no episodes have all their devices online" if only_available else "selected tasks have no recorded episodes") raise HTTPException(409, {"message": msg}) return roles, plan async def _run_dataset_job(owner: str, job: DatasetJob, private: bool, processor_device_id: str) -> None: """Background: wait for every device's upload to complete, then have ONE device trigger the processing Space (raw → LeRobot, mono/bimanual per the device set) with ITS OWN long-lived token and report the result back. The fleet never handles an HF token for the Space call: the device→Space channel is the same one the SLAM flow already uses, so no token is cached on the fleet nor forwarded through it (avoids widening the token's blast radius and the short-lived-OAuth-token expiry problem). The Space downloads the raw, builds + pushes the dataset, and deletes the raw itself.""" fleet = _fleet_of(owner) deadline = time.time() + 1800.0 # 30-min cap for the whole upload phase try: pending = dict(job.upload_cmds) # device_id -> command id n = len(pending) while pending: if time.time() > deadline: raise RuntimeError("upload timed out") await asyncio.sleep(2.0) for dev_id, cmd_id in list(pending.items()): dev = fleet.get(dev_id) c = _command_status(dev, cmd_id) if dev else None if c is not None and c.status == "done": res = c.result or {} if res.get("status") != "ok": raise RuntimeError(f"{(dev.name if dev else dev_id)}: {res.get('message', 'upload failed')}") pending.pop(dev_id) job.progress = (n - len(pending)) / n if n else 1.0 job.message = f"Uploaded {n - len(pending)}/{n} device(s)… (this can take several minutes)" continue # Not done yet — bail fast if the device dropped offline (it went # dark before finishing, so its upload will never complete). An # actively-uploading device keeps polling → stays online, so this # only fires on a real disconnect (within ~ONLINE_WINDOW). if dev is None or not dev.online: who = dev.name if dev else dev_id raise RuntimeError(f"device '{who}' went offline before finishing its upload — reconnect it and retry") # Raw dataset complete → one device runs the processing (calls the Space # with its own token; the command completes when the Space is done). job.status = "processing" job.progress = None # opaque Space conversion → indeterminate bar job.message = "Converting to LeRobot… (this can take several minutes)" dev = fleet.get(processor_device_id) if dev is None or not dev.online: raise RuntimeError("no online device available to run processing") proc = Command(id=uuid.uuid4().hex[:12], type="process_dataset", args={"space_url": LEROBOT_SPACE_URL, "source_repo": job.raw_repo, "target_repo": job.target_repo, "roles": job.roles, "private": private, "task": job.target_repo.split("/")[-1]}) _enqueue(dev, proc) proc_deadline = time.time() + 3600.0 # 60-min cap for processing while True: if time.time() > proc_deadline: raise RuntimeError("processing timed out") await asyncio.sleep(3.0) c = _command_status(dev, proc.id) if c is None or c.status != "done": # Bail fast if the processing device dropped offline mid-run # (it keeps polling while working, so this is a real disconnect). d = fleet.get(processor_device_id) if d is None or not d.online: who = d.name if d else processor_device_id raise RuntimeError(f"device '{who}' went offline during processing — reconnect it and retry") continue res = c.result or {} if res.get("status") != "ok": raise RuntimeError(res.get("message", "processing failed")) job.result_url = res.get("result_url") or f"https://huggingface.co/datasets/{job.target_repo}" job.status, job.message, job.progress = "done", "Dataset ready.", 1.0 return except Exception as e: # noqa: BLE001 job.status, job.error = "error", str(e) job.message = f"Failed: {e}" logger.warning("dataset job %s failed: %s", job.id, e, exc_info=True) @app.post("/api/fleet/lerobot-dataset") async def create_lerobot_dataset(req: DatasetReq, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) roles, plan = _resolve_dataset_plan(owner, req.task_ids, req.only_available, req.roles or None) fleet = _fleet_of(owner) # With only_available the plan already excludes offline-device episodes; this # guard then bites only in the default mode (require the whole set up). offline = [d for d in plan if d not in fleet or not fleet[d].online] if offline: raise HTTPException(409, {"message": "some recording devices are offline", "devices": offline}) # One member device runs the processing (calls the Space with its own token). processor = next(iter(plan)) job_id = uuid.uuid4().hex[:8] name = (req.name or f"grabette-dataset-{job_id}").strip() target_repo = name if "/" in name else f"{owner}/{name}" # Intermediate raw dataset named after the target (…-raw) instead of random, # so it's easy to spot and correlate. Same namespace as the target. raw_repo = f"{target_repo}-raw" job = DatasetJob(id=job_id, task_ids=list(req.task_ids), roles=roles, raw_repo=raw_repo, target_repo=target_repo, message=f"Uploading episodes from {len(plan)} device(s)… (this can take several minutes)") _dataset_jobs_of(owner)[job_id] = job for dev_id, p in plan.items(): cmd = Command(id=uuid.uuid4().hex[:12], type="upload_episodes", args={"raw_repo": raw_repo, "role": p["role"], "episode_ids": sorted(p["episode_ids"]), "private": req.private}) _enqueue(fleet[dev_id], cmd) job.upload_cmds[dev_id] = cmd.id t = asyncio.create_task(_run_dataset_job(owner, job, req.private, processor)) _bg_tasks.add(t) t.add_done_callback(_bg_tasks.discard) return {"status": "ok", "job_id": job_id, "raw_repo": raw_repo, "target_repo": target_repo, "roles": roles, "devices": list(plan.keys())} @app.get("/api/fleet/lerobot-dataset/{job_id}") async def lerobot_dataset_status(job_id: str, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) job = _dataset_jobs_of(owner).get(job_id) if job is None: raise HTTPException(404, "job not found") return {"id": job.id, "status": job.status, "message": job.message, "progress": job.progress, "raw_repo": job.raw_repo, "target_repo": job.target_repo, "result_url": job.result_url, "error": job.error} # === device-facing sync API (Bearer auth) — physical-button episode start/stop # A button press records an episode into the OPEN session containing this # device (found by membership). It behaves exactly like the operator starting # an episode from the dashboard: same task, same synchronized T0 across the # session's members. No open session → solo (the device records locally). ====== @app.post("/api/devices/{device_id}/sync/start") async def device_sync_start(device_id: str, auth: tuple[str, str] = Depends(device_auth)) -> dict[str, Any]: owner, _token = auth if device_id not in _fleet_of(owner): raise HTTPException(404, "Device not registered") s = _open_session_for_device(owner, device_id) if s is None: return {"status": "solo"} task_name = _task_name(owner, s.task_id) _task = _tasks_of(owner).get(s.task_id) signature = _task.device_signature if _task else [] target_iso = _schedule_episode_start( owner, task_name, s.members, _lead_for(s), exclude_device_id=device_id, signature=signature, ) episode_id = _episode_id_for_target(target_iso) s.episodes.append({"episode_id": episode_id, "roles": dict(s.members), "started_at": datetime.now(timezone.utc).isoformat(), "start_at_utc": target_iso}) s.recording = True peers = [d for d in s.members.values() if d != device_id] # Full membership + signature so the pressing device persists who it recorded # with (it self-schedules from scheduled_start_utc, so it's excluded above). return {"status": "scheduled", "scheduled_start_utc": target_iso, "task_name": task_name, "peers": peers, "members": _named_members(owner, s.members), "signature": signature} @app.post("/api/devices/{device_id}/sync/stop") async def device_sync_stop(device_id: str, auth: tuple[str, str] = Depends(device_auth)) -> dict[str, Any]: owner, _token = auth if device_id not in _fleet_of(owner): raise HTTPException(404, "Device not registered") s = _open_session_for_device(owner, device_id) if s is None: return {"status": "solo"} ids = _dispatch_episode_stop(owner, s.members, exclude_device_id=device_id) _begin_episode_stop(s, ids) return {"status": "ok", "peers": [d for d in s.members.values() if d != device_id]} @app.post("/api/fleet/dispatch") async def dispatch(req: DispatchReq, request: Request) -> dict[str, Any]: owner = await _operator_loaded(request) dev = _fleet_of(owner).get(req.device_id) if dev is None: raise HTTPException(404, "Device not found in your fleet") cmd = Command(id=uuid.uuid4().hex[:12], type=req.type, args=dict(req.args)) _enqueue(dev, cmd) return {"status": "queued", "command_id": cmd.id} # === OAuth relay for grabette devices ======================================== @app.get("/oauth/grabette/callback", response_model=None) async def grabette_oauth_relay( code: str | None = None, state: str | None = None, error: str | None = None, error_description: str | None = None, ) -> RedirectResponse | HTMLResponse: """Relay the HF OAuth callback to the originating grabette on the local network. The grabette encodes its mDNS hostname into the OAuth state as ``{hostname}|{session_id}``. This endpoint splits that apart and issues a 302 redirect so the user's browser (on the same LAN as the grabette) reaches ``http://{hostname}.local:8000/api/hf-auth/oauth/callback``. Only this Space URL needs to be registered as a redirect_uri in the HF OAuth app — one entry covers every grabette regardless of hostname. """ if not state or "|" not in state: return HTMLResponse("Missing or invalid state parameter.", status_code=400) hostname, session_id = state.split("|", 1) from urllib.parse import urlencode base = f"http://{hostname}.local:8000/api/hf-auth/oauth/callback" if error: # Forward the error to the grabette so it can mark the session as failed # and stop the polling loop on the frontend. params: dict = {"error": error, "state": session_id} if error_description: params["error_description"] = error_description return RedirectResponse(f"{base}?{urlencode(params)}", status_code=302) if not code: return HTMLResponse("Missing code.", status_code=400) return RedirectResponse( f"{base}?{urlencode({'code': code, 'state': session_id})}", status_code=302, ) # === UI ====================================================================== @app.get("/") async def index() -> HTMLResponse: return HTMLResponse(_INDEX) _INDEX = """ Grabette fleet

Grabette fleet — operator dashboard

This dashboard lists every Grabette, Gripette and Casquette that has been detected and is connected to your HuggingFace account. Each device is grouped by type below; sign in with HuggingFace to see and control the devices that report to your fleet.

HuggingFace login

Checking…

Fleet

Tasks

0

These tasks come from the grabettes connected to this account — those online now, plus any connected earlier in this session. Episode counts reflect only the grabettes currently online.

No tasks yet.
"""