--- title: Splasher emoji: 💦 colorFrom: indigo colorTo: blue sdk: docker app_port: 7860 pinned: false license: mit short_description: Label synchronized 3D/image channels into a 2D BEV grid --- # Splasher > **Live demo** — this Hugging Face Space runs `splasher demo --serve` (the synthetic > source) inside a Docker container. The 3D cloud, camera and BEV grid are all interactive. > Note: the demo serves a **single shared session**, so concurrent visitors edit the same > grid. See the [GitHub repo](https://github.com/augustin-bresset/splasher) to run it locally. A **labeling** tool with a generic core: you give it a *synchronous dataset* — at each timestamp, a **pack of named channels** (3D point cloud, camera image, pose, …) — and you label either a **top-down 2D grid (BEV)**, the **3D points** directly, or both. First use case: **traversability**. But nothing is hard-wired: no imposed class schema, no imposed world semantics, and **no mandatory dependency on any dataset format**. apairo is just one optional input adapter. ## Idea - Several synchronized channels, shown as references: you roam freely in the 3D cloud and look at the camera images. - You choose which **channels** to display (the *Channels* dock: show/hide each cloud or camera available in the source — multiple cameras and clouds supported). - You **draw the grid of cells** (top-down view): its extent and cell size, created explicitly via **"New grid"**. Undo is **per frame**. - You **drag a rectangle** with the mouse on the top-down view. Depending on the target: - **Grid**: fills the covered cells with the active class (output = raster of IDs). - **Points**: assigns the class to the 3D points in the rectangle (output = per-point labels). - Interaction: **left-drag = apply**, **right-drag = erase / deselect**, **Shift-drag or middle-button = pan**, **wheel = zoom**. An **ego marker** (X = forward, Y = left) is drawn at the origin in both the BEV and the 3D views, using the same axis convention so the two read consistently. - **Selection** mode (desktop-style): left-drag adds cells to the selection, right-drag removes them (non-contiguous selections allowed), then you **apply** the class to the whole selection at once. Changing the grid asks for **confirmation** if a labeling already exists. - **Accumulation**: you can accumulate ±N frames **registered by their poses** into the current frame's frame of reference (a denser cloud helps labeling). The grid and labels stay **per frame**: a brush stroke on the accumulated cloud is **de-accumulated** back to each source frame. (Requires a `POSE` channel.) ## Architecture (swappable GUI) Layers from generic to specific — each depends only on the previous one: ``` splasher/ core/ pure numpy model (grid, label targets, BEV projection, accumulation…) engine/ headless Session: all the state + operations, with no UI dependency. Returns a *semantic* ViewState (points + per-point labels + channel, BEV field, grid raster, selection, images) — not pixels. server/ FastAPI backend on the same Session + serves the web front; desktop app. web/ web front (vanilla, zero build) — the ONLY front (packaged in the wheel). vendor/ Three.js hosted locally (offline). ``` **One front, one engine.** The web front is served by the backend; the desktop app (`splasher` without `--serve`) opens that same front in a **native webview** (pywebview, Spotify/Electron style) — so the desktop *is* the web front, in a window. The `Session` (`splasher.engine.Session`) is the flexibility point: it draws nothing, it exposes `view_state()` + commands (`paint_rect`, `select_rect`, `apply_selection`, `set_frame`, `set_accum_radius`, `commit_grid`, `save`/`load`, …). Plugging in another front = consuming this `Session` (the web does it over HTTP). **Colorization stays on the front side**: each front draws the primitives its own way (reference helpers in `splasher.engine.render`, mirrored in JS under `splasher/web/src`). ## Installation ```bash cd ~/dev/splasher uv sync # core + engine only (numpy) uv sync --extra app # + desktop app (FastAPI + uvicorn + pywebview) uv sync --extra api # + headless web/API server (FastAPI + uvicorn) uv sync --extra apairo # + apairo adapter (optional) ``` ## Demo (zero external data) ```bash uv run python examples/demo_arraysource.py # desktop app (native window, `app` extra) splasher # empty: file-viewer mode (browse + open files) splasher demo # synthetic dataset splasher demo --serve # web server → http://127.0.0.1:8077 (`api` extra) ``` **File viewer**: launched empty, *Open file…* browses the filesystem and opens individual point clouds (`.npy`/`.bin`/`.pcd`) and images into resizable views, with a clear error for unreadable / unsupported files. Independent from the dataset/labeling mode. Dark brutalist black-&-blue design. Labelable top-down (BEV) view (underlay mode: height / density / intensity) + **3D/camera panels you can add, resize, and each bind to a channel** (3D color-by: height / intensity). Sensor placements (from `ChannelSpec.placement`) are drawn as markers in 3D. Classes are fully editable (⚙ in the *Classes* panel). Three.js is hosted locally (`splasher/web/vendor`) → works offline. > Native desktop window: the `app` extra ships a Qt WebEngine (Chromium) backend, so > `uv sync --extra app && splasher demo` opens a real native window out of the box. If no > backend can start, it falls back to the default browser and keeps serving. ## Input The core consumes a `Source`: `__len__`, `__getitem__(i) -> Frame`, `channels()`. `ArraySource` builds one from in-memory numpy arrays. `ApairoSource` (`apairo` extra) wraps any synchronous apairo dataset. ### apairo ```bash uv sync --extra apairo --extra app # adapter + desktop splasher /path/to/dataset --adapter apairo # all channels splasher /path/to/dataset --adapter apairo --channels lidar,cam_front,pose # only these splasher /path/to/dataset --adapter apairo --reference lidar --tolerance 0.05 # sync an async dataset ``` Two ways to pick what you work on: **load everything** (a synchronous dataset) and choose what to *display* in the UI (the *Clouds (BEV)* toggles + **Add view** per channel), or **select channels at load time** with `--channels a,b,c`. An asynchronous dataset needs `--reference ` (and optionally `--tolerance`) to be synchronized first. ## API (back/front mode) `splasher demo --serve [--host H --port P]` starts a FastAPI server driven by the same `Session` as the desktop app. Main endpoints (each command returns the updated `ViewState` so a front renders in a single round-trip): | Method | Route | Purpose | |--------|-------|---------| | `GET` | `/api/session` | ~static description (channels, classes, n_frames) | | `GET` | `/api/view` | current render state | | `POST` | `/api/frame`, `/api/class`, `/api/tool`, `/api/targets`, `/api/accum`, `/api/visibility` | settings | | `POST` | `/api/paint`, `/api/select`, `/api/selection/apply`, `/api/selection/clear`, `/api/clear`, `/api/undo` | labeling | | `POST` | `/api/grid`, `/api/save`, `/api/load` | grid & I/O | numpy arrays travel as `{dtype, shape, data(base64)}` (`splasher.server.protocol`), decodable directly into a `TypedArray` on the JavaScript side. Interactive docs at `/docs`. ## File-viewer mode Launched empty (`splasher`), *Open file…* browses the filesystem (type a path, **Tab** to complete) and opens point clouds (`.npy`/`.bin`/`.pcd`) and images (`.png`/`.jpg`/… or `.npy` `HxWxC`) into resizable views. The **Clouds (BEV)** panel selects which open clouds feed the BEV (multiple = combined), with a color mode (height/intensity/normal). The BEV grid and its labels are **independent** of the displayed cloud — switching/combining clouds never wipes them. **Export** writes the grid raster to a single `.npy` (default name `_bev.npy`); **Save** writes a full session folder. ## Development ```bash uv sync --extra api # core + engine + server (for the test suite) uv run --extra api pytest -q # run the tests ``` The core/engine import without any UI dependency; tests are pure-numpy + FastAPI's `TestClient`. CI runs the suite on Python 3.11 and 3.12 (`.github/workflows/ci.yml`).