README.md

---
title: Splasher
emoji: 💦
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: Label synchronized 3D/image channels into a 2D BEV grid
---

# Splasher

> **Live demo** — this Hugging Face Space runs `splasher demo --serve` (the synthetic
> source) inside a Docker container. The 3D cloud, camera and BEV grid are all interactive.
> Note: the demo serves a **single shared session**, so concurrent visitors edit the same
> grid. See the [GitHub repo](https://github.com/augustin-bresset/splasher) to run it locally.

A **labeling** tool with a generic core: you give it a *synchronous dataset* — at each
timestamp, a **pack of named channels** (3D point cloud, camera image, pose, …) — and you
label either a **top-down 2D grid (BEV)**, the **3D points** directly, or both.

First use case: **traversability**. But nothing is hard-wired: no imposed class schema, no
imposed world semantics, and **no mandatory dependency on any dataset format**. apairo is
just one optional input adapter.

## Idea

- Several synchronized channels, shown as references: you roam freely in the 3D cloud and
  look at the camera images.
- You choose which **channels** to display (the *Channels* dock: show/hide each cloud or
  camera available in the source — multiple cameras and clouds supported).
- You **draw the grid of cells** (top-down view): its extent and cell size, created
  explicitly via **"New grid"**. Undo is **per frame**.
- You **drag a rectangle** with the mouse on the top-down view. Depending on the target:
  - **Grid**: fills the covered cells with the active class (output = raster of IDs).
  - **Points**: assigns the class to the 3D points in the rectangle (output = per-point labels).
- Interaction: **left-drag = apply**, **right-drag = erase / deselect**,
  **Shift-drag or middle-button = pan**, **wheel = zoom**. An **ego marker** (X = forward,
  Y = left) is drawn at the origin in both the BEV and the 3D views, using the same axis
  convention so the two read consistently.
- **Selection** mode (desktop-style): left-drag adds cells to the selection, right-drag
  removes them (non-contiguous selections allowed), then you **apply** the class to the whole
  selection at once. Changing the grid asks for **confirmation** if a labeling already exists.
- **Accumulation**: you can accumulate ±N frames **registered by their poses** into the
  current frame's frame of reference (a denser cloud helps labeling). The grid and labels
  stay **per frame**: a brush stroke on the accumulated cloud is **de-accumulated** back to
  each source frame. (Requires a `POSE` channel.)

## Architecture (swappable GUI)

Layers from generic to specific — each depends only on the previous one:

```
splasher/
  core/      pure numpy model (grid, label targets, BEV projection, accumulation…)
  engine/    headless Session: all the state + operations, with no UI dependency.
             Returns a *semantic* ViewState (points + per-point labels + channel,
             BEV field, grid raster, selection, images) — not pixels.
  server/    FastAPI backend on the same Session + serves the web front; desktop app.
  web/       web front (vanilla, zero build) — the ONLY front (packaged in the wheel).
    vendor/  Three.js hosted locally (offline).
```

**One front, one engine.** The web front is served by the backend; the desktop app
(`splasher` without `--serve`) opens that same front in a **native webview** (pywebview,
Spotify/Electron style) — so the desktop *is* the web front, in a window.

The `Session` (`splasher.engine.Session`) is the flexibility point: it draws nothing, it
exposes `view_state()` + commands (`paint_rect`, `select_rect`, `apply_selection`,
`set_frame`, `set_accum_radius`, `commit_grid`, `save`/`load`, …). Plugging in another front
= consuming this `Session` (the web does it over HTTP). **Colorization stays on the front
side**: each front draws the primitives its own way (reference helpers in
`splasher.engine.render`, mirrored in JS under `splasher/web/src`).

## Installation

```bash
cd ~/dev/splasher
uv sync                 # core + engine only (numpy)
uv sync --extra app     # + desktop app (FastAPI + uvicorn + pywebview)
uv sync --extra api     # + headless web/API server (FastAPI + uvicorn)
uv sync --extra apairo  # + apairo adapter (optional)
```

## Demo (zero external data)

```bash
uv run python examples/demo_arraysource.py   # desktop app (native window, `app` extra)
splasher                                      # empty: file-viewer mode (browse + open files)
splasher demo                                 # synthetic dataset
splasher demo --serve                         # web server → http://127.0.0.1:8077 (`api` extra)
```

**File viewer**: launched empty, *Open file…* browses the filesystem and opens individual
point clouds (`.npy`/`.bin`/`.pcd`) and images into resizable views, with a clear error for
unreadable / unsupported files. Independent from the dataset/labeling mode.

Dark brutalist black-&-blue design. Labelable top-down (BEV) view (underlay mode:
height / density / intensity) + **3D/camera panels you can add, resize, and each bind to a
channel** (3D color-by: height / intensity). Sensor placements (from `ChannelSpec.placement`)
are drawn as markers in 3D. Classes are fully editable (⚙ in the *Classes* panel). Three.js
is hosted locally (`splasher/web/vendor`) → works offline.

> Native desktop window: the `app` extra ships a Qt WebEngine (Chromium) backend, so
> `uv sync --extra app && splasher demo` opens a real native window out of the box. If no
> backend can start, it falls back to the default browser and keeps serving.

## Input

The core consumes a `Source`: `__len__`, `__getitem__(i) -> Frame`, `channels()`.
`ArraySource` builds one from in-memory numpy arrays. `ApairoSource` (`apairo` extra) wraps
any synchronous apairo dataset.

### apairo

```bash
uv sync --extra apairo --extra app                     # adapter + desktop
splasher /path/to/dataset --adapter apairo             # all channels
splasher /path/to/dataset --adapter apairo --channels lidar,cam_front,pose   # only these
splasher /path/to/dataset --adapter apairo --reference lidar --tolerance 0.05  # sync an async dataset
```

Two ways to pick what you work on: **load everything** (a synchronous dataset) and choose
what to *display* in the UI (the *Clouds (BEV)* toggles + **Add view** per channel), or
**select channels at load time** with `--channels a,b,c`. An asynchronous dataset needs
`--reference <channel>` (and optionally `--tolerance`) to be synchronized first.

## API (back/front mode)

`splasher demo --serve [--host H --port P]` starts a FastAPI server driven by the same
`Session` as the desktop app. Main endpoints (each command returns the updated `ViewState`
so a front renders in a single round-trip):

| Method | Route | Purpose |
|--------|-------|---------|
| `GET`  | `/api/session` | ~static description (channels, classes, n_frames) |
| `GET`  | `/api/view`    | current render state |
| `POST` | `/api/frame`, `/api/class`, `/api/tool`, `/api/targets`, `/api/accum`, `/api/visibility` | settings |
| `POST` | `/api/paint`, `/api/select`, `/api/selection/apply`, `/api/selection/clear`, `/api/clear`, `/api/undo` | labeling |
| `POST` | `/api/grid`, `/api/save`, `/api/load` | grid & I/O |

numpy arrays travel as `{dtype, shape, data(base64)}` (`splasher.server.protocol`), decodable
directly into a `TypedArray` on the JavaScript side. Interactive docs at `/docs`.

## File-viewer mode

Launched empty (`splasher`), *Open file…* browses the filesystem (type a path, **Tab** to
complete) and opens point clouds (`.npy`/`.bin`/`.pcd`) and images (`.png`/`.jpg`/… or
`.npy` `HxWxC`) into resizable views. The **Clouds (BEV)** panel selects which open clouds
feed the BEV (multiple = combined), with a color mode (height/intensity/normal). The BEV
grid and its labels are **independent** of the displayed cloud — switching/combining clouds
never wipes them. **Export** writes the grid raster to a single `.npy` (default name
`<cloud>_bev.npy`); **Save** writes a full session folder.

## Development

```bash
uv sync --extra api          # core + engine + server (for the test suite)
uv run --extra api pytest -q # run the tests
```

The core/engine import without any UI dependency; tests are pure-numpy + FastAPI's
`TestClient`. CI runs the suite on Python 3.11 and 3.12 (`.github/workflows/ci.yml`).