| --- |
| title: Splasher |
| emoji: 💦 |
| colorFrom: indigo |
| colorTo: blue |
| sdk: docker |
| app_port: 7860 |
| pinned: false |
| license: mit |
| short_description: Label synchronized 3D/image channels into a 2D BEV grid |
| --- |
| |
| # Splasher |
|
|
| > **Live demo** — this Hugging Face Space runs `splasher demo --serve` (the synthetic |
| > source) inside a Docker container. The 3D cloud, camera and BEV grid are all interactive. |
| > Note: the demo serves a **single shared session**, so concurrent visitors edit the same |
| > grid. See the [GitHub repo](https://github.com/augustin-bresset/splasher) to run it locally. |
|
|
| A **labeling** tool with a generic core: you give it a *synchronous dataset* — at each |
| timestamp, a **pack of named channels** (3D point cloud, camera image, pose, …) — and you |
| label either a **top-down 2D grid (BEV)**, the **3D points** directly, or both. |
|
|
| First use case: **traversability**. But nothing is hard-wired: no imposed class schema, no |
| imposed world semantics, and **no mandatory dependency on any dataset format**. apairo is |
| just one optional input adapter. |
|
|
| ## Idea |
|
|
| - Several synchronized channels, shown as references: you roam freely in the 3D cloud and |
| look at the camera images. |
| - You choose which **channels** to display (the *Channels* dock: show/hide each cloud or |
| camera available in the source — multiple cameras and clouds supported). |
| - You **draw the grid of cells** (top-down view): its extent and cell size, created |
| explicitly via **"New grid"**. Undo is **per frame**. |
| - You **drag a rectangle** with the mouse on the top-down view. Depending on the target: |
| - **Grid**: fills the covered cells with the active class (output = raster of IDs). |
| - **Points**: assigns the class to the 3D points in the rectangle (output = per-point labels). |
| - Interaction: **left-drag = apply**, **right-drag = erase / deselect**, |
| **Shift-drag or middle-button = pan**, **wheel = zoom**. An **ego marker** (X = forward, |
| Y = left) is drawn at the origin in both the BEV and the 3D views, using the same axis |
| convention so the two read consistently. |
| - **Selection** mode (desktop-style): left-drag adds cells to the selection, right-drag |
| removes them (non-contiguous selections allowed), then you **apply** the class to the whole |
| selection at once. Changing the grid asks for **confirmation** if a labeling already exists. |
| - **Accumulation**: you can accumulate ±N frames **registered by their poses** into the |
| current frame's frame of reference (a denser cloud helps labeling). The grid and labels |
| stay **per frame**: a brush stroke on the accumulated cloud is **de-accumulated** back to |
| each source frame. (Requires a `POSE` channel.) |
|
|
| ## Architecture (swappable GUI) |
|
|
| Layers from generic to specific — each depends only on the previous one: |
|
|
| ``` |
| splasher/ |
| core/ pure numpy model (grid, label targets, BEV projection, accumulation…) |
| engine/ headless Session: all the state + operations, with no UI dependency. |
| Returns a *semantic* ViewState (points + per-point labels + channel, |
| BEV field, grid raster, selection, images) — not pixels. |
| server/ FastAPI backend on the same Session + serves the web front; desktop app. |
| web/ web front (vanilla, zero build) — the ONLY front (packaged in the wheel). |
| vendor/ Three.js hosted locally (offline). |
| ``` |
|
|
| **One front, one engine.** The web front is served by the backend; the desktop app |
| (`splasher` without `--serve`) opens that same front in a **native webview** (pywebview, |
| Spotify/Electron style) — so the desktop *is* the web front, in a window. |
|
|
| The `Session` (`splasher.engine.Session`) is the flexibility point: it draws nothing, it |
| exposes `view_state()` + commands (`paint_rect`, `select_rect`, `apply_selection`, |
| `set_frame`, `set_accum_radius`, `commit_grid`, `save`/`load`, …). Plugging in another front |
| = consuming this `Session` (the web does it over HTTP). **Colorization stays on the front |
| side**: each front draws the primitives its own way (reference helpers in |
| `splasher.engine.render`, mirrored in JS under `splasher/web/src`). |
|
|
| ## Installation |
|
|
| ```bash |
| cd ~/dev/splasher |
| uv sync # core + engine only (numpy) |
| uv sync --extra app # + desktop app (FastAPI + uvicorn + pywebview) |
| uv sync --extra api # + headless web/API server (FastAPI + uvicorn) |
| uv sync --extra apairo # + apairo adapter (optional) |
| ``` |
|
|
| ## Demo (zero external data) |
|
|
| ```bash |
| uv run python examples/demo_arraysource.py # desktop app (native window, `app` extra) |
| splasher # empty: file-viewer mode (browse + open files) |
| splasher demo # synthetic dataset |
| splasher demo --serve # web server → http://127.0.0.1:8077 (`api` extra) |
| ``` |
|
|
| **File viewer**: launched empty, *Open file…* browses the filesystem and opens individual |
| point clouds (`.npy`/`.bin`/`.pcd`) and images into resizable views, with a clear error for |
| unreadable / unsupported files. Independent from the dataset/labeling mode. |
|
|
| Dark brutalist black-&-blue design. Labelable top-down (BEV) view (underlay mode: |
| height / density / intensity) + **3D/camera panels you can add, resize, and each bind to a |
| channel** (3D color-by: height / intensity). Sensor placements (from `ChannelSpec.placement`) |
| are drawn as markers in 3D. Classes are fully editable (⚙ in the *Classes* panel). Three.js |
| is hosted locally (`splasher/web/vendor`) → works offline. |
|
|
| > Native desktop window: the `app` extra ships a Qt WebEngine (Chromium) backend, so |
| > `uv sync --extra app && splasher demo` opens a real native window out of the box. If no |
| > backend can start, it falls back to the default browser and keeps serving. |
|
|
| ## Input |
|
|
| The core consumes a `Source`: `__len__`, `__getitem__(i) -> Frame`, `channels()`. |
| `ArraySource` builds one from in-memory numpy arrays. `ApairoSource` (`apairo` extra) wraps |
| any synchronous apairo dataset. |
|
|
| ### apairo |
|
|
| ```bash |
| uv sync --extra apairo --extra app # adapter + desktop |
| splasher /path/to/dataset --adapter apairo # all channels |
| splasher /path/to/dataset --adapter apairo --channels lidar,cam_front,pose # only these |
| splasher /path/to/dataset --adapter apairo --reference lidar --tolerance 0.05 # sync an async dataset |
| ``` |
|
|
| Two ways to pick what you work on: **load everything** (a synchronous dataset) and choose |
| what to *display* in the UI (the *Clouds (BEV)* toggles + **Add view** per channel), or |
| **select channels at load time** with `--channels a,b,c`. An asynchronous dataset needs |
| `--reference <channel>` (and optionally `--tolerance`) to be synchronized first. |
|
|
| ## API (back/front mode) |
|
|
| `splasher demo --serve [--host H --port P]` starts a FastAPI server driven by the same |
| `Session` as the desktop app. Main endpoints (each command returns the updated `ViewState` |
| so a front renders in a single round-trip): |
|
|
| | Method | Route | Purpose | |
| |--------|-------|---------| |
| | `GET` | `/api/session` | ~static description (channels, classes, n_frames) | |
| | `GET` | `/api/view` | current render state | |
| | `POST` | `/api/frame`, `/api/class`, `/api/tool`, `/api/targets`, `/api/accum`, `/api/visibility` | settings | |
| | `POST` | `/api/paint`, `/api/select`, `/api/selection/apply`, `/api/selection/clear`, `/api/clear`, `/api/undo` | labeling | |
| | `POST` | `/api/grid`, `/api/save`, `/api/load` | grid & I/O | |
| |
| numpy arrays travel as `{dtype, shape, data(base64)}` (`splasher.server.protocol`), decodable |
| directly into a `TypedArray` on the JavaScript side. Interactive docs at `/docs`. |
| |
| ## File-viewer mode |
| |
| Launched empty (`splasher`), *Open file…* browses the filesystem (type a path, **Tab** to |
| complete) and opens point clouds (`.npy`/`.bin`/`.pcd`) and images (`.png`/`.jpg`/… or |
| `.npy` `HxWxC`) into resizable views. The **Clouds (BEV)** panel selects which open clouds |
| feed the BEV (multiple = combined), with a color mode (height/intensity/normal). The BEV |
| grid and its labels are **independent** of the displayed cloud — switching/combining clouds |
| never wipes them. **Export** writes the grid raster to a single `.npy` (default name |
| `<cloud>_bev.npy`); **Save** writes a full session folder. |
|
|
| ## Development |
|
|
| ```bash |
| uv sync --extra api # core + engine + server (for the test suite) |
| uv run --extra api pytest -q # run the tests |
| ``` |
|
|
| The core/engine import without any UI dependency; tests are pure-numpy + FastAPI's |
| `TestClient`. CI runs the suite on Python 3.11 and 3.12 (`.github/workflows/ci.yml`). |
|
|
