docs/api_usage.md

# Scene Graph Manager API Usage

This document gives a concise, high-signal overview of the HTTP API so other services can integrate without reading the whole codebase.

## Refresh Checklist (for future updates)

Use this prompt when endpoints change:

> Run `rg "@app" -n api/app.py` and list new/modified routes. Update `docs/api_usage.md` with:
> - Endpoint table (method, path, summary)
> - Auth / header requirements
> - Request & response JSON samples (curl when useful)
> - Notes on query params, error behaviors, and streaming endpoints.
> Re-run `python -m compileall api/app.py` and ensure the doc still reflects reality.

## Base URL

All paths below are relative to:

```
https://scene-graph-mgr-api.fly.dev
```

## Authentication

- Public endpoints are unauthenticated but subject to Fly.io rate limits.
- Internal write endpoints require a shared secret header: `x-internal-secret: <secret>`.
- Standard error payload:

```json
{
  "detail": "Human readable message",
  "error_code": "optional-machine-code"
}
```

4xx indicates caller issues (validation, missing scene). 5xx means server-side failure.

## Endpoint Map

| Method | Path | Summary |
| --- | --- | --- |
| GET | `/healthz` | Liveness check. |
| GET | `/scenes` | List scenes with summary metadata (optional `include_empty`). |
| PUT | `/scenes/{scene_id}` | Overwrite a scene with a full graph payload. |
| PATCH | `/scenes/{scene_id}` | Apply an RFC 6902 patch to the latest graph. |
| POST | `/scenes/{scene_id}/create` | Seed an empty scene (optionally overwrite). |
| GET | `/scenes/{scene_id}/versions/latest` | Latest graph version. |
| GET | `/scenes/{scene_id}/versions` | Metadata for all versions. |
| GET | `/scenes/{scene_id}/versions/{version_id}` | Raw version record (JSON). |
| POST | `/scenes/{scene_id}/add_image` | Upload image bytes and enqueue processing. |
| POST | `/scenes/{scene_id}/add_images_from_keys` | Enqueue processing for existing S3 keys. |
| POST | `/scenes/{scene_id}/upload_video` | Upload video and queue frame extraction batch. |
| POST | `/scenes/{scene_id}/upload_image` | Browser upload → WebP resize → presigned URL. |
| GET | `/scenes/{scene_id}/instances` | List stored instances for the scene (status filters optional). |
| GET | `/scenes/{scene_id}/objects/{obj_id}/beliefs` | Latest improver belief for an object. |
| GET | `/scenes/{scene_id}/objects/{obj_id}/instances` | List stored instances for an object (status filters optional). |
| GET | `/scenes/{scene_id}/improver/backlog` | Inspect improver queue/backlog for a scene. |
| POST | `/scenes/{scene_id}/objects/{obj_id}/instances/{instance_id}/improve_instance` | Queue improver workflow for an instance. |
| GET | `/scenes/{scene_id}/runlogs` | Recent improver run logs (filterable). |
| GET | `/scenes/{scene_id}/change-requests` | Inspect change request queue (pending/applied). |
| GET | `/scenes/{scene_id}/change-summaries` | Stream applied change summaries (newest first). |
| GET | `/scenes/{scene_id}/diff` | Structural diff between two versions (machine patch & stats). |
| GET | `/scenes/{scene_id}/diff-semantic` | Semantic diff (ID-aware ops). |
| GET | `/scenes/{scene_id}/images/presign` | Generate presigned GET URL for an image key. |
| GET | `/scenes/{scene_id}/media` | List recorded media assets (images/videos). |
| POST | `/scenes/{scene_id}/media` | Upsert scene media entries supplied by workers or services. |
| GET | `/jobs/{job_id}` | Inspect queued/finished RQ jobs. |
| POST | `/stream3r/jobs` | Submit a reconstruction job (pose pointmap or model build). |
| GET | `/stream3r/jobs/{job_id}` | Inspect a Stream3R job record. |
| GET | `/stream3r/jobs` | List Stream3R jobs (filter by scene, type, status). |
| GET | `/stream3r/jobs/{job_id}/events` | Server-Sent Events feed for job lifecycle updates. |
| GET | `/stream3r/models/{scene_id}/presign` | Presign the latest model_build scene.glb for download. |
| POST | `/internal/commit-version` | Worker/internal scene commit (requires secret). |
| GET | `/debug/s3-ping` | Smoke-test S3/B2 credentials. |
| WS | `/ws?channel=all|{scene_id}` | Broadcasts scene events over WebSocket. |

---

## Endpoint Details & Examples

### Health & Diagnostics

#### `GET /healthz`
Returns `{ "ok": true, "ts": "ISO timestamp" }` for uptime monitoring.

#### `GET /debug/s3-ping`
Verifies object storage connectivity using a put/delete round-trip. Good for smoke tests.

### Scene Lifecycle

#### `GET /scenes`
List scenes. Optional `include_empty=true` includes scenes with no versions yet.

```bash
curl "https://scene-graph-mgr-api.fly.dev/scenes?include_empty=true"
```

Response contains summaries with counts and most recent version metadata.

#### `PUT /scenes/{scene_id}`
Replace entire scene graph.

Request body (`SceneGraphEnvelope`):
```json
{
  "scene_location_id": "scene-123",
  "scene_graph": {"objects": [], "relations": []},
  "meta": {"source": "manual"}
}
```
Creates a new version record and broadcasts `scene.put` on WebSocket.

#### `PATCH /scenes/{scene_id}`
Apply RFC 6902 patch to latest graph. Optional `base_version` guard prevents lost updates.

```json
{
  "scene_location_id": "scene-123",
  "json_patch": [
    {"op": "add", "path": "/objects/-", "value": {"id": "chair-1", "attributes": {}}}
  ],
  "base_version": 1728400000000
}
```

#### `POST /scenes/{scene_id}/create`
Seeds an empty graph. `overwrite=true` allows reseeding existing scenes.

#### Version Reads
- `GET /scenes/{scene_id}/versions/latest` → latest full graph.
- `GET /scenes/{scene_id}/versions` → list of `{version_id, created_at, bytes}`.
- `GET /scenes/{scene_id}/versions/{version_id}` → raw record (graph + metadata).

### Image Upload & Processing

#### `POST /scenes/{scene_id}/add_image`
Multipart upload (`file=@image.jpg`). Stores bytes via S3-compatible API, seeds scene if empty, enqueues `worker.tasks.process_image_for_scene`. Response includes RQ `job_id` and filename.

#### `POST /scenes/{scene_id}/add_images_from_keys`
JSON body:
```json
{
  "keys": ["scenes/scene-123/images/20241008/a.png"],
  "room_hint": "living_room",
  "prompt": "describe objects",
  "bounding_boxes": {"a.png": [[0.1,0.2,0.5,0.8]]}
}
```
Seeds scene if needed and enqueues batch worker job.

Response (`AddImagesFromKeysResponse`):

```json
{
  "scene_location_id": "scene-123",
  "queued_at": "2024-10-08T14:12:05Z",
  "job_id": "rq-job-id",
  "keys": [
    "scenes/scene-123/images/20241008/a.png",
    "scenes/scene-123/images/20241008/b.png"
  ]
}
```

#### `POST /scenes/{scene_id}/upload_image`
For browser uploads. Accepts multipart file, resizes to WebP (max width 1024), uploads, and returns:
```json
{
  "scene_location_id": "scene-123",
  "key": "scenes/scene-123/images/20241008/abc.webp",
  "url": "https://...presigned...",
  "width": 768,
  "height": 512,
  "bytes": 123456
}
```

#### `GET /scenes/{scene_id}/images/presign`
Query parameters: `key` (must reside under `scenes/{scene_id}/images/…` **or** `scenes/{scene_id}/videos/…`) and optional `expires`. Returns `{ "url": "...", "expires_in": 900 }`.

### Video Upload & Frame Extraction

#### `POST /scenes/{scene_id}/upload_video`
Accepts a multipart video upload (e.g., `file=@walkthrough.mp4`). Stores the binary in object storage and seeds an empty scene when needed. The worker enqueues `process_video_for_scene`, which:

- extracts WebP frames at 2fps by default (`frame_interval=0.5` seconds) and stores them under `scenes/{scene_id}/images/video_frames/...` (or keeps the source under `scenes/{scene_id}/videos/...`) so the `/images/presign` endpoint can serve them;
- retries with a software H.264 transcode if AV1 or other codecs fail to decode on the host;
- publishes the same `scene.update` event stream as still-image uploads.

Response payload (`UploadVideoResponse`):

```json
{
  "scene_location_id": "scene-123",
  "key": "scenes/scene-123/videos/20241008/abcd1234.mp4",
  "filename": "walkthrough.mp4",
  "size_bytes": 456789012,
  "content_type": "video/mp4",
  "queued_at": "2024-10-08T14:12:05Z",
  "job_id": "rq-job-id"
}
```
Clients should poll `GET /jobs/{job_id}` to track progress. When `job.result.frame_keys` is present the frame extraction succeeded.

### Scene Media

#### `GET /scenes/{scene_id}/media`
Lists stored media records for a scene. Use `media_type=image` to filter to keyframes vs. `media_type=video` for source clips.

```bash
curl "https://scene-graph-mgr-api.fly.dev/scenes/scene-123/media?media_type=image&limit=50"
```

Response (`SceneMediaListResponse`) includes the total slice returned plus ISO timestamps normalized to UTC.

#### `POST /scenes/{scene_id}/media`
Upserts media entries (usually called by workers after uploading files). Payload:

```json
{
  "entries": [
    {
      "file": "scenes/scene-123/keyframes/frame_000012.jpg",
      "media_type": "image",
      "captured_at": "2024-10-08T14:12:05Z"
    },
    {
      "file": "scenes/scene-123/videos/20241008/abcd1234.mp4",
      "media_type": "video"
    }
  ]
}
```

- `media_type` is optional—when omitted the server infers it from the filename (`image` vs `video`).
- `captured_at` accepts ISO-8601 strings (UTC preferred). If omitted, the server stores `now()`.
- Existing rows are updated in-place thanks to an upsert on the `file` column.

Response (`SceneMediaBatchResponse`) summarizes how many entries were accepted:

```json
{
  "scene_id": "scene-123",
  "accepted": 2,
  "skipped": 0,
  "files": [
    "scenes/scene-123/keyframes/frame_000012.jpg",
    "scenes/scene-123/videos/20241008/abcd1234.mp4"
  ]
}
```

### Improver Monitoring

#### `GET /scenes/{scene_id}/improver/backlog`
Summarizes outstanding improver work for a scene. Default statuses are `pending`, `queued`, and `processing`, plus records that do not yet have a status. Use this to detect when the improver has drained the backlog.

Query parameters:
- `limit` (default `200`, max `2000`) — cap the number of instances returned.
- `status` — optional list to override which statuses count as “not yet processed.” Omit to use the defaults.
- `include_missing_status` (default `true`) — include rows with `status IS NULL`.

Response:

```json
{
  "scene_id": "scene-123",
  "count": 2,
  "instances": [
    {
      "scene_id": "scene-123",
      "obj_id": "obj-1",
      "instance_id": "inst-42",
      "status": "pending",
      "status_reason": "ambient",
      "status_changed_at": "2024-10-08T14:15:06Z",
      "last_event_at": "2024-10-08T14:15:06Z",
      "created_at": "2024-10-08T13:59:40Z",
      "data": {
        "image_id": "img-998",
        "bbox_xyxy": [0.1, 0.2, 0.5, 0.8]
      }
    }
  ]
}
```
When `count` is zero the improver has no pending work for that scene.

### Improver & Beliefs

#### `POST /scenes/{scene_id}/objects/{obj_id}/instances/{instance_id}/improve_instance`
Queues improver workflow. Body adheres to `InstanceEvent` schema. Example:

```bash
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
        "scene_id": "scene-123",
        "obj_id": "sofa-1",
        "instance_id": "inst-456",
        "image_id": "scenes/scene-123/images/20241008/sofa.png",
        "bbox_xyxy": [0.1, 0.3, 0.6, 0.9]
      }' \
  https://scene-graph-mgr-api.fly.dev/scenes/scene-123/objects/sofa-1/instances/inst-456/improve_instance
```

Response: `{ "enqueued": true, "job_id": "rq-job-id" }`.

The worker embeds, seeds Qdrant, and calls `scene_improver.tasks.improve_scene`.

#### `GET /scenes/{scene_id}/objects/{obj_id}/beliefs`
Returns latest belief payload (name Dirichlet, attribute betas, relations) or 404 if none stored.

#### `GET /scenes/{scene_id}/instances`
Returns persisted instances across every object in the scene. Query params:
- `limit` (1–5000, default 1000)
- `status` (optional, repeatable) — only include instances whose `status` matches one of the provided values.
- `exclude_status` (optional, repeatable) — omit instances whose `status` matches any of the provided values (e.g., `exclude_status=superseded` to skip reassigned instances).

Response mirrors the storage map:

```json
{
  "scene_id": "scene-123",
  "count": 5,
  "objects": {
    "sofa-1": [{...}, {...}],
    "lamp-4": [{...}]
  }
}
```

#### `GET /scenes/{scene_id}/objects/{obj_id}/instances`
Returns the persisted instance rows for the object. Query params:
- `limit` (1–1000, default 100)
- `status` (optional, repeatable) — only include instances whose `status` matches one of the provided values.
- `exclude_status` (optional, repeatable) — omit instances whose `status` matches any of the provided values.

Example request skipping superseded items:

```bash
curl "https://scene-graph-mgr-api.fly.dev/scenes/scene-123/objects/sofa-1/instances?limit=20&exclude_status=superseded"
```

Response:

```json
{
  "scene_id": "scene-123",
  "obj_id": "sofa-1",
  "count": 2,
  "instances": [
    {
      "id": "inst-456",
      "image_id": "scenes/...",
      "bbox_xyxy": [0.1,0.3,0.6,0.9],
      "captured_at": 1728401000,
      "status": "processed"
    },
    {
      "id": "inst-123",
      "image_id": "scenes/...",
      "bbox_xyxy": [0.05,0.2,0.4,0.7],
      "status": "pending"
    }
  ]
}
```

#### `GET /scenes/{scene_id}/runlogs`
Query params:
- `limit` (1–1000, default 100)
- `obj_id` (optional filter)
- `instance_id` (optional filter)

Response contains `records` newest-first with `step`, `message`, `data`, timestamps, and run IDs.

#### `GET /scenes/{scene_id}/change-requests`
Inspect the change request queue. Query params:
- `state` (optional: `pending`, `applied`, `stale`)
- `limit` (1–200, default 50)

Returns an array mirroring the DB record with preconditions, payload, result, and the new `applied_summary` field:

```json
[
  {
    "request_id": "95e8...",
    "scene_id": "scene-123",
    "obj_id": "table-1",
    "requested_by": "belief-agent",
    "state": "applied",
    "confidence": 0.92,
    "payload": {"operations": [...]},
    "result": {"summary": "Renamed object \"wooden table\" from \"table\" to \"dining table\""},
    "applied_at": "2024-11-19T20:14:32.123Z",
    "applied_summary": "Renamed object \"wooden table\" from \"table\" to \"dining table\""
  }
]
```

#### `GET /scenes/{scene_id}/change-summaries`
Lightweight feed of applied change blurbs, ordered by `applied_at` descending. Supports `limit` (1–200, default 50).

```bash
curl "https://scene-graph-mgr-api.fly.dev/scenes/scene-123/change-summaries?limit=20"
```

```json
[
  {
    "request_id": "95e8...",
    "scene_id": "scene-123",
    "obj_id": "table-1",
    "applied_version": 1729300042000,
    "applied_at": "2024-11-19T20:14:32.123Z",
    "summary": "Updated object \"wooden table\" attribute \"size\" from \"medium\" to \"large\""
  },
  {
    "request_id": "73f1...",
    "scene_id": "scene-123",
    "summary": "Renamed object \"wooden table\" from \"table\" to \"dining table\""
  }
]
```

Use this endpoint for ambient “stream of consciousness” UIs showing how the improver evolves the scene.

### Diffs

#### `GET /scenes/{scene_id}/diff`
Compare two versions (`from_version`, `to_version`). Optional `mode` query (`patch`, `summary`, `both`). Returns machine patch plus stats when requested.

#### `GET /scenes/{scene_id}/diff-semantic`
ID-aware diff returning ordered operations (append/remove/replace) with summary stats.

### Stream3R Reconstruction Jobs

The Stream3R API wraps the reconstruction workers (pose pointmaps and full models) and provides idempotent enqueue, polling, and event streaming.

#### `POST /stream3r/jobs`
Submit a reconstruction job. Supported `job_type` values are `pose_pointmap` and `model_build`. Provide at least one frame (`url` or `path`) and optional `client_request_id` for idempotency (subsequent calls reuse the existing job and return `200 OK`).

```bash
curl -X POST https://scene-graph-mgr-api.fly.dev/stream3r/jobs \
  -H "Content-Type: application/json" \
  -d '{
        "job_type": "pose_pointmap",
        "scene_id": "scene-123",
        "frames": [
          {"url": "https://cdn.example/scene-123/frame_0000.webp"},
          {"path": "/mnt/captures/scene-123/frame_0001.png"}
        ],
        "session_settings": {"prediction_mode": "pointmap"},
        "client_request_id": "scene-123-20241008-run1"
      }'
```

Response (`202 Accepted` on first submission):

```json
{
  "job_id": "d8f8a3fc-3aed-441c-ac78-2b953a9229bf",
  "job_type": "pose_pointmap",
  "scene_id": "scene-123",
  "status": "queued",
  "created_at": "2024-10-08T14:12:05.417Z",
  "payload": {
    "job_type": "pose_pointmap",
    "scene_id": "scene-123",
    "frames": [
      {"url": "https://cdn.example/scene-123/frame_0000.webp"},
      {"path": "/mnt/captures/scene-123/frame_0001.png"}
    ],
    "session_settings": {"prediction_mode": "pointmap"}
  }
}
```

#### `GET /stream3r/jobs/{job_id}`
Fetch the canonical job record (backed by Postgres). Fields include:
- `status`: `queued`, `started`, `progress`, `finished`, or `failed`
- `result`: worker-published artifact manifest (S3 URLs, local paths)
- `error`: error string when status is `failed`
- timestamps (`created_at`, `started_at`, `completed_at`)

Typical successful response:

```json
{
  "job_id": "d8f8a3fc-3aed-441c-ac78-2b953a9229bf",
  "job_type": "model_build",
  "scene_id": "scene-123",
  "status": "finished",
  "created_at": "2024-10-08T14:12:05.417Z",
  "started_at": "2024-10-08T14:12:12.998Z",
  "completed_at": "2024-10-08T14:24:39.221Z",
  "result": {
    "model_dir": "s3://bucket/scene-123/stream3r/models/20241008",
    "summary_url": "s3://bucket/scene-123/stream3r/models/20241008/summary.json"
  },
  "error": null,
  "client_request_id": "scene-123-20241008-run1"
}
```

#### `GET /stream3r/jobs`
List jobs with optional filters:
- `scene_id`
- `job_type`
- `status`
- `limit` (1–200, default 50)
- `offset` (default 0)

Returns `{ "jobs": [...], "limit": 50, "offset": 0 }` with the same schema as the single-job response.

#### `GET /stream3r/jobs/{job_id}/events`
Server-Sent Events feed backed by Redis Streams. Use it for near-real-time updates in browser dashboards.

```bash
curl --no-buffer \
  -H "Accept: text/event-stream" \
  "https://scene-graph-mgr-api.fly.dev/stream3r/jobs/d8f8a3fc-3aed-441c-ac78-2b953a9229bf/events"
```

Events are emitted as standard SSE payloads:

```
id: 1728409930123-0
event: progress
data: {"job_id":"d8f8a3fc-3aed-441c-ac78-2b953a9229bf","status":"progress","progress":65}

id: 1728409960456-0
event: finished
data: {"job_id":"d8f8a3fc-3aed-441c-ac78-2b953a9229bf","status":"finished","result_url":"s3://bucket/.../summary.json"}
```

Reconnect with the last `id` to resume (`?last_id=<redis-stream-id>`). When the worker encounters an error the stream emits `event: failed` with an `error` field.

> **Implementation note:** the current worker stub still returns `status="failed"` with `error="stream3r worker not implemented"` until the GPU-backed handler ships. Downstream clients should surface the error text to operators and may retry later.

#### `GET /stream3r/models/{scene_id}/presign`
Fetch a presigned download URL for the most recent `model_build` job's `scene.glb`. Optional query params:
- `job_id` — force a specific job (must belong to the scene).
- `expires` — TTL in seconds (default 900, range 60–86400).

```bash
curl "https://scene-graph-mgr-api.fly.dev/stream3r/models/scene-123/presign?expires=600"
```

Response:

```json
{
  "scene_id": "scene-123",
  "job_id": "d8f8a3fc-3aed-441c-ac78-2b953a9229bf",
  "key": "scenes/scene-123/stream3r/models/20241008/scene.glb",
  "url": "https://s3.amazonaws.com/...signature...",
  "expires_in": 600
}
```

Returns `404` if no successful model build exists or the job did not publish a GLB artifact.

### Jobs & Internal Ops

#### `GET /jobs/{job_id}`
Inspect RQ job status. Returns timestamps, result payload (if finished), and truncated stack trace when failed.

#### `POST /internal/commit-version`
Worker-only commit. Body:
```json
{
  "scene_location_id": "scene-123",
  "scene_graph": {...},
  "base_version": 1728400000000,
  "meta": {"source": "worker"}
}
```
Requires `x-internal-secret` header when enabled. Creates new version, broadcasts `scene.update`, and publishes on Redis pub/sub.

### WebSocket Stream

#### `WS /ws?channel=all|{scene_id}`
Receives JSON events when scenes change (`scene.create`, `scene.put`, `scene.patch`, `scene.update`). Use to refresh UI state in real time.

---

## Notes & Best Practices

- All timestamps are UTC ISO strings.
- Scene writes (`PUT`, `PATCH`, `create`, `commit-version`) broadcast on WebSocket and publish to Redis channel `scene_events`.
- Object storage paths follow `scenes/{scene_id}/images/...`; presign endpoint enforces this prefix.
- Scene graph payloads no longer embed instance blobs; use the `/instances` endpoint for that data.
- Postgres storage is required; filesystem fallbacks have been removed from the API/worker flows.
- Queue jobs default to 15-minute timeout (image) or 30-minute (batch). Track job progress via `/jobs/{job_id}` or Redis CLI.
- Improver run logs are persisted to Postgres (if configured) and mirrored to JSONL under `SCENE_RUN_LOG_DIR`.