docs/api_usage.md

Scene Graph Manager API Usage

This document gives a concise, high-signal overview of the HTTP API so other services can integrate without reading the whole codebase.

Refresh Checklist (for future updates)

Use this prompt when endpoints change:

Run rg "@app" -n api/app.py and list new/modified routes. Update docs/api_usage.md with:

• Endpoint table (method, path, summary)
• Auth / header requirements
• Request & response JSON samples (curl when useful)
• Notes on query params, error behaviors, and streaming endpoints. Re-run python -m compileall api/app.py and ensure the doc still reflects reality.

Base URL

All paths below are relative to:

https://scene-graph-mgr-api.fly.dev

Authentication

• Public endpoints are unauthenticated but subject to Fly.io rate limits.
• Internal write endpoints require a shared secret header: x-internal-secret: <secret>.
• Standard error payload:

{
  "detail": "Human readable message",
  "error_code": "optional-machine-code"
}

4xx indicates caller issues (validation, missing scene). 5xx means server-side failure.

Endpoint Map

Method	Path	Summary
GET	/healthz	Liveness check.
GET	/scenes	List scenes with summary metadata (optional include_empty).
PUT	/scenes/{scene_id}	Overwrite a scene with a full graph payload.
PATCH	/scenes/{scene_id}	Apply an RFC 6902 patch to the latest graph.
POST	/scenes/{scene_id}/create	Seed an empty scene (optionally overwrite).
GET	/scenes/{scene_id}/versions/latest	Latest graph version.
GET	/scenes/{scene_id}/versions	Metadata for all versions.
GET	/scenes/{scene_id}/versions/{version_id}	Raw version record (JSON).
POST	/scenes/{scene_id}/add_image	Upload image bytes and enqueue processing.
POST	/scenes/{scene_id}/add_images_from_keys	Enqueue processing for existing S3 keys.
POST	/scenes/{scene_id}/upload_video	Upload video and queue frame extraction batch.
POST	/scenes/{scene_id}/upload_image	Browser upload → WebP resize → presigned URL.
GET	/scenes/{scene_id}/instances	List stored instances for the scene (status filters optional).
GET	/scenes/{scene_id}/objects/{obj_id}/beliefs	Latest improver belief for an object.
GET	/scenes/{scene_id}/objects/{obj_id}/instances	List stored instances for an object (status filters optional).
GET	/scenes/{scene_id}/improver/backlog	Inspect improver queue/backlog for a scene.
POST	/scenes/{scene_id}/objects/{obj_id}/instances/{instance_id}/improve_instance	Queue improver workflow for an instance.
GET	/scenes/{scene_id}/runlogs	Recent improver run logs (filterable).
GET	/scenes/{scene_id}/change-requests	Inspect change request queue (pending/applied).
GET	/scenes/{scene_id}/change-summaries	Stream applied change summaries (newest first).
GET	/scenes/{scene_id}/diff	Structural diff between two versions (machine patch & stats).
GET	/scenes/{scene_id}/diff-semantic	Semantic diff (ID-aware ops).
GET	/scenes/{scene_id}/images/presign	Generate presigned GET URL for an image key.
GET	/scenes/{scene_id}/media	List recorded media assets (images/videos).
POST	/scenes/{scene_id}/media	Upsert scene media entries supplied by workers or services.
GET	/jobs/{job_id}	Inspect queued/finished RQ jobs.
POST	/stream3r/jobs	Submit a reconstruction job (pose pointmap or model build).
GET	/stream3r/jobs/{job_id}	Inspect a Stream3R job record.
GET	/stream3r/jobs	List Stream3R jobs (filter by scene, type, status).
GET	/stream3r/jobs/{job_id}/events	Server-Sent Events feed for job lifecycle updates.
GET	/stream3r/models/{scene_id}/presign	Presign the latest model_build scene.glb for download.
POST	/internal/commit-version	Worker/internal scene commit (requires secret).
GET	/debug/s3-ping	Smoke-test S3/B2 credentials.
WS	`/ws?channel=all	{scene_id}`

---

Endpoint Details & Examples

Health & Diagnostics

GET /healthz

Returns { "ok": true, "ts": "ISO timestamp" } for uptime monitoring.

GET /debug/s3-ping

Verifies object storage connectivity using a put/delete round-trip. Good for smoke tests.

Scene Lifecycle

GET /scenes

List scenes. Optional include_empty=true includes scenes with no versions yet.

curl "https://scene-graph-mgr-api.fly.dev/scenes?include_empty=true"

Response contains summaries with counts and most recent version metadata.

PUT /scenes/{scene_id}

Replace entire scene graph.

Request body (SceneGraphEnvelope):

{
  "scene_location_id": "scene-123",
  "scene_graph": {"objects": [], "relations": []},
  "meta": {"source": "manual"}
}

Creates a new version record and broadcasts scene.put on WebSocket.

PATCH /scenes/{scene_id}

Apply RFC 6902 patch to latest graph. Optional base_version guard prevents lost updates.

{
  "scene_location_id": "scene-123",
  "json_patch": [
    {"op": "add", "path": "/objects/-", "value": {"id": "chair-1", "attributes": {}}}
  ],
  "base_version": 1728400000000
}

POST /scenes/{scene_id}/create

Seeds an empty graph. overwrite=true allows reseeding existing scenes.

Version Reads

• GET /scenes/{scene_id}/versions/latest → latest full graph.
• GET /scenes/{scene_id}/versions → list of {version_id, created_at, bytes}.
• GET /scenes/{scene_id}/versions/{version_id} → raw record (graph + metadata).

Image Upload & Processing

POST /scenes/{scene_id}/add_image

Multipart upload (file=@image.jpg). Stores bytes via S3-compatible API, seeds scene if empty, enqueues worker.tasks.process_image_for_scene. Response includes RQ job_id and filename.

POST /scenes/{scene_id}/add_images_from_keys

JSON body:

{
  "keys": ["scenes/scene-123/images/20241008/a.png"],
  "room_hint": "living_room",
  "prompt": "describe objects",
  "bounding_boxes": {"a.png": [[0.1,0.2,0.5,0.8]]}
}

Seeds scene if needed and enqueues batch worker job.

Response (AddImagesFromKeysResponse):

{
  "scene_location_id": "scene-123",
  "queued_at": "2024-10-08T14:12:05Z",
  "job_id": "rq-job-id",
  "keys": [
    "scenes/scene-123/images/20241008/a.png",
    "scenes/scene-123/images/20241008/b.png"
  ]
}

POST /scenes/{scene_id}/upload_image

For browser uploads. Accepts multipart file, resizes to WebP (max width 1024), uploads, and returns:

{
  "scene_location_id": "scene-123",
  "key": "scenes/scene-123/images/20241008/abc.webp",
  "url": "https://...presigned...",
  "width": 768,
  "height": 512,
  "bytes": 123456
}

GET /scenes/{scene_id}/images/presign

Query parameters: key (must reside under scenes/{scene_id}/images/… or scenes/{scene_id}/videos/…) and optional expires. Returns { "url": "...", "expires_in": 900 }.

Video Upload & Frame Extraction

POST /scenes/{scene_id}/upload_video

Accepts a multipart video upload (e.g., file=@walkthrough.mp4). Stores the binary in object storage and seeds an empty scene when needed. The worker enqueues process_video_for_scene, which:

• extracts WebP frames at 2fps by default (frame_interval=0.5 seconds) and stores them under scenes/{scene_id}/images/video_frames/... (or keeps the source under scenes/{scene_id}/videos/...) so the /images/presign endpoint can serve them;
• retries with a software H.264 transcode if AV1 or other codecs fail to decode on the host;
• publishes the same scene.update event stream as still-image uploads.

Response payload (UploadVideoResponse):

{
  "scene_location_id": "scene-123",
  "key": "scenes/scene-123/videos/20241008/abcd1234.mp4",
  "filename": "walkthrough.mp4",
  "size_bytes": 456789012,
  "content_type": "video/mp4",
  "queued_at": "2024-10-08T14:12:05Z",
  "job_id": "rq-job-id"
}

Clients should poll GET /jobs/{job_id} to track progress. When job.result.frame_keys is present the frame extraction succeeded.

Scene Media

GET /scenes/{scene_id}/media

Lists stored media records for a scene. Use media_type=image to filter to keyframes vs. media_type=video for source clips.

curl "https://scene-graph-mgr-api.fly.dev/scenes/scene-123/media?media_type=image&limit=50"

Response (SceneMediaListResponse) includes the total slice returned plus ISO timestamps normalized to UTC.

POST /scenes/{scene_id}/media

Upserts media entries (usually called by workers after uploading files). Payload:

{
  "entries": [
    {
      "file": "scenes/scene-123/keyframes/frame_000012.jpg",
      "media_type": "image",
      "captured_at": "2024-10-08T14:12:05Z"
    },
    {
      "file": "scenes/scene-123/videos/20241008/abcd1234.mp4",
      "media_type": "video"
    }
  ]
}

• media_type is optional—when omitted the server infers it from the filename (image vs video).
• captured_at accepts ISO-8601 strings (UTC preferred). If omitted, the server stores now().
• Existing rows are updated in-place thanks to an upsert on the file column.

Response (SceneMediaBatchResponse) summarizes how many entries were accepted:

{
  "scene_id": "scene-123",
  "accepted": 2,
  "skipped": 0,
  "files": [
    "scenes/scene-123/keyframes/frame_000012.jpg",
    "scenes/scene-123/videos/20241008/abcd1234.mp4"
  ]
}

Improver Monitoring

GET /scenes/{scene_id}/improver/backlog

Summarizes outstanding improver work for a scene. Default statuses are pending, queued, and processing, plus records that do not yet have a status. Use this to detect when the improver has drained the backlog.

Query parameters:

• limit (default 200, max 2000) — cap the number of instances returned.
• status — optional list to override which statuses count as “not yet processed.” Omit to use the defaults.
• include_missing_status (default true) — include rows with status IS NULL.

Response:

{
  "scene_id": "scene-123",
  "count": 2,
  "instances": [
    {
      "scene_id": "scene-123",
      "obj_id": "obj-1",
      "instance_id": "inst-42",
      "status": "pending",
      "status_reason": "ambient",
      "status_changed_at": "2024-10-08T14:15:06Z",
      "last_event_at": "2024-10-08T14:15:06Z",
      "created_at": "2024-10-08T13:59:40Z",
      "data": {
        "image_id": "img-998",
        "bbox_xyxy": [0.1, 0.2, 0.5, 0.8]
      }
    }
  ]
}

When count is zero the improver has no pending work for that scene.

Improver & Beliefs

POST /scenes/{scene_id}/objects/{obj_id}/instances/{instance_id}/improve_instance

Queues improver workflow. Body adheres to InstanceEvent schema. Example:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
        "scene_id": "scene-123",
        "obj_id": "sofa-1",
        "instance_id": "inst-456",
        "image_id": "scenes/scene-123/images/20241008/sofa.png",
        "bbox_xyxy": [0.1, 0.3, 0.6, 0.9]
      }' \
  https://scene-graph-mgr-api.fly.dev/scenes/scene-123/objects/sofa-1/instances/inst-456/improve_instance

Response: { "enqueued": true, "job_id": "rq-job-id" }.

The worker embeds, seeds Qdrant, and calls scene_improver.tasks.improve_scene.

GET /scenes/{scene_id}/objects/{obj_id}/beliefs

Returns latest belief payload (name Dirichlet, attribute betas, relations) or 404 if none stored.

GET /scenes/{scene_id}/instances

Returns persisted instances across every object in the scene. Query params:

• limit (1–5000, default 1000)
• status (optional, repeatable) — only include instances whose status matches one of the provided values.
• exclude_status (optional, repeatable) — omit instances whose status matches any of the provided values (e.g., exclude_status=superseded to skip reassigned instances).

Response mirrors the storage map:

{
  "scene_id": "scene-123",
  "count": 5,
  "objects": {
    "sofa-1": [{...}, {...}],
    "lamp-4": [{...}]
  }
}

GET /scenes/{scene_id}/objects/{obj_id}/instances

Returns the persisted instance rows for the object. Query params:

• limit (1–1000, default 100)
• status (optional, repeatable) — only include instances whose status matches one of the provided values.
• exclude_status (optional, repeatable) — omit instances whose status matches any of the provided values.

Example request skipping superseded items:

curl "https://scene-graph-mgr-api.fly.dev/scenes/scene-123/objects/sofa-1/instances?limit=20&exclude_status=superseded"

Response:

{
  "scene_id": "scene-123",
  "obj_id": "sofa-1",
  "count": 2,
  "instances": [
    {
      "id": "inst-456",
      "image_id": "scenes/...",
      "bbox_xyxy": [0.1,0.3,0.6,0.9],
      "captured_at": 1728401000,
      "status": "processed"
    },
    {
      "id": "inst-123",
      "image_id": "scenes/...",
      "bbox_xyxy": [0.05,0.2,0.4,0.7],
      "status": "pending"
    }
  ]
}

GET /scenes/{scene_id}/runlogs

Query params:

• limit (1–1000, default 100)
• obj_id (optional filter)
• instance_id (optional filter)

Response contains records newest-first with step, message, data, timestamps, and run IDs.

GET /scenes/{scene_id}/change-requests

Inspect the change request queue. Query params:

• state (optional: pending, applied, stale)
• limit (1–200, default 50)

Returns an array mirroring the DB record with preconditions, payload, result, and the new applied_summary field:

[
  {
    "request_id": "95e8...",
    "scene_id": "scene-123",
    "obj_id": "table-1",
    "requested_by": "belief-agent",
    "state": "applied",
    "confidence": 0.92,
    "payload": {"operations": [...]},
    "result": {"summary": "Renamed object \"wooden table\" from \"table\" to \"dining table\""},
    "applied_at": "2024-11-19T20:14:32.123Z",
    "applied_summary": "Renamed object \"wooden table\" from \"table\" to \"dining table\""
  }
]

GET /scenes/{scene_id}/change-summaries

Lightweight feed of applied change blurbs, ordered by applied_at descending. Supports limit (1–200, default 50).

curl "https://scene-graph-mgr-api.fly.dev/scenes/scene-123/change-summaries?limit=20"

[
  {
    "request_id": "95e8...",
    "scene_id": "scene-123",
    "obj_id": "table-1",
    "applied_version": 1729300042000,
    "applied_at": "2024-11-19T20:14:32.123Z",
    "summary": "Updated object \"wooden table\" attribute \"size\" from \"medium\" to \"large\""
  },
  {
    "request_id": "73f1...",
    "scene_id": "scene-123",
    "summary": "Renamed object \"wooden table\" from \"table\" to \"dining table\""
  }
]

Use this endpoint for ambient “stream of consciousness” UIs showing how the improver evolves the scene.

Diffs

GET /scenes/{scene_id}/diff

Compare two versions (from_version, to_version). Optional mode query (patch, summary, both). Returns machine patch plus stats when requested.

GET /scenes/{scene_id}/diff-semantic

ID-aware diff returning ordered operations (append/remove/replace) with summary stats.

Stream3R Reconstruction Jobs

The Stream3R API wraps the reconstruction workers (pose pointmaps and full models) and provides idempotent enqueue, polling, and event streaming.

POST /stream3r/jobs

Submit a reconstruction job. Supported job_type values are pose_pointmap and model_build. Provide at least one frame (url or path) and optional client_request_id for idempotency (subsequent calls reuse the existing job and return 200 OK).

curl -X POST https://scene-graph-mgr-api.fly.dev/stream3r/jobs \
  -H "Content-Type: application/json" \
  -d '{
        "job_type": "pose_pointmap",
        "scene_id": "scene-123",
        "frames": [
          {"url": "https://cdn.example/scene-123/frame_0000.webp"},
          {"path": "/mnt/captures/scene-123/frame_0001.png"}
        ],
        "session_settings": {"prediction_mode": "pointmap"},
        "client_request_id": "scene-123-20241008-run1"
      }'

Response (202 Accepted on first submission):

{
  "job_id": "d8f8a3fc-3aed-441c-ac78-2b953a9229bf",
  "job_type": "pose_pointmap",
  "scene_id": "scene-123",
  "status": "queued",
  "created_at": "2024-10-08T14:12:05.417Z",
  "payload": {
    "job_type": "pose_pointmap",
    "scene_id": "scene-123",
    "frames": [
      {"url": "https://cdn.example/scene-123/frame_0000.webp"},
      {"path": "/mnt/captures/scene-123/frame_0001.png"}
    ],
    "session_settings": {"prediction_mode": "pointmap"}
  }
}

GET /stream3r/jobs/{job_id}

Fetch the canonical job record (backed by Postgres). Fields include:

• status: queued, started, progress, finished, or failed
• result: worker-published artifact manifest (S3 URLs, local paths)
• error: error string when status is failed
• timestamps (created_at, started_at, completed_at)

Typical successful response:

{
  "job_id": "d8f8a3fc-3aed-441c-ac78-2b953a9229bf",
  "job_type": "model_build",
  "scene_id": "scene-123",
  "status": "finished",
  "created_at": "2024-10-08T14:12:05.417Z",
  "started_at": "2024-10-08T14:12:12.998Z",
  "completed_at": "2024-10-08T14:24:39.221Z",
  "result": {
    "model_dir": "s3://bucket/scene-123/stream3r/models/20241008",
    "summary_url": "s3://bucket/scene-123/stream3r/models/20241008/summary.json"
  },
  "error": null,
  "client_request_id": "scene-123-20241008-run1"
}

GET /stream3r/jobs

List jobs with optional filters:

• scene_id
• job_type
• status
• limit (1–200, default 50)
• offset (default 0)

Returns { "jobs": [...], "limit": 50, "offset": 0 } with the same schema as the single-job response.

GET /stream3r/jobs/{job_id}/events

Server-Sent Events feed backed by Redis Streams. Use it for near-real-time updates in browser dashboards.

curl --no-buffer \
  -H "Accept: text/event-stream" \
  "https://scene-graph-mgr-api.fly.dev/stream3r/jobs/d8f8a3fc-3aed-441c-ac78-2b953a9229bf/events"

Events are emitted as standard SSE payloads:

id: 1728409930123-0
event: progress
data: {"job_id":"d8f8a3fc-3aed-441c-ac78-2b953a9229bf","status":"progress","progress":65}

id: 1728409960456-0
event: finished
data: {"job_id":"d8f8a3fc-3aed-441c-ac78-2b953a9229bf","status":"finished","result_url":"s3://bucket/.../summary.json"}

Reconnect with the last id to resume (?last_id=<redis-stream-id>). When the worker encounters an error the stream emits event: failed with an error field.

Implementation note: the current worker stub still returns status="failed" with error="stream3r worker not implemented" until the GPU-backed handler ships. Downstream clients should surface the error text to operators and may retry later.

GET /stream3r/models/{scene_id}/presign

Fetch a presigned download URL for the most recent model_build job's scene.glb. Optional query params:

• job_id — force a specific job (must belong to the scene).
• expires — TTL in seconds (default 900, range 60–86400).

curl "https://scene-graph-mgr-api.fly.dev/stream3r/models/scene-123/presign?expires=600"

Response:

{
  "scene_id": "scene-123",
  "job_id": "d8f8a3fc-3aed-441c-ac78-2b953a9229bf",
  "key": "scenes/scene-123/stream3r/models/20241008/scene.glb",
  "url": "https://s3.amazonaws.com/...signature...",
  "expires_in": 600
}

Returns 404 if no successful model build exists or the job did not publish a GLB artifact.

Jobs & Internal Ops

GET /jobs/{job_id}

Inspect RQ job status. Returns timestamps, result payload (if finished), and truncated stack trace when failed.

POST /internal/commit-version

Worker-only commit. Body:

{
  "scene_location_id": "scene-123",
  "scene_graph": {...},
  "base_version": 1728400000000,
  "meta": {"source": "worker"}
}

Requires x-internal-secret header when enabled. Creates new version, broadcasts scene.update, and publishes on Redis pub/sub.

WebSocket Stream

WS /ws?channel=all|{scene_id}

Receives JSON events when scenes change (scene.create, scene.put, scene.patch, scene.update). Use to refresh UI state in real time.

---

Notes & Best Practices

• All timestamps are UTC ISO strings.
• Scene writes (PUT, PATCH, create, commit-version) broadcast on WebSocket and publish to Redis channel scene_events.
• Object storage paths follow scenes/{scene_id}/images/...; presign endpoint enforces this prefix.
• Scene graph payloads no longer embed instance blobs; use the /instances endpoint for that data.
• Postgres storage is required; filesystem fallbacks have been removed from the API/worker flows.
• Queue jobs default to 15-minute timeout (image) or 30-minute (batch). Track job progress via /jobs/{job_id} or Redis CLI.
• Improver run logs are persisted to Postgres (if configured) and mirrored to JSONL under SCENE_RUN_LOG_DIR.