docs/agent-integration.md

Direct and agent integration

Use this guide when adding local image-to-3D generation to an existing web product. The stable application boundary is intentionally small:

Blob/File/ImageBitmap/ImageData/canvas
  -> WebGPU browser generation
  -> { ply: Blob, splat: Blob, metadata, count }

The reusable implementation is examples/direct/image-to-splat.ts. It accepts browser image sources, exports both portable formats, and disposes the large in-memory Gaussian scene deterministically.

Runtime boundary: inference runs in a current WebGPU-capable Chromium browser. Node.js, serverless functions, React Server Components, and edge workers may provide configuration or host assets, but they must not call generate(). Do not upload a user's image to a backend unless the product explicitly requires and discloses that behavior.

Five-minute onboarding

1. Install the alpha tarballs as described in the package README, or install @ai3d/triposplat-webgpu after publication.
2. Copy examples/direct/image-to-splat.ts into the client-side portion of the app.
3. Host the complete, immutable model manifest and artifacts on an object CDN. This is currently about 6.465 GB; do not proxy it through an application function.
4. Construct one ImageToSplatGenerator per browser tab and reuse it for queued requests.
5. Pass an alpha-bearing image, await generate(), and save or upload the returned PLY/.splat blobs according to the host product's policy.
6. Call dispose() when the page/component is torn down.

import { ImageToSplatGenerator } from './image-to-splat'

const generator = new ImageToSplatGenerator({
  modelBaseUrl: 'https://cdn.example.com/triposplat/v1/',
  cache: 'opfs',
})

const report = await ImageToSplatGenerator.checkCompatibility()
if (!report.supported) throw new Error(report.blockers.join('\n'))

const output = await generator.generate(file, {
  steps: 4,
  seed: 42,
  onProgress: ({ message }) => console.info(message),
})

saveBlob(output.ply, 'object.ply')
saveBlob(output.splat, 'object.splat')
await generator.dispose()

Inputs

The package accepts Blob, File, ImageBitmap, ImageData, HTMLImageElement, HTMLCanvasElement, and OffscreenCanvas. Prefer a File/Blob from a local picker or drag-and-drop surface; no base64 conversion is needed.

An alpha-bearing subject image works directly. A normal opaque JPEG does not contain a foreground mask and fails with BackgroundRemovalRequiredError. Until a qualified remover is bundled, either:

• request a transparent PNG/WebP from the user;
• provide a validated browser-local removeBackground callback in the constructor; or
• pass inputIsPrepared: true only for the exact preprocessed 1024×1024 opaque RGB-on-black representation.

Never set inputIsPrepared merely to bypass the error for a raw photograph.

Outputs

ImageToSplatResult owns two browser Blobs and JSON-compatible metadata:

Field	Meaning
ply	Binary little-endian 17-float 3DGS PLY
splat	De-facto 32-byte browser .splat records
count	Number of exported Gaussians; currently 262,144
metadata	Coordinate, encoding, model revision, seed, and generation settings
elapsedMs	Generation plus both encodes

Use blob.arrayBuffer() for APIs that need bytes, URL.createObjectURL(blob) for browser downloads/viewers, or send the blob to object storage only when remote persistence is an intentional product feature. The .splat extension is ambiguous across tools; consumers must support the layout documented in Gaussian conventions.

Platform recipes

Vite, Vue, Svelte, Angular, and plain bundled apps

Import the adapter only from client code. Keep one instance in application scope, generate from event handlers, and dispose it on application teardown. Vite resolves the package worker automatically.

React

Create the generator in a client component or provider, retain it in a ref, and dispose it in the effect cleanup. Keep output blobs or object URLs in state; never put the canonical scene's typed arrays in React state.

Next.js

Place the adapter behind a file containing 'use client', or dynamically import the client component with SSR disabled. A Route Handler may return the public model URL, but generation cannot run in a Server Component or Route Handler.

Server-rendered and multi-tenant products

Expose only non-secret public configuration to the client. examples/direct/model-config-server.mjs shows a dependency-free Node endpoint:

TRIPOSPLAT_MODEL_BASE_URL=https://cdn.example.com/triposplat/v1/ \
  node examples/direct/model-config-server.mjs

Mount createTripoSplatConfigHandler() in an existing Node server, or implement the same JSON response in another backend. Do not expose signing secrets. Use the package's custom fetch, request-init options, or short-lived signed artifact URLs when model access is private.

Native mobile and desktop shells

Run the client integration inside a Chromium WebView that exposes WebGPU, module workers, Canvas/OffscreenCanvas, and adequate storage. A native Node/Electron main process cannot substitute for the browser renderer. Always execute checkCompatibility() on the target shell.

Model hosting requirements

• Preserve every external-data path exactly as declared by its ONNX graph and manifest.
• Serve immutable versioned files from an object CDN with CORS, byte ranges, and correct content lengths.
• Allow the app origin to GET the manifest, graph files, and sidecars.
• Prefer cache: 'opfs'; the first run downloads and verifies the declared artifacts.
• Do not bundle model payloads into npm packages, serverless functions, or the application JavaScript bundle.
• Treat the model URL as public unless access is enforced with short-lived browser-compatible authorization.

See Model hosting for deployment details.

Error and lifecycle contract

• Call checkCompatibility() before a large model download and present its blockers to the user.
• Use one AbortController per request. Cancellation terminates unsafe in-flight worker state; the same generator can retry and lazily creates a clean worker.
• BackgroundRemovalRequiredError is an input/action error, not a WebGPU failure.
• Show load and generation progress; initial verified artifact loading can be long.
• Generation calls on one model are serialized. Implement a visible queue or cancel/replace policy rather than creating concurrent model instances.
• Keep returned blobs only as long as needed. Revoke object URLs created for previews/downloads.
• Call dispose() exactly once during teardown; it is idempotent.

Instructions for coding agents

When adapting this example to another codebase, preserve these invariants:

1. Put every import and TripoSplatWebGPU call behind the platform's browser/client boundary.
2. Accept native browser image objects; do not invent a base64-only API.
3. Run compatibility checking and expose cancellation plus progress.
4. Return or download both PLY and .splat unless the product explicitly requests one format.
5. Export before disposing the scene. Prefer the supplied adapter, which already guarantees this ordering.
6. Reuse one generator and dispose it on teardown; do not construct a model per render or request.
7. Do not claim opaque JPEG support unless a real local background-removal callback is configured.
8. Do not proxy the 6+ GB model through application compute, and do not upload input images by default.
9. Keep alpha-status and model-quality warnings visible while the package remains pre-release.
10. Validate the final integration in a real target browser against a complete immutable manifest.

A useful implementation request for another agent is:

Integrate @ai3d/triposplat-webgpu using docs/agent-integration.md and
examples/direct/image-to-splat.ts. Keep inference client-only, reuse one
ImageToSplatGenerator, accept File/Blob input, show compatibility/progress/
cancellation, expose both PLY and .splat downloads, handle opaque-input errors,
and dispose resources on teardown. Do not proxy models or upload user images.

Acceptance checks:

• an alpha-bearing image reaches all five generation stages;
• cancel followed by retry works without a page reload;
• both non-empty blobs download with the expected extensions;
• the reported count and metadata are available to the host app;
• opaque input gives a clear background-removal action;
• page/component teardown disposes the generator;
• network inspection shows only configured app/model requests unless remote output storage was deliberately added.

This remains an alpha integration. Review Current status before making production quality, parity, memory, or cross-browser claims.