examples/direct/README.md

Direct image-to-splat example

This is the shortest reusable integration for turning one browser image into both a 3D Gaussian Splatting PLY file and a browser .splat file.

The example is framework-neutral. It keeps the input image in browser memory, reports progress, supports cancellation, exports both formats, and releases the generated scene after encoding.

Alpha status: generation currently runs in a WebGPU-capable browser and the package is still experimental. Use an alpha-bearing image or a prepared RGB-on-black image. A normal opaque JPEG requires a caller-provided local background-removal adapter.

Files

File	Purpose
image-to-splat.ts	Reusable image-in/files-out adapter
browser.ts	File picker, compatibility check, progress, cancellation, and downloads
index.html	Minimal Vite-compatible page
model-config-server.mjs	Optional Node endpoint for public model configuration

The Node script serves configuration only. It does not perform inference and does not receive user images.

Runtime architecture

User image (File/Blob)
        │
        ▼
Browser WebGPU + module worker
        │
        ▼
GaussianScene
        │
        ├── binary PLY Blob
        └── browser .splat Blob

The model manifest and ONNX artifacts are downloaded by the browser from modelBaseUrl. The image is not uploaded by this package. A backend can render the model URL into a page or return it from an API, but generation must remain behind the browser/client boundary.

Run in an existing Vite app

The package is not published to npm yet. From this repository, build the two public workspace packages and pack them into a temporary directory:

pnpm install --frozen-lockfile
pnpm --filter @ai3d/gaussian-scene build
pnpm --filter @ai3d/triposplat-webgpu build
mkdir -p /tmp/triposplat-packs
pnpm --filter @ai3d/gaussian-scene pack --pack-destination /tmp/triposplat-packs
pnpm --filter @ai3d/triposplat-webgpu pack --pack-destination /tmp/triposplat-packs

In the consuming Vite application, install the generated tarballs:

npm install /tmp/triposplat-packs/ai3d-gaussian-scene-*.tgz \
  /tmp/triposplat-packs/ai3d-triposplat-webgpu-*.tgz

After publication, use:

npm install @ai3d/triposplat-webgpu

Copy these files into the client-side portion of the application:

examples/direct/image-to-splat.ts
examples/direct/browser.ts
examples/direct/index.html

Set the model directory in index.html:

<meta
  name="triposplat-model-base-url"
  content="https://cdn.example.com/triposplat/v1/"
/>

The URL must point to the directory containing manifest.json. The manifest must resolve every graph and external-data artifact. Do not point this value at an application API route that proxies multi-gigabyte model files.

Start the Vite app normally and open the page in a current Chrome or Edge build with WebGPU enabled. Choose an alpha-bearing PNG/WebP and click Generate. The first run downloads and verifies the model artifacts; the current fp32 manifest declares approximately 6.465 GB.

The example defaults to the 4-step schedule and produces:

• triposplat.ply — binary little-endian 17-float 3DGS PLY;
• triposplat.splat — 32-byte browser splat records;
• Gaussian count, output sizes, and elapsed time in the page status.

Use the adapter directly

For an application that already has its own UI, import only the reusable adapter:

import { ImageToSplatGenerator } from './image-to-splat'

const generator = new ImageToSplatGenerator({
  modelBaseUrl: 'https://cdn.example.com/triposplat/v1/',
  manifestUrl: 'manifest.json',
  executionProviders: ['webgpu'],
  cache: 'opfs',
})

const compatibility = await ImageToSplatGenerator.checkCompatibility()
if (!compatibility.supported) {
  throw new Error(compatibility.blockers.join('\n') || 'WebGPU is unavailable.')
}

const output = await generator.generate(file, {
  steps: 4,
  guidanceScale: 3,
  gaussianCount: 262144,
  seed: 42,
  onProgress: ({ stage, message, progress }) => {
    console.info(stage, message, progress)
  },
})

await saveBlob(output.ply, 'object.ply')
await saveBlob(output.splat, 'object.splat')
await generator.dispose()

saveBlob is application code. A browser download helper can use URL.createObjectURL; a viewer can use the object URL or blob.arrayBuffer(); a deliberate remote-storage integration can upload the blob after generation.

Do not retain the internal GaussianScene in UI state. The adapter exports both files and disposes the scene in a finally block.

Input contract

The adapter accepts the browser image sources supported by TripoSplatWebGPU.generate():

• File or Blob;
• ImageBitmap;
• ImageData;
• HTMLImageElement;
• HTMLCanvasElement;
• OffscreenCanvas.

Use a transparent subject image when possible. The preprocessing pipeline uses the alpha channel to isolate the subject and composites the result onto the required black background.

A raw opaque photo does not contain a foreground mask. It raises BackgroundRemovalRequiredError unless the generator is configured with a validated browser-local removeBackground callback. Do not set inputIsPrepared: true as a generic bypass; that flag is only for the exact official 1024×1024 RGB-on-black representation.

A remover can be provided at construction time:

const generator = new ImageToSplatGenerator({
  modelBaseUrl,
  removeBackground: async (resizedRgba, { signal } = {}) => {
    return runLocalBackgroundRemover(resizedRgba, signal)
  },
})

The remover must return same-size straight-alpha RGBA pixels and remain local to the browser. This repository does not bundle a qualified background-removal model.

Output contract

generate() returns:

interface ImageToSplatResult {
  count: number
  metadata: GaussianSceneMetadata
  ply: Blob
  splat: Blob
  elapsedMs: number
}

ply is the canonical binary little-endian 3DGS PLY export. splat uses the de-facto 32-byte browser record layout documented in docs/gaussian-conventions.md. The .splat extension is not a universal format identifier; confirm that the target viewer supports this layout.

The current published decoder contract emits 262,144 Gaussians. The adapter passes that value explicitly so a changed decoder contract cannot silently produce a different output shape.

Cancellation and lifecycle

Use one AbortController per user request:

const controller = new AbortController()
const pending = generator.generate(file, {
  signal: controller.signal,
  onProgress: ({ message }) => updateStatus(message),
})

cancelButton.onclick = () => controller.abort()
await pending

A cancelled in-flight WebGPU operation tears down the one-shot worker runtime. The same generator can be retried and will lazily create a clean runtime. Calls on one generator are serialized; reuse one generator instead of creating one per click.

Dispose the generator when the page, route, or component is destroyed:

window.addEventListener('pagehide', () => {
  void generator.dispose()
}, { once: true })

Revoke any object URLs created for previews or downloads. Keep only the output blobs that the host application actually needs.

Server-rendered and multi-tenant apps

The optional Node endpoint exposes only a public model URL:

TRIPOSPLAT_MODEL_BASE_URL=https://cdn.example.com/triposplat/v1/ \
  node examples/direct/model-config-server.mjs

It responds at /api/triposplat-config with:

{
  "modelBaseUrl": "https://cdn.example.com/triposplat/v1/",
  "manifestUrl": "manifest.json"
}

Mount createTripoSplatConfigHandler() in an existing Node server, or implement the same response in another backend. A client can consume it before constructing the generator:

const config = await fetch('/api/triposplat-config').then((response) => {
  if (!response.ok) throw new Error('Unable to load TripoSplat configuration.')
  return response.json() as Promise<{
    modelBaseUrl: string
    manifestUrl: string
  }>
})

const generator = new ImageToSplatGenerator(config)

Do not put CDN signing secrets in this response. For private model access, use short-lived browser-compatible signed URLs or the package fetch, manifestRequestInit, and artifactRequestInit hooks. Avoid logging signed URLs.

Model hosting checklist

Host the complete immutable model release on an object CDN or static origin:

• serve manifest.json, every graph, and every external-data sidecar;
• preserve the external-data paths embedded in each ONNX graph;
• enable CORS for the application origin;
• support byte-range requests and correct Content-Length values;
• keep versioned artifacts immutable;
• do not route model payloads through serverless functions or an application proxy;
• do not include model payloads in npm tarballs or the JavaScript bundle.

See docs/model-hosting.md for the deployment layout and cache behavior.

Platform boundaries

• Vite, Vue, Svelte, Angular, plain web: import the adapter from client code and let the bundler resolve the package worker.
• React: keep one generator in a ref/provider and call dispose() in effect cleanup. Do not put typed Gaussian arrays in React state.
• Next.js: place the adapter in a 'use client' module or a client component loaded with SSR disabled. Server Components and Route Handlers cannot run generation.
• Node/serverless/edge: serve configuration or model metadata only. These runtimes are not an inference fallback.
• Electron/native shells: use a renderer/WebView with WebGPU, module workers, Canvas/OffscreenCanvas, and sufficient storage; check compatibility on the target shell.

Coding-agent handoff

Give an implementation agent this instruction when integrating the example:

Integrate @ai3d/triposplat-webgpu using examples/direct/image-to-splat.ts.
Keep inference client-only. Accept File/Blob input, run compatibility checks,
show load and generation progress, support AbortController cancellation, expose
both PLY and .splat downloads, handle BackgroundRemovalRequiredError clearly,
and dispose the generator on teardown. Reuse one generator instance. Do not
proxy model artifacts through application compute or upload user images by
default. Preserve the package's alpha and browser/WebGPU limitations.

Acceptance checks:

• an alpha-bearing image reaches all five generation stages;
• cancellation followed by retry works without a page reload;
• both output blobs are non-empty and use the expected file extensions;
• count and metadata are exposed to the host application;
• opaque input produces a clear background-removal action;
• route/component teardown disposes the generator;
• network inspection shows only configured application/model requests unless remote output storage was deliberately added.

Validation

For the repository-level checks:

pnpm typecheck
pnpm lint
pnpm test
pnpm test:package-consumer

For a real integration, use a complete immutable manifest in current Chrome or Edge and verify a generated PLY in a compatible viewer plus a .splat viewer. Recorded structural passes do not establish final image parity, production memory safety, or support on every WebGPU device.