omote-ai/lam-a2e

LAM A2E — Lip Animation Model

Wav2Vec2-based model fine-tuned for real-time audio-to-expression (A2E) inference. Outputs 52 ARKit blendshape weights at 30fps for avatar lip sync.

Part of the @omote/core SDK - the first open source client-side AI character SDK for the web.

Variants

Desktop (WebGPU)

File	Size	Description
model_fp16.onnx	385 KB	ONNX graph (external data format)
model_fp16.onnx.data	192 MB	fp16 weights

• Backend: WebGPU (preferred) / WASM
• Input: 16kHz audio, fixed 16000 samples
• Output: 52 ARKit blendshapes + CTC ASR (dual-head)
• Precision: Surgical mixed fp16 -- decomposed LayerNorm subgraphs kept in fp32

iOS / Safari (WASM)

File	Size	Description
lam_ios.onnx	230 KB	ONNX graph (external data format, opset 18)
lam_ios.onnx.data	192 MB	fp16 weights

• Backend: WASM only
• Input: 16kHz audio, fixed 16000 samples
• Output: 52 ARKit blendshapes (no CTC head)
• Optimizations: 32 fused native LayerNormalization ops, 54.6% node reduction (1339→608)

Quick Start

import { createA2E } from '@omote/core';

// Zero-config — auto-detects platform, uses this HuggingFace CDN
const a2e = createA2E();
await a2e.load();

const { blendshapes } = await a2e.infer(audioSamples);
// blendshapes: Float32Array[52] — ARKit weights at 30fps

ARKit Blendshapes (52)

eyeBlinkLeft, eyeLookDownLeft, eyeLookInLeft, eyeLookOutLeft, eyeLookUpLeft,
eyeSquintLeft, eyeWideLeft, eyeBlinkRight, eyeLookDownRight, eyeLookInRight,
eyeLookOutRight, eyeLookUpRight, eyeSquintRight, eyeWideRight,
jawForward, jawLeft, jawRight, jawOpen,
mouthClose, mouthFunnel, mouthPucker, mouthLeft, mouthRight,
mouthSmileLeft, mouthSmileRight, mouthFrownLeft, mouthFrownRight,
mouthDimpleLeft, mouthDimpleRight, mouthStretchLeft, mouthStretchRight,
mouthRollLower, mouthRollUpper, mouthShrugLower, mouthShrugUpper,
mouthPressLeft, mouthPressRight, mouthLowerDownLeft, mouthLowerDownRight,
mouthUpperUpLeft, mouthUpperUpRight,
browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight,
cheekPuff, cheekSquintLeft, cheekSquintRight,
noseSneerLeft, noseSneerRight, tongueOut

License

Apache 2.0