README.md

metadata

title: Gemma Avatar
emoji: 🗣️
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
short_description: Talk to Gemma 4 face to face, with a 3D lip-synced avatar
models:
  - google/gemma-4-31B-it
  - nvidia/parakeet-tdt-1.1b
  - Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice

Gemma Avatar

Realtime voice chat with a 3D talking-head avatar. Same AI stack as the smolagents/hf-realtime-voice Space, but the orb visualization is replaced by a TalkingHead 3D avatar with real-time audio-driven lip-sync.

Rebuild note: this Space was reconstructed after the original repo was lost. The frontend (avatar, worklets, WebSocket client, HeadAudio lip-sync) is a faithful recovery, vendored from smolagents/hf-realtime-voice and met4citizen/HeadAudio. What's genuinely new: the avatar model itself (generated by MeshForge from a real photo, not the original's Ready Player Me brunette.glb), and the backend — this rebuild ships direct-connect only (no load-balancer session proxy), since that needs backend credentials that weren't recoverable. Paste a running ws://.../v1/realtime endpoint into Settings to talk to it.

The pipeline

you speak → silero-VAD → parakeet-tdt-1.1b (STT) → gemma-4-31B-it on Cerebras → Qwen3-TTS → avatar speaks

Transport is the OpenAI Realtime GA protocol over WebSocket against an s2s (speech-to-speech) backend: mic PCM16 @ 16 kHz goes up as input_audio_buffer.append, TTS PCM16 @ 16 kHz comes back as response.output_audio.delta, transcripts stream alongside.

How the avatar works

• Rendering / body language — TalkingHead (three.js). Blinking, breathing, idle sway, moods, hand gestures, and emoji expressions are its built-in animation system.
• Lip-sync — the backend sends raw PCM only (no word timings, no visemes), so the mouth is driven from the audio itself with HeadAudio: an AudioWorklet that classifies MFCC frames into Oculus visemes (~50 ms latency, fully in-browser). The s2s playback worklet is routed into TalkingHead's audio graph (audioAnalyzerNode → audioSpeechGainNode → reverb → speakers) and HeadAudio taps the speech gain node.
• The model plays the avatar — three function tools are declared to the backend: set_mood, make_hand_gesture, make_facial_expression. Gemma calls them mid-conversation (smiles when greeting, shrugs when unsure, thumbs-up when agreeing).
• Choreography — client statuses drive presence: the avatar makes eye contact when you start talking, gestures with its hands on new utterances, and barge-in clears the playback buffer so the mouth settles instantly.

Run it

bun install
bun run dev

Open http://localhost:7860, tap the settings icon, paste a direct ws://.../v1/realtime endpoint URL, then tap Start talking.

?fakemic=1 starts a session with a silent synthetic mic (useful for testing the full loop without a microphone — trigger a reply from the console with getClient().requestResponse()).

Layout

index.ts                    Bun server: HTML import + static assets + /api/config
index.html                  App shell (avatar hero, caption, subtitles, settings)
src/app.js                  Session wiring, tool executor, UI state
src/avatar.js               AvatarStage: TalkingHead + HeadAudio + choreography
src/s2s/s2s-ws-client.js    Realtime WS client (vendored from smolagents/hf-realtime-voice;
                            orb removed, injectable output node + worklet base URL + shared-ctx close)
src/s2s/codec.js            PCM/base64 + transcript helpers (vendored, unchanged)
src/vendor/headaudio.min.mjs  HeadAudio node class (vendored, unchanged)
public/worklets/             mic-capture + audio-playback AudioWorklets (vendored, unchanged)
public/vendor/               HeadAudio worklet processor + viseme model (vendored, unchanged)
public/avatars/meshforge.glb Default avatar (generated from a photo by MeshForge)

Notes

• The avatar GLB must have a Mixamo-compatible rig plus ARKit and Oculus-viseme blend shapes — see Daankular/meshforge-avatar for how this one was generated.
• TalkingHead owns the AudioContext; the s2s client is handed head.audioCtx and never closes it.
• Everything animation-related runs on requestAnimationFrame — a backgrounded tab freezes the avatar (audio keeps playing).