| --- |
| title: Gemma Avatar |
| emoji: 🗣️ |
| colorFrom: indigo |
| colorTo: purple |
| sdk: docker |
| app_port: 7860 |
| pinned: false |
| thumbnail: https://huggingface.co/spaces/victor/gemma-avatar/resolve/main/thumbnail.webp |
| short_description: Talk to Gemma 4 face to face, with a 3D lip-synced avatar |
| models: |
| - google/gemma-4-31B-it |
| - nvidia/parakeet-tdt-1.1b |
| - Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice |
| tags: |
| - ml-intern |
| --- |
| |
| # Gemma Avatar |
|
|
| Realtime voice chat with a 3D talking-head avatar. Same AI stack as the |
| [smolagents/hf-realtime-voice](https://huggingface.co/spaces/smolagents/hf-realtime-voice) |
| Space ([blog post](https://huggingface.co/blog/cerebras-gemma4-voice-ai)), but the |
| orb visualization is replaced by a [TalkingHead](https://github.com/met4citizen/TalkingHead) |
| 3D avatar with real-time audio-driven lip-sync. |
|
|
| ## The pipeline |
|
|
| ``` |
| you speak → silero-VAD → parakeet-tdt-1.1b (STT) → gemma-4-31B-it on Cerebras → Qwen3-TTS → avatar speaks |
| ``` |
|
|
| Transport is the OpenAI Realtime GA protocol over WebSocket against Hugging |
| Face's speech-to-speech backend: mic PCM16 @ 16 kHz goes up as |
| `input_audio_buffer.append`, TTS PCM16 @ 16 kHz comes back as |
| `response.output_audio.delta`, transcripts stream alongside. |
|
|
| ## How the avatar works |
|
|
| - **Rendering / body language** — [TalkingHead](https://github.com/met4citizen/TalkingHead) |
| (three.js). Blinking, breathing, idle sway, moods, hand gestures, and emoji |
| expressions are its built-in animation system. |
| - **Lip-sync** — the backend sends raw PCM only (no word timings, no visemes), |
| so the mouth is driven from the audio itself with |
| [HeadAudio](https://github.com/met4citizen/HeadAudio): an AudioWorklet that |
| classifies MFCC frames into Oculus visemes (~50 ms latency, fully in-browser). |
| The s2s playback worklet is routed into TalkingHead's audio graph |
| (`audioAnalyzerNode → audioSpeechGainNode → reverb → speakers`) and HeadAudio |
| taps the speech gain node. |
| - **The model plays the avatar** — three function tools are declared to the |
| backend: `set_mood`, `make_hand_gesture`, `make_facial_expression`. Gemma |
| calls them mid-conversation (smiles when greeting, shrugs when unsure, |
| thumbs-up when agreeing). |
| - **Choreography** — client statuses drive presence: the avatar makes eye |
| contact when you start talking, gestures with its hands on new utterances, |
| and barge-in clears the playback buffer so the mouth settles instantly. |
|
|
| ## Run it |
|
|
| ```bash |
| bun install |
| |
| # Pick a backend (one of): |
| LOAD_BALANCER_URL=https://… bun run dev # a speech-to-speech load balancer |
| SESSION_PROXY_URL=https://…/api bun run dev # piggyback another deployment's /api (dev) |
| bun run dev # direct mode: paste a ws:// URL in Settings |
| ``` |
|
|
| Open http://localhost:3000 and tap **Start talking**. |
|
|
| `?fakemic=1` starts a session with a silent synthetic mic (useful for testing |
| the full loop without a microphone — trigger a reply from the console with |
| `getClient().requestResponse()`). |
|
|
| ## Layout |
|
|
| ``` |
| index.ts Bun server: HTML import + /api/session proxy + static assets |
| index.html App shell (avatar hero, caption, subtitles, settings) |
| src/app.js Session wiring, tool executor, UI state |
| src/avatar.js AvatarStage: TalkingHead + HeadAudio + choreography |
| src/s2s/s2s-ws-client.js Realtime WS client (vendored from the Space; orb removed, |
| injectable output node + worklet base URL + shared-ctx close) |
| src/s2s/codec.js PCM/base64 + transcript helpers (vendored, unchanged) |
| src/vendor/headaudio.min.mjs HeadAudio node class (bundled) |
| public/worklets/ mic-capture + audio-playback AudioWorklets (vendored, unchanged) |
| public/vendor/ HeadAudio worklet processor + viseme model (runtime-loaded) |
| public/avatars/brunette.glb Default avatar (Ready Player Me; CC BY-NC 4.0 — non-commercial) |
| ``` |
|
|
| ## Notes |
|
|
| - The avatar GLB must have a Mixamo-compatible rig plus ARKit and Oculus-viseme |
| blend shapes. Ready Player Me avatars work with |
| `?morphTargets=ARKit,Oculus%20Visemes` on the GLB URL. |
| - TalkingHead owns the AudioContext; the s2s client is handed `head.audioCtx` |
| and never closes it. |
| - Everything animation-related runs on requestAnimationFrame — a backgrounded |
| tab freezes the avatar (audio keeps playing). |
|
|
| <!-- ml-intern-provenance --> |
| ## Generated by ML Intern |
|
|
| This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub. |
|
|
| - Try ML Intern: https://smolagents-ml-intern.hf.space |
| - Source code: https://github.com/huggingface/ml-intern |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "bep40/gemma-avatar-copy" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id) |
| ``` |
|
|
| For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class. |
|
|
