niloy629/bangla-gemma-s2s

Bangla Gemma speech-to-speech + lipsync — recreate this system

This folder is the minimal tree that rebuilds the working assistant:

audio in → Gemma 4 E4B → spoken Bangla reply + talking-head video

Architecture write-ups (mechanisms and why, not scores):

• docs/speech_to_speech_architecture.tex
• docs/speech_lipsync_architecture.tex

---

Hugging Face

What	Repo	Link
Our weights + assets + this code	model	https://huggingface.co/niloy629/bangla-gemma-s2s
22 735-row training set (slim)	dataset	https://huggingface.co/datasets/niloy629/bangla-gemma-s2s-data

Download:

hf download niloy629/bangla-gemma-s2s --local-dir ./bangla-gemma-s2s
hf download niloy629/bangla-gemma-s2s-data --repo-type dataset --local-dir ./bangla-gemma-s2s-data

---

Models

Already on Hugging Face (do not re-upload)

Role	Id / loader
Language model + audio tower	google/gemma-4-E4B-it — must load as Gemma4ForConditionalGeneration
Mimi vocoder	Kyutai Moshi via moshi.models.loaders.get_mimi() (loaders.DEFAULT_REPO / MIMI_NAME)
Wav2Vec2 teacher (only if you re-extract lipsync targets)	facebook/wav2vec2-base-960h
Silero VAD	loaded at runtime by silero_vad

Uploaded here (trained or vendored for one-stop restore)

All at the root of https://huggingface.co/niloy629/bangla-gemma-s2s :

File	What	Trained by us?
ctc.pt	Gemma char states → mHuBERT units (contextual-state CTC)	yes (runs/ctc_ctx_scratch)
runlength.pt	Per-unit 50 Hz hold lengths	yes (runs/runlen)
units2mimi.pt	50 Hz units → 12.5 Hz Mimi latents	yes (runs/units2mimi_bridge)
gemma_unitalk_adapter.pt	Gemma chars + durations → wav2vec 12×768	yes (runs/gemma_w2v_realdur/phase2)
unitalk.pt	PersonaPlex UniTalk body (Helium 4096 → wav2vec layers)	warm-start, not trained from scratch
imtalker_generator.ckpt	IMTalker flow-matching motion generator	upstream, frozen
imtalker_renderer.ckpt	IMTalker talking-head renderer	upstream, frozen

Session assets (repo root):

File	Use
ref_cropped.png	Identity portrait (already square)
idle_clip.mp4	Looping idle face between turns

---

Dataset

https://huggingface.co/datasets/niloy629/bangla-gemma-s2s-data

22 735 utterances (21 098 train / 1 637 val): IndicTTS speech + 10 k synthetic rows.

The Hub copy is a handful of tar shards (HF does not like 120k tiny .pt files). After download:

cd bangla-gemma-s2s-data
for t in audio gemma_ctx durations durations_aligned mimi_latents hubert_units metadata; do
  tar -xf ${t}.tar
done

Manifest: metadata/manifest.jsonl (paths relative to the extracted folders).

Folder	Contents	Why it is here
audio/	16 kHz wav	re-extract features; listen
gemma_ctx/	generation-time Gemma char states (2560-d)	train CTC + lipsync adapter
durations/	per-character durations	length regulator
durations_aligned/	CTC forced-align durations (IndicTTS rows)	prefer these over uniform T/n
hubert_units/	mHuBERT unit sequences	train CTC + run-length + bridge
mimi_latents/	512-d continuous Mimi latents	train the bridge

Not uploaded (huge, regenerable):

• Isolated gemma_path states — superseded by gemma_ctx
• 12-layer wav2vec teachers (~123 GB) — re-run extract_w2v_teacher.py / fill_w2v_teacher.py on audio/ if you retrain the face adapter
• OpenSLR dumps, GLM experiments, GAN/mel runs, direct Gemma→Mimi collapse runs

---

Essential code (code/)

Live server

• duplex_server.py + duplex_ui.html — conversational A/V (VAD, barge-in, H.264/Opus)
• voice_server.py + voice_ui.html — shared engine / push-to-talk
• voice_loop_audio.py, stream_voice.py — audio-in / streaming decode
• tournament_7sent.py — load_ctc_stack / synth_ctc
• face_inproc.py, gemma_face_driver.py, make_idle_clip.py

Train (voice)

• train_gemma2units_ctc.py
• train_runlength.py
• train_units2mimi_bridge.py
• train_gemma2mimi_cont.py — shared helpers (decode_continuous, etc.)
• extract_gemma_context.py, extract_gemma_ljspeech.py
• align_char_durations.py
• precompute_hubert_units.py, precompute_mimi_latents.py, fill_mimi_latents.py

Train (lipsync)

• gemma_wav2vec_adapter.py
• train_gemma_wav2vec_adapter.py
• run_gemma_w2v_pipeline.sh
• extract_w2v_teacher.py, fill_w2v_teacher.py

You still need an IMTalker code checkout for UniTalkWav2VecAdapter and IMTRenderer (weights are in the model repo). Point IMTalker/ at the original generator/renderer tree.

Inference command (after download)

python duplex_server.py --host 0.0.0.0 --port 8999 \
  --ctc-ckpt ctc.pt \
  --bridge-ckpt units2mimi.pt \
  --runlength-ckpt runlength.pt \
  --adapter-ckpt gemma_unitalk_adapter.pt \
  --unitalk-ckpt unitalk.pt \
  --generator-ckpt imtalker_generator.ckpt \
  --renderer-ckpt imtalker_renderer.ckpt \
  --face-ref ref_cropped.png \
  --idle-clip idle_clip.mp4 \
  --face-inproc

---

What this is not

• Not one giant audiovisual network. Gemma writes once; voice and face are two readouts of the same character states.
• Not a drop-in replacement for Whisper-then-TTS. Gemma hears; quality of understanding is the audio tower.
• Do not load Gemma with AutoModelForCausalLM — that drops the audio tower silently.