| # SmolDuplex: Complete Architecture & PRD |
|
|
| ## 1. SYSTEM OVERVIEW |
|
|
| **SmolDuplex** is a 139M-parameter full-duplex spoken interaction model. It simultaneously listens and speaks with 200ms turn-taking granularity. |
|
|
| | Metric | Target | |
| |--------|--------| |
| | Trainable Parameters | ~139M | |
| | Turn-taking latency | <400ms | |
| | Chunk size | 200ms | |
| | Simultaneous listen+speak | Yes | |
| | Hardware (inference) | Any 8GB+ GPU | |
| | Training cost | ~$44 cloud | |
| | Training time | ~38 GPU-hours | |
|
|
| ## 2. ARCHITECTURE DIAGRAM |
|
|
| ``` |
| USER AUDIO IN (mic) AGENT AUDIO OUT (speaker) |
| | ^ |
| v | |
| +------------+ 5 tokens/200ms +-----------+ 5 tokens/200ms +------------+ |
| | CosyVoice |-------------------->| SmolLM2 |--------------------->| CosyVoice | |
| | Encoder | | 135M | | Decoder | |
| | (frozen) | | (trained) | | (frozen) | |
| | ~70M | | ~139M | | ~80M | |
| +------------+ +-----------+ +------------+ |
| | |
| Standard causal |
| next-token prediction |
| ``` |
|
|
| ## 3. LLM BACKBONE |
|
|
| **Base**: [HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Architecture | LlamaForCausalLM | |
| | Layers | 30 | |
| | Hidden size | 576 | |
| | Attention heads | 9 (GQA, 3 KV heads) | |
| | FFN intermediate | 1536 | |
| | Activation | SiLU | |
| | Context window | 8192 tokens | |
| | Position encoding | RoPE (theta=100000) | |
| | Original vocab | 49,152 | |
| | **Expanded vocab** | **53,258** (+4096 speech + 10 special) | |
| | **Final trainable params** | **~139M** | |
|
|
| ## 4. VOCABULARY (53,258 tokens) |
|
|
| ``` |
| Tokens 0-49151: BPE text tokens (original SmolLM2) |
| Tokens 49152-53247: CosyVoice speech codes (4096 codebook) |
| Token 53248: [CHUNK] — 200ms chunk boundary |
| Token 53249: [ASR] — ASR task prefix |
| Token 53250: [TTS] — TTS task prefix |
| Token 53251: [SOS] — Start of speech |
| Token 53252: [EOS] — End of speech |
| Token 53253: [SOT] — Start of text |
| Token 53254: [EOT] — End of text |
| Token 53255: <sil_sp> — Silent speech (agent listening) |
| Token 53256: <sil_txt> — No text this chunk |
| Token 53257: <bch> — Backchannel trigger |
| ``` |
|
|
| ## 5. TOKEN SEQUENCE FORMATS |
|
|
| ### Stage 1 — ASR: |
| ``` |
| [ASR] [SOS] sp1 sp2 sp3 ... spN [EOS] [SOT] txt1 txt2 ... txtM [EOT] |
| ``` |
|
|
| ### Stage 1 — TTS: |
| ``` |
| [TTS] [SOT] txt1 txt2 ... txtM [EOT] [SOS] sp1 sp2 sp3 ... spN [EOS] |
| ``` |
|
|
| ### Stage 2 — Half-Duplex Dialogue: |
| ``` |
| [SOS] user_sp1..N [EOS] [SOT] agent_txt1..M [EOT] [SOS] agent_sp1..K [EOS] [SOS] user_sp... [EOS] ... |
| ``` |
|
|
| ### Stage 3 — Full-Duplex (200ms chunks): |
| ``` |
| Per chunk (13 tokens): |
| [CHUNK] usr_sp1 usr_sp2 usr_sp3 usr_sp4 usr_sp5 | agt_txt1 agt_txt2 | agt_sp1 agt_sp2 agt_sp3 agt_sp4 agt_sp5 |
| |
| Agent speaking, user silent: |
| [CHUNK] <sil_sp> <sil_sp> <sil_sp> <sil_sp> <sil_sp> | txt1 txt2 | sp1 sp2 sp3 sp4 sp5 |
| |
| Agent listening, user speaking: |
| [CHUNK] usr1 usr2 usr3 usr4 usr5 | <sil_txt> <sil_txt> | <sil_sp> <sil_sp> <sil_sp> <sil_sp> <sil_sp> |
| |
| Backchannel: |
| [CHUNK] usr1 usr2 usr3 usr4 usr5 | <bch> <sil_txt> | bch_sp1 bch_sp2 bch_sp3 bch_sp4 bch_sp5 |
| ``` |
|
|
| ### Context window math: |
| - 8192 tokens / 13 per chunk = 630 chunks |
| - 630 x 200ms = **126 seconds** max conversation |
|
|
| ## 6. TRAINING STAGES |
|
|
| ### Stage 1: Modality Alignment |
|
|
| | | | |
| |--|--| |
| | **Goal** | Teach speech <-> text mapping | |
| | **Tasks** | ASR + TTS (50/50 mix) | |
| | **Data** | LibriSpeech 960h (free, HuggingFace) | |
| | **Format** | `[ASR] speech -> text` and `[TTS] text -> speech` | |
| | **Init** | SmolLM2-135M pretrained (expanded vocab) | |
| | **LR** | 5e-5, cosine schedule, 500 warmup steps | |
| | **Batch** | 32 (x2 grad accum = 64 effective) | |
| | **Seq len** | 1024 | |
| | **Epochs** | 3 | |
| | **Loss mask** | Target tokens only (text for ASR, speech for TTS) | |
| | **Duration** | ~2h on A10G | |
| | **Success** | Model can do basic ASR + TTS | |
|
|
| ### Stage 2: Half-Duplex Dialogue |
|
|
| | | | |
| |--|--| |
| | **Goal** | Learn turn-based conversation | |
| | **Tasks** | User speaks -> Agent thinks (text) -> Agent speaks | |
| | **Data** | 10K synthetic conversations (LLM-generated text + CosyVoice TTS) | |
| | **Format** | Sequential: user_speech -> agent_text -> agent_speech -> ... | |
| | **Init** | Stage 1 checkpoint | |
| | **LR** | 2e-5, cosine, 200 warmup | |
| | **Batch** | 16 (x4 grad accum = 64 effective) | |
| | **Seq len** | 4096 | |
| | **Epochs** | 5 | |
| | **Loss mask** | Agent tokens only (not user speech) | |
| | **Duration** | ~3h on A10G | |
| | **Success** | Coherent responses to spoken queries | |
| |
| ### Stage 3: Full-Duplex Interaction |
| |
| | | | |
| |--|--| |
| | **Goal** | Simultaneous listen+speak, 200ms turn-taking | |
| | **Tasks** | Turn-taking, backchanneling, interruption handling | |
| | **Data** | 15K conversations chunked at 200ms (augmented from Stage 2 + fresh) | |
| | **Format** | Flattened chunks: `[CHUNK] user_sp5 agent_txt2 agent_sp5` | |
| | **Sub-stage 3a** | Three-stream (user_sp + agent_txt + agent_sp) — 6 epochs | |
| | **Sub-stage 3b** | Two-stream (user_sp + agent_sp, no text) — 4 epochs | |
| | **Init** | Stage 2 checkpoint | |
| | **LR** | 1e-5, cosine, 100 warmup | |
| | **Batch** | 8 (x8 grad accum = 64 effective) | |
| | **Seq len** | 8192 | |
| | **Epochs** | 10 total | |
| | **Loss mask** | Agent tokens only | |
| | **Duration** | ~8h on A10G | |
| | **Success** | <400ms turn-taking, natural backchannels, handles interrupts | |
| |
| ## 7. DATA GENERATION |
| |
| ### Stage 1 (download only): |
| ```bash |
| # LibriSpeech — free, ready |
| from datasets import load_dataset |
| ds = load_dataset("openslr/librispeech_asr", "all") |
| # Then tokenize audio with CosyVoice encoder (~10h on A10G) |
| ``` |
| |
| ### Stage 2 (synthetic generation): |
| ```python |
| # Step 1: Generate 10K text dialogues with any LLM |
| # Step 2: Synthesize each turn with CosyVoice TTS |
| # Step 3: Tokenize all audio with CosyVoice encoder |
| # Step 4: Format as half-duplex sequences |
| # Time: ~10h total, Cost: ~$14 |
| ``` |
| |
| ### Stage 3 (augmentation): |
| ```python |
| # Step 1: Take Stage 2 dialogues |
| # Step 2: Chunk into 200ms segments |
| # Step 3: Inject backchannels (p=0.10 per chunk during user speech) |
| # Step 4: Inject interruptions (p=0.05 per agent turn) |
| # Step 5: Add natural pauses (1-4 chunks between turns) |
| # Step 6: Generate 5K fresh conversations for diversity |
| # Step 7: Flatten into [CHUNK] user_sp5 agent_txt2 agent_sp5 format |
| # Time: ~5h for fresh generation, ~1h for augmentation |
| ``` |
| |
| ## 8. INFERENCE |
| |
| ### Latency Budget (per 200ms chunk): |
| ``` |
| Audio capture: ~1ms |
| CosyVoice encode: ~10ms |
| LLM forward (7 tok): ~20-30ms (RTX 3060) |
| CosyVoice decode: ~15ms |
| Audio playback: ~1ms |
| ───────────────────────────── |
| TOTAL: ~50-60ms ✓ (within 200ms budget) |
| ``` |
| |
| ### Memory: |
| ``` |
| Model (bf16): 270 MB |
| KV cache (8192 ctx): ~1.5 GB |
| CosyVoice enc+dec: ~300 MB |
| ───────────────────────────── |
| TOTAL VRAM: ~2.5 GB (fits on any modern GPU) |
| ``` |
| |
| ### Realtime loop: |
| ```python |
| while True: |
| user_audio = capture_200ms() # 200ms mic input |
| user_tokens = cosyvoice_encode(user_audio) # -> 5 tokens |
| context.extend([CHUNK] + user_tokens) # append to context |
| agent_tokens = llm.generate(context, max_new=7) # predict 7 tokens |
| context.extend(agent_tokens) # append generated |
| agent_sp = agent_tokens[2:] # last 5 = speech |
| if not all_silent(agent_sp): |
| play(cosyvoice_decode(agent_sp)) # output audio |
| ``` |
| |
| ## 9. COSTS |
|
|
| | Item | GPU Hours | Cost (A10G) | |
| |------|-----------|-------------| |
| | Tokenize LibriSpeech | 10h | $11 | |
| | Generate Stage 2 text | 2h | $2 | |
| | CosyVoice TTS (Stage 2) | 8h | $9 | |
| | CosyVoice TTS (Stage 3 extra) | 5h | $6 | |
| | Train Stage 1 | 2h | $2 | |
| | Train Stage 2 | 3h | $3 | |
| | Train Stage 3 | 8h | $9 | |
| | **TOTAL** | **~38h** | **~$44** | |
|
|
| Own RTX 4090: ~$2 electricity. |
|
|
| ## 10. EVALUATION METRICS |
|
|
| | Metric | Target | Stage | |
| |--------|--------|-------| |
| | ASR WER (LibriSpeech dev) | <30% | 1 | |
| | TTS intelligibility | >70% words correct | 1 | |
| | Response relevance (GPT-4 judge) | >3.0/5.0 | 2 | |
| | Turn-taking latency | <400ms | 3 | |
| | Backchannel F1 | >0.5 | 3 | |
| | Interruption yield rate | >80% within 400ms | 3 | |
| | FD-Bench score | Report (no target) | 3 | |
|
|
| ## 11. REFERENCES |
|
|
| - [OmniFlatten (2410.17799)](https://arxiv.org/abs/2410.17799) — Core arch, proven at 500M |
| - [SyncLLM (2409.15594)](https://arxiv.org/abs/2409.15594) — Time-sync mechanism |
| - [Chronological Thinking (2510.05150)](https://arxiv.org/abs/2510.05150) — Think-while-listen, 1.5B |
| - [Full-Duplex-Bench (2503.04721)](https://arxiv.org/abs/2503.04721) — Evaluation standard |
| - [Sommelier (2603.25750)](https://arxiv.org/abs/2603.25750) — Data processing pipeline |
| - [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) — Speech tokenizer/detokenizer |
| - [SmolLM2 (2502.02737)](https://arxiv.org/abs/2502.02737) — LLM backbone |
|
|
