Abstract
Marimba uses frozen Qwen3-ASR features and lightweight readouts to make lexical, turn-end, language, speaker-change, and personal-information decisions from speech, achieving strong detection and calibration on conversational and read-speech benchmarks.
Speech applications require decisions about term presence, turn completion, and language within an audio window. We present Marimba, an interface for typed speech decisions built on frozen Qwen3-ASR 0.6B features. Lexical questions use zero-shot scoring of ASR transcripts; non-lexical questions use a question-conditioned head and a separate layer-18 turn-end readout. Language identification uses the ASR model's own language scores. On private conversational test sets, decoder AUROC is 0.80-0.92; on public read speech it is 0.92-0.97. Speech Commands four-way accuracy is 0.93; binary lexical ECE is at most 0.08. Personal-information detection, which passes the greedy transcript to a private, curated PII pipeline, reaches item-level F1 0.904 on synthetic Brazilian Portuguese speech (1.000 on reference transcripts). Layer-12 speaker readouts detect speaker change with AUROC 0.966-0.980 on AMI and VoxConverse, above an ECAPA baseline, and count speakers (0, 1, 2, or 3+) more accurately than pyannote 3.1. The turn-end readout reaches AUROC 0.930 versus Smart Turn v3.2 at 0.982 on Smart Turn's own test set. On a subset of ICSI meetings labeled by next speaker, AUROC is 0.716 versus 0.712; the paired confidence interval includes zero.
Community
Marimba is a System One model for speech: ask typed questions about live audio, get typed answers back every 250 ms.
Is someone speaking? Did the turn end? Which language? Did they say "refund"? Did the speaker change? How many speakers?
Answers come back as yes/no, a choice, or a score.
- One frozen Qwen3-ASR-0.6B serves every question, so it runs on a laptop CPU.
- Counts speakers better than pyannote 3.1 on AMI and VoxConverse; 100% language ID on 300 FLEURS clips.
- Open weights, benchmark and code (Apache-2.0).
Latency (p50 per question, 8 s window, batch 1):
| Question | Mac M4 Pro CPU | NVIDIA L4 GPU |
|---|---|---|
| Turn end | ~100 ms | ~12 ms |
| Voice activity | ~105 ms | ~14 ms |
| Language | ~1.3 s | ~150 ms |
| Term ("did they say X?") | not measured | ~1.4 s |
Turn end and voice activity are fast on CPU; term detection needs ASR decoding, so a GPU helps there. Protocol: https://github.com/Charcot-AI/marimba-api/blob/main/docs/benchmarks.md
Try it live: https://huggingface.co/spaces/charcot-ai/marimba
Model: https://huggingface.co/charcot-ai/marimba
Benchmark: https://huggingface.co/datasets/charcot-ai/marimba-bench
Code: https://github.com/Charcot-AI/marimba-api
Author here, ask me anything!
Get this paper in your agent:
hf papers read 2611.18330 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 1
charcot-ai/marimba-bench
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper