Marimba

Typed decisions about short audio windows: ask whether someone is speaking, whether a turn has ended, which language is spoken, or whether a term occurs. Marimba uses a frozen Qwen3-ASR-0.6B backbone; one encoder pass is shared across questions on the same window.

By Charcot · Try the Space · Source code · Documentation

Only the last eight seconds are scored. Speaker questions require eight valid seconds. Term scores are uncalibrated; no real-time guarantee is claimed.

Quickstart

Use Python 3.10 or newer. Install the package and its torch backend:

python -m pip install "marimba-audio[torch]"

This example downloads a bundled synthetic speech clip from the public Space. The first model load downloads the frozen backbone and readout weights.

from huggingface_hub import hf_hub_download
from marimba import Marimba, Question

audio_path = hf_hub_download(
    "charcot-ai/marimba",
    "space/examples/03_term.wav",
    repo_type="space",
)
model = Marimba.from_pretrained(backend="torch", device="cpu", dtype="float32")
questions = [
    Question.score("Has the speaker finished?"),
    Question.noul("Is someone speaking?"),
    Question.choice("What language?", ["en", "pt", "es"]),
    Question.noul('Did someone say "marimba"?'),
    Question.choice("How many speakers?", ["0", "1", "2", "3+"]),
]
for decision in model(audio_path, questions):
    print(decision.to_dict())

Replace audio_path with your own audio file when ready. Files are mixed to mono, resampled to 16 kHz, and cropped to the last eight seconds; shorter clips are left-padded with a validity mask. See audio inputs and streaming.

Outputs

Question kind	value	Probability fields
noul (yes/no)	Boolean decision at a threshold	probability for the positive class
score	Numeric probability	probability for the positive class
choice	Selected option	probabilities over the supplied options

Decision.to_dict() returns a JSON-safe dictionary with task, kind, status, value, probability fields, threshold, window timestamps, latency_ms, and calibrated. Unavailable tasks return no prediction, not a simulated answer. Speaker tasks return None values on windows with fewer than eight valid seconds.

Language is closed-set. Without Other, probabilities are normalized over your supplied options. Adding Other aggregates the remaining supported languages; it does not detect every unknown language.

Evaluation results

These are the existing checkpoint results, not measurements from the quickstart. validated describes the evaluated task, not arbitrary prompts or deployment conditions.

Task	Output	Status	Reported result
Turn end	score	validated	AUROC 0.930 on the Smart Turn v3.2 test (Smart Turn v3.2: 0.982, in-distribution for it); 0.716 vs. 0.712 on ICSI meetings, a tie
Voice activity	noul	validated	AUROC 0.995 on AMI, 0.9996 on VoxConverse
Language	choice	validated	30 languages; closed-set accuracy 100% on FLEURS, 93% on Common Voice (300 clips each)
Term	noul	validated	AUROC 0.92 on Common Voice, 0.97 on FLEURS; scores are uncalibrated (raw fraction of greedy + four beam hypotheses)
Same speaker	noul / score	validated	AUROC 0.998 on AMI, 0.996 on VoxConverse; on par with ECAPA (0.985, 0.995)
Speaker change	noul / score	validated	AUROC 0.966 on AMI, 0.980 on VoxConverse; ECAPA 0.924, 0.962
Speaker count (0/1/2/3+)	choice	validated	Accuracy 0.778 on AMI, 0.859 on VoxConverse; pyannote 3.1 0.738, 0.836; undercounts 3+
Personal information	noul	experimental	Needs your text PII detector (none included); F1 0.904 on 200 synthetic pt-BR clips with an in-house detector

Speaker questions are per-window decisions. Same speaker compares two four-second halves, each containing one speaker; speaker change compares the last two seconds with the previous six. They do not identify people or produce diarization timelines.

On one NVIDIA L4 (bf16, batch size 1), p95 latency per question is 13 ms for turn end, 16 ms for voice activity, 161 ms for language and 2.35 s for term detection. See the measurement protocol for conditions and exclusions; these are not Space response-time guarantees.

How it works

Path	Method
Turn end and speaker questions	Linear or small MLP readouts on frozen encoder layers
Voice activity	Small question-conditioned head
Language	Qwen3-ASR's native language scores
Term	Exact token-sequence match over greedy decoding plus four beam hypotheses; score is the fraction of five hypotheses containing the term
Personal information	Greedy transcript passed to a text detector you supply

Limitations

• Decoder context above eight seconds is unvalidated. Longer recordings are not evaluated in full by one call.
• Term detection inherits ASR misses on rare words. Hypothesis fractions are not calibrated occurrence probabilities.
• Voice-activity prompts were evaluated in Portuguese and in held-out Arabic, Hindi, Korean and Russian (AUROC at least 0.98); recognized questions use the canonical prompt.
• Personal information was evaluated only on synthetic Brazilian Portuguese speech. No PII detector is included.
• Sound and scene classification is not supported; the head is at chance on CochlScene.
• Calibration shifts across domains: on ICSI, 200 labeled windows per fold reduced turn-end ECE from 0.200 to 0.011. See domain calibration.
• Not for diagnosis, identity verification, surveillance or safety-critical automation.

License and credits

Head and readout weights are Apache-2.0. Dataset licenses are separate; no dataset audio is distributed here.

Source	License	Use and pinned revision
Qwen/Qwen3-ASR-0.6B	Apache-2.0	Frozen encoder and decoder · 5eb144179a02acc5e5ba31e748d22b0cf3e303b0
pipecat-ai/smart-turn-data-v3.2-train	CC BY 4.0	Turn-end readout training data · f8c95d284595defb6b88a715c356cbc5c31b7ad1

Citation

Solano Amadori, Marimba: A System One Model for Speech. arXiv identifier: [PENDING]. This is a provisional record; replace [PENDING] once the identifier is assigned.

@misc{amadori_marimba,
  author = {Solano Amadori},
  title = {Marimba: A System One Model for Speech},
  archivePrefix = {arXiv},
  eprint = {[PENDING]},
  url = {https://huggingface.co/charcot-ai/marimba}
}

Model and weights · Code