0x8badbeef/molmo-audio-slap-diar

Blaster — SLAP diarization head

Private frame-level multi-speaker activity model. Emits arrival-ordered tracks so Stage C can build:

<soa> <spk1> [frames…] <spk2> [frames…] <eoa>

Hub metadata (correct typing)

Field	Value	Why
pipeline_tag	voice-activity-detection	Closest Hub tag: per-frame speech activity over ≤4 speakers (not single-label audio-classification)
library_name	pytorch	Raw encoder.pt — custom DiarEncoder, not Transformers
Tags	speaker-diarization	Full diarization (segments + speaker ids) is built in diar_segments

There is no first-class Hub speaker-diarization pipeline tag; do not use audio-classification (implies a single clip-level class).

Files

File	Role
encoder.pt	Canonical checkpoint for serve (MOLMO_DIAR_CHECKPOINT) — arrival phase ~step 28k
encoder_fork4_arrival_28k.pt	Named copy of the arrival checkpoint
encoder_fork4_pit.pt	PIT separation phase backup

Architecture

• DiarEncoder(fork_blocks=4) — trainable copy of Stage A’s lower blocks + DiarBranch transformer
• ASR trunk path stays frozen (WER preserved from Stage A)
• Output: per-frame logits over ≤4 speakers @ ~25 Hz; segments via diar_segments.segments_from_logits

Gate B (held-out mixtures)

NIST-collar DER ~18% (stated gate was 15%; accepted to proceed). Live diar costs ~4 pp on Gate D spoken attribution vs oracle markers.

Serve

export MOLMO_DIAR_CHECKPOINT=/path/to/encoder.pt
# or compose: DIAR_MODEL_DIR pointing at this repo root

Aux worker: AudioEmbeddingProcessor → process_diarized → {embeds, speakers, spk_lens}.

Load

from diar_infer import DiarEncoder
import torch
m = DiarEncoder(fork_blocks=4)
state = torch.load("encoder.pt", map_location="cpu", weights_only=False)
m.load_state_dict(state["model"], strict=True)

Related

• Stage A trunk: molmo-audio-slap-stage-a
• Serving LM: molmo-audio-serving-diar-d
• Index: molmo-audio-phase-h-restore

License

CC BY-NC-SA 4.0 (Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International).

Marketing name: Blaster. Technical Hub id remains molmo-audio-*.

• Attribution required.
• NonCommercial — no commercial use of these weights without a separate grant.
• ShareAlike — adaptations must use the same license.

Base backbone allenai/Molmo-7B-D-0924 remains Apache-2.0. Training mixes that include People’s Speech / Dolly are themselves under CC-BY-SA; this weight license is intentionally more restrictive (adds NC). See project README_TRAINING.md licensing summary.