🕵 CoLMbo

Speaker Language Model for Descriptive Profiling

Who is this speaker? CoLMbo will tell you.

📖 Paper · 💻 GitHub · 📦 TEARS Dataset · 📧 Contact

---

What is CoLMbo?

Traditional speaker recognition answers one question: "Is this speaker A or B?"

CoLMbo asks something richer: "What is this speaker like?"

Given a few seconds of audio and a natural language prompt, CoLMbo generates free-form descriptions of the speaker their gender, age, dialect, height, education, and more directly from voice alone, with no labels or metadata required.

"The speaker is a male. He is likely between 26 and 35 years old. He speaks with a New England dialect. He has a Bachelor's Degree."

CoLMbo integrates a speaker encoder with prompt-conditioned GPT-2 decoding, enabling zero-shot generalization across diverse speaker populations and datasets.

---

Table of Contents

• Quick Start
• Example Prompts
• Dataset: TEARS
• Use Cases
• Citation
• Authors

---

Quick Start

Installation

pip install transformers torch torchaudio

Load & Run

from transformers import AutoModel
import torchaudio

# Load CoLMbo
model = AutoModel.from_pretrained("cmu-mlsp/CoLMbo", trust_remote_code=True)
model.eval()

# Option A — from a waveform tensor
waveform, sr = torchaudio.load("speaker.wav")
print(model.describe(waveform, "Please describe the speaker."))

# Option B — directly from a file path
print(model.describe_file("speaker.wav", "What is the speaker's dialect?"))

⚠️ Audio should be mono, 16 kHz. The model will automatically resample if needed.

---

Example Prompts

CoLMbo accepts any natural language question about the speaker:

prompts = [
    "What is the speaker's gender?",
    "What is the speaker's age?",
    "What is the speaker's dialect?",
    "What is the speaker's race?",
    "What is the speaker's height?",
    "What is the speaker's education level?",
    "Please describe the speaker.",
]

for prompt in prompts:
    print(f"Q: {prompt}")
    print(f"A: {model.describe_file('speaker.wav', prompt)}\n")

Example output:

Q: What is the speaker's gender?
A: The speaker's gender is male.

Q: What is the speaker's age?
A: The speaker is between 26 and 35 years old.

Q: What is the speaker's dialect?
A: The speaker's dialect is from the New England region.

Q: Please describe the speaker.
A: The speaker is a male. He is likely between 26 and 35 years old.
   He speaks with a New England dialect. He has a Bachelor's Degree.

Dataset: TEARS

CoLMbo is trained and evaluated on TEARS — a large-scale speaker captioning corpus with rich per-speaker annotations.

📦 cmu-mlsp/TEARS on Hugging Face

Split	Utterances
Train	71,100
Test	44,900
Total	116,000

Each example pairs an audio file with a set of (prompt, response) pairs covering:

Attribute	Example Response
Gender	"The speaker's gender is male."
Age	"The speaker is between 26 and 35 years old."
Dialect	"The speaker's dialect is from the New England region."
Height	"The speaker is between 5'8 and 5'11."
Education	"The speaker has a Bachelor's Degree."
Description	"The speaker is a male. He is likely between 26 and 35..."

Audio sources:

• EARS — expressive, studio-quality single-speaker recordings
• TIMIT (LDC93S1) — phonetically balanced American English speech

---

Use Cases

	Use Case	Description
🔍	Speaker Profiling	Predict age, gender, dialect, education from voice
🧩	Explainable Speaker Verification	Human-readable justifications alongside verification decisions
🎙️	Speaker-Aware Captioning	Enrich ASR transcripts with speaker metadata
🔬	Zero-Shot Attribute Prediction	Query any speaker attribute without task-specific heads
🕵️	Forensic Audio Analysis	Generate structured speaker descriptions for investigative use

---

Citation

If you find CoLMbo useful in your research, please cite:

@misc{CoLMbo,
  title        = {CoLMbo: Speaker Language Model for Descriptive Profiling},
  author       = {Massa Baali and Shuo Han and Syed Abdul Hannan and Purusottam Samal and
                  Karanveer Singh and Soham Deshmukh and Rita Singh and Bhiksha Raj},
  year         = {2025},
  eprint       = {2506.09375},
  archivePrefix= {arXiv},
  url          = {https://arxiv.org/abs/2506.09375},
  primaryClass = {cs.CL}
}

---

Authors

Massa Baali · Shuo Han · Syed Abdul Hannan · Purusottam Samal · Karanveer Singh · Soham Deshmukh · Rita Singh · Bhiksha Raj

Carnegie Mellon University — Language Technologies Institute Machine Learning for Signal Processing Group

📧 mbaali@cs.cmu.edu