---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: audio-classification
tags:
  - audio
  - topic-segmentation
  - whisper
datasets:
  - retkowski/ytseg
---

# AudioSeg

AudioSeg predicts **topic boundaries directly from audio**, on a 6-second grid,
and was introduced in our paper _**Beyond Transcripts: A Renewed Perspective on
Audio Chaptering**_ ([acl](https://aclanthology.org/2026.acl-long.396/) | [arXiv](https://arxiv.org/abs/2602.08979)).
Given an audio file (e.g. a podcast, lecture, or YouTube video), the model
outputs a boundary probability for every 6-second segment, which can be
thresholded into chapter/topic boundaries.

## Architecture

- **Encoder (frozen):** [`openai/whisper-large-v3`](https://huggingface.co/openai/whisper-large-v3)
  encoder, applied to consecutive 30-second chunks of the audio.
- **Head (trained, ~32M params):**
  - a local segment transformer that pools encoder frames into one embedding per 6-second segment via a learned `[SEG]` token
  - a document encoder over the segment sequence producing per-segment boundary probabilities

Only the head weights are stored in this repository; the Whisper encoder is downloaded from `openai/whisper-large-v3` on first use.

## Usage

```bash
pip install transformers torch torchaudio
```

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("retkowski/audioseg", trust_remote_code=True)
model = model.to("cuda")

result = model.segment("episode.mp3")

result["ts_boundaries"]     # boundary timestamps in seconds, e.g. [315.0, 747.0, ...]
result["probs"]             # boundary probability per 6s segment
result["segment_size_sec"]  # 6.0
```

`segment()` also accepts a waveform tensor (`model.segment(waveform, sample_rate=sr)`), a custom decision `threshold` (default 0.5), and `batch_chunks` to control how many 30-second chunks are encoded per Whisper batch.

A reported timestamp `t` is the midpoint of the 6-second segment in which a topic change was detected, i.e. the change lies within `[t - 3, t + 3)` seconds. The start of the audio is not considered a boundary, so the first segment is never reported.

## Training

The model was trained on [YTSeg](https://huggingface.co/datasets/retkowski/ytseg), YouTube videos with human-created chapter markers, using the chapter start times as boundary supervision (binary cross-entropy on the 6-second grid). The Whisper encoder was kept frozen.

## Citing

We kindly request you to cite our corresponding paper if you use our model:

```bibtex
@inproceedings{retkowski-etal-2026-beyond,
    title = "Beyond Transcripts: A Renewed Perspective on Audio Chaptering",
    author = {Retkowski, Fabian  and
      Z{\"u}fle, Maike  and
      Nguyen, Thai Binh  and
      Niehues, Jan  and
      Waibel, Alexander},
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.396/",
    doi = "10.18653/v1/2026.acl-long.396",
    pages = "8765--8787",
    ISBN = "979-8-89176-390-6",
    abstract = "Audio chaptering, the task of automatically segmenting long-form audio into coherent sections, is increasingly important for navigating podcasts, lectures, and videos. Despite its relevance, research remains limited and text-based, leaving key questions unresolved about leveraging audio information, handling ASR errors, and transcript-free evaluation. We address these gaps through three contributions: (1) a systematic comparison between text-based models with acoustic features, a novel audio-only architecture (AudioSeg) operating on learned audio representations, and multimodal LLMs; (2) empirical analysis of factors affecting performance, including transcript quality, acoustic features, duration, and speaker composition; and (3) formalized evaluation protocols contrasting transcript-dependent text-space protocols with transcript-invariant time-space protocols. Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and current MLLMs struggle due to context limitations and weak instruction following."
}
```

## License

CC-BY-NC-4.0 (non-commercial use).