README.md

---
library_name: espnet
pipeline_tag: any-to-any
base_model: Qwen/Qwen3-8B-Base
language:
- en
tags:
- audio
- multimodal
- speech-language-model
- audio-understanding
- audio-generation
- audio-to-text
- text-to-audio
- rich-caption
- speech
- music
- environmental-sound
- espnet
- pytorch
inference: false
---

# Bagpiper-Base

Bagpiper-Base is the pretrained 8B checkpoint from
[Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://openreview.net/forum?id=FuHs64E3X6).
It learns a bidirectional mapping between physical audio and detailed natural-language
descriptions spanning speech, music, environmental sound, and mixtures of them.

> **This is the Base checkpoint, not the instruction-tuned assistant.** Use
> [`espnet/bagpiper-sft`](https://huggingface.co/espnet/bagpiper-sft) for the
> caption-then-process model trained to answer open-ended audio requests. Bagpiper-Base
> is intended for audio-to-rich-caption probing, rich-caption-conditioned audio
> generation, and downstream initialization.

## Repository contents

| File | Description |
|---|---|
| `base.pt` | Native ESPnet weight-only checkpoint |
| `train_stage2_qwen3_base.yaml` | Reconstructed checkpoint-compatible model configuration |

`base.pt` is a PyTorch mapping with model state under the top-level `module` key. It
contains 1,382 tensors: 1,381 BF16 tensors from training and one deterministic FP32
`vocab_weight` buffer required by the current strict ESPnet loader. It contains no
optimizer, scheduler, or exact-resume state. This is not a Transformers
`from_pretrained` directory, and no vLLM compatibility is claimed.

The YAML reconstructs the architecture and vocabulary of the released checkpoint. The
original stage-2 launch referenced this filename but did not commit it, so its trainer
fields should not be treated as an archival record of every training setting.

## Download and integrity

```bash
hf download espnet/bagpiper \
  --revision d7b91ab261116973fb65cf44ff9590f6f242323b \
  --local-dir bagpiper-base

echo "c9917c1237bd44ea0ea8b41746e5ec5aecae92273538c9b6e4040d26f69a6521  bagpiper-base/base.pt" \
  | sha256sum -c -
```

## Architecture

| Component | Released configuration |
|---|---|
| Text backbone and tokenizer | [`Qwen/Qwen3-8B-Base`](https://huggingface.co/Qwen/Qwen3-8B-Base) |
| Audio input encoder | [`Qwen/Qwen3-Omni-30B-A3B-Instruct`](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) |
| Audio output tokenizer | Eight-stream, delay-interleaved [`hf-audio/xcodec-hubert-general`](https://huggingface.co/hf-audio/xcodec-hubert-general) |
| Interface | Interleaved text and audio tokens in ESPnet SpeechLM |
| Checkpoint dtype | BF16 model tensors; FP32 deterministic vocabulary-weight buffer |

The audio encoder converts input waveforms into continuous representations. X-Codec
provides discrete output tokens that can be decoded back into audio. Expanded text and
audio embeddings allow the Qwen3 backbone to model both directions through one shared
sequence interface.

## ESPnet runtime

The native consumer is `espnet2/speechlm/bin/inference.py`. Reconstruct the model with
the supplied training YAML, pair it with the task-appropriate inference YAML from the
[Bagpiper recipe](https://github.com/jctian98/espnet/tree/1992821bfb6d68c6636d1b75340b8b2b591bc015/egs2/bagpiper/speechlm1),
and pass `base.pt` through `--model-checkpoint`.

The released state dict was loaded through ESPnet's strict inference path on an NVIDIA
H100 using [ESPnet PR #6503](https://github.com/espnet/espnet/pull/6503) at commit
`1992821bfb6d68c6636d1b75340b8b2b591bc015` and Transformers 5.5.4. A complete
clean-room installation and end-to-end quickstart have not yet been validated; the Hub
Inference API is therefore disabled. The paper estimates roughly 24 GB of GPU memory for
BF16 inference, with longer inputs, decoding, or additional workers requiring more.

## Pretraining

Bagpiper-Base is the result of the first two stages of the paper's curriculum: connector
and embedding warmup followed by full pretraining. The audio encoder and codec remain
frozen during full pretraining while the backbone, connectors, and expanded vocabulary
are optimized. The paper reports training on 80 NVIDIA GH200 GPUs.

The 600B figure in the paper is a **token budget**, not a row count: 300B text-to-audio,
150B audio-to-text, and 150B text-only tokens. The audio pipeline began from roughly
422M raw audio-caption pairs, capped clips at 30 seconds, and used machine-generated rich
captions plus quality, alignment, and deduplication filters.

The associated public
[`Bagpiper_PreTrain_Data`](https://huggingface.co/datasets/espnet/Bagpiper_PreTrain_Data)
snapshot is a large artifact from this pipeline, not a claim of a complete one-to-one dump
of every raw pair or text-only source used in training.

## Paper-reported Base probes

These results evaluate the pretrained audio-rich-caption mapping. They were not rerun as
part of this Hub card update and must not be interpreted as end-to-end task-solving scores.

| Direction / probe | Result |
|---|---:|
| Audio -> rich caption -> external-LLM ASR, LibriSpeech test-clean WER | 5.0 |
| Audio -> rich caption -> external-LLM QA, MMAU-Mini accuracy | 69.0 |
| Rich caption -> speech, LibriSpeech test-clean WER | 1.8 |
| Rich caption -> general audio, MMAU-Mini FAD | 2.98 |
| Rich caption -> general audio, MMAU-Mini audio-audio CLAP | 0.55 |

For the understanding probes, an external text LLM answers using only Bagpiper-Base's
generated caption; the model does not solve the final task itself. The speech-generation
WER uses only the roughly 60% of test-clean examples whose conditioning rich captions
preserve the full transcription. Generation baselines receive transcripts or short
captions, whereas Bagpiper-Base receives full rich captions. See the paper for the complete
protocol, comparisons, cycle-consistency results, and text-capability probes.

## Intended use

- Research on bidirectional audio-rich-caption representations.
- Rich-caption generation for analysis or as input to a separately evaluated downstream
  system.
- Rich-caption-conditioned generation of speech, music, sound, and mixtures.
- Initialization for downstream ESPnet SpeechLM fine-tuning.

For general instruction following and open-ended audio question answering or generation,
start from [`espnet/bagpiper-sft`](https://huggingface.co/espnet/bagpiper-sft). For
speech-specialized synthesis, see
[`espnet/bagpiper-tts-sft`](https://huggingface.co/espnet/bagpiper-tts-sft).

## Limitations and responsible use

Rich captions are an explicit information bottleneck: omissions, hallucinations, or
incorrect acoustic details can propagate into downstream answers or generated audio.
Generated audio may contain artifacts, wrong words, unsafe content, or unintended
resemblance to people or styles. Most headline probes are English-focused, some depend on
model-based evaluation, and performance outside the evaluated distributions is unknown.

Training sources and machine-generated captions may contain copyrighted, identifiable,
sensitive, biased, or offensive material. Do not use this checkpoint for impersonation,
deceptive media, surveillance, biometric authentication, or high-stakes automated
decisions. Obtain appropriate rights and consent, disclose synthetic media where relevant,
and apply human review and domain-specific safeguards.

No blanket model-weight license is declared in this card. The checkpoint does not override
the terms of its upstream components or training sources; review those terms for the
intended use and jurisdiction.

## Provenance and validation

- Selected checkpoint: `step_260000/global_step259988`.
- Raw source: `JinchuanTian/Bagpiper_checkpoints` at revision
  `f226d242bb0d595b02e2289f8784693e88470c18`.
- Raw model-state SHA-256:
  `58c64ff430e2d54170403b77471c28e49b432669a9b264f41efaf0bedd52fd7a`.
- Released `base.pt`: 18,188,844,317 bytes; SHA-256
  `c9917c1237bd44ea0ea8b41746e5ec5aecae92273538c9b6e4040d26f69a6521`.
- Hub revision before this card update:
  `d7b91ab261116973fb65cf44ff9590f6f242323b`.
- The converter preserves all 1,381 source tensors and adds only the deterministic
  `vocab_weight` buffer; the resulting state dict passes the native strict ESPnet loader.

## Related resources

- [Project page and audio gallery](https://bagpiper-cmu.github.io/)
- [Accepted COLM 2026 paper](https://openreview.net/forum?id=FuHs64E3X6)
- [Bagpiper ESPnet recipe](https://github.com/jctian98/espnet/tree/1992821bfb6d68c6636d1b75340b8b2b591bc015/egs2/bagpiper/speechlm1)
- [Bagpiper SFT model](https://huggingface.co/espnet/bagpiper-sft)
- [Bagpiper-TTS SFT model](https://huggingface.co/espnet/bagpiper-tts-sft)
- [Bagpiper pretraining data](https://huggingface.co/datasets/espnet/Bagpiper_PreTrain_Data)
- [Bagpiper SFT data](https://huggingface.co/datasets/espnet/Bagpiper_SFT_Data)

## Citation

```bibtex
@inproceedings{tian2026bagpiper,
  title     = {Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions},
  author    = {Jinchuan Tian and Haoran Wang and Bo-Hao Su and Chien-yu Huang and
               Qingzheng Wang and Jiatong Shi and William Chen and Xun Gong and
               Siddhant Arora and Chin-Jou Li and Masao Someki and Takashi Maekaku and
               Keita Goto and Yusuke Shinohara and Jin Sakuma and
               Chao-Han Huck Yang and Shinji Watanabe},
  booktitle = {Third Conference on Language Modeling},
  year      = {2026},
  url       = {https://openreview.net/forum?id=FuHs64E3X6}
}
```