---
library_name: espnet
pipeline_tag: any-to-any
base_model: espnet/bagpiper
language:
- en
tags:
- audio
- multimodal
- speech-language-model
- audio-understanding
- audio-generation
- text-to-audio
- espnet
- pytorch
inference: false
---

# Bagpiper SFT

Bagpiper is an 8B audio foundation model for open-ended audio understanding and
generation through rich captions. It accepts text and audio and can return text
or generate speech, music, environmental sounds, and mixtures of them.

This repository contains the paper-selected general-SFT checkpoint from
[Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://openreview.net/forum?id=FuHs64E3X6).

## Files and format

- `model.pt`: native ESPnet weight-only checkpoint, stored as
  `{"module": state_dict}`.
- `train_stage3_qwen3_base.yaml`: checkpoint-compatible model configuration,
  named consistently with the pre-trained `espnet/bagpiper` release.
- `inference_text.yaml`: audio-understanding/text-output decoding.
- `inference_audio.yaml`: text-then-audio generation decoding.
- `MANIFEST.json` and `SHA256SUMS`: provenance and integrity records.

`model.pt` contains 1,382 tensors: the 1,381 BF16 tensors from training plus the
deterministically reconstructed FP32 `vocab_weight` buffer required by the current
strict ESPnet loader. It contains no optimizer or exact-resume state. It is not a
Transformers `from_pretrained` directory and no vLLM compatibility is claimed.

## Download

```bash
hf download espnet/bagpiper-sft --local-dir bagpiper-sft
sha256sum -c bagpiper-sft/SHA256SUMS
```

## ESPnet runtime

The release was strict-loaded with:

- ESPnet PR [#6503](https://github.com/espnet/espnet/pull/6503), commit
  `1992821bfb6d68c6636d1b75340b8b2b591bc015`;
- Python 3.11.15, PyTorch 2.12.1+cu129, Transformers 5.5.4;
- FlashAttention-3 3.0.0 at source commit
  `77aacb68d194ba9af1010eda5eac3e7c0df8e6f6`; and
- one NVIDIA H100 80GB HBM3.

```bash
git clone --branch titan-pt-weightonly-init https://github.com/jctian98/espnet.git
cd espnet
git checkout 1992821bfb6d68c6636d1b75340b8b2b591bc015
python -m pip install -e '.[speechlm]'
python -m pip install --no-build-isolation \
  'flash-attn-3 @ git+https://github.com/Dao-AILab/flash-attention.git@77aacb68d194ba9af1010eda5eac3e7c0df8e6f6#subdirectory=hopper'
```

Use `espnet2/speechlm/bin/inference.py` with
`train_stage3_qwen3_base.yaml`, one of the supplied inference YAMLs, and
`model.pt`. Training backend configuration belongs to the ESPnet recipe and is
not duplicated in this model repository.

## Paper-reported results

These values are from the paper's checkpoint and evaluation protocols; they were
not rerun as part of the Hub upload.

| Evaluation | Result |
|---|---:|
| LibriSpeech test-clean WER | 2.5 |
| MMAU-Mini | 74.5 |
| MMAU | 73.1 |
| MMAR | 57.0 |
| AIR-Bench chat | 6.57 |
| AudioBench aggregate | 70.39 |
| General-model TTS WER | 2.7 |

The paper reports audio-fingerprint overlap between the SFT pool and **12.8% of
AudioBench** and **3.2% of AIR-Bench** evaluation clips. Interpret those results
with this disclosure.

## Provenance and validation

- Selected checkpoint: `step_270000/global_step269985`.
- Raw source: `JinchuanTian/bagpiper_sft` at revision
  `b11d5a0c11ad488edd04e3734d4bdff764977f57`.
- Raw SHA-256: `e0d8fdd57c8cb8c938cd61ada43b20e469d9cea33ad8450e0971f80d4f9667b6`.
- `model.pt` SHA-256:
  `50980756fad38bbefeb326ac0012e39f431ec5af8c0caaac8bd254f179f33746`.
- All 1,381 source tensors matched the converted file exactly with
  `torch.equal`; the only added key was `vocab_weight`.
- The 1,382-entry state dict passed the native ESPnet strict inference loader.

## Limitations and responsible use

Understanding can hallucinate or mistranscribe. Generated audio can contain
artifacts, wrong words, unsafe content, or unintended resemblance to people and
styles. Most reported evaluations are English and some rely on model judges. The
model has no built-in watermark, factuality verifier, copyright filter, or
speaker-consent check. Do not use it for impersonation, deceptive media,
surveillance, biometric authentication, or high-stakes automated decisions.

Training-data availability and terms are documented at
[Bagpiper SFT Data](https://huggingface.co/datasets/espnet/Bagpiper_SFT_Data).
Third-party component terms are listed in `THIRD_PARTY_NOTICES.md`.

## Citation

```bibtex
@inproceedings{anonymous2026bagpiper,
  title={Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions},
  author={Jinchuan Tian and Haoran Wang and Bo-Hao Su and Chien-yu Huang and
          Qingzheng Wang and Jiatong Shi and William Chen and Xun Gong and
          Siddhant Arora and Chin-Jou Li and Masao Someki and Takashi Maekaku and
          Keita Goto and Yusuke Shinohara and Jin Sakuma and
          Chao-Han Huck Yang and Shinji Watanabe},
  booktitle={Third Conference on Language Modeling},
  year={2026},
  url={https://openreview.net/forum?id=FuHs64E3X6}
}
```