File size: 4,959 Bytes
df2e49a 5646600 df2e49a 5646600 df2e49a 5646600 df2e49a 5646600 df2e49a 5646600 df2e49a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | ---
library_name: espnet
pipeline_tag: any-to-any
base_model: espnet/bagpiper
language:
- en
tags:
- audio
- multimodal
- speech-language-model
- audio-understanding
- audio-generation
- text-to-audio
- espnet
- pytorch
inference: false
---
# Bagpiper SFT
Bagpiper is an 8B audio foundation model for open-ended audio understanding and
generation through rich captions. It accepts text and audio and can return text
or generate speech, music, environmental sounds, and mixtures of them.
This repository contains the paper-selected general-SFT checkpoint from
[Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://openreview.net/forum?id=FuHs64E3X6).
## Files and format
- `model.pt`: native ESPnet weight-only checkpoint, stored as
`{"module": state_dict}`.
- `train_stage3_qwen3_base.yaml`: checkpoint-compatible model configuration,
named consistently with the pre-trained `espnet/bagpiper` release.
- `inference_text.yaml`: audio-understanding/text-output decoding.
- `inference_audio.yaml`: text-then-audio generation decoding.
- `MANIFEST.json` and `SHA256SUMS`: provenance and integrity records.
`model.pt` contains 1,382 tensors: the 1,381 BF16 tensors from training plus the
deterministically reconstructed FP32 `vocab_weight` buffer required by the current
strict ESPnet loader. It contains no optimizer or exact-resume state. It is not a
Transformers `from_pretrained` directory and no vLLM compatibility is claimed.
## Download
```bash
hf download espnet/bagpiper-sft --local-dir bagpiper-sft
sha256sum -c bagpiper-sft/SHA256SUMS
```
## ESPnet runtime
The release was strict-loaded with:
- ESPnet PR [#6503](https://github.com/espnet/espnet/pull/6503), commit
`1992821bfb6d68c6636d1b75340b8b2b591bc015`;
- Python 3.11.15, PyTorch 2.12.1+cu129, Transformers 5.5.4;
- FlashAttention-3 3.0.0 at source commit
`77aacb68d194ba9af1010eda5eac3e7c0df8e6f6`; and
- one NVIDIA H100 80GB HBM3.
```bash
git clone --branch titan-pt-weightonly-init https://github.com/jctian98/espnet.git
cd espnet
git checkout 1992821bfb6d68c6636d1b75340b8b2b591bc015
python -m pip install -e '.[speechlm]'
python -m pip install --no-build-isolation \
'flash-attn-3 @ git+https://github.com/Dao-AILab/flash-attention.git@77aacb68d194ba9af1010eda5eac3e7c0df8e6f6#subdirectory=hopper'
```
Use `espnet2/speechlm/bin/inference.py` with
`train_stage3_qwen3_base.yaml`, one of the supplied inference YAMLs, and
`model.pt`. Training backend configuration belongs to the ESPnet recipe and is
not duplicated in this model repository.
## Paper-reported results
These values are from the paper's checkpoint and evaluation protocols; they were
not rerun as part of the Hub upload.
| Evaluation | Result |
|---|---:|
| LibriSpeech test-clean WER | 2.5 |
| MMAU-Mini | 74.5 |
| MMAU | 73.1 |
| MMAR | 57.0 |
| AIR-Bench chat | 6.57 |
| AudioBench aggregate | 70.39 |
| General-model TTS WER | 2.7 |
The paper reports audio-fingerprint overlap between the SFT pool and **12.8% of
AudioBench** and **3.2% of AIR-Bench** evaluation clips. Interpret those results
with this disclosure.
## Provenance and validation
- Selected checkpoint: `step_270000/global_step269985`.
- Raw source: `JinchuanTian/bagpiper_sft` at revision
`b11d5a0c11ad488edd04e3734d4bdff764977f57`.
- Raw SHA-256: `e0d8fdd57c8cb8c938cd61ada43b20e469d9cea33ad8450e0971f80d4f9667b6`.
- `model.pt` SHA-256:
`50980756fad38bbefeb326ac0012e39f431ec5af8c0caaac8bd254f179f33746`.
- All 1,381 source tensors matched the converted file exactly with
`torch.equal`; the only added key was `vocab_weight`.
- The 1,382-entry state dict passed the native ESPnet strict inference loader.
## Limitations and responsible use
Understanding can hallucinate or mistranscribe. Generated audio can contain
artifacts, wrong words, unsafe content, or unintended resemblance to people and
styles. Most reported evaluations are English and some rely on model judges. The
model has no built-in watermark, factuality verifier, copyright filter, or
speaker-consent check. Do not use it for impersonation, deceptive media,
surveillance, biometric authentication, or high-stakes automated decisions.
Training-data availability and terms are documented at
[Bagpiper SFT Data](https://huggingface.co/datasets/espnet/Bagpiper_SFT_Data).
Third-party component terms are listed in `THIRD_PARTY_NOTICES.md`.
## Citation
```bibtex
@inproceedings{anonymous2026bagpiper,
title={Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions},
author={Jinchuan Tian and Haoran Wang and Bo-Hao Su and Chien-yu Huang and
Qingzheng Wang and Jiatong Shi and William Chen and Xun Gong and
Siddhant Arora and Chin-Jou Li and Masao Someki and Takashi Maekaku and
Keita Goto and Yusuke Shinohara and Jin Sakuma and
Chao-Han Huck Yang and Shinji Watanabe},
booktitle={Third Conference on Language Modeling},
year={2026},
url={https://openreview.net/forum?id=FuHs64E3X6}
}
```
|