| --- |
| library_name: espnet |
| pipeline_tag: any-to-any |
| base_model: espnet/bagpiper |
| language: |
| - en |
| tags: |
| - audio |
| - multimodal |
| - speech-language-model |
| - audio-understanding |
| - audio-generation |
| - text-to-audio |
| - espnet |
| - pytorch |
| inference: false |
| --- |
| |
| # Bagpiper SFT |
|
|
| Bagpiper is an 8B audio foundation model for open-ended audio understanding and |
| generation through rich captions. It accepts text and audio and can return text |
| or generate speech, music, environmental sounds, and mixtures of them. |
|
|
| This repository contains the paper-selected general-SFT checkpoint from |
| [Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://openreview.net/forum?id=FuHs64E3X6). |
|
|
| ## Files and format |
|
|
| - `model.pt`: native ESPnet weight-only checkpoint, stored as |
| `{"module": state_dict}`. |
| - `train_stage3_qwen3_base.yaml`: checkpoint-compatible model configuration, |
| named consistently with the pre-trained `espnet/bagpiper` release. |
| - `inference_text.yaml`: audio-understanding/text-output decoding. |
| - `inference_audio.yaml`: text-then-audio generation decoding. |
| - `MANIFEST.json` and `SHA256SUMS`: provenance and integrity records. |
|
|
| `model.pt` contains 1,382 tensors: the 1,381 BF16 tensors from training plus the |
| deterministically reconstructed FP32 `vocab_weight` buffer required by the current |
| strict ESPnet loader. It contains no optimizer or exact-resume state. It is not a |
| Transformers `from_pretrained` directory and no vLLM compatibility is claimed. |
|
|
| ## Download |
|
|
| ```bash |
| hf download espnet/bagpiper-sft --local-dir bagpiper-sft |
| sha256sum -c bagpiper-sft/SHA256SUMS |
| ``` |
|
|
| ## ESPnet runtime |
|
|
| The release was strict-loaded with: |
|
|
| - ESPnet PR [#6503](https://github.com/espnet/espnet/pull/6503), commit |
| `1992821bfb6d68c6636d1b75340b8b2b591bc015`; |
| - Python 3.11.15, PyTorch 2.12.1+cu129, Transformers 5.5.4; |
| - FlashAttention-3 3.0.0 at source commit |
| `77aacb68d194ba9af1010eda5eac3e7c0df8e6f6`; and |
| - one NVIDIA H100 80GB HBM3. |
|
|
| ```bash |
| git clone --branch titan-pt-weightonly-init https://github.com/jctian98/espnet.git |
| cd espnet |
| git checkout 1992821bfb6d68c6636d1b75340b8b2b591bc015 |
| python -m pip install -e '.[speechlm]' |
| python -m pip install --no-build-isolation \ |
| 'flash-attn-3 @ git+https://github.com/Dao-AILab/flash-attention.git@77aacb68d194ba9af1010eda5eac3e7c0df8e6f6#subdirectory=hopper' |
| ``` |
|
|
| Use `espnet2/speechlm/bin/inference.py` with |
| `train_stage3_qwen3_base.yaml`, one of the supplied inference YAMLs, and |
| `model.pt`. Training backend configuration belongs to the ESPnet recipe and is |
| not duplicated in this model repository. |
|
|
| ## Paper-reported results |
|
|
| These values are from the paper's checkpoint and evaluation protocols; they were |
| not rerun as part of the Hub upload. |
|
|
| | Evaluation | Result | |
| |---|---:| |
| | LibriSpeech test-clean WER | 2.5 | |
| | MMAU-Mini | 74.5 | |
| | MMAU | 73.1 | |
| | MMAR | 57.0 | |
| | AIR-Bench chat | 6.57 | |
| | AudioBench aggregate | 70.39 | |
| | General-model TTS WER | 2.7 | |
|
|
| The paper reports audio-fingerprint overlap between the SFT pool and **12.8% of |
| AudioBench** and **3.2% of AIR-Bench** evaluation clips. Interpret those results |
| with this disclosure. |
|
|
| ## Provenance and validation |
|
|
| - Selected checkpoint: `step_270000/global_step269985`. |
| - Raw source: `JinchuanTian/bagpiper_sft` at revision |
| `b11d5a0c11ad488edd04e3734d4bdff764977f57`. |
| - Raw SHA-256: `e0d8fdd57c8cb8c938cd61ada43b20e469d9cea33ad8450e0971f80d4f9667b6`. |
| - `model.pt` SHA-256: |
| `50980756fad38bbefeb326ac0012e39f431ec5af8c0caaac8bd254f179f33746`. |
| - All 1,381 source tensors matched the converted file exactly with |
| `torch.equal`; the only added key was `vocab_weight`. |
| - The 1,382-entry state dict passed the native ESPnet strict inference loader. |
|
|
| ## Limitations and responsible use |
|
|
| Understanding can hallucinate or mistranscribe. Generated audio can contain |
| artifacts, wrong words, unsafe content, or unintended resemblance to people and |
| styles. Most reported evaluations are English and some rely on model judges. The |
| model has no built-in watermark, factuality verifier, copyright filter, or |
| speaker-consent check. Do not use it for impersonation, deceptive media, |
| surveillance, biometric authentication, or high-stakes automated decisions. |
|
|
| Training-data availability and terms are documented at |
| [Bagpiper SFT Data](https://huggingface.co/datasets/espnet/Bagpiper_SFT_Data). |
| Third-party component terms are listed in `THIRD_PARTY_NOTICES.md`. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{anonymous2026bagpiper, |
| title={Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions}, |
| author={Jinchuan Tian and Haoran Wang and Bo-Hao Su and Chien-yu Huang and |
| Qingzheng Wang and Jiatong Shi and William Chen and Xun Gong and |
| Siddhant Arora and Chin-Jou Li and Masao Someki and Takashi Maekaku and |
| Keita Goto and Yusuke Shinohara and Jin Sakuma and |
| Chao-Han Huck Yang and Shinji Watanabe}, |
| booktitle={Third Conference on Language Modeling}, |
| year={2026}, |
| url={https://openreview.net/forum?id=FuHs64E3X6} |
| } |
| ``` |
|
|
