| --- |
| library_name: espnet |
| pipeline_tag: any-to-any |
| base_model: Qwen/Qwen3-8B-Base |
| language: |
| - en |
| tags: |
| - audio |
| - multimodal |
| - speech-language-model |
| - audio-understanding |
| - audio-generation |
| - audio-to-text |
| - text-to-audio |
| - rich-caption |
| - speech |
| - music |
| - environmental-sound |
| - espnet |
| - pytorch |
| inference: false |
| --- |
| |
| # Bagpiper-Base |
|
|
| Bagpiper-Base is the pretrained 8B checkpoint from |
| [Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://openreview.net/forum?id=FuHs64E3X6). |
| It learns a bidirectional mapping between physical audio and detailed natural-language |
| descriptions spanning speech, music, environmental sound, and mixtures of them. |
|
|
| > **This is the Base checkpoint, not the instruction-tuned assistant.** Use |
| > [`espnet/bagpiper-sft`](https://huggingface.co/espnet/bagpiper-sft) for the |
| > caption-then-process model trained to answer open-ended audio requests. Bagpiper-Base |
| > is intended for audio-to-rich-caption probing, rich-caption-conditioned audio |
| > generation, and downstream initialization. |
|
|
| ## Repository contents |
|
|
| | File | Description | |
| |---|---| |
| | `base.pt` | Native ESPnet weight-only checkpoint | |
| | `train_stage2_qwen3_base.yaml` | Reconstructed checkpoint-compatible model configuration | |
|
|
| `base.pt` is a PyTorch mapping with model state under the top-level `module` key. It |
| contains 1,382 tensors: 1,381 BF16 tensors from training and one deterministic FP32 |
| `vocab_weight` buffer required by the current strict ESPnet loader. It contains no |
| optimizer, scheduler, or exact-resume state. This is not a Transformers |
| `from_pretrained` directory, and no vLLM compatibility is claimed. |
|
|
| The YAML reconstructs the architecture and vocabulary of the released checkpoint. The |
| original stage-2 launch referenced this filename but did not commit it, so its trainer |
| fields should not be treated as an archival record of every training setting. |
|
|
| ## Download and integrity |
|
|
| ```bash |
| hf download espnet/bagpiper \ |
| --revision d7b91ab261116973fb65cf44ff9590f6f242323b \ |
| --local-dir bagpiper-base |
| |
| echo "c9917c1237bd44ea0ea8b41746e5ec5aecae92273538c9b6e4040d26f69a6521 bagpiper-base/base.pt" \ |
| | sha256sum -c - |
| ``` |
|
|
| ## Architecture |
|
|
| | Component | Released configuration | |
| |---|---| |
| | Text backbone and tokenizer | [`Qwen/Qwen3-8B-Base`](https://huggingface.co/Qwen/Qwen3-8B-Base) | |
| | Audio input encoder | [`Qwen/Qwen3-Omni-30B-A3B-Instruct`](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) | |
| | Audio output tokenizer | Eight-stream, delay-interleaved [`hf-audio/xcodec-hubert-general`](https://huggingface.co/hf-audio/xcodec-hubert-general) | |
| | Interface | Interleaved text and audio tokens in ESPnet SpeechLM | |
| | Checkpoint dtype | BF16 model tensors; FP32 deterministic vocabulary-weight buffer | |
|
|
| The audio encoder converts input waveforms into continuous representations. X-Codec |
| provides discrete output tokens that can be decoded back into audio. Expanded text and |
| audio embeddings allow the Qwen3 backbone to model both directions through one shared |
| sequence interface. |
|
|
| ## ESPnet runtime |
|
|
| The native consumer is `espnet2/speechlm/bin/inference.py`. Reconstruct the model with |
| the supplied training YAML, pair it with the task-appropriate inference YAML from the |
| [Bagpiper recipe](https://github.com/jctian98/espnet/tree/1992821bfb6d68c6636d1b75340b8b2b591bc015/egs2/bagpiper/speechlm1), |
| and pass `base.pt` through `--model-checkpoint`. |
|
|
| The released state dict was loaded through ESPnet's strict inference path on an NVIDIA |
| H100 using [ESPnet PR #6503](https://github.com/espnet/espnet/pull/6503) at commit |
| `1992821bfb6d68c6636d1b75340b8b2b591bc015` and Transformers 5.5.4. A complete |
| clean-room installation and end-to-end quickstart have not yet been validated; the Hub |
| Inference API is therefore disabled. The paper estimates roughly 24 GB of GPU memory for |
| BF16 inference, with longer inputs, decoding, or additional workers requiring more. |
|
|
| ## Pretraining |
|
|
| Bagpiper-Base is the result of the first two stages of the paper's curriculum: connector |
| and embedding warmup followed by full pretraining. The audio encoder and codec remain |
| frozen during full pretraining while the backbone, connectors, and expanded vocabulary |
| are optimized. The paper reports training on 80 NVIDIA GH200 GPUs. |
|
|
| The 600B figure in the paper is a **token budget**, not a row count: 300B text-to-audio, |
| 150B audio-to-text, and 150B text-only tokens. The audio pipeline began from roughly |
| 422M raw audio-caption pairs, capped clips at 30 seconds, and used machine-generated rich |
| captions plus quality, alignment, and deduplication filters. |
|
|
| The associated public |
| [`Bagpiper_PreTrain_Data`](https://huggingface.co/datasets/espnet/Bagpiper_PreTrain_Data) |
| snapshot is a large artifact from this pipeline, not a claim of a complete one-to-one dump |
| of every raw pair or text-only source used in training. |
|
|
| ## Paper-reported Base probes |
|
|
| These results evaluate the pretrained audio-rich-caption mapping. They were not rerun as |
| part of this Hub card update and must not be interpreted as end-to-end task-solving scores. |
|
|
| | Direction / probe | Result | |
| |---|---:| |
| | Audio -> rich caption -> external-LLM ASR, LibriSpeech test-clean WER | 5.0 | |
| | Audio -> rich caption -> external-LLM QA, MMAU-Mini accuracy | 69.0 | |
| | Rich caption -> speech, LibriSpeech test-clean WER | 1.8 | |
| | Rich caption -> general audio, MMAU-Mini FAD | 2.98 | |
| | Rich caption -> general audio, MMAU-Mini audio-audio CLAP | 0.55 | |
|
|
| For the understanding probes, an external text LLM answers using only Bagpiper-Base's |
| generated caption; the model does not solve the final task itself. The speech-generation |
| WER uses only the roughly 60% of test-clean examples whose conditioning rich captions |
| preserve the full transcription. Generation baselines receive transcripts or short |
| captions, whereas Bagpiper-Base receives full rich captions. See the paper for the complete |
| protocol, comparisons, cycle-consistency results, and text-capability probes. |
|
|
| ## Intended use |
|
|
| - Research on bidirectional audio-rich-caption representations. |
| - Rich-caption generation for analysis or as input to a separately evaluated downstream |
| system. |
| - Rich-caption-conditioned generation of speech, music, sound, and mixtures. |
| - Initialization for downstream ESPnet SpeechLM fine-tuning. |
|
|
| For general instruction following and open-ended audio question answering or generation, |
| start from [`espnet/bagpiper-sft`](https://huggingface.co/espnet/bagpiper-sft). For |
| speech-specialized synthesis, see |
| [`espnet/bagpiper-tts-sft`](https://huggingface.co/espnet/bagpiper-tts-sft). |
|
|
| ## Limitations and responsible use |
|
|
| Rich captions are an explicit information bottleneck: omissions, hallucinations, or |
| incorrect acoustic details can propagate into downstream answers or generated audio. |
| Generated audio may contain artifacts, wrong words, unsafe content, or unintended |
| resemblance to people or styles. Most headline probes are English-focused, some depend on |
| model-based evaluation, and performance outside the evaluated distributions is unknown. |
|
|
| Training sources and machine-generated captions may contain copyrighted, identifiable, |
| sensitive, biased, or offensive material. Do not use this checkpoint for impersonation, |
| deceptive media, surveillance, biometric authentication, or high-stakes automated |
| decisions. Obtain appropriate rights and consent, disclose synthetic media where relevant, |
| and apply human review and domain-specific safeguards. |
|
|
| No blanket model-weight license is declared in this card. The checkpoint does not override |
| the terms of its upstream components or training sources; review those terms for the |
| intended use and jurisdiction. |
|
|
| ## Provenance and validation |
|
|
| - Selected checkpoint: `step_260000/global_step259988`. |
| - Raw source: `JinchuanTian/Bagpiper_checkpoints` at revision |
| `f226d242bb0d595b02e2289f8784693e88470c18`. |
| - Raw model-state SHA-256: |
| `58c64ff430e2d54170403b77471c28e49b432669a9b264f41efaf0bedd52fd7a`. |
| - Released `base.pt`: 18,188,844,317 bytes; SHA-256 |
| `c9917c1237bd44ea0ea8b41746e5ec5aecae92273538c9b6e4040d26f69a6521`. |
| - Hub revision before this card update: |
| `d7b91ab261116973fb65cf44ff9590f6f242323b`. |
| - The converter preserves all 1,381 source tensors and adds only the deterministic |
| `vocab_weight` buffer; the resulting state dict passes the native strict ESPnet loader. |
|
|
| ## Related resources |
|
|
| - [Project page and audio gallery](https://bagpiper-cmu.github.io/) |
| - [Accepted COLM 2026 paper](https://openreview.net/forum?id=FuHs64E3X6) |
| - [Bagpiper ESPnet recipe](https://github.com/jctian98/espnet/tree/1992821bfb6d68c6636d1b75340b8b2b591bc015/egs2/bagpiper/speechlm1) |
| - [Bagpiper SFT model](https://huggingface.co/espnet/bagpiper-sft) |
| - [Bagpiper-TTS SFT model](https://huggingface.co/espnet/bagpiper-tts-sft) |
| - [Bagpiper pretraining data](https://huggingface.co/datasets/espnet/Bagpiper_PreTrain_Data) |
| - [Bagpiper SFT data](https://huggingface.co/datasets/espnet/Bagpiper_SFT_Data) |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{tian2026bagpiper, |
| title = {Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions}, |
| author = {Jinchuan Tian and Haoran Wang and Bo-Hao Su and Chien-yu Huang and |
| Qingzheng Wang and Jiatong Shi and William Chen and Xun Gong and |
| Siddhant Arora and Chin-Jou Li and Masao Someki and Takashi Maekaku and |
| Keita Goto and Yusuke Shinohara and Jin Sakuma and |
| Chao-Han Huck Yang and Shinji Watanabe}, |
| booktitle = {Third Conference on Language Modeling}, |
| year = {2026}, |
| url = {https://openreview.net/forum?id=FuHs64E3X6} |
| } |
| ``` |
|
|
