WhisperPro

Speech → transcript + compact prosody.

WhisperPro is the unified speech encoder behind TextPro-SLM. It extracts synchronized transcript tokens and continuous prosody features, preserving acoustic information that a transcript alone can miss.

Paper · GitHub project · Paired checkpoint

Release and pairing

This repository contains the original WhisperPro checkpoint (step 9092) and is paired with TextPro-SLM-7B-Global.

The original interface supports audio clips of at most 30 seconds. Use the separate 30Plus release for the segmentation-based long-audio extension.

Component	Specification
Backbone	OpenAI Whisper large-v3
Prosody features	Decoder layer 16, 1280 dimensions
Compact representation	Mean pooling over valid generated-token positions
Weights	Inference model in safetensors; training-only Mel reconstructor excluded

Method

The accompanying paper studies how reducing the speech–text mismatch from the input side can preserve semantic capabilities while learning paralinguistic understanding, using roughly 1,000 hours of LLM training audio for the paper models. This model card documents the released artifact; it does not introduce new benchmark measurements.

Download

pip install "huggingface-hub>=0.36,<1"
hf download qqjz/WhisperPro --local-dir WhisperPro

This Hub repository contains checkpoint artifacts and model documentation only. Inference code, dependencies, and usage examples are being prepared separately in the GitHub project; they are not included in this checkpoint release. Prosody-aware inference requires that implementation rather than loading the text backbone alone.

Files and export

model.safetensors        WhisperPro inference weights
config.json              Whisper architecture configuration
codec_config.json        Prosody-layer selection
generation_config.json   ASR generation configuration
*token* / vocab / merges  Processor and tokenizer artifacts
SHA256SUMS.json          Artifact sizes and SHA-256 hashes

The processor matches the one used by the paired inference pipeline. All retained weight tensors are unchanged; excluding the training-only Mel reconstructor does not change transcript/prosody extraction.

Model family

Encoder	Paired 7B model	Audio handling
WhisperPro	TextPro-SLM-7B-Global	Original, at most 30 seconds
WhisperPro-30Plus	TextPro-SLM-7B-30Plus	Segmentation-based long-audio extension

Limitations

Transcription errors and shifts in language or recording conditions can affect the extracted features. Compact mean pooling compresses time-varying prosody. The 30Plus processing design is subject to segmentation effects and the LLM's finite transcript context; no unlimited-duration or long-audio accuracy claim is implied.

License and acknowledgements

Apache-2.0; see LICENSE and NOTICE. This checkpoint is derived from OpenAI Whisper large-v3.

Citation

@inproceedings{cui2026textproslm,
  title={Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM},
  author={Cui, Wenqian and Li, Xiao-Hui and Tan, Daxin and Zheng, Qiyong and King, Irwin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=Jsjmo8eX29}
}