TextPro-SLM-7B-30Plus

Your speech LLM can be a prosody-aware text LLM.

TextPro-SLM combines transcript tokens with a compact prosody embedding. Transcript tokens carry linguistic content, while prosody carries complementary acoustic information. This input-side factorization brings spoken input closer to the native input format of a text LLM.

Paper · GitHub project · Paired checkpoint

Release and pairing

This repository contains the 30Plus extension and is paired with WhisperPro-30Plus.

The paired inference design handles recordings longer than 30 seconds through VAD segmentation, previous-transcript context, transcript concatenation, and equal-weight averaging of segment prosody vectors. Each Whisper segment remains at most 30 seconds; this is not a native extension of Whisper's context window. This release is separate from the original checkpoints, and original-paper scores are not measurements of the 30Plus extension.

Component	Specification
Text backbone	Qwen2.5-7B-Instruct, fully fine-tuned
Prosody projector	1280 → 3584 → 3584, GELU, BF16
Fusion	One global prosody embedding prepended to transcript tokens
Audio preprocessing	Mono, 16 kHz
LLM context	32,768 tokens
Output	Text response

Method

The accompanying paper studies how reducing the speech–text mismatch from the input side can preserve semantic capabilities while learning paralinguistic understanding, using roughly 1,000 hours of LLM training audio for the paper models. This model card documents the released artifact; it does not introduce new benchmark measurements.

Download

pip install "huggingface-hub>=0.36,<1"
hf download qqjz/TextPro-SLM-7B-30Plus --local-dir TextPro-SLM-7B-30Plus

This Hub repository contains checkpoint artifacts and model documentation only. Inference code, dependencies, and usage examples are being prepared separately in the GitHub project; they are not included in this checkpoint release. Prosody-aware inference requires that implementation rather than loading the text backbone alone.

Files and export

llm/                     Fine-tuned LLM weights, configuration, and tokenizer
projector.safetensors    Compact trained prosody projector
textpro_config.json      Encoder pairing, fusion, and prosody-token settings
SHA256SUMS.json          Artifact sizes and SHA-256 hashes

The supplied tokenizer preserves the trained <|repo_name|> prosody marker (ID 151663). The configuration pins the paired encoder revision. The projector is stored as compact safetensors rather than a serialized training storage buffer.

Model family

Encoder	Paired 7B model	Audio handling
WhisperPro	TextPro-SLM-7B-Global	Original, at most 30 seconds
WhisperPro-30Plus	TextPro-SLM-7B-30Plus	Segmentation-based long-audio extension

Limitations

Responses may contain transcription errors or hallucinations. Performance can vary with language, recording conditions, speaker characteristics, and domain. Global pooling compresses time-varying prosody. The model returns text; it does not synthesize speech. The 30Plus processing design is subject to segmentation effects and the LLM's finite transcript context; no unlimited-duration or long-audio accuracy claim is implied.

License and acknowledgements

Apache-2.0; see LICENSE and NOTICE. The LLM is derived from Qwen2.5-7B-Instruct. The paired WhisperPro encoder is derived from OpenAI Whisper large-v3.

Citation

@inproceedings{cui2026textproslm,
  title={Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM},
  author={Cui, Wenqian and Li, Xiao-Hui and Tan, Daxin and Zheng, Qiyong and King, Irwin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://openreview.net/forum?id=Jsjmo8eX29}
}