File size: 1,814 Bytes
fdfdc1c 145da28 fdfdc1c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | ---
license: mit
---
## SparQLe – Speech Queries to Text via Instruction‑Tuned LLM ⚡
**What it does:**
SparQLe (Speech Routing to Query LLMs) enables direct speech-to-text understanding by aligning self‑supervised speech representations (e.g., HuBERT-like features) with instruction‑tuned Large Language Models (LLMs). This is achieved using a lightweight *modality adapter*, bridging the modalities without retraining the whole LLM. ([Moonlight][1])
**Key strengths:**
* **Preserves semantic content** of spoken input in the produced text
* **Efficiently leverages frozen SSL models**, avoiding heavy ASR backbones like Whisper
* **Modular design** with a query‑former (Q‑former) adapter and LLM backend
**Architecture:**
1. **Speech encoder** (SSL) transforms raw input into latent features.
2. **Modality adapter / Q‑former** aligns these with the LLM’s text embedding space.
3. **Instruction‑tuned LLM** processes the adapted input to generate semantic text.
## Citation
If you use SparQLe in your research, please cite:
```bibtex
@misc{djanibekov2025sparqlespeechqueriestext,
title={SparQLe: Speech Queries to Text Translation Through LLMs},
author={Amirbek Djanibekov and Hanan Aldarmaki},
year={2025},
eprint={2502.09284},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.09284},
}
```
📄 Read the full paper on arXiv: [https://arxiv.org/abs/2502.09284](https://arxiv.org/abs/2502.09284)
---
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
---
## Acknowledgments
- This work builds upon [fairseq](https://github.com/facebookresearch/fairseq) 💙
- The Qformer architecture is inspired by [BLIP-2](https://github.com/salesforce/BLIP-2) ✨
|