| --- |
| license: mit |
| --- |
| |
|
|
| ## SparQLe – Speech Queries to Text via Instruction‑Tuned LLM ⚡ |
|
|
| **What it does:** |
| SparQLe (Speech Routing to Query LLMs) enables direct speech-to-text understanding by aligning self‑supervised speech representations (e.g., HuBERT-like features) with instruction‑tuned Large Language Models (LLMs). This is achieved using a lightweight *modality adapter*, bridging the modalities without retraining the whole LLM. ([Moonlight][1]) |
|
|
| **Key strengths:** |
|
|
| * **Preserves semantic content** of spoken input in the produced text |
| * **Efficiently leverages frozen SSL models**, avoiding heavy ASR backbones like Whisper |
| * **Modular design** with a query‑former (Q‑former) adapter and LLM backend |
|
|
| **Architecture:** |
|
|
| 1. **Speech encoder** (SSL) transforms raw input into latent features. |
| 2. **Modality adapter / Q‑former** aligns these with the LLM’s text embedding space. |
| 3. **Instruction‑tuned LLM** processes the adapted input to generate semantic text. |
|
|
|
|
| ## Citation |
|
|
| If you use SparQLe in your research, please cite: |
|
|
| ```bibtex |
| @misc{djanibekov2025sparqlespeechqueriestext, |
| title={SparQLe: Speech Queries to Text Translation Through LLMs}, |
| author={Amirbek Djanibekov and Hanan Aldarmaki}, |
| year={2025}, |
| eprint={2502.09284}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2502.09284}, |
| } |
| ``` |
|
|
| 📄 Read the full paper on arXiv: [https://arxiv.org/abs/2502.09284](https://arxiv.org/abs/2502.09284) |
|
|
| --- |
|
|
| ## License |
|
|
| This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
|
|
| --- |
|
|
| ## Acknowledgments |
|
|
| - This work builds upon [fairseq](https://github.com/facebookresearch/fairseq) 💙 |
| - The Qformer architecture is inspired by [BLIP-2](https://github.com/salesforce/BLIP-2) ✨ |
|
|
