README.md

---

library_name: transformers
license: mit
datasets:
- openslr/librispeech_asr
- slprl/SpokenSwag
- slprl/sTinyStories
base_model:
- Qwen/Qwen2.5-0.5B
pipeline_tag: audio-to-audio
language:
- zho
- eng
- fra
- spa
- por
- deu
- ita
- rus
- jpn
- kor
- vie
- tha
- ara
---


# Model Card for SLAM

This is a Speech Language Model trained for generating speech continuations over discrete [Hubert tokens](https://huggingface.co/slprl/mhubert-base-25hz).


## Model Details

### Model Description
This is a Speech Language Model, introduced in "[_Slamming_: Training a Speech Language Model on One GPU in a Day](https://arxiv.org/abs/2502.15814)", focusing on efficient training. 
It was fine-tuned from [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) over a vocabulary of 500 speech tokens extracted from 
the 11-th layer of [mhubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz). For a stronger version of the model trained with 
slightly more compute - 2*A100 for 2 days, see [slam_scaled](https://huggingface.co/slprl/slam_scaled).



The model was trained by next-token prediction over a subset of LibriSpeech, Libri-Light and a synthetic data 

[sTinyStories](https://huggingface.co/datasets/slprl/sTinyStories). It was then trained with DPO over 

[SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).



- **Developed by:** [SLP-RL](https://huggingface.co/slprl)

- **Model type:** SpeechLM

- **License:** MIT

- **Finetuned from model:** [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)



### Model Sources



- **Repository:** [https://github.com/slp-rl/slamkit](https://github.com/slp-rl/slamkit)

- **Paper:** [https://arxiv.org/abs/2502.15814](https://arxiv.org/abs/2502.15814)

- **Demo:** [https://pages.cs.huji.ac.il/adiyoss-lab/slamming/](https://pages.cs.huji.ac.il/adiyoss-lab/slamming/)



## Uses

This is a base SpeechLM and as such can be used to generate continuations for speech segments, or as base for further tuning. See the _SlamKit_

[codebase](https://github.com/slp-rl/slamkit) for more details on usage, and checkout the [demo page](https://pages.cs.huji.ac.il/adiyoss-lab/slamming/) for some generation examples



### Out-of-Scope Use

This model was trained on curated speech datasets which contain mainly audio-books and stories, as such the outputs should not be treated as factual in any way.







## How to Get Started with the Model

We refer users to the official repository for full usage explanations - [github](https://github.com/slp-rl/slamkit).





## Training Details

We highly encourage users to read the full [paper](https://arxiv.org/abs/2502.15814), for full training details, a brief overview is provided below.





### Training Data

This model was trained on a subset of [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr) train, 

[Libri-Light](https://ai.meta.com/tools/libri-light/) and the synthetic dataset

[sTinyStories](https://huggingface.co/datasets/slprl/sTinyStories) for the pre-training phase. It was also trained with DPO on the synthetic

dataset [SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).



### Training Procedure

This model was trained by next token prediction over several datasets, and then trained with DPO over [SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).

Please refer to the [paper]() or [code](https://github.com/slp-rl/slamkit) for the full training recipes.



#### Preprocessing

Speech tokens are extracted from the audio using [Hubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz), and quantised using the 

official kmeans released with the model in [textlesslib](https://github.com/facebookresearch/textlesslib/tree/main). Units are de-duplicated.

We encourage you to explore the official repository for full details - [github](https://github.com/slp-rl/slamkit).





## Evaluation

The paper provides full results, we do give here some results and also refer to the [demo page]() to listen to some samples.

| Model                                    | Compute (GPU days) | Parameters | sBLIMP ↑ | sStoryCloze ↑ | tStoryCloze ↑ | GenPPL ↓ | Auto-BLEU ↓ |

|------------------------------------------|--------------------|------------|----------|--------------|--------------|---------|------------|

| [TWIST-1.3B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                      | 160xV100          | 1B         | 57.00    | 52.4         | 70.6         | 131.8   | 3.20       |

| [TWIST-7B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                        | ?                 | 7B         | 59.00    | 55.3         | 74.1         | 93.7    | 3.06       |

| [TWIST-13B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                       | ?                 | 13B        | 59.20    | 55.4         | 76.4         | -       | -          |

| [Scaled Optimal](https://arxiv.org/abs/2404.00685)      | ?                 | 823M       | **61.3** | **56.7**     | **78.0**     | -       | -          |

| [Predicted Optimal]((https://arxiv.org/abs/2404.00685))   | 1xA5000           | 78M        | 56.85    | 54.09        | 70.49        | -       | -          |

| TWIST-350M (Original recipe)            | 1xA5000           | 305M       | 51.52 ± .19 | 53.65 ± .57 | 68.80 ± .47 | 259.2 ± 6.7 | 3.26 ± .46 |

| *Slam (-DPO) (ours)*                    | 1xA5000           | 358M       | *56.45* ± .17 | *55.59* ± .30 | *78.01* ± .27 | *88.3* ± 1.0 | 3.47 ± .17 |
| **Slam (ours)**                       | 1xA5000           | 358M       | **58.86** ± .20 | **58.04** ± .51 | **82.04** ± .21 | **62.8** ± 4.1 | 3.88 ± .11 |



### Compute Infrastructure
This model was trained as part of ["*Slamming*: Training a Speech Language Model on One GPU in a Day"](https://arxiv.org/abs/2502.15814), focusing on efficient training.

#### Hardware
This model was trained using **only a single Nvidia A5000 GPU**, 16 CPU cores and 24 GB of RAM for **24 hours**.

#### Software
The model was trained using the [*SlamKit*](https://github.com/slp-rl/slamkit) codebase which builds upon 🤗transformers extending it to support
easy and efficient training of Speech Language Models.

## Citation

**BibTeX:**
```

@misc{maimon2025slamming,

      title={Slamming: Training a Speech Language Model on One GPU in a Day}, 

      author={Gallil Maimon and Avishai Elmakies and Yossi Adi},

      year={2025},

      eprint={2502.15814},

      archivePrefix={arXiv},

      primaryClass={cs.LG},

      url={https://arxiv.org/abs/2502.15814}, 

}

```