Update README.md

README.md

@@ -10,15 +10,15 @@ pipeline_tag: audio-to-audio
 ---
 
 # Model Card for Model ID
-This is a Speech Lanaguage Model trained for generating audio contiuations over discrete [Hubert tokens](https://huggingface.co/slprl/mhubert-base-25hz).
+This is a Speech Lanaguage Model trained for generating speech contiuations over discrete [Hubert tokens](https://huggingface.co/slprl/mhubert-base-25hz).
 
 
 ## Model Details
 
 ### Model Description
-This is a Speech Lanaguage Model, fine-tuned from [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) over a vocabulary of 500 
-speech tokens extracted from the 11-th layer of [mhubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz). It was trained as part of
-["*Slamming*: Training a Speech Language Model on One GPU in a Day"], focusing on efficient training. For a stronger model trained with 
+This is a Speech Lanaguage Model, introduced in "_Slamming_: Training a Speech Language Model on One GPU in a Day", focusing on efficient training. 
+It was fine-tuned from [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B) over a vocabulary of 500 speech tokens extracted from 
+the 11-th layer of [mhubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz). For a stronger version of the model trained with 
 slightly more compute - 2*A100 for 2 days, see [slam_scaled](https://huggingface.co/slprl/slam_scaled).
 
 The model was trained by next-token prediction over a subset of LibriSpeech, Libri-Light and a synthetic data 
@@ -37,13 +37,8 @@ The model was trained by next-token prediction over a subset of LibriSpeech, Lib
 - **Demo:** [Link](https://pages.cs.huji.ac.il/adiyoss-lab/slamming/)
 
 ## Uses
-This is a base SpeechLM and as such can be used to generate contiuations for speech segments, or as base for further tuning.
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+This is a base SpeechLM and as such can be used to generate contiuations for speech segments, or as base for further tuning. See the _slam_
+[codebase](https://github.com/slp-rl/slam) for more details on usage, and checkout the [demo page](https://pages.cs.huji.ac.il/adiyoss-lab/slamming/) for some generation examples
 
 ### Out-of-Scope Use
 This model was trained on curated speech datasets which contain mainly audio-books and stories, as such the outputs should not be treated as factual in any way.
@@ -59,13 +54,14 @@ We highly encourage users to read the full [paper](), for full training details,
 
 
 ### Training Data
-This model was trained on a subset of [LibriSpeech] train, [Libri-Light]() and the synthetic dataset
+This model was trained on a subset of [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr) train, 
+[Libri-Light](https://ai.meta.com/tools/libri-light/) and the synthetic dataset
 [sTinyStories](https://huggingface.co/datasets/slprl/sTinyStories) for the pre-training phase. It was also trained with DPO on the synthetic
 dataset [SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).
 
 ### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+This model was trained by next token prediction over several dataset, and then trained with DPO over [SpokenSwag](https://huggingface.co/datasets/slprl/SpokenSwag).
+Please refer to the [paper]() or [code](https://github.com/slp-rl/slam) for the full training recipes.
 
 #### Preprocessing
 Speech tokens are extracted from the audio using [Hubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz), and quantised using the 
@@ -73,21 +69,20 @@ official kmeans released with the model in [textlesslib](https://github.com/face
 We encourage you to explore the official repository for full details - [github](https://github.com/slp-rl/slam).
 
 
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-
 ## Evaluation
 The paper provides full results, we do give here some results and also refer to the [demo page]() to listen to some samples.
+| Model                                    | Compute (GPU days) | Parameters | sBLIMP ↑ | sStoryCloze ↑ | tStoryCloze ↑ | GenPPL ↓ | Auto-BLEU ↓ |
+|------------------------------------------|--------------------|------------|----------|--------------|--------------|---------|------------|
+| [TWIST-1.3B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                      | 160xV100          | 1B         | 57.00    | 52.4         | 70.6         | 131.8   | 3.20       |
+| [TWIST-7B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                        | ?                 | 7B         | 59.00    | 55.3         | 74.1         | 93.7    | 3.06       |
+| [TWIST-13B](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)                       | ?                 | 13B        | 59.20    | 55.4         | 76.4         | -       | -          |
+| [Scaled Optimal](https://arxiv.org/abs/2404.00685)      | ?                 | 823M       | **61.3** | **56.7**     | **78.0**     | -       | -          |
+| [Predicted Optimal]((https://arxiv.org/abs/2404.00685))   | 1xA5000           | 78M        | 56.85    | 54.09        | 70.49        | -       | -          |
+| TWIST-350M (Original recipe)            | 1xA5000           | 305M       | 51.52 ± .19 | 53.65 ± .57 | 68.80 ± .47 | 259.2 ± 6.7 | 3.26 ± .46 |
+| *Slam (-DPO) (ours)*                    | 1xA5000           | 358M       | *56.45* ± .17 | *55.59* ± .30 | *78.01* ± .27 | *88.3* ± 1.0 | 3.47 ± .17 |
+| **Slam (ours)**                       | 1xA5000           | 358M       | **58.86** ± .20 | **58.04** ± .51 | **82.04** ± .21 | **62.8** ± 4.1 | 3.88 ± .11 |
 
-**ADD Table**
-
-
-
-### Model Architecture and Objective
 
-[More Information Needed]
 
 ### Compute Infrastructure
 This model was trained as part of ["*Slamming*: Training a Speech Language Model on One GPU in a Day"], focusing on efficient training.