Upload 14 files

README.md

@@ -0,0 +1,117 @@
+# SPIRIT-LM Expressive Interleaved (Corrected Teacher, Libri-Light)
+
+**SPIRIT-LM Expressive Interleaved (Corrected)** is a fine-tuned version of the 7B SPIRIT-LM teacher model adapted to the **Libri-Light** domain. It supports **interleaved speech and text inputs**, and was used as the **teacher model for distilling TinyWave**.
+
+This checkpoint was fine-tuned for 10k steps with **LoRA adapters** on synthetic interleaved data created from Libri-Light and Whisper transcriptions. The resulting model improves alignment with the target distribution and provides stronger supervision for expressive speech–text generation.
+
+> 📖 This checkpoint is part of the *TinyWave* distillation framework. See [arXiv:2506.23670](https://arxiv.org/abs/2506.23670) for details.
+
+---
+
+## 🧠 Model Purpose
+
+| Role             | Distillation Teacher                     |
+|------------------|-------------------------------------------|
+| Base Model       | `spirit-lm-expressive-7b` (SPIRIT-LM)     |
+| Fine-tuned on    | Libri-Light (10k steps with LoRA)         |
+| Input Modalities | Interleaved speech + text                 |
+| Output           | Speech tokens                             |
+| Used for         | Training `tinywave/interleaved-expressive-2b` |
+
+---
+
+## 🔧 Usage
+
+### 1. Install SPIRIT-LM and Load Expressive Tokenizer
+
+```bash
+git clone https://github.com/facebookresearch/spiritlm
+cd spiritlm
+pip install -e '.[eval]'
+````
+
+```python
+from spiritlm.speech_tokenizer import spiritlm_expressive
+speech_tokenizer = spiritlm_expressive()
+```
+
+---
+
+### 2. Inference (Speech or Interleaved)
+
+```python
+from transformers import LlamaForCausalLM, AutoTokenizer
+import torchaudio
+import torch
+
+MODEL_PATH = "tinywave/expressive-spirit-lm-interleaved-librilight"
+tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
+model = LlamaForCausalLM.from_pretrained(MODEL_PATH, device_map="auto", torch_dtype=torch.bfloat16)
+
+# Interleaved speech input
+speech_tokenizer = spiritlm_expressive()
+
+def get_inference(audio_path):
+    audio, _ = torchaudio.load(audio_path)
+    input_values = audio.view(1, 1, -1).to(speech_tokenizer.hubert_model.device).float()
+    tokens = speech_tokenizer.encode_string(input_values)
+    input_ids = tokenizer(tokens, return_tensors="pt").input_ids.to(model.device)
+    output = model.generate(input_ids, max_new_tokens=256, do_sample=True, temperature=0.9, top_p=0.9)
+    return tokenizer.decode(output[0])
+
+def get_inference_text(prompt):
+    input_ids = tokenizer(prompt + " [Speech]", return_tensors="pt").input_ids.to(model.device)
+    output = model.generate(input_ids, max_new_tokens=256, do_sample=True, temperature=0.9, top_p=0.9)
+    return tokenizer.decode(output[0])
+```
+
+---
+
+## 🎧 Inference Modes
+
+### 💬 Text + Speech Interleaving
+
+Input:
+
+```text
+"The astronaut stepped outside the capsule— [Speech]"
+```
+
+Output:
+Expressive speech continuation in WAV format.
+
+---
+
+### 🔄 Speech Continuation
+
+Input: `speech.wav`
+Output: Semantically and stylistically aligned spoken continuation.
+
+---
+
+## 📂 Files
+
+* `pytorch_model.bin`: LoRA-adapted SPIRIT-LM 7B weights
+* `config.json`, `tokenizer.json`: Compatible with Hugging Face Transformers
+* Compatible with `spiritlm_expressive` tokenizer only
+
+---
+
+## 📎 Citation
+
+```bibtex
+@article{nouriborji2025tinywave,
+  title={Efficient Interleaved Speech Modeling through Knowledge Distillation},
+  author={Nouriborji, Mohammadmahdi and Rohanian, Morteza},
+  journal={arXiv preprint arXiv:2506.23670},
+  year={2025}
+}
+```
+
+---
+
+## 🔗 Related
+
+* 🔬 Paper: [arXiv:2506.23670](https://arxiv.org/abs/2506.23670)
+* 🧠 Student model: [`tinywave/interleaved-expressive-2b`](https://huggingface.co/tinywave/interleaved-expressive-2b)
+* 🌐 [Project Website](https://mohammadmahdinoori.github.io/tinywave-landing/)

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 16384,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 100000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "vocab_size": 32768
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.51.3"
+}

gitattributes.txt

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

model-00001-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ac270904565dd8f9aeaa83d24467eae348e72eb8cdce5e4dd380d2b8f71004a
+size 4852979328

model-00002-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64ec253cd3ef03c34ce7d3d0496923cfa01d84ef801028ad1022dbf7174195b6
+size 4857206856

model-00003-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dff7bdedeaafb4d9d3627da5876ba2014f7d5806c2cf89baa0e30961b3220e20
+size 4857206904

model-00004-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:728d45e2a8492a349906c7f2107d3a2b4db8bfe62f1cde1c55d208318ec5c564
+size 4857206904

model-00005-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d90cd61a40c5d8f533380bc2fcbc1b85c144ce6fa738bda97236a6198346efed
+size 4857206904

model-00006-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86bca5dfd3e4217fc1bdca507efec3f1ad83a68feb7096c9ecfbb6b8df6eff7e
+size 2697055024

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1608b90876103c6c3c67ca079ab9d3c5ee4e7707acf869103e728a4d30626643
+size 514364