Create README.md

README.md

@@ -0,0 +1,62 @@
+# lina-speech (beta)
+
+Exploring "linear attention" for text-to-speech.
+
+It predicts audio codec "à la" [MusicGen](https://arxiv.org/abs/2306.05284) : delayed residual vector quantizers so that we do not need multiple models.
+
+Featuring [RWKV](https://github.com/BlinkDL/RWKV-LM), [Mamba](https://github.com/state-spaces/mamba), [Gated Linear Attention](https://github.com/sustcsonglin/flash-linear-attention).
+
+Compared to other LM TTS model :
+- Can be easily pretrained and finetuned on midrange GPUs.
+- Tiny memory footprint.
+- Trained on long context (up to 2000 tokens : ~27s).
+
+### Models
+
+| Model | #Params | Dataset | Checkpoint | Steps | Note |
+| :---: | :---: |:---: |:---: |:---: |:---: |
+| GLA | 60M, 130M | Librilight-medium | [Download](https://nubo.ircam.fr/index.php/s/wjNYLb54m7L8xf9) | 300k | GPU inference only |
+| Mamba| 60M | Librilight-medium |[Download](https://nubo.ircam.fr/index.php/s/wjNYLb54m7L8xf9)| 300k | GPU inference only |
+| RWKV v6 | 60M | LibriTTS |[Download](https://nubo.ircam.fr/index.php/s/wjNYLb54m7L8xf9) | 150k | GPU inference only |
+
+### Installation
+Following the linear complexity LM you choose, follow respective instructions first:
+- For Mamba check the [official repo](https://github.com/state-spaces/mamba).
+- For GLA/RWKV inference check [flash-linear-attention](https://github.com/sustcsonglin/flash-linear-attention).
+- For RWKV training check [RWKV-LM](https://github.com/BlinkDL/RWKV-LM)
+
+### Inference
+
+Download configuration and weights above, then check `Inference.ipynb`.
+
+### TODO
+
+- [x] Fix RWKV6 inference and/or switch to FLA implem.
+- [ ] Provide a Datamodule for training (_lhotse_ might also work well).
+- [ ] Implement CFG.
+- [ ] Scale up.
+
+### Acknowledgment
+
+- The RWKV authors and the community around for carrying high-level truly opensource research.
+- @SmerkyG for making my life easy at testing cutting edge language model.
+- @lucidrains for its huge codebase.
+- @sustcsonglin who made [GLA and FLA](https://github.com/sustcsonglin/flash-linear-attention).
+- @harrisonvanderbyl fixing RWKV inference.
+
+### Cite
+```bib
+@software{lemerle2024linaspeech,
+  title  = {LinaSpeech: Exploring "linear attention" for text-to-speech.},
+  author = {Lemerle, Théodor},
+  url    = {https://github.com/theodorblackbird/lina-speech},
+  month  = april,
+  year   = {2024}
+}
+```
+### IRCAM
+
+This work takes place at IRCAM, and is part of the following project :
+[ANR Exovoices](https://anr.fr/Projet-ANR-21-CE23-0040)
+
+<img align="left" width="200" height="200" src="logo_ircam.jpeg">