Create README.md

README.md

@@ -0,0 +1,84 @@
+---
+datasets:
+- DaveLoay/NSynth_Bass_Captions
+language:
+- en
+---
+# Riffsuion Fine-Tune
+
+This is a Fine-Tuned version of **Rifussion**, trained on **bass** samples extracted from the **NSynth** dataset.
+The porpuse of this work is to evaluate the performance of the model to generate bass audio samples.
+
+## Notes
+
+This is the way I found to achieve this goal, if you have a better idea for doing this, please share it with me.
+
+## Quickstart Guide
+
+Clone the **Riffusion** repository and install the requirements.txt file from: [Riffusion Github](https://github.com/riffusion/riffusion) 
+```python
+import torch
+from diffusers import DiffusionPipeline
+
+pipe = DiffusionPipeline.from_pretrained("DaveLoay/Riffusion_FT_Bass_512_4000", torch_dtype=torch.float16).to(device)
+prompt = "Your desired prompt"
+image = pipe(prompt).images[0]
+```
+
+After that, you would have been generated an spectrogram saved on image. So if you want to convert this image into an audio file, you could use the **spectrogram_image_converter** mehtod contained in the **Rifussion** repo.
+
+```python
+from riffusion.spectrogram_image_converter import SpectrogramImageConverter
+from riffusion.spectrogram_params import SpectrogramParams
+
+params = SpectrogramParams()
+converter = SpectrogramImageConverter(params)
+audio = converter.audio_from_spectrogram_image(image)
+```
+
+## Fine Tuning
+
+For the Fine-Tuning process, I used the bass samples from the test split in the NSynth dataset, which you can check out here: [NSynth Dataset](https://magenta.tensorflow.org/nsynth) 
+
+You can find the pre-processed files in my repo, here: [DaveLoay/NSynth_Bass_Captions](DaveLoay/NSynth_Bass_Captions)
+
+And as mention in the official **Rifussion** HF repo, I used the **train_text_to_image** script contained in the **Diffusers** repo, which you can check out here: [Diffusers Repo](https://github.com/huggingface/diffusers/tree/main/examples/text_to_image)
+
+After configuring all dependencies, I used the following code to train the model:
+
+```bash
+  accelerate launch --mixed_precision="fp16"  train_text_to_image.py \
+    --pretrained_model_name_or_path=riffusion/riffusion-model-v1 \
+    --dataset_name=DaveLoay/NSynth_Bass_Captions \
+    --resolution=512 \
+    --use_ema \
+    --train_batch_size=3 \
+    --gradient_accumulation_steps=4 \
+    --gradient_checkpointing \
+    --max_train_steps=4000 \
+    --learning_rate=1e-05 \
+    --max_grad_norm=1 \
+    --lr_scheduler="constant" --lr_warmup_steps=0 \
+    --output_dir="Riffusion_FT_Bass_512_4000" \
+    --push_to_hub
+```
+
+## Hardware
+
+The hardware I used to fine-tune this model is:
+
+* NVIDIA A100 40 GB vRAM hosted in Google Colab Pro
+
+It took about 3 hours to complete the training process, and used about ~26 GB of vRAM.
+
+## Credits
+
+You can check the original repositories here:
+
+[Riffusion](https://www.riffusion.com/)
+
+[NSynth Dataset](https://magenta.tensorflow.org/nsynth)
+
+[Diffusers](https://huggingface.co/docs/diffusers/index)
+
+