Add model metadata

README.md

@@ -0,0 +1,114 @@
+---
+metrics:
+- MFCC-DTW
+- ZCR
+- Chroma Score
+- Spectral Score
+model-index:
+- name: SEE-2-SOUND
+  results:
+  - task:
+      type: spatial-audio-generation
+      name: Spatial Audio Generation
+    dataset:
+      type: rishitdagli/see-2-sound-eval
+      name: SEE-2-SOUND Evaluation Dataset
+    metrics:
+      - type: MFCC-DTW
+        value: 0.03 × 10^-3
+        name: AViTAR Marginal Scene Guidance - Mel-Frequency Cepstral Coefficient - Dynamic Time Warping
+      - type: ZCR
+        value: 0.95
+        name: AViTAR Marginal Scene Guidance - Zero Crossing Rate
+      - type: Chroma
+        value: 0.77
+        name: Chroma Feature
+      - type: Spectral Score
+        value: 0.95
+        name: AViTAR Marginal Scene Guidance - Spectral Score
+    source:
+      name: arXiv
+      url: https://arxiv.org/abs/2406.06612
+tags:
+- vision
+- audio
+- spatial audio
+- audio generation
+- music
+- art
+---
+
+<h2>SEE-2-SOUND🔊: Zero-Shot Spatial Environment-to-Spatial Sound</h2>
+
+[**Rishit Dagli**](https://rishitdagli.com/)<sup>1</sup> · [**Shivesh Prakash**](https://shivesh777.github.io/)<sup>1</sup> · [**Rupert Wu**](https://www.cs.toronto.edu/~rupert/)<sup>1</sup> · [**Houman Khosravani**](https://scholar.google.ca/citations?user=qzhk98YAAAAJ&hl=en)<sup>1,2,3</sup>
+
+<sup>1</sup>University of Toronto&emsp;&emsp;&emsp;&emsp;<sup>2</sup>Temerty Centre for Artificial Intelligence Research and Education in Medicine&emsp;&emsp;&emsp;&emsp;<sup>3</sup>Sunnybrook Research Institute
+
+| | | | |
+|---|---|---|---|
+| [![Paper PDF](https://img.shields.io/badge/arXiv-See2Sound-red)](https://arxiv.org/abs/2406.06612) | [![Project Page](https://img.shields.io/badge/Project_Page-See2Sound-green)](https://see2sound.github.io) | [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/rishitdagli/see-2-sound) | [![Hugging Face Paper](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Paper-yellow)](https://huggingface.co/papers/2406.06612) |
+
+This work presents **SEE-2-SOUND**, a method to generate spatial audio from images, animated images, and videos to accompany the visual content. Check out our [website](https://see2sound.github.io) to view some results of this work.
+
+![teaser](https://raw.githubusercontent.com/see2sound/see2sound/main/assets/teaser.png
+
+These checkpoints are meant to be used with our code: [SEE-2-SOUND](https://github.com/see2sound/see2sound).
+
+## Installation
+
+First, install the pip package and download these checkpoints (needs Git LFS):
+
+```sh
+pip install -e git+https://github.com/see2sound/see2sound.git#egg=see2sound
+git clone https://huggingface.co/rishitdagli/see-2-sound
+cd see-2-sound
+```
+
+View the full installation instructions as well a tips on dependencies in the [repository README](https://github.com/see2sound/see2sound/tree/main?tab=readme-ov-file#installation).
+
+## Running the Models
+
+Now, we can start by making a configuration file, make a file called `config.yaml`:
+
+```yaml
+codi_encoder: 'codi/codi_encoder.pth'
+codi_text: 'codi/codi_text.pth'
+codi_audio: 'codi/codi_audio.pth'
+codi_video: 'codi/codi_video.pth'
+
+sam: 'sam/sam.pth'
+# H, L or B in decreasing performance
+sam_size: 'H'
+
+depth: '/depth/depth.pth'
+# L, B, or S in decreasing performance
+depth_size: 'L'
+
+download: False
+
+# Change to True if your GPU has < 40 GB vRAM
+low_mem: False
+fp16: False
+gpu: True
+steps: 500
+num_audios: 3
+prompt: ''
+verbose: True
+```
+
+Now, we can start running inference:
+
+```py
+import see2sound
+
+
+config_file_path = "config.yaml"
+
+model = see2sound.See2Sound(config_path = config_file_path)
+model.setup()
+model.run(path = "test.png", output_path = "test.wav")
+```
+
+## More Information
+
+Feel free to take a look at the full [dcoumentation](https://github.com/see2sound/see2sound/blob/main/README.md) for extra information and tips on running the model.