update

README.md

@@ -6,32 +6,182 @@ tags:
   - speech
   - diffusion
   - multimodal
+  - audio-generation
+  - audio-editing
+  - video-to-audio
+  - text-to-speech
 ---
 
-# Audio-Omni
+# 🎛️ Audio-Omni
 
 **Unified Audio Understanding, Generation, and Editing** (SIGGRAPH 2026)
 
-## License
+[![GitHub](https://img.shields.io/badge/GitHub-Repository-black?logo=github)](https://github.com/ZeyueT/Audio-Omni)
+[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://zeyuet.github.io/Audio-Omni/)
+[![arXiv](https://img.shields.io/badge/arXiv-Paper-red)](https://arxiv.org/abs/XXXX.XXXXX)
 
-The model weights are released under **CC-BY-NC-4.0** (non-commercial use only).
+## 📖 Overview
 
-**Commercial use requires explicit written authorization.** Contact: ztianad@connect.ust.hk
+Audio-Omni is the first end-to-end framework that unifies **understanding**, **generation**, and **editing** across general sound, music, and speech domains. It combines a frozen Multimodal Large Language Model (Qwen2.5-Omni) for high-level reasoning with a trainable Diffusion Transformer for high-fidelity synthesis.
 
-## Links
+## 🎯 Capabilities
 
-- [GitHub Repository](https://github.com/Audio-Omni/Audio-Omni)
-- [Project Page](https://zeyuet.github.io/Audio-Omni/)
+- **Understanding**: Audio/video captioning, question answering
+- **Generation**: Text-to-Audio, Text-to-Music, Video-to-Audio, Video-to-Music, Text-to-Speech, Voice Conversion
+- **Editing**: Add, Remove, Extract, Style Transfer
 
-## Files
+## 📦 Model Files
 
 - `Audio-Omni.json` — Model configuration
 - `model.ckpt` — Model checkpoint (~21 GB)
+- `synchformer_state_dict.pth` — Synchformer checkpoint for video conditioning
 
-## Quick Start
+## 🚀 Quick Start
+
+### Installation
 
 ```bash
+# Clone the GitHub repository
+git clone https://github.com/ZeyueT/Audio-Omni.git
+cd Audio-Omni
+
+# Install dependencies
+pip install -e .
+conda install -c conda-forge ffmpeg libsndfile
+
+# Download model from Hugging Face
 huggingface-cli download HKUSTAudio/Audio-Omni --local-dir model/
 ```
 
-See the [GitHub repository](https://github.com/Audio-Omni/Audio-Omni) for full usage instructions.
+### Python API
+
+```python
+from audio_omni import AudioOmni
+import torchaudio
+
+# Load model
+model = AudioOmni("model/Audio-Omni.json", "model/model.ckpt")
+```
+
+### 1️⃣ Understanding
+
+```python
+# Audio understanding
+response = model.understand(
+    "Describe the sounds in this audio.", 
+    audio="example.wav"
+)
+
+# Video understanding
+response = model.understand(
+    "What is happening in this video?", 
+    video="example.mp4"
+)
+
+# Audio + Video understanding
+response = model.understand(
+    "Does the audio match the video?", 
+    audio="example.wav", 
+    video="example.mp4"
+)
+```
+
+### 2️⃣ Generation
+
+```python
+# Text-to-Audio
+audio = model.generate("T2A", prompt="A clock ticking.")
+torchaudio.save("output.wav", audio, model.sample_rate)
+
+# Text-to-Music
+audio = model.generate(
+    "T2M", 
+    prompt="Compose a bright jazz swing instrumental with walking bass."
+)
+torchaudio.save("music.wav", audio, model.sample_rate)
+
+# Video-to-Audio
+audio = model.generate("V2A", video_path="example.mp4")
+torchaudio.save("v2a_output.wav", audio, model.sample_rate)
+
+# Text-to-Speech
+audio = model.generate("TTS", prompt="Hello, welcome to Audio-Omni.")
+torchaudio.save("tts_output.wav", audio, model.sample_rate)
+
+# Text-to-Speech with voice cloning
+audio = model.generate(
+    "TTS",
+    prompt="Hello, welcome to Audio-Omni.",
+    voice_prompt_path="ref_voice.wav",
+    voice_ref_text="This is the reference transcript."
+)
+torchaudio.save("tts_cloned.wav", audio, model.sample_rate)
+```
+
+### 3️⃣ Editing
+
+```python
+# Add a sound
+audio = model.edit("Add", "input.wav", desc="skateboarding")
+torchaudio.save("output_add.wav", audio, model.sample_rate)
+
+# Remove a sound
+audio = model.edit("Remove", "input.wav", desc="female singing")
+torchaudio.save("output_remove.wav", audio, model.sample_rate)
+
+# Extract a sound
+audio = model.edit("Extract", "input.wav", desc="wood thrush calling")
+torchaudio.save("output_extract.wav", audio, model.sample_rate)
+
+# Style transfer
+audio = model.edit(
+    "Style Transfer", 
+    "input.wav",
+    source_category="playing electric guitar", 
+    target_category="playing saxophone"
+)
+torchaudio.save("output_transfer.wav", audio, model.sample_rate)
+```
+
+## 🖥️ Gradio Demo
+
+```bash
+# Launch interactive demo
+python run_gradio.py \
+    --model-config model/Audio-Omni.json \
+    --ckpt-path model/model.ckpt \
+    --server-port 7777
+```
+
+Visit `http://localhost:7777` to access the web interface.
+
+## 📚 Documentation
+
+For detailed documentation, training instructions, and more examples, visit the [GitHub repository](https://github.com/ZeyueT/Audio-Omni).
+
+## 📝 Citation
+
+```bibtex
+@inproceedings{tian2026audioomni,
+  title={Audio-Omni: Unified Audio Understanding, Generation, and Editing},
+  author={Tian, Zeyue and Yang, Binxin and Liu, Zhaoyang and Zhang, Jiexuan and Yuan, Ruibin and Yin, Hubery and Chen, Qifeng and Li, Chen and Lv, Jing and Xue, Wei and Guo, Yike},
+  booktitle={ACM Transactions on Graphics (SIGGRAPH 2026)},
+  year={2026}
+}
+```
+
+## 📄 License
+
+**Code**: Apache-2.0 License  
+**Model Weights**: CC-BY-NC-4.0 (Non-commercial use only)
+
+Commercial use of the model weights requires explicit written authorization from the authors.  
+For commercial licensing inquiries, contact: ztianad@connect.ust.hk
+
+## 📭 Contact
+
+- **Zeyue Tian**: ztianad@connect.ust.hk
+
+---
+
+**For full installation guide, API reference, and advanced usage, see the [GitHub repository](https://github.com/ZeyueT/Audio-Omni).**

model.ckpt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8ddfdd6c965b7b1fc5c5af01e306cdfd78e44f56fd584789b8987233438e65b3
-size 22220882720