Delete README (1).md

README (1).md

@@ -1,216 +0,0 @@
----
-license: mit
-language:
-- en
-- zh
-base_model:
-- Qwen/Qwen2-0.5B
-pipeline_tag: feature-extraction
-library_name: sentence-transformers
-tags:
-- MoE
-- Unified Generation
-- Speech and Music
-- Multi-modal
-datasets:
----
-
-<h1 align="center">UniMoE-Audio</h1>
-
-**UniMoE-Audio** is a unified framework that seamlessly combines speech and music generation. Powered by a novel dynamic-capacity Mixture-of-Experts design, it adapts intelligently to input complexity, enabling high-fidelity voice and expressive music within a single model.
-
-## Key Innovations
-
-#### **Top-P Dynamic Routing Strategy**
-We introduce a **Top-P routing strategy** that overcomes the limitations of conventional static Top-K routing:
-
-- **Dynamic Expert Allocation**: Instead of assigning a fixed number of experts to every token, our approach dynamically determines the number of experts based on token complexity
-- **Resource Efficiency**: Simple tokens don't consume unnecessary resources, while complex tokens receive sufficient processing power
-- **Performance Optimization**: Results in improved overall efficiency and performance
-
-#### **Three-Stage Training Curriculum**
-We employ a comprehensive training approach to enable effective joint learning from imbalanced data:
-
-1. **Independent Specialist Training** - Initial expert specialization
-2. **Integration with Warm-up** - Gradual system integration  
-3. **Synergistic Joint Training** - Collaborative optimization
-
-## Model Information
-- **Base Model**: Qwen2.5-VL with MoE extensions
-- **Audio Codec**: DAC (Descript Audio Codec) with 12 channels
-- **Expert Configuration**: 8 dynamic experts + 2 shared experts
-- **Audio Sampling Rate**: 16kHz
-- Usage:
-  - Text-to-Speech (TTS)
-  - Speech-to-Text (STT)
-  - Music Generation
-- GPU Requirements:
-  - Memory: 16GB+
-  - CUDA-enabled GPU
-
-## Open-source Plan
-- [☑️] Model Checkpoint 
-    - [☑️] [UniMoE-Audio-preview](https://huggingface.co/foggyforest/UniMoE-Audio-preview)
-- [☑️] Inference Code: [HITsz-TMG/UniMoE-Audio](https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs/tree/master/UniMoE-Audio)
-- [☑️] Technical Report: [UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoE]()
-
-## Evaluation
-### Speech Synthesis
-![Speech Synthesis](./imgs/Speech_Generation.png)
-### Text to Music Generation
-![Text to Music Generation](./imgs/T2M.png)
-### Video-Text to Music Generation
-![Video-Text to Music Generation](./imgs/VT2M.png)
-
-## Requirements
-We recommend using conda to install the environment.
-```bash
-conda env create -f configs/enviroment.yml      # add -n for your name
-conda activate unimoe-audio                     # default name
-```
-then install the torch packages
-  ```bash
-   # Use the official index
-   pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu121
-   
-   # Use Tsinghua mirror source
-   pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 -i https://pypi.tuna.tsinghua.edu.cn/simple/ --extra-index-url https://download.pytorch.org/whl/cu121
-   
-   # Use Alibaba Cloud mirror source
-   pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 -i https://mirrors.aliyun.com/pypi/simple/ --extra-index-url https://download.pytorch.org/whl/cu121
-   ```
-A `dac model` is also required to be downloaded in '/path/to/UniMoE-Audio/utils/dac_model'.
-It will be automatically downloaded when running the first time.
-
-
-## Usage
-Please move to the `utils` folder to your working directory.
-Then you can use the model like this:
-
-```python
-from modeling import UniMoEAudio
-
-MODEL_NAME= "HIT-TMG/UniMoE-Audio-Preview"
-
-# Load model
-unimoe_audio = UniMoEAudio.from_pretrained(
-    MODEL_NAME,
-    cache_dir='./cache',
-    torch_dtype='bfloat16',
-    device_id=0
-) 
-
-```
-
-### TTS Example:
-```python
-# TTS/Voice Cloning
-target_text = "Target Text"
-prompt_audio = "/path/to/your/prompt_audio.wav"
-prompt_text  = "Prompt Text"
-
-# Encode prompt audio
-prompt_codec = unimoe_audio.dac.encode(prompt_audio)
-
-prompt_codec_input_ids = unimoe_audio._preprocess_codec(
-      codec=prompt_codec,
-      codec_delay_pattern=unimoe_audio.model.config.codec_delay_pattern,
-      codec_channels=unimoe_audio.model.num_channels,
-      codec_bos_value=unimoe_audio.model.config.codec_bos_value,
-      codec_eos_value=unimoe_audio.model.config.codec_eos_value,
-      codec_pad_value=unimoe_audio.model.config.codec_pad_value
-  )
-
-# Construct prompt text
-text_input, _, _ = unimoe_audio._prepare_prompt(task="speech", caption=target_text, prompt_text=prompt_text, prompt_codec_input_ids=prompt_codec_input_ids)
-
-# Tokenize input text
-source_input = unimoe_audio.tokenizer(text_input, add_special_tokens=False, return_tensors="pt", padding=True)
-prompt_codec_input_ids = prompt_codec_input_ids.unsqueeze(0).expand(len(text_input), -1, -1).reshape(-1, prompt_codec_input_ids.shape[1])
-
-#Speech Generation
-unimoe_audio._generate_core(
-        source_input, 
-        prompt_codec_input_ids, 
-        save_name = "speech", 
-        output_dir = "./",
-        cfg_scale = 1.0, 
-        temperature = 1.0, 
-        top_p = 1.0, 
-        cfg_filter_top_k = 45, 
-        eos_prob_mul_factor = 1.0, 
-        do_sample = True, 
-        debug_guidance_step = -1, 
-        use_cache = True
-        )
-```
-### T2M Example:
-```python
-caption = "music deccription"
-
-# Construct prompt text
-text_input, _, _ = unimoe_audio._prepare_prompt(task="music", caption=caption)
-
-# Tokenize input text
-source_input = unimoe_audio.tokenizer(text_input, add_special_tokens=False, return_tensors="pt", padding=True)
-
-#music generation with prompt text
-unimoe_audio._generate_core(
-        source_input, 
-        None, 
-        save_name = "music", 
-        output_dir = "./",
-        cfg_scale = 10.0, 
-        temperature = 1.0, 
-        top_p = 1.0, 
-        cfg_filter_top_k = 45, 
-        eos_prob_mul_factor = 0.6, 
-        do_sample = True, 
-        debug_guidance_step = -1, 
-        use_cache = True
-        )
-```
-
-### VT2M Example:
-```python
-# VT2M
-caption = "music deccription"
-prompt_video = "/path/to/your/video.mp4"
-
-#prepare prompt
-text_input, video_inputs, fps_inputs = unimoe_audio._prepare_prompt(task="music", caption=caption, video=prompt_video, fps=1, sampling_fps=1, max_frames=1)
-
-#input processor
-source_input = unimoe_audio.processor(
-            text=text_input, 
-            images=None, 
-            videos=video_inputs, 
-            fps=fps_inputs, 
-            padding=True, 
-            return_tensors="pt", 
-            do_resize=False
-        )
-
-#music generation with prompt video
-unimoe_audio._generate_core(
-        source_input, 
-        None, 
-        save_name = "video_music", 
-        output_dir = "./",
-        rebuild_codec=None, 
-        cfg_scale = 10.0, 
-        temperature = 1.0, 
-        top_p = 1.0, 
-        cfg_filter_top_k = 45, 
-        eos_prob_mul_factor = 0.6, 
-        do_sample = True, 
-        debug_guidance_step = -1, 
-        use_cache = True
-        )
-```
-
-
-
-
-
-