Create README.md

README.md

@@ -1,3 +1,160 @@
 ---
+language:
+- en
+- zh
 license: apache-2.0
+pipeline_tag: audio-text-to-text
+tags:
+- speech-language-model
+- streaming
+- audio
+- multimodal
+- qwen2.5-omni
+datasets:
+- zhifeixie/StreamAudio-2M
+base_model:
+- Qwen/Qwen2.5-Omni-3B
 ---
+# Mini-Omni3: Streaming Audio-In, Text-Out Conversational Model
+
+[**Code**](https://github.com/xzf-thu/Mini-Omni3) <!-- TODO: confirm repo URL -->
+
+Mini-Omni3 is a streaming speech-language model that listens to audio in real time and decides, at each audio chunk, whether to keep listening or to start replying with text. The model alternates between a **LISTENING** state, where it consumes one encoder-output chunk per step and emits either `KEEP_SILENCE` or `TEXT_BEGIN`, and a **SPEAKING** state, where it autoregressively generates a text turn until `TEXT_END` and then returns to listening for the next chunk.
+
+This design lets the model handle both spoken questions ("answer it") and ambient sounds ("decide based on the sound whether help is needed") within a single streaming session, without an external VAD or turn-taking heuristic.
+
+The release contains the Mini-Omni3 language-model weights (sharded safetensors), a chunk-wise audio encoder adapted from Qwen2.5-Omni, and the matching tokenizer and model config.
+
+## Model Details
+
+- **Model name:** Mini-Omni3
+- **Task:** Streaming audio-conditioned text generation (audio in, text out)
+- **Audio encoder:** Qwen2.5-Omni audio tower (chunk-wise)
+- **Audio framing:** 16 kHz, padded to 0.4-second (6400-sample) boundaries; 10 encoder-output frames per chunk
+- **Decoding states:** LISTENING (emits `KEEP_SILENCE` / `TEXT_BEGIN`) and SPEAKING (emits text until `TEXT_END`)
+- **Default sampling:** temperature 0.3, top-k 3
+- **Default max new tokens:** 4096 per session
+- **License:** Apache-2.0
+
+## Repository Contents
+
+```text
+Mini-Omni3/
+├── MiniOmni3_LM_sharded/                  # Sharded safetensors of the LM weights
+│   ├── model.safetensors.index.json
+│   └── model-0000N-of-0000N.safetensors
+├── MiniOmni3_ChunkwisedEncoder.pth        # Audio encoder weights (Qwen2.5-Omni audio tower)
+├── qwen_2_5_omni_config/                  # Audio-encoder config (nested: thinker_config.audio_config)
+├── model_config.yaml                      # GPT config consumed by Config.from_file
+├── tokenizer.json                         # Tokenizer
+└── README.md                              # This card
+```
+
+## Intended Use
+
+Mini-Omni3 is intended for streaming conversational agents that need to react to audio as it arrives — for example, voice assistants that may interject mid-utterance, alarms that respond to ambient sound, or low-latency dialogue systems where waiting for a full utterance before replying is too slow. The model is not a transcription system; it produces a conversational reply (or silence) rather than a verbatim transcript.
+
+## Quick Start
+
+### Installation
+
+```bash
+git clone https://github.com/xzf-thu/Mini-Omni3.git  # TODO: confirm repo URL
+cd Mini-Omni3
+conda create -n mini-omni3 python=3.10 -y
+conda activate mini-omni3
+pip install -r requirements.txt
+```
+
+### Download the checkpoint
+
+```python
+from huggingface_hub import snapshot_download
+
+local_dir = snapshot_download(
+    repo_id="zhifeixie/Mini-Omni3",
+    repo_type="model",
+)
+print(local_dir)
+```
+
+### Python Usage
+
+```python
+from src.miniomni3.generate.run import run_inference
+
+run_inference(
+    checkpoint_dir=local_dir,           # the path snapshot_download returned
+    audio_paths=["/path/to/audio.wav"], # offline mode: one round per path
+    device="cuda:0",                    # or "mps" / "cpu"
+)
+```
+
+For interactive use, omit `audio_paths` and `run_inference` will prompt for an audio path each round:
+
+```python
+run_inference(checkpoint_dir=local_dir, rounds=5, device="cuda:0")
+```
+
+## Streaming Protocol
+
+A single session looks like:
+
+```text
+[system prompt tokens]
+  ┌─── LISTENING ───┐
+  │ AUDIO_BEGIN PAD*10 ASSISTANT  →  KEEP_SILENCE          (keep listening)
+  │ AUDIO_BEGIN PAD*10 ASSISTANT  →  TEXT_BEGIN EMOTION    (start replying)
+  └─────────────────┘
+  ┌─── SPEAKING ────┐
+  │ … text tokens … TEXT_END                                (reply finished)
+  └─────────────────┘
+  ┌─── LISTENING ───┐  (next audio chunk)
+  …
+```
+
+The model is trained to emit at most one `TEXT_BEGIN` per audio chunk. Each assistant turn begins with `TEXT_BEGIN`, followed by an emotion token, the reply tokens, and `TEXT_END`. Turns starting with `KEEP_SILENCE` indicate the model chose not to respond to that chunk.
+
+## Training Summary
+
+<!-- TODO: fill in once details are public.
+Suggested fields:
+- Pretraining base
+- SFT / instruction-tuning data
+- Streaming-objective data construction (how KEEP_SILENCE / TEXT_BEGIN supervision was generated)
+- Total tokens / hours of audio
+- Hardware and duration
+-->
+
+## Evaluation
+
+<!-- TODO: fill in once benchmarks are decided.
+Candidate metrics:
+- Spoken-QA accuracy on held-out audio prompts
+- False-trigger rate on ambient / non-speech audio (lower is better)
+- Response-onset latency in encoder chunks from end of question
+- Text quality of replies (e.g. GPT-judge or human preference)
+-->
+
+## Limitations
+
+- The model produces text, not speech. Pair it with a TTS system for end-to-end voice interaction.
+- Audio must be 16 kHz mono; non-conforming inputs are resampled by `whisper.load_audio` and padded to 0.4-second boundaries before encoding.
+- Decisions are made at 0.4-second granularity (one encoder chunk), which sets a floor on response-onset latency.
+- Trailing partial audio chunks shorter than 10 encoder frames are dropped before generation.
+
+## Citation
+
+<!-- TODO: replace with the real arxiv id and year once published. -->
+```bibtex
+@misc{xie_miniomni3,
+  title  = {Mini-Omni3: Streaming Audio-In, Text-Out Conversational Modeling},
+  author = {Zhifei Xie and collaborators},
+  year   = {2026},
+  note   = {Preprint in preparation}
+}
+```
+
+## Acknowledgements
+
+Mini-Omni3 builds on the Qwen2.5-Omni audio encoder. We thank the Qwen team and the maintainers of OpenAI Whisper for the audio-loading utilities used in this project.