Update README.md

README.md

@@ -64,12 +64,14 @@ library_name: transformers
 
 ## VibeVoice-ASR (Transformers-compatible version)
 [![GitHub](https://img.shields.io/badge/GitHub-Repo-black?logo=github)](https://github.com/microsoft/VibeVoice)
+[![Deploy on Foundry](https://img.shields.io/badge/Deploy-on_Foundry-blue?logo=microsoft)](https://huggingface.co/docs/microsoft-azure/foundry/guides/vibevoice-asr)
 [![Live Playground](https://img.shields.io/badge/Live-Playground-green?logo=gradio)](https://aka.ms/vibevoice-asr)
 [![Technical Report](https://img.shields.io/badge/arXiv-2601.18184-b31b1b?logo=arxiv)](https://arxiv.org/pdf/2601.18184)
 
 **VibeVoice-ASR** is a unified speech-to-text model designed to handle **60-minute long-form audio** in a single pass, generating structured transcriptions containing **Who (Speaker), When (Timestamps), and What (Content)**, with support for **Customized Hotwords** and over **50 languages**.
 
 ➡️ **Demo:** [VibeVoice-ASR-Demo](https://aka.ms/vibevoice-asr)<br>
+➡️ **Deploy:** [Deploy VibeVoice ASR on Microsoft Foundry: Long-Form Transcription, +50 Languages Supported & Speaker Diarization](https://huggingface.co/docs/microsoft-azure/foundry/guides/vibevoice-asr)<br>
 ➡️ **Report:** [VibeVoice-ASR Technical Report](https://arxiv.org/pdf/2601.18184)<br>
 
 <p align="left">
@@ -99,32 +101,14 @@ library_name: transformers
 
 VibeVoice-ASR is available as of v5.3.0 of Transformers!
 
-```
+```bash
 pip install "transformers>=5.3.0"
-```
-
-### Loading model
-
-```python
-from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
+Loading modelPythonfrom transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
 model_id = "microsoft/VibeVoice-ASR-HF"
 processor = AutoProcessor.from_pretrained(model_id)
 model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id)
-```
-
-### Speaker-timestamped transcription
-
-A notable feature of VibeVoice ASR is its ability to transcribe multi-speaker content, denoting who spoke and when.
-
-The example below transcribes the following audio.
-
-<audio controls>
-  <source src="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav" type="audio/wav">
-</audio>
-
-```python
-from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
+Speaker-timestamped transcriptionA notable feature of VibeVoice ASR is its ability to transcribe multi-speaker content, denoting who spoke and when.The example below transcribes the following audio.<audio controls><source src="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav" type="audio/wav"></audio>Pythonfrom transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
 model_id = "microsoft/VibeVoice-ASR-HF"
 processor = AutoProcessor.from_pretrained(model_id)
@@ -133,7 +117,7 @@ print(f"Model loaded on {model.device} with dtype {model.dtype}")
 
 # Prepare inputs using `apply_transcription_request`
 inputs = processor.apply_transcription_request(
-    audio="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
+    audio="[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav)",
 ).to(model.device, model.dtype)
 
 # Apply model
@@ -180,22 +164,7 @@ TRANSCRIPTION ONLY
 ============================================================
 Hello everyone and welcome to the Vibe Voice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me. Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings. Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible. Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it.
 """
-```
-
-The VibeVoice ASR model is trained to generate a string that resembles a JSON structure. The flag `return_format="parsed"` tries to return the generated output as a list of dicts, while `return_format="transcription_only"` tries to extract only the transcribed audio. If they fail, the generated output is returned as-is.
-
-### Providing context
-
-It is also possible to provide context. This can be useful if certain words cannot be transcribed correctly, such as proper nouns.
-
-Below we transcribe an audio where the speaker (with a German accent) talks about VibeVoice, comparing with and without the context "About VibeVoice".
-
-<audio controls>
-  <source src="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav" type="audio/wav">
-</audio>
-
-```python
-from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
+Providing contextIt is also possible to provide context. This can be useful if certain words cannot be transcribed correctly, such as proper nouns.Below we transcribe an audio where the speaker (with a German accent) talks about VibeVoice, comparing with and without the context "About VibeVoice".<audio controls><source src="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav" type="audio/wav"></audio>Pythonfrom transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
 model_id = "microsoft/VibeVoice-ASR-HF"
 processor = AutoProcessor.from_pretrained(model_id)
@@ -204,7 +173,7 @@ print(f"Model loaded on {model.device} with dtype {model.dtype}")
 
 # Without context
 inputs = processor.apply_transcription_request(
-    audio="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
+    audio="[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
 ).to(model.device, model.dtype)
 output_ids = model.generate(**inputs)
 generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
@@ -213,7 +182,7 @@ print(f"WITHOUT CONTEXT: {transcription}")
 
 # With context
 inputs = processor.apply_transcription_request(
-    audio="https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
+    audio="[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
     prompt="About VibeVoice",
 ).to(model.device, model.dtype)
 output_ids = model.generate(**inputs)
@@ -225,20 +194,12 @@ print(f"WITH CONTEXT   : {transcription}")
 WITHOUT CONTEXT: Revevoices is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio.
 WITH CONTEXT   : VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker, conversational audio.
 """
-```
-
-
-### Batch inference
-
-Batch inference is possible by passing a list of audio and (if provided) a list of prompts of equal length.
-
-```python
-from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
+Batch inferenceBatch inference is possible by passing a list of audio and (if provided) a list of prompts of equal length.Pythonfrom transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
 model_id = "microsoft/VibeVoice-ASR-HF"
 audio = [
-    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
-    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav"
+    "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
+    "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav)"
 ]
 prompts = ["About VibeVoice", None]
 
@@ -252,22 +213,13 @@ generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
 transcription = processor.decode(generated_ids, return_format="transcription_only")
 
 print(transcription)
-```
-
-### Adjusting tokenizer chunk (e.g. if out-of-memory)
-
-A key feature of VibeVoice ASR is that it can transcribe up to 60 minutes of continuous audio. This is done by chunking audio into 60-second segments (1440000 samples at 24kHz) and caching the convolution states between each segment.
-
-However, if chunks of 60 seconds are too large for your device, the `tokenizer_chunk_size` argument passed to `generate` can be adjusted. *Note it should be a multiple of the hop length (3200 for the original acoustic tokenizer).*
-
-```python
-from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
+Adjusting tokenizer chunk (e.g. if out-of-memory)A key feature of VibeVoice ASR is that it can transcribe up to 60 minutes of continuous audio. This is done by chunking audio into 60-second segments (1440000 samples at 24kHz) and caching the convolution states between each segment.However, if chunks of 60 seconds are too large for your device, the tokenizer_chunk_size argument passed to generate can be adjusted. Note it should be a multiple of the hop length (3200 for the original acoustic tokenizer).Pythonfrom transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
 tokenizer_chunk_size = 64000    # default is 1440000 (60s @ 24kHz)
 model_id = "microsoft/VibeVoice-ASR-HF"
 audio = [
-    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
-    "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav"
+    "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
+    "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav)"
 ]
 prompts = ["About VibeVoice", None]
 
@@ -280,13 +232,7 @@ output_ids = model.generate(**inputs, tokenizer_chunk_size=tokenizer_chunk_size)
 generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
 transcription = processor.decode(generated_ids, return_format="transcription_only")
 print(transcription)
-```
-
-### Chat template
-
-VibeVoice ASR also accepts chat template inputs (`apply_transcription_request` is actually a wrapper for `apply_chat_template` for convenience):
-```python
-from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
+Chat templateVibeVoice ASR also accepts chat template inputs (apply_transcription_request is actually a wrapper for apply_chat_template for convenience):Pythonfrom transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
 model_id = "microsoft/VibeVoice-ASR-HF"
 processor = AutoProcessor.from_pretrained(model_id)
@@ -300,7 +246,7 @@ chat_template = [
                 {"type": "text", "text": "About VibeVoice"},
                 {
                     "type": "audio",
-                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
+                    "path": "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
                 },
             ],
         }
@@ -311,7 +257,7 @@ chat_template = [
             "content": [
                 {
                     "type": "audio",
-                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
+                    "path": "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav)",
                 },
             ],
         }
@@ -328,14 +274,7 @@ output_ids = model.generate(**inputs)
 generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
 transcription = processor.decode(generated_ids, return_format="transcription_only")
 print(transcription)
-```
-
-### Training
-
-VibeVoice ASR can be trained with the loss outputted by the model.
-
-```python
-from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
+TrainingVibeVoice ASR can be trained with the loss outputted by the model.Pythonfrom transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
 model_id = "microsoft/VibeVoice-ASR-HF"
 processor = AutoProcessor.from_pretrained(model_id)
@@ -352,7 +291,7 @@ chat_template = [
                 {"type": "text", "text": "VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker, conversational audio."},
                 {
                     "type": "audio",
-                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
+                    "path": "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
                 },
             ],
         }
@@ -364,7 +303,7 @@ chat_template = [
                 {"type": "text", "text": "Hello everyone and welcome to the VibeVoice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me. Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings. Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible. Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it."},
                 {
                     "type": "audio",
-                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
+                    "path": "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav)",
                 },
             ],
         }
@@ -380,13 +319,7 @@ inputs = processor.apply_chat_template(
 loss = model(**inputs).loss
 print("Loss:", loss.item())
 loss.backward()
-```
-
-### Torch compile
-
-The model can be compiled for faster inference/training.
-```python
-import time
+Torch compileThe model can be compiled for faster inference/training.Pythonimport time
 import torch
 from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
 
@@ -411,7 +344,7 @@ chat_template = [
                 },
                 {
                     "type": "audio",
-                    "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
+                    "path": "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
                 },
             ],
         }
@@ -464,14 +397,7 @@ print(f"Average time with compile: {compile_time:.4f}s")
 
 speedup = no_compile_time / compile_time
 print(f"\nSpeedup: {speedup:.2f}x")
-```
-
-### Pipeline usage
-
-The model can be used as a pipeline, but you will have to define your own methods for parsing the raw output.
-
-```python
-from transformers import pipeline
+Pipeline usageThe model can be used as a pipeline, but you will have to define your own methods for parsing the raw output.Pythonfrom transformers import pipeline
 
 model_id = "microsoft/VibeVoice-ASR-HF"
 pipe = pipeline("any-to-any", model=model_id, device_map="auto")
@@ -482,7 +408,7 @@ chat_template = [
             {"type": "text", "text": "About VibeVoice"},
             {
                 "type": "audio",
-                "path": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
+                "path": "[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)",
             },
         ],
     }
@@ -498,47 +424,6 @@ print(outputs)
 ============================================================
 RAW PIPELINE OUTPUT
 ============================================================
-[{'input_text': [{'role': 'user', 'content': [{'type': 'text', 'text': 'About VibeVoice'}, {'type': 'audio', 'path': 'https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav'}]}], 'generated_text': 'assistant\n[{"Start":0.0,"End":7.56,"Speaker":0,"Content":"VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker conversational audio."}]\n'}]
+[{'input_text': [{'role': 'user', 'content': [{'type': 'text', 'text': 'About VibeVoice'}, {'type': 'audio', 'path': '[https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav](https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav)'}]}], 'generated_text': 'assistant\n[{"Start":0.0,"End":7.56,"Speaker":0,"Content":"VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker conversational audio."}]\n'}]
 """
-```
-
-## Evaluation
-
-Below are results from the [technical report](https://arxiv.org/pdf/2601.18184).
-
-<p align="center">
-  <img src="figures/DER.jpg" alt="DER" width="70%">
-  <img src="figures/cpWER.jpg" alt="cpWER" width="70%">
-  <img src="figures/tcpWER.jpg" alt="tcpWER" width="70%">
-</p>
-
-
-### Open ASR Leaderboard
-
-On the [Open ASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard), the following results were obtained:
-
-| Dataset                | WER (%)  |
-| ---------------------- | -------- |
-| ami_test               | 17.20    |
-| earnings22_test        | 13.17    |
-| gigaspeech_test        | 9.67     |
-| librispeech_test.clean | 2.20     |
-| librispeech_test.other | 5.51     |
-| spgispeech_test        | 3.80     |
-| tedlium_test           | 2.57     |
-| voxpopuli_test         | 8.01     |
-| **Average**            | **7.77** |
-| **RTFx**               | **51.80** |
-
-## Language Distribution
-<p align="center">
-  <img src="figures/language_distribution_horizontal.png" alt="Language Distribution" width="80%">
-</p>
-
-
-## License
-This project is licensed under the MIT License.
-
-## Contact
-This project was conducted by members of Microsoft Research. We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at VibeVoice@microsoft.com.
-If the team receives reports of undesired behavior or identifies issues independently, we will update this repository with appropriate mitigations.
+EvaluationBelow are results from the technical report.<p align="center"><img src="figures/DER.jpg" alt="DER" width="70%"><img src="figures/cpWER.jpg" alt="cpWER" width="70%"><img src="figures/tcpWER.jpg" alt="tcpWER" width="70%"></p>Open ASR LeaderboardOn the Open ASR leaderboard, the following results were obtained:DatasetWER (%)ami_test17.20earnings22_test13.17gigaspeech_test9.67librispeech_test.clean2.20librispeech_test.other5.51spgispeech_test3.80tedlium_test2.57voxpopuli_test8.01Average7.77RTFx51.80Language Distribution<p align="center"><img src="figures/language_distribution_horizontal.png" alt="Language Distribution" width="80%"></p>LicenseThis project is licensed under the MIT License.ContactThis project was conducted by members of Microsoft Research. We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at VibeVoice@microsoft.com.If the team receives reports of undesired behavior or identifies issues independently, we will update this repository with appropriate mitigations.