| # Audio and speech | |
| The OpenAI API provides a range of audio capabilities. If you know what you want to build, find your use case below to get started. If you're not sure where to start, read this page as an overview. | |
| ## Build with audio | |
| <div className="w-full max-w-full overflow-hidden"> | |
| </div> | |
| ## A tour of audio use cases | |
| LLMs can process audio by using sound as input, creating sound as output, or both. OpenAI has several API endpoints that help you build audio applications or voice agents. | |
| ### Voice agents | |
| Voice agents understand audio to handle tasks and respond back in natural language. There are two main ways to approach voice agents: either with speech-to-speech models and the [Realtime API](https://developers.openai.com/api/docs/guides/realtime), or by chaining together a speech-to-text model, a text language model to process the request, and a text-to-speech model to respond. Speech-to-speech is lower latency and more natural, but chaining together a voice agent is a reliable way to extend a text-based agent into a voice agent. If you are already using the [Agents SDK](https://developers.openai.com/api/docs/guides/agents), you can [extend your existing agents with voice capabilities](https://openai.github.io/openai-agents-python/voice/quickstart/) using the chained approach. | |
| ### Streaming audio | |
| Process audio in real time to build voice agents and other low-latency applications, including transcription use cases. You can stream audio in and out of a model with the [Realtime API](https://developers.openai.com/api/docs/guides/realtime). Our advanced speech models provide automatic speech recognition for improved accuracy, low-latency interactions, and multilingual support. | |
| ### Text to speech | |
| For turning text into speech, use the [Audio API](https://developers.openai.com/api/docs/api-reference/audio/) `audio/speech` endpoint. Models compatible with this endpoint are `gpt-4o-mini-tts`, `tts-1`, and `tts-1-hd`. With `gpt-4o-mini-tts`, you can ask the model to speak a certain way or with a certain tone of voice. | |
| ### Speech to text | |
| For speech to text, use the [Audio API](https://developers.openai.com/api/docs/api-reference/audio/) `audio/transcriptions` endpoint. Models compatible with this endpoint are `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`, `whisper-1`, and `gpt-4o-transcribe-diarize`. `gpt-4o-transcribe-diarize` adds speaker labels and timestamps for HTTP requests and is intended for non-latency-sensitive workloads, while the other models focus on transcription only. With streaming, you can continuously pass in audio and get a continuous stream of text back. | |
| ## Choosing the right API | |
| There are multiple APIs for transcribing or generating audio: | |
| | API | Supported modalities | Streaming support | | |
| | ---------------------------------------------------- | --------------------------------- | ------------------------------------------------ | | |
| | [Realtime API](https://developers.openai.com/api/docs/api-reference/realtime) | Audio and text inputs and outputs | Audio streaming in, audio and text streaming out | | |
| | [Chat Completions API](https://developers.openai.com/api/docs/api-reference/chat) | Audio and text inputs and outputs | Audio and text streaming out | | |
| | [Transcription API](https://developers.openai.com/api/docs/api-reference/audio) | Audio inputs | Text streaming out | | |
| | [Speech API](https://developers.openai.com/api/docs/api-reference/audio) | Text inputs and audio outputs | Audio streaming out | | |
| ### General use APIs vs. specialized APIs | |
| The main distinction is general use APIs vs. specialized APIs. With the Realtime and Chat Completions APIs, you can use our latest models' native audio understanding and generation capabilities and combine them with other features like function calling. These APIs can be used for a wide range of use cases, and you can select the model you want to use. | |
| On the other hand, the Transcription, Translation and Speech APIs are specialized to work with specific models and only meant for one purpose. | |
| ### Talking with a model vs. controlling the script | |
| Another way to select the right API is asking yourself how much control you need. To design conversational interactions, where the model thinks and responds in speech, use the Realtime or Chat Completions API, depending if you need low-latency or not. | |
| You won't know exactly what the model will say ahead of time, as it will generate audio responses directly, but the conversation will feel natural. | |
| For more control and predictability, you can use the Speech-to-text / LLM / Text-to-speech pattern, so you know exactly what the model will say and can control the response. Please note that with this method, there will be added latency. | |
| This is what the Audio APIs are for: pair an LLM with the `audio/transcriptions` and `audio/speech` endpoints to take spoken user input, process and generate a text response, and then convert that to speech that the user can hear. | |
| ### Recommendations | |
| - If you need [real-time interactions](https://developers.openai.com/api/docs/guides/realtime-conversations) or [transcription](https://developers.openai.com/api/docs/guides/realtime-transcription), use the Realtime API. | |
| - If realtime is not a requirement but you're looking to build a [voice agent](https://developers.openai.com/api/docs/guides/voice-agents) or an audio-based application that requires features such as [function calling](https://developers.openai.com/api/docs/guides/function-calling), use the Chat Completions API. | |
| - For use cases with one specific purpose, use the Transcription, Translation, or Speech APIs. | |
| ## Add audio to your existing application | |
| Models such as `gpt-realtime` and `gpt-audio` are natively multimodal, meaning they can understand and generate multiple modalities as input and output. | |
| If you already have a text-based LLM application with the [Chat Completions endpoint](https://developers.openai.com/api/docs/api-reference/chat/), you may want to add audio capabilities. For example, if your chat application supports text input, you can add audio input and output—just include `audio` in the `modalities` array and use an audio model, like `gpt-audio`. | |
| Audio is not yet supported in the [Responses | |
| API](https://developers.openai.com/api/docs/api-reference/chat/completions/responses). | |
| <div data-content-switcher-pane data-value="audio-out"> | |
| <div class="hidden">Audio output from model</div> | |
| Create a human-like audio response to a prompt | |
| ```javascript | |
| import { writeFileSync } from "node:fs"; | |
| import OpenAI from "openai"; | |
| const openai = new OpenAI(); | |
| // Generate an audio response to the given prompt | |
| const response = await openai.chat.completions.create({ | |
| model: "gpt-audio", | |
| modalities: ["text", "audio"], | |
| audio: { voice: "alloy", format: "wav" }, | |
| messages: [ | |
| { | |
| role: "user", | |
| content: "Is a golden retriever a good family dog?" | |
| } | |
| ], | |
| store: true, | |
| }); | |
| // Inspect returned data | |
| console.log(response.choices[0]); | |
| // Write audio data to a file | |
| writeFileSync( | |
| "dog.wav", | |
| Buffer.from(response.choices[0].message.audio.data, 'base64'), | |
| { encoding: "utf-8" } | |
| ); | |
| ``` | |
| ```python | |
| import base64 | |
| from openai import OpenAI | |
| client = OpenAI() | |
| completion = client.chat.completions.create( | |
| model="gpt-audio", | |
| modalities=["text", "audio"], | |
| audio={"voice": "alloy", "format": "wav"}, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": "Is a golden retriever a good family dog?" | |
| } | |
| ] | |
| ) | |
| print(completion.choices[0]) | |
| wav_bytes = base64.b64decode(completion.choices[0].message.audio.data) | |
| with open("dog.wav", "wb") as f: | |
| f.write(wav_bytes) | |
| ``` | |
| ```bash | |
| curl "https://api.openai.com/v1/chat/completions" \\ | |
| -H "Content-Type: application/json" \\ | |
| -H "Authorization: Bearer $OPENAI_API_KEY" \\ | |
| -d '{ | |
| "model": "gpt-audio", | |
| "modalities": ["text", "audio"], | |
| "audio": { "voice": "alloy", "format": "wav" }, | |
| "messages": [ | |
| { | |
| "role": "user", | |
| "content": "Is a golden retriever a good family dog?" | |
| } | |
| ] | |
| }' | |
| ``` | |
| </div> | |
| <div data-content-switcher-pane data-value="audio-in" hidden> | |
| <div class="hidden">Audio input to model</div> | |
| Use audio inputs for prompting a model | |
| ```javascript | |
| import OpenAI from "openai"; | |
| const openai = new OpenAI(); | |
| // Fetch an audio file and convert it to a base64 string | |
| const url = "https://cdn.openai.com/API/docs/audio/alloy.wav"; | |
| const audioResponse = await fetch(url); | |
| const buffer = await audioResponse.arrayBuffer(); | |
| const base64str = Buffer.from(buffer).toString("base64"); | |
| const response = await openai.chat.completions.create({ | |
| model: "gpt-audio", | |
| modalities: ["text", "audio"], | |
| audio: { voice: "alloy", format: "wav" }, | |
| messages: [ | |
| { | |
| role: "user", | |
| content: [ | |
| { type: "text", text: "What is in this recording?" }, | |
| { type: "input_audio", input_audio: { data: base64str, format: "wav" }} | |
| ] | |
| } | |
| ], | |
| store: true, | |
| }); | |
| console.log(response.choices[0]); | |
| ``` | |
| ```python | |
| import base64 | |
| import requests | |
| from openai import OpenAI | |
| client = OpenAI() | |
| # Fetch the audio file and convert it to a base64 encoded string | |
| url = "https://cdn.openai.com/API/docs/audio/alloy.wav" | |
| response = requests.get(url) | |
| response.raise_for_status() | |
| wav_data = response.content | |
| encoded_string = base64.b64encode(wav_data).decode('utf-8') | |
| completion = client.chat.completions.create( | |
| model="gpt-audio", | |
| modalities=["text", "audio"], | |
| audio={"voice": "alloy", "format": "wav"}, | |
| messages=[ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { | |
| "type": "text", | |
| "text": "What is in this recording?" | |
| }, | |
| { | |
| "type": "input_audio", | |
| "input_audio": { | |
| "data": encoded_string, | |
| "format": "wav" | |
| } | |
| } | |
| ] | |
| }, | |
| ] | |
| ) | |
| print(completion.choices[0].message) | |
| ``` | |
| ```bash | |
| curl "https://api.openai.com/v1/chat/completions" \\ | |
| -H "Content-Type: application/json" \\ | |
| -H "Authorization: Bearer $OPENAI_API_KEY" \\ | |
| -d '{ | |
| "model": "gpt-audio", | |
| "modalities": ["text", "audio"], | |
| "audio": { "voice": "alloy", "format": "wav" }, | |
| "messages": [ | |
| { | |
| "role": "user", | |
| "content": [ | |
| { "type": "text", "text": "What is in this recording?" }, | |
| { | |
| "type": "input_audio", | |
| "input_audio": { | |
| "data": "<base64 bytes here>", | |
| "format": "wav" | |
| } | |
| } | |
| ] | |
| } | |
| ] | |
| }' | |
| ``` | |
| </div> |