File size: 6,976 Bytes
a9bd396 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 | <!--Copyright 2025 The HuggingFace Team. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->
# Chat message patterns
Chat models expect conversations as a list of dictionaries. Each dictionary uses `role` and `content` keys. The `content` key holds the user message passed to the model. Large language models accept text and tools and multimodal models combine text with images, videos, and audio.
Transformers uses a unified format where each modality type is specified explicitly, making it straightforward to mix and match inputs in a single message.
This guide covers message formatting patterns for each modality, tools, batch inference, and multi-turn conversations.
## Text
Text is the most basic content type. It's the foundation for all other patterns. Pass your message to `"content"` as a string.
```py
message = [
{
"role": "user",
"content": "Explain the French Bread Law."
}
]
```
You could also use the explicit `"type": "text"` format to keep your code consistent when you add images, videos, or audio later.
```py
message = [
{
"role": "user",
"content": [{"type": "text", "text": "Explain the French Bread Law."}]
}
]
```
## Tools
[Tools](./chat_extras) are functions a chat model can call, like getting real-time weather data, instead of generating it on its own.
The `assistant` role handles the tool request. Set `"type": "function"` in the `"tool_calls"` key and provide your tool to the `"function"` key. Append the assistant's tool request to your message.
```py
weather = {"name": "get_current_temperature", "arguments": {"location": "Paris, France", "unit": "celsius"}}
message.append(
{
"role": "assistant",
"tool_calls": [{"type": "function", "function": weather}]
}
)
```
The `tool` role handles the result. Append it in `"content"`. This value should always be a string.
```py
message.append({"role": "tool", "content": "22"})
```
## Multimodal
Multimodal models extend this format to handle images, videos, and audio. Each input specifies its `"type"` and provides the media with `"url"` or `"path"`.
### Image
Set `"type": "image"` and use `"url"` for links or `"path"` for local files.
```py
message = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "text", "text": "What pastry is shown in the image?"}
]
}
]
```
### Video
Set `"type": "video"` and use `"url"` for links or `"path"` for local files.
```py
message = [
{
"role": "user",
"content": [
{"type": "video", "url": "https://static01.nyt.com/images/2019/10/01/dining/01Sourdough-GIF-1/01Sourdough-GIF-1-superJumbo.gif"},
{"type": "text", "text": "What is shown in this video?"}
]
}
]
```
### Audio
Set `"type": "audio"` and use `"url"` for links or `"path"` for local files.
```py
message = [
{
"role": "user",
"content": [
{"type": "audio", "url": "https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac"},
{"type": "text", "text": "Transcribe the speech."}
]
}
]
```
### Mixed multiple
The `content` list accepts any combination of types. The model processes all inputs together, enabling comparisons or cross-modal reasoning.
```py
message = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "video", "url": "https://static01.nyt.com/images/2019/10/01/dining/01Sourdough-GIF-1/01Sourdough-GIF-1-superJumbo.gif"},
{"type": "text", "text": "What does the image and video share in common?"},
],
},
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "image", "url": "https://assets.bonappetit.com/photos/57e191f49f19b4610e6b7693/master/w_1600%2Cc_limit/undefined"},
{"type": "text", "text": "What type of pastries are these?"},
],
}
]
```
## Batched
Batched inference processes multiple conversations in a single forward pass to improve throughput and efficiency. Wrap each conversation in its own list, then pass them together as a list of lists.
```py
messages = [
[
{"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "text", "text": "What type of pastry is this?"}
]
},
],
[
{"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57e191f49f19b4610e6b7693/master/w_1600%2Cc_limit/undefined"},
{"type": "text", "text": "What type of pastry is this?"}
]
},
],
]
```
## Multi-turn
Conversations span multiple exchanges, alternating between `"user"` and `"assistant"` roles. Each turn adds a new message to the list, giving the model access to the full conversation history. This context helps the model generate more appropriate responses.
```py
message = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://assets.bonappetit.com/photos/57ad4ebc53e63daf11a4ddc7/master/w_1280,c_limit/kouign-amann.jpg"},
{"type": "text", "text": "What pastry is shown in the image?"}
]
},
{
"role": "assistant",
"content": [{"type": "text", "text": "This is kouign amann, a laminated dough pastry (i.e., dough folded with layers of butter) that also incorporates sugar between layers so that during baking the sugar caramelizes."}]
},
{
"role": "user",
"content": [
{"type": "image", "url": "https://static01.nyt.com/images/2023/07/21/multimedia/21baguettesrex-hbkc/21baguettesrex-hbkc-videoSixteenByNineJumbo1600.jpg"},
{"type": "text", "text": "Compare it to this image now."}
]
}
]
```
|