Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Multimodal templates","local":"multimodal-templates","sections":[{"title":"ImageTextToTextPipeline","local":"imagetexttotextpipeline","sections":[],"depth":2},{"title":"Image inputs","local":"image-inputs","sections":[],"depth":2},{"title":"Video inputs","local":"video-inputs","sections":[],"depth":2},{"title":"Template configuration","local":"template-configuration","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_36839/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/scheduler.01eeda35.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/singletons.177df05e.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.4862150a.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/paths.517376d1.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.6dd51b66.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/0.8897c14d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/11.d5d8d591.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/Tip.de9bae2b.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/CodeBlock.864da1b0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/EditOnGithub.7faefd25.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/HfOption.f7f04550.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/stores.318eade7.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Multimodal templates","local":"multimodal-templates","sections":[{"title":"ImageTextToTextPipeline","local":"imagetexttotextpipeline","sections":[],"depth":2},{"title":"Image inputs","local":"image-inputs","sections":[],"depth":2},{"title":"Video inputs","local":"video-inputs","sections":[],"depth":2},{"title":"Template configuration","local":"template-configuration","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="multimodal-templates" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#multimodal-templates"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Multimodal templates</span></h1> <p data-svelte-h="svelte-1agojlz">Multimodal model chat templates expect a similar <a href="./chat_templating">template</a> as text-only models. It needs <code>messages</code> that includes a dictionary of the <code>role</code> and <code>content</code>.</p> <p data-svelte-h="svelte-kacsd5">Multimodal templates are included in the <a href="./processors">Processor</a> class and requires an additional <code>type</code> key for specifying whether the included content is an image, video, or text.</p> <p data-svelte-h="svelte-gm7wkx">This guide will show you how to format chat templates for multimodal models as well as some best practices for configuring the template</p> <h2 class="relative group"><a id="imagetexttotextpipeline" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#imagetexttotextpipeline"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>ImageTextToTextPipeline</span></h2> <p data-svelte-h="svelte-3ommlp"><a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.ImageTextToTextPipeline">ImageTextToTextPipeline</a> is a high-level image and text generation class with a “chat mode”. Chat mode is enabled when a conversational model is detected and the chat prompt is <a href="./llm_tutorial#wrong-prompt-format">properly formatted</a>.</p> <p data-svelte-h="svelte-1rwc0gb">Start by building a chat history with the following two roles.</p> <ul data-svelte-h="svelte-dldyqw"><li><code>system</code> describes how the model should behave and respond when you’re chatting with it. This role isn’t supported by all chat models.</li> <li><code>user</code> is where you enter your first message to the model.</li></ul> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->messages = [ | |
| { | |
| <span class="hljs-string">"role"</span>: <span class="hljs-string">"system"</span>, | |
| <span class="hljs-string">"content"</span>: [{<span class="hljs-string">"type"</span>: <span class="hljs-string">"text"</span>, <span class="hljs-string">"text"</span>: <span class="hljs-string">"You are a friendly chatbot who always responds in the style of a pirate"</span>}], | |
| }, | |
| { | |
| <span class="hljs-string">"role"</span>: <span class="hljs-string">"user"</span>, | |
| <span class="hljs-string">"content"</span>: [ | |
| {<span class="hljs-string">"type"</span>: <span class="hljs-string">"image"</span>, <span class="hljs-string">"url"</span>: <span class="hljs-string">"http://images.cocodataset.org/val2017/000000039769.jpg"</span>}, | |
| {<span class="hljs-string">"type"</span>: <span class="hljs-string">"text"</span>, <span class="hljs-string">"text"</span>: <span class="hljs-string">"What are these?"</span>}, | |
| ], | |
| }, | |
| ]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-feobkr">Create a <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.ImageTextToTextPipeline">ImageTextToTextPipeline</a> and pass the chat to it. For large models, setting <a href="./models#big-model-inference">device_map=“auto”</a> helps load the model quicker and automatically places it on the fastest device available. Changing the data type to <a href="./models#model-data-type">torch.bfloat16</a> also helps save memory.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1id4m6u">The <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.ImageTextToTextPipeline">ImageTextToTextPipeline</a> accepts chats in the OpenAI format to make inference easier and more accessible.</p></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| pipeline = pipeline(<span class="hljs-string">"image-text-to-text"</span>, model=<span class="hljs-string">"llava-hf/llava-onevision-qwen2-0.5b-ov-hf"</span>, device=<span class="hljs-string">"cuda"</span>, torch_dtype=torch.float16) | |
| pipeline(text=messages, max_new_tokens=<span class="hljs-number">50</span>, return_full_text=<span class="hljs-literal">False</span>) | |
| [{<span class="hljs-string">'input_text'</span>: [{<span class="hljs-string">'role'</span>: <span class="hljs-string">'system'</span>, | |
| <span class="hljs-string">'content'</span>: [{<span class="hljs-string">'type'</span>: <span class="hljs-string">'text'</span>, | |
| <span class="hljs-string">'text'</span>: <span class="hljs-string">'You are a friendly chatbot who always responds in the style of a pirate'</span>}]}, | |
| {<span class="hljs-string">'role'</span>: <span class="hljs-string">'user'</span>, | |
| <span class="hljs-string">'content'</span>: [{<span class="hljs-string">'type'</span>: <span class="hljs-string">'image'</span>, | |
| <span class="hljs-string">'url'</span>: <span class="hljs-string">'http://images.cocodataset.org/val2017/000000039769.jpg'</span>}, | |
| {<span class="hljs-string">'type'</span>: <span class="hljs-string">'text'</span>, <span class="hljs-string">'text'</span>: <span class="hljs-string">'What are these?'</span>}]}], | |
| <span class="hljs-string">'generated_text'</span>: <span class="hljs-string">'The image shows two cats lying on a pink surface, which appears to be a cushion or a soft blanket. The cat on the left has a striped coat, typical of tabby cats, and is lying on its side with its head resting on the'</span>}]<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="image-inputs" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#image-inputs"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Image inputs</span></h2> <p data-svelte-h="svelte-bjpyoz">For multimodal models that accept images like <a href="./model_doc/llava">LLaVA</a>, include the following in <code>content</code> as shown below.</p> <ul data-svelte-h="svelte-17i21u2"><li>The content <code>"type"</code> can be an <code>"image"</code> or <code>"text"</code>.</li> <li>For images, it can be a link to the image (<code>"url"</code>), a file path (<code>"path"</code>), or <code>"base64"</code>. Images are automatically loaded, processed, and prepared into pixel values as inputs to the model.</li></ul> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoProcessor, LlavaOnevisionForConditionalGeneration | |
| model = LlavaOnevisionForConditionalGeneration.from_pretrained(<span class="hljs-string">"llava-hf/llava-onevision-qwen2-0.5b-ov-hf"</span>) | |
| processor = AutoProcessor.from_pretrained(<span class="hljs-string">"llava-hf/llava-onevision-qwen2-0.5b-ov-hf"</span>) | |
| messages = [ | |
| { | |
| <span class="hljs-string">"role"</span>: <span class="hljs-string">"system"</span>, | |
| <span class="hljs-string">"content"</span>: [{<span class="hljs-string">"type"</span>: <span class="hljs-string">"text"</span>, <span class="hljs-string">"text"</span>: <span class="hljs-string">"You are a friendly chatbot who always responds in the style of a pirate"</span>}], | |
| }, | |
| { | |
| <span class="hljs-string">"role"</span>: <span class="hljs-string">"user"</span>, | |
| <span class="hljs-string">"content"</span>: [ | |
| {<span class="hljs-string">"type"</span>: <span class="hljs-string">"image"</span>, <span class="hljs-string">"url"</span>: <span class="hljs-string">"http://images.cocodataset.org/val2017/000000039769.jpg"</span>}, | |
| {<span class="hljs-string">"type"</span>: <span class="hljs-string">"text"</span>, <span class="hljs-string">"text"</span>: <span class="hljs-string">"What are these?"</span>}, | |
| ], | |
| }, | |
| ]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1f65dnd">Pass <code>messages</code> to <a href="/docs/transformers/pr_36839/en/main_classes/processors#transformers.ProcessorMixin.apply_chat_template">apply_chat_template()</a> to tokenize the input content and return the <code>input_ids</code> and <code>pixel_values</code>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->processed_chat = processor.apply_chat_template(messages, add_generation_prompt=<span class="hljs-literal">True</span>, tokenize=<span class="hljs-literal">True</span>, return_dict=<span class="hljs-literal">True</span>, return_tensors=<span class="hljs-string">"pt"</span>) | |
| <span class="hljs-built_in">print</span>(processed_chat.keys())<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-2dx3mn">These inputs are now ready to be used in <a href="/docs/transformers/pr_36839/en/main_classes/text_generation#transformers.GenerationMixin.generate">generate()</a>.</p> <h2 class="relative group"><a id="video-inputs" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#video-inputs"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Video inputs</span></h2> <p data-svelte-h="svelte-1ldrrkv">Some vision models also support video inputs. The message format is very similar to the format for <a href="#image-inputs">image inputs</a>.</p> <ul data-svelte-h="svelte-lze7lb"><li>The content <code>"type"</code> should be <code>"video"</code> to indicate the the content is a video.</li> <li>For videos, it can be a link to the video (<code>"url"</code>) or it could be a file path (<code>"path"</code>). Videos loaded from a URL can only be decoded with <a href="https://pyav.basswood-io.com/docs/stable/" rel="nofollow">PyAV</a> or <a href="https://github.com/dmlc/decord" rel="nofollow">Decord</a>.</li></ul> <div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p data-svelte-h="svelte-1ddif0x">Loading a video from <code>"url"</code> is only supported by the PyAV or Decord backends.</p></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoProcessor, LlavaOnevisionForConditionalGeneration | |
| model_id = <span class="hljs-string">"llava-hf/llava-onevision-qwen2-0.5b-ov-hf"</span> | |
| model = LlavaOnevisionForConditionalGeneration.from_pretrained(model_id) | |
| processor = AutoProcessor.from_pretrained(model_id) | |
| messages = [ | |
| { | |
| <span class="hljs-string">"role"</span>: <span class="hljs-string">"system"</span>, | |
| <span class="hljs-string">"content"</span>: [{<span class="hljs-string">"type"</span>: <span class="hljs-string">"text"</span>, <span class="hljs-string">"text"</span>: <span class="hljs-string">"You are a friendly chatbot who always responds in the style of a pirate"</span>}], | |
| }, | |
| { | |
| <span class="hljs-string">"role"</span>: <span class="hljs-string">"user"</span>, | |
| <span class="hljs-string">"content"</span>: [ | |
| {<span class="hljs-string">"type"</span>: <span class="hljs-string">"video"</span>, <span class="hljs-string">"url"</span>: <span class="hljs-string">"https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/720/Big_Buck_Bunny_720_10s_10MB.mp4"</span>}, | |
| {<span class="hljs-string">"type"</span>: <span class="hljs-string">"text"</span>, <span class="hljs-string">"text"</span>: <span class="hljs-string">"What do you see in this video?"</span>}, | |
| ], | |
| }, | |
| ]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1vjvhc2">Pass <code>messages</code> to <a href="/docs/transformers/pr_36839/en/main_classes/processors#transformers.ProcessorMixin.apply_chat_template">apply_chat_template()</a> to tokenize the input content. There are a few extra parameters to include in <a href="/docs/transformers/pr_36839/en/main_classes/processors#transformers.ProcessorMixin.apply_chat_template">apply_chat_template()</a> that controls the sampling process.</p> <p data-svelte-h="svelte-mmdb9c">The <code>video_load_backend</code> parameter refers to a specific framework to load a video. It supports <a href="https://pyav.basswood-io.com/docs/stable/" rel="nofollow">PyAV</a>, <a href="https://github.com/dmlc/decord" rel="nofollow">Decord</a>, <a href="https://github.com/opencv/opencv" rel="nofollow">OpenCV</a>, and <a href="https://pytorch.org/vision/stable/index.html" rel="nofollow">torchvision</a>.</p> <p data-svelte-h="svelte-d4t5se">The examples below uses Decord as the backend because it is a bit faster than PyAV.</p> <div class="flex space-x-2 items-center my-1.5 mr-8 h-7 !pl-0 -mx-3 md:mx-0"><div class="flex items-center border rounded-lg px-1.5 py-1 leading-none select-none text-smd border-gray-800 bg-black dark:bg-gray-700 text-white">fixed number of frames </div><div class="flex items-center border rounded-lg px-1.5 py-1 leading-none select-none text-smd text-gray-500 cursor-pointer opacity-90 hover:text-gray-700 dark:hover:text-gray-200 hover:shadow-sm">fps </div><div class="flex items-center border rounded-lg px-1.5 py-1 leading-none select-none text-smd text-gray-500 cursor-pointer opacity-90 hover:text-gray-700 dark:hover:text-gray-200 hover:shadow-sm">custom frame sampling </div><div class="flex items-center border rounded-lg px-1.5 py-1 leading-none select-none text-smd text-gray-500 cursor-pointer opacity-90 hover:text-gray-700 dark:hover:text-gray-200 hover:shadow-sm">list of image frames </div></div> <div class="language-select"><p data-svelte-h="svelte-1ilhsfw">The <code>num_frames</code> parameter controls how many frames to uniformly sample from the video. Each checkpoint has a maximum frame count it was pretrained with and exceeding this count can significantly lower generation quality. It’s important to choose a frame count that fits both the model capacity and your hardware resources. If <code>num_frames</code> isn’t specified, the entire video is loaded without any frame sampling.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->processed_chat = processor.apply_chat_template( | |
| messages, | |
| add_generation_prompt=<span class="hljs-literal">True</span>, | |
| tokenize=<span class="hljs-literal">True</span>, | |
| return_dict=<span class="hljs-literal">True</span>, | |
| return_tensors=<span class="hljs-string">"pt"</span>, | |
| num_frames=<span class="hljs-number">32</span>, | |
| video_load_backend=<span class="hljs-string">"decord"</span>, | |
| ) | |
| <span class="hljs-built_in">print</span>(processed_chat.keys())<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-2dx3mn">These inputs are now ready to be used in <a href="/docs/transformers/pr_36839/en/main_classes/text_generation#transformers.GenerationMixin.generate">generate()</a>.</p> </div> <h2 class="relative group"><a id="template-configuration" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#template-configuration"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Template configuration</span></h2> <p data-svelte-h="svelte-3krzhl">You can create a custom chat template with <a href="https://jinja.palletsprojects.com/en/3.1.x/templates/" rel="nofollow">Jinja</a> and set it with <a href="/docs/transformers/pr_36839/en/main_classes/processors#transformers.ProcessorMixin.apply_chat_template">apply_chat_template()</a>. Refer to the <a href="./chat_templating_writing">Template writing</a> guide for more details.</p> <p data-svelte-h="svelte-1gbnbdg">For example, to enable a template to handle a <em>list of content</em> from multiple modalities while still supporting plain strings for text-only inference, specify how to handle the <code>content['type']</code> if it is an image or text as shown below in the Llama 3.2 Vision Instruct <a href="https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct/blob/main/chat_template.json" rel="nofollow">template</a>.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">for</span></span> message <span class="hljs-keyword">in</span> messages %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">if</span></span> loop.index0 == 0 %}</span><span class="hljs-template-variable">{{ bos_token }}</span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">endif</span></span> %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-variable">{{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' }}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">if</span></span> message['content'] is string %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-variable">{{ message['content'] }}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">else</span></span> %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">for</span></span> content <span class="hljs-keyword">in</span> message['content'] %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">if</span></span> content['type'] == 'image' %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-variable">{{ '<|image|>' }}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">elif</span></span> content['type'] == 'text' %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-variable">{{ content['text'] }}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">endif</span></span> %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">endfor</span></span> %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">endif</span></span> %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-variable">{{ '<|eot_id|>' }}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">endfor</span></span> %}</span><span class="language-xml"> | |
| </span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">if</span></span> add_generation_prompt %}</span><span class="hljs-template-variable">{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}</span><span class="hljs-template-tag">{% <span class="hljs-name"><span class="hljs-name">endif</span></span> %}</span><!-- HTML_TAG_END --></pre></div> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/chat_templating_multimodal.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1bm5psi = { | |
| assets: "/docs/transformers/pr_36839/en", | |
| base: "/docs/transformers/pr_36839/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"), | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 11], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 36.8 kB
- Xet hash:
- 28ad6f9c028e35439b2a77e7f7b3b54d0343e7fd964cf57603e2570182839f2a
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.