Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Chat basics","local":"chat-basics","sections":[{"title":"transformers-cli","local":"transformers-cli","sections":[],"depth":2},{"title":"TextGenerationPipeline","local":"textgenerationpipeline","sections":[],"depth":2},{"title":"Performance","local":"performance","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_36839/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/scheduler.01eeda35.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/singletons.177df05e.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.4862150a.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/paths.517376d1.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.6dd51b66.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/0.8897c14d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/15.075a06f8.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/Tip.de9bae2b.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/CodeBlock.864da1b0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/EditOnGithub.7faefd25.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Chat basics","local":"chat-basics","sections":[{"title":"transformers-cli","local":"transformers-cli","sections":[],"depth":2},{"title":"TextGenerationPipeline","local":"textgenerationpipeline","sections":[],"depth":2},{"title":"Performance","local":"performance","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="chat-basics" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#chat-basics"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Chat basics</span></h1> <p data-svelte-h="svelte-jrujnx">Chat models are conversational models you can send and receive messages from. There are many chat models available to choose from, but in general, larger models tend to be better though that’s not always the case. The model size is often included in the name, like “8B” or “70B”, and it describes the number of parameters. Mixture-of-expert (MoE) models have names like “8x7B” or “141B-A35B” which means it’s a 56B and 141B parameter model. You can try quantizing larger models to reduce memory requirements, otherwise you’ll need ~2 bytes of memory per parameter.</p> <p data-svelte-h="svelte-glfytc">Check model leaderboards like <a href="https://hf.co/spaces/HuggingFaceH4/open_llm_leaderboard" rel="nofollow">OpenLLM</a> and <a href="https://chat.lmsys.org/?leaderboard" rel="nofollow">LMSys Chatbot Arena</a> to further help you identify the best chat models for your use case. Models that are specialized in certain domains (medical, legal text, non-English languages, etc.) may sometimes outperform larger general purpose models.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1owhzhh">Chat with a number of open-source models for free on <a href="https://hf.co/chat/" rel="nofollow">HuggingChat</a>!</p></div> <p data-svelte-h="svelte-1ymhiw4">This guide shows you how to quickly start chatting with Transformers from the command line, how build and format a conversation, and how to chat using the <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.TextGenerationPipeline">TextGenerationPipeline</a>.</p> <h2 class="relative group"><a id="transformers-cli" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#transformers-cli"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>transformers-cli</span></h2> <p data-svelte-h="svelte-1stsitg">Chat with a model directly from the command line as shown below. It launches an interactive session with a model. Enter <code>clear</code> to reset the conversation, <code>exit</code> to terminate the session, and <code>help</code> to display all the command options.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->transformers-cli chat --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct<!-- HTML_TAG_END --></pre></div> <div class="flex justify-center" data-svelte-h="svelte-1q6uti0"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/transformers-chat-cli.png"></div> <p data-svelte-h="svelte-1dduaqi">For a full list of options, run the command below.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->transformers-cli chat -h<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ct2dmy">The chat is implemented on top of the <a href="./model_doc/auto">AutoClass</a>, using tooling from <a href="./llm_tutorial">text generation</a> and <a href="./chat_templating">chat</a>.</p> <h2 class="relative group"><a id="textgenerationpipeline" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#textgenerationpipeline"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>TextGenerationPipeline</span></h2> <p data-svelte-h="svelte-1meafzr"><a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.TextGenerationPipeline">TextGenerationPipeline</a> is a high-level text generation class with a “chat mode”. Chat mode is enabled when a conversational model is detected and the chat prompt is <a href="./llm_tutorial#wrong-prompt-format">properly formatted</a>.</p> <p data-svelte-h="svelte-1dss5mr">To start, build a chat history with the following two roles.</p> <ul data-svelte-h="svelte-dldyqw"><li><code>system</code> describes how the model should behave and respond when you’re chatting with it. This role isn’t supported by all chat models.</li> <li><code>user</code> is where you enter your first message to the model.</li></ul> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->chat = [ | |
| {<span class="hljs-string">"role"</span>: <span class="hljs-string">"system"</span>, <span class="hljs-string">"content"</span>: <span class="hljs-string">"You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."</span>}, | |
| {<span class="hljs-string">"role"</span>: <span class="hljs-string">"user"</span>, <span class="hljs-string">"content"</span>: <span class="hljs-string">"Hey, can you tell me any fun things to do in New York?"</span>} | |
| ]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-16yewtn">Create the <a href="/docs/transformers/pr_36839/en/main_classes/pipelines#transformers.TextGenerationPipeline">TextGenerationPipeline</a> and pass <code>chat</code> to it. For large models, setting <a href="./models#big-model-inference">device_map=“auto”</a> helps load the model quicker and automatically places it on the fastest device available. Changing the data type to <a href="./models#model-data-type">torch.bfloat16</a> also helps save memory.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch | |
| <span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline | |
| pipeline = pipeline(task=<span class="hljs-string">"text-generation"</span>, model=<span class="hljs-string">"meta-llama/Meta-Llama-3-8B-Instruct"</span>, torch_dtype=torch.bfloat16, device_map=<span class="hljs-string">"auto"</span>) | |
| response = pipeline(chat, max_new_tokens=<span class="hljs-number">512</span>) | |
| <span class="hljs-built_in">print</span>(response[<span class="hljs-number">0</span>][<span class="hljs-string">"generated_text"</span>][-<span class="hljs-number">1</span>][<span class="hljs-string">"content"</span>])<!-- HTML_TAG_END --></pre></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->(sigh) Oh boy, you're asking me for advice? You're gonna need a map, pal! Alright, | |
| alright, I'll give you the lowdown. But don't say I didn't warn you, I'm a robot, not a tour guide! | |
| So, you wanna know what's fun to do in the Big Apple? Well, let me tell you, there's a million | |
| things to do, but I'll give you the highlights. First off, you gotta see the sights: the Statue of | |
| Liberty, Central Park, Times Square... you know, the usual tourist traps. But if you're lookin' for | |
| something a little more... unusual, I'd recommend checkin' out the Museum of Modern Art. It's got | |
| some wild stuff, like that Warhol guy's soup cans and all that jazz. | |
| And if you're feelin' adventurous, take a walk across the Brooklyn Bridge. Just watch out for | |
| those pesky pigeons, they're like little feathered thieves! (laughs) Get it? Thieves? Ah, never mind. | |
| Now, if you're lookin' for some serious fun, hit up the comedy clubs in Greenwich Village. You might | |
| even catch a glimpse of some up-and-coming comedians... or a bunch of wannabes tryin' to make it big. (winks) | |
| And finally, if you're feelin' like a real New Yorker, grab a slice of pizza from one of the many amazing | |
| pizzerias around the city. Just don't try to order a "robot-sized" slice, trust me, it won't end well. (laughs) | |
| So, there you have it, pal! That's my expert advice on what to do in New York. Now, if you'll | |
| excuse me, I've got some oil changes to attend to. (winks)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-hacrsz">Use the <code>append</code> method on <code>chat</code> to respond to the models message.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->chat = response[<span class="hljs-number">0</span>][<span class="hljs-string">"generated_text"</span>] | |
| chat.append( | |
| {<span class="hljs-string">"role"</span>: <span class="hljs-string">"user"</span>, <span class="hljs-string">"content"</span>: <span class="hljs-string">"Wait, what's so wild about soup cans?"</span>} | |
| ) | |
| response = pipeline(chat, max_new_tokens=<span class="hljs-number">512</span>) | |
| <span class="hljs-built_in">print</span>(response[<span class="hljs-number">0</span>][<span class="hljs-string">"generated_text"</span>][-<span class="hljs-number">1</span>][<span class="hljs-string">"content"</span>])<!-- HTML_TAG_END --></pre></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->(laughs) Oh, you're killin' me, pal! You don't get it, do you? Warhol's soup cans are like, art, man! | |
| It's like, he took something totally mundane, like a can of soup, and turned it into a masterpiece. It's | |
| like, "Hey, look at me, I'm a can of soup, but I'm also a work of art!" | |
| (sarcastically) Oh, yeah, real original, Andy. | |
| But, you know, back in the '60s, it was like, a big deal. People were all about challenging the | |
| status quo, and Warhol was like, the king of that. He took the ordinary and made it extraordinary. | |
| And, let me tell you, it was like, a real game-changer. I mean, who would've thought that a can of soup could be art? (laughs) | |
| But, hey, you're not alone, pal. I mean, I'm a robot, and even I don't get it. (winks) | |
| But, hey, that's what makes art, art, right? (laughs)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="performance" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#performance"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Performance</span></h2> <p data-svelte-h="svelte-icjqz6">Transformers load models in full precision by default, and for a 8B model, this requires ~32GB of memory! Reduce memory usage by loading a model in half-precision or bfloat16 (only uses ~2 bytes per parameter). You can even quantize the model to a lower precision like 8-bit or 4-bit with <a href="https://hf.co/docs/bitsandbytes/index" rel="nofollow">bitsandbytes</a>.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-inxikh">Refer to the <a href="./quantization/overview">Quantization</a> docs for more information about the different quantization backends available.</p></div> <p data-svelte-h="svelte-1l05y34">Create a <a href="/docs/transformers/pr_36839/en/main_classes/quantization#transformers.BitsAndBytesConfig">BitsAndBytesConfig</a> with your desired quantization settings and pass it to the pipelines <code>model_kwargs</code> parameter. The example below quantizes a model to 8-bits.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> pipeline, BitsAndBytesConfig | |
| quantization_config = BitsAndBytesConfig(load_in_8bit=<span class="hljs-literal">True</span>) | |
| pipeline = pipeline(task=<span class="hljs-string">"text-generation"</span>, model=<span class="hljs-string">"meta-llama/Meta-Llama-3-8B-Instruct"</span>, device_map=<span class="hljs-string">"auto"</span>, model_kwargs={<span class="hljs-string">"quantization_config"</span>: quantization_config})<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-95wc3r">In general, larger models are slower in addition to requiring more memory because text generation is bottlenecked by <strong>memory bandwidth</strong> instead of compute power. Each active parameter must be read from memory for every generated token. For a 16GB model, 16GB must be read from memory for every generated token.</p> <p data-svelte-h="svelte-debr7a">The number of generated tokens/sec is proportional to the total memory bandwidth of the system divided by the model size. Depending on your hardware, total memory bandwidth can vary. Refer to the table below for approximate generation speeds for different hardware types.</p> <table data-svelte-h="svelte-c36prw"><thead><tr><th>Hardware</th> <th>Memory bandwidth</th></tr></thead> <tbody><tr><td>consumer CPU</td> <td>20-100GB/sec</td></tr> <tr><td>specialized CPU (Intel Xeon, AMD Threadripper/Epyc, Apple silicon)</td> <td>200-900GB/sec</td></tr> <tr><td>data center GPU (NVIDIA A100/H100)</td> <td>2-3TB/sec</td></tr></tbody></table> <p data-svelte-h="svelte-pj190v">The easiest solution for improving generation speed is to either quantize a model or use hardware with higher memory bandwidth.</p> <p data-svelte-h="svelte-1sesjc6">You can also try techniques like <a href="./generation_strategies#speculative-decoding">speculative decoding</a>, where a smaller model generates candidate tokens that are verified by the larger model. If the candidate tokens are correct, the larger model can generate more than one token per <code>forward</code> pass. This significantly alleviates the bandwidth bottleneck and improves generation speed.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-oacgq6">Parameters may not be active for every generated token in MoE models such as <a href="./model_doc/mixtral">Mixtral</a>, <a href="./model_doc/qwen2_moe.md">Qwen2MoE</a>, and <a href="./model_doc/dbrx">DBRX</a>. As a result, MoE models generally have much lower memory bandwidth requirements and can be faster than a regular LLM of the same size. However, techniques like speculative decoding are ineffective with MoE models because parameters become activated with each new speculated token.</p></div> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/conversations.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1bm5psi = { | |
| assets: "/docs/transformers/pr_36839/en", | |
| base: "/docs/transformers/pr_36839/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"), | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 15], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 30.6 kB
- Xet hash:
- d8aef171d6ae22c0fd04bdfd3ced4fe3eb7d60a1cdc4af4c48c4645ad55fb5e4
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.