Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Transformers Agents","local":"transformers-agents","sections":[{"title":"Quickstart","local":"quickstart","sections":[{"title":"Single execution (run)","local":"single-execution-run","sections":[],"depth":3},{"title":"Chat-based execution (chat)","local":"chat-based-execution-chat","sections":[],"depth":3},{"title":"⚠️ Remote execution","local":"-remote-execution","sections":[],"depth":3},{"title":"What’s happening here? What are tools, and what are agents?","local":"whats-happening-here-what-are-tools-and-what-are-agents","sections":[{"title":"Agents","local":"agents","sections":[],"depth":4},{"title":"Tools","local":"tools","sections":[],"depth":4},{"title":"Code-execution?!","local":"code-execution","sections":[],"depth":4}],"depth":3},{"title":"A curated set of tools","local":"a-curated-set-of-tools","sections":[],"depth":3},{"title":"Custom tools","local":"custom-tools","sections":[],"depth":3},{"title":"Code generation","local":"code-generation","sections":[],"depth":3}],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/v4.40.2/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/entry/start.617cb5b4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/scheduler.9bc65507.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/singletons.d2baa3f3.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/index.3b203c72.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/paths.5efb232c.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/entry/app.6f576df8.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/index.707bf1b6.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/nodes/0.90e84d27.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/nodes/396.b21e380c.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/Tip.c2ecdbf4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/CodeBlock.54a9f38d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/v4.40.2/en/_app/immutable/chunks/EditOnGithub.922df6ba.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Transformers Agents","local":"transformers-agents","sections":[{"title":"Quickstart","local":"quickstart","sections":[{"title":"Single execution (run)","local":"single-execution-run","sections":[],"depth":3},{"title":"Chat-based execution (chat)","local":"chat-based-execution-chat","sections":[],"depth":3},{"title":"⚠️ Remote execution","local":"-remote-execution","sections":[],"depth":3},{"title":"What’s happening here? What are tools, and what are agents?","local":"whats-happening-here-what-are-tools-and-what-are-agents","sections":[{"title":"Agents","local":"agents","sections":[],"depth":4},{"title":"Tools","local":"tools","sections":[],"depth":4},{"title":"Code-execution?!","local":"code-execution","sections":[],"depth":4}],"depth":3},{"title":"A curated set of tools","local":"a-curated-set-of-tools","sections":[],"depth":3},{"title":"Custom tools","local":"custom-tools","sections":[],"depth":3},{"title":"Code generation","local":"code-generation","sections":[],"depth":3}],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="transformers-agents" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#transformers-agents"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Transformers Agents</span></h1> <div class="course-tip course-tip-orange bg-gradient-to-br dark:bg-gradient-to-r before:border-orange-500 dark:before:border-orange-800 from-orange-50 dark:from-gray-900 to-white dark:to-gray-950 border border-orange-50 text-orange-700 dark:text-gray-400"><p data-svelte-h="svelte-1fgfei3">Transformers Agents is an experimental API which is subject to change at any time. Results returned by the agents | |
| can vary as the APIs or underlying models are prone to change.</p></div> <p data-svelte-h="svelte-hggcnq">Transformers version v4.29.0, building on the concept of <em>tools</em> and <em>agents</em>. You can play with in | |
| <a href="https://colab.research.google.com/drive/1c7MHD-T1forUPGcC_jlwsIptOzpG3hSj" rel="nofollow">this colab</a>.</p> <p data-svelte-h="svelte-g8wjpq">In short, it provides a natural language API on top of transformers: we define a set of curated tools and design an | |
| agent to interpret natural language and to use these tools. It is extensible by design; we curated some relevant tools, | |
| but we’ll show you how the system can be extended easily to use any tool developed by the community.</p> <p data-svelte-h="svelte-ijizhx">Let’s start with a few examples of what can be achieved with this new API. It is particularly powerful when it comes | |
| to multimodal tasks, so let’s take it for a spin to generate images and read text out loud.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run(<span class="hljs-string">"Caption the following image"</span>, image=image)<!-- HTML_TAG_END --></pre></div> <table data-svelte-h="svelte-gt1bz9"><thead><tr><th><strong>Input</strong></th> <th><strong>Output</strong></th></tr></thead> <tbody><tr><td><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/beaver.png" width="200"></td> <td>A beaver is swimming in the water</td></tr></tbody></table> <hr> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run(<span class="hljs-string">"Read the following text out loud"</span>, text=text)<!-- HTML_TAG_END --></pre></div> <table data-svelte-h="svelte-1a2vad0"><thead><tr><th><strong>Input</strong></th> <th><strong>Output</strong></th></tr></thead> <tbody><tr><td>A beaver is swimming in the water</td> <td><audio controls><source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tts_example.wav" type="audio/wav"> your browser does not support the audio element.</audio></td></tr></tbody></table> <hr> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run( | |
| <span class="hljs-string">"In the following `document`, where will the TRRF Scientific Advisory Council Meeting take place?"</span>, | |
| document=document, | |
| )<!-- HTML_TAG_END --></pre></div> <table data-svelte-h="svelte-1iw5xoi"><thead><tr><th><strong>Input</strong></th> <th><strong>Output</strong></th></tr></thead> <tbody><tr><td><img src="https://datasets-server.huggingface.co/assets/hf-internal-testing/example-documents/--/hf-internal-testing--example-documents/test/0/image/image.jpg" width="200"></td> <td>ballroom foyer</td></tr></tbody></table> <h2 class="relative group"><a id="quickstart" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#quickstart"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Quickstart</span></h2> <p data-svelte-h="svelte-1dmdrm1">Before being able to use <code>agent.run</code>, you will need to instantiate an agent, which is a large language model (LLM). | |
| We provide support for openAI models as well as opensource alternatives from BigCode and OpenAssistant. The openAI | |
| models perform better (but require you to have an openAI API key, so cannot be used for free); Hugging Face is | |
| providing free access to endpoints for BigCode and OpenAssistant models.</p> <p data-svelte-h="svelte-1dv7sdf">To start with, please install the <code>agents</code> extras in order to install all default dependencies.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->pip install transformers[agents]<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-gocwan">To use openAI models, you instantiate an <a href="/docs/transformers/v4.40.2/en/main_classes/agent#transformers.OpenAiAgent">OpenAiAgent</a> after installing the <code>openai</code> dependency:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->pip install openai<!-- HTML_TAG_END --></pre></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> OpenAiAgent | |
| agent = OpenAiAgent(model=<span class="hljs-string">"text-davinci-003"</span>, api_key=<span class="hljs-string">"<your_api_key>"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1jiytcn">To use BigCode or OpenAssistant, start by logging in to have access to the Inference API:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> huggingface_hub <span class="hljs-keyword">import</span> login | |
| login(<span class="hljs-string">"<YOUR_TOKEN>"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ezmlbn">Then, instantiate the agent</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> HfAgent | |
| <span class="hljs-comment"># Starcoder</span> | |
| agent = HfAgent(<span class="hljs-string">"https://api-inference.huggingface.co/models/bigcode/starcoder"</span>) | |
| <span class="hljs-comment"># StarcoderBase</span> | |
| <span class="hljs-comment"># agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoderbase")</span> | |
| <span class="hljs-comment"># OpenAssistant</span> | |
| <span class="hljs-comment"># agent = HfAgent(url_endpoint="https://api-inference.huggingface.co/models/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5")</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-4zoxn5">This is using the inference API that Hugging Face provides for free at the moment. If you have your own inference | |
| endpoint for this model (or another one) you can replace the URL above with your URL endpoint.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-13io30c">StarCoder and OpenAssistant are free to use and perform admirably well on simple tasks. However, the checkpoints | |
| don’t hold up when handling more complex prompts. If you’re facing such an issue, we recommend trying out the OpenAI | |
| model which, while sadly not open-source, performs better at this given time.</p></div> <p data-svelte-h="svelte-1pylma2">You’re now good to go! Let’s dive into the two APIs that you now have at your disposal.</p> <h3 class="relative group"><a id="single-execution-run" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#single-execution-run"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Single execution (run)</span></h3> <p data-svelte-h="svelte-10trlmm">The single execution method is when using the <a href="/docs/transformers/v4.40.2/en/main_classes/agent#transformers.Agent.run">run()</a> method of the agent:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run(<span class="hljs-string">"Draw me a picture of rivers and lakes."</span>)<!-- HTML_TAG_END --></pre></div> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png" width="200"> <p data-svelte-h="svelte-aarw7i">It automatically selects the tool (or tools) appropriate for the task you want to perform and runs them appropriately. It | |
| can perform one or several tasks in the same instruction (though the more complex your instruction, the more likely | |
| the agent is to fail).</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run(<span class="hljs-string">"Draw me a picture of the sea then transform the picture to add an island"</span>)<!-- HTML_TAG_END --></pre></div> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/sea_and_island.png" width="200"> <br> <p data-svelte-h="svelte-1y2g2zx">Every <a href="/docs/transformers/v4.40.2/en/main_classes/agent#transformers.Agent.run">run()</a> operation is independent, so you can run it several times in a row with different tasks.</p> <p data-svelte-h="svelte-1kxdglx">Note that your <code>agent</code> is just a large-language model, so small variations in your prompt might yield completely | |
| different results. It’s important to explain as clearly as possible the task you want to perform. We go more in-depth | |
| on how to write good prompts <a href="custom_tools#writing-good-user-inputs">here</a>.</p> <p data-svelte-h="svelte-pifkum">If you’d like to keep a state across executions or to pass non-text objects to the agent, you can do so by specifying | |
| variables that you would like the agent to use. For example, you could generate the first image of rivers and lakes, | |
| and ask the model to update that picture to add an island by doing the following:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->picture = agent.run(<span class="hljs-string">"Generate a picture of rivers and lakes."</span>) | |
| updated_picture = agent.run(<span class="hljs-string">"Transform the image in `picture` to add an island to it."</span>, picture=picture)<!-- HTML_TAG_END --></pre></div> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-xtdcfc">This can be helpful when the model is unable to understand your request and mixes tools. An example would be:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run(<span class="hljs-string">"Draw me the picture of a capybara swimming in the sea"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1d3eg7h">Here, the model could interpret in two ways:</p> <ul data-svelte-h="svelte-6h7gti"><li>Have the <code>text-to-image</code> generate a capybara swimming in the sea</li> <li>Or, have the <code>text-to-image</code> generate capybara, then use the <code>image-transformation</code> tool to have it swim in the sea</li></ul> <p data-svelte-h="svelte-17vcsom">In case you would like to force the first scenario, you could do so by passing it the prompt as an argument:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run(<span class="hljs-string">"Draw me a picture of the `prompt`"</span>, prompt=<span class="hljs-string">"a capybara swimming in the sea"</span>)<!-- HTML_TAG_END --></pre></div></div> <h3 class="relative group"><a id="chat-based-execution-chat" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#chat-based-execution-chat"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Chat-based execution (chat)</span></h3> <p data-svelte-h="svelte-zsx8hx">The agent also has a chat-based approach, using the <a href="/docs/transformers/v4.40.2/en/main_classes/agent#transformers.Agent.chat">chat()</a> method:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.chat(<span class="hljs-string">"Generate a picture of rivers and lakes"</span>)<!-- HTML_TAG_END --></pre></div> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png" width="200"> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.chat(<span class="hljs-string">"Transform the picture so that there is a rock in there"</span>)<!-- HTML_TAG_END --></pre></div> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes_and_beaver.png" width="200"> <br> <p data-svelte-h="svelte-1uy6dyg">This is an interesting approach when you want to keep the state across instructions. It’s better for experimentation, | |
| but will tend to be much better at single instructions rather than complex instructions (which the <a href="/docs/transformers/v4.40.2/en/main_classes/agent#transformers.Agent.run">run()</a> | |
| method is better at handling).</p> <p data-svelte-h="svelte-16wygo5">This method can also take arguments if you would like to pass non-text types or specific prompts.</p> <h3 class="relative group"><a id="-remote-execution" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#-remote-execution"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>⚠️ Remote execution</span></h3> <p data-svelte-h="svelte-1y37z6">For demonstration purposes and so that it could be used with all setups, we had created remote executors for several | |
| of the default tools the agent has access for the release. These are created using | |
| <a href="https://huggingface.co/inference-endpoints" rel="nofollow">inference endpoints</a>.</p> <p data-svelte-h="svelte-h6dg90">We have turned these off for now, but in order to see how to set up remote executors tools yourself, | |
| we recommend reading the <a href="./custom_tools">custom tool guide</a>.</p> <h3 class="relative group"><a id="whats-happening-here-what-are-tools-and-what-are-agents" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#whats-happening-here-what-are-tools-and-what-are-agents"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>What’s happening here? What are tools, and what are agents?</span></h3> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/diagram.png"> <h4 class="relative group"><a id="agents" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#agents"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Agents</span></h4> <p data-svelte-h="svelte-1h1hsmq">The “agent” here is a large language model, and we’re prompting it so that it has access to a specific set of tools.</p> <p data-svelte-h="svelte-1yih65e">LLMs are pretty good at generating small samples of code, so this API takes advantage of that by prompting the | |
| LLM gives a small sample of code performing a task with a set of tools. This prompt is then completed by the | |
| task you give your agent and the description of the tools you give it. This way it gets access to the doc of the | |
| tools you are using, especially their expected inputs and outputs, and can generate the relevant code.</p> <h4 class="relative group"><a id="tools" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#tools"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Tools</span></h4> <p data-svelte-h="svelte-1pl6quz">Tools are very simple: they’re a single function, with a name, and a description. We then use these tools’ descriptions | |
| to prompt the agent. Through the prompt, we show the agent how it would leverage tools to perform what was | |
| requested in the query.</p> <p data-svelte-h="svelte-owbr46">This is using brand-new tools and not pipelines, because the agent writes better code with very atomic tools. | |
| Pipelines are more refactored and often combine several tasks in one. Tools are meant to be focused on | |
| one very simple task only.</p> <h4 class="relative group"><a id="code-execution" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#code-execution"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Code-execution?!</span></h4> <p data-svelte-h="svelte-17ipo1r">This code is then executed with our small Python interpreter on the set of inputs passed along with your tools. | |
| We hear you screaming “Arbitrary code execution!” in the back, but let us explain why that is not the case.</p> <p data-svelte-h="svelte-or0pg0">The only functions that can be called are the tools you provided and the print function, so you’re already | |
| limited in what can be executed. You should be safe if it’s limited to Hugging Face tools.</p> <p data-svelte-h="svelte-fb990x">Then, we don’t allow any attribute lookup or imports (which shouldn’t be needed anyway for passing along | |
| inputs/outputs to a small set of functions) so all the most obvious attacks (and you’d need to prompt the LLM | |
| to output them anyway) shouldn’t be an issue. If you want to be on the super safe side, you can execute the | |
| run() method with the additional argument return_code=True, in which case the agent will just return the code | |
| to execute and you can decide whether to do it or not.</p> <p data-svelte-h="svelte-1ri56my">The execution will stop at any line trying to perform an illegal operation or if there is a regular Python error | |
| with the code generated by the agent.</p> <h3 class="relative group"><a id="a-curated-set-of-tools" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#a-curated-set-of-tools"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>A curated set of tools</span></h3> <p data-svelte-h="svelte-lbnuw">We identify a set of tools that can empower such agents. Here is an updated list of the tools we have integrated | |
| in <code>transformers</code>:</p> <ul data-svelte-h="svelte-5hcxax"><li><strong>Document question answering</strong>: given a document (such as a PDF) in image format, answer a question on this document (<a href="./model_doc/donut">Donut</a>)</li> <li><strong>Text question answering</strong>: given a long text and a question, answer the question in the text (<a href="./model_doc/flan-t5">Flan-T5</a>)</li> <li><strong>Unconditional image captioning</strong>: Caption the image! (<a href="./model_doc/blip">BLIP</a>)</li> <li><strong>Image question answering</strong>: given an image, answer a question on this image (<a href="./model_doc/vilt">VILT</a>)</li> <li><strong>Image segmentation</strong>: given an image and a prompt, output the segmentation mask of that prompt (<a href="./model_doc/clipseg">CLIPSeg</a>)</li> <li><strong>Speech to text</strong>: given an audio recording of a person talking, transcribe the speech into text (<a href="./model_doc/whisper">Whisper</a>)</li> <li><strong>Text to speech</strong>: convert text to speech (<a href="./model_doc/speecht5">SpeechT5</a>)</li> <li><strong>Zero-shot text classification</strong>: given a text and a list of labels, identify to which label the text corresponds the most (<a href="./model_doc/bart">BART</a>)</li> <li><strong>Text summarization</strong>: summarize a long text in one or a few sentences (<a href="./model_doc/bart">BART</a>)</li> <li><strong>Translation</strong>: translate the text into a given language (<a href="./model_doc/nllb">NLLB</a>)</li></ul> <p data-svelte-h="svelte-u6begh">These tools have an integration in transformers, and can be used manually as well, for example:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> load_tool | |
| tool = load_tool(<span class="hljs-string">"text-to-speech"</span>) | |
| audio = tool(<span class="hljs-string">"This is a text to speech tool"</span>)<!-- HTML_TAG_END --></pre></div> <h3 class="relative group"><a id="custom-tools" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#custom-tools"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Custom tools</span></h3> <p data-svelte-h="svelte-1qgu66u">While we identify a curated set of tools, we strongly believe that the main value provided by this implementation is | |
| the ability to quickly create and share custom tools.</p> <p data-svelte-h="svelte-32uplk">By pushing the code of a tool to a Hugging Face Space or a model repository, you’re then able to leverage the tool | |
| directly with the agent. We’ve added a few | |
| <strong>transformers-agnostic</strong> tools to the <a href="https://huggingface.co/huggingface-tools" rel="nofollow"><code>huggingface-tools</code> organization</a>:</p> <ul data-svelte-h="svelte-18r1pun"><li><strong>Text downloader</strong>: to download a text from a web URL</li> <li><strong>Text to image</strong>: generate an image according to a prompt, leveraging stable diffusion</li> <li><strong>Image transformation</strong>: modify an image given an initial image and a prompt, leveraging instruct pix2pix stable diffusion</li> <li><strong>Text to video</strong>: generate a small video according to a prompt, leveraging damo-vilab</li></ul> <p data-svelte-h="svelte-np3cvk">The text-to-image tool we have been using since the beginning is a remote tool that lives in | |
| <a href="https://huggingface.co/spaces/huggingface-tools/text-to-image" rel="nofollow"><em>huggingface-tools/text-to-image</em></a>! We will | |
| continue releasing such tools on this and other organizations, to further supercharge this implementation.</p> <p data-svelte-h="svelte-10s0ieo">The agents have by default access to tools that reside on <a href="https://huggingface.co/huggingface-tools" rel="nofollow"><code>huggingface-tools</code></a>. | |
| We explain how to you can write and share your tools as well as leverage any custom tool that resides on the Hub in <a href="custom_tools">following guide</a>.</p> <h3 class="relative group"><a id="code-generation" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#code-generation"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Code generation</span></h3> <p data-svelte-h="svelte-er9qni">So far we have shown how to use the agents to perform actions for you. However, the agent is only generating code | |
| that we then execute using a very restricted Python interpreter. In case you would like to use the code generated in | |
| a different setting, the agent can be prompted to return the code, along with tool definition and accurate imports.</p> <p data-svelte-h="svelte-16j3iq1">For example, the following instruction</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->agent.run(<span class="hljs-string">"Draw me a picture of rivers and lakes"</span>, return_code=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-pibxgg">returns the following code</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> load_tool | |
| image_generator = load_tool(<span class="hljs-string">"huggingface-tools/text-to-image"</span>) | |
| image = image_generator(prompt=<span class="hljs-string">"rivers and lakes"</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-rbbu3k">that you can then modify and execute yourself.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/transformers_agents.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_q46aet = { | |
| assets: "/docs/transformers/v4.40.2/en", | |
| base: "/docs/transformers/v4.40.2/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/v4.40.2/en/_app/immutable/entry/start.617cb5b4.js"), | |
| import("/docs/transformers/v4.40.2/en/_app/immutable/entry/app.6f576df8.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 396], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 59.9 kB
- Xet hash:
- 8edce5d64458be4bf945a142336ae06ce43a6ad1ebbc863ab0a38a615bbe5295
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.