Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Response Parsing","local":"response-parsing","sections":[{"title":"The parse_response method","local":"the-parseresponse-method","sections":[],"depth":2},{"title":"Developers: Understanding a simple response schema","local":"developers-understanding-a-simple-response-schema","sections":[],"depth":2},{"title":"Developers: Complex schemas","local":"developers-complex-schemas","sections":[],"depth":2},{"title":"Developers: Understanding the parser logic","local":"developers-understanding-the-parser-logic","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_36895/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/entry/start.b88698fb.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/scheduler.31fdf58d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/singletons.18caffe4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/index.252883d5.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/paths.a82d371a.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/entry/app.ac300a2c.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/preload-helper.643b8c40.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/index.2f76fdf0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/nodes/0.1669b2ea.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/nodes/14.8d6dc697.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/CopyLLMTxtMenu.9cbc081e.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/MermaidChart.svelte_svelte_type_style_lang.945f6700.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/IconCopy.ac192424.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36895/en/_app/immutable/chunks/CodeBlock.e52df5d6.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Response Parsing","local":"response-parsing","sections":[{"title":"The parse_response method","local":"the-parseresponse-method","sections":[],"depth":2},{"title":"Developers: Understanding a simple response schema","local":"developers-understanding-a-simple-response-schema","sections":[],"depth":2},{"title":"Developers: Complex schemas","local":"developers-complex-schemas","sections":[],"depth":2},{"title":"Developers: Understanding the parser logic","local":"developers-understanding-the-parser-logic","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <div class="items-center shrink-0 min-w-[100px] max-sm:min-w-[50px] justify-end ml-auto flex" style="float: right; margin-left: 10px; display: inline-flex; position: relative; z-index: 10;"><div class="inline-flex rounded-md max-sm:rounded-sm"><button class="inline-flex items-center gap-1 h-7 max-sm:h-7 px-2 max-sm:px-1.5 text-sm font-medium text-gray-800 border border-r-0 rounded-l-md max-sm:rounded-l-sm border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-live="polite"><span class="inline-flex items-center justify-center rounded-md p-0.5 max-sm:p-0 hover:text-gray-800 dark:hover:text-gray-200"><svg class="sm:size-3.5 size-3" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg></span> <span>Copy page</span></button> <button class="inline-flex items-center justify-center w-6 max-sm:w-5 h-7 max-sm:h-7 disabled:pointer-events-none text-sm text-gray-500 hover:text-gray-700 dark:hover:text-white rounded-r-md max-sm:rounded-r-sm border border-l transition border-gray-200 bg-white hover:shadow-inner dark:border-gray-850 dark:bg-gray-950 dark:text-gray-200 dark:hover:bg-gray-800" aria-haspopup="menu" aria-expanded="false" aria-label="Open copy menu"><svg class="transition-transform text-gray-400 overflow-visible sm:size-3.5 size-3 rotate-0" width="1em" height="1em" viewBox="0 0 12 7" fill="none" xmlns="http://www.w3.org/2000/svg"><path d="M1 1L6 6L11 1" stroke="currentColor"></path></svg></button></div> </div> <h1 class="relative group"><a id="response-parsing" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#response-parsing"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Response Parsing</span></h1> <p data-svelte-h="svelte-yj9icr">It is increasingly common for chat models to generate structured outputs, rather than just a single reply string. | |
| The most common uses for structured outputs are <a href="./chat_extras">tool calling</a> and <a href="https://huggingface.co/reasoning-course" rel="nofollow">reasoning models</a>. | |
| Tool calling models can output tool calls, containing the name of the tool to call and any arguments to be passed to it, | |
| while reasoning models often output reasoning steps as a “chain of thought”. Some recent models even use both of these, | |
| and may output reasoning and/or one or more tool calls before their final answer.</p> <p data-svelte-h="svelte-1d15j2x">Models with structured outputs pose a challenge for chat templating, because the output needs to be parsed before it | |
| can be appended to the chat. For a concrete example, let’s say we ask <a href="https://huggingface.co/openai/gpt-oss-120b" rel="nofollow">GPT-OSS</a> | |
| what the weather is like, and it thinks and decides to call a tool. Here’s what the raw model output might look like:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-txt "><!-- HTML_TAG_START --><|start|>analysis<|message|>The user asks: "What is the weather like in SF?" The user explicitly asks about SF (San Francisco). | |
| So we need to get the current weather in San Francisco, CA. We need to call get_current_weather function. | |
| So we should call get_current_weather with location "San Francisco, CA". Let's do that. | |
| We will call function get_current_weather.<|end|><|start|>commentary to=functions.get_current_weather<|channel|>commentary <|constrain|>json<|message|>{"location":"San Francisco, CA"}<|call|><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-pr40h6">But if you want to append this to a chat, you’ll need to format it as a chat message dict, like this:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-json "><!-- HTML_TAG_START --><span class="hljs-punctuation">{</span> | |
| <span class="hljs-attr">"role"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"assistant"</span><span class="hljs-punctuation">,</span> | |
| <span class="hljs-attr">"thinking"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"The user asks: \"What is the weather like in SF?\" We need to get the location of the user? The user explicitly asks about SF (San Francisco). So we need to get the current weather in San Francisco, CA. We need to call get_current_weather function. But we need to call function to get weather data. So we should call get_current_weather with location \"San Francisco, CA\". Let's do that."</span><span class="hljs-punctuation">,</span> | |
| <span class="hljs-attr">"tool_calls"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">[</span> | |
| <span class="hljs-punctuation">{</span> | |
| <span class="hljs-attr">"name"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"get_current_weather"</span><span class="hljs-punctuation">,</span> | |
| <span class="hljs-attr">"arguments"</span><span class="hljs-punctuation">:</span> <span class="hljs-punctuation">{</span> | |
| <span class="hljs-attr">"location"</span><span class="hljs-punctuation">:</span> <span class="hljs-string">"San Francisco, CA"</span> | |
| <span class="hljs-punctuation">}</span> | |
| <span class="hljs-punctuation">}</span> | |
| <span class="hljs-punctuation">]</span> | |
| <span class="hljs-punctuation">}</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-oqk4e6">Chat <strong>templates</strong> give us a way to turn messages into formatted input for a model, but we need something else to | |
| parse model output back into a standard message dict. This is what chat <strong>parsing</strong> is for.</p> <h2 class="relative group"><a id="the-parseresponse-method" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#the-parseresponse-method"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>The parse_response method</span></h2> <p data-svelte-h="svelte-ghr51">Parsing a chat response on a model that supports it is straightforward. Simply take the raw, decoded output from | |
| <a href="%60~generation.GenerationMixin.generate%60">generate</a>, and pass it to the tokenizer’s <a href="~PreTrainedTokenizerBase.parse_response">parse_response</a> method:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-python "><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoModelForCausalLM, AutoTokenizer | |
| checkpoint = <span class="hljs-string">"HuggingFaceTB/SmolLM3-3B"</span> | |
| tokenizer = AutoTokenizer.from_pretrained(checkpoint) | |
| model = AutoModelForCausalLM.from_pretrained(checkpoint, dtype=<span class="hljs-string">"auto"</span>, device_map=<span class="hljs-string">"auto"</span>) | |
| messages = [ | |
| { | |
| <span class="hljs-string">"role"</span>: <span class="hljs-string">"user"</span>, | |
| <span class="hljs-string">"content"</span>: <span class="hljs-string">"Hey! Can you summarize the end of the Cold War as briefly as possible? Like, comically briefly. It should really leave out almost most of the relevant information."</span> | |
| } | |
| ] | |
| input_ids = tokenizer.apply_chat_template( | |
| messages, | |
| add_generation_prompt=<span class="hljs-literal">True</span>, | |
| tokenize=<span class="hljs-literal">True</span>, | |
| return_tensors=<span class="hljs-string">"pt"</span> | |
| ).to(model.device) | |
| outputs = model.generate(input_ids, max_new_tokens=<span class="hljs-number">1024</span>)[<span class="hljs-number">0</span>, input_ids.shape[<span class="hljs-number">1</span>]:] | |
| out_text = tokenizer.decode(outputs) | |
| parsed = tokenizer.parse_response(out_text) | |
| <span class="hljs-built_in">print</span>(parsed.keys())<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-9qbsg1">And you should get:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-text "><!-- HTML_TAG_START -->dict_keys(['role', 'thinking', 'content'])<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-ankz23">And that’s all you need to start using response parsing! <code>parse_response</code> should return a complete message dict that is ready to be appended to the chat history. | |
| When the tokenizer does not support response parsing, <code>parse_response</code> will throw an error. We hope to add support | |
| to more tokenizers over time.</p> <h2 class="relative group"><a id="developers-understanding-a-simple-response-schema" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#developers-understanding-a-simple-response-schema"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Developers: Understanding a simple response schema</span></h2> <p data-svelte-h="svelte-1bf5k84">Under the hood, <code>parse_response</code> uses a <strong>JSON schema</strong> to parse the model output. A JSON schema represents | |
| the structure of the output message dict. The schema is augmented with additional fields that indicate how the | |
| output message string should be parsed into the expected format. Let’s take a look at the schema for a SmolLM response, | |
| excluding tool calls for now:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-python "><!-- HTML_TAG_START -->{ | |
| <span class="hljs-string">"x-regex"</span>: <span class="hljs-string">"(?:<think>\n?(?P<thinking>.+?)\n?</think>)?\s*(?P<content>.+?)?\s*(?:<\|im_end\|>|$)"</span>, | |
| <span class="hljs-string">"type"</span>: <span class="hljs-string">"object"</span>, | |
| <span class="hljs-string">"properties"</span>: { | |
| <span class="hljs-string">"role"</span>: {<span class="hljs-string">"const"</span>: <span class="hljs-string">"assistant"</span>}, | |
| <span class="hljs-string">"content"</span>: {<span class="hljs-string">"type"</span>: <span class="hljs-string">"string"</span>}, | |
| <span class="hljs-string">"thinking"</span>: {<span class="hljs-string">"type"</span>: <span class="hljs-string">"string"</span>} | |
| } | |
| }<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-8t74od">We can see that the schema describes a JSON “object” (a <code>dict</code>, in other words) with three keys: <code>role</code>, <code>content</code>, and <code>thinking</code>. | |
| Because all assistant responses have the role “assistant”, the <code>role</code> key is a <code>const</code>(ant). The other two keys are strings, extracted | |
| from the named groups in the regex in the <code>x-regex</code> field.</p> <p data-svelte-h="svelte-1qkniqw">Like chat templates, response schemas are set as a property of the tokenizer. To enable response parsing, all you need | |
| to do is set <code>tokenizer.response_schema</code> to a valid schema dict, and <code>tokenizer.parse_response()</code> will work! Again, like | |
| chat templates, this schema will be saved with the processor, so once you set it, you can use <code>save_pretrained()</code> or <code>push_to_hub()</code> to | |
| save and share the schema.</p> <h2 class="relative group"><a id="developers-complex-schemas" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#developers-complex-schemas"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Developers: Complex schemas</span></h2> <p data-svelte-h="svelte-1tm8w2m">Now, let’s look at a more complex schema, which includes tool calls, to gain more of an understanding of the parser | |
| internals. For this, we’ll use the <code>GPT-OSS</code> schema. GPT-OSS emits both tool calls and thinking blocks, and it uses | |
| an unusual format where model responses are tagged with one of three “channels”: <code>commentary</code> for things like | |
| tool calls, <code>analysis</code> for chain of thought blocks, and <code>final</code> for messages intended to be sent to the user. | |
| A full message where the model calls a tool named <code>get_current_weather</code> might look like this, with some extra linebreaks added for clarity:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-text "><!-- HTML_TAG_START --><|channel|>analysis<|message|> | |
| The user asks: "What is the weather like in SF?" So we need to get the current weather in San Francisco, CA. | |
| We need to call get_current_weather function. So we should call get_current_weather with location "San Francisco, CA". | |
| <|end|> | |
| <|start|>assistant<|channel|>commentary | |
| to=functions.get_current_weather <|constrain|>json<|message|> | |
| { | |
| "location": "San Francisco, CA" | |
| } | |
| <|call|><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-16sxd71">Parsing proceeds recursively; the output of a regex (or other parser) at one level becomes the input to the nodes below it. | |
| In other words, don’t feel like you have to parse the entire output in one enormous regex! Instead, start with the schema, | |
| and then add regexes to extract the relevant chunks as you go. Here’s a schema that will parse it, with some | |
| explanatory comments:</p> <div class="code-block relative "><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class="language-python "><!-- HTML_TAG_START -->{ | |
| <span class="hljs-string">"type"</span>: <span class="hljs-string">"object"</span>, | |
| <span class="hljs-string">"properties"</span>: { | |
| <span class="hljs-string">"role"</span>: {<span class="hljs-string">"const"</span>: <span class="hljs-string">"assistant"</span>}, | |
| <span class="hljs-comment"># "content" and "thinking" are both similar to the previous example, and just extract a single string</span> | |
| <span class="hljs-comment"># However, rather than using a single regex with named groups to extract both, we use a regex in each subkey.</span> | |
| <span class="hljs-comment"># When an object node has no parser/regex, the entire input string is passed to all of its children, so </span> | |
| <span class="hljs-comment"># parsing can either be done with named groups at the object level, or with separate regexes at the property level.</span> | |
| <span class="hljs-string">"content"</span>: {<span class="hljs-string">"type"</span>: <span class="hljs-string">"string"</span>, <span class="hljs-string">"x-regex"</span>: <span class="hljs-string">r"<\|channel\|>final<\|message\|>(.*?)(?:<\|end\|>|$)"</span>}, | |
| <span class="hljs-string">"thinking"</span>: {<span class="hljs-string">"type"</span>: <span class="hljs-string">"string"</span>, <span class="hljs-string">"x-regex"</span>: <span class="hljs-string">r"<\|channel\|>analysis<\|message\|>(.*?)<\|end\|>"</span>}, | |
| <span class="hljs-string">"tool_calls"</span>: { | |
| <span class="hljs-comment"># "x-regex-iterator" uses re.finditer to find multiple possible manages, and returns them as an</span> | |
| <span class="hljs-comment"># array/list. You don't need to worry about array handling, though - each item in the array will be</span> | |
| <span class="hljs-comment"># parsed by the `items` schema, so just write the schema for a single item.</span> | |
| <span class="hljs-string">"x-regex-iterator"</span>: <span class="hljs-string">r"<\|channel\|>commentary (to=functions\..*?<\|message\|>.*?)(?:<\|call\|>|$)"</span>, | |
| <span class="hljs-string">"type"</span>: <span class="hljs-string">"array"</span>, | |
| <span class="hljs-string">"items"</span>: { | |
| <span class="hljs-string">"type"</span>: <span class="hljs-string">"object"</span>, | |
| <span class="hljs-string">"properties"</span>: { | |
| <span class="hljs-comment"># A const property is a fixed value, and the input has no effect on it.</span> | |
| <span class="hljs-string">"type"</span>: {<span class="hljs-string">"const"</span>: <span class="hljs-string">"function"</span>}, | |
| <span class="hljs-comment"># Here, we wrap the entire tool call dict in a `{"function": ...}` block. The input string is passed through to it unchanged.</span> | |
| <span class="hljs-string">"function"</span>: { | |
| <span class="hljs-string">"type"</span>: <span class="hljs-string">"object"</span>, | |
| <span class="hljs-string">"properties"</span>: { | |
| <span class="hljs-string">"name"</span>: {<span class="hljs-string">"type"</span>: <span class="hljs-string">"string"</span>, <span class="hljs-string">"x-regex"</span>: <span class="hljs-string">r"^to=functions\.(\w+)"</span>}, | |
| <span class="hljs-string">"arguments"</span>: { | |
| <span class="hljs-string">"type"</span>: <span class="hljs-string">"object"</span>, | |
| <span class="hljs-string">"x-regex"</span>: <span class="hljs-string">"<\|message\|>(.*)"</span>, | |
| <span class="hljs-comment"># The "x-parser" field indicates that the extracted string should be parsed as JSON.</span> | |
| <span class="hljs-comment"># The output is then passed to the schema nodes below and recursive parsing continues.</span> | |
| <span class="hljs-string">"x-parser"</span>: <span class="hljs-string">"json"</span>, | |
| <span class="hljs-comment"># additionalProperties: True allows the parser to accept arbitrary keys </span> | |
| <span class="hljs-comment"># that are not specified in the schema.</span> | |
| <span class="hljs-string">"additionalProperties"</span>: <span class="hljs-literal">True</span>, | |
| }, | |
| }, | |
| }, | |
| }, | |
| }, | |
| }, | |
| }, | |
| }<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="developers-understanding-the-parser-logic" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#developers-understanding-the-parser-logic"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Developers: Understanding the parser logic</span></h2> <p data-svelte-h="svelte-vcbjhw">The parser follows a few simple rules:</p> <ol data-svelte-h="svelte-1ln2c8h"><li>Each level of the schema receives input from the level above, applies any regex or parser it has, and then passes the output to its children.</li> <li>The root level receives the entire decoded model output string as input.</li> <li>If a node has structured content after parsing (for example, if the regex has named groups and returns a dict, or if the parser returns a dict or list), | |
| then that structured content is mapped to the node’s children, and each child node receives its corresponding value as input.</li> <li>If an <code>object</code> (dict) node has unstructured (string) output, then the entire string is passed to all of its children. This allows child nodes | |
| to handle parsing individually rather than requiring a single parent regex to extract all keys at once.</li> <li>If an <code>array</code> (list) node has unstructured (string) output, then this throws an error.</li></ol> <p data-svelte-h="svelte-tkq3ps">There is a small set of allowable <code>x-</code> keys that indicate how parsing should be done at each node:</p> <ul data-svelte-h="svelte-pismj9"><li><code>x-regex</code>: A regex string to apply to the input. If the regex has named groups, the output is a dict of group names to values. Named groups should only be used in <code>object</code> nodes. | |
| Otherwise, the regex must have exactly one unnamed capturing group, and the output is the value of that group as a string.</li> <li><code>x-regex-iterator</code>: A regex string to apply to the input using <code>re.finditer()</code>. The output is a list of all matches. | |
| This should only be used in <code>array</code> nodes, and the regex must have exactly one unnamed capturing group. The output is distributed to | |
| the node’s <code>items</code> schema.</li> <li><code>x-parser</code>: Calls a built-in parser to apply to the input. Currently, the only supported parser is <code>json</code>, which parses the input string as JSON. | |
| The output is passed to the child nodes for further parsing. Note that the <code>json</code> parser can return deeply nested output - in this case, the output | |
| will be progressively unwrapped as it is passed through child nodes. The child nodes do not need additional <code>x-parser</code> or <code>x-regex</code> fields in this case, | |
| but their structure must match the structure of the parsed JSON.</li> <li><code>x-parser-args</code>: Only allowed in conjunction with <code>x-parser</code>. This is a dict of additional arguments that control parsing. Right now, the only supported | |
| argument is <code>transform</code>, which specifies a <code>jmespath</code> transformation to apply to the output. This is useful when the JSON parser returns a structure | |
| that needs to be modified to match the schema.</li> <li><code>x-regex-key-value</code>: This is rarely necessary, but it can be useful when parsing key-value pairs in non-JSON format where the names of the keys are not known | |
| in advance, such as when a model emits XML tool calls with arbitrary argument names. The regex must have exactly two named capturing groups, | |
| <code>key</code> and <code>value</code>, and the output is a dict mapping keys to values. This should only be used in <code>object</code> nodes.</li></ul> <p data-svelte-h="svelte-dj88z4">In general, multiple regexes/parsers cannot be combined at the same level. The exception is that <code>x-regex</code>, returning a single string, can be combined with the other parsers. In this case, | |
| <code>x-regex</code> is applied first, and then the output is passed to the other parser, either <code>x-regex-iterator</code>, <code>x-parser</code>, or <code>x-regex-key-value</code>. | |
| All regexes are applied with the <code>DOTALL</code> flag, since model outputs often contain newlines. This means that <code>.</code> matches all characters, including newlines.</p> <p data-svelte-h="svelte-15ad99z">Putting these ideas together, you can see that the input flows through the schema, being parsed at each level and then distributed to child nodes. Each level | |
| only needs to extract the input content that is relevant for that part of the schema, and can then let its child nodes handle the rest. Internally, this is handled | |
| with a parser function that receives input, applies any regexes/parsers at the current level, then maps the result to its child nodes before recursively calling itself on each of them. | |
| Recursion terminates when it reaches leaf nodes, usually primitive types like <code>string</code> or <code>number</code>, which simply return the input they receive.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/chat_response_parsing.md" target="_blank"><svg class="mr-1" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M31,16l-7,7l-1.41-1.41L28.17,16l-5.58-5.59L24,9l7,7z"></path><path d="M1,16l7-7l1.41,1.41L3.83,16l5.58,5.59L8,23l-7-7z"></path><path d="M12.419,25.484L17.639,6.552l1.932,0.518L14.351,26.002z"></path></svg> <span data-svelte-h="svelte-zjs2n5"><span class="underline">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1s77840 = { | |
| assets: "/docs/transformers/pr_36895/en", | |
| base: "/docs/transformers/pr_36895/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_36895/en/_app/immutable/entry/start.b88698fb.js"), | |
| import("/docs/transformers/pr_36895/en/_app/immutable/entry/app.ac300a2c.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 14], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 40.9 kB
- Xet hash:
- f7361168ffa862ca94ab680e037e04618f629130a15a9df231c73276d620f547
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.