Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Generation features","local":"generation-features","sections":[{"title":"Streaming","local":"streaming","sections":[],"depth":2},{"title":"Watermarking","local":"watermarking","sections":[],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_36839/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/scheduler.01eeda35.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/singletons.177df05e.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.4862150a.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/paths.517376d1.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/index.6dd51b66.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/0.8897c14d.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/nodes/23.f02c6b22.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/Tip.de9bae2b.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/CodeBlock.864da1b0.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36839/en/_app/immutable/chunks/EditOnGithub.7faefd25.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Generation features","local":"generation-features","sections":[{"title":"Streaming","local":"streaming","sections":[],"depth":2},{"title":"Watermarking","local":"watermarking","sections":[],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="generation-features" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#generation-features"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Generation features</span></h1> <p data-svelte-h="svelte-sb9i59">The <a href="/docs/transformers/pr_36839/en/main_classes/text_generation#transformers.GenerationMixin.generate">generate()</a> API supports a couple features for building applications on top of it.</p> <p data-svelte-h="svelte-fugxhy">This guide will show you how to use these features.</p> <h2 class="relative group"><a id="streaming" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#streaming"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Streaming</span></h2> <p data-svelte-h="svelte-1crgzf8">Streaming starts returning text as soon as it is generated so you don’t have to wait to see the entire generated response all at once. It is important in user-facing applications because it reduces perceived latency and allows users to see the generation progression.</p> <div class="flex justify-center" data-svelte-h="svelte-cjtnoe"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/tgi/streaming-generation-visual-dark_360.gif"></div> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-16cvhv6">Learn more about streaming in the <a href="https://huggingface.co/docs/text-generation-inference/en/conceptual/streaming" rel="nofollow">Text Generation Inference</a> docs.</p></div> <p data-svelte-h="svelte-rc5aw0">Create an instance of <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.TextStreamer">TextStreamer</a> with the tokenizer. Pass <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.TextStreamer">TextStreamer</a> to the <code>streamer</code> parameter in <a href="/docs/transformers/pr_36839/en/main_classes/text_generation#transformers.GenerationMixin.generate">generate()</a> to stream the output one word at a time.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoModelForCausalLM, AutoTokenizer, TextStreamer | |
| tokenizer = AutoTokenizer.from_pretrained(<span class="hljs-string">"openai-community/gpt2"</span>) | |
| model = AutoModelForCausalLM.from_pretrained(<span class="hljs-string">"openai-community/gpt2"</span>) | |
| inputs = tokenizer([<span class="hljs-string">"The secret to baking a good cake is "</span>], return_tensors=<span class="hljs-string">"pt"</span>) | |
| streamer = TextStreamer(tokenizer) | |
| _ = model.generate(**inputs, streamer=streamer, max_new_tokens=<span class="hljs-number">20</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1rvij9b">The <code>streamer</code> parameter is compatible with any class with a <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.TextStreamer.put">put()</a> and <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.TextStreamer.end">end()</a> method. <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.TextStreamer.put">put()</a> pushes new tokens and <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.TextStreamer.end">end()</a> flags the end of generation. You can create your own streamer class as long as they include these two methods, or you can use Transformers’ basic streamer classes.</p> <h2 class="relative group"><a id="watermarking" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#watermarking"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Watermarking</span></h2> <p data-svelte-h="svelte-len99d">Watermarking is useful for detecting whether text is generated. The <a href="https://hf.co/papers/2306.04634" rel="nofollow">watermarking strategy</a> in Transformers randomly “colors” a subset of the tokens green. When green tokens are generated, they have a small bias added to their logits, and a higher probability of being generated. You can detect generated text by comparing the proportion of green tokens to the amount of green tokens typically found in human-generated text.</p> <p data-svelte-h="svelte-aajjni">Watermarking is supported for any generative model in Transformers and doesn’t require an extra classification model to detect the watermarked text.</p> <p data-svelte-h="svelte-wb1ob5">Create a <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.WatermarkingConfig">WatermarkingConfig</a> with the bias value to add to the logits and watermarking algorithm. The example below uses the <code>"selfhash"</code> algorithm, where the green token selection only depends on the current token. Pass the <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.WatermarkingConfig">WatermarkingConfig</a> to <a href="/docs/transformers/pr_36839/en/main_classes/text_generation#transformers.GenerationMixin.generate">generate()</a>.</p> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-yl3kjq">The <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.WatermarkDetector">WatermarkDetector</a> class detects the proportion of green tokens in generated text, which is why it is recommended to strip the prompt text, if it is much longer than the generated text. Padding can also have an effect on <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.WatermarkDetector">WatermarkDetector</a>.</p></div> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoTokenizer, AutoModelForCausalLM, WatermarkDetector, WatermarkingConfig | |
| model = AutoModelForCausalLM.from_pretrained(<span class="hljs-string">"openai-community/gpt2"</span>) | |
| tokenizer = AutoTokenizer.from_pretrained(<span class="hljs-string">"openai-community/gpt2"</span>) | |
| tokenizer.pad_token_id = tokenizer.eos_token_id | |
| tokenizer.padding_side = <span class="hljs-string">"left"</span> | |
| inputs = tokenizer([<span class="hljs-string">"This is the beginning of a long story"</span>, <span class="hljs-string">"Alice and Bob are"</span>], padding=<span class="hljs-literal">True</span>, return_tensors=<span class="hljs-string">"pt"</span>) | |
| input_len = inputs[<span class="hljs-string">"input_ids"</span>].shape[-<span class="hljs-number">1</span>] | |
| watermarking_config = WatermarkingConfig(bias=<span class="hljs-number">2.5</span>, seeding_scheme=<span class="hljs-string">"selfhash"</span>) | |
| out = model.generate(**inputs, watermarking_config=watermarking_config, do_sample=<span class="hljs-literal">False</span>, max_length=<span class="hljs-number">20</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1dl6fim">Create an instance of <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.WatermarkDetector">WatermarkDetector</a> and pass the model output to it to detect whether the text is machine-generated. The <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.WatermarkDetector">WatermarkDetector</a> must have the same <a href="/docs/transformers/pr_36839/en/internal/generation_utils#transformers.WatermarkingConfig">WatermarkingConfig</a> used during generation.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->detector = WatermarkDetector(model_config=model.config, device=<span class="hljs-string">"cpu"</span>, watermarking_config=watermarking_config) | |
| detection_out = detector(out, return_dict=<span class="hljs-literal">True</span>) | |
| detection_out.prediction | |
| array([<span class="hljs-literal">True</span>, <span class="hljs-literal">True</span>])<!-- HTML_TAG_END --></pre></div> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/generation_features.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_1bm5psi = { | |
| assets: "/docs/transformers/pr_36839/en", | |
| base: "/docs/transformers/pr_36839/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/start.6be8d590.js"), | |
| import("/docs/transformers/pr_36839/en/_app/immutable/entry/app.09748b4b.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 23], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 17.8 kB
- Xet hash:
- f7c087bfb8cab0d81d29a44fc2b69b28c8895f5320782488e1cf2804e4ac9e89
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.