text-generation-inference/main/en/conceptual/speculation.html

<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Speculation&quot;,&quot;local&quot;:&quot;speculation&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Medusa&quot;,&quot;local&quot;:&quot;medusa&quot;,&quot;sections&quot;:[],&quot;depth&quot;:3},{&quot;title&quot;:&quot;N-gram&quot;,&quot;local&quot;:&quot;n-gram&quot;,&quot;sections&quot;:[],&quot;depth&quot;:3}],&quot;depth&quot;:2}">
		<link href="/docs/text-generation-inference/main/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/entry/start.1810066f.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/chunks/scheduler.362310b7.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/chunks/singletons.fa2b0eb7.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/chunks/index.7f53ec41.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/chunks/paths.284aef40.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/entry/app.8cfc1931.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/chunks/index.57dfc70d.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/nodes/0.543c9bd9.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/chunks/each.e59479a4.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/nodes/20.0db16f84.js">
		<link rel="modulepreload" href="/docs/text-generation-inference/main/en/_app/immutable/chunks/EditOnGithub.9633c464.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;Speculation&quot;,&quot;local&quot;:&quot;speculation&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Medusa&quot;,&quot;local&quot;:&quot;medusa&quot;,&quot;sections&quot;:[],&quot;depth&quot;:3},{&quot;title&quot;:&quot;N-gram&quot;,&quot;local&quot;:&quot;n-gram&quot;,&quot;sections&quot;:[],&quot;depth&quot;:3}],&quot;depth&quot;:2}"><!-- HEAD_svelte-u9bgzb_END -->      <p></p>  <h2 class="relative group"><a id="speculation" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#speculation"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Speculation</span></h2> <p data-svelte-h="svelte-17a28u5">Speculative decoding, assisted generation, Medusa, and others are a few different names for the same idea.
The idea is to generate tokens <em>before</em> the large model actually runs, and only <em>check</em> if those tokens where valid.</p> <p data-svelte-h="svelte-1tzgo4g">So you are making <em>more</em> computations on your LLM, but if you are correct you produce 1, 2, 3 etc.. tokens on a single LLM pass. Since LLMs are usually memory bound (and not compute bound), provided your guesses are correct enough, this is a 2-3x faster inference (It can be much more for code oriented tasks for instance).</p> <p data-svelte-h="svelte-qhbwif">You can check a more <a href="https://huggingface.co/blog/assisted-generation" rel="nofollow">detailed explanation</a>.</p> <p data-svelte-h="svelte-1sqs0g7">Text-generation inference supports 2 main speculative methods:</p> <ul data-svelte-h="svelte-1d4jtwt"><li>Medusa</li> <li>N-gram</li></ul>  <h3 class="relative group"><a id="medusa" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#medusa"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Medusa</span></h3> <p data-svelte-h="svelte-1j9vs1z">Medusa is a <a href="https://arxiv.org/abs/2401.10774" rel="nofollow">simple method</a> to create many tokens in a single pass using fine-tuned LM heads in addition to your existing models.</p> <p data-svelte-h="svelte-85cbh3">You can check a few existing  fine-tunes for popular models:</p> <ul data-svelte-h="svelte-m7tzce"><li><a href="https://huggingface.co/text-generation-inference/gemma-7b-it-medusa" rel="nofollow">text-generation-inference/gemma-7b-it-medusa</a></li> <li><a href="https://huggingface.co/text-generation-inference/Mixtral-8x7B-Instruct-v0.1-medusa" rel="nofollow">text-generation-inference/Mixtral-8x7B-Instruct-v0.1-medusa</a></li> <li><a href="https://huggingface.co/text-generation-inference/Mistral-7B-Instruct-v0.2-medusa" rel="nofollow">text-generation-inference/Mistral-7B-Instruct-v0.2-medusa</a></li></ul> <p data-svelte-h="svelte-1otaesp">In order to create your own medusa heads for your own finetune, you should check own the original medusa repo. <a href="../basic_tutorials/train_medusa.md">../basic_tutorials/train_medusa.md</a></p> <p data-svelte-h="svelte-x0oydl">In order to use medusa models in TGI, simply point to a medusa enabled model, and everything will load automatically.</p>  <h3 class="relative group"><a id="n-gram" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#n-gram"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>N-gram</span></h3> <p data-svelte-h="svelte-nr08mo">If you don’t have a medusa model, or don’t have the resource to fine-tune, you can try to use <code>n-gram</code>.
N-gram works by trying to find matching tokens in the previous sequence, and use those as speculation for generating new tokens. For example, if the tokens “np.mean” appear multiple times in the sequence, the model can speculate that the next continuation of the tokens “np.” is probably also “mean”.</p> <p data-svelte-h="svelte-1nvhqv5">This is an extremely simple method, which works best for code, or highly repetitive text. This might not be beneficial, if the speculation misses too much.</p> <p data-svelte-h="svelte-nu16xj">In order to enable n-gram speculation simply use</p> <p data-svelte-h="svelte-1wnslsy"><code>--speculate 2</code> in your flags.</p> <p data-svelte-h="svelte-1aes5um"><a href="https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher#speculate" rel="nofollow">Details about the flag</a></p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/text-generation-inference/blob/main/docs/source/conceptual/speculation.md" target="_blank"><span data-svelte-h="svelte-1kd6by1">&lt;</span> <span data-svelte-h="svelte-x0xyl0">&gt;</span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a>  <p></p> 
			
			<script>
				{
					__sveltekit_1dfb6m4 = {
						assets: "/docs/text-generation-inference/main/en",
						base: "/docs/text-generation-inference/main/en",
						env: {}
					};

					const element = document.currentScript.parentElement;

					const data = [null,null];

					Promise.all([
						import("/docs/text-generation-inference/main/en/_app/immutable/entry/start.1810066f.js"),
						import("/docs/text-generation-inference/main/en/_app/immutable/entry/app.8cfc1931.js")
					]).then(([kit, app]) => {
						kit.start(app, element, {
							node_ids: [0, 20],
							data,
							form: null,
							error: null
						});
					});
				}
			</script>