Buckets:
| <meta charset="utf-8" /><meta name="hf:doc:metadata" content="{"title":"Introduction to Attention Mechanisms in Transformers","local":"introduction-to-attention-mechanisms-in-transformers","sections":[{"title":"Query, Key, and Value","local":"query-key-and-value","sections":[],"depth":2},{"title":"Scaled Dot-Product Attention","local":"scaled-dot-product-attention","sections":[],"depth":2},{"title":"Multi-Head Attention","local":"multi-head-attention","sections":[],"depth":2},{"title":"Self-attention, causal self-attention, and cross-attention","local":"self-attention-causal-self-attention-and-cross-attention","sections":[],"depth":2},{"title":"Sparse Attention Mechanisms","local":"sparse-attention-mechanisms","sections":[{"title":"Locality-Sensitive Hashing Attention","local":"locality-sensitive-hashing-attention","sections":[],"depth":3},{"title":"Local Attention for Efficient Sequence Processing","local":"local-attention-for-efficient-sequence-processing","sections":[{"title":"Other tricks","local":"other-tricks","sections":[{"title":"Axial positional encodings","local":"axial-positional-encodings","sections":[],"depth":5}],"depth":4}],"depth":3}],"depth":2}],"depth":1}"> | |
| <link href="/docs/transformers/pr_36597/en/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/entry/start.b2a3acf6.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/chunks/scheduler.01eeda35.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/chunks/singletons.01837060.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/chunks/index.4862150a.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/chunks/paths.0dffc251.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/entry/app.ad1361e9.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/chunks/index.6dd51b66.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/nodes/0.0c85cba2.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/chunks/each.e59479a4.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/nodes/6.9c4bdb67.js"> | |
| <link rel="modulepreload" href="/docs/transformers/pr_36597/en/_app/immutable/chunks/EditOnGithub.7faefd25.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{"title":"Introduction to Attention Mechanisms in Transformers","local":"introduction-to-attention-mechanisms-in-transformers","sections":[{"title":"Query, Key, and Value","local":"query-key-and-value","sections":[],"depth":2},{"title":"Scaled Dot-Product Attention","local":"scaled-dot-product-attention","sections":[],"depth":2},{"title":"Multi-Head Attention","local":"multi-head-attention","sections":[],"depth":2},{"title":"Self-attention, causal self-attention, and cross-attention","local":"self-attention-causal-self-attention-and-cross-attention","sections":[],"depth":2},{"title":"Sparse Attention Mechanisms","local":"sparse-attention-mechanisms","sections":[{"title":"Locality-Sensitive Hashing Attention","local":"locality-sensitive-hashing-attention","sections":[],"depth":3},{"title":"Local Attention for Efficient Sequence Processing","local":"local-attention-for-efficient-sequence-processing","sections":[{"title":"Other tricks","local":"other-tricks","sections":[{"title":"Axial positional encodings","local":"axial-positional-encodings","sections":[],"depth":5}],"depth":4}],"depth":3}],"depth":2}],"depth":1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="introduction-to-attention-mechanisms-in-transformers" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#introduction-to-attention-mechanisms-in-transformers"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Introduction to Attention Mechanisms in Transformers</span></h1> <p data-svelte-h="svelte-1kuh7rx">Attention is at the core of every transformer architecture. The title of the groundbreaking paper ”<strong>Attention Is All You Need</strong>” by Vaswani et al. is no coincidence. In this work, they presented the “Scaled Dot-Product Attention” mechanism, which was then extended by the splitting of the tensors into multiple heads (Multi-Head Attention), enabling parallelization and improved feature extraction. Since then, many variations have emerged, each addressing specific challenges such as computational efficiency, sequence length limitations, and others.</p> <p data-svelte-h="svelte-1dj7bhs">At a high level, attention is an ingenious mechanism through which the model learns to focus on the most relevant parts of the input data when making predictions or generating inputs. Here, we provide a brief introduction on the workings of the most popular attention mechanisms.</p> <h2 class="relative group"><a id="query-key-and-value" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#query-key-and-value"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Query, Key, and Value</span></h2> <p data-svelte-h="svelte-xgxd11">Imagine you are particularly interested in learning about the Big Bang — this is your query, $Q$; your focus. You walk into a library, and each book has a detailed index that summarizes its content — this is the key, $K$. The value, $V$, is the actual information you extract from the books; paragraphs, or whole chapters. First, you compare the match between your query and key. This comparison provides you with attention scores, i.e., which books are the most relevant. Finally, to accomplish your task, you gather information from various different books that may have some similarity with your query — for example, cosmology, biology, and even some religious texts.</p> <p data-svelte-h="svelte-j6gxzj">Technically, $Q$, $K$, $V$ are individually obtained by multiplying the input embedding $X$ with their respective projection matrices $W^Q$ for the query, $W^K$ for the key, and $W^V$ for the value. Additionally, $Q, K \in \mathbb{R}^{t\times d_k}$, and $V \in \mathbb{R}^{t\times d_v}$. To simplify the discussion and notation, we will assume $d_v = d_k = d$.</p> <h2 class="relative group"><a id="scaled-dot-product-attention" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#scaled-dot-product-attention"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Scaled Dot-Product Attention</span></h2> <p data-svelte-h="svelte-1giso2q">The dot product between two vectors is a measure of their similarity. Orthogonal vectors have a dot product equal to 0, as their projection onto each other is null. This notion of similarity can be extended to matrices, as their rows and columns can be viewed as vectors.</p> <p>The first step in the Scaled Dot-Product Attention (SDPA) involves computing the <em data-svelte-h="svelte-1v443iz">attention scores</em> $Q\cdot K^T$, or the level of similarity between the query and the key. This product is then divided by a scaling factor $\sqrt{d_k}$, yielding a zero-mean and unit-variance product (provided the elements of $Q$ and $K$ are already normalized). The <em data-svelte-h="svelte-i999kw">softmax</em> function, which acts as a normalizer, is then applied to this product to yield the <em data-svelte-h="svelte-117invb">attention weights</em>: $\text{softmax}(\frac{Q\cdot K^T}{\sqrt{d_k}})$. | |
| Finally, the attention weights matrix is multiplied by the value $V$:<!-- HTML_TAG_START --><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>Attention</mtext><mo stretchy="false">(</mo><mi>Q</mi><mo separator="true">,</mo><mi>K</mi><mo separator="true">,</mo><mi>V</mi><mo stretchy="false">)</mo><mo>=</mo><mtext>softmax</mtext><mo stretchy="false">(</mo><mfrac><mrow><mi>Q</mi><mo>⋅</mo><msup><mi>K</mi><mi>T</mi></msup></mrow><msqrt><msub><mi>d</mi><mi>k</mi></msub></msqrt></mfrac><mo stretchy="false">)</mo><mi>V</mi></mrow><annotation encoding="application/x-tex">\text{Attention}(Q, K, V) = \text{softmax}(\frac{Q\cdot K^T}{\sqrt{d_k}})V</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord text"><span class="mord">Attention</span></span><span class="mopen">(</span><span class="mord mathnormal">Q</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.07153em;">K</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.22222em;">V</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:2.4483em;vertical-align:-0.93em;"></span><span class="mord text"><span class="mord">softmax</span></span><span class="mopen">(</span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.5183em;"><span style="top:-2.2528em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord sqrt"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8572em;"><span class="svg-align" style="top:-3em;"><span class="pstrut" style="height:3em;"></span><span class="mord" style="padding-left:0.833em;"><span class="mord"><span class="mord mathnormal">d</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.03148em;">k</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span><span style="top:-2.8172em;"><span class="pstrut" style="height:3em;"></span><span class="hide-tail" style="min-width:0.853em;height:1.08em;"><svg xmlns="http://www.w3.org/2000/svg" width='400em' height='1.08em' viewBox='0 0 400000 1080' preserveAspectRatio='xMinYMin slice'><path d='M95,702 | |
| c-2.7,0,-7.17,-2.7,-13.5,-8c-5.8,-5.3,-9.5,-10,-9.5,-14 | |
| c0,-2,0.3,-3.3,1,-4c1.3,-2.7,23.83,-20.7,67.5,-54 | |
| c44.2,-33.3,65.8,-50.3,66.5,-51c1.3,-1.3,3,-2,5,-2c4.7,0,8.7,3.3,12,10 | |
| s173,378,173,378c0.7,0,35.3,-71,104,-213c68.7,-142,137.5,-285,206.5,-429 | |
| c69,-144,104.5,-217.7,106.5,-221 | |
| l0 -0 | |
| c5.3,-9.3,12,-14,20,-14 | |
| H400000v40H845.2724 | |
| s-225.272,467,-225.272,467s-235,486,-235,486c-2.7,4.7,-9,7,-19,7 | |
| c-6,0,-10,-1,-12,-3s-194,-422,-194,-422s-65,47,-65,47z | |
| M834 80h400000v40h-400000z'/></svg></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.1828em;"><span></span></span></span></span></span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord mathnormal">Q</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">⋅</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.07153em;">K</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8413em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em;">T</span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.93em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mclose">)</span><span class="mord mathnormal" style="margin-right:0.22222em;">V</span></span></span></span></span><!-- HTML_TAG_END --></p> <p data-svelte-h="svelte-9pn409">In the context of sequence processing, for a sequence of length $t$, the attention weights matrix will be $t \times t$. Note that the $ij$ element of this matrix comes from the multiplication between the $i$-th row vector of $Q$ and the $j$-th column vector of $K^T$, i.e., the $j$-th row of $K$. Since $Q$, $K$ and $V$ have dimensions $t \times d$, each row is a vector of a token embedded in a $d$-dimensional space. Thus, the $ij$ element of the attention weight matrix represents the similarity between the latent (embedded) representation between the $i$-th query token and the $j$-th key token.</p> <div align="center" data-svelte-h="svelte-1tqad8j"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/a5fce677dfff6da5a76534e53a7c89ea1ae19bd9/query-key-matmul.svg" alt="Query and Key Matrix multiplication"></div> <p data-svelte-h="svelte-a9s9j0">Finally, the attention weight matrix multiplies the value matrix $V$, effectively taking a weighted sum of the sequence tokens embedded in the value space, extracting the information it learns to be most important.</p> <p data-svelte-h="svelte-1xn7eb6">It is noteworthy to observe that the query, key and value are not static but rather different representations learned throughout the course of the training process.</p> <p data-svelte-h="svelte-1i57g34">In sum, the attention weights matrix guides how much attention the $i$-th query token should pay to the $j$-th key token for the downstream task. This allows the model to dynamically determine which parts of the sequence are most relevant when processing each token.</p> <h2 class="relative group"><a id="multi-head-attention" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#multi-head-attention"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Multi-Head Attention</span></h2> <p>The original transformer paper not only presented the Scaled Dot-Product Attention, but also its implementation applied to multiple “heads”, which is known as <strong data-svelte-h="svelte-gzwfn4">Multi-head attention</strong>. A <em data-svelte-h="svelte-vkbg94">head</em> is an individual attention mechanism that operates in parallel with other heads. Its mathematical formulation is as follows: | |
| <!-- HTML_TAG_START --><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>MultiHead</mtext><mo stretchy="false">(</mo><mi>Q</mi><mo separator="true">,</mo><mi>K</mi><mo separator="true">,</mo><mi>V</mi><mo stretchy="false">)</mo><mo>=</mo><mtext>Concat</mtext><mo stretchy="false">(</mo><msub><mtext>head</mtext><mn>1</mn></msub><mo separator="true">,</mo><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mi mathvariant="normal">.</mi><mo separator="true">,</mo><msub><mtext>head</mtext><mi>h</mi></msub><mo stretchy="false">)</mo><msup><mi>W</mi><mi>O</mi></msup></mrow><annotation encoding="application/x-tex"> \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord text"><span class="mord">MultiHead</span></span><span class="mopen">(</span><span class="mord mathnormal">Q</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.07153em;">K</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord mathnormal" style="margin-right:0.22222em;">V</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1.1413em;vertical-align:-0.25em;"></span><span class="mord text"><span class="mord">Concat</span></span><span class="mopen">(</span><span class="mord"><span class="mord text"><span class="mord">head</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">1</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">...</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mord text"><span class="mord">head</span></span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">h</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mclose">)</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.13889em;">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8913em;"><span style="top:-3.113em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.02778em;">O</span></span></span></span></span></span></span></span></span></span></span></span><!-- HTML_TAG_END --> | |
| where $$ \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) $$</p> <p data-svelte-h="svelte-1htm0uo">$AW_i^A$ represents the projection of $A$ in the $i$-th head. It is important to note that if the query, key, and value lie in a $d$-dimensional space and the model has $n$ heads, the projections at each of the heads will convert the respective query, key and value to a space of $d/n$ dimensions. After the concatenation of the heads, they are projected again to a $d$-dimensional space ($W^O$ matrix).</p> <p data-svelte-h="svelte-1se0abb">The effectiveness of employing multiple heads is similar to that of an ensemble, in which each head learns different aspects of the sequence. Moreover, it enables parallel computation, which makes it more computionally efficient. In language modeling, for example, one could say that one head focuses on subject-verb interactions, while another head focus on noun-subject. However, in reality, the interactions during the learning process are much more complex and not as well-defined as in this analogy.</p> <div align="center" data-svelte-h="svelte-f0hff3"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/a5fce677dfff6da5a76534e53a7c89ea1ae19bd9/multi-head-attention.svg" alt="Multi-Head Attention" style="width:29%;"></div> <h2 class="relative group"><a id="self-attention-causal-self-attention-and-cross-attention" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#self-attention-causal-self-attention-and-cross-attention"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Self-attention, causal self-attention, and cross-attention</span></h2> <p data-svelte-h="svelte-4woieb"><strong>Self-attention</strong> occurs when a token attends to other tokens in the same sequence, meaning the query, key, and value all originate from the same source. In the original Transformer model, self-attention is used in both the encoder and decoder. However, the decoder employs <strong>causal self-attention</strong>, also known as ‘Masked Multi-Head Attention’. This ensures, the model is prevented from attending tokens it has not seen yet, i.e., future tokens. Therefore, it is commonly present in autoregressive or <em>causal</em> language models, whose main purpose is sequence generation. The causal mask assigns large negative values (e.g., $-\infty$) to attention scores of future tokens before applying the softmax. This forces their attention weights to be zero, ensuring that no information leaks from future tokens.</p> <p data-svelte-h="svelte-1ak5nq9">In contrast to self-attention, <strong>cross-attention</strong> is an attention mechanism in which the query, key and value do not come from the same source. In the original Transformer model, the output of the final encoder layer is fed as the key and value for the second multi-head attention module of the transformer. The query comes from the decoder (causal) self-attention module. In cross-attention, the query comes from the decoder (based on the current token in the target sequence), while the key (which provides the “index”) and value (which carries actual encoded information) come from the encoder. Therefore, the network is trying to make sense of how the past tokens influence the token being currently generated.</p> <p data-svelte-h="svelte-1xh6fcr">In the context of machine translation, suppose the task is to translate “The cat sat on the mat” from English to French, “Le chat était assis sur le tapis”, which is the target sequence. During token generation, the query (from the decoder) attends to the entire encoded English sentence, which serves as the key and value. This allows the model to determine how different parts of the source sentence influence the next token to be generated in the target language.</p> <p data-svelte-h="svelte-1nyxtn7">The figure below illustrates where each of these attention mechanisms is present in the transformer model.</p> <div align="center" data-svelte-h="svelte-m6lh0e"><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/a5fce677dfff6da5a76534e53a7c89ea1ae19bd9/transformer-attentions.svg" alt="Highlighted attention mechanisms of a Transformer"></div> <h2 class="relative group"><a id="sparse-attention-mechanisms" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#sparse-attention-mechanisms"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Sparse Attention Mechanisms</span></h2> <p data-svelte-h="svelte-wnzxyk">Most transformer models use full attention in the sense that the attention matrix is square. It can be a big computational bottleneck when you have long sequences. Longformer and reformer are models that try to be more efficient and | |
| use a sparse version of the attention matrix to speed up training.</p> <h3 class="relative group"><a id="locality-sensitive-hashing-attention" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#locality-sensitive-hashing-attention"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Locality-Sensitive Hashing Attention</span></h3> <p data-svelte-h="svelte-16oyjk2"><a href="model_doc/reformer">Reformer</a> uses Locality-Sensitive Hashing (LSH) Attention. In the $ \text{softmax}(Q\cdot K^T)$, only the biggest elements (in the softmax | |
| dimension) of the matrix $Q\cdot K^T$ are going to give useful contributions. So for each query q in Q, we can consider only | |
| the keys $k$ in $K$ that are close to $q$. A hash function is used to determine if $q$ and $k$ are close. The attention mask is | |
| modified to mask the current token (except at the first position), because it will give a query and a key equal (so | |
| very similar to each other). Since the hash can be a bit random, several hash functions are used in practice | |
| (determined by a n_rounds parameter) and then are averaged together.</p> <h3 class="relative group"><a id="local-attention-for-efficient-sequence-processing" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#local-attention-for-efficient-sequence-processing"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Local Attention for Efficient Sequence Processing</span></h3> <p data-svelte-h="svelte-1vyuflu"><a href="model_doc/longformer">Longformer</a> uses local attention: often, the local context (e.g., what are the two tokens to the left and right?) is enough to take action for a given token. Also, by stacking attention layers that have a small window, the last layer will have a receptive field of more than just the tokens in the window, allowing them to build a representation of the whole sentence.</p> <p data-svelte-h="svelte-1x90fhu">Some preselected input tokens are also given global attention: for those few tokens, the attention matrix can access | |
| all tokens and this process is symmetric: all other tokens have access to those specific tokens (on top of the ones in | |
| their local window). This is shown in Figure 2d of the paper, see below for a sample attention mask:</p> <div align="center" data-svelte-h="svelte-1yd4xve"><img scale="50 %" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/local_attention_mask.png"></div> <p data-svelte-h="svelte-11i8g0o">Using those attention matrices with less parameters then allows the model to have inputs having a bigger sequence | |
| length.</p> <h4 class="relative group"><a id="other-tricks" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#other-tricks"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Other tricks</span></h4> <h5 class="relative group"><a id="axial-positional-encodings" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#axial-positional-encodings"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Axial positional encodings</span></h5> <p data-svelte-h="svelte-1l6o2az"><a href="model_doc/reformer">Reformer</a> uses axial positional encodings: in traditional transformer models, the positional encoding | |
| $E$ is a matrix of size $l$ by $d$, where $l$ is the sequence length and $d$ is the dimension of the | |
| hidden state. If you have very long texts, this matrix can be huge and take way too much space on the GPU. To alleviate | |
| that, axial positional encodings consist of factorizing that big matrix $E$ into two smaller matrices $E_1$ and $E_2$, with | |
| dimensions $l_1 \times d_1$ and $l_2 \times d_2$, such that $l_1 \times l_2 = l$ and | |
| $d_1 + d_2 = d$ (with the product for the lengths, this ends up being way smaller). The embedding for time | |
| step $j$ in $E$ is obtained by concatenating the embeddings for timestep $j\mod l_1$ in $E_1$ and $j // l_1$ | |
| in $E_2$.</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/transformers/blob/main/docs/source/en/attention.md" target="_blank"><span data-svelte-h="svelte-1kd6by1"><</span> <span data-svelte-h="svelte-x0xyl0">></span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p> | |
| <script> | |
| { | |
| __sveltekit_si9ik5 = { | |
| assets: "/docs/transformers/pr_36597/en", | |
| base: "/docs/transformers/pr_36597/en", | |
| env: {} | |
| }; | |
| const element = document.currentScript.parentElement; | |
| const data = [null,null]; | |
| Promise.all([ | |
| import("/docs/transformers/pr_36597/en/_app/immutable/entry/start.b2a3acf6.js"), | |
| import("/docs/transformers/pr_36597/en/_app/immutable/entry/app.ad1361e9.js") | |
| ]).then(([kit, app]) => { | |
| kit.start(app, element, { | |
| node_ids: [0, 6], | |
| data, | |
| form: null, | |
| error: null | |
| }); | |
| }); | |
| } | |
| </script> | |
Xet Storage Details
- Size:
- 37.8 kB
- Xet hash:
- 097f742d485dc5a6413efbd5825135a44abda85095c920e317ed757c1723981f
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.