eval/eval_results.csv

question,answer,contexts,ground_truth,faithfulness,answer_relevancy,context_precision,context_recall
What is the computational complexity of self-attention with respect to sequence length?,"The computational complexity of self-attention with respect to sequence length is O(n^2*d), where n is the sequence length and d is the dimensionality of the input representations. [Page 477]","['Attention Mechanisms and Transformers\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n11.6.1 Self-Attention\nGiven a sequence of input tokens x1, . . . , x𝑛where any x𝑖∈R𝑑(1 ≤𝑖≤𝑛), its selfattention outputs a sequence of the same length y1, . . . , y𝑛, where\ny𝑖= 𝑓(x𝑖, (x1, x1), . . . , (x𝑛, x𝑛)) ∈R𝑑\n(11.6.1)\naccording to the definition of attention pooling in (11.1.1). Using multi-head attention,\nthe following code snippet computes the self-attention of a tensor with shape (batch size,\nnumber of time steps or sequence length in tokens, 𝑑). The output tensor has the same\nshape.\nnum_hiddens, num_heads = 100, 5\nattention = d2l.MultiHeadAttention(num_hiddens, num_heads, 0.5)\nbatch_size, num_queries, valid_lens = 2, 4, torch.tensor([3, 2])\nX = torch.ones((batch_size, num_queries, num_hiddens))\nd2l.check_shape(attention(X, X, X, valid_lens),\n(batch_size, num_queries, num_hiddens))\n11.6.2 Comparing CNNs, RNNs, and Self-Attention\nLet’s compare architectures for mapping a sequence of 𝑛tokens to another one of equal\nlength, where each input or output token is represented by a 𝑑-dimensional vector. Specifically, we will consider CNNs, RNNs, and self-attention. We will compare their computational complexity, sequential operations, and maximum path lengths. Note that sequential\noperations prevent parallel computation, while a shorter path between any combination of\nsequence positions makes it easier to learn long-range dependencies within the sequence\n(Hochreiter et al., 2001).\nLet’s regard any text sequence as a “one-dimensional image”. Similarly, one-dimensional'
 '(Hochreiter et al., 2001).\nLet’s regard any text sequence as a “one-dimensional image”. Similarly, one-dimensional\nCNNs can process local features such as 𝑛-grams in text. Given a sequence of length 𝑛, consider a convolutional layer whose kernel size is 𝑘, and whose numbers of input and output\nchannels are both 𝑑. The computational complexity of the convolutional layer is O(𝑘𝑛𝑑2).\nAs Fig. 11.6.1 shows, CNNs are hierarchical, so there are O(1) sequential operations and\nthe maximum path length is O(𝑛/𝑘). For example, x1 and x5 are within the receptive field\nof a two-layer CNN with kernel size 3 in Fig. 11.6.1.\nWhen updating the hidden state of RNNs, multiplication of the 𝑑×𝑑weight matrix and the\n𝑑-dimensional hidden state has a computational complexity of O(𝑑2). Since the sequence\nlength is 𝑛, the computational complexity of the recurrent layer is O(𝑛𝑑2). According\nto Fig. 11.6.1, there are O(𝑛) sequential operations that cannot be parallelized and the\nmaximum path length is also O(𝑛).'
 'Self-Attention and Positional Encoding\nt\nFig. 11.6.1\nComparing CNN (padding tokens are omitted), RNN, and self-attention architectures.\nIn self-attention, the queries, keys, and values are all 𝑛× 𝑑matrices. Consider the scaled\ndot product attention in (11.3.6), where an 𝑛×𝑑matrix is multiplied by a 𝑑×𝑛matrix, then\nthe output 𝑛×𝑛matrix is multiplied by an 𝑛× 𝑑matrix. As a result, the self-attention has a\nO(𝑛2𝑑) computational complexity. As we can see from Fig. 11.6.1, each token is directly\nconnected to any other token via self-attention. Therefore, computation can be parallel with\nO(1) sequential operations and the maximum path length is also O(1).\nAll in all, both CNNs and self-attention enjoy parallel computation and self-attention has\nthe shortest maximum path length. However, the quadratic computational complexity with\nrespect to the sequence length makes self-attention prohibitively slow for very long sequences.\n11.6.3 Positional Encoding\nUnlike RNNs, which recurrently process tokens of a sequence one-by-one, self-attention\nditches sequential operations in favor of parallel computation. Note that self-attention by\nitself does not preserve the order of the sequence. What do we do if it really matters that\nthe model knows in which order the input sequence arrived?\nThe dominant approach for preserving information about the order of tokens is to represent\nthis to the model as an additional input associated with each token. These inputs are called\npositional encodings, and they can either be learned or fixed a priori. We now describe a\nsimple scheme for fixed positional encodings based on sine and cosine functions (Vaswani\net al., 2017).\nSuppose that the input representation X ∈R𝑛×𝑑contains the 𝑑-dimensional embeddings'
 'embeddings (Huang et al., 2018, Shaw et al., 2018).\nDiscussions161.\n11.7 The Transformer Architecture\nWe have compared CNNs, RNNs, and self-attention in Section 11.6.2. Notably, selfattention enjoys both parallel computation and the shortest maximum path length. Therefore, it is appealing to design deep architectures by using self-attention. Unlike earlier\nself-attention models that still rely on RNNs for input representations (Cheng et al., 2016,'
 ""Attention Mechanisms and Transformers\nBecause of the autoregressive property of the decoder self-attention, no query attends to\nkey–value pairs after the query position.\nd2l.show_heatmaps(\ndec_self_attention_weights[:, :, :, :],\nxlabel='Key positions', ylabel='Query positions',\ntitles=['Head %d' % i for i in range(1, 5)], figsize=(7, 3.5))\nSimilar to the case in the encoder self-attention, via the specified valid length of the input\nsequence, no query from the output sequence attends to those padding tokens from the input\nsequence.\nd2l.show_heatmaps(\ndec_inter_attention_weights, xlabel='Key positions',\nylabel='Query positions', titles=['Head %d' % i for i in range(1, 5)],\nfigsize=(7, 3.5))\nAlthough the Transformer architecture was originally proposed for sequence-to-sequence\nlearning, as we will discover later in the book, either the Transformer encoder or the Transformer decoder is often individually used for different deep learning tasks.\n11.7.7 Summary""]","Self-attention has a computational complexity of O(n²d), where n is the sequence length and d is the dimensionality of the token representations. This quadratic complexity with respect to sequence length makes self-attention prohibitively slow for very long sequences.",1.0,,0.8041666666465626,1.0
How many attention heads does the MultiHeadAttention implementation in the book use in its toy example test?,The MultiHeadAttention implementation in the book uses 5 attention heads in its toy example test. [Page 475],"['Attention Mechanisms and Transformers\nvalues V ∈R𝑚×𝑣thus can be written as\nsoftmax\n\x12QK⊤\n√\n𝑑\n\x13\nV ∈R𝑛×𝑣.\n(11.3.6)\nNote that when applying this to a minibatch, we need the batch matrix multiplication introduced in (11.3.5). In the following implementation of the scaled dot product attention, we\nuse dropout for model regularization.\nclass DotProductAttention(nn.Module):\n#@save\n""""""Scaled dot product attention.""""""\ndef __init__(self, dropout):\nsuper().__init__()\nself.dropout = nn.Dropout(dropout)\n# Shape of queries: (batch_size, no. of queries, d)\n# Shape of keys: (batch_size, no. of key-value pairs, d)\n# Shape of values: (batch_size, no. of key-value pairs, value dimension)\n# Shape of valid_lens: (batch_size,) or (batch_size, no. of queries)\ndef forward(self, queries, keys, values, valid_lens=None):\nd = queries.shape[-1]\n# Swap the last two dimensions of keys with keys.transpose(1, 2)\nscores = torch.bmm(queries, keys.transpose(1, 2)) / math.sqrt(d)\nself.attention_weights = masked_softmax(scores, valid_lens)\nreturn torch.bmm(self.dropout(self.attention_weights), values)\nTo illustrate how the DotProductAttention class works, we use the same keys, values,\nand valid lengths from the earlier toy example for additive attention. For the purpose of\nour example we assume that we have a minibatch size of 2, a total of 10 keys and values,\nand that the dimensionality of the values is 4. Lastly, we assume that the valid length per\nobservation is 2 and 6 respectively. Given that, we expect the output to be a 2×1×4 tensor,\ni.e., one row per example of the minibatch.\nqueries = torch.normal(0, 1, (2, 1, 2))\nkeys = torch.normal(0, 1, (2, 10, 2))\nvalues = torch.normal(0, 1, (2, 10, 4))\nvalid_lens = torch.tensor([2, 6])'
 '# no. of queries, no. of key-value pairs)\nscores = self.w_v(features).squeeze(-1)\nself.attention_weights = masked_softmax(scores, valid_lens)\n# Shape of values: (batch_size, no. of key-value pairs, value\n# dimension)\nreturn torch.bmm(self.dropout(self.attention_weights), values)\nLet’s see how AdditiveAttention works. In our toy example we pick queries, keys and\nvalues of size (2, 1, 20), (2, 10, 2) and (2, 10, 4), respectively. This is identical to our choice\nfor DotProductAttention, except that now the queries are 20-dimensional. Likewise, we\npick (2, 6) as the valid lengths for the sequences in the minibatch.\nqueries = torch.normal(0, 1, (2, 1, 20))\n(continues on next page)'
 'Attention Mechanisms and Transformers\n\nsequence. This is achieved by treating the state (context variable) as an output of additive\nattention pooling. In the RNN encoder–decoder, the Bahdanau attention mechanism treats\nthe decoder hidden state at the previous time step as the query, and the encoder hidden\nstates at all the time steps as both the keys and values.\n11.4.5 Exercises\n1. Replace GRU with LSTM in the experiment.\n2. Modify the experiment to replace the additive attention scoring function with the scaled\ndot-product. How does it influence the training efficiency?\nDiscussions159.\n11.5 Multi-Head Attention\nIn practice, given the same set of queries, keys, and values we may want our model to\ncombine knowledge from different behaviors of the same attention mechanism, such as\ncapturing dependencies of various ranges (e.g., shorter-range vs. longer-range) within a sequence. Thus, it may be beneficial to allow our attention mechanism to jointly use different\nrepresentation subspaces of queries, keys, and values.\nTo this end, instead of performing a single attention pooling, queries, keys, and values can\nbe transformed with ℎindependently learned linear projections. Then these ℎprojected\nqueries, keys, and values are fed into attention pooling in parallel. In the end, ℎattentionpooling outputs are concatenated and transformed with another learned linear projection to\nproduce the final output. This design is called multi-head attention, where each of the ℎ\nattention pooling outputs is a head (Vaswani et al., 2017). Using fully connected layers to\nperform learnable linear transformations, Fig. 11.5.1 describes multi-head attention.\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l'
 'Multi-Head Attention\nt\nFig. 11.5.1\nMulti-head attention, where multiple heads are concatenated then linearly transformed.\n11.5.1 Model\nBefore providing the implementation of multi-head attention, let’s formalize this model\nmathematically. Given a query q ∈R𝑑𝑞, a key k ∈R𝑑𝑘, and a value v ∈R𝑑𝑣, each\nattention head h𝑖(𝑖= 1, . . . , ℎ) is computed as\nh𝑖= 𝑓(W(𝑞)\n𝑖\nq, W(𝑘)\n𝑖\nk, W(𝑣)\n𝑖\nv) ∈R𝑝𝑣,\n(11.5.1)\nwhere W(𝑞)\n𝑖\n∈R𝑝𝑞×𝑑𝑞, W(𝑘)\n𝑖\n∈R𝑝𝑘×𝑑𝑘, and W(𝑣)\n𝑖\n∈R𝑝𝑣×𝑑𝑣are learnable parameters\nand 𝑓is attention pooling, such as additive attention and scaled dot product attention in\nSection 11.3. The multi-head attention output is another linear transformation via learnable\nparameters W𝑜∈R𝑝𝑜×ℎ𝑝𝑣of the concatenation of ℎheads:\nW𝑜\n\uf8ee\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8f0\nh1\n.\n.\n.\nhℎ\n\uf8f9\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fb\n∈R𝑝𝑜.\n(11.5.2)\nBased on this design, each head may attend to different parts of the input. More sophisticated functions than the simple weighted average can be expressed.\n11.5.2 Implementation\nIn our implementation, we choose the scaled dot product attention for each head of the\nmulti-head attention. To avoid significant growth of computational cost and parametrization cost, we set 𝑝𝑞= 𝑝𝑘= 𝑝𝑣= 𝑝𝑜/ℎ. Note that ℎheads can be computed in parallel\nif we set the number of outputs of linear transformations for the query, key, and value to\n𝑝𝑞ℎ= 𝑝𝑘ℎ= 𝑝𝑣ℎ= 𝑝𝑜. In the following implementation, 𝑝𝑜is specified via the argument\nnum_hiddens.\nclass MultiHeadAttention(d2l.Module):\n#@save\n""""""Multi-head attention.""""""\ndef __init__(self, num_hiddens, num_heads, dropout, bias=False, **kwargs):\nsuper().__init__()\nself.num_heads = num_heads\nself.attention = d2l.DotProductAttention(dropout)\nself.W_q = nn.LazyLinear(num_hiddens, bias=bias)\nself.W_k = nn.LazyLinear(num_hiddens, bias=bias)'
 'and values are the same.\nAs a result, the shape of the multi-head attention output is\n(batch_size, num_queries, num_hiddens).\nnum_hiddens, num_heads = 100, 5\nattention = MultiHeadAttention(num_hiddens, num_heads, 0.5)\nbatch_size, num_queries, num_kvpairs = 2, 4, 6\nvalid_lens = torch.tensor([3, 2])\nX = torch.ones((batch_size, num_queries, num_hiddens))\nY = torch.ones((batch_size, num_kvpairs, num_hiddens))\nd2l.check_shape(attention(X, Y, Y, valid_lens),\n(batch_size, num_queries, num_hiddens))\n11.5.3 Summary\nMulti-head attention combines knowledge of the same attention pooling via different representation subspaces of queries, keys, and values. To compute multiple heads of multi-head\nattention in parallel, proper tensor manipulation is needed.\n11.5.4 Exercises\n1. Visualize attention weights of multiple heads in this experiment.\n2. Suppose that we have a trained model based on multi-head attention and we want to\nprune less important attention heads to increase the prediction speed. How can we design experiments to measure the importance of an attention head?\nDiscussions160.\n11.6 Self-Attention and Positional Encoding\nIn deep learning, we often use CNNs or RNNs to encode sequences. Now with attention\nmechanisms in mind, imagine feeding a sequence of tokens into an attention mechanism\nsuch that at every step, each token has its own query, keys, and values. Here, when computing the value of a token’s representation at the next layer, the token can attend (via its query\nvector) to any other’s token (matching based on their key vectors). Using the full set of']","The toy example test uses 5 attention heads, with num_hiddens set to 100 and num_heads set to 5.",,,0.99999999998,
What are the two pretraining tasks used in BERT?,The two pretraining tasks used in BERT are masked language modeling and next sentence prediction. [Page 772],"['the art for natural language tasks such as single text classification, text pair classification or\nregression, text tagging, and question answering.\nYou may note that these downstream tasks include text pair understanding. BERT pretraining has another loss for predicting whether one sentence immediately follows the other.\nHowever, this loss was later found to be less useful when pretraining RoBERTa, a BERT\nvariant of the same size, on 2000 billion tokens (Liu et al., 2019). Other derivatives of'
 'language processing tasks (Devlin et al., 2018). Using a pretrained Transformer encoder,\nBERT is able to represent any token based on its bidirectional context. During supervised'
 'Natural Language Processing: Pretraining\nlearning of downstream tasks, BERT is similar to GPT in two aspects. First, BERT representations will be fed into an added output layer, with minimal changes to the model\narchitecture depending on nature of tasks, such as predicting for every token vs. predicting\nfor the entire sequence. Second, all the parameters of the pretrained Transformer encoder\nare fine-tuned, while the additional output layer will be trained from scratch. Fig. 15.8.1\ndepicts the differences among ELMo, GPT, and BERT.\nt\nFig. 15.8.1\nA comparison of ELMo, GPT, and BERT.\nBERT further improved the state of the art on eleven natural language processing tasks\nunder broad categories of (i) single text classification (e.g., sentiment analysis), (ii) text pair\nclassification (e.g., natural language inference), (iii) question answering, (iv) text tagging\n(e.g., named entity recognition). All proposed in 2018, from context-sensitive ELMo to\ntask-agnostic GPT and BERT, conceptually simple yet empirically powerful pretraining of\ndeep representations for natural languages have revolutionized solutions to various natural\nlanguage processing tasks.\nIn the rest of this chapter, we will dive into the pretraining of BERT. When natural language\nprocessing applications are explained in Chapter 16, we will illustrate fine-tuning of BERT\nfor downstream applications.\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n15.8.4 Input Representation\nIn natural language processing, some tasks (e.g., sentiment analysis) take single text as\ninput, while in some other tasks (e.g., natural language inference), the input is a pair of\ntext sequences. The BERT input sequence unambiguously represents both single text and'
 'model task of BERT pretraining. The prediction uses a one-hidden-layer MLP (self.mlp).\nIn forward inference, it takes two inputs: the encoded result of BERTEncoder and the token\npositions for prediction. The output is the prediction results at these positions.\n#@save\nclass MaskLM(nn.Module):\n""""""The masked language model task of BERT.""""""\ndef __init__(self, vocab_size, num_hiddens, **kwargs):\nsuper(MaskLM, self).__init__(**kwargs)\nself.mlp = nn.Sequential(nn.LazyLinear(num_hiddens),\nnn.ReLU(),\nnn.LayerNorm(num_hiddens),\nnn.LazyLinear(vocab_size))\ndef forward(self, X, pred_positions):\nnum_pred_positions = pred_positions.shape[1]\npred_positions = pred_positions.reshape(-1)\nbatch_size = X.shape[0]\nbatch_idx = torch.arange(0, batch_size)\n# Suppose that `batch_size` = 2, `num_pred_positions` = 3, then\n# `batch_idx` is `torch.tensor([0, 0, 0, 1, 1, 1])`\nbatch_idx = torch.repeat_interleave(batch_idx, num_pred_positions)\nmasked_X = X[batch_idx, pred_positions]\n(continues on next page)'
 '• BERT combines the best of both worlds: it encodes context bidirectionally and requires\nminimal architecture changes for a wide range of natural language processing tasks.\n• The embeddings of the BERT input sequence are the sum of the token embeddings,\nsegment embeddings, and positional embeddings.\n• Pretraining BERT is composed of two tasks: masked language modeling and next sentence prediction. The former is able to encode bidirectional context for representing\nwords, while the latter explicitly models the logical relationship between text pairs.']",BERT is pretrained on two tasks: masked language modeling and next sentence prediction.,1.0,,0.8041666666465626,0.6
What training configuration does the book use for the sequence to sequence model with Bahdanau attention?,I could not find the answer to that question in the uploaded document.,"['Attention Mechanisms and Transformers\nsequence-to-sequence applications, such as machine translations (Bahdanau et al., 2014).\nYou might recall that in the first sequence-to-sequence models for machine translation\n(Sutskever et al., 2014), the entire input was compressed by the encoder into a single fixedlength vector to be fed into the decoder. The intuition behind attention is that rather than\ncompressing the input, it might be better for the decoder to revisit the input sequence at\nevery step. Moreover, rather than always seeing the same representation of the input, one\nmight imagine that the decoder should selectively focus on particular parts of the input sequence at particular decoding steps. Bahdanau’s attention mechanism provided a simple\nmeans by which the decoder could dynamically attend to different parts of the input at each\ndecoding step. The high-level idea is that the encoder could produce a representation of\nlength equal to the original input sequence. Then, at decoding time, the decoder can (via\nsome control mechanism) receive as input a context vector consisting of a weighted sum\nof the representations on the input at each time step. Intuitively, the weights determine the\nextent to which each step’s context “focuses” on each input token, and the key is to make\nthis process for assigning the weights differentiable so that it can be learned along with all\nof the other neural network parameters.\nInitially, the idea was a remarkably successful enhancement to the recurrent neural networks that already dominated machine translation applications. The models performed\nbetter than the original encoder–decoder sequence-to-sequence architectures. Furthermore,'
 'The Bahdanau Attention Mechanism\n11.4 The Bahdanau Attention Mechanism\nWhen we encountered machine translation in Section 10.7, we designed an encoder–decoder\narchitecture for sequence-to-sequence learning based on two RNNs (Sutskever et al., 2014).\nSpecifically, the RNN encoder transforms a variable-length sequence into a fixed-shape\ncontext variable. Then, the RNN decoder generates the output (target) sequence token by\ntoken based on the generated tokens and the context variable.\nRecall Fig. 10.7.2 which we repeat (Fig. 11.4.1) with some additional detail. Conventionally, in an RNN all relevant information about a source sequence is translated into some\ninternal fixed-dimensional state representation by the encoder. It is this very state that is\nused by the decoder as the complete and exclusive source of information for generating the\ntranslated sequence. In other words, the sequence-to-sequence mechanism treats the intermediate state as a sufficient statistic of whatever string might have served as input.\nt\nFig. 11.4.1\nSequence-to-sequence model. The state, as generated by the encoder, is the only piece of\ninformation shared between the encoder and the decoder.\nWhile this is quite reasonable for short sequences, it is clear that it is infeasible for long ones,\nsuch as a book chapter or even just a very long sentence. After all, before too long there will\nsimply not be enough “space” in the intermediate representation to store all that is important\nin the source sequence. Consequently the decoder will fail to translate long and complex\nsentences. One of the first to encounter this was Graves (2013) who tried to design an\nRNN to generate handwritten text. Since the source text has arbitrary length they designed a'
 'RNN to generate handwritten text. Since the source text has arbitrary length they designed a\ndifferentiable attention model to align text characters with the much longer pen trace, where\nthe alignment moves only in one direction. This, in turn, draws on decoding algorithms in\nspeech recognition, e.g., hidden Markov models (Rabiner and Juang, 1993).\nInspired by the idea of learning to align, Bahdanau et al. (2014) proposed a differentiable\nattention model without the unidirectional alignment limitation. When predicting a token,\nif not all the input tokens are relevant, the model aligns (or attends) only to parts of the input\nsequence that are deemed relevant to the current prediction. This is then used to update the\ncurrent state before generating the next token. While quite innocuous in its description, this\nBahdanau attention mechanism has arguably turned into one of the most influential ideas\nof the past decade in deep learning, giving rise to Transformers (Vaswani et al., 2017) and\nmany related new architectures.'
 'The Bahdanau Attention Mechanism\nengs = [\'go .\', \'i lost .\', \'he\\\'s calm .\', \'i\\\'m home .\']\nfras = [\'va !\', \'j\\\'ai perdu .\', \'il est calme .\', \'je suis chez moi .\']\npreds, _ = model.predict_step(\ndata.build(engs, fras), d2l.try_gpu(), data.num_steps)\nfor en, fr, p in zip(engs, fras, preds):\ntranslation = []\nfor token in data.tgt_vocab.to_tokens(p):\nif token == \'<eos>\':\nbreak\ntranslation.append(token)\nprint(f\'{en} => {translation}, bleu,\'\nf\'{d2l.bleu("" "".join(translation), fr, k=2):.3f}\')\ngo . => [\'va\', \'!\'], bleu,1.000\ni lost . => [""j\'ai"", \'perdu\', \'.\'], bleu,1.000\nhe\'s calm . => [\'il\', \'court\', \'.\'], bleu,0.000\ni\'m home . => [\'je\', \'suis\', \'chez\', \'moi\', \'.\'], bleu,1.000\nLet’s visualize the attention weights when translating the last English sentence. We see that\neach query assigns non-uniform weights over key–value pairs. It shows that at each decoding step, different parts of the input sequences are selectively aggregated in the attention\npooling.\n_, dec_attention_weights = model.predict_step(\ndata.build([engs[-1]], [fras[-1]]), d2l.try_gpu(), data.num_steps, True)\nattention_weights = torch.cat(\n[step[0][0][0] for step in dec_attention_weights], 0)\nattention_weights = attention_weights.reshape((1, 1, -1, data.num_steps))\n# Plus one to include the end-of-sequence token\nd2l.show_heatmaps(\nattention_weights[:, :, :, :len(engs[-1].split()) + 1].cpu(),\nxlabel=\'Key positions\', ylabel=\'Query positions\')\n11.4.4 Summary\nWhen predicting a token, if not all the input tokens are relevant, the RNN encoder–decoder\nwith the Bahdanau attention mechanism selectively aggregates different parts of the input'
 'Attention Mechanisms and Transformers\n\nsequence. This is achieved by treating the state (context variable) as an output of additive\nattention pooling. In the RNN encoder–decoder, the Bahdanau attention mechanism treats\nthe decoder hidden state at the previous time step as the query, and the encoder hidden\nstates at all the time steps as both the keys and values.\n11.4.5 Exercises\n1. Replace GRU with LSTM in the experiment.\n2. Modify the experiment to replace the additive attention scoring function with the scaled\ndot-product. How does it influence the training efficiency?\nDiscussions159.\n11.5 Multi-Head Attention\nIn practice, given the same set of queries, keys, and values we may want our model to\ncombine knowledge from different behaviors of the same attention mechanism, such as\ncapturing dependencies of various ranges (e.g., shorter-range vs. longer-range) within a sequence. Thus, it may be beneficial to allow our attention mechanism to jointly use different\nrepresentation subspaces of queries, keys, and values.\nTo this end, instead of performing a single attention pooling, queries, keys, and values can\nbe transformed with ℎindependently learned linear projections. Then these ℎprojected\nqueries, keys, and values are fed into attention pooling in parallel. In the end, ℎattentionpooling outputs are concatenated and transformed with another learned linear projection to\nproduce the final output. This design is called multi-head attention, where each of the ℎ\nattention pooling outputs is a head (Vaswani et al., 2017). Using fully connected layers to\nperform learnable linear transformations, Fig. 11.5.1 describes multi-head attention.\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l']","The book trains the model with a learning rate of 0.005, gradient clipping at 1, for 30 epochs on 1 GPU, using a batch size of 128.",1.0,,0.4499999999775,0.0
What are the steps in the Transformer encoder block?,"The Transformer encoder block consists of two sublayers: self-attention and positionwise feed-forward networks, with a residual connection and layer normalization around them. [Page 485] 

These sublayers are implemented in the TransformerEncoderBlock class, which is added to the TransformerEncoder class in a loop to create multiple identical layers. [Page 485] 

The self-attention sublayer is an instance of the MultiHeadAttention class, and the positionwise feed-forward network is an instance of the PositionWiseFFN class. [Page 486]","['step as the current input to the decoder.\nIn the following, we explain the design depicted in Fig. 10.7.1 in greater detail. We will\ntrain this model for machine translation on the English–French dataset as introduced in\nSection 10.5.\n10.7.2 Encoder\nRecall that the encoder transforms an input sequence of variable length into a fixed-shape\ncontext variable c (see Fig. 10.7.1).\nConsider a single sequence example (batch size 1). Suppose the input sequence is 𝑥1, . . . , 𝑥𝑇,\nsuch that 𝑥𝑡is the 𝑡th token. At time step 𝑡, the RNN transforms the input feature vector x𝑡'
 'The Transformer Architecture\nwhose values are always between −1 and 1, we multiply values of the learnable input embeddings by the square root of the embedding dimension to rescale before summing up the\ninput embedding and the positional encoding.\nclass TransformerEncoder(d2l.Encoder):\n#@save\n""""""The Transformer encoder.""""""\ndef __init__(self, vocab_size, num_hiddens, ffn_num_hiddens,\nnum_heads, num_blks, dropout, use_bias=False):\nsuper().__init__()\nself.num_hiddens = num_hiddens\nself.embedding = nn.Embedding(vocab_size, num_hiddens)\nself.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout)\nself.blks = nn.Sequential()\nfor i in range(num_blks):\nself.blks.add_module(""block""+str(i), TransformerEncoderBlock(\nnum_hiddens, ffn_num_hiddens, num_heads, dropout, use_bias))\ndef forward(self, X, valid_lens):\n# Since positional encoding values are between -1 and 1, the embedding\n# values are multiplied by the square root of the embedding dimension\n# to rescale before they are summed up\nX = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))\nself.attention_weights = [None] * len(self.blks)\nfor i, blk in enumerate(self.blks):\nX = blk(X, valid_lens)\nself.attention_weights[\ni] = blk.attention.attention.attention_weights\nreturn X\nBelow we specify hyperparameters to create a two-layer Transformer encoder. The shape of\nthe Transformer encoder output is (batch size, number of time steps, num_hiddens).\nencoder = TransformerEncoder(200, 24, 48, 8, 2, 0.5)\nd2l.check_shape(encoder(torch.ones((2, 100), dtype=torch.long), valid_lens),\n(2, 100, 24))\n11.7.5 Decoder'
 'd2l.check_shape(encoder(torch.ones((2, 100), dtype=torch.long), valid_lens),\n(2, 100, 24))\n11.7.5 Decoder\nAs shown in Fig. 11.7.1, the Transformer decoder is composed of multiple identical layers. Each layer is implemented in the following TransformerDecoderBlock class, which\ncontains three sublayers: decoder self-attention, encoder–decoder attention, and positionwise feed-forward networks. These sublayers employ a residual connection around them\nfollowed by layer normalization.\nAs we described earlier in this section, in the masked multi-head decoder self-attention\n(the first sublayer), queries, keys, and values all come from the outputs of the previous\ndecoder layer. When training sequence-to-sequence models, tokens at all the positions\n(time steps) of the output sequence are known. However, during prediction the output\nsequence is generated token by token; thus, at any decoder time step only the generated\ntokens can be used in the decoder self-attention. To preserve autoregression in the decoder,'
 'Attention Mechanisms and Transformers\nits masked self-attention specifies dec_valid_lens so that any query only attends to all\npositions in the decoder up to the query position.\nclass TransformerDecoderBlock(nn.Module):\n# The i-th block in the Transformer decoder\ndef __init__(self, num_hiddens, ffn_num_hiddens, num_heads, dropout, i):\nsuper().__init__()\nself.i = i\nself.attention1 = d2l.MultiHeadAttention(num_hiddens, num_heads,\ndropout)\nself.addnorm1 = AddNorm(num_hiddens, dropout)\nself.attention2 = d2l.MultiHeadAttention(num_hiddens, num_heads,\ndropout)\nself.addnorm2 = AddNorm(num_hiddens, dropout)\nself.ffn = PositionWiseFFN(ffn_num_hiddens, num_hiddens)\nself.addnorm3 = AddNorm(num_hiddens, dropout)\ndef forward(self, X, state):\nenc_outputs, enc_valid_lens = state[0], state[1]\n# During training, all the tokens of any output sequence are processed\n# at the same time, so state[2][self.i] is None as initialized. When\n# decoding any output sequence token by token during prediction,\n# state[2][self.i] contains representations of the decoded output at\n# the i-th block up to the current time step\nif state[2][self.i] is None:\nkey_values = X\nelse:\nkey_values = torch.cat((state[2][self.i], X), dim=1)\nstate[2][self.i] = key_values\nif self.training:\nbatch_size, num_steps, _ = X.shape\n# Shape of dec_valid_lens: (batch_size, num_steps), where every\n# row is [1, 2, ..., num_steps]\ndec_valid_lens = torch.arange(\n1, num_steps + 1, device=X.device).repeat(batch_size, 1)\nelse:\ndec_valid_lens = None\n# Self-attention\nX2 = self.attention1(X, key_values, key_values, dec_valid_lens)\nY = self.addnorm1(X, X2)\n# Encoder-decoder attention. Shape of enc_outputs:\n# (batch_size, num_steps, num_hiddens)'
 ""The Transformer Architecture\n(continued from previous page)\nd2l.check_shape(enc_attention_weights,\n(num_blks, num_heads, data.num_steps, data.num_steps))\nIn the encoder self-attention, both queries and keys come from the same input sequence.\nSince padding tokens do not carry meaning, with specified valid length of the input sequence no query attends to positions of padding tokens. In the following, two layers of\nmulti-head attention weights are presented row by row. Each head independently attends\nbased on a separate representation subspace of queries, keys, and values.\nd2l.show_heatmaps(\nenc_attention_weights.cpu(), xlabel='Key positions',\nylabel='Query positions', titles=['Head %d' % i for i in range(1, 5)],\nfigsize=(7, 3.5))\nTo visualize the decoder self-attention weights and the encoder–decoder attention weights,\nwe need more data manipulations. For example, we fill the masked attention weights\nwith zero. Note that the decoder self-attention weights and the encoder–decoder attention weights both have the same queries: the beginning-of-sequence token followed by the\noutput tokens and possibly end-of-sequence tokens.\ndec_attention_weights_2d = [head[0].tolist()\nfor step in dec_attention_weights\nfor attn in step for blk in attn for head in blk]\ndec_attention_weights_filled = torch.tensor(\npd.DataFrame(dec_attention_weights_2d).fillna(0.0).values)\nshape = (-1, 2, num_blks, num_heads, data.num_steps)\ndec_attention_weights = dec_attention_weights_filled.reshape(shape)\ndec_self_attention_weights, dec_inter_attention_weights = \\\ndec_attention_weights.permute(1, 2, 3, 0, 4)\nd2l.check_shape(dec_self_attention_weights,\n(num_blks, num_heads, data.num_steps, data.num_steps))\nd2l.check_shape(dec_inter_attention_weights,""]","The Transformer encoder block performs the following steps: first, multi-head self-attention is applied to the input, then a residual connection is added and layer normalization is applied. Second, the result passes through a positionwise feed-forward network, followed by another residual connection and layer normalization. The residual connection requires that the two inputs have the same shape so the output tensor has the same shape after addition.",,,0.99999999998,0.6666666666666666
How does the masked softmax operation handle sequences of different lengths in a minibatch?,"The masked softmax operation handles sequences of different lengths in a minibatch by setting the values of the invalid sequences to a very large negative number, such as -1e6, which makes their contribution to gradients and values vanish in practice, effectively masking them. 

[Page 462] [Page 463]","['(11.1.3) by using the softmax operation:\n𝛼(q, k𝑖) = softmax(𝑎(q, k𝑖)) =\nexp(q⊤k𝑖/\n√\n𝑑)\nÍ\n𝑗=1 exp(q⊤k 𝑗/\n√\n𝑑)\n.\n(11.3.3)\nAs it turns out, all popular attention mechanisms use the softmax, hence we will limit\nourselves to that in the remainder of this chapter.\n11.3.2 Convenience Functions\nWe need a few functions to make the attention mechanism efficient to deploy. This includes\ntools for dealing with strings of variable lengths (common for natural language processing)\nand tools for efficient evaluation on minibatches (batch matrix multiplication).\nMasked Softmax Operation\nOne of the most popular applications of the attention mechanism is to sequence models.\nHence we need to be able to deal with sequences of different lengths. In some cases, such'
 'Attention Mechanisms and Transformers\nsequences may end up in the same minibatch, necessitating padding with dummy tokens\nfor shorter sequences (see Section 10.5 for an example). These special tokens do not carry\nmeaning. For instance, assume that we have the following three sentences:\nDive\ninto\nDeep\nLearning\nLearn to\ncode\n<blank>\nHello world <blank> <blank>\nSince we do not want blanks in our attention model we simply need to limit Í𝑛\n𝑖=1 𝛼(q, k𝑖)v𝑖\nto Í𝑙\n𝑖=1 𝛼(q, k𝑖)v𝑖for however long, 𝑙≤𝑛, the actual sentence is. Since it is such a common\nproblem, it has a name: the masked softmax operation.\nLet’s implement it. Actually, the implementation cheats ever so slightly by setting the values\nof v𝑖, for 𝑖> 𝑙, to zero. Moreover, it sets the attention weights to a large negative number,\nsuch as −106, in order to make their contribution to gradients and values vanish in practice.\nThis is done since linear algebra kernels and operators are heavily optimized for GPUs and\nit is faster to be slightly wasteful in computation rather than to have code with conditional\n(if then else) statements.\ndef masked_softmax(X, valid_lens):\n#@save\n""""""Perform softmax operation by masking elements on the last axis.""""""\n# X: 3D tensor, valid_lens: 1D or 2D tensor\ndef _sequence_mask(X, valid_len, value=0):\nmaxlen = X.size(1)\nmask = torch.arange((maxlen), dtype=torch.float32,\ndevice=X.device)[None, :] < valid_len[:, None]\nX[~mask] = value\nreturn X\nif valid_lens is None:\nreturn nn.functional.softmax(X, dim=-1)\nelse:\nshape = X.shape\nif valid_lens.dim() == 1:\nvalid_lens = torch.repeat_interleave(valid_lens, shape[1])\nelse:\nvalid_lens = valid_lens.reshape(-1)\n# On the last axis, replace masked elements with a very large negative'
 'else:\nvalid_lens = valid_lens.reshape(-1)\n# On the last axis, replace masked elements with a very large negative\n# value, whose exponentiation outputs 0\nX = _sequence_mask(X.reshape(-1, shape[-1]), valid_lens, value=-1e6)\nreturn nn.functional.softmax(X.reshape(shape), dim=-1)\nTo illustrate how this function works, consider a minibatch of two examples of size 2 × 4,\nwhere their valid lengths are 2 and 3, respectively. As a result of the masked softmax operation, values beyond the valid lengths for each pair of vectors are all masked as zero.\nmasked_softmax(torch.rand(2, 2, 4), torch.tensor([2, 3]))\ntensor([[[0.4448, 0.5552, 0.0000, 0.0000],\n[0.4032, 0.5968, 0.0000, 0.0000]],\n(continues on next page)'
 'Attention Scoring Functions\n(continued from previous page)\n[[0.2795, 0.2805, 0.4400, 0.0000],\n[0.2798, 0.3092, 0.4110, 0.0000]]])\nIf we need more fine-grained control to specify the valid length for each of the two vectors of every example, we simply use a two-dimensional tensor of valid lengths. This\nyields:\nmasked_softmax(torch.rand(2, 2, 4), torch.tensor([[1, 3], [2, 4]]))\ntensor([[[1.0000, 0.0000, 0.0000, 0.0000],\n[0.4109, 0.2794, 0.3097, 0.0000]],\n[[0.3960, 0.6040, 0.0000, 0.0000],\n[0.2557, 0.1833, 0.2420, 0.3190]]])\nBatch Matrix Multiplication\nAnother commonly used operation is to multiply batches of matrices by one another. This\ncomes in handy when we have minibatches of queries, keys, and values. More specifically,\nassume that\nQ = [Q1, Q2, . . . , Q𝑛] ∈R𝑛×𝑎×𝑏,\nK = [K1, K2, . . . , K𝑛] ∈R𝑛×𝑏×𝑐.\n(11.3.4)\nThen the batch matrix multiplication (BMM) computes the elementwise product\nBMM(Q, K) = [Q1K1, Q2K2, . . . , Q𝑛K𝑛] ∈R𝑛×𝑎×𝑐.\n(11.3.5)\nLet’s see this in action in a deep learning framework.\nQ = torch.ones((2, 3, 4))\nK = torch.ones((2, 4, 6))\nd2l.check_shape(torch.bmm(Q, K), (2, 3, 6))\n11.3.3 Scaled Dot Product Attention\nLet’s return to the dot product attention introduced in (11.3.2). In general, it requires that\nboth the query and the key have the same vector length, say 𝑑, even though this can be\naddressed easily by replacing q⊤k with q⊤Mk where M is a matrix suitably chosen for\ntranslating between both spaces. For now assume that the dimensions match.\nIn practice, we often think of minibatches for efficiency, such as computing attention for\n𝑛queries and 𝑚key-value pairs, where queries and keys are of length 𝑑and values are of\nlength 𝑣. The scaled dot product attention of queries Q ∈R𝑛×𝑑, keys K ∈R𝑚×𝑑, and'
 '# no. of queries, no. of key-value pairs)\nscores = self.w_v(features).squeeze(-1)\nself.attention_weights = masked_softmax(scores, valid_lens)\n# Shape of values: (batch_size, no. of key-value pairs, value\n# dimension)\nreturn torch.bmm(self.dropout(self.attention_weights), values)\nLet’s see how AdditiveAttention works. In our toy example we pick queries, keys and\nvalues of size (2, 1, 20), (2, 10, 2) and (2, 10, 4), respectively. This is identical to our choice\nfor DotProductAttention, except that now the queries are 20-dimensional. Likewise, we\npick (2, 6) as the valid lengths for the sequences in the minibatch.\nqueries = torch.normal(0, 1, (2, 1, 20))\n(continues on next page)']","The masked softmax operation limits the attention sum to only the valid tokens in each sequence. It sets the values of tokens beyond the valid length to zero and sets their attention weights to a very large negative number such as negative one million, so their contribution to gradients and values vanishes. This is faster than using conditional if-then-else statements because linear algebra kernels are optimized for GPUs.",1.0,,0.99999999998,
What steps does multi-head attention perform to compute its output?,"Multi-head attention performs the following steps to compute its output: 
1. It transforms queries, keys, and values with independently learned linear projections. 
2. These projected queries, keys, and values are fed into attention pooling in parallel. 
3. The attention pooling outputs are concatenated. 
4. The concatenated outputs are transformed with another learned linear projection to produce the final output.

[Page 472] [Page 473]","['Attention Mechanisms and Transformers\n\nsequence. This is achieved by treating the state (context variable) as an output of additive\nattention pooling. In the RNN encoder–decoder, the Bahdanau attention mechanism treats\nthe decoder hidden state at the previous time step as the query, and the encoder hidden\nstates at all the time steps as both the keys and values.\n11.4.5 Exercises\n1. Replace GRU with LSTM in the experiment.\n2. Modify the experiment to replace the additive attention scoring function with the scaled\ndot-product. How does it influence the training efficiency?\nDiscussions159.\n11.5 Multi-Head Attention\nIn practice, given the same set of queries, keys, and values we may want our model to\ncombine knowledge from different behaviors of the same attention mechanism, such as\ncapturing dependencies of various ranges (e.g., shorter-range vs. longer-range) within a sequence. Thus, it may be beneficial to allow our attention mechanism to jointly use different\nrepresentation subspaces of queries, keys, and values.\nTo this end, instead of performing a single attention pooling, queries, keys, and values can\nbe transformed with ℎindependently learned linear projections. Then these ℎprojected\nqueries, keys, and values are fed into attention pooling in parallel. In the end, ℎattentionpooling outputs are concatenated and transformed with another learned linear projection to\nproduce the final output. This design is called multi-head attention, where each of the ℎ\nattention pooling outputs is a head (Vaswani et al., 2017). Using fully connected layers to\nperform learnable linear transformations, Fig. 11.5.1 describes multi-head attention.\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l'
 'Multi-Head Attention\nt\nFig. 11.5.1\nMulti-head attention, where multiple heads are concatenated then linearly transformed.\n11.5.1 Model\nBefore providing the implementation of multi-head attention, let’s formalize this model\nmathematically. Given a query q ∈R𝑑𝑞, a key k ∈R𝑑𝑘, and a value v ∈R𝑑𝑣, each\nattention head h𝑖(𝑖= 1, . . . , ℎ) is computed as\nh𝑖= 𝑓(W(𝑞)\n𝑖\nq, W(𝑘)\n𝑖\nk, W(𝑣)\n𝑖\nv) ∈R𝑝𝑣,\n(11.5.1)\nwhere W(𝑞)\n𝑖\n∈R𝑝𝑞×𝑑𝑞, W(𝑘)\n𝑖\n∈R𝑝𝑘×𝑑𝑘, and W(𝑣)\n𝑖\n∈R𝑝𝑣×𝑑𝑣are learnable parameters\nand 𝑓is attention pooling, such as additive attention and scaled dot product attention in\nSection 11.3. The multi-head attention output is another linear transformation via learnable\nparameters W𝑜∈R𝑝𝑜×ℎ𝑝𝑣of the concatenation of ℎheads:\nW𝑜\n\uf8ee\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8f0\nh1\n.\n.\n.\nhℎ\n\uf8f9\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fb\n∈R𝑝𝑜.\n(11.5.2)\nBased on this design, each head may attend to different parts of the input. More sophisticated functions than the simple weighted average can be expressed.\n11.5.2 Implementation\nIn our implementation, we choose the scaled dot product attention for each head of the\nmulti-head attention. To avoid significant growth of computational cost and parametrization cost, we set 𝑝𝑞= 𝑝𝑘= 𝑝𝑣= 𝑝𝑜/ℎ. Note that ℎheads can be computed in parallel\nif we set the number of outputs of linear transformations for the query, key, and value to\n𝑝𝑞ℎ= 𝑝𝑘ℎ= 𝑝𝑣ℎ= 𝑝𝑜. In the following implementation, 𝑝𝑜is specified via the argument\nnum_hiddens.\nclass MultiHeadAttention(d2l.Module):\n#@save\n""""""Multi-head attention.""""""\ndef __init__(self, num_hiddens, num_heads, dropout, bias=False, **kwargs):\nsuper().__init__()\nself.num_heads = num_heads\nself.attention = d2l.DotProductAttention(dropout)\nself.W_q = nn.LazyLinear(num_hiddens, bias=bias)\nself.W_k = nn.LazyLinear(num_hiddens, bias=bias)'
 'and values are the same.\nAs a result, the shape of the multi-head attention output is\n(batch_size, num_queries, num_hiddens).\nnum_hiddens, num_heads = 100, 5\nattention = MultiHeadAttention(num_hiddens, num_heads, 0.5)\nbatch_size, num_queries, num_kvpairs = 2, 4, 6\nvalid_lens = torch.tensor([3, 2])\nX = torch.ones((batch_size, num_queries, num_hiddens))\nY = torch.ones((batch_size, num_kvpairs, num_hiddens))\nd2l.check_shape(attention(X, Y, Y, valid_lens),\n(batch_size, num_queries, num_hiddens))\n11.5.3 Summary\nMulti-head attention combines knowledge of the same attention pooling via different representation subspaces of queries, keys, and values. To compute multiple heads of multi-head\nattention in parallel, proper tensor manipulation is needed.\n11.5.4 Exercises\n1. Visualize attention weights of multiple heads in this experiment.\n2. Suppose that we have a trained model based on multi-head attention and we want to\nprune less important attention heads to increase the prediction speed. How can we design experiments to measure the importance of an attention head?\nDiscussions160.\n11.6 Self-Attention and Positional Encoding\nIn deep learning, we often use CNNs or RNNs to encode sequences. Now with attention\nmechanisms in mind, imagine feeding a sequence of tokens into an attention mechanism\nsuch that at every step, each token has its own query, keys, and values. Here, when computing the value of a token’s representation at the next layer, the token can attend (via its query\nvector) to any other’s token (matching based on their key vectors). Using the full set of'
 'Attention Mechanisms and Transformers\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n11.6.1 Self-Attention\nGiven a sequence of input tokens x1, . . . , x𝑛where any x𝑖∈R𝑑(1 ≤𝑖≤𝑛), its selfattention outputs a sequence of the same length y1, . . . , y𝑛, where\ny𝑖= 𝑓(x𝑖, (x1, x1), . . . , (x𝑛, x𝑛)) ∈R𝑑\n(11.6.1)\naccording to the definition of attention pooling in (11.1.1). Using multi-head attention,\nthe following code snippet computes the self-attention of a tensor with shape (batch size,\nnumber of time steps or sequence length in tokens, 𝑑). The output tensor has the same\nshape.\nnum_hiddens, num_heads = 100, 5\nattention = d2l.MultiHeadAttention(num_hiddens, num_heads, 0.5)\nbatch_size, num_queries, valid_lens = 2, 4, torch.tensor([3, 2])\nX = torch.ones((batch_size, num_queries, num_hiddens))\nd2l.check_shape(attention(X, X, X, valid_lens),\n(batch_size, num_queries, num_hiddens))\n11.6.2 Comparing CNNs, RNNs, and Self-Attention\nLet’s compare architectures for mapping a sequence of 𝑛tokens to another one of equal\nlength, where each input or output token is represented by a 𝑑-dimensional vector. Specifically, we will consider CNNs, RNNs, and self-attention. We will compare their computational complexity, sequential operations, and maximum path lengths. Note that sequential\noperations prevent parallel computation, while a shorter path between any combination of\nsequence positions makes it easier to learn long-range dependencies within the sequence\n(Hochreiter et al., 2001).\nLet’s regard any text sequence as a “one-dimensional image”. Similarly, one-dimensional'
 ""The Transformer Architecture\n(continued from previous page)\nd2l.check_shape(enc_attention_weights,\n(num_blks, num_heads, data.num_steps, data.num_steps))\nIn the encoder self-attention, both queries and keys come from the same input sequence.\nSince padding tokens do not carry meaning, with specified valid length of the input sequence no query attends to positions of padding tokens. In the following, two layers of\nmulti-head attention weights are presented row by row. Each head independently attends\nbased on a separate representation subspace of queries, keys, and values.\nd2l.show_heatmaps(\nenc_attention_weights.cpu(), xlabel='Key positions',\nylabel='Query positions', titles=['Head %d' % i for i in range(1, 5)],\nfigsize=(7, 3.5))\nTo visualize the decoder self-attention weights and the encoder–decoder attention weights,\nwe need more data manipulations. For example, we fill the masked attention weights\nwith zero. Note that the decoder self-attention weights and the encoder–decoder attention weights both have the same queries: the beginning-of-sequence token followed by the\noutput tokens and possibly end-of-sequence tokens.\ndec_attention_weights_2d = [head[0].tolist()\nfor step in dec_attention_weights\nfor attn in step for blk in attn for head in blk]\ndec_attention_weights_filled = torch.tensor(\npd.DataFrame(dec_attention_weights_2d).fillna(0.0).values)\nshape = (-1, 2, num_blks, num_heads, data.num_steps)\ndec_attention_weights = dec_attention_weights_filled.reshape(shape)\ndec_self_attention_weights, dec_inter_attention_weights = \\\ndec_attention_weights.permute(1, 2, 3, 0, 4)\nd2l.check_shape(dec_self_attention_weights,\n(num_blks, num_heads, data.num_steps, data.num_steps))\nd2l.check_shape(dec_inter_attention_weights,""]","Multi-head attention performs the following steps: queries, keys, and values are each transformed with h independently learned linear projections. These projected queries, keys, and values are then fed into attention pooling in parallel across all h heads. The h attention pooling outputs are concatenated and transformed with another learned linear projection via the output weight matrix to produce the final output.",1.0,,0.99999999998,
How is positional encoding computed and added to the input in the Transformer?,"2𝑗
(𝑖+𝛿)𝜔𝑗
cos 2𝑗
(𝑖+𝛿)𝜔𝑗

(11.6.3)
Page Number:481
In addition to being able to represent position offset linearly, this form has other nice
properties. It also enables to represent absolute position in the input sequence with a
fixed set of learned embedding parameters. The number of parameters in this positional
encoding model is O(𝑑), where 𝑑 is the input embedding size. 
Page Number:482
The absolute position information can be encoded with a set of learned embedding
parameters. These parameters can be used to represent any absolute position in the input
sequence with a fixed set of parameters. This is because the absolute position is
incorporated in the input embeddings of the tokens by adding the corresponding
positional encoding matrix P.
Page Number:483
As stated above, the positional encoding matrix P is added to the input embeddings
matrix X to get the output X + P. The absolute position of each token is incorporated
into its input embedding by adding its corresponding positional encoding vector to it.
This is the simplest way to encode the absolute position in the sequence. 

    WHAT CAN THE USER ASK ABOUT THE CONTENT OF THESE DOCUMENT SECTIONS? 
    USER QUESTION: How does positional encoding work and what properties does it have? 

    ANSWER THE USER'S QUESTION BASED ON THE CONTENT OF THESE DOCUMENT SECTIONS. 
    TO ANSWER THE QUESTION, YOU MUST FOLLOW THIS FORMAT: 
    ""To answer your question, [answer the question based on the content of the document]. [Page X] [Page Y]...""","['et al., 2017).\nSuppose that the input representation X ∈R𝑛×𝑑contains the 𝑑-dimensional embeddings\nfor 𝑛tokens of a sequence. The positional encoding outputs X + P using a positional\nembedding matrix P ∈R𝑛×𝑑of the same shape, whose element on the 𝑖th row and the'
 'position offset 𝛿, the positional encoding at position 𝑖+ 𝛿can be represented by a linear\nprojection of that at position 𝑖.\nThis projection can be explained mathematically. Denoting 𝜔𝑗= 1/100002𝑗/𝑑, any pair\nof (𝑝𝑖,2𝑗, 𝑝𝑖,2 𝑗+1) in (11.6.2) can be linearly projected to (𝑝𝑖+𝛿,2𝑗, 𝑝𝑖+𝛿,2𝑗+1) for any fixed\noffset 𝛿:\n\x14 cos(𝛿𝜔𝑗)\nsin(𝛿𝜔𝑗)\n−sin(𝛿𝜔𝑗)\ncos(𝛿𝜔𝑗)\n\x15 \x14 𝑝𝑖,2𝑗\n𝑝𝑖,2𝑗+1\n\x15\n=\n\x14 cos(𝛿𝜔𝑗) sin(𝑖𝜔𝑗) + sin(𝛿𝜔𝑗) cos(𝑖𝜔𝑗)\n−sin(𝛿𝜔𝑗) sin(𝑖𝜔𝑗) + cos(𝛿𝜔𝑗) cos(𝑖𝜔𝑗)\n\x15\n=\n\x14sin \x00(𝑖+ 𝛿)𝜔𝑗\n\x01\ncos \x00(𝑖+ 𝛿)𝜔𝑗\n\x01\n\x15\n=\n\x14 𝑝𝑖+𝛿,2𝑗\n𝑝𝑖+𝛿,2 𝑗+1\n\x15\n,\n(11.6.3)\nwhere the 2 × 2 projection matrix does not depend on any position index 𝑖.\n11.6.4 Summary\nIn self-attention, the queries, keys, and values all come from the same place. Both CNNs\nand self-attention enjoy parallel computation and self-attention has the shortest maximum\npath length. However, the quadratic computational complexity with respect to the sequence\nlength makes self-attention prohibitively slow for very long sequences. To use the sequence\norder information, we can inject absolute or relative positional information by adding positional encoding to the input representations.\n11.6.5 Exercises\n1. Suppose that we design a deep architecture to represent a sequence by stacking selfattention layers with positional encoding. What could the possible issues be?\n2. Can you design a learnable positional encoding method?\n3. Can we assign different learned embeddings according to different offsets between queries\nand keys that are compared in self-attention? Hint: you may refer to relative position\nembeddings (Huang et al., 2018, Shaw et al., 2018).\nDiscussions161.\n11.7 The Transformer Architecture'
 'we require that sublayer(x) ∈R𝑑so that the residual connection x + sublayer(x) ∈R𝑑is\nfeasible. This addition from the residual connection is immediately followed by layer normalization (Ba et al., 2016). As a result, the Transformer encoder outputs a 𝑑-dimensional\nvector representation for each position of the input sequence.\nThe Transformer decoder is also a stack of multiple identical layers with residual connections and layer normalizations. As well as the two sublayers described in the encoder, the\ndecoder inserts a third sublayer, known as the encoder–decoder attention, between these\ntwo. In the encoder–decoder attention, queries are from the outputs of the decoder’s selfattention sublayer, and the keys and values are from the Transformer encoder outputs. In\nthe decoder self-attention, queries, keys, and values are all from the outputs of the previous\ndecoder layer. However, each position in the decoder is allowed only to attend to all positions in the decoder up to that position. This masked attention preserves the autoregressive\nproperty, ensuring that the prediction only depends on those output tokens that have been\ngenerated.\nWe have already described and implemented multi-head attention based on scaled dot products in Section 11.5 and positional encoding in Section 11.6.3. In the following, we will\nimplement the rest of the Transformer model.'
 'The Transformer Architecture\nwhose values are always between −1 and 1, we multiply values of the learnable input embeddings by the square root of the embedding dimension to rescale before summing up the\ninput embedding and the positional encoding.\nclass TransformerEncoder(d2l.Encoder):\n#@save\n""""""The Transformer encoder.""""""\ndef __init__(self, vocab_size, num_hiddens, ffn_num_hiddens,\nnum_heads, num_blks, dropout, use_bias=False):\nsuper().__init__()\nself.num_hiddens = num_hiddens\nself.embedding = nn.Embedding(vocab_size, num_hiddens)\nself.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout)\nself.blks = nn.Sequential()\nfor i in range(num_blks):\nself.blks.add_module(""block""+str(i), TransformerEncoderBlock(\nnum_hiddens, ffn_num_hiddens, num_heads, dropout, use_bias))\ndef forward(self, X, valid_lens):\n# Since positional encoding values are between -1 and 1, the embedding\n# values are multiplied by the square root of the embedding dimension\n# to rescale before they are summed up\nX = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))\nself.attention_weights = [None] * len(self.blks)\nfor i, blk in enumerate(self.blks):\nX = blk(X, valid_lens)\nself.attention_weights[\ni] = blk.attention.attention.attention_weights\nreturn X\nBelow we specify hyperparameters to create a two-layer Transformer encoder. The shape of\nthe Transformer encoder output is (batch size, number of time steps, num_hiddens).\nencoder = TransformerEncoder(200, 24, 48, 8, 2, 0.5)\nd2l.check_shape(encoder(torch.ones((2, 100), dtype=torch.long), valid_lens),\n(2, 100, 24))\n11.7.5 Decoder'
 '11.8.4 Putting It All Together\nThe forward pass of vision Transformers below is straightforward. First, input images are\nfed into an PatchEmbedding instance, whose output is concatenated with the “<cls>” token\nembedding. They are summed with learnable positional embeddings before dropout. Then\nthe output is fed into the Transformer encoder that stacks num_blks instances of the ViTBlock class. Finally, the representation of the “<cls>” token is projected by the network\nhead.']","A positional embedding matrix P of the same shape as the input representation X is created. The element on the i-th row and the 2j-th column is sin(i divided by 10000 to the power of 2j over d) and the 2j+1 th column is cos(i divided by 10000 to the power of 2j over d). The positional encoding output is X plus P, after which dropout is applied.",,,0.999999999975,1.0
How does self-attention differ from cross-attention in the Transformer decoder?,"To answer your question, self-attention in the Transformer decoder differs from cross-attention in that self-attention only allows a query to attend to key-value pairs from previous positions in the same output sequence, whereas cross-attention (also known as encoder-decoder attention) allows a query from the output sequence to attend to all key-value pairs from the input sequence. [Page 481] [Page 490] [Page 500]","['we require that sublayer(x) ∈R𝑑so that the residual connection x + sublayer(x) ∈R𝑑is\nfeasible. This addition from the residual connection is immediately followed by layer normalization (Ba et al., 2016). As a result, the Transformer encoder outputs a 𝑑-dimensional\nvector representation for each position of the input sequence.\nThe Transformer decoder is also a stack of multiple identical layers with residual connections and layer normalizations. As well as the two sublayers described in the encoder, the\ndecoder inserts a third sublayer, known as the encoder–decoder attention, between these\ntwo. In the encoder–decoder attention, queries are from the outputs of the decoder’s selfattention sublayer, and the keys and values are from the Transformer encoder outputs. In\nthe decoder self-attention, queries, keys, and values are all from the outputs of the previous\ndecoder layer. However, each position in the decoder is allowed only to attend to all positions in the decoder up to that position. This masked attention preserves the autoregressive\nproperty, ensuring that the prediction only depends on those output tokens that have been\ngenerated.\nWe have already described and implemented multi-head attention based on scaled dot products in Section 11.5 and positional encoding in Section 11.6.3. In the following, we will\nimplement the rest of the Transformer model.'
 'd2l.check_shape(encoder(torch.ones((2, 100), dtype=torch.long), valid_lens),\n(2, 100, 24))\n11.7.5 Decoder\nAs shown in Fig. 11.7.1, the Transformer decoder is composed of multiple identical layers. Each layer is implemented in the following TransformerDecoderBlock class, which\ncontains three sublayers: decoder self-attention, encoder–decoder attention, and positionwise feed-forward networks. These sublayers employ a residual connection around them\nfollowed by layer normalization.\nAs we described earlier in this section, in the masked multi-head decoder self-attention\n(the first sublayer), queries, keys, and values all come from the outputs of the previous\ndecoder layer. When training sequence-to-sequence models, tokens at all the positions\n(time steps) of the output sequence are known. However, during prediction the output\nsequence is generated token by token; thus, at any decoder time step only the generated\ntokens can be used in the decoder self-attention. To preserve autoregression in the decoder,'
 ""Attention Mechanisms and Transformers\nBecause of the autoregressive property of the decoder self-attention, no query attends to\nkey–value pairs after the query position.\nd2l.show_heatmaps(\ndec_self_attention_weights[:, :, :, :],\nxlabel='Key positions', ylabel='Query positions',\ntitles=['Head %d' % i for i in range(1, 5)], figsize=(7, 3.5))\nSimilar to the case in the encoder self-attention, via the specified valid length of the input\nsequence, no query from the output sequence attends to those padding tokens from the input\nsequence.\nd2l.show_heatmaps(\ndec_inter_attention_weights, xlabel='Key positions',\nylabel='Query positions', titles=['Head %d' % i for i in range(1, 5)],\nfigsize=(7, 3.5))\nAlthough the Transformer architecture was originally proposed for sequence-to-sequence\nlearning, as we will discover later in the book, either the Transformer encoder or the Transformer decoder is often individually used for different deep learning tasks.\n11.7.7 Summary""
 'The Transformer is an instance of the encoder–decoder architecture, though either the encoder or the decoder can be used individually in practice. In the Transformer architecture, multi-head self-attention is used for representing the input sequence and the output\nsequence, though the decoder has to preserve the autoregressive property via a masked\nversion. Both the residual connections and the layer normalization in the Transformer are\nimportant for training a very deep model. The positionwise feed-forward network in the\nTransformer model transforms the representation at all the sequence positions using the\nsame MLP.\n11.7.8 Exercises\n1. Train a deeper Transformer in the experiments. How does it affect the training speed\nand the translation performance?\n2. Is it a good idea to replace scaled dot product attention with additive attention in the\nTransformer? Why?\n3. For language modeling, should we use the Transformer encoder, decoder, or both? How\nwould you design this method?\n4. What challenges can Transformers face if input sequences are very long? Why?\n5. How would you improve the computational and memory efficiency of Transformers?\nHint: you may refer to the survey paper by Tay et al. (2020).\nDiscussions162.\n11.8 Transformers for Vision\nThe Transformer architecture was initially proposed for sequence-to-sequence learning,\nwith a focus on machine translation. Subsequently, Transformers emerged as the model\nof choice in various natural language processing tasks (Brown et al., 2020, Devlin et al.,\n2018, Radford et al., 2018, Radford et al., 2019, Raffel et al., 2020). However, in the field of\ncomputer vision the dominant architecture has remained the CNN (Chapter 8). Naturally,'
 'Attention Mechanisms and Transformers\nt\nFig. 11.9.3\nLeft: Pretraining T5 by predicting consecutive spans. The original sentence is “I”, “love”,\n“this”, “red”, “car”, where “love” is replaced by a special “<X>” token, and consecutive\n“red”, “car” are replaced by a special “<Y>” token. The target sequence ends with a\nspecial “<Z>” token. Right: Attention pattern in the Transformer encoder–decoder. In the\nencoder self-attention (lower square), all input tokens attend to each other; In the\nencoder–decoder cross-attention (upper rectangle), each target token attends to all input\ntokens; In the decoder self-attention (upper triangle), each target token attends to present\nand past target tokens only (causal).\nthe decoder has a causal attention pattern to prevent itself from attending to future tokens\nduring sequence prediction.\nIn T5, predicting consecutive span is also referred to as reconstructing corrupted text.\nWith this objective, T5 is pretrained with 1000 billion tokens from the C4 (Colossal Clean\nCrawled Corpus) data, which consists of clean English text from the web (Raffel et al.,\n2020).\nFine-Tuning T5\nSimilar to BERT, T5 needs to be fine-tuned (updating T5 parameters) on task-specific training data to perform this task. Major differences from BERT fine-tuning include: (i) T5\ninput includes task descriptions; (ii) T5 can generate sequences with arbitrary length with\nits Transformer decoder; (iii) No additional layers are required.\nFig. 11.9.4 explains fine-tuning T5 using text summarization as an example. In this downstream task, the task description tokens “Summarize”, “:” followed by the article tokens\nare input to the encoder.']","In self-attention, the queries, keys, and values all come from the same place, meaning each token attends to every other token in the same sequence. In cross-attention in the Transformer decoder, the queries come from the outputs of the decoder self-attention sublayer, while the keys and values come from the Transformer encoder outputs. This allows the decoder to attend to the encoder representations of the input sequence while generating output tokens.",0.7,,0.99999999998,1.0
"How do CNNs, RNNs, and self-attention compare in terms of sequential operations and maximum path length?","To answer your question, CNNs, RNNs, and self-attention compare as follows: CNNs have O(1) sequential operations and a maximum path length of O(n/k), RNNs have O(n) sequential operations and a maximum path length of O(n), and self-attention has O(1) sequential operations and a maximum path length of O(1). [Page 476] [Page 477]","['Recurrent Neural Networks\nnetworks where each layer’s parameters (both conventional and recurrent) are shared across\ntime steps.\nt\nFig. 9.1\nOn the left recurrent connections are depicted via cyclic edges. On the right, we unfold\nthe RNN over time steps. Here, recurrent edges span adjacent time steps, while\nconventional connections are computed synchronously.\nLike neural networks more broadly, RNNs have a long discipline-spanning history, originating as models of the brain popularized by cognitive scientists and subsequently adopted\nas practical modeling tools employed by the machine learning community. As we do for\ndeep learning more broadly, in this book we adopt the machine learning perspective, focusing on RNNs as practical tools that rose to popularity in the 2010s owing to breakthrough\nresults on such diverse tasks as handwriting recognition (Graves et al., 2008), machine\ntranslation (Sutskever et al., 2014), and recognizing medical diagnoses (Lipton et al., 2016).\nWe point the reader interested in more background material to a publicly available comprehensive review (Lipton et al., 2015). We also note that sequentiality is not unique to RNNs.\nFor example, the CNNs that we already introduced can be adapted to handle data of varying\nlength, e.g., images of varying resolution. Moreover, RNNs have recently ceded considerable market share to Transformer models, which will be covered in Chapter 11. However,\nRNNs rose to prominence as the default models for handling complex sequential structure\nin deep learning, and remain staple models for sequential modeling to this day. The stories\nof RNNs and of sequence modeling are inextricably linked, and this is as much a chapter'
 'Queries, Keys, and Values\nthe simplest instantiations of the idea. We then work our way up to the Transformer architecture, the vision Transformer, and the landscape of modern Transformer-based pretrained\nmodels.\n11.1 Queries, Keys, and Values\nSo far all the networks we have reviewed crucially relied on the input being of a welldefined size. For instance, the images in ImageNet are of size 224 × 224 pixels and CNNs\nare specifically tuned to this size. Even in natural language processing the input size for\nRNNs is well defined and fixed. Variable size is addressed by sequentially processing one\ntoken at a time, or by specially designed convolution kernels (Kalchbrenner et al., 2014).\nThis approach can lead to significant problems when the input is truly of varying size with\nvarying information content, such as in Section 10.7 in the transformation of text (Sutskever\net al., 2014). In particular, for long sequences it becomes quite difficult to keep track of\neverything that has already been generated or even viewed by the network. Even explicit\ntracking heuristics such as proposed by Yang et al. (2016) only offer limited benefit.\nCompare this to databases. In their simplest form they are collections of keys (𝑘) and values\n(𝑣). For instance, our database D might consist of tuples {(“Zhang”, “Aston”), (“Lipton”,\n“Zachary”), (“Li”, “Mu”), (“Smola”, “Alex”), (“Hu”, “Rachel”), (“Werness”, “Brent”)}\nwith the last name being the key and the first name being the value. We can operate on\nD, for instance with the exact query (𝑞) for “Li” which would return the value “Mu”. If\n(“Li”, “Mu”) was not a record in D, there would be no valid answer. If we also allowed for'
 '(Hochreiter et al., 2001).\nLet’s regard any text sequence as a “one-dimensional image”. Similarly, one-dimensional\nCNNs can process local features such as 𝑛-grams in text. Given a sequence of length 𝑛, consider a convolutional layer whose kernel size is 𝑘, and whose numbers of input and output\nchannels are both 𝑑. The computational complexity of the convolutional layer is O(𝑘𝑛𝑑2).\nAs Fig. 11.6.1 shows, CNNs are hierarchical, so there are O(1) sequential operations and\nthe maximum path length is O(𝑛/𝑘). For example, x1 and x5 are within the receptive field\nof a two-layer CNN with kernel size 3 in Fig. 11.6.1.\nWhen updating the hidden state of RNNs, multiplication of the 𝑑×𝑑weight matrix and the\n𝑑-dimensional hidden state has a computational complexity of O(𝑑2). Since the sequence\nlength is 𝑛, the computational complexity of the recurrent layer is O(𝑛𝑑2). According\nto Fig. 11.6.1, there are O(𝑛) sequential operations that cannot be parallelized and the\nmaximum path length is also O(𝑛).'
 'Self-Attention and Positional Encoding\nt\nFig. 11.6.1\nComparing CNN (padding tokens are omitted), RNN, and self-attention architectures.\nIn self-attention, the queries, keys, and values are all 𝑛× 𝑑matrices. Consider the scaled\ndot product attention in (11.3.6), where an 𝑛×𝑑matrix is multiplied by a 𝑑×𝑛matrix, then\nthe output 𝑛×𝑛matrix is multiplied by an 𝑛× 𝑑matrix. As a result, the self-attention has a\nO(𝑛2𝑑) computational complexity. As we can see from Fig. 11.6.1, each token is directly\nconnected to any other token via self-attention. Therefore, computation can be parallel with\nO(1) sequential operations and the maximum path length is also O(1).\nAll in all, both CNNs and self-attention enjoy parallel computation and self-attention has\nthe shortest maximum path length. However, the quadratic computational complexity with\nrespect to the sequence length makes self-attention prohibitively slow for very long sequences.\n11.6.3 Positional Encoding\nUnlike RNNs, which recurrently process tokens of a sequence one-by-one, self-attention\nditches sequential operations in favor of parallel computation. Note that self-attention by\nitself does not preserve the order of the sequence. What do we do if it really matters that\nthe model knows in which order the input sequence arrived?\nThe dominant approach for preserving information about the order of tokens is to represent\nthis to the model as an additional input associated with each token. These inputs are called\npositional encodings, and they can either be learned or fixed a priori. We now describe a\nsimple scheme for fixed positional encodings based on sine and cosine functions (Vaswani\net al., 2017).\nSuppose that the input representation X ∈R𝑛×𝑑contains the 𝑑-dimensional embeddings'
 'embeddings (Huang et al., 2018, Shaw et al., 2018).\nDiscussions161.\n11.7 The Transformer Architecture\nWe have compared CNNs, RNNs, and self-attention in Section 11.6.2. Notably, selfattention enjoys both parallel computation and the shortest maximum path length. Therefore, it is appealing to design deep architectures by using self-attention. Unlike earlier\nself-attention models that still rely on RNNs for input representations (Cheng et al., 2016,']","CNNs have O(1) sequential operations and a maximum path length of O(n/k) where k is the kernel size. RNNs have O(n) sequential operations that cannot be parallelized and a maximum path length of O(n). Self-attention has O(1) sequential operations allowing full parallel computation and a maximum path length of O(1), meaning any two tokens are directly connected. However self-attention has a higher computational complexity of O(n²d) compared to O(nd²) for RNNs.",1.0,,0.99999999998,1.0
How does layer normalization differ from batch normalization and why is layer normalization preferred in Transformers?,"To answer your question, layer normalization differs from batch normalization in that it normalizes across the feature dimension, whereas batch normalization normalizes across the examples within a minibatch. This makes layer normalization scale independent and batch size independent, which is beneficial for variable-length sequences in natural language processing tasks. Layer normalization is preferred in Transformers because it prevents divergence, is independent of minibatch size, and is more effective than batch normalization in natural language processing tasks. [Page 336] [Page 341] [Page 483]","['numerical stability, and regularization.\nBatch normalization is applied to individual layers, or optionally, to all of them: In each\ntraining iteration, we first normalize the inputs (of batch normalization) by subtracting their\nmean and dividing by their standard deviation, where both are estimated based on the statistics of the current minibatch. Next, we apply a scale coefficient and an offset to recover the'
 'Batch Normalization\nFixing a trained model, you might think that we would prefer using the entire dataset to\nestimate the mean and variance. Once training is complete, why would we want the same\nimage to be classified differently, depending on the batch in which it happens to reside?\nDuring training, such exact calculation is infeasible because the intermediate variables for\nall data examples change every time we update our model. However, once the model is\ntrained, we can calculate the means and variances of each layer’s variables based on the\nentire dataset. Indeed this is standard practice for models employing batch normalization;\nthus batch normalization layers function differently in training mode (normalizing by minibatch statistics) than in prediction mode (normalizing by dataset statistics). In this form they\nclosely resemble the behavior of dropout regularization of Section 5.6, where noise is only\ninjected during training.\n8.5.2 Batch Normalization Layers\nBatch normalization implementations for fully connected layers and convolutional layers\nare slightly different. One key difference between batch normalization and other layers is\nthat because the former operates on a full minibatch at a time, we cannot just ignore the\nbatch dimension as we did before when introducing other layers.\nFully Connected Layers\nWhen applying batch normalization to fully connected layers, Ioffe and Szegedy (2015), in\ntheir original paper inserted batch normalization after the affine transformation and before\nthe nonlinear activation function. Later applications experimented with inserting batch\nnormalization right after activation functions. Denoting the input to the fully connected'
 'Modern Convolutional Neural Networks\nwe collect the values over all spatial locations when computing the mean and variance and\nconsequently apply the same mean and variance within a given channel to normalize the\nvalue at each spatial location. Each channel has its own scale and shift parameters, both of\nwhich are scalars.\nLayer Normalization\nNote that in the context of convolutions the batch normalization is well defined even for\nminibatches of size 1: after all, we have all the locations across an image to average. Consequently, mean and variance are well defined, even if it is just within a single observation.\nThis consideration led Ba et al. (2016) to introduce the notion of layer normalization. It\nworks just like a batch norm, only that it is applied to one observation at a time. Consequently both the offset and the scaling factor are scalars. For an 𝑛-dimensional vector x,\nlayer norms are given by\nx →LN(x) = x −ˆ\n𝜇\nˆ\n𝜎\n,\n(8.5.4)\nwhere scaling and offset are applied coefficient-wise and given by\nˆ\n𝜇\ndef\n= 1\n𝑛\n𝑛\nÕ\n𝑖=1\n𝑥𝑖and ˆ\n𝜎2 def\n= 1\n𝑛\n𝑛\nÕ\n𝑖=1\n(𝑥𝑖−ˆ\n𝜇)2 + 𝜖.\n(8.5.5)\nAs before we add a small offset 𝜖> 0 to prevent division by zero. One of the major benefits\nof using layer normalization is that it prevents divergence. After all, ignoring 𝜖, the output\nof the layer normalization is scale independent. That is, we have LN(x) ≈LN(𝛼x) for any\nchoice of 𝛼≠0. This becomes an equality for |𝛼| →∞(the approximate equality is due\nto the offset 𝜖for the variance).\nAnother advantage of the layer normalization is that it does not depend on the minibatch\nsize. It is also independent of whether we are in training or test regime. In other words, it is'
 'lead to further inventions of layers and techniques in the future.\nOn a more practical note, there are a number of aspects worth remembering about batch\nnormalization:\n• During model training, batch normalization continuously adjusts the intermediate output\nof the network by utilizing the mean and standard deviation of the minibatch, so that\nthe values of the intermediate output in each layer throughout the neural network are\nmore stable.\n• Batch normalization is slightly different for fully connected layers than for convolutional\nlayers. In fact, for convolutional layers, layer normalization can sometimes be used as\nan alternative.\n• Like a dropout layer, batch normalization layers have different behaviors in training mode\nthan in prediction mode.\n• Batch normalization is useful for regularization and improving convergence in optimization. By contrast, the original motivation of reducing internal covariate shift seems\nnot to be a valid explanation.\n• For more robust models that are less sensitive to input perturbations, consider removing\nbatch normalization (Wang et al., 2022).\n8.5.7 Exercises\n1. Should we remove the bias parameter from the fully connected layer or the convolutional\nlayer before the batch normalization? Why?'
 ""The Transformer Architecture\nber of outputs in the positionwise feed-forward network. Since the same MLP transforms\nat all the positions, when the inputs at all these positions are the same, their outputs are also\nidentical.\nffn = PositionWiseFFN(4, 8)\nffn.eval()\nffn(torch.ones((2, 3, 4)))[0]\ntensor([[ 0.6300,\n0.7739,\n0.0278,\n0.2508, -0.0519,\n0.4881, -0.4105,\n0.\n↩→5163],\n[ 0.6300,\n0.7739,\n0.0278,\n0.2508, -0.0519,\n0.4881, -0.4105,\n0.\n↩→5163],\n[ 0.6300,\n0.7739,\n0.0278,\n0.2508, -0.0519,\n0.4881, -0.4105,\n0.\n↩→5163]],\ngrad_fn=<SelectBackward0>)\n11.7.3 Residual Connection and Layer Normalization\nNow let’s focus on the “add & norm” component in Fig. 11.7.1. As we described at the\nbeginning of this section, this is a residual connection immediately followed by layer normalization. Both are key to effective deep architectures.\nIn Section 8.5, we explained how batch normalization recenters and rescales across the\nexamples within a minibatch. As discussed in Section 8.5.2, layer normalization is the same\nas batch normalization except that the former normalizes across the feature dimension, thus\nenjoying benefits of scale independence and batch size independence. Despite its pervasive\napplications in computer vision, batch normalization is usually empirically less effective\nthan layer normalization in natural language processing tasks, where the inputs are often\nvariable-length sequences.\nThe following code snippet compares the normalization across different dimensions by\nlayer normalization and batch normalization.\nln = nn.LayerNorm(2)\nbn = nn.LazyBatchNorm1d()\nX = torch.tensor([[1, 2], [2, 3]], dtype=torch.float32)\n# Compute mean and variance from X in the training mode\nprint('layer norm:', ln(X), '\\nbatch norm:', bn(X))""]","Batch normalization normalizes across the examples within a minibatch while layer normalization normalizes across the feature dimension. Layer normalization enjoys scale independence and batch size independence. Despite batch normalization being widely used in computer vision, layer normalization is usually empirically more effective in natural language processing tasks where inputs are often variable-length sequences.",0.875,,0.99999999998,0.6666666666666666
How does the Bahdanau attention mechanism differ from the fixed context vector approach in original sequence to sequence models?,"The Bahdanau attention mechanism differs from the fixed context vector approach in that it dynamically updates the context vector at each decoding step, allowing the decoder to selectively focus on different parts of the input sequence. In contrast, the fixed context vector approach compresses the entire input into a single fixed-length vector that is used throughout the decoding process. [Page 450] [Page 467]","['Attention Mechanisms and Transformers\nsequence-to-sequence applications, such as machine translations (Bahdanau et al., 2014).\nYou might recall that in the first sequence-to-sequence models for machine translation\n(Sutskever et al., 2014), the entire input was compressed by the encoder into a single fixedlength vector to be fed into the decoder. The intuition behind attention is that rather than\ncompressing the input, it might be better for the decoder to revisit the input sequence at\nevery step. Moreover, rather than always seeing the same representation of the input, one\nmight imagine that the decoder should selectively focus on particular parts of the input sequence at particular decoding steps. Bahdanau’s attention mechanism provided a simple\nmeans by which the decoder could dynamically attend to different parts of the input at each\ndecoding step. The high-level idea is that the encoder could produce a representation of\nlength equal to the original input sequence. Then, at decoding time, the decoder can (via\nsome control mechanism) receive as input a context vector consisting of a weighted sum\nof the representations on the input at each time step. Intuitively, the weights determine the\nextent to which each step’s context “focuses” on each input token, and the key is to make\nthis process for assigning the weights differentiable so that it can be learned along with all\nof the other neural network parameters.\nInitially, the idea was a remarkably successful enhancement to the recurrent neural networks that already dominated machine translation applications. The models performed\nbetter than the original encoder–decoder sequence-to-sequence architectures. Furthermore,'
 'The Bahdanau Attention Mechanism\n11.4 The Bahdanau Attention Mechanism\nWhen we encountered machine translation in Section 10.7, we designed an encoder–decoder\narchitecture for sequence-to-sequence learning based on two RNNs (Sutskever et al., 2014).\nSpecifically, the RNN encoder transforms a variable-length sequence into a fixed-shape\ncontext variable. Then, the RNN decoder generates the output (target) sequence token by\ntoken based on the generated tokens and the context variable.\nRecall Fig. 10.7.2 which we repeat (Fig. 11.4.1) with some additional detail. Conventionally, in an RNN all relevant information about a source sequence is translated into some\ninternal fixed-dimensional state representation by the encoder. It is this very state that is\nused by the decoder as the complete and exclusive source of information for generating the\ntranslated sequence. In other words, the sequence-to-sequence mechanism treats the intermediate state as a sufficient statistic of whatever string might have served as input.\nt\nFig. 11.4.1\nSequence-to-sequence model. The state, as generated by the encoder, is the only piece of\ninformation shared between the encoder and the decoder.\nWhile this is quite reasonable for short sequences, it is clear that it is infeasible for long ones,\nsuch as a book chapter or even just a very long sentence. After all, before too long there will\nsimply not be enough “space” in the intermediate representation to store all that is important\nin the source sequence. Consequently the decoder will fail to translate long and complex\nsentences. One of the first to encounter this was Graves (2013) who tried to design an\nRNN to generate handwritten text. Since the source text has arbitrary length they designed a'
 'RNN to generate handwritten text. Since the source text has arbitrary length they designed a\ndifferentiable attention model to align text characters with the much longer pen trace, where\nthe alignment moves only in one direction. This, in turn, draws on decoding algorithms in\nspeech recognition, e.g., hidden Markov models (Rabiner and Juang, 1993).\nInspired by the idea of learning to align, Bahdanau et al. (2014) proposed a differentiable\nattention model without the unidirectional alignment limitation. When predicting a token,\nif not all the input tokens are relevant, the model aligns (or attends) only to parts of the input\nsequence that are deemed relevant to the current prediction. This is then used to update the\ncurrent state before generating the next token. While quite innocuous in its description, this\nBahdanau attention mechanism has arguably turned into one of the most influential ideas\nof the past decade in deep learning, giving rise to Transformers (Vaswani et al., 2017) and\nmany related new architectures.'
 'Attention Mechanisms and Transformers\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n11.4.1 Model\nWe follow the notation introduced by the sequence-to-sequence architecture of Section\n10.7, in particular (10.7.3). The key idea is that instead of keeping the state, i.e., the context variable c summarizing the source sentence, as fixed, we dynamically update it, as a\nfunction of both the original text (encoder hidden states h𝑡) and the text that was already\ngenerated (decoder hidden states s𝑡′−1). This yields c𝑡′, which is updated after any decoding time step 𝑡′. Suppose that the input sequence is of length 𝑇. In this case the context\nvariable is the output of attention pooling:\nc𝑡′ =\n𝑇\nÕ\n𝑡=1\n𝛼(s𝑡′−1, h𝑡)h𝑡.\n(11.4.1)\nWe used s𝑡′−1 as the query, and h𝑡as both the key and the value. Note that c𝑡′ is then\nused to generate the state s𝑡′ and to generate a new token: see (10.7.3). In particular, the\nattention weight 𝛼is computed as in (11.3.3) using the additive attention scoring function\ndefined by (11.3.7). This RNN encoder–decoder architecture using attention is depicted in\nFig. 11.4.2. Note that later this model was modified so as to include the already generated\ntokens in the decoder as further context (i.e., the attention sum does not stop at 𝑇but rather\nit proceeds up to 𝑡′ −1). For instance, see Chan et al. (2015) for a description of this\nstrategy, as applied to speech recognition.\nt\nFig. 11.4.2\nLayers in an RNN encoder–decoder model with the Bahdanau attention mechanism.\n11.4.2 Defining the Decoder with Attention\nTo implement the RNN encoder–decoder with attention, we only need to redefine the decoder (omitting the generated symbols from the attention function simplifies the design).'
 'Attention Mechanisms and Transformers\n\nsequence. This is achieved by treating the state (context variable) as an output of additive\nattention pooling. In the RNN encoder–decoder, the Bahdanau attention mechanism treats\nthe decoder hidden state at the previous time step as the query, and the encoder hidden\nstates at all the time steps as both the keys and values.\n11.4.5 Exercises\n1. Replace GRU with LSTM in the experiment.\n2. Modify the experiment to replace the additive attention scoring function with the scaled\ndot-product. How does it influence the training efficiency?\nDiscussions159.\n11.5 Multi-Head Attention\nIn practice, given the same set of queries, keys, and values we may want our model to\ncombine knowledge from different behaviors of the same attention mechanism, such as\ncapturing dependencies of various ranges (e.g., shorter-range vs. longer-range) within a sequence. Thus, it may be beneficial to allow our attention mechanism to jointly use different\nrepresentation subspaces of queries, keys, and values.\nTo this end, instead of performing a single attention pooling, queries, keys, and values can\nbe transformed with ℎindependently learned linear projections. Then these ℎprojected\nqueries, keys, and values are fed into attention pooling in parallel. In the end, ℎattentionpooling outputs are concatenated and transformed with another learned linear projection to\nproduce the final output. This design is called multi-head attention, where each of the ℎ\nattention pooling outputs is a head (Vaswani et al., 2017). Using fully connected layers to\nperform learnable linear transformations, Fig. 11.5.1 describes multi-head attention.\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l']","In original sequence to sequence models, the entire input was compressed by the encoder into a single fixed-length context vector fed into the decoder. The Bahdanau attention mechanism dynamically updates the context variable at each decoding step as a function of both the encoder hidden states at all time steps and the decoder hidden state at the previous time step. This allows the decoder to selectively focus on different parts of the input sequence at each decoding step rather than always seeing the same fixed representation.",0.8,,0.99999999998,
What is attention pooling?,"Attention pooling is an operation that generates a linear combination of values contained in a database, where the weights are computed based on the similarity between a query and keys, and the output is a weighted average of the values. [Page 452] [Page 455] [Page 460]","['Attention Mechanisms and Transformers\nas\nAttention(q, D)\ndef\n=\n𝑚\nÕ\n𝑖=1\n𝛼(q, k𝑖)v𝑖,\n(11.1.1)\nwhere 𝛼(q, k𝑖) ∈R (𝑖= 1, . . . , 𝑚) are scalar attention weights. The operation itself is\ntypically referred to as attention pooling. The name attention derives from the fact that the\noperation pays particular attention to the terms for which the weight 𝛼is significant (i.e.,\nlarge). As such, the attention over D generates a linear combination of values contained in\nthe database. In fact, this contains the above example as a special case where all but one\nweight is zero. We have a number of special cases:\n• The weights 𝛼(q, k𝑖) are nonnegative. In this case the output of the attention mechanism\nis contained in the convex cone spanned by the values v𝑖.\n• The weights 𝛼(q, k𝑖) form a convex combination, i.e., Í\n𝑖𝛼(q, k𝑖) = 1 and 𝛼(q, k𝑖) ≥0\nfor all 𝑖. This is the most common setting in deep learning.\n• Exactly one of the weights 𝛼(q, k𝑖) is 1, while all others are 0. This is akin to a traditional\ndatabase query.\n• All weights are equal, i.e., 𝛼(q, k𝑖) = 1\n𝑚for all 𝑖. This amounts to averaging across the\nentire database, also called average pooling in deep learning.\nA common strategy for ensuring that the weights sum up to 1 is to normalize them via\n𝛼(q, k𝑖) =\n𝛼(q, k𝑖)\nÍ\n𝑗𝛼(q, k 𝑗) .\n(11.1.2)\nIn particular, to ensure that the weights are also nonnegative, one can resort to exponentiation. This means that we can now pick any function 𝑎(q, k) and then apply the softmax\noperation used for multinomial models to it via\n𝛼(q, k𝑖) =\nexp(𝑎(q, k𝑖))\nÍ\n𝑗exp(𝑎(q, k 𝑗)) .\n(11.1.3)\nThis operation is readily available in all deep learning frameworks. It is differentiable and'
 'Attention Pooling by Similarity'
 'Discussions154.\n11.2 Attention Pooling by Similarity\nNow that we have introduced the primary components of the attention mechanism, let’s\nuse them in a rather classical setting, namely regression and classification via kernel density estimation (Nadaraya, 1964, Watson, 1964). This detour simply provides additional\nbackground: it is entirely optional and can be skipped if needed. At their core, Nadaraya–\nWatson estimators rely on some similarity kernel 𝛼(q, k) relating queries q to keys k. Some\ncommon kernels are\n𝛼(q, k) = exp\n\x12\n−1\n2 ∥q −k∥2\n\x13\nGaussian;\n𝛼(q, k) = 1 if ∥q −k∥≤1\nBoxcar;\n𝛼(q, k) = max (0, 1 −∥q −k∥)\nEpanechikov.\n(11.2.1)\nThere are many more choices that we could pick. See a Wikipedia article 155 for a more\nextensive review and how the choice of kernels is related to kernel density estimation, sometimes also called Parzen Windows (Parzen, 1957). All of the kernels are heuristic and can\nbe tuned. For instance, we can adjust the width, not only on a global basis but even on a\nper-coordinate basis. Regardless, all of them lead to the following equation for regression\nand classification alike:\n𝑓(q) =\nÕ\n𝑖\nv𝑖\n𝛼(q, k𝑖)\nÍ\n𝑗𝛼(q, k 𝑗) .\n(11.2.2)\nIn the case of a (scalar) regression with observations (x𝑖, 𝑦𝑖) for features and labels respectively, v𝑖= 𝑦𝑖are scalars, k𝑖= x𝑖are vectors, and the query q denotes the new location\nwhere 𝑓should be evaluated. In the case of (multiclass) classification, we use one-hotencoding of 𝑦𝑖to obtain v𝑖. One of the convenient properties of this estimator is that it requires no training. Even more so, if we suitably narrow the kernel with increasing amounts\nof data, the approach is consistent (Mack and Silverman, 1982), i.e., it will converge to'
 'Attention Pooling by Similarity\nClearly, the narrower the kernel, the less smooth the estimate. At the same time, it adapts\nbetter to the local variations. Let’s look at the corresponding attention weights.\nplot(x_train, y_train, x_val, y_val, kernels, names, attention=True)\nAs we would expect, the narrower the kernel, the narrower the range of large attention\nweights. It is also clear that picking the same width might not be ideal. In fact, Silverman\n(1986) proposed a heuristic that depends on the local density. Many more such “tricks”\nhave been proposed. For instance, Norelli et al. (2022) used a similar nearest-neighbor\ninterpolation technique for designing cross-modal image and text representations.\nThe astute reader might wonder why we are providing this deep dive for a method that is over\nhalf a century old. First, it is one of the earliest precursors of modern attention mechanisms.\nSecond, it is great for visualization. Third, and just as importantly, it demonstrates the limits\nof hand-crafted attention mechanisms. A much better strategy is to learn the mechanism,\nby learning the representations for queries and keys. This is what we will embark on in the\nfollowing sections.\n11.2.4 Summary\nNadaraya–Watson kernel regression is an early precursor of the current attention mechanisms. It can be used directly with little to no training or tuning, either for classification or\nregression. The attention weight is assigned according to the similarity (or distance) between query and key, and according to how many similar observations are available.\n11.2.5 Exercises'
 't\nFig. 11.3.1\nComputing the output of attention pooling as a weighted average of values, where weights\nare computed with the attention scoring function a and the softmax operation.']","Attention pooling is the operation that computes a weighted sum of values from a database of key-value pairs, where the weights are determined by the compatibility between a query and each key. Formally, given a database of m key-value tuples and a query q, attention pooling computes the sum over all i of alpha(q, ki) times vi, where alpha(q, ki) are scalar attention weights. The name derives from the fact that the operation pays particular attention to terms for which the weight alpha is significant or large.",1.0,,0.99999999998,
What is positional encoding and why is it needed in the Transformer?,"Positional encoding is a technique used to add positional information to the input embeddings in the Transformer model, as the self-attention mechanism is permutation-equivalent and does not inherently capture the order of the input sequence. It is needed to preserve the sequence order and provide the model with information about the position of each token in the sequence. [Page 481] [Page 485]","['Self-Attention and Positional Encoding'
 'The Transformer Architecture\nLin et al., 2017, Paulus et al., 2017), the Transformer model is solely based on attention\nmechanisms without any convolutional or recurrent layer (Vaswani et al., 2017). Though\noriginally proposed for sequence-to-sequence learning on text data, Transformers have been\npervasive in a wide range of modern deep learning applications, such as in areas to do with\nlanguage, vision, speech, and reinforcement learning.\nimport math\nimport pandas as pd\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n11.7.1 Model\nAs an instance of the encoder–decoder architecture, the overall architecture of the Transformer is presented in Fig. 11.7.1. As we can see, the Transformer is composed of an encoder and a decoder. In contrast to Bahdanau attention for sequence-to-sequence learning\nin Fig. 11.4.2, the input (source) and output (target) sequence embeddings are added with\npositional encoding before being fed into the encoder and the decoder that stack modules\nbased on self-attention.\nNow we provide an overview of the Transformer architecture in Fig. 11.7.1. At a high level,\nthe Transformer encoder is a stack of multiple identical layers, where each layer has two\nsublayers (either is denoted as sublayer). The first is a multi-head self-attention pooling\nand the second is a positionwise feed-forward network. Specifically, in the encoder selfattention, queries, keys, and values are all from the outputs of the previous encoder layer.\nInspired by the ResNet design of Section 8.6, a residual connection is employed around\nboth sublayers. In the Transformer, for any input x ∈R𝑑at any position of the sequence,\nwe require that sublayer(x) ∈R𝑑so that the residual connection x + sublayer(x) ∈R𝑑is'
 'we require that sublayer(x) ∈R𝑑so that the residual connection x + sublayer(x) ∈R𝑑is\nfeasible. This addition from the residual connection is immediately followed by layer normalization (Ba et al., 2016). As a result, the Transformer encoder outputs a 𝑑-dimensional\nvector representation for each position of the input sequence.\nThe Transformer decoder is also a stack of multiple identical layers with residual connections and layer normalizations. As well as the two sublayers described in the encoder, the\ndecoder inserts a third sublayer, known as the encoder–decoder attention, between these\ntwo. In the encoder–decoder attention, queries are from the outputs of the decoder’s selfattention sublayer, and the keys and values are from the Transformer encoder outputs. In\nthe decoder self-attention, queries, keys, and values are all from the outputs of the previous\ndecoder layer. However, each position in the decoder is allowed only to attend to all positions in the decoder up to that position. This masked attention preserves the autoregressive\nproperty, ensuring that the prediction only depends on those output tokens that have been\ngenerated.\nWe have already described and implemented multi-head attention based on scaled dot products in Section 11.5 and positional encoding in Section 11.6.3. In the following, we will\nimplement the rest of the Transformer model.'
 'The Transformer Architecture\nwhose values are always between −1 and 1, we multiply values of the learnable input embeddings by the square root of the embedding dimension to rescale before summing up the\ninput embedding and the positional encoding.\nclass TransformerEncoder(d2l.Encoder):\n#@save\n""""""The Transformer encoder.""""""\ndef __init__(self, vocab_size, num_hiddens, ffn_num_hiddens,\nnum_heads, num_blks, dropout, use_bias=False):\nsuper().__init__()\nself.num_hiddens = num_hiddens\nself.embedding = nn.Embedding(vocab_size, num_hiddens)\nself.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout)\nself.blks = nn.Sequential()\nfor i in range(num_blks):\nself.blks.add_module(""block""+str(i), TransformerEncoderBlock(\nnum_hiddens, ffn_num_hiddens, num_heads, dropout, use_bias))\ndef forward(self, X, valid_lens):\n# Since positional encoding values are between -1 and 1, the embedding\n# values are multiplied by the square root of the embedding dimension\n# to rescale before they are summed up\nX = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))\nself.attention_weights = [None] * len(self.blks)\nfor i, blk in enumerate(self.blks):\nX = blk(X, valid_lens)\nself.attention_weights[\ni] = blk.attention.attention.attention_weights\nreturn X\nBelow we specify hyperparameters to create a two-layer Transformer encoder. The shape of\nthe Transformer encoder output is (batch size, number of time steps, num_hiddens).\nencoder = TransformerEncoder(200, 24, 48, 8, 2, 0.5)\nd2l.check_shape(encoder(torch.ones((2, 100), dtype=torch.long), valid_lens),\n(2, 100, 24))\n11.7.5 Decoder'
 'The Transformer is an instance of the encoder–decoder architecture, though either the encoder or the decoder can be used individually in practice. In the Transformer architecture, multi-head self-attention is used for representing the input sequence and the output\nsequence, though the decoder has to preserve the autoregressive property via a masked\nversion. Both the residual connections and the layer normalization in the Transformer are\nimportant for training a very deep model. The positionwise feed-forward network in the\nTransformer model transforms the representation at all the sequence positions using the\nsame MLP.\n11.7.8 Exercises\n1. Train a deeper Transformer in the experiments. How does it affect the training speed\nand the translation performance?\n2. Is it a good idea to replace scaled dot product attention with additive attention in the\nTransformer? Why?\n3. For language modeling, should we use the Transformer encoder, decoder, or both? How\nwould you design this method?\n4. What challenges can Transformers face if input sequences are very long? Why?\n5. How would you improve the computational and memory efficiency of Transformers?\nHint: you may refer to the survey paper by Tay et al. (2020).\nDiscussions162.\n11.8 Transformers for Vision\nThe Transformer architecture was initially proposed for sequence-to-sequence learning,\nwith a focus on machine translation. Subsequently, Transformers emerged as the model\nof choice in various natural language processing tasks (Brown et al., 2020, Devlin et al.,\n2018, Radford et al., 2018, Radford et al., 2019, Raffel et al., 2020). However, in the field of\ncomputer vision the dominant architecture has remained the CNN (Chapter 8). Naturally,']","Positional encoding is additional input associated with each token that represents the order of tokens in a sequence. It is needed because unlike RNNs which process tokens sequentially and implicitly encode position, self-attention discards sequential operations in favor of parallel computation and by itself does not preserve the order of the sequence. Positional encodings can either be learned or fixed. The Transformer uses fixed encodings based on sine and cosine functions of different frequencies.",0.5,,0.99999999998,
What is scaled dot product attention?,"Scaled dot product attention is an attention function that is used in Transformers, which rescales the dot product of query and key vectors by the square root of the dimensionality of the vectors. [Page 461] This is done to ensure that the variance of the dot product remains 1, regardless of the vector length. The scaled dot product attention scoring function is given by 𝑎(q, k𝑖) = q⊤k𝑖/ √𝑑, where q is the query vector, k𝑖 is the key vector, and 𝑑 is the dimensionality of the vectors. [Page 461] The attention weights are then computed by applying the softmax operation to the scaled dot product attention scores. [Page 461] [Page 464]","['Attention Scoring Functions\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n11.3.1 Dot Product Attention\nLet’s review the attention function (without exponentiation) from the Gaussian kernel for\na moment:\n𝑎(q, k𝑖) = −1\n2 ∥q −k𝑖∥2 = q⊤k𝑖−1\n2 ∥k𝑖∥2 −1\n2 ∥q∥2.\n(11.3.1)\nFirst, note that the final term depends on q only. As such it is identical for all (q, k𝑖)\npairs. Normalizing the attention weights to 1, as is done in (11.1.3), ensures that this term\ndisappears entirely. Second, note that both batch and layer normalization (to be discussed\nlater) lead to activations that have well-bounded, and often constant, norms ∥k𝑖∥. This is\nthe case, for instance, whenever the keys k𝑖were generated by a layer norm. As such, we\ncan drop it from the definition of 𝑎without any major change in the outcome.\nLast, we need to keep the order of magnitude of the arguments in the exponential function\nunder control. Assume that all the elements of the query q ∈R𝑑and the key k𝑖∈R𝑑\nare independent and identically drawn random variables with zero mean and unit variance.\nThe dot product between both vectors has zero mean and a variance of 𝑑. To ensure that\nthe variance of the dot product still remains 1 regardless of vector length, we use the scaled\ndot product attention scoring function. That is, we rescale the dot product by 1/\n√\n𝑑. We\nthus arrive at the first commonly used attention function that is used, e.g., in Transformers\n(Vaswani et al., 2017):\n𝑎(q, k𝑖) = q⊤k𝑖/\n√\n𝑑.\n(11.3.2)\nNote that attention weights 𝛼still need normalizing. We can simplify this further via\n(11.1.3) by using the softmax operation:\n𝛼(q, k𝑖) = softmax(𝑎(q, k𝑖)) =\nexp(q⊤k𝑖/\n√\n𝑑)\nÍ\n𝑗=1 exp(q⊤k 𝑗/\n√\n𝑑)\n.\n(11.3.3)'
 'Attention Mechanisms and Transformers\nvalues V ∈R𝑚×𝑣thus can be written as\nsoftmax\n\x12QK⊤\n√\n𝑑\n\x13\nV ∈R𝑛×𝑣.\n(11.3.6)\nNote that when applying this to a minibatch, we need the batch matrix multiplication introduced in (11.3.5). In the following implementation of the scaled dot product attention, we\nuse dropout for model regularization.\nclass DotProductAttention(nn.Module):\n#@save\n""""""Scaled dot product attention.""""""\ndef __init__(self, dropout):\nsuper().__init__()\nself.dropout = nn.Dropout(dropout)\n# Shape of queries: (batch_size, no. of queries, d)\n# Shape of keys: (batch_size, no. of key-value pairs, d)\n# Shape of values: (batch_size, no. of key-value pairs, value dimension)\n# Shape of valid_lens: (batch_size,) or (batch_size, no. of queries)\ndef forward(self, queries, keys, values, valid_lens=None):\nd = queries.shape[-1]\n# Swap the last two dimensions of keys with keys.transpose(1, 2)\nscores = torch.bmm(queries, keys.transpose(1, 2)) / math.sqrt(d)\nself.attention_weights = masked_softmax(scores, valid_lens)\nreturn torch.bmm(self.dropout(self.attention_weights), values)\nTo illustrate how the DotProductAttention class works, we use the same keys, values,\nand valid lengths from the earlier toy example for additive attention. For the purpose of\nour example we assume that we have a minibatch size of 2, a total of 10 keys and values,\nand that the dimensionality of the values is 4. Lastly, we assume that the valid length per\nobservation is 2 and 6 respectively. Given that, we expect the output to be a 2×1×4 tensor,\ni.e., one row per example of the minibatch.\nqueries = torch.normal(0, 1, (2, 1, 2))\nkeys = torch.normal(0, 1, (2, 10, 2))\nvalues = torch.normal(0, 1, (2, 10, 4))\nvalid_lens = torch.tensor([2, 6])'
 'Attention Mechanisms and Transformers\n\nsequence. This is achieved by treating the state (context variable) as an output of additive\nattention pooling. In the RNN encoder–decoder, the Bahdanau attention mechanism treats\nthe decoder hidden state at the previous time step as the query, and the encoder hidden\nstates at all the time steps as both the keys and values.\n11.4.5 Exercises\n1. Replace GRU with LSTM in the experiment.\n2. Modify the experiment to replace the additive attention scoring function with the scaled\ndot-product. How does it influence the training efficiency?\nDiscussions159.\n11.5 Multi-Head Attention\nIn practice, given the same set of queries, keys, and values we may want our model to\ncombine knowledge from different behaviors of the same attention mechanism, such as\ncapturing dependencies of various ranges (e.g., shorter-range vs. longer-range) within a sequence. Thus, it may be beneficial to allow our attention mechanism to jointly use different\nrepresentation subspaces of queries, keys, and values.\nTo this end, instead of performing a single attention pooling, queries, keys, and values can\nbe transformed with ℎindependently learned linear projections. Then these ℎprojected\nqueries, keys, and values are fed into attention pooling in parallel. In the end, ℎattentionpooling outputs are concatenated and transformed with another learned linear projection to\nproduce the final output. This design is called multi-head attention, where each of the ℎ\nattention pooling outputs is a head (Vaswani et al., 2017). Using fully connected layers to\nperform learnable linear transformations, Fig. 11.5.1 describes multi-head attention.\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l'
 'Self-Attention and Positional Encoding\nt\nFig. 11.6.1\nComparing CNN (padding tokens are omitted), RNN, and self-attention architectures.\nIn self-attention, the queries, keys, and values are all 𝑛× 𝑑matrices. Consider the scaled\ndot product attention in (11.3.6), where an 𝑛×𝑑matrix is multiplied by a 𝑑×𝑛matrix, then\nthe output 𝑛×𝑛matrix is multiplied by an 𝑛× 𝑑matrix. As a result, the self-attention has a\nO(𝑛2𝑑) computational complexity. As we can see from Fig. 11.6.1, each token is directly\nconnected to any other token via self-attention. Therefore, computation can be parallel with\nO(1) sequential operations and the maximum path length is also O(1).\nAll in all, both CNNs and self-attention enjoy parallel computation and self-attention has\nthe shortest maximum path length. However, the quadratic computational complexity with\nrespect to the sequence length makes self-attention prohibitively slow for very long sequences.\n11.6.3 Positional Encoding\nUnlike RNNs, which recurrently process tokens of a sequence one-by-one, self-attention\nditches sequential operations in favor of parallel computation. Note that self-attention by\nitself does not preserve the order of the sequence. What do we do if it really matters that\nthe model knows in which order the input sequence arrived?\nThe dominant approach for preserving information about the order of tokens is to represent\nthis to the model as an additional input associated with each token. These inputs are called\npositional encodings, and they can either be learned or fixed a priori. We now describe a\nsimple scheme for fixed positional encodings based on sine and cosine functions (Vaswani\net al., 2017).\nSuppose that the input representation X ∈R𝑛×𝑑contains the 𝑑-dimensional embeddings'
 'Y = self.addnorm1(X, X2)\n# Encoder-decoder attention. Shape of enc_outputs:\n# (batch_size, num_steps, num_hiddens)\nY2 = self.attention2(Y, enc_outputs, enc_outputs, enc_valid_lens)\nZ = self.addnorm2(Y, Y2)\nreturn self.addnorm3(Z, self.ffn(Z)), state\nTo facilitate scaled dot product operations in the encoder–decoder attention and addition\noperations in the residual connections, the feature dimension (num_hiddens) of the decoder\nis the same as that of the encoder.\ndecoder_blk = TransformerDecoderBlock(24, 48, 8, 0.5, 0)\nX = torch.ones((2, 100, 24))\nstate = [encoder_blk(X, valid_lens), valid_lens, [None]]\nd2l.check_shape(decoder_blk(X, state)[0], X.shape)']","Scaled dot product attention is an attention scoring function that computes the dot product between a query and a key and divides by the square root of the key dimensionality d. This scaling ensures that when all elements of the query and key are independent random variables with zero mean and unit variance, the variance of the dot product remains 1 regardless of vector length. The softmax is then applied to the scaled dot products to produce the attention weights.",,,0.99999999998,
What is multi-head attention?,"Multi-head attention is a design that allows a model to jointly use different representation subspaces of queries, keys, and values by transforming them with independently learned linear projections and then feeding these projected queries, keys, and values into attention pooling in parallel. The outputs from each attention pooling are concatenated and transformed with another learned linear projection to produce the final output [Page 472] [Page 473]. 

This design enables the model to capture dependencies of various ranges within a sequence and attend to different parts of the input [Page 472]. Each head may attend to different parts of the input, allowing for more sophisticated functions than a simple weighted average [Page 473].","['Attention Mechanisms and Transformers\n\nsequence. This is achieved by treating the state (context variable) as an output of additive\nattention pooling. In the RNN encoder–decoder, the Bahdanau attention mechanism treats\nthe decoder hidden state at the previous time step as the query, and the encoder hidden\nstates at all the time steps as both the keys and values.\n11.4.5 Exercises\n1. Replace GRU with LSTM in the experiment.\n2. Modify the experiment to replace the additive attention scoring function with the scaled\ndot-product. How does it influence the training efficiency?\nDiscussions159.\n11.5 Multi-Head Attention\nIn practice, given the same set of queries, keys, and values we may want our model to\ncombine knowledge from different behaviors of the same attention mechanism, such as\ncapturing dependencies of various ranges (e.g., shorter-range vs. longer-range) within a sequence. Thus, it may be beneficial to allow our attention mechanism to jointly use different\nrepresentation subspaces of queries, keys, and values.\nTo this end, instead of performing a single attention pooling, queries, keys, and values can\nbe transformed with ℎindependently learned linear projections. Then these ℎprojected\nqueries, keys, and values are fed into attention pooling in parallel. In the end, ℎattentionpooling outputs are concatenated and transformed with another learned linear projection to\nproduce the final output. This design is called multi-head attention, where each of the ℎ\nattention pooling outputs is a head (Vaswani et al., 2017). Using fully connected layers to\nperform learnable linear transformations, Fig. 11.5.1 describes multi-head attention.\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l'
 'Multi-Head Attention\nt\nFig. 11.5.1\nMulti-head attention, where multiple heads are concatenated then linearly transformed.\n11.5.1 Model\nBefore providing the implementation of multi-head attention, let’s formalize this model\nmathematically. Given a query q ∈R𝑑𝑞, a key k ∈R𝑑𝑘, and a value v ∈R𝑑𝑣, each\nattention head h𝑖(𝑖= 1, . . . , ℎ) is computed as\nh𝑖= 𝑓(W(𝑞)\n𝑖\nq, W(𝑘)\n𝑖\nk, W(𝑣)\n𝑖\nv) ∈R𝑝𝑣,\n(11.5.1)\nwhere W(𝑞)\n𝑖\n∈R𝑝𝑞×𝑑𝑞, W(𝑘)\n𝑖\n∈R𝑝𝑘×𝑑𝑘, and W(𝑣)\n𝑖\n∈R𝑝𝑣×𝑑𝑣are learnable parameters\nand 𝑓is attention pooling, such as additive attention and scaled dot product attention in\nSection 11.3. The multi-head attention output is another linear transformation via learnable\nparameters W𝑜∈R𝑝𝑜×ℎ𝑝𝑣of the concatenation of ℎheads:\nW𝑜\n\uf8ee\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8ef\n\uf8f0\nh1\n.\n.\n.\nhℎ\n\uf8f9\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fa\n\uf8fb\n∈R𝑝𝑜.\n(11.5.2)\nBased on this design, each head may attend to different parts of the input. More sophisticated functions than the simple weighted average can be expressed.\n11.5.2 Implementation\nIn our implementation, we choose the scaled dot product attention for each head of the\nmulti-head attention. To avoid significant growth of computational cost and parametrization cost, we set 𝑝𝑞= 𝑝𝑘= 𝑝𝑣= 𝑝𝑜/ℎ. Note that ℎheads can be computed in parallel\nif we set the number of outputs of linear transformations for the query, key, and value to\n𝑝𝑞ℎ= 𝑝𝑘ℎ= 𝑝𝑣ℎ= 𝑝𝑜. In the following implementation, 𝑝𝑜is specified via the argument\nnum_hiddens.\nclass MultiHeadAttention(d2l.Module):\n#@save\n""""""Multi-head attention.""""""\ndef __init__(self, num_hiddens, num_heads, dropout, bias=False, **kwargs):\nsuper().__init__()\nself.num_heads = num_heads\nself.attention = d2l.DotProductAttention(dropout)\nself.W_q = nn.LazyLinear(num_hiddens, bias=bias)\nself.W_k = nn.LazyLinear(num_hiddens, bias=bias)'
 'and values are the same.\nAs a result, the shape of the multi-head attention output is\n(batch_size, num_queries, num_hiddens).\nnum_hiddens, num_heads = 100, 5\nattention = MultiHeadAttention(num_hiddens, num_heads, 0.5)\nbatch_size, num_queries, num_kvpairs = 2, 4, 6\nvalid_lens = torch.tensor([3, 2])\nX = torch.ones((batch_size, num_queries, num_hiddens))\nY = torch.ones((batch_size, num_kvpairs, num_hiddens))\nd2l.check_shape(attention(X, Y, Y, valid_lens),\n(batch_size, num_queries, num_hiddens))\n11.5.3 Summary\nMulti-head attention combines knowledge of the same attention pooling via different representation subspaces of queries, keys, and values. To compute multiple heads of multi-head\nattention in parallel, proper tensor manipulation is needed.\n11.5.4 Exercises\n1. Visualize attention weights of multiple heads in this experiment.\n2. Suppose that we have a trained model based on multi-head attention and we want to\nprune less important attention heads to increase the prediction speed. How can we design experiments to measure the importance of an attention head?\nDiscussions160.\n11.6 Self-Attention and Positional Encoding\nIn deep learning, we often use CNNs or RNNs to encode sequences. Now with attention\nmechanisms in mind, imagine feeding a sequence of tokens into an attention mechanism\nsuch that at every step, each token has its own query, keys, and values. Here, when computing the value of a token’s representation at the next layer, the token can attend (via its query\nvector) to any other’s token (matching based on their key vectors). Using the full set of'
 'Attention Mechanisms and Transformers\nimport math\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n11.6.1 Self-Attention\nGiven a sequence of input tokens x1, . . . , x𝑛where any x𝑖∈R𝑑(1 ≤𝑖≤𝑛), its selfattention outputs a sequence of the same length y1, . . . , y𝑛, where\ny𝑖= 𝑓(x𝑖, (x1, x1), . . . , (x𝑛, x𝑛)) ∈R𝑑\n(11.6.1)\naccording to the definition of attention pooling in (11.1.1). Using multi-head attention,\nthe following code snippet computes the self-attention of a tensor with shape (batch size,\nnumber of time steps or sequence length in tokens, 𝑑). The output tensor has the same\nshape.\nnum_hiddens, num_heads = 100, 5\nattention = d2l.MultiHeadAttention(num_hiddens, num_heads, 0.5)\nbatch_size, num_queries, valid_lens = 2, 4, torch.tensor([3, 2])\nX = torch.ones((batch_size, num_queries, num_hiddens))\nd2l.check_shape(attention(X, X, X, valid_lens),\n(batch_size, num_queries, num_hiddens))\n11.6.2 Comparing CNNs, RNNs, and Self-Attention\nLet’s compare architectures for mapping a sequence of 𝑛tokens to another one of equal\nlength, where each input or output token is represented by a 𝑑-dimensional vector. Specifically, we will consider CNNs, RNNs, and self-attention. We will compare their computational complexity, sequential operations, and maximum path lengths. Note that sequential\noperations prevent parallel computation, while a shorter path between any combination of\nsequence positions makes it easier to learn long-range dependencies within the sequence\n(Hochreiter et al., 2001).\nLet’s regard any text sequence as a “one-dimensional image”. Similarly, one-dimensional'
 ""The Transformer Architecture\n(continued from previous page)\nd2l.check_shape(enc_attention_weights,\n(num_blks, num_heads, data.num_steps, data.num_steps))\nIn the encoder self-attention, both queries and keys come from the same input sequence.\nSince padding tokens do not carry meaning, with specified valid length of the input sequence no query attends to positions of padding tokens. In the following, two layers of\nmulti-head attention weights are presented row by row. Each head independently attends\nbased on a separate representation subspace of queries, keys, and values.\nd2l.show_heatmaps(\nenc_attention_weights.cpu(), xlabel='Key positions',\nylabel='Query positions', titles=['Head %d' % i for i in range(1, 5)],\nfigsize=(7, 3.5))\nTo visualize the decoder self-attention weights and the encoder–decoder attention weights,\nwe need more data manipulations. For example, we fill the masked attention weights\nwith zero. Note that the decoder self-attention weights and the encoder–decoder attention weights both have the same queries: the beginning-of-sequence token followed by the\noutput tokens and possibly end-of-sequence tokens.\ndec_attention_weights_2d = [head[0].tolist()\nfor step in dec_attention_weights\nfor attn in step for blk in attn for head in blk]\ndec_attention_weights_filled = torch.tensor(\npd.DataFrame(dec_attention_weights_2d).fillna(0.0).values)\nshape = (-1, 2, num_blks, num_heads, data.num_steps)\ndec_attention_weights = dec_attention_weights_filled.reshape(shape)\ndec_self_attention_weights, dec_inter_attention_weights = \\\ndec_attention_weights.permute(1, 2, 3, 0, 4)\nd2l.check_shape(dec_self_attention_weights,\n(num_blks, num_heads, data.num_steps, data.num_steps))\nd2l.check_shape(dec_inter_attention_weights,""]","Multi-head attention is a design where instead of performing a single attention pooling, queries, keys, and values are transformed with h independently learned linear projections and fed into attention pooling in parallel. The h attention pooling outputs called heads are concatenated and transformed with another learned linear projection to produce the final output. This allows each head to attend to different parts of the input and capture dependencies of various ranges within a sequence.",,,0.99999999998,