| top_k_chunks = {'ids': ['348ae231-e272-409b-9fe3-93be619e25f5_1107', '348ae231-e272-409b-9fe3-93be619e25f5_1110', '348ae231-e272-409b-9fe3-93be619e25f5_1124', '348ae231-e272-409b-9fe3-93be619e25f5_1125', '348ae231-e272-409b-9fe3-93be619e25f5_1134'], 'embeddings': None, 'documents': ['are not mutually exclusive, and have been successfully combined for phoneme classification\n(Graves and Schmidhuber, 2005) and handwriting recognition (Graves et al., 2008).\nThe first sections in this chapter will explain the LSTM architecture, a lighter-weight version\ncalled the gated recurrent unit (GRU), the key ideas behind bidirectional RNNs and a brief\nexplanation of how RNN layers are stacked together to form deep RNNs. Subsequently,\nwe will explore the application of RNNs in sequence-to-sequence tasks, introducing machine translation along with key ideas such as encoder–decoder architectures and beam\nsearch.', '10.1.1 Gated Memory Cell\nEach memory cell is equipped with an internal state and a number of multiplicative gates\nthat determine whether (i) a given input should impact the internal state (the input gate),\n(ii) the internal state should be flushed to 0 (the forget gate), and (iii) the internal state of a\ngiven neuron should be allowed to impact the cell’s output (the output gate).\nGated Hidden State\nThe key distinction between vanilla RNNs and LSTMs is that the latter support gating of\nthe hidden state. This means that we have dedicated mechanisms for when a hidden state\nshould be updated and also for when it should be reset. These mechanisms are learned and\nthey address the concerns listed above. For instance, if the first token is of great importance\nwe will learn not to update the hidden state after the first observation. Likewise, we will\nlearn to skip irrelevant temporary observations. Last, we will learn to reset the latent state\nwhenever needed. We discuss this in detail below.', 'gating mechanisms but with the aim of speeding up computation. The gated recurrent unit', 'Gated Recurrent Units (GRU)\n(GRU) (Cho et al., 2014) offered a streamlined version of the LSTM memory cell that often achieves comparable performance but with the advantage of being faster to compute\n(Chung et al., 2014).\nimport torch\nfrom torch import nn\nfrom d2l import torch as d2l\n10.2.1 Reset Gate and Update Gate\nHere, the LSTM’s three gates are replaced by two: the reset gate and the update gate. As\nwith LSTMs, these gates are given sigmoid activations, forcing their values to lie in the\ninterval (0, 1). Intuitively, the reset gate controls how much of the previous state we might\nstill want to remember. Likewise, an update gate would allow us to control how much of\nthe new state is just a copy of the old one. Fig. 10.2.1 illustrates the inputs for both the reset\nand update gates in a GRU, given the input of the current time step and the hidden state\nof the previous time step. The outputs of the gates are given by two fully connected layers\nwith a sigmoid activation function.\nt\nFig. 10.2.1\nComputing the reset gate and the update gate in a GRU model.', "The code is significantly faster in training as it uses compiled operators rather than Python.\ngru = GRU(num_inputs=len(data.vocab), num_hiddens=32)\nmodel = d2l.RNNLM(gru, vocab_size=len(data.vocab), lr=4)\ntrainer.fit(model, data)\nAfter training, we print out the perplexity on the training set and the predicted sequence\nfollowing the provided prefix.\nmodel.predict('it has', 20, data.vocab, d2l.try_gpu())\n'it has so it and the time '\n10.2.6 Summary\nCompared with LSTMs, GRUs achieve similar performance but tend to be lighter computationally. Generally, compared with simple RNNs, gated RNNS, just like LSTMs and\nGRUs, can better capture dependencies for sequences with large time step distances. GRUs\ncontain basic RNNs as their extreme case whenever the reset gate is switched on. They can\nalso skip subsequences by turning on the update gate.\n10.2.7 Exercises\n1. Assume that we only want to use the input at time step 𝑡′ to predict the output at time\nstep 𝑡> 𝑡′. What are the best values for the reset and update gates for each time step?"], 'uris': None, 'included': ['metadatas', 'documents'], 'data': None, 'metadatas': [{'page_number': 409}, {'page_number': 410}, {'page_number': 416}, {'page_number': 417}, {'page_number': 421}]} |
