function of some inputs at that time step and its state at the previous time step: h(t) =
f(h(t–1), x(t)). Its output at time step t, denoted y(t), is also a function of the previous
state and the current inputs. In the case of the basic cells we have discussed so far, the
output is simply equal to the state, but in more complex cells this is not always the
case, as shown in Figure 14-3.
Figure 14-3. A cell’s hidden state and its output may be different
Input and Output Sequences
An RNN can simultaneously take a sequence of inputs and produce a sequence of
outputs (see Figure 14-4, top-left network). For example, this type of network is use‐
ful for predicting time series such as stock prices: you feed it the prices over the last N
days, and it must output the prices shifted by one day into the future (i.e., from N – 1
days ago to tomorrow).
Alternatively, you could feed the network a sequence of inputs, and ignore all outputs


Alternatively, you could feed the network a sequence of inputs, and ignore all outputs
except for the last one (see the top-right network). In other words, this is a sequence-
to-vector network. For example, you could feed the network a sequence of words cor‐
382
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


responding to a movie review, and the network would output a sentiment score (e.g.,
from –1 [hate] to +1 [love]).
Conversely, you could feed the network a single input at the first time step (and zeros
for all other time steps), and let it output a sequence (see the bottom-left network).
This is a vector-to-sequence network. For example, the input could be an image, and
the output could be a caption for that image.
Lastly, you could have a sequence-to-vector network, called an encoder, followed by a
vector-to-sequence network, called a decoder (see the bottom-right network). For
example, this can be used for translating a sentence from one language to another.
You would feed the network a sentence in one language, the encoder would convert
this sentence into a single vector representation, and then the decoder would decode
this vector into a sentence in another language. This two-step model, called an


this vector into a sentence in another language. This two-step model, called an
Encoder–Decoder, works much better than trying to translate on the fly with a single
sequence-to-sequence RNN (like the one represented on the top left), since the last
words of a sentence can affect the first words of the translation, so you need to wait
until you have heard the whole sentence before translating it.
Figure 14-4. Seq to seq (top left), seq to vector (top right), vector to seq (bottom left),
delayed seq to seq (bottom right)
Sounds promising, so let’s start coding!
Recurrent Neurons
|
383
Download from finelybook www.finelybook.com


Basic RNNs in TensorFlow
First, let’s implement a very simple RNN model, without using any of TensorFlow’s
RNN operations, to better understand what goes on under the hood. We will create
an RNN composed of a layer of five recurrent neurons (like the RNN represented in
Figure 14-2), using the tanh activation function. We will assume that the RNN runs
over only two time steps, taking input vectors of size 3 at each time step. The follow‐
ing code builds this RNN, unrolled through two time steps:
n_inputs = 3
n_neurons = 5
X0 = tf.placeholder(tf.float32, [None, n_inputs])
X1 = tf.placeholder(tf.float32, [None, n_inputs])
Wx = tf.Variable(tf.random_normal(shape=[n_inputs, n_neurons],dtype=tf.float32))
Wy = tf.Variable(tf.random_normal(shape=[n_neurons,n_neurons],dtype=tf.float32))
b = tf.Variable(tf.zeros([1, n_neurons], dtype=tf.float32))
Y0 = tf.tanh(tf.matmul(X0, Wx) + b)
Y1 = tf.tanh(tf.matmul(Y0, Wy) + tf.matmul(X1, Wx) + b)
init = tf.global_variables_initializer()


Y1 = tf.tanh(tf.matmul(Y0, Wy) + tf.matmul(X1, Wx) + b)
init = tf.global_variables_initializer()
This network looks much like a two-layer feedforward neural network, with a few
twists: first, the same weights and bias terms are shared by both layers, and second,
we feed inputs at each layer, and we get outputs from each layer. To run the model, we
need to feed it the inputs at both time steps, like so:
import numpy as np
# Mini-batch:        instance 0,instance 1,instance 2,instance 3
X0_batch = np.array([[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 0, 1]]) # t = 0
X1_batch = np.array([[9, 8, 7], [0, 0, 0], [6, 5, 4], [3, 2, 1]]) # t = 1
with tf.Session() as sess:
    init.run()
    Y0_val, Y1_val = sess.run([Y0, Y1], feed_dict={X0: X0_batch, X1: X1_batch})
This mini-batch contains four instances, each with an input sequence composed of
exactly two inputs. At the end, Y0_val and Y1_val contain the outputs of the network
at both time steps for all neurons and all instances in the mini-batch:


at both time steps for all neurons and all instances in the mini-batch:
>>> print(Y0_val)  # output at t = 0
[[-0.2964572   0.82874775 -0.34216955 -0.75720584  0.19011548]  # instance 0
 [-0.12842922  0.99981797  0.84704727 -0.99570125  0.38665548]  # instance 1
 [ 0.04731077  0.99999976  0.99330056 -0.999933    0.55339795]  # instance 2
 [ 0.70323634  0.99309105  0.99909431 -0.85363263  0.7472108 ]] # instance 3
>>> print(Y1_val)  # output at t = 1
[[ 0.51955646  1.          0.99999022 -0.99984968 -0.24616946]  # instance 0
 [-0.70553327 -0.11918639  0.48885304  0.08917919 -0.26579669]  # instance 1
384
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


[-0.32477224  0.99996376  0.99933046 -0.99711186  0.10981458]  # instance 2
 [-0.43738723  0.91517633  0.97817528 -0.91763324  0.11047263]] # instance 3
That wasn’t too hard, but of course if you want to be able to run an RNN over 100
time steps, the graph is going to be pretty big. Now let’s look at how to create the
same model using TensorFlow’s RNN operations.
Static Unrolling Through Time
The static_rnn() function creates an unrolled RNN network by chaining cells. The
following code creates the exact same model as the previous one:
X0 = tf.placeholder(tf.float32, [None, n_inputs])
X1 = tf.placeholder(tf.float32, [None, n_inputs])
basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
output_seqs, states = tf.contrib.rnn.static_rnn(
                          basic_cell, [X0, X1], dtype=tf.float32)
Y0, Y1 = output_seqs
First we create the input placeholders, as before. Then we create a BasicRNNCell,


First we create the input placeholders, as before. Then we create a BasicRNNCell,
which you can think of as a factory that creates copies of the cell to build the unrolled
RNN (one for each time step). Then we call static_rnn(), giving it the cell factory
and the input tensors, and telling it the data type of the inputs (this is used to create
the initial state matrix, which by default is full of zeros). The static_rnn() function
calls the cell factory’s __call__() function once per input, creating two copies of the
cell (each containing a layer of five recurrent neurons), with shared weights and bias
terms, and it chains them just like we did earlier. The static_rnn() function returns
two objects. The first is a Python list containing the output tensors for each time step.
The second is a tensor containing the final states of the network. When you are using
basic cells, the final state is simply equal to the last output.


basic cells, the final state is simply equal to the last output.
If there were 50 time steps, it would not be very convenient to have to define 50 input
placeholders and 50 output tensors. Moreover, at execution time you would have to
feed each of the 50 placeholders and manipulate the 50 outputs. Let’s simplify this.
The following code builds the same RNN again, but this time it takes a single input
placeholder of shape [None, n_steps, n_inputs] where the first dimension is the
mini-batch size. Then it extracts the list of input sequences for each time step. X_seqs
is a Python list of n_steps tensors of shape [None, n_inputs], where once again the
first dimension is the mini-batch size. To do this, we first swap the first two dimen‐
sions using the transpose() function, so that the time steps are now the first dimen‐
sion. Then we extract a Python list of tensors along the first dimension (i.e., one
tensor per time step) using the unstack() function. The next two lines are the same


tensor per time step) using the unstack() function. The next two lines are the same
as before. Finally, we merge all the output tensors into a single tensor using the
stack() function, and we swap the first two dimensions to get a final outputs tensor
Basic RNNs in TensorFlow
|
385
Download from finelybook www.finelybook.com


of shape [None, n_steps, n_neurons] (again the first dimension is the mini-batch
size).
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
X_seqs = tf.unstack(tf.transpose(X, perm=[1, 0, 2]))
basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
output_seqs, states = tf.contrib.rnn.static_rnn(
                          basic_cell, X_seqs, dtype=tf.float32)
outputs = tf.transpose(tf.stack(output_seqs), perm=[1, 0, 2])
Now we can run the network by feeding it a single tensor that contains all the mini-
batch sequences:
X_batch = np.array([
         # t = 0     t = 1
        [[0, 1, 2], [9, 8, 7]], # instance 0
        [[3, 4, 5], [0, 0, 0]], # instance 1
        [[6, 7, 8], [6, 5, 4]], # instance 2
        [[9, 0, 1], [3, 2, 1]], # instance 3
    ])
with tf.Session() as sess:
    init.run()
    outputs_val = outputs.eval(feed_dict={X: X_batch})
And we get a single outputs_val tensor for all instances, all time steps, and all neu‐
rons:
>>> print(outputs_val)


rons:
>>> print(outputs_val)
[[[-0.2964572   0.82874775 -0.34216955 -0.75720584  0.19011548]
  [ 0.51955646  1.          0.99999022 -0.99984968 -0.24616946]]
 [[-0.12842922  0.99981797  0.84704727 -0.99570125  0.38665548]
  [-0.70553327 -0.11918639  0.48885304  0.08917919 -0.26579669]]
 [[ 0.04731077  0.99999976  0.99330056 -0.999933    0.55339795]
  [-0.32477224  0.99996376  0.99933046 -0.99711186  0.10981458]]
 [[ 0.70323634  0.99309105  0.99909431 -0.85363263  0.7472108 ]
  [-0.43738723  0.91517633  0.97817528 -0.91763324  0.11047263]]]
However, this approach still builds a graph containing one cell per time step. If there
were 50 time steps, the graph would look pretty ugly. It is a bit like writing a program
without ever using loops (e.g., Y0=f(0, X0); Y1=f(Y0, X1); Y2=f(Y1, X2); ...;
Y50=f(Y49, X50)). With such as large graph, you may even get out-of-memory
(OOM) errors during backpropagation (especially with the limited memory of GPU


(OOM) errors during backpropagation (especially with the limited memory of GPU
cards), since it must store all tensor values during the forward pass so it can use them
to compute gradients during the reverse pass.
Fortunately, there is a better solution: the dynamic_rnn() function.
386
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


Dynamic Unrolling Through Time
The dynamic_rnn() function uses a while_loop() operation to run over the cell the
appropriate number of times, and you can set swap_memory=True if you want it to
swap the GPU’s memory to the CPU’s memory during backpropagation to avoid
OOM errors. Conveniently, it also accepts a single tensor for all inputs at every time
step (shape [None, n_steps, n_inputs]) and it outputs a single tensor for all out‐
puts at every time step (shape [None, n_steps, n_neurons]); there is no need to
stack, unstack, or transpose. The following code creates the same RNN as earlier
using the dynamic_rnn() function. It’s so much nicer!
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)
During backpropagation, the while_loop() operation does the
appropriate magic: it stores the tensor values for each iteration dur‐


appropriate magic: it stores the tensor values for each iteration dur‐
ing the forward pass so it can use them to compute gradients dur‐
ing the reverse pass.
Handling Variable Length Input Sequences
So far we have used only fixed-size input sequences (all exactly two steps long). What
if the input sequences have variable lengths (e.g., like sentences)? In this case you
should set the sequence_length parameter when calling the dynamic_rnn() (or
static_rnn()) function; it must be a 1D tensor indicating the length of the input
sequence for each instance. For example:
seq_length = tf.placeholder(tf.int32, [None])
[...]
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32,
                                    sequence_length=seq_length)
For example, suppose the second input sequence contains only one input instead of
two. It must be padded with a zero vector in order to fit in the input tensor X (because


two. It must be padded with a zero vector in order to fit in the input tensor X (because
the input tensor’s second dimension is the size of the longest sequence—i.e., 2).
X_batch = np.array([
        # step 0     step 1
        [[0, 1, 2], [9, 8, 7]], # instance 0
        [[3, 4, 5], [0, 0, 0]], # instance 1 (padded with a zero vector)
        [[6, 7, 8], [6, 5, 4]], # instance 2
        [[9, 0, 1], [3, 2, 1]], # instance 3
    ])
seq_length_batch = np.array([2, 1, 2, 2])
Basic RNNs in TensorFlow
|
387
Download from finelybook www.finelybook.com


Of course, you now need to feed values for both placeholders X and seq_length:
with tf.Session() as sess:
    init.run()
    outputs_val, states_val = sess.run(
        [outputs, states], feed_dict={X: X_batch, seq_length: seq_length_batch})
Now the RNN outputs zero vectors for every time step past the input sequence length
(look at the second instance’s output for the second time step):
>>> print(outputs_val)
[[[-0.2964572   0.82874775 -0.34216955 -0.75720584  0.19011548]
  [ 0.51955646  1.          0.99999022 -0.99984968 -0.24616946]]  # final state
 [[-0.12842922  0.99981797  0.84704727 -0.99570125  0.38665548]   # final state
  [ 0.          0.          0.          0.          0.        ]]  # zero vector
 [[ 0.04731077  0.99999976  0.99330056 -0.999933    0.55339795]
  [-0.32477224  0.99996376  0.99933046 -0.99711186  0.10981458]]  # final state
 [[ 0.70323634  0.99309105  0.99909431 -0.85363263  0.7472108 ]


[[ 0.70323634  0.99309105  0.99909431 -0.85363263  0.7472108 ]
  [-0.43738723  0.91517633  0.97817528 -0.91763324  0.11047263]]] # final state
Moreover, the states tensor contains the final state of each cell (excluding the zero
vectors):
>>> print(states_val)
[[ 0.51955646  1.          0.99999022 -0.99984968 -0.24616946]    # t = 1
 [-0.12842922  0.99981797  0.84704727 -0.99570125  0.38665548]    # t = 0 !!!
 [-0.32477224  0.99996376  0.99933046 -0.99711186  0.10981458]    # t = 1
 [-0.43738723  0.91517633  0.97817528 -0.91763324  0.11047263]]   # t = 1
Handling Variable-Length Output Sequences
What if the output sequences have variable lengths as well? If you know in advance
what length each sequence will have (for example if you know that it will be the same
length as the input sequence), then you can set the sequence_length parameter as
described above. Unfortunately, in general this will not be possible: for example, the


described above. Unfortunately, in general this will not be possible: for example, the
length of a translated sentence is generally different from the length of the input sen‐
tence. In this case, the most common solution is to define a special output called an
end-of-sequence token (EOS token). Any output past the EOS should be ignored (we
will discuss this later in this chapter).
Okay, now you know how to build an RNN network (or more precisely an RNN net‐
work unrolled through time). But how do you train it?
388
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


Training RNNs
To train an RNN, the trick is to unroll it through time (like we just did) and then
simply use regular backpropagation (see Figure 14-5). This strategy is called backpro‐
pagation through time (BPTT).
Figure 14-5. Backpropagation through time
Just like in regular backpropagation, there is a first forward pass through the unrolled
network (represented by the dashed arrows); then the output sequence is evaluated
using a cost function C �tmin
, �tmin + 1 , ⋯, �tmax
 (where tmin and tmax are the first
and last output time steps, not counting the ignored outputs), and the gradients of
that cost function are propagated backward through the unrolled network (repre‐
sented by the solid arrows); and finally the model parameters are updated using the
gradients computed during BPTT. Note that the gradients flow backward through all
the outputs used by the cost function, not just through the final output (for example,


the outputs used by the cost function, not just through the final output (for example,
in Figure 14-5 the cost function is computed using the last three outputs of the net‐
work, Y(2), Y(3), and Y(4), so gradients flow through these three outputs, but not
through Y(0) and Y(1)). Moreover, since the same parameters W and b are used at each
time step, backpropagation will do the right thing and sum over all time steps.
Training a Sequence Classifier
Let’s train an RNN to classify MNIST images. A convolutional neural network would
be better suited for image classification (see Chapter 13), but this makes for a simple
example that you are already familiar with. We will treat each image as a sequence of
28 rows of 28 pixels each (since each MNIST image is 28 × 28 pixels). We will use
cells of 150 recurrent neurons, plus a fully connected layer containing 10 neurons
Training RNNs
|
389
Download from finelybook www.finelybook.com


(one per class) connected to the output of the last time step, followed by a softmax
layer (see Figure 14-6).
Figure 14-6. Sequence classifier
The construction phase is quite straightforward; it’s pretty much the same as the
MNIST classifier we built in Chapter 10 except that an unrolled RNN replaces the
hidden layers. Note that the fully connected layer is connected to the states tensor,
which contains only the final state of the RNN (i.e., the 28th output). Also note that y
is a placeholder for the target classes.
from tensorflow.contrib.layers import fully_connected
n_steps = 28
n_inputs = 28
n_neurons = 150
n_outputs = 10
learning_rate = 0.001
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.int32, [None])
basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
outputs, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)
logits = fully_connected(states, n_outputs, activation_fn=None)


logits = fully_connected(states, n_outputs, activation_fn=None)
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(
               labels=y, logits=logits)
loss = tf.reduce_mean(xentropy)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(loss)
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
390
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


init = tf.global_variables_initializer()
Now let’s load the MNIST data and reshape the test data to [batch_size, n_steps,
n_inputs] as is expected by the network. We will take care of reshaping the training
data in a moment.
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/")
X_test = mnist.test.images.reshape((-1, n_steps, n_inputs))
y_test = mnist.test.labels
Now we are ready to train the RNN. The execution phase is exactly the same as for
the MNIST classifier in Chapter 10, except that we reshape each training batch before
feeding it to the network.
n_epochs = 100
batch_size = 150
with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            X_batch = X_batch.reshape((-1, n_steps, n_inputs))
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})


sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_test = accuracy.eval(feed_dict={X: X_test, y: y_test})
        print(epoch, "Train accuracy:", acc_train, "Test accuracy:", acc_test)
The output should look like this:
0 Train accuracy: 0.713333 Test accuracy: 0.7299
1 Train accuracy: 0.766667 Test accuracy: 0.7977
...
98 Train accuracy: 0.986667 Test accuracy: 0.9777
99 Train accuracy: 0.986667 Test accuracy: 0.9809
We get over 98% accuracy—not bad! Plus you would certainly get a better result by
tuning the hyperparameters, initializing the RNN weights using He initialization,
training longer, or adding a bit of regularization (e.g., dropout).
You can specify an initializer for the RNN by wrapping its
construction
code
in
a
variable
scope
(e.g.,
use
variable_scope("rnn", initializer=variance_scaling_ini
tializer()) to use He initialization).
Training RNNs
|
391


tializer()) to use He initialization).
Training RNNs
|
391
Download from finelybook www.finelybook.com


Training to Predict Time Series
Now let’s take a look at how to handle time series, such as stock prices, air tempera‐
ture, brain wave patterns, and so on. In this section we will train an RNN to predict
the next value in a generated time series. Each training instance is a randomly
selected sequence of 20 consecutive values from the time series, and the target
sequence is the same as the input sequence, except it is shifted by one time step into
the future (see Figure 14-7).
Figure 14-7. Time series (left), and a training instance from that series (right)
First, let’s create the RNN. It will contain 100 recurrent neurons and we will unroll it
over 20 time steps since each training instance will be 20 inputs long. Each input will
contain only one feature (the value at that time). The targets are also sequences of 20
inputs, each containing a single value. The code is almost the same as earlier:
n_steps = 20
n_inputs = 1
n_neurons = 100
n_outputs = 1


n_steps = 20
n_inputs = 1
n_neurons = 100
n_outputs = 1
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.float32, [None, n_steps, n_outputs])
cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons, activation=tf.nn.relu)
outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)
In general you would have more than just one input feature. For
example, if you were trying to predict stock prices, you would
likely have many other input features at each time step, such as pri‐
ces of competing stocks, ratings from analysts, or any other feature
that might help the system make its predictions.
At each time step we now have an output vector of size 100. But what we actually
want is a single output value at each time step. The simplest solution is to wrap the
cell in an OutputProjectionWrapper. A cell wrapper acts like a normal cell, proxying
392
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


every method call to an underlying cell, but it also adds some functionality. The Out
putProjectionWrapper adds a fully connected layer of linear neurons (i.e., without
any activation function) on top of each output (but it does not affect the cell state).
All these fully connected layers share the same (trainable) weights and bias terms.
The resulting RNN is represented in Figure 14-8.
Figure 14-8. RNN cells using output projections
Wrapping a cell is quite easy. Let’s tweak the preceding code by wrapping the
BasicRNNCell into an OutputProjectionWrapper:
cell = tf.contrib.rnn.OutputProjectionWrapper(
    tf.contrib.rnn.BasicRNNCell(num_units=n_neurons, activation=tf.nn.relu),
    output_size=n_outputs)
So far, so good. Now we need to define the cost function. We will use the Mean
Squared Error (MSE), as we did in previous regression tasks. Next we will create an
Adam optimizer, the training op, and the variable initialization op, as usual:
learning_rate = 0.001


learning_rate = 0.001
loss = tf.reduce_mean(tf.square(outputs - y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
Now on to the execution phase:
n_iterations = 10000
batch_size = 50
with tf.Session() as sess:
Training RNNs
|
393
Download from finelybook www.finelybook.com


init.run()
    for iteration in range(n_iterations):
        X_batch, y_batch = [...]  # fetch the next training batch
        sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        if iteration % 100 == 0:
            mse = loss.eval(feed_dict={X: X_batch, y: y_batch})
            print(iteration, "\tMSE:", mse)
The program’s output should look like this:
0    MSE: 379.586
100  MSE: 14.58426
200  MSE: 7.14066
300  MSE: 3.98528
400  MSE: 2.00254
[...]
Once the model is trained, you can make predictions:
X_new = [...]  # New sequences
y_pred = sess.run(outputs, feed_dict={X: X_new})
Figure 14-9 shows the predicted sequence for the instance we looked at earlier (in
Figure 14-7), after just 1,000 training iterations.
Figure 14-9. Time series prediction
Although using an OutputProjectionWrapper is the simplest solution to reduce the
dimensionality of the RNN’s output sequences down to just one value per time step


dimensionality of the RNN’s output sequences down to just one value per time step
(per instance), it is not the most efficient. There is a trickier but more efficient solu‐
tion: you can reshape the RNN outputs from [batch_size, n_steps, n_neurons]
to [batch_size * n_steps, n_neurons], then apply a single fully connected layer
with the appropriate output size (in our case just 1), which will result in an output
tensor of shape [batch_size * n_steps, n_outputs], and then reshape this tensor
394
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


to [batch_size, n_steps, n_outputs]. These operations are represented in
Figure 14-10.
Figure 14-10. Stack all the outputs, apply the projection, then unstack the result
To implement this solution, we first revert to a basic cell, without the OutputProjec
tionWrapper:
cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons, activation=tf.nn.relu)
rnn_outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)
Then we stack all the outputs using the reshape() operation, apply the fully connec‐
ted linear layer (without using any activation function; this is just a projection), and
finally unstack all the outputs, again using reshape():
stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons])
stacked_outputs = fully_connected(stacked_rnn_outputs, n_outputs,
                                  activation_fn=None)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
Training RNNs
|
395
Download from finelybook www.finelybook.com


The rest of the code is the same as earlier. This can provide a significant speed boost
since there is just one fully connected layer instead of one per time step.
Creative RNN
Now that we have a model that can predict the future, we can use it to generate some
creative sequences, as explained at the beginning of the chapter. All we need is to pro‐
vide it a seed sequence containing n_steps values (e.g., full of zeros), use the model to
predict the next value, append this predicted value to the sequence, feed the last
n_steps values to the model to predict the next value, and so on. This process gener‐
ates a new sequence that has some resemblance to the original time series (see
Figure 14-11).
sequence = [0.] * n_steps
for iteration in range(300):
    X_batch = np.array(sequence[-n_steps:]).reshape(1, n_steps, 1)
    y_pred = sess.run(outputs, feed_dict={X: X_batch})
    sequence.append(y_pred[0, -1, 0])


y_pred = sess.run(outputs, feed_dict={X: X_batch})
    sequence.append(y_pred[0, -1, 0])
Figure 14-11. Creative sequences, seeded with zeros (left) or with an instance (right)
Now you can try to feed all your John Lennon albums to an RNN and see if it can
generate the next “Imagine.” However, you will probably need a much more powerful
RNN, with more neurons, and also much deeper. Let’s look at deep RNNs now.
Deep RNNs
It is quite common to stack multiple layers of cells, as shown in Figure 14-12. This
gives you a deep RNN.
396
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


Figure 14-12. Deep RNN (left), unrolled through time (right)
To implement a deep RNN in TensorFlow, you can create several cells and stack them
into a MultiRNNCell. In the following code we stack three identical cells (but you
could very well use various kinds of cells with a different number of neurons):
n_neurons = 100
n_layers = 3
basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
multi_layer_cell = tf.contrib.rnn.MultiRNNCell([basic_cell] * n_layers)
outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
That’s all there is to it! The states variable is a tuple containing one tensor per layer,
each representing the final state of that layer’s cell (with shape [batch_size, n_neu
rons]). If you set state_is_tuple=False when creating the MultiRNNCell, then
states becomes a single tensor containing the states from every layer, concatenated
along the column axis (i.e., its shape is [batch_size, n_layers * n_neurons]).


along the column axis (i.e., its shape is [batch_size, n_layers * n_neurons]).
Note that before TensorFlow 0.11.0, this behavior was the default.
Distributing a Deep RNN Across Multiple GPUs
Chapter 12 pointed out that we can efficiently distribute deep RNNs across multiple
GPUs by pinning each layer to a different GPU (see Figure 12-16). However, if you
try to create each cell in a different device() block, it will not work:
with tf.device("/gpu:0"):  # BAD! This is ignored.
    layer1 = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
with tf.device("/gpu:1"):  # BAD! Ignored again.
    layer2 = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
This fails because a BasicRNNCell is a cell factory, not a cell per se (as mentioned ear‐
lier); no cells get created when you create the factory, and thus no variables do either.
Deep RNNs
|
397
Download from finelybook www.finelybook.com


2 This uses the decorator design pattern.
The device block is simply ignored. The cells actually get created later. When you call
dynamic_rnn(), it calls the MultiRNNCell, which calls each individual BasicRNNCell,
which create the actual cells (including their variables). Unfortunately, none of these
classes provide any way to control the devices on which the variables get created. If
you try to put the dynamic_rnn() call within a device block, the whole RNN gets pin‐
ned to a single device. So are you stuck? Fortunately not! The trick is to create your
own cell wrapper:
import tensorflow as tf
class DeviceCellWrapper(tf.contrib.rnn.RNNCell):
  def __init__(self, device, cell):
    self._cell = cell
    self._device = device
  @property
  def state_size(self):
    return self._cell.state_size
  @property
  def output_size(self):
    return self._cell.output_size
  def __call__(self, inputs, state, scope=None):
    with tf.device(self._device):


def __call__(self, inputs, state, scope=None):
    with tf.device(self._device):
        return self._cell(inputs, state, scope)
This wrapper simply proxies every method call to another cell, except it wraps the
__call__() function within a device block.2 Now you can distribute each layer on a
different GPU:
devices = ["/gpu:0", "/gpu:1", "/gpu:2"]
cells = [DeviceCellWrapper(dev,tf.contrib.rnn.BasicRNNCell(num_units=n_neurons))
         for dev in devices]
multi_layer_cell = tf.contrib.rnn.MultiRNNCell(cells)
outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
Do not set state_is_tuple=False, or the MultiRNNCell will con‐
catenate all the cell states into a single tensor, on a single GPU.
398
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


Applying Dropout
If you build a very deep RNN, it may end up overfitting the training set. To prevent
that, a common technique is to apply dropout (introduced in Chapter 11). You can
simply add a dropout layer before or after the RNN as usual, but if you also want to
apply dropout between the RNN layers, you need to use a DropoutWrapper. The fol‐
lowing code applies dropout to the inputs of each layer in the RNN, dropping each
input with a 50% probability:
keep_prob = 0.5
cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
cell_drop = tf.contrib.rnn.DropoutWrapper(cell, input_keep_prob=keep_prob)
multi_layer_cell = tf.contrib.rnn.MultiRNNCell([cell_drop] * n_layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
Note that it is also possible to apply dropout to the outputs by setting out
put_keep_prob.
The main problem with this code is that it will apply dropout not only during train‐


put_keep_prob.
The main problem with this code is that it will apply dropout not only during train‐
ing but also during testing, which is not what you want (recall that dropout should be
applied only during training). Unfortunately, the DropoutWrapper does not support
an is_training placeholder (yet?), so you must either write your own dropout wrap‐
per class, or have two different graphs: one for training, and the other for testing. The
second option looks like this:
import sys
is_training = (sys.argv[-1] == "train")
X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.float32, [None, n_steps, n_outputs])
cell = tf.contrib.rnn.BasicRNNCell(num_units=n_neurons)
if is_training:
    cell = tf.contrib.rnn.DropoutWrapper(cell, input_keep_prob=keep_prob)
multi_layer_cell = tf.contrib.rnn.MultiRNNCell([cell] * n_layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
[...] # build the rest of the graph


[...] # build the rest of the graph
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session() as sess:
    if is_training:
        init.run()
        for iteration in range(n_iterations):
            [...] # train the model
        save_path = saver.save(sess, "/tmp/my_model.ckpt")
    else:
        saver.restore(sess, "/tmp/my_model.ckpt")
        [...] # use the model
Deep RNNs
|
399
Download from finelybook www.finelybook.com


With that you should be able to train all sorts of RNNs! Unfortunately, if you want to
train an RNN on long sequences, things will get a bit harder. Let’s see why and what
you can do about it.
The Difficulty of Training over Many Time Steps
To train an RNN on long sequences, you will need to run it over many time steps,
making the unrolled RNN a very deep network. Just like any deep neural network it
may suffer from the vanishing/exploding gradients problem (discussed in Chap‐
ter 11) and take forever to train. Many of the tricks we discussed to alleviate this
problem can be used for deep unrolled RNNs as well: good parameter initialization,
nonsaturating activation functions (e.g., ReLU), Batch Normalization, Gradient Clip‐
ping, and faster optimizers. However, if the RNN needs to handle even moderately
long sequences (e.g., 100 inputs), then training will still be very slow.
The simplest and most common solution to this problem is to unroll the RNN only


The simplest and most common solution to this problem is to unroll the RNN only
over a limited number of time steps during training. This is called truncated backpro‐
pagation through time. In TensorFlow you can implement it simply by truncating the
input sequences. For example, in the time series prediction problem, you would sim‐
ply reduce n_steps during training. The problem, of course, is that the model will
not be able to learn long-term patterns. One workaround could be to make sure that
these shortened sequences contain both old and recent data, so that the model can
learn to use both (e.g., the sequence could contain monthly data for the last five
months, then weekly data for the last five weeks, then daily data over the last five
days). But this workaround has its limits: what if fine-grained data from last year is
actually useful? What if there was a brief but significant event that absolutely must be
taken into account, even years later (e.g., the result of an election)?


taken into account, even years later (e.g., the result of an election)?
Besides the long training time, a second problem faced by long-running RNNs is the
fact that the memory of the first inputs gradually fades away. Indeed, due to the trans‐
formations that the data goes through when traversing an RNN, some information is
lost after each time step. After a while, the RNN’s state contains virtually no trace of
the first inputs. This can be a showstopper. For example, say you want to perform
sentiment analysis on a long review that starts with the four words “I loved this
movie,” but the rest of the review lists the many things that could have made the
movie even better. If the RNN gradually forgets the first four words, it will completely
misinterpret the review. To solve this problem, various types of cells with long-term
memory have been introduced. They have proved so successful that the basic cells are


memory have been introduced. They have proved so successful that the basic cells are
not much used anymore. Let’s first look at the most popular of these long memory
cells: the LSTM cell.
400
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


3 “Long Short-Term Memory,” S. Hochreiter and J. Schmidhuber (1997).
4 “Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling,” H.
Sak et al. (2014).
5 “Recurrent Neural Network Regularization,” W. Zaremba et al. (2015).
LSTM Cell
The Long Short-Term Memory (LSTM) cell was proposed in 19973 by Sepp Hochreiter
and Jürgen Schmidhuber, and it was gradually improved over the years by several
researchers, such as Alex Graves, Haşim Sak,4 Wojciech Zaremba,5 and many more. If
you consider the LSTM cell as a black box, it can be used very much like a basic cell,
except it will perform much better; training will converge faster and it will detect
long-term dependencies in the data. In TensorFlow, you can simply use a BasicLSTM
Cell instead of a BasicRNNCell:
lstm_cell = tf.contrib.rnn.BasicLSTMCell(num_units=n_neurons)
LSTM cells manage two state vectors, and for performance reasons they are kept


LSTM cells manage two state vectors, and for performance reasons they are kept
separate by default. You can change this default behavior by setting
state_is_tuple=False when creating the BasicLSTMCell.
So how does an LSTM cell work? The architecture of a basic LSTM cell is shown in
Figure 14-13.
Figure 14-13. LSTM cell
LSTM Cell
|
401
Download from finelybook www.finelybook.com


If you don’t look at what’s inside the box, the LSTM cell looks exactly like a regular
cell, except that its state is split in two vectors: h(t) and c(t) (“c” stands for “cell”). You
can think of h(t) as the short-term state and c(t) as the long-term state.
Now let’s open the box! The key idea is that the network can learn what to store in the
long-term state, what to throw away, and what to read from it. As the long-term state
c(t–1) traverses the network from left to right, you can see that it first goes through a
forget gate, dropping some memories, and then it adds some new memories via the
addition operation (which adds the memories that were selected by an input gate).
The result c(t) is sent straight out, without any further transformation. So, at each time
step, some memories are dropped and some memories are added. Moreover, after the
addition operation, the long-term state is copied and passed through the tanh func‐


addition operation, the long-term state is copied and passed through the tanh func‐
tion, and then the result is filtered by the output gate. This produces the short-term
state h(t) (which is equal to the cell’s output for this time step y(t)). Now let’s look at
where new memories come from and how the gates work.
First, the current input vector x(t) and the previous short-term state h(t–1) are fed to
four different fully connected layers. They all serve a different purpose:
• The main layer is the one that outputs g(t). It has the usual role of analyzing the
current inputs x(t) and the previous (short-term) state h(t–1). In a basic cell, there is
nothing else than this layer, and its output goes straight out to y(t) and h(t). In con‐
trast, in an LSTM cell this layer’s output does not go straight out, but instead it is
partially stored in the long-term state.
• The three other layers are gate controllers. Since they use the logistic activation


• The three other layers are gate controllers. Since they use the logistic activation
function, their outputs range from 0 to 1. As you can see, their outputs are fed to
element-wise multiplication operations, so if they output 0s, they close the gate,
and if they output 1s, they open it. Specifically:
— The forget gate (controlled by f(t)) controls which parts of the long-term state
should be erased.
— The input gate (controlled by i(t)) controls which parts of g(t) should be added
to the long-term state (this is why we said it was only “partially stored”).
— Finally, the output gate (controlled by o(t)) controls which parts of the long-
term state should be read and output at this time step (both to h(t)) and y(t).
In short, an LSTM cell can learn to recognize an important input (that’s the role of the
input gate), store it in the long-term state, learn to preserve it for as long as it is
needed (that’s the role of the forget gate), and learn to extract it whenever it is needed.


needed (that’s the role of the forget gate), and learn to extract it whenever it is needed.
This explains why they have been amazingly successful at capturing long-term pat‐
terns in time series, long texts, audio recordings, and more.
402
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


6 “Recurrent Nets that Time and Count,” F. Gers and J. Schmidhuber (2000).
Equation 14-3 summarizes how to compute the cell’s long-term state, its short-term
state, and its output at each time step for a single instance (the equations for a whole
mini-batch are very similar).
Equation 14-3. LSTM computations
�t = σ �xi
T · �t + �hi
T · �t −1 + �i
�t = σ �xf
T · �t + �hf
T · �t −1 + �f
�t = σ �xo
T · �t + �ho
T · �t −1 + �o
�t = tanh �xg
T · �t + �hg
T · �t −1 + �g
�t = �t ⊗�t −1 + �t ⊗�t
�t = �t = �t ⊗tanh �t
• Wxi, Wxf, Wxo, Wxg are the weight matrices of each of the four layers for their con‐
nection to the input vector x(t).
• Whi, Whf, Who, and Whg are the weight matrices of each of the four layers for their
connection to the previous short-term state h(t–1).
• bi, bf, bo, and bg are the bias terms for each of the four layers. Note that Tensor‐
Flow initializes bf to a vector full of 1s instead of 0s. This prevents forgetting
everything at the beginning of training.


everything at the beginning of training.
Peephole Connections
In a basic LSTM cell, the gate controllers can look only at the input x(t) and the previ‐
ous short-term state h(t–1). It may be a good idea to give them a bit more context by
letting them peek at the long-term state as well. This idea was proposed by Felix Gers
and Jürgen Schmidhuber in 2000.6 They proposed an LSTM variant with extra con‐
nections called peephole connections: the previous long-term state c(t–1) is added as an
input to the controllers of the forget gate and the input gate, and the current long-
term state c(t) is added as input to the controller of the output gate.
To implement peephole connections in TensorFlow, you must use the LSTMCell
instead of the BasicLSTMCell and set use_peepholes=True:
lstm_cell = tf.contrib.rnn.LSTMCell(num_units=n_neurons, use_peepholes=True)
LSTM Cell
|
403
Download from finelybook www.finelybook.com


7 “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” K. Cho
et al. (2014).
8 A 2015 paper by Klaus Greff et al., “LSTM: A Search Space Odyssey,” seems to show that all LSTM variants
perform roughly the same.
There are many other variants of the LSTM cell. One particularly popular variant is
the GRU cell, which we will look at now.
GRU Cell
The Gated Recurrent Unit (GRU) cell (see Figure 14-14) was proposed by Kyunghyun
Cho et al. in a 2014 paper7 that also introduced the Encoder–Decoder network we
mentioned earlier.
Figure 14-14. GRU cell
The GRU cell is a simplified version of the LSTM cell, and it seems to perform just as
well8 (which explains its growing popularity). The main simplifications are:
• Both state vectors are merged into a single vector h(t).
• A single gate controller controls both the forget gate and the input gate. If the
gate controller outputs a 1, the input gate is open and the forget gate is closed. If
404
|


gate controller outputs a 1, the input gate is open and the forget gate is closed. If
404
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


it outputs a 0, the opposite happens. In other words, whenever a memory must
be stored, the location where it will be stored is erased first. This is actually a fre‐
quent variant to the LSTM cell in and of itself.
• There is no output gate; the full state vector is output at every time step. How‐
ever, there is a new gate controller that controls which part of the previous state
will be shown to the main layer.
Equation 14-4 summarizes how to compute the cell’s state at each time step for a sin‐
gle instance.
Equation 14-4. GRU computations
�t = σ �xz
T · �t + �hz
T · �t −1
�t = σ �xr
T · �t + �hr
T · �t −1
�t = tanh �xg
T · �t + �hg
T · �t ⊗�t −1
�t = 1 −�t
⊗tanh �xg
T · �t −1 + �t ⊗�t
Creating a GRU cell in TensorFlow is trivial:
gru_cell = tf.contrib.rnn.GRUCell(num_units=n_neurons)
LSTM or GRU cells are one of the main reasons behind the success of RNNs in recent
years, in particular for applications in natural language processing (NLP).
Natural Language Processing


Natural Language Processing
Most of the state-of-the-art NLP applications, such as machine translation, automatic
summarization, parsing, sentiment analysis, and more, are now based (at least in
part) on RNNs. In this last section, we will take a quick look at what a machine trans‐
lation model looks like. This topic is very well covered by TensorFlow’s awesome
Word2Vec and Seq2Seq tutorials, so you should definitely check them out.
Word Embeddings
Before we start, we need to choose a word representation. One option could be to
represent each word using a one-hot vector. Suppose your vocabulary contains
50,000 words, then the nth word would be represented as a 50,000-dimensional vector,
full of 0s except for a 1 at the nth position. However, with such a large vocabulary, this
sparse representation would not be efficient at all. Ideally, you want similar words to
have similar representations, making it easy for the model to generalize what it learns


have similar representations, making it easy for the model to generalize what it learns
about a word to all similar words. For example, if the model is told that “I drink milk”
is a valid sentence, and if it knows that “milk” is close to “water” but far from “shoes,”
Natural Language Processing
|
405
Download from finelybook www.finelybook.com


9 For more details, check out Christopher Olah’s great post, or Sebastian Ruder’s series of posts.
then it will know that “I drink water” is probably a valid sentence as well, while “I
drink shoes” is probably not. But how can you come up with such a meaningful rep‐
resentation?
The most common solution is to represent each word in the vocabulary using a fairly
small and dense vector (e.g., 150 dimensions), called an embedding, and just let the
neural network learn a good embedding for each word during training. At the begin‐
ning of training, embeddings are simply chosen randomly, but during training, back‐
propagation automatically moves the embeddings around in a way that helps the
neural network perform its task. Typically this means that similar words will gradu‐
ally cluster close to one another, and even end up organized in a rather meaningful
way. For example, embeddings may end up placed along various axes that represent


way. For example, embeddings may end up placed along various axes that represent
gender, singular/plural, adjective/noun, and so on. The result can be truly amazing.9
In TensorFlow, you first need to create the variable representing the embeddings for
every word in your vocabulary (initialized randomly):
vocabulary_size = 50000
embedding_size = 150
embeddings = tf.Variable(
    tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
Now suppose you want to feed the sentence “I drink milk” to your neural network.
You should first preprocess the sentence and break it into a list of known words. For
example you may remove unnecessary characters, replace unknown words by a pre‐
defined token word such as “[UNK]”, replace numerical values by “[NUM]”, replace
URLs by “[URL]”, and so on. Once you have a list of known words, you can look up
each word’s integer identifier (from 0 to 49999) in a dictionary, for example [72, 3335,


each word’s integer identifier (from 0 to 49999) in a dictionary, for example [72, 3335,
288]. At that point, you are ready to feed these word identifiers to TensorFlow using a
placeholder, and apply the embedding_lookup() function to get the corresponding
embeddings:
train_inputs = tf.placeholder(tf.int32, shape=[None])  # from ids...
embed = tf.nn.embedding_lookup(embeddings, train_inputs)  # ...to embeddings
Once your model has learned good word embeddings, they can actually be reused
fairly efficiently in any NLP application: after all, “milk” is still close to “water” and far
from “shoes” no matter what your application is. In fact, instead of training your own
word embeddings, you may want to download pretrained word embeddings. Just like
when reusing pretrained layers (see Chapter 11), you can choose to freeze the pre‐
trained embeddings (e.g., creating the embeddings variable using trainable=False)


trained embeddings (e.g., creating the embeddings variable using trainable=False)
or let backpropagation tweak them for your application. The first option will speed
up training, but the second may lead to slightly higher performance.
406
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


10 “Sequence to Sequence learning with Neural Networks,” I. Sutskever et al. (2014).
Embeddings are also useful for representing categorical attributes
that can take on a large number of different values, especially when
there are complex similarities between values. For example, con‐
sider professions, hobbies, dishes, species, brands, and so on.
You now have almost all the tools you need to implement a machine translation sys‐
tem. Let’s look at this now.
An Encoder–Decoder Network for Machine Translation
Let’s take a look at a simple machine translation model10 that will translate English
sentences to French (see Figure 14-15).
Figure 14-15. A simple machine translation model
The English sentences are fed to the encoder, and the decoder outputs the French
translations. Note that the French translations are also used as inputs to the decoder,
but pushed back by one step. In other words, the decoder is given as input the word


but pushed back by one step. In other words, the decoder is given as input the word
that it should have output at the previous step (regardless of what it actually output).
For the very first word, it is given a token that represents the beginning of the sen‐
Natural Language Processing
|
407
Download from finelybook www.finelybook.com


tence (e.g., “<go>”). The decoder is expected to end the sentence with an end-of-
sequence (EOS) token (e.g., “<eos>”).
Note that the English sentences are reversed before they are fed to the encoder. For
example “I drink milk” is reversed to “milk drink I.” This ensures that the beginning
of the English sentence will be fed last to the encoder, which is useful because that’s
generally the first thing that the decoder needs to translate.
Each word is initially represented by a simple integer identifier (e.g., 288 for the word
“milk”). Next, an embedding lookup returns the word embedding (as explained ear‐
lier, this is a dense, fairly low-dimensional vector). These word embeddings are what
is actually fed to the encoder and the decoder.
At each step, the decoder outputs a score for each word in the output vocabulary (i.e.,
French), and then the Softmax layer turns these scores into probabilities. For exam‐


French), and then the Softmax layer turns these scores into probabilities. For exam‐
ple, at the first step the word “Je” may have a probability of 20%, “Tu” may have a
probability of 1%, and so on. The word with the highest probability is output. This is
very much like a regular classification task, so you can train the model using the soft
max_cross_entropy_with_logits() function.
Note that at inference time (after training), you will not have the target sentence to
feed to the decoder. Instead, simply feed the decoder the word that it output at the
previous step, as shown in Figure 14-16 (this will require an embedding lookup that
is not shown on the diagram).
Figure 14-16. Feeding the previous output word as input at inference time
Okay, now you have the big picture. However, if you go through TensorFlow’s
sequence-to-sequence tutorial and you look at the code in rnn/translate/
seq2seq_model.py (in the TensorFlow models), you will notice a few important differ‐
ences:
408
|


seq2seq_model.py (in the TensorFlow models), you will notice a few important differ‐
ences:
408
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


11 The bucket sizes used in the tutorial are different.
12 “On Using Very Large Target Vocabulary for Neural Machine Translation,” S. Jean et al. (2015).
13 “Neural Machine Translation by Jointly Learning to Align and Translate,” D. Bahdanau et al. (2014).
14 “Long Short-Term Memory-Networks for Machine Reading,” J. Cheng (2016).
15 “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” K. Xu et al. (2015).
• First, so far we have assumed that all input sequences (to the encoder and to the
decoder) have a constant length. But obviously sentence lengths may vary. There
are several ways that this can be handled—for example, using the
sequence_length argument to the static_rnn() or dynamic_rnn() functions to
specify each sentence’s length (as discussed earlier). However, another approach
is used in the tutorial (presumably for performance reasons): sentences are grou‐
ped into buckets of similar lengths (e.g., a bucket for the 1- to 6-word sentences,


ped into buckets of similar lengths (e.g., a bucket for the 1- to 6-word sentences,
another for the 7- to 12-word sentences, and so on11), and the shorter sentences
are padded using a special padding token (e.g., “<pad>”). For example “I drink
milk” becomes “<pad> <pad> <pad> milk drink I”, and its translation becomes
“Je bois du lait <eos> <pad>”. Of course, we want to ignore any output past the
EOS token. For this, the tutorial’s implementation uses a target_weights vector.
For example, for the target sentence “Je bois du lait <eos> <pad>”, the weights
would be set to [1.0, 1.0, 1.0, 1.0, 1.0, 0.0] (notice the weight 0.0 that
corresponds to the padding token in the target sentence). Simply multiplying the
losses by the target weights will zero out the losses that correspond to words past
EOS tokens.
• Second, when the output vocabulary is large (which is the case here), outputting
a probability for each and every possible word would be terribly slow. If the tar‐


a probability for each and every possible word would be terribly slow. If the tar‐
get vocabulary contains, say, 50,000 French words, then the decoder would out‐
put 50,000-dimensional vectors, and then computing the softmax function over
such a large vector would be very computationally intensive. To avoid this, one
solution is to let the decoder output much smaller vectors, such as 1,000-
dimensional vectors, then use a sampling technique to estimate the loss without
having to compute it over every single word in the target vocabulary. This Sam‐
pled Softmax technique was introduced in 2015 by Sébastien Jean et al.12 In Ten‐
sorFlow you can use the sampled_softmax_loss() function.
• Third, the tutorial’s implementation uses an attention mechanism that lets the
decoder peek into the input sequence. Attention augmented RNNs are beyond
the scope of this book, but if you are interested there are helpful papers about


the scope of this book, but if you are interested there are helpful papers about
machine translation,13 machine reading,14 and image captions15 using attention.
• Finally, the tutorial’s implementation makes use of the tf.nn.legacy_seq2seq
module, which provides tools to build various Encoder–Decoder models easily.
Natural Language Processing
|
409
Download from finelybook www.finelybook.com


For example, the embedding_rnn_seq2seq() function creates a simple Encoder–
Decoder model that automatically takes care of word embeddings for you, just
like the one represented in Figure 14-15. This code will likely be updated quickly
to use the new tf.nn.seq2seq module.
You now have all the tools you need to understand the sequence-to-sequence tutor‐
ial’s implementation. Check it out and train your own English-to-French translator!
Exercises
1. Can you think of a few applications for a sequence-to-sequence RNN? What
about a sequence-to-vector RNN? And a vector-to-sequence RNN?
2. Why do people use encoder–decoder RNNs rather than plain sequence-to-
sequence RNNs for automatic translation?
3. How could you combine a convolutional neural network with an RNN to classify
videos?
4. What are the advantages of building an RNN using dynamic_rnn() rather than
static_rnn()?
5. How can you deal with variable-length input sequences? What about variable-
length output sequences?


length output sequences?
6. What is a common way to distribute training and execution of a deep RNN
across multiple GPUs?
7. Embedded Reber grammars were used by Hochreiter and Schmidhuber in their
paper about LSTMs. They are artificial grammars that produce strings such as
“BPBTSXXVPSEPE.” Check out Jenny Orr’s nice introduction to this topic.
Choose a particular embedded Reber grammar (such as the one represented on
Jenny Orr’s page), then train an RNN to identify whether a string respects that
grammar or not. You will first need to write a function capable of generating a
training batch containing about 50% strings that respect the grammar, and 50%
that don’t.
8. Tackle the “How much did it rain? II” Kaggle competition. This is a time series
prediction task: you are given snapshots of polarimetric radar values and asked to
predict the hourly rain gauge total. Luis Andre Dutra e Silva’s interview gives


predict the hourly rain gauge total. Luis Andre Dutra e Silva’s interview gives
some interesting insights into the techniques he used to reach second place in the
competition. In particular, he used an RNN composed of two LSTM layers.
9. Go through TensorFlow’s Word2Vec tutorial to create word embeddings, and
then go through the Seq2Seq tutorial to train an English-to-French translation
system.
Solutions to these exercises are available in Appendix A.
410
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


CHAPTER 15
Autoencoders
Autoencoders are artificial neural networks capable of learning efficient representa‐
tions of the input data, called codings, without any supervision (i.e., the training set is
unlabeled). These codings typically have a much lower dimensionality than the input
data, making autoencoders useful for dimensionality reduction (see Chapter 8). More
importantly, autoencoders act as powerful feature detectors, and they can be used for
unsupervised pretraining of deep neural networks (as we discussed in Chapter 11).
Lastly, they are capable of randomly generating new data that looks very similar to the
training data; this is called a generative model. For example, you could train an
autoencoder on pictures of faces, and it would then be able to generate new faces.
Surprisingly, autoencoders work by simply learning to copy their inputs to their out‐
puts. This may sound like a trivial task, but we will see that constraining the network


puts. This may sound like a trivial task, but we will see that constraining the network
in various ways can make it rather difficult. For example, you can limit the size of the
internal representation, or you can add noise to the inputs and train the network to
recover the original inputs. These constraints prevent the autoencoder from trivially
copying the inputs directly to the outputs, which forces it to learn efficient ways of
representing the data. In short, the codings are byproducts of the autoencoder’s
attempt to learn the identity function under some constraints.
In this chapter we will explain in more depth how autoencoders work, what types of
constraints can be imposed, and how to implement them using TensorFlow, whether
it is for dimensionality reduction, feature extraction, unsupervised pretraining, or as
generative models.
411
Download from finelybook www.finelybook.com


1 “Perception in chess,” W. Chase and H. Simon (1973).
Efficient Data Representations
Which of the following number sequences do you find the easiest to memorize?
• 40, 27, 25, 36, 81, 57, 10, 73, 19, 68
• 50, 25, 76, 38, 19, 58, 29, 88, 44, 22, 11, 34, 17, 52, 26, 13, 40, 20
At first glance, it would seem that the first sequence should be easier, since it is much
shorter. However, if you look carefully at the second sequence, you may notice that it
follows two simple rules: even numbers are followed by their half, and odd numbers
are followed by their triple plus one (this is a famous sequence known as the hailstone
sequence). Once you notice this pattern, the second sequence becomes much easier to
memorize than the first because you only need to memorize the two rules, the first
number, and the length of the sequence. Note that if you could quickly and easily
memorize very long sequences, you would not care much about the existence of a


memorize very long sequences, you would not care much about the existence of a
pattern in the second sequence. You would just learn every number by heart, and that
would be that. It is the fact that it is hard to memorize long sequences that makes it
useful to recognize patterns, and hopefully this clarifies why constraining an autoen‐
coder during training pushes it to discover and exploit patterns in the data.
The relationship between memory, perception, and pattern matching was famously
studied by William Chase and Herbert Simon in the early 1970s.1 They observed that
expert chess players were able to memorize the positions of all the pieces in a game by
looking at the board for just 5 seconds, a task that most people would find impossible.
However, this was only the case when the pieces were placed in realistic positions
(from actual games), not when the pieces were placed randomly. Chess experts don’t


(from actual games), not when the pieces were placed randomly. Chess experts don’t
have a much better memory than you and I, they just see chess patterns more easily
thanks to their experience with the game. Noticing patterns helps them store infor‐
mation efficiently.
Just like the chess players in this memory experiment, an autoencoder looks at the
inputs, converts them to an efficient internal representation, and then spits out some‐
thing that (hopefully) looks very close to the inputs. An autoencoder is always com‐
posed of two parts: an encoder (or recognition network) that converts the inputs to an
internal representation, followed by a decoder (or generative network) that converts
the internal representation to the outputs (see Figure 15-1).
As you can see, an autoencoder typically has the same architecture as a Multi-Layer
Perceptron (MLP; see Chapter 10), except that the number of neurons in the output


Perceptron (MLP; see Chapter 10), except that the number of neurons in the output
layer must be equal to the number of inputs. In this example, there is just one hidden
412
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


layer composed of two neurons (the encoder), and one output layer composed of
three neurons (the decoder). The outputs are often called the reconstructions since the
autoencoder tries to reconstruct the inputs, and the cost function contains a recon‐
struction loss that penalizes the model when the reconstructions are different from the
inputs.
Figure 15-1. The chess memory experiment (left) and a simple autoencoder (right)
Because the internal representation has a lower dimensionality than the input data (it
is 2D instead of 3D), the autoencoder is said to be undercomplete. An undercomplete
autoencoder cannot trivially copy its inputs to the codings, yet it must find a way to
output a copy of its inputs. It is forced to learn the most important features in the
input data (and drop the unimportant ones).
Let’s see how to implement a very simple undercomplete autoencoder for dimension‐
ality reduction.
Performing PCA with an Undercomplete Linear
Autoencoder


ality reduction.
Performing PCA with an Undercomplete Linear
Autoencoder
If the autoencoder uses only linear activations and the cost function is the Mean
Squared Error (MSE), then it can be shown that it ends up performing Principal
Component Analysis (see Chapter 8).
The following code builds a simple linear autoencoder to perform PCA on a 3D data‐
set, projecting it to 2D:
import tensorflow as tf
from tensorflow.contrib.layers import fully_connected
n_inputs = 3  # 3D inputs
n_hidden = 2  # 2D codings
Performing PCA with an Undercomplete Linear Autoencoder
|
413
Download from finelybook www.finelybook.com


n_outputs = n_inputs
learning_rate = 0.01
X = tf.placeholder(tf.float32, shape=[None, n_inputs])
hidden = fully_connected(X, n_hidden, activation_fn=None)
outputs = fully_connected(hidden, n_outputs, activation_fn=None)
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))  # MSE
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(reconstruction_loss)
init = tf.global_variables_initializer()
This code is really not very different from all the MLPs we built in past chapters. The
two things to note are:
• The number of outputs is equal to the number of inputs.
• To perform simple PCA, we set activation_fn=None (i.e., all neurons are linear)
and the cost function is the MSE. We will see more complex autoencoders
shortly.
Now let’s load the dataset, train the model on the training set, and use it to encode the
test set (i.e., project it to 2D):
X_train, X_test = [...] # load the dataset
n_iterations = 1000


test set (i.e., project it to 2D):
X_train, X_test = [...] # load the dataset
n_iterations = 1000
codings = hidden  # the output of the hidden layer provides the codings
with tf.Session() as sess:
    init.run()
    for iteration in range(n_iterations):
        training_op.run(feed_dict={X: X_train})  # no labels (unsupervised)
    codings_val = codings.eval(feed_dict={X: X_test})
Figure 15-2 shows the original 3D dataset (at the left) and the output of the autoen‐
coder’s hidden layer (i.e., the coding layer, at the right). As you can see, the autoen‐
coder found the best 2D plane to project the data onto, preserving as much variance
in the data as it could (just like PCA).
414
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


Figure 15-2. PCA performed by an undercomplete linear autoencoder
Stacked Autoencoders
Just like other neural networks we have discussed, autoencoders can have multiple
hidden layers. In this case they are called stacked autoencoders (or deep autoencoders).
Adding more layers helps the autoencoder learn more complex codings. However,
one must be careful not to make the autoencoder too powerful. Imagine an encoder
so powerful that it just learns to map each input to a single arbitrary number (and the
decoder learns the reverse mapping). Obviously such an autoencoder will reconstruct
the training data perfectly, but it will not have learned any useful data representation
in the process (and it is unlikely to generalize well to new instances).
The architecture of a stacked autoencoder is typically symmetrical with regards to the
central hidden layer (the coding layer). To put it simply, it looks like a sandwich. For


central hidden layer (the coding layer). To put it simply, it looks like a sandwich. For
example, an autoencoder for MNIST (introduced in Chapter 3) may have 784 inputs,
followed by a hidden layer with 300 neurons, then a central hidden layer of 150 neu‐
rons, then another hidden layer with 300 neurons, and an output layer with 784 neu‐
rons. This stacked autoencoder is represented in Figure 15-3.
Figure 15-3. Stacked autoencoder
Stacked Autoencoders
|
415
Download from finelybook www.finelybook.com


TensorFlow Implementation
You can implement a stacked autoencoder very much like a regular deep MLP. In par‐
ticular, the same techniques we used in Chapter 11 for training deep nets can be
applied. For example, the following code builds a stacked autoencoder for MNIST,
using He initialization, the ELU activation function, and ℓ2 regularization. The code
should look very familiar, except that there are no labels (no y):
n_inputs = 28 * 28  # for MNIST
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs
learning_rate = 0.01
l2_reg = 0.001
X = tf.placeholder(tf.float32, shape=[None, n_inputs])
with tf.contrib.framework.arg_scope(
        [fully_connected],
        activation_fn=tf.nn.elu,
        weights_initializer=tf.contrib.layers.variance_scaling_initializer(),
        weights_regularizer=tf.contrib.layers.l2_regularizer(l2_reg)):
    hidden1 = fully_connected(X, n_hidden1)
    hidden2 = fully_connected(hidden1, n_hidden2)  # codings


hidden2 = fully_connected(hidden1, n_hidden2)  # codings
    hidden3 = fully_connected(hidden2, n_hidden3)
    outputs = fully_connected(hidden3, n_outputs, activation_fn=None)
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))  # MSE
reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
loss = tf.add_n([reconstruction_loss] + reg_losses)
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
You can then train the model normally. Note that the digit labels (y_batch) are
unused:
n_epochs = 5
batch_size = 150
with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
416
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


Tying Weights
When an autoencoder is neatly symmetrical, like the one we just built, a common
technique is to tie the weights of the decoder layers to the weights of the encoder lay‐
ers. This halves the number of weights in the model, speeding up training and limit‐
ing the risk of overfitting. Specifically, if the autoencoder has a total of N layers (not
counting the input layer), and WL represents the connection weights of the Lth layer
(e.g., layer 1 is the first hidden layer, layer N
2  is the coding layer, and layer N is the
output layer), then the decoder layer weights can be defined simply as: WN–L+1 = WL
T
(with L = 1, 2, ⋯, N
2 ).
Unfortunately, implementing tied weights in TensorFlow using the fully_connec
ted() function is a bit cumbersome; it’s actually easier to just define the layers man‐
ually. The code ends up significantly more verbose:
activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)


activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
initializer = tf.contrib.layers.variance_scaling_initializer()
X = tf.placeholder(tf.float32, shape=[None, n_inputs])
weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_hidden2])
weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")
weights3 = tf.transpose(weights2, name="weights3")  # tied weights
weights4 = tf.transpose(weights1, name="weights4")  # tied weights
biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_hidden2), name="biases2")
biases3 = tf.Variable(tf.zeros(n_hidden3), name="biases3")
biases4 = tf.Variable(tf.zeros(n_outputs), name="biases4")
hidden1 = activation(tf.matmul(X, weights1) + biases1)
hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)
hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)


hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)
outputs = tf.matmul(hidden3, weights4) + biases4
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))
reg_loss = regularizer(weights1) + regularizer(weights2)
loss = reconstruction_loss + reg_loss
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
This code is fairly straightforward, but there are a few important things to note:
Stacked Autoencoders
|
417
Download from finelybook www.finelybook.com


• First, weight3 and weights4 are not variables, they are respectively the transpose
of weights2 and weights1 (they are “tied” to them).
• Second, since they are not variables, it’s no use regularizing them: we only regula‐
rize weights1 and weights2.
• Third, biases are never tied, and never regularized.
Training One Autoencoder at a Time
Rather than training the whole stacked autoencoder in one go like we just did, it is
often much faster to train one shallow autoencoder at a time, then stack all of them
into a single stacked autoencoder (hence the name), as shown on Figure 15-4. This is
especially useful for very deep autoencoders.
Figure 15-4. Training one autoencoder at a time
During the first phase of training, the first autoencoder learns to reconstruct the
inputs. During the second phase, the second autoencoder learns to reconstruct the
output of the first autoencoder’s hidden layer. Finally, you just build a big sandwich


output of the first autoencoder’s hidden layer. Finally, you just build a big sandwich
using all these autoencoders, as shown in Figure 15-4 (i.e., you first stack the hidden
layers of each autoencoder, then the output layers in reverse order). This gives you
the final stacked autoencoder. You could easily train more autoencoders this way,
building a very deep stacked autoencoder.
To implement this multiphase training algorithm, the simplest approach is to use a
different TensorFlow graph for each phase. After training an autoencoder, you just
run the training set through it and capture the output of the hidden layer. This output
then serves as the training set for the next autoencoder. Once all autoencoders have
been trained this way, you simply copy the weights and biases from each autoencoder
and use them to build the stacked autoencoder. Implementing this approach is quite
418
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


straightforward, so we won’t detail it here, but please check out the code in the
Jupyter notebooks for an example.
Another approach is to use a single graph containing the whole stacked autoencoder,
plus some extra operations to perform each training phase, as shown in Figure 15-5.
Figure 15-5. A single graph to train a stacked autoencoder
This deserves a bit of explanation:
• The central column in the graph is the full stacked autoencoder. This part can be
used after training.
• The left column is the set of operations needed to run the first phase of training.
It creates an output layer that bypasses hidden layers 2 and 3. This output layer
shares the same weights and biases as the stacked autoencoder’s output layer. On
top of that are the training operations that will aim at making the output as close
as possible to the inputs. Thus, this phase will train the weights and biases for the
hidden layer 1 and the output layer (i.e., the first autoencoder).


hidden layer 1 and the output layer (i.e., the first autoencoder).
• The right column in the graph is the set of operations needed to run the second
phase of training. It adds the training operation that will aim at making the out‐
put of hidden layer 3 as close as possible to the output of hidden layer 1. Note
that we must freeze hidden layer 1 while running phase 2. This phase will train
the weights and biases for hidden layers 2 and 3 (i.e., the second autoencoder).
The TensorFlow code looks like this:
[...] # Build the whole stacked autoencoder normally.
      # In this example, the weights are not tied.
Stacked Autoencoders
|
419
Download from finelybook www.finelybook.com


optimizer = tf.train.AdamOptimizer(learning_rate)
with tf.name_scope("phase1"):
    phase1_outputs = tf.matmul(hidden1, weights4) + biases4
    phase1_reconstruction_loss = tf.reduce_mean(tf.square(phase1_outputs - X))
    phase1_reg_loss = regularizer(weights1) + regularizer(weights4)
    phase1_loss = phase1_reconstruction_loss + phase1_reg_loss
    phase1_training_op = optimizer.minimize(phase1_loss)
with tf.name_scope("phase2"):
    phase2_reconstruction_loss = tf.reduce_mean(tf.square(hidden3 - hidden1))
    phase2_reg_loss = regularizer(weights2) + regularizer(weights3)
    phase2_loss = phase2_reconstruction_loss + phase2_reg_loss
    train_vars = [weights2, biases2, weights3, biases3]
    phase2_training_op = optimizer.minimize(phase2_loss, var_list=train_vars)
The first phase is rather straightforward: we just create an output layer that skips hid‐
den layers 2 and 3, then build the training operations to minimize the distance


den layers 2 and 3, then build the training operations to minimize the distance
between the outputs and the inputs (plus some regularization).
The second phase just adds the operations needed to minimize the distance between
the output of hidden layer 3 and hidden layer 1 (also with some regularization). Most
importantly, we provide the list of trainable variables to the minimize() method,
making sure to leave out weights1 and biases1; this effectively freezes hidden layer 1
during phase 2.
During the execution phase, all you need to do is run the phase 1 training op for a
number of epochs, then the phase 2 training op for some more epochs.
Since hidden layer 1 is frozen during phase 2, its output will always
be the same for any given training instance. To avoid having to
recompute the output of hidden layer 1 at every single epoch, you
can compute it for the whole training set at the end of phase 1, then
directly feed the cached output of hidden layer 1 during phase 2.


directly feed the cached output of hidden layer 1 during phase 2.
This can give you a nice performance boost.
Visualizing the Reconstructions
One way to ensure that an autoencoder is properly trained is to compare the inputs
and the outputs. They must be fairly similar, and the differences should be unimpor‐
tant details. Let’s plot two random digits and their reconstructions:
n_test_digits = 2
X_test = mnist.test.images[:n_test_digits]
with tf.Session() as sess:
    [...] # Train the Autoencoder
    outputs_val = outputs.eval(feed_dict={X: X_test})
420
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


def plot_image(image, shape=[28, 28]):
    plt.imshow(image.reshape(shape), cmap="Greys", interpolation="nearest")
    plt.axis("off")
for digit_index in range(n_test_digits):
    plt.subplot(n_test_digits, 2, digit_index * 2 + 1)
    plot_image(X_test[digit_index])
    plt.subplot(n_test_digits, 2, digit_index * 2 + 2)
    plot_image(outputs_val[digit_index])
Figure 15-6 shows the resulting images.
Figure 15-6. Original digits (left) and their reconstructions (right)
Looks close enough. So the autoencoder has properly learned to reproduce its inputs,
but has it learned useful features? Let’s take a look.
Visualizing Features
Once your autoencoder has learned some features, you may want to take a look at
them. There are various techniques for this. Arguably the simplest technique is to
consider each neuron in every hidden layer, and find the training instances that acti‐
vate it the most. This is especially useful for the top hidden layers since they often


vate it the most. This is especially useful for the top hidden layers since they often
capture relatively large features that you can easily spot in a group of training instan‐
ces that contain them. For example, if a neuron strongly activates when it sees a cat in
a picture, it will be pretty obvious that the pictures that activate it the most all contain
cats. However, for lower layers, this technique does not work so well, as the features
are smaller and more abstract, so it’s often hard to understand exactly what the neu‐
ron is getting all excited about.
Let’s look at another technique. For each neuron in the first hidden layer, you can cre‐
ate an image where a pixel’s intensity corresponds to the weight of the connection to
the given neuron. For example, the following code plots the features learned by five
neurons in the first hidden layer:
with tf.Session() as sess:
    [...] # train autoencoder
Stacked Autoencoders
|
421
Download from finelybook www.finelybook.com


weights1_val = weights1.eval()
for i in range(5):
    plt.subplot(1, 5, i + 1)
    plot_image(weights1_val.T[i])
You may get low-level features such as the ones shown in Figure 15-7.
Figure 15-7. Features learned by five neurons from the first hidden layer
The first four features seem to correspond to small patches, while the fifth feature
seems to look for vertical strokes (note that these features come from the stacked
denoising autoencoder that we will discuss later).
Another technique is to feed the autoencoder a random input image, measure the
activation of the neuron you are interested in, and then perform backpropagation to
tweak the image in such a way that the neuron will activate even more. If you iterate
several times (performing gradient ascent), the image will gradually turn into the
most exciting image (for the neuron). This is a useful technique to visualize the kinds
of inputs that a neuron is looking for.


of inputs that a neuron is looking for.
Finally, if you are using an autoencoder to perform unsupervised pretraining—for
example, for a classification task—a simple way to verify that the features learned by
the autoencoder are useful is to measure the performance of the classifier.
Unsupervised Pretraining Using Stacked Autoencoders
As we discussed in Chapter 11, if you are tackling a complex supervised task but you
do not have a lot of labeled training data, one solution is to find a neural network that
performs a similar task, and then reuse its lower layers. This makes it possible to train
a high-performance model using only little training data because your neural net‐
work won’t have to learn all the low-level features; it will just reuse the feature detec‐
tors learned by the existing net.
Similarly, if you have a large dataset but most of it is unlabeled, you can first train a
stacked autoencoder using all the data, then reuse the lower layers to create a neural


stacked autoencoder using all the data, then reuse the lower layers to create a neural
network for your actual task, and train it using the labeled data. For example,
Figure 15-8 shows how to use a stacked autoencoder to perform unsupervised pre‐
training for a classification neural network. The stacked autoencoder itself is typically
trained one autoencoder at a time, as discussed earlier. When training the classifier, if
422
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


2 “Greedy Layer-Wise Training of Deep Networks,” Y. Bengio et al. (2007).
you really don’t have much labeled training data, you may want to freeze the pre‐
trained layers (at least the lower ones).
Figure 15-8. Unsupervised pretraining using autoencoders
This situation is actually quite common, because building a large
unlabeled dataset is often cheap (e.g., a simple script can download
millions of images off the internet), but labeling them can only be
done reliably by humans (e.g., classifying images as cute or not).
Labeling instances is time-consuming and costly, so it is quite com‐
mon to have only a few thousand labeled instances.
As we discussed earlier, one of the triggers of the current Deep Learning tsunami is
the discovery in 2006 by Geoffrey Hinton et al. that deep neural networks can be pre‐
trained in an unsupervised fashion. They used restricted Boltzmann machines for
that (see Appendix E), but in 2007 Yoshua Bengio et al. showed2 that autoencoders
worked just as well.


worked just as well.
There is nothing special about the TensorFlow implementation: just train an autoen‐
coder using all the training data, then reuse its encoder layers to create a new neural
network (see Chapter 11 for more details on how to reuse pretrained layers, or check
out the code examples in the Jupyter notebooks).
Up to now, in order to force the autoencoder to learn interesting features, we have
limited the size of the coding layer, making it undercomplete. There are actually
many other kinds of constraints that can be used, including ones that allow the cod‐
Unsupervised Pretraining Using Stacked Autoencoders
|
423
Download from finelybook www.finelybook.com


3 “Extracting and Composing Robust Features with Denoising Autoencoders,” P. Vincent et al. (2008).
4 “Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denois‐
ing Criterion,” P. Vincent et al. (2010).
ing layer to be just as large as the inputs, or even larger, resulting in an overcomplete
autoencoder. Let’s look at some of those approaches now.
Denoising Autoencoders
Another way to force the autoencoder to learn useful features is to add noise to its
inputs, training it to recover the original, noise-free inputs. This prevents the autoen‐
coder from trivially copying its inputs to its outputs, so it ends up having to find pat‐
terns in the data.
The idea of using autoencoders to remove noise has been around since the 1980s
(e.g., it is mentioned in Yann LeCun’s 1987 master’s thesis). In a 2008 paper,3 Pascal
Vincent et al. showed that autoencoders could also be used for feature extraction. In a


Vincent et al. showed that autoencoders could also be used for feature extraction. In a
2010 paper,4 Vincent et al. introduced stacked denoising autoencoders.
The noise can be pure Gaussian noise added to the inputs, or it can be randomly
switched off inputs, just like in dropout (introduced in Chapter 11). Figure 15-9
shows both options.
Figure 15-9. Denoising autoencoders, with Gaussian noise (left) or dropout (right)
424
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


TensorFlow Implementation
Implementing denoising autoencoders in TensorFlow is not too hard. Let’s start with
Gaussian noise. It’s really just like training a regular autoencoder, except you add
noise to the inputs, and the reconstruction loss is calculated based on the original
inputs:
X = tf.placeholder(tf.float32, shape=[None, n_inputs])
X_noisy = X + tf.random_normal(tf.shape(X))
[...]
hidden1 = activation(tf.matmul(X_noisy, weights1) + biases1)
[...]
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))  # MSE
[...]
Since the shape of X is only partially defined during the construc‐
tion phase, we cannot know in advance the shape of the noise that
we must add to X. We cannot call X.get_shape() because this
would just return the partially defined shape of X ([None,
n_inputs]), and random_normal() expects a fully defined shape so
it would raise an exception. Instead, we call tf.shape(X), which
creates an operation that will return the shape of X at runtime,


creates an operation that will return the shape of X at runtime,
which will be fully defined at that point.
Implementing the dropout version, which is more common, is not much harder:
from tensorflow.contrib.layers import dropout
keep_prob = 0.7
is_training = tf.placeholder_with_default(False, shape=(), name='is_training')
X = tf.placeholder(tf.float32, shape=[None, n_inputs])
X_drop = dropout(X, keep_prob, is_training=is_training)
[...]
hidden1 = activation(tf.matmul(X_drop, weights1) + biases1)
[...]
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))  # MSE
[...]
During training we must set is_training to True (as explained in Chapter 11) using
the feed_dict:
sess.run(training_op, feed_dict={X: X_batch, is_training: True})
However, during testing it is not necessary to set is_training to False, since we set
that as the default in the call to the placeholder_with_default() function.
Denoising Autoencoders
|
425
Download from finelybook www.finelybook.com


Sparse Autoencoders
Another kind of constraint that often leads to good feature extraction is sparsity: by
adding an appropriate term to the cost function, the autoencoder is pushed to reduce
the number of active neurons in the coding layer. For example, it may be pushed to
have on average only 5% significantly active neurons in the coding layer. This forces
the autoencoder to represent each input as a combination of a small number of acti‐
vations. As a result, each neuron in the coding layer typically ends up representing a
useful feature (if you could speak only a few words per month, you would probably
try to make them worth listening to).
In order to favor sparse models, we must first measure the actual sparsity of the cod‐
ing layer at each training iteration. We do so by computing the average activation of
each neuron in the coding layer, over the whole training batch. The batch size must
not be too small, or else the mean will not be accurate.


not be too small, or else the mean will not be accurate.
Once we have the mean activation per neuron, we want to penalize the neurons that
are too active by adding a sparsity loss to the cost function. For example, if we meas‐
ure that a neuron has an average activation of 0.3, but the target sparsity is 0.1, it must
be penalized to activate less. One approach could be simply adding the squared error
(0.3 – 0.1)2 to the cost function, but in practice a better approach is to use the Kull‐
back–Leibler divergence (briefly discussed in Chapter 4), which has much stronger
gradients than the Mean Squared Error, as you can see in Figure 15-10.
Figure 15-10. Sparsity loss
426
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


Given two discrete probability distributions P and Q, the KL divergence between
these distributions, noted DKL(P ∥ Q), can be computed using Equation 15-1.
Equation 15-1. Kullback–Leibler divergence
DKL P ∥Q = ∑
i P i log P i
Q i
In our case, we want to measure the divergence between the target probability p that a
neuron in the coding layer will activate, and the actual probability q (i.e., the mean
activation over the training batch). So the KL divergence simplifies to Equation 15-2.
Equation 15-2. KL divergence between the target sparsity p and the actual sparsity q
DKL p ∥q = p log p
q + 1 −p log 1 −p
1 −q
Once we have computed the sparsity loss for each neuron in the coding layer, we just
sum up these losses, and add the result to the cost function. In order to control the
relative importance of the sparsity loss and the reconstruction loss, we can multiply
the sparsity loss by a sparsity weight hyperparameter. If this weight is too high, the


the sparsity loss by a sparsity weight hyperparameter. If this weight is too high, the
model will stick closely to the target sparsity, but it may not reconstruct the inputs
properly, making the model useless. Conversely, if it is too low, the model will mostly
ignore the sparsity objective and it will not learn any interesting features.
TensorFlow Implementation
We now have all we need to implement a sparse autoencoder using TensorFlow:
def kl_divergence(p, q):
    return p * tf.log(p / q) + (1 - p) * tf.log((1 - p) / (1 - q))
learning_rate = 0.01
sparsity_target = 0.1
sparsity_weight = 0.2
[...] # Build a normal autoencoder (in this example the coding layer is hidden1)
optimizer = tf.train.AdamOptimizer(learning_rate)
hidden1_mean = tf.reduce_mean(hidden1, axis=0)  # batch mean
sparsity_loss = tf.reduce_sum(kl_divergence(sparsity_target, hidden1_mean))
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))  # MSE
loss = reconstruction_loss + sparsity_weight * sparsity_loss


loss = reconstruction_loss + sparsity_weight * sparsity_loss
training_op = optimizer.minimize(loss)
An important detail is the fact that the activations of the coding layer must be
between 0 and 1 (but not equal to 0 or 1), or else the KL divergence will return NaN
Sparse Autoencoders
|
427
Download from finelybook www.finelybook.com


5 “Auto-Encoding Variational Bayes,” D. Kingma and M. Welling (2014).
(Not a Number). A simple solution is to use the logistic activation function for the
coding layer:
hidden1 = tf.nn.sigmoid(tf.matmul(X, weights1) + biases1)
One simple trick can speed up convergence: instead of using the MSE, we can choose
a reconstruction loss that will have larger gradients. Cross entropy is often a good
choice. To use it, we must normalize the inputs to make them take on values from 0
to 1, and use the logistic activation function in the output layer so the outputs also
take on values from 0 to 1. TensorFlow’s sigmoid_cross_entropy_with_logits()
function takes care of efficiently applying the logistic (sigmoid) activation function to
the outputs and computing the cross entropy:
[...]
logits = tf.matmul(hidden1, weights2) + biases2)
outputs = tf.nn.sigmoid(logits)
reconstruction_loss = tf.reduce_sum(
    tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits))


tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits))
Note that the outputs operation is not needed during training (we use it only when
we want to look at the reconstructions).
Variational Autoencoders
Another important category of autoencoders was introduced in 2014 by Diederik
Kingma and Max Welling,5 and has quickly become one of the most popular types of
autoencoders: variational autoencoders.
They are quite different from all the autoencoders we have discussed so far, in partic‐
ular:
• They are probabilistic autoencoders, meaning that their outputs are partly deter‐
mined by chance, even after training (as opposed to denoising autoencoders,
which use randomness only during training).
• Most importantly, they are generative autoencoders, meaning that they can gener‐
ate new instances that look like they were sampled from the training set.
Both these properties make them rather similar to RBMs (see Appendix E), but they


Both these properties make them rather similar to RBMs (see Appendix E), but they
are easier to train and the sampling process is much faster (with RBMs you need to
wait for the network to stabilize into a “thermal equilibrium” before you can sample a
new instance).
428
|
Chapter 15: Autoencoders
Download from finelybook www.finelybook.com


6 Variational autoencoders are actually more general; the codings are not limited to Gaussian distributions.
Let’s take a look at how they work. Figure 15-11 (left) shows a variational autoen‐
coder. You can recognize, of course, the basic structure of all autoencoders, with an
encoder followed by a decoder (in this example, they both have two hidden layers),
but there is a twist: instead of directly producing a coding for a given input, the
encoder produces a mean coding μ and a standard deviation σ. The actual coding is
then sampled randomly from a Gaussian distribution with mean μ and standard devi‐
ation σ. After that the decoder just decodes the sampled coding normally. The right
part of the diagram shows a training instance going through this autoencoder. First,
the encoder produces μ and σ, then a coding is sampled randomly (notice that it is
not exactly located at μ), and finally this coding is decoded, and the final output
resembles the training instance.


resembles the training instance.
Figure 15-11. Variational autoencoder (left), and an instance going through it (right)
As you can see on the diagram, although the inputs may have a very convoluted dis‐
tribution, a variational autoencoder tends to produce codings that look as though
they were sampled from a simple Gaussian distribution:6 during training, the cost
function (discussed next) pushes the codings to gradually migrate within the coding
space (also called the latent space) to occupy a roughly (hyper)spherical region that
looks like a cloud of Gaussian points. One great consequence is that after training a
Variational Autoencoders
|
429
Download from finelybook www.finelybook.com


7 For more mathematical details, check out the original paper on variational autoencoders, or Carl Doersch’s
great tutorial (2016).
variational autoencoder, you can very easily generate a new instance: just sample a
random coding from the Gaussian distribution, decode it, and voilà!
So let’s look at the cost function. It is composed of two parts. The first is the usual
reconstruction loss that pushes the autoencoder to reproduce its inputs (we can use
cross entropy for this, as discussed earlier). The second is the latent loss that pushes
the autoencoder to have codings that look as though they were sampled from a simple
Gaussian distribution, for which we use the KL divergence between the target distri‐
bution (the Gaussian distribution) and the actual distribution of the codings. The
math is a bit more complex than earlier, in particular because of the Gaussian noise,
which limits the amount of information that can be transmitted to the coding layer


which limits the amount of information that can be transmitted to the coding layer
(thus pushing the autoencoder to learn useful features). Luckily, the equations sim‐
plify to the following code for the latent loss:7
eps = 1e-10  # smoothing term to avoid computing log(0) which is NaN
latent_loss = 0.5 * tf.reduce_sum(
    tf.square(hidden3_sigma) + tf.square(hidden3_mean)
    - 1 - tf.log(eps + tf.square(hidden3_sigma)))
One common variant is to train the encoder to output γ = log(σ2) rather than σ.
Wherever we need σ we can just compute σ = exp
γ
2 . This makes it a bit easier for
the encoder to capture sigmas of different scales, and thus it helps speed up conver‐
gence. The latent loss ends up a bit simpler:
latent_loss = 0.5 * tf.reduce_sum(
    tf.exp(hidden3_gamma) + tf.square(hidden3_mean) - 1 - hidden3_gamma)
The following code builds the variational autoencoder shown in Figure 15-11 (left),
using the log(σ2) variant:
n_inputs = 28 * 28  # for MNIST
n_hidden1 = 500