sess.run(enqueue, feed_dict={v: [[7., 8., 9., 5.], [6., 7., 8., 9.]]}) # 2x4
If we just dequeue one item at a time, we get the exact same tensors that were
enqueued. But if we dequeue several items at a time (using dequeue_many() or
dequeue_up_to()), the queue automatically pads the tensors appropriately. For exam‐
ple, if we dequeue all three items at once, all tensors will be padded with zeros to
become 3 × 4 tensors, since the maximum size for the first dimension is 3 (first item)
and the maximum size for the second dimension is 4 (third item):
>>> q = [...]
>>> dequeue = q.dequeue_many(3)
>>> with tf.Session([...]) as sess:
...     print(sess.run(dequeue))
[[[ 1.  2.  0.  0.]
  [ 3.  4.  0.  0.]
  [ 5.  6.  0.  0.]]
 [[ 1.  0.  0.  0.]
  [ 0.  0.  0.  0.]
  [ 0.  0.  0.  0.]]
 [[ 7.  8.  9.  5.]
  [ 6.  7.  8.  9.]
  [ 0.  0.  0.  0.]]]
This type of queue can be useful when you are dealing with variable length inputs,
such as sequences of words (see Chapter 14).


such as sequences of words (see Chapter 14).
Okay, now let’s pause for a second: so far you have learned to distribute computations
across multiple devices and servers, share variables across sessions, and communicate
asynchronously using queues. Before you start training neural networks, though,
there’s one last topic we need to discuss: how to efficiently load training data.
334
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


Loading Data Directly from the Graph
So far we have assumed that the clients would load the training data and feed it to the
cluster using placeholders. This is simple and works quite well for simple setups, but
it is rather inefficient since it transfers the training data several times:
1. From the filesystem to the client
2. From the client to the master task
3. Possibly from the master task to other tasks where the data is needed
It gets worse if you have several clients training various neural networks using the
same training data (for example, for hyperparameter tuning): if every client loads the
data simultaneously, you may end up even saturating your file server or the network’s
bandwidth.
Preload the data into a variable
For datasets that can fit in memory, a better option is to load the training data once
and assign it to a variable, then just use that variable in your graph. This is called
preloading the training set. This way the data will be transferred only once from the


preloading the training set. This way the data will be transferred only once from the
client to the cluster (but it may still need to be moved around from task to task
depending on which operations need it). The following code shows how to load the
full training set into a variable:
training_set_init = tf.placeholder(tf.float32, shape=(None, n_features))
training_set = tf.Variable(training_set_init, trainable=False, collections=[],
                           name="training_set")
with tf.Session([...]) as sess:
    data = [...]  # load the training data from the datastore
    sess.run(training_set.initializer, feed_dict={training_set_init: data})
You must set trainable=False so the optimizers don’t try to tweak this variable. You
should also set collections=[] to ensure that this variable won’t get added to the
GraphKeys.GLOBAL_VARIABLES collection, which is used for saving and restoring
checkpoints.
This example assumes that all of your training set (including the


checkpoints.
This example assumes that all of your training set (including the
labels) consists only of float32 values. If that’s not the case, you
will need one variable per type.
Reading the training data directly from the graph
If the training set does not fit in memory, a good solution is to use reader operations:
these are operations capable of reading data directly from the filesystem. This way the
Multiple Devices Across Multiple Servers
|
335
Download from finelybook www.finelybook.com


training data never needs to flow through the clients at all. TensorFlow provides read‐
ers for various file formats:
• CSV
• Fixed-length binary records
• TensorFlow’s own TFRecords format, based on protocol buffers
Let’s look at a simple example reading from a CSV file (for other formats, please
check out the API documentation). Suppose you have file named my_test.csv that
contains training instances, and you want to create operations to read it. Suppose it
has the following content, with two float features x1 and x2 and one integer target
representing a binary class:
x1,  x2,  target
1. , 2. , 0
4. , 5  , 1
7. ,    , 0
First, let’s create a TextLineReader to read this file. A TextLineReader opens a file
(once we tell it which one to open) and reads lines one by one. It is a stateful opera‐
tion, like variables and queues: it preserves its state across multiple runs of the graph,
keeping track of which file it is currently reading and what its current position is in
this file.


keeping track of which file it is currently reading and what its current position is in
this file.
reader = tf.TextLineReader(skip_header_lines=1)
Next, we create a queue that the reader will pull from to know which file to read next.
We also create an enqueue operation and a placeholder to push any filename we want
to the queue, and we create an operation to close the queue once we have no more
files to read:
filename_queue = tf.FIFOQueue(capacity=10, dtypes=[tf.string], shapes=[()])
filename = tf.placeholder(tf.string)
enqueue_filename = filename_queue.enqueue([filename])
close_filename_queue = filename_queue.close()
Now we are ready to create a read operation that will read one record (i.e., a line) at a
time and return a key/value pair. The key is the record’s unique identifier—a string
composed of the filename, a colon (:), and the line number—and the value is simply
a string containing the content of the line:
key, value = reader.read(filename_queue)


a string containing the content of the line:
key, value = reader.read(filename_queue)
We have all we need to read the file line by line! But we are not quite done yet—we
need to parse this string to get the features and target:
x1, x2, target = tf.decode_csv(value, record_defaults=[[-1.], [-1.], [-1]])
features = tf.stack([x1, x2])
336
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


The first line uses TensorFlow’s CSV parser to extract the values from the current
line. The default values are used when a field is missing (in this example the third
training instance’s x2 feature), and they are also used to determine the type of each
field (in this case two floats and one integer).
Finally, we can push this training instance and its target to a RandomShuffleQueue
that we will share with the training graph (so it can pull mini-batches from it), and we
create an operation to close that queue when we are done pushing instances to it:
instance_queue = tf.RandomShuffleQueue(
    capacity=10, min_after_dequeue=2,
    dtypes=[tf.float32, tf.int32], shapes=[[2],[]],
    name="instance_q", shared_name="shared_instance_q")
enqueue_instance = instance_queue.enqueue([features, target])
close_instance_queue = instance_queue.close()
Wow! That was a lot of work just to read a file. Plus we only created the graph, so now
we need to run it:
with tf.Session([...]) as sess:


we need to run it:
with tf.Session([...]) as sess:
    sess.run(enqueue_filename, feed_dict={filename: "my_test.csv"})
    sess.run(close_filename_queue)
    try:
        while True:
            sess.run(enqueue_instance)
    except tf.errors.OutOfRangeError as ex:
        pass # no more records in the current file and no more files to read
    sess.run(close_instance_queue)
First we open the session, and then we enqueue the filename "my_test.csv" and
immediately close that queue since we will not enqueue any more filenames. Then we
run an infinite loop to enqueue instances one by one. The enqueue_instance
depends on the reader reading the next line, so at every iteration a new record is read
until it reaches the end of the file. At that point it tries to read the filename queue to
know which file to read next, and since the queue is closed it throws an OutOfRan
geError exception (if we did not close the queue, it would just remain blocked until


geError exception (if we did not close the queue, it would just remain blocked until
we pushed another filename or closed the queue). Lastly, we close the instance queue
so that the training operations pulling from it won’t get blocked forever. Figure 12-9
summarizes what we have learned; it represents a typical graph for reading training
instances from a set of CSV files.
Multiple Devices Across Multiple Servers
|
337
Download from finelybook www.finelybook.com


Figure 12-9. A graph dedicated to reading training instances from CSV files
In the training graph, you need to create the shared instance queue and simply
dequeue mini-batches from it:
instance_queue = tf.RandomShuffleQueue([...], shared_name="shared_instance_q")
mini_batch_instances, mini_batch_targets = instance_queue.dequeue_up_to(2)
[...] # use the mini_batch instances and targets to build the training graph
training_op = [...]
with tf.Session([...]) as sess:
    try:
        for step in range(max_steps):
            sess.run(training_op)
    except tf.errors.OutOfRangeError as ex:
        pass # no more training instances
In this example, the first mini-batch will contain the first two instances of the CSV
file, and the second mini-batch will contain the last instance.
TensorFlow queues don’t handle sparse tensors well, so if your
training instances are sparse you should parse the records after the
instance queue.


training instances are sparse you should parse the records after the
instance queue.
This architecture will only use one thread to read records and push them to the
instance queue. You can get a much higher throughput by having multiple threads
read simultaneously from multiple files using multiple readers. Let’s see how.
Multithreaded readers using a Coordinator and a QueueRunner
To have multiple threads read instances simultaneously, you could create Python
threads (using the threading module) and manage them yourself. However, Tensor‐
Flow provides some tools to make this simpler: the Coordinator class and the QueueR
unner class.
338
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


A Coordinator is a very simple object whose sole purpose is to coordinate stopping
multiple threads. First you create a Coordinator:
coord = tf.train.Coordinator()
Then you give it to all threads that need to stop jointly, and their main loop looks like
this:
while not coord.should_stop():
    [...] # do something
Any thread can request that every thread stop by calling the Coordinator’s
request_stop() method:
coord.request_stop()
Every thread will stop as soon as it finishes its current iteration. You can wait for all of
the threads to finish by calling the Coordinator’s join() method, passing it the list of
threads:
coord.join(list_of_threads)
A QueueRunner runs multiple threads that each run an enqueue operation repeatedly,
filling up a queue as fast as possible. As soon as the queue is closed, the next thread
that tries to push an item to the queue will get an OutOfRangeError; this thread
catches the error and immediately tells other threads to stop using a Coordinator.


catches the error and immediately tells other threads to stop using a Coordinator.
The following code shows how you can use a QueueRunner to have five threads read‐
ing instances simultaneously and pushing them to an instance queue:
[...] # same construction phase as earlier
queue_runner = tf.train.QueueRunner(instance_queue, [enqueue_instance] * 5)
with tf.Session() as sess:
    sess.run(enqueue_filename, feed_dict={filename: "my_test.csv"})
    sess.run(close_filename_queue)
    coord = tf.train.Coordinator()
    enqueue_threads = queue_runner.create_threads(sess, coord=coord, start=True)
The first line creates the QueueRunner and tells it to run five threads, all running the
same enqueue_instance operation repeatedly. Then we start a session and we
enqueue the name of the files to read (in this case just "my_test.csv"). Next we cre‐
ate a Coordinator that the QueueRunner will use to stop gracefully, as just explained.


ate a Coordinator that the QueueRunner will use to stop gracefully, as just explained.
Finally, we tell the QueueRunner to create the threads and start them. The threads will
read all training instances and push them to the instance queue, and then they will all
stop gracefully.
This will be a bit more efficient than earlier, but we can do better. Currently all
threads are reading from the same file. We can make them read simultaneously from
separate files instead (assuming the training data is sharded across multiple CSV files)
by creating multiple readers (see Figure 12-10).
Multiple Devices Across Multiple Servers
|
339
Download from finelybook www.finelybook.com


Figure 12-10. Reading simultaneously from multiple files
For this we need to write a small function to create a reader and the nodes that will
read and push one instance to the instance queue:
def read_and_push_instance(filename_queue, instance_queue):
    reader = tf.TextLineReader(skip_header_lines=1)
    key, value = reader.read(filename_queue)
    x1, x2, target = tf.decode_csv(value, record_defaults=[[-1.], [-1.], [-1]])
    features = tf.stack([x1, x2])
    enqueue_instance = instance_queue.enqueue([features, target])
    return enqueue_instance
Next we define the queues:
filename_queue = tf.FIFOQueue(capacity=10, dtypes=[tf.string], shapes=[()])
filename = tf.placeholder(tf.string)
enqueue_filename = filename_queue.enqueue([filename])
close_filename_queue = filename_queue.close()
instance_queue = tf.RandomShuffleQueue([...])
And finally we create the QueueRunner, but this time we give it a list of different


And finally we create the QueueRunner, but this time we give it a list of different
enqueue operations. Each operation will use a different reader, so the threads will
simultaneously read from different files:
read_and_enqueue_ops = [
    read_and_push_instance(filename_queue, instance_queue)
    for i in range(5)]
queue_runner = tf.train.QueueRunner(instance_queue, read_and_enqueue_ops)
The execution phase is then the same as before: first push the names of the files to
read, then create a Coordinator and create and start the QueueRunner threads. This
time all threads will read from different files simultaneously until all files are read
entirely, and then the QueueRunner will close the instance queue so that other ops
pulling from it don’t get blocked.
340
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Other convenience functions
TensorFlow also offers a few convenience functions to simplify some common tasks
when reading training instances. We will go over just a few (see the API documenta‐
tion for the full list).
The string_input_producer() takes a 1D tensor containing a list of filenames, cre‐
ates a thread that pushes one filename at a time to the filename queue, and then
closes the queue. If you specify a number of epochs, it will cycle through the file‐
names once per epoch before closing the queue. By default, it shuffles the filenames at
each epoch. It creates a QueueRunner to manage its thread, and adds it to the Graph
Keys.QUEUE_RUNNERS collection. To start every QueueRunner in that collection, you
can call the tf.train.start_queue_runners() function. Note that if you forget to
start the QueueRunner, the filename queue will be open and empty, and your readers
will be blocked forever.
There are a few other producer functions that similarly create a queue and a corre‐


There are a few other producer functions that similarly create a queue and a corre‐
sponding QueueRunner for running an enqueue operation (e.g., input_producer(),
range_input_producer(), and slice_input_producer()).
The shuffle_batch() function takes a list of tensors (e.g., [features, target]) and
creates:
• A RandomShuffleQueue
• A QueueRunner to enqueue the tensors to the queue (added to the Graph
Keys.QUEUE_RUNNERS collection)
• A dequeue_many operation to extract a mini-batch from the queue
This makes it easy to manage in a single process a multithreaded input pipeline feed‐
ing a queue and a training pipeline reading mini-batches from that queue. Also check
out the batch(), batch_join(), and shuffle_batch_join() functions that provide
similar functionality.
Okay! You now have all the tools you need to start training and running neural net‐
works efficiently across multiple devices and servers on a TensorFlow cluster. Let’s
review what you have learned:


review what you have learned:
• Using multiple GPU devices
• Setting up and starting a TensorFlow cluster
• Distributing computations across multiple devices and servers
• Sharing variables (and other stateful ops such as queues and readers) across ses‐
sions using containers
• Coordinating multiple graphs working asynchronously using queues
Multiple Devices Across Multiple Servers
|
341
Download from finelybook www.finelybook.com


4 Not 100% linear if you wait for all devices to finish, since the total time will be the time taken by the slowest
device.
• Reading inputs efficiently using readers, queue runners, and coordinators
Now let’s use all of this to parallelize neural networks!
Parallelizing Neural Networks on a TensorFlow Cluster
In this section, first we will look at how to parallelize several neural networks by sim‐
ply placing each one on a different device. Then we will look at the much trickier
problem of training a single neural network across multiple devices and servers.
One Neural Network per Device
The most trivial way to train and run neural networks on a TensorFlow cluster is to
take the exact same code you would use for a single device on a single machine, and
specify the master server’s address when creating the session. That’s it—you’re done!
Your code will be running on the server’s default device. You can change the device


Your code will be running on the server’s default device. You can change the device
that will run your graph simply by putting your code’s construction phase within a
device block.
By running several client sessions in parallel (in different threads or different pro‐
cesses), connecting them to different servers, and configuring them to use different
devices, you can quite easily train or run many neural networks in parallel, across all
devices and all machines in your cluster (see Figure 12-11). The speedup is almost
linear.4 Training 100 neural networks across 50 servers with 2 GPUs each will not take
much longer than training just 1 neural network on 1 GPU.
Figure 12-11. Training one neural network per device
342
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


This solution is perfect for hyperparameter tuning: each device in the cluster will
train a different model with its own set of hyperparameters. The more computing
power you have, the larger the hyperparameter space you can explore.
It also works perfectly if you host a web service that receives a large number of queries
per second (QPS) and you need your neural network to make a prediction for each
query. Simply replicate the neural network across all devices on the cluster and dis‐
patch queries across all devices. By adding more servers you can handle an unlimited
number of QPS (however, this will not reduce the time it takes to process a single
request since it will still have to wait for a neural network to make a prediction).
Another option is to serve your neural networks using TensorFlow
Serving. It is an open source system, released by Google in Febru‐
ary 2016, designed to serve a high volume of queries to Machine


ary 2016, designed to serve a high volume of queries to Machine
Learning models (typically built with TensorFlow). It handles
model versioning, so you can easily deploy a new version of your
network to production, or experiment with various algorithms
without interrupting your service, and it can sustain a heavy load
by adding more servers. For more details, check out https://tensor
flow.github.io/serving/.
In-Graph Versus Between-Graph Replication
You can also parallelize the training of a large ensemble of neural networks by simply
placing every neural network on a different device (ensembles were introduced in
Chapter 7). However, once you want to run the ensemble, you will need to aggregate
the individual predictions made by each neural network to produce the ensemble’s
prediction, and this requires a bit of coordination.
There are two major approaches to handling a neural network ensemble (or any other
graph that contains large chunks of independent computations):


graph that contains large chunks of independent computations):
• You can create one big graph, containing every neural network, each pinned to a
different device, plus the computations needed to aggregate the individual pre‐
dictions from all the neural networks (see Figure 12-12). Then you just create
one session to any server in the cluster and let it take care of everything (includ‐
ing waiting for all individual predictions to be available before aggregating them).
This approach is called in-graph replication.
Parallelizing Neural Networks on a TensorFlow Cluster
|
343
Download from finelybook www.finelybook.com


Figure 12-12. In-graph replication
• Alternatively, you can create one separate graph for each neural network and
handle synchronization between these graphs yourself. This approach is called
between-graph replication. One typical implementation is to coordinate the exe‐
cution of these graphs using queues (see Figure 12-13). A set of clients handles
one neural network each, reading from its dedicated input queue, and writing to
its dedicated prediction queue. Another client is in charge of reading the inputs
and pushing them to all the input queues (copying all inputs to every queue).
Finally, one last client is in charge of reading one prediction from each prediction
queue and aggregating them to produce the ensemble’s prediction.
Figure 12-13. Between-graph replication
344
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


These solutions have their pros and cons. In-graph replication is somewhat simpler to
implement since you don’t have to manage multiple clients and multiple queues.
However, between-graph replication is a bit easier to organize into well-bounded and
easy-to-test modules. Moreover, it gives you more flexibility. For example, you could
add a dequeue timeout in the aggregator client so that the ensemble would not fail
even if one of the neural network clients crashes or if one neural network takes too
long to produce its prediction. TensorFlow lets you specify a timeout when calling the
run() function by passing a RunOptions with timeout_in_ms:
with tf.Session([...]) as sess:
    [...]
    run_options = tf.RunOptions()
    run_options.timeout_in_ms = 1000  # 1s timeout
    try:
        pred = sess.run(dequeue_prediction, options=run_options)
    except tf.errors.DeadlineExceededError as ex:
        [...] # the dequeue operation timed out after 1s


[...] # the dequeue operation timed out after 1s
Another way you can specify a timeout is to set the session’s operation_time
out_in_ms configuration option, but in this case the run() function times out if any
operation takes longer than the timeout delay:
config = tf.ConfigProto()
config.operation_timeout_in_ms = 1000  # 1s timeout for every operation
with tf.Session([...], config=config) as sess:
    [...]
    try:
        pred = sess.run(dequeue_prediction)
    except tf.errors.DeadlineExceededError as ex:
        [...]  # the dequeue operation timed out after 1s
Model Parallelism
So far we have run each neural network on a single device. What if we want to run a
single neural network across multiple devices? This requires chopping your model
into separate chunks and running each chunk on a different device. This is called
model parallelism. Unfortunately, model parallelism turns out to be pretty tricky, and


model parallelism. Unfortunately, model parallelism turns out to be pretty tricky, and
it really depends on the architecture of your neural network. For fully connected net‐
works, there is generally not much to be gained from this approach (see
Figure 12-14). Intuitively, it may seem that an easy way to split the model is to place
each layer on a different device, but this does not work since each layer needs to wait
for the output of the previous layer before it can do anything. So perhaps you can
slice it vertically—for example, with the left half of each layer on one device, and the
right part on another device? This is slightly better, since both halves of each layer can
indeed work in parallel, but the problem is that each half of the next layer requires the
output of both halves, so there will be a lot of cross-device communication (repre‐
Parallelizing Neural Networks on a TensorFlow Cluster
|
345
Download from finelybook www.finelybook.com


sented by the dashed arrows). This is likely to completely cancel out the benefit of the
parallel computation, since cross-device communication is slow (especially if it is
across separate machines).
Figure 12-14. Splitting a fully connected neural network
However, as we will see in Chapter 13, some neural network architectures, such as
convolutional neural networks, contain layers that are only partially connected to the
lower layers, so it is much easier to distribute chunks across devices in an efficient
way.
Figure 12-15. Splitting a partially connected neural network
Moreover, as we will see in Chapter 14, some deep recurrent neural networks are
composed of several layers of memory cells (see the left side of Figure 12-16). A cell’s
output at time t is fed back to its input at time t + 1 (as you can see more clearly on
the right side of Figure 12-16). If you split such a network horizontally, placing each


the right side of Figure 12-16). If you split such a network horizontally, placing each
layer on a different device, then at the first step only one device will be active, at the
second step two will be active, and by the time the signal propagates to the output
layer all devices will be active simultaneously. There is still a lot of cross-device com‐
munication going on, but since each cell may be fairly complex, the benefit of run‐
ning multiple cells in parallel often outweighs the communication penalty.
346
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Figure 12-16. Splitting a deep recurrent neural network
In short, model parallelism can speed up running or training some types of neural
networks, but not all, and it requires special care and tuning, such as making sure
that devices that need to communicate the most run on the same machine.
Data Parallelism
Another way to parallelize the training of a neural network is to replicate it on each
device, run a training step simultaneously on all replicas using a different mini-batch
for each, and then aggregate the gradients to update the model parameters. This is
called data parallelism (see Figure 12-17).
Figure 12-17. Data parallelism
There are two variants of this approach: synchronous updates and asynchronous
updates.
Parallelizing Neural Networks on a TensorFlow Cluster
|
347
Download from finelybook www.finelybook.com


5 This name is slightly confusing since it sounds like some replicas are special, doing nothing. In reality, all rep‐
licas are equivalent: they all work hard to be among the fastest at each training step, and the losers vary at
every step (unless some devices are really slower than others).
Synchronous updates
With synchronous updates, the aggregator waits for all gradients to be available before
computing the average and applying the result (i.e., using the aggregated gradients to
update the model parameters). Once a replica has finished computing its gradients, it
must wait for the parameters to be updated before it can proceed to the next mini-
batch. The downside is that some devices may be slower than others, so all other
devices will have to wait for them at every step. Moreover, the parameters will be
copied to every device almost at the same time (immediately after the gradients are
applied), which may saturate the parameter servers’ bandwidth.


applied), which may saturate the parameter servers’ bandwidth.
To reduce the waiting time at each step, you could ignore the gradi‐
ents from the slowest few replicas (typically ~10%). For example,
you could run 20 replicas, but only aggregate the gradients from
the fastest 18 replicas at each step, and just ignore the gradients
from the last 2. As soon as the parameters are updated, the first 18
replicas can start working again immediately, without having to
wait for the 2 slowest replicas. This setup is generally described as
having 18 replicas plus 2 spare replicas.5
Asynchronous updates
With asynchronous updates, whenever a replica has finished computing the gradi‐
ents, it immediately uses them to update the model parameters. There is no aggrega‐
tion (remove the “mean” step in Figure 12-17), and no synchronization. Replicas just
work independently of the other replicas. Since there is no waiting for the other repli‐


work independently of the other replicas. Since there is no waiting for the other repli‐
cas, this approach runs more training steps per minute. Moreover, although the
parameters still need to be copied to every device at every step, this happens at differ‐
ent times for each replica so the risk of bandwidth saturation is reduced.
Data parallelism with asynchronous updates is an attractive choice, because of its
simplicity, the absence of synchronization delay, and a better use of the bandwidth.
However, although it works reasonably well in practice, it is almost surprising that it
works at all! Indeed, by the time a replica has finished computing the gradients based
on some parameter values, these parameters will have been updated several times by
other replicas (on average N – 1 times if there are N replicas) and there is no guaran‐
tee that the computed gradients will still be pointing in the right direction (see


tee that the computed gradients will still be pointing in the right direction (see
Figure 12-18). When gradients are severely out-of-date, they are called stale gradients:
they can slow down convergence, introducing noise and wobble effects (the learning
348
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


curve may contain temporary oscillations), or they can even make the training algo‐
rithm diverge.
Figure 12-18. Stale gradients when using asynchronous updates
There are a few ways to reduce the effect of stale gradients:
• Reduce the learning rate.
• Drop stale gradients or scale them down.
• Adjust the mini-batch size.
• Start the first few epochs using just one replica (this is called the warmup phase).
Stale gradients tend to be more damaging at the beginning of training, when gra‐
dients are typically large and the parameters have not settled into a valley of the
cost function yet, so different replicas may push the parameters in quite different
directions.
A paper published by the Google Brain team in April 2016 benchmarked various
approaches and found that data parallelism with synchronous updates using a few
spare replicas was the most efficient, not only converging faster but also producing a


spare replicas was the most efficient, not only converging faster but also producing a
better model. However, this is still an active area of research, so you should not rule
out asynchronous updates quite yet.
Bandwidth saturation
Whether you use synchronous or asynchronous updates, data parallelism still
requires communicating the model parameters from the parameter servers to every
replica at the beginning of every training step, and the gradients in the other direction
at the end of each training step. Unfortunately, this means that there always comes a
point where adding an extra GPU will not improve performance at all because the
Parallelizing Neural Networks on a TensorFlow Cluster
|
349
Download from finelybook www.finelybook.com


time spent moving the data in and out of GPU RAM (and possibly across the net‐
work) will outweigh the speedup obtained by splitting the computation load. At that
point, adding more GPUs will just increase saturation and slow down training.
For some models, typically relatively small and trained on a very
large training set, you are often better off training the model on a
single machine with a single GPU.
Saturation is more severe for large dense models, since they have a lot of parameters
and gradients to transfer. It is less severe for small models (but the parallelization gain
is small) and also for large sparse models since the gradients are typically mostly
zeros, so they can be communicated efficiently. Jeff Dean, initiator and lead of the
Google Brain project, reported typical speedups of 25–40x when distributing compu‐
tations across 50 GPUs for dense models, and 300x speedup for sparser models


tations across 50 GPUs for dense models, and 300x speedup for sparser models
trained across 500 GPUs. As you can see, sparse models really do scale better. Here
are a few concrete examples:
• Neural Machine Translation: 6x speedup on 8 GPUs
• Inception/ImageNet: 32x speedup on 50 GPUs
• RankBrain: 300x speedup on 500 GPUs
These numbers represent the state of the art in Q1 2016. Beyond a few dozen GPUs
for a dense model or few hundred GPUs for a sparse model, saturation kicks in and
performance degrades. There is plenty of research going on to solve this problem
(exploring peer-to-peer architectures rather than centralized parameter servers, using
lossy model compression, optimizing when and what the replicas need to communi‐
cate, and so on), so there will likely be a lot of progress in parallelizing neural net‐
works in the next few years.
In the meantime, here are a few simple steps you can take to reduce the saturation
problem:


In the meantime, here are a few simple steps you can take to reduce the saturation
problem:
• Group your GPUs on a few servers rather than scattering them across many
servers. This will avoid unnecessary network hops.
• Shard the parameters across multiple parameter servers (as discussed earlier).
• Drop the model parameters’ float precision from 32 bits (tf.float32) to 16 bits
(tf.bfloat16). This will cut in half the amount of data to transfer, without much
impact on the convergence rate or the model’s performance.
350
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Although 16-bit precision is the minimum for training neural net‐
work, you can actually drop down to 8-bit precision after training
to reduce the size of the model and speed up computations. This is
called quantizing the neural network. It is particularly useful for
deploying and running pretrained models on mobile phones. See
Pete Warden’s great post on the subject.
TensorFlow implementation
To implement data parallelism using TensorFlow, you first need to choose whether
you want in-graph replication or between-graph replication, and whether you want
synchronous updates or asynchronous updates. Let’s look at how you would imple‐
ment each combination (see the exercises and the Jupyter notebooks for complete
code examples).
With in-graph replication + synchronous updates, you build one big graph contain‐
ing all the model replicas (placed on different devices), and a few nodes to aggregate
all their gradients and feed them to an optimizer. Your code opens a session to the


all their gradients and feed them to an optimizer. Your code opens a session to the
cluster and simply runs the training operation repeatedly.
With in-graph replication + asynchronous updates, you also create one big graph, but
with one optimizer per replica, and you run one thread per replica, repeatedly run‐
ning the replica’s optimizer.
With between-graph replication + asynchronous updates, you run multiple inde‐
pendent clients (typically in separate processes), each training the model replica as if
it were alone in the world, but the parameters are actually shared with other replicas
(using a resource container).
With between-graph replication + synchronous updates, once again you run multiple
clients, each training a model replica based on shared parameters, but this time you
wrap the optimizer (e.g., a MomentumOptimizer) within a SyncReplicasOptimizer.
Each replica uses this optimizer as it would use any other optimizer, but under the


Each replica uses this optimizer as it would use any other optimizer, but under the
hood this optimizer sends the gradients to a set of queues (one per variable), which is
read by one of the replica’s SyncReplicasOptimizer, called the chief. The chief aggre‐
gates the gradients and applies them, then writes a token to a token queue for each
replica, signaling it that it can go ahead and compute the next gradients. This
approach supports having spare replicas.
If you go through the exercises, you will implement each of these four solutions. You
will easily be able to apply what you have learned to train large deep neural networks
across dozens of servers and GPUs! In the following chapters we will go through a
few more important neural network architectures before we tackle Reinforcement
Learning.
Parallelizing Neural Networks on a TensorFlow Cluster
|
351
Download from finelybook www.finelybook.com


Exercises
1. If you get a CUDA_ERROR_OUT_OF_MEMORY when starting your TensorFlow pro‐
gram, what is probably going on? What can you do about it?
2. What is the difference between pinning an operation on a device and placing an
operation on a device?
3. If you are running on a GPU-enabled TensorFlow installation, and you just use
the default placement, will all operations be placed on the first GPU?
4. If you pin a variable to "/gpu:0", can it be used by operations placed on /gpu:1?
Or by operations placed on "/cpu:0"? Or by operations pinned to devices loca‐
ted on other servers?
5. Can two operations placed on the same device run in parallel?
6. What is a control dependency and when would you want to use one?
7. Suppose you train a DNN for days on a TensorFlow cluster, and immediately
after your training program ends you realize that you forgot to save the model
using a Saver. Is your trained model lost?


using a Saver. Is your trained model lost?
8. Train several DNNs in parallel on a TensorFlow cluster, using different hyper‐
parameter values. This could be DNNs for MNIST classification or any other task
you are interested in. The simplest option is to write a single client program that
trains only one DNN, then run this program in multiple processes in parallel,
with different hyperparameter values for each client. The program should have
command-line options to control what server and device the DNN should be
placed on, and what resource container and hyperparameter values to use (make
sure to use a different resource container for each DNN). Use a validation set or
cross-validation to select the top three models.
9. Create an ensemble using the top three models from the previous exercise.
Define it in a single graph, ensuring that each DNN runs on a different device.
Evaluate it on the validation set: does the ensemble perform better than the indi‐
vidual DNNs?


Evaluate it on the validation set: does the ensemble perform better than the indi‐
vidual DNNs?
10. Train a DNN using between-graph replication and data parallelism with asyn‐
chronous updates, timing how long it takes to reach a satisfying performance.
Next, try again using synchronous updates. Do synchronous updates produce a
better model? Is training faster? Split the DNN vertically and place each vertical
slice on a different device, and train the model again. Is training any faster? Is the
performance any different?
Solutions to these exercises are available in Appendix A.
352
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


CHAPTER 13
Convolutional Neural Networks
Although IBM’s Deep Blue supercomputer beat the chess world champion Garry Kas‐
parov back in 1996, until quite recently computers were unable to reliably perform
seemingly trivial tasks such as detecting a puppy in a picture or recognizing spoken
words. Why are these tasks so effortless to us humans? The answer lies in the fact that
perception largely takes place outside the realm of our consciousness, within special‐
ized visual, auditory, and other sensory modules in our brains. By the time sensory
information reaches our consciousness, it is already adorned with high-level features;
for example, when you look at a picture of a cute puppy, you cannot choose not to see
the puppy, or not to notice its cuteness. Nor can you explain how you recognize a cute
puppy; it’s just obvious to you. Thus, we cannot trust our subjective experience: per‐
ception is not trivial at all, and to understand it we must look at how the sensory
modules work.


ception is not trivial at all, and to understand it we must look at how the sensory
modules work.
Convolutional neural networks (CNNs) emerged from the study of the brain’s visual
cortex, and they have been used in image recognition since the 1980s. In the last few
years, thanks to the increase in computational power, the amount of available training
data, and the tricks presented in Chapter 11 for training deep nets, CNNs have man‐
aged to achieve superhuman performance on some complex visual tasks. They power
image search services, self-driving cars, automatic video classification systems, and
more. Moreover, CNNs are not restricted to visual perception: they are also successful
at other tasks, such as voice recognition or natural language processing (NLP); however,
we will focus on visual applications for now.
In this chapter we will present where CNNs came from, what their building blocks
look like, and how to implement them using TensorFlow. Then we will present some


look like, and how to implement them using TensorFlow. Then we will present some
of the best CNN architectures.
353
Download from finelybook www.finelybook.com


1 “Single Unit Activity in Striate Cortex of Unrestrained Cats,” D. Hubel and T. Wiesel (1958).
2 “Receptive Fields of Single Neurones in the Cat’s Striate Cortex,” D. Hubel and T. Wiesel (1959).
3 “Receptive Fields and Functional Architecture of Monkey Striate Cortex,” D. Hubel and T. Wiesel (1968).
4 “Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected
by Shift in Position,” K. Fukushima (1980).
5 “Gradient-Based Learning Applied to Document Recognition,” Y. LeCun et al. (1998).
The Architecture of the Visual Cortex
David H. Hubel and Torsten Wiesel performed a series of experiments on cats in
19581 and 19592 (and a few years later on monkeys3), giving crucial insights on the
structure of the visual cortex (the authors received the Nobel Prize in Physiology or
Medicine in 1981 for their work). In particular, they showed that many neurons in
the visual cortex have a small local receptive field, meaning they react only to visual


the visual cortex have a small local receptive field, meaning they react only to visual
stimuli located in a limited region of the visual field (see Figure 13-1, in which the
local receptive fields of five neurons are represented by dashed circles). The receptive
fields of different neurons may overlap, and together they tile the whole visual field.
Moreover, the authors showed that some neurons react only to images of horizontal
lines, while others react only to lines with different orientations (two neurons may
have the same receptive field but react to different line orientations). They also
noticed that some neurons have larger receptive fields, and they react to more com‐
plex patterns that are combinations of the lower-level patterns. These observations
led to the idea that the higher-level neurons are based on the outputs of neighboring
lower-level neurons (in Figure 13-1, notice that each neuron is connected only to a


lower-level neurons (in Figure 13-1, notice that each neuron is connected only to a
few neurons from the previous layer). This powerful architecture is able to detect all
sorts of complex patterns in any area of the visual field.
Figure 13-1. Local receptive fields in the visual cortex
These studies of the visual cortex inspired the neocognitron, introduced in 1980,4
which gradually evolved into what we now call convolutional neural networks. An
important milestone was a 1998 paper5 by Yann LeCun, Léon Bottou, Yoshua Bengio,
354
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


6 A convolution is a mathematical operation that slides one function over another and measures the integral of
their pointwise multiplication. It has deep connections with the Fourier transform and the Laplace transform,
and is heavily used in signal processing. Convolutional layers actually use cross-correlations, which are very
similar to convolutions (see http://goo.gl/HAfxXd for more details).
and Patrick Haffner, which introduced the famous LeNet-5 architecture, widely used
to recognize handwritten check numbers. This architecture has some building blocks
that you already know, such as fully connected layers and sigmoid activation func‐
tions, but it also introduces two new building blocks: convolutional layers and pooling
layers. Let’s look at them now.
Why not simply use a regular deep neural network with fully con‐
nected layers for image recognition tasks? Unfortunately, although
this works fine for small images (e.g., MNIST), it breaks down for


this works fine for small images (e.g., MNIST), it breaks down for
larger images because of the huge number of parameters it
requires. For example, a 100 × 100 image has 10,000 pixels, and if
the first layer has just 1,000 neurons (which already severely
restricts the amount of information transmitted to the next layer),
this means a total of 10 million connections. And that’s just the first
layer. CNNs solve this problem using partially connected layers.
Convolutional Layer
The most important building block of a CNN is the convolutional layer:6 neurons in
the first convolutional layer are not connected to every single pixel in the input image
(like they were in previous chapters), but only to pixels in their receptive fields (see
Figure 13-2). In turn, each neuron in the second convolutional layer is connected
only to neurons located within a small rectangle in the first layer. This architecture
allows the network to concentrate on low-level features in the first hidden layer, then


allows the network to concentrate on low-level features in the first hidden layer, then
assemble them into higher-level features in the next hidden layer, and so on. This
hierarchical structure is common in real-world images, which is one of the reasons
why CNNs work so well for image recognition.
Convolutional Layer
|
355
Download from finelybook www.finelybook.com


Figure 13-2. CNN layers with rectangular local receptive fields
Until now, all multilayer neural networks we looked at had layers
composed of a long line of neurons, and we had to flatten input
images to 1D before feeding them to the neural network. Now each
layer is represented in 2D, which makes it easier to match neurons
with their corresponding inputs.
A neuron located in row i, column j of a given layer is connected to the outputs of the
neurons in the previous layer located in rows i to i + fh – 1, columns j to j + fw – 1,
where fh and fw are the height and width of the receptive field (see Figure 13-3). In
order for a layer to have the same height and width as the previous layer, it is com‐
mon to add zeros around the inputs, as shown in the diagram. This is called zero pad‐
ding.
Figure 13-3. Connections between layers and zero padding
356
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


It is also possible to connect a large input layer to a much smaller layer by spacing out
the receptive fields, as shown in Figure 13-4. The distance between two consecutive
receptive fields is called the stride. In the diagram, a 5 × 7 input layer (plus zero pad‐
ding) is connected to a 3 × 4 layer, using 3 × 3 receptive fields and a stride of 2 (in this
example the stride is the same in both directions, but it does not have to be so). A
neuron located in row i, column j in the upper layer is connected to the outputs of the
neurons in the previous layer located in rows i × sh to i × sh + fh – 1, columns j × sw +
fw – 1, where sh and sw are the vertical and horizontal strides.
Figure 13-4. Reducing dimensionality using a stride
Filters
A neuron’s weights can be represented as a small image the size of the receptive field.
For example, Figure 13-5 shows two possible sets of weights, called filters (or convolu‐


For example, Figure 13-5 shows two possible sets of weights, called filters (or convolu‐
tion kernels). The first one is represented as a black square with a vertical white line in
the middle (it is a 7 × 7 matrix full of 0s except for the central column, which is full of
1s); neurons using these weights will ignore everything in their receptive field except
for the central vertical line (since all inputs will get multiplied by 0, except for the
ones located in the central vertical line). The second filter is a black square with a
horizontal white line in the middle. Once again, neurons using these weights will
ignore everything in their receptive field except for the central horizontal line.
Now if all neurons in a layer use the same vertical line filter (and the same bias term),
and you feed the network the input image shown in Figure 13-5 (bottom image), the
layer will output the top-left image. Notice that the vertical white lines get enhanced


layer will output the top-left image. Notice that the vertical white lines get enhanced
while the rest gets blurred. Similarly, the upper-right image is what you get if all neu‐
rons use the horizontal line filter; notice that the horizontal white lines get enhanced
while the rest is blurred out. Thus, a layer full of neurons using the same filter gives
Convolutional Layer
|
357
Download from finelybook www.finelybook.com


you a feature map, which highlights the areas in an image that are most similar to the
filter. During training, a CNN finds the most useful filters for its task, and it learns to
combine them into more complex patterns (e.g., a cross is an area in an image where
both the vertical filter and the horizontal filter are active).
Figure 13-5. Applying two different filters to get two feature maps
Stacking Multiple Feature Maps
Up to now, for simplicity, we have represented each convolutional layer as a thin 2D
layer, but in reality it is composed of several feature maps of equal sizes, so it is more
accurately represented in 3D (see Figure 13-6). Within one feature map, all neurons
share the same parameters (weights and bias term), but different feature maps may
have different parameters. A neuron’s receptive field is the same as described earlier,
but it extends across all the previous layers’ feature maps. In short, a convolutional


but it extends across all the previous layers’ feature maps. In short, a convolutional
layer simultaneously applies multiple filters to its inputs, making it capable of detect‐
ing multiple features anywhere in its inputs.
The fact that all neurons in a feature map share the same parame‐
ters dramatically reduces the number of parameters in the model,
but most importantly it means that once the CNN has learned to
recognize a pattern in one location, it can recognize it in any other
location. In contrast, once a regular DNN has learned to recognize
a pattern in one location, it can recognize it only in that particular
location.
358
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


Moreover, input images are also composed of multiple sublayers: one per color chan‐
nel. There are typically three: red, green, and blue (RGB). Grayscale images have just
one channel, but some images may have much more—for example, satellite images
that capture extra light frequencies (such as infrared).
Figure 13-6. Convolution layers with multiple feature maps, and images with three
channels
Specifically, a neuron located in row i, column j of the feature map k in a given convo‐
lutional layer l is connected to the outputs of the neurons in the previous layer l – 1,
located in rows i × sw to i × sw + fw – 1 and columns j × sh to j × sh + fh – 1, across all
feature maps (in layer l – 1). Note that all neurons located in the same row i and col‐
umn j but in different feature maps are connected to the outputs of the exact same
neurons in the previous layer.
Equation 13-1 summarizes the preceding explanations in one big mathematical equa‐


Equation 13-1 summarizes the preceding explanations in one big mathematical equa‐
tion: it shows how to compute the output of a given neuron in a convolutional layer.
Convolutional Layer
|
359
Download from finelybook www.finelybook.com


It is a bit ugly due to all the different indices, but all it does is calculate the weighted
sum of all the inputs, plus the bias term.
Equation 13-1. Computing the output of a neuron in a convolutional layer
zi, j, k = bk + ∑
u = 1
f h
∑
v = 1
f w
∑
k′ = 1
f n′
xi′, j′, k′ . wu, v, k′, k
with
i′ = u . sh + f h −1
j′ = v . sw + f w −1
• zi, j, k is the output of the neuron located in row i, column j in feature map k of the
convolutional layer (layer l).
• As explained earlier, sh and sw are the vertical and horizontal strides, fh and fw are
the height and width of the receptive field, and fn′ is the number of feature maps
in the previous layer (layer l – 1).
• xi′, j′, k′ is the output of the neuron located in layer l – 1, row i′, column j′, feature
map k′ (or channel k′ if the previous layer is the input layer).
• bk is the bias term for feature map k (in layer l). You can think of it as a knob that
tweaks the overall brightness of the feature map k.


tweaks the overall brightness of the feature map k.
• wu, v, k′ ,k is the connection weight between any neuron in feature map k of the layer
l and its input located at row u, column v (relative to the neuron’s receptive field),
and feature map k′.
TensorFlow Implementation
In TensorFlow, each input image is typically represented as a 3D tensor of shape
[height, width, channels]. A mini-batch is represented as a 4D tensor of shape
[mini-batch size, height, width, channels]. The weights of a convolutional
layer are represented as a 4D tensor of shape [fh, fw, fn, fn′]. The bias terms of a convo‐
lutional layer are simply represented as a 1D tensor of shape [fn].
Let’s look at a simple example. The following code loads two sample images, using
Scikit-Learn’s load_sample_images() (which loads two color images, one of a Chi‐
nese temple, and the other of a flower). Then it creates two 7 × 7 filters (one with a


nese temple, and the other of a flower). Then it creates two 7 × 7 filters (one with a
vertical white line in the middle, and the other with a horizontal white line), and
applies them to both images using a convolutional layer built using TensorFlow’s
conv2d() function (with zero padding and a stride of 2). Finally, it plots one of the
resulting feature maps (similar to the top-right image in Figure 13-5).
360
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


import numpy as np
from sklearn.datasets import load_sample_images
# Load sample images
dataset = np.array(load_sample_images().images, dtype=np.float32)
batch_size, height, width, channels = dataset.shape
# Create 2 filters
filters_test = np.zeros(shape=(7, 7, channels, 2), dtype=np.float32)
filters_test[:, 3, :, 0] = 1  # vertical line
filters_test[3, :, :, 1] = 1  # horizontal line
# Create a graph with input X plus a convolutional layer applying the 2 filters
X = tf.placeholder(tf.float32, shape=(None, height, width, channels))
convolution = tf.nn.conv2d(X, filters, strides=[1,2,2,1], padding="SAME")
with tf.Session() as sess:
    output = sess.run(convolution, feed_dict={X: dataset})
plt.imshow(output[0, :, :, 1])  # plot 1st image's 2nd feature map
plt.show()
Most of this code is self-explanatory, but the conv2d() line deserves a bit of explana‐
tion:
• X is the input mini-batch (a 4D tensor, as explained earlier).


tion:
• X is the input mini-batch (a 4D tensor, as explained earlier).
• filters is the set of filters to apply (also a 4D tensor, as explained earlier).
• strides is a four-element 1D array, where the two central elements are the verti‐
cal and horizontal strides (sh and sw). The first and last elements must currently
be equal to 1. They may one day be used to specify a batch stride (to skip some
instances) and a channel stride (to skip some of the previous layer’s feature maps
or channels).
• padding must be either "VALID" or "SAME":
— If set to "VALID", the convolutional layer does not use zero padding, and may
ignore some rows and columns at the bottom and right of the input image,
depending on the stride, as shown in Figure 13-7 (for simplicity, only the hor‐
izontal dimension is shown here, but of course the same logic applies to the
vertical dimension).
— If set to "SAME", the convolutional layer uses zero padding if necessary. In this


— If set to "SAME", the convolutional layer uses zero padding if necessary. In this
case, the number of output neurons is equal to the number of input neurons
divided by the stride, rounded up (in this example, ceil (13 / 5) = 3). Then
zeros are added as evenly as possible around the inputs.
Convolutional Layer
|
361
Download from finelybook www.finelybook.com


7 A fully connected layer with 150 × 100 neurons, each connected to all 150 × 100 × 3 inputs, would have 1502
× 1002 × 3 = 675 million parameters!
Figure 13-7. Padding options—input width: 13, filter width: 6, stride: 5
Unfortunately, convolutional layers have quite a few hyperparameters: you must
choose the number of filters, their height and width, the strides, and the padding
type. As always, you can use cross-validation to find the right hyperparameter values,
but this is very time-consuming. We will discuss common CNN architectures later, to
give you some idea of what hyperparameter values work best in practice.
Memory Requirements
Another problem with CNNs is that the convolutional layers require a huge amount
of RAM, especially during training, because the reverse pass of backpropagation
requires all the intermediate values computed during the forward pass.
For example, consider a convolutional layer with 5 × 5 filters, outputting 200 feature


For example, consider a convolutional layer with 5 × 5 filters, outputting 200 feature
maps of size 150 × 100, with stride 1 and SAME padding. If the input is a 150 × 100
RGB image (three channels), then the number of parameters is (5 × 5 × 3 + 1) × 200
= 15,200 (the +1 corresponds to the bias terms), which is fairly small compared to a
fully connected layer.7 However, each of the 200 feature maps contains 150 × 100 neu‐
rons, and each of these neurons needs to compute a weighted sum of its 5 × 5 × 3 =
75 inputs: that’s a total of 225 million float multiplications. Not as bad as a fully con‐
362
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


8 1 MB = 1,024 kB = 1,024 × 1,024 bytes = 1,024 × 1,024 × 8 bits.
nected layer, but still quite computationally intensive. Moreover, if the feature maps
are represented using 32-bit floats, then the convolutional layer’s output will occupy
200 × 150 × 100 × 32 = 96 million bits (about 11.4 MB) of RAM.8 And that’s just for
one instance! If a training batch contains 100 instances, then this layer will use up
over 1 GB of RAM!
During inference (i.e., when making a prediction for a new instance) the RAM occu‐
pied by one layer can be released as soon as the next layer has been computed, so you
only need as much RAM as required by two consecutive layers. But during training
everything computed during the forward pass needs to be preserved for the reverse
pass, so the amount of RAM needed is (at least) the total amount of RAM required by
all layers.
If training crashes because of an out-of-memory error, you can try
reducing the mini-batch size. Alternatively, you can try reducing


reducing the mini-batch size. Alternatively, you can try reducing
dimensionality using a stride, or removing a few layers. Or you can
try using 16-bit floats instead of 32-bit floats. Or you could distrib‐
ute the CNN across multiple devices.
Now let’s look at the second common building block of CNNs: the pooling layer.
Pooling Layer
Once you understand how convolutional layers work, the pooling layers are quite
easy to grasp. Their goal is to subsample (i.e., shrink) the input image in order to
reduce the computational load, the memory usage, and the number of parameters
(thereby limiting the risk of overfitting). Reducing the input image size also makes
the neural network tolerate a little bit of image shift (location invariance).
Just like in convolutional layers, each neuron in a pooling layer is connected to the
outputs of a limited number of neurons in the previous layer, located within a small


outputs of a limited number of neurons in the previous layer, located within a small
rectangular receptive field. You must define its size, the stride, and the padding type,
just like before. However, a pooling neuron has no weights; all it does is aggregate the
inputs using an aggregation function such as the max or mean. Figure 13-8 shows a
max pooling layer, which is the most common type of pooling layer. In this example,
we use a 2 × 2 pooling kernel, a stride of 2, and no padding. Note that only the max
input value in each kernel makes it to the next layer. The other inputs are dropped.
Pooling Layer
|
363
Download from finelybook www.finelybook.com


Figure 13-8. Max pooling layer (2 × 2 pooling kernel, stride 2, no padding)
This is obviously a very destructive kind of layer: even with a tiny 2 × 2 kernel and a
stride of 2, the output will be two times smaller in both directions (so its area will be
four times smaller), simply dropping 75% of the input values.
A pooling layer typically works on every input channel independently, so the output
depth is the same as the input depth. You may alternatively pool over the depth
dimension, as we will see next, in which case the image’s spatial dimensions (height
and width) remain unchanged, but the number of channels is reduced.
Implementing a max pooling layer in TensorFlow is quite easy. The following code
creates a max pooling layer using a 2 × 2 kernel, stride 2, and no padding, then
applies it to all the images in the dataset:
[...] # load the image dataset, just like above
# Create a graph with input X plus a max pooling layer


# Create a graph with input X plus a max pooling layer
X = tf.placeholder(tf.float32, shape=(None, height, width, channels))
max_pool = tf.nn.max_pool(X, ksize=[1,2,2,1], strides=[1,2,2,1],padding="VALID")
with tf.Session() as sess:
    output = sess.run(max_pool, feed_dict={X: dataset})
plt.imshow(output[0].astype(np.uint8))  # plot the output for the 1st image
plt.show()
The ksize argument contains the kernel shape along all four dimensions of the input
tensor: [batch size, height, width, channels]. TensorFlow currently does not
support pooling over multiple instances, so the first element of ksize must be equal
to 1. Moreover, it does not support pooling over both the spatial dimensions (height
and width) and the depth dimension, so either ksize[1] and ksize[2] must both be
equal to 1, or ksize[3] must be equal to 1.
To create an average pooling layer, just use the avg_pool() function instead of
max_pool().
364
|
Chapter 13: Convolutional Neural Networks


max_pool().
364
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


Now you know all the building blocks to create a convolutional neural network. Let’s
see how to assemble them.
CNN Architectures
Typical CNN architectures stack a few convolutional layers (each one generally fol‐
lowed by a ReLU layer), then a pooling layer, then another few convolutional layers
(+ReLU), then another pooling layer, and so on. The image gets smaller and smaller
as it progresses through the network, but it also typically gets deeper and deeper (i.e.,
with more feature maps) thanks to the convolutional layers (see Figure 13-9). At the
top of the stack, a regular feedforward neural network is added, composed of a few
fully connected layers (+ReLUs), and the final layer outputs the prediction (e.g., a
softmax layer that outputs estimated class probabilities).
Figure 13-9. Typical CNN architecture
A common mistake is to use convolution kernels that are too large.
You can often get the same effect as a 9 × 9 kernel by stacking two 3


You can often get the same effect as a 9 × 9 kernel by stacking two 3
× 3 kernels on top of each other, for a lot less compute.
Over the years, variants of this fundamental architecture have been developed, lead‐
ing to amazing advances in the field. A good measure of this progress is the error rate
in competitions such as the ILSVRC ImageNet challenge. In this competition the
top-5 error rate for image classification fell from over 26% to barely over 3% in just
five years. The top-five error rate is the number of test images for which the system’s
top 5 predictions did not include the correct answer. The images are large (256 pixels
high) and there are 1,000 classes, some of which are really subtle (try distinguishing
120 dog breeds). Looking at the evolution of the winning entries is a good way to
understand how CNNs work.
We will first look at the classical LeNet-5 architecture (1998), then three of the win‐
ners of the ILSVRC challenge: AlexNet (2012), GoogLeNet (2014), and ResNet


ners of the ILSVRC challenge: AlexNet (2012), GoogLeNet (2014), and ResNet
(2015).
CNN Architectures
|
365
Download from finelybook www.finelybook.com


Other Visual Tasks
There was stunning progress as well in other visual tasks such as object detection and
localization, and image segmentation. In object detection and localization, the neural
network typically outputs a sequence of bounding boxes around various objects in
the image. For example, see Maxine Oquab et al.’s 2015 paper that outputs a heat map
for each object class, or Russell Stewart et al.’s 2015 paper that uses a combination of a
CNN to detect faces and a recurrent neural network to output a sequence of bound‐
ing boxes around them. In image segmentation, the net outputs an image (usually of
the same size as the input) where each pixel indicates the class of the object to which
the corresponding input pixel belongs. For example, check out Evan Shelhamer et al.’s
2016 paper.
LeNet-5
The LeNet-5 architecture is perhaps the most widely known CNN architecture. As
mentioned earlier, it was created by Yann LeCun in 1998 and widely used for hand‐


mentioned earlier, it was created by Yann LeCun in 1998 and widely used for hand‐
written digit recognition (MNIST). It is composed of the layers shown in Table 13-1.
Table 13-1. LeNet-5 architecture
Layer Type
Maps Size
Kernel size Stride
Activation
Out
Fully Connected
–
10
–
–
RBF
F6
Fully Connected
–
84
–
–
tanh
C5
Convolution
120
1 × 1
5 × 5
1
tanh
S4
Avg Pooling
16
5 × 5
2 × 2
2
tanh
C3
Convolution
16
10 × 10 5 × 5
1
tanh
S2
Avg Pooling
6
14 × 14 2 × 2
2
tanh
C1
Convolution
6
28 × 28 5 × 5
1
tanh
In
Input
1
32 × 32 –
–
–
There are a few extra details to be noted:
• MNIST images are 28 × 28 pixels, but they are zero-padded to 32 × 32 pixels and
normalized before being fed to the network. The rest of the network does not use
any padding, which is why the size keeps shrinking as the image progresses
through the network.
• The average pooling layers are slightly more complex than usual: each neuron
computes the mean of its inputs, then multiplies the result by a learnable coeffi‐


computes the mean of its inputs, then multiplies the result by a learnable coeffi‐
cient (one per map) and adds a learnable bias term (again, one per map), then
finally applies the activation function.
366
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


9 “ImageNet Classification with Deep Convolutional Neural Networks,” A. Krizhevsky et al. (2012).
• Most neurons in C3 maps are connected to neurons in only three or four S2
maps (instead of all six S2 maps). See table 1 in the original paper for details.
• The output layer is a bit special: instead of computing the dot product of the
inputs and the weight vector, each neuron outputs the square of the Euclidian
distance between its input vector and its weight vector. Each output measures
how much the image belongs to a particular digit class. The cross entropy cost
function is now preferred, as it penalizes bad predictions much more, producing
larger gradients and thus converging faster.
Yann LeCun’s website (“LENET” section) features great demos of LeNet-5 classifying
digits.
AlexNet
The AlexNet CNN architecture9 won the 2012 ImageNet ILSVRC challenge by a large
margin: it achieved 17% top-5 error rate while the second best achieved only 26%! It


margin: it achieved 17% top-5 error rate while the second best achieved only 26%! It
was developed by Alex Krizhevsky (hence the name), Ilya Sutskever, and Geoffrey
Hinton. It is quite similar to LeNet-5, only much larger and deeper, and it was the
first to stack convolutional layers directly on top of each other, instead of stacking a
pooling layer on top of each convolutional layer. Table 13-2 presents this architecture.
Table 13-2. AlexNet architecture
Layer Type
Maps
Size
Kernel size
Stride
Padding
Activation
Out
Fully Connected
–
1,000
–
–
–
Softmax
F9
Fully Connected
–
4,096
–
–
–
ReLU
F8
Fully Connected
–
4,096
–
–
–
ReLU
C7
Convolution
256
13 × 13
3 × 3
1
SAME
ReLU
C6
Convolution
384
13 × 13
3 × 3
1
SAME
ReLU
C5
Convolution
384
13 × 13
3 × 3
1
SAME
ReLU
S4
Max Pooling
256
13 × 13
3 × 3
2
VALID
–
C3
Convolution
256
27 × 27
5 × 5
1
SAME
ReLU
S2
Max Pooling
96
27 × 27
3 × 3
2
VALID
–
C1
Convolution
96
55 × 55
11 × 11
4
SAME
ReLU
In
Input
3 (RGB)
224 × 224 –
–
–
–


3 × 3
2
VALID
–
C1
Convolution
96
55 × 55
11 × 11
4
SAME
ReLU
In
Input
3 (RGB)
224 × 224 –
–
–
–
To reduce overfitting, the authors used two regularization techniques we discussed in
previous chapters: first they applied dropout (with a 50% dropout rate) during train‐
ing to the outputs of layers F8 and F9. Second, they performed data augmentation by
CNN Architectures
|
367
Download from finelybook www.finelybook.com


10 “Going Deeper with Convolutions,” C. Szegedy et al. (2015).
randomly shifting the training images by various offsets, flipping them horizontally,
and changing the lighting conditions.
AlexNet also uses a competitive normalization step immediately after the ReLU step
of layers C1 and C3, called local response normalization. This form of normalization
makes the neurons that most strongly activate inhibit neurons at the same location
but in neighboring feature maps (such competitive activation has been observed in
biological neurons). This encourages different feature maps to specialize, pushing
them apart and forcing them to explore a wider range of features, ultimately improv‐
ing generalization. Equation 13-2 shows how to apply LRN.
Equation 13-2. Local response normalization
bi = ai k + α ∑
j = jlow
jhigh
aj
2
−β
with
jhigh = min i + r
2, f n −1
jlow = max 0, i −r
2
• bi is the normalized output of the neuron located in feature map i, at some row u


2
• bi is the normalized output of the neuron located in feature map i, at some row u
and column v (note that in this equation we consider only neurons located at this
row and column, so u and v are not shown).
• ai is the activation of that neuron after the ReLU step, but before normalization.
• k, α, β, and r are hyperparameters. k is called the bias, and r is called the depth
radius.
• fn is the number of feature maps.
For example, if r = 2 and a neuron has a strong activation, it will inhibit the activation
of the neurons located in the feature maps immediately above and below its own.
In AlexNet, the hyperparameters are set as follows: r = 2, α = 0.00002, β = 0.75, and k
= 1. This step can be implemented using TensorFlow’s local_response_normaliza
tion() operation.
A variant of AlexNet called ZF Net was developed by Matthew Zeiler and Rob Fergus
and won the 2013 ILSVRC challenge. It is essentially AlexNet with a few tweaked


and won the 2013 ILSVRC challenge. It is essentially AlexNet with a few tweaked
hyperparameters (number of feature maps, kernel size, stride, etc.).
GoogLeNet
The GoogLeNet architecture was developed by Christian Szegedy et al. from Google
Research,10 and it won the ILSVRC 2014 challenge by pushing the top-5 error rate
368
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


11 In the 2010 movie Inception, the characters keep going deeper and deeper into multiple layers of dreams,
hence the name of these modules.
below 7%. This great performance came in large part from the fact that the network
was much deeper than previous CNNs (see Figure 13-11). This was made possible by
sub-networks called inception modules,11 which allow GoogLeNet to use parameters
much more efficiently than previous architectures: GoogLeNet actually has 10 times
fewer parameters than AlexNet (roughly 6 million instead of 60 million).
Figure 13-10 shows the architecture of an inception module. The notation “3 × 3 +
2(S)” means that the layer uses a 3 × 3 kernel, stride 2, and SAME padding. The input
signal is first copied and fed to four different layers. All convolutional layers use the
ReLU activation function. Note that the second set of convolutional layers uses differ‐
ent kernel sizes (1 × 1, 3 × 3, and 5 × 5), allowing them to capture patterns at different


ent kernel sizes (1 × 1, 3 × 3, and 5 × 5), allowing them to capture patterns at different
scales. Also note that every single layer uses a stride of 1 and SAME padding (even
the max pooling layer), so their outputs all have the same height and width as their
inputs. This makes it possible to concatenate all the outputs along the depth dimen‐
sion in the final depth concat layer (i.e., stack the feature maps from all four top con‐
volutional layers). This concatenation layer can be implemented in TensorFlow using
the concat() operation, with axis=3 (axis 3 is the depth).
Figure 13-10. Inception module
You may wonder why inception modules have convolutional layers with 1 × 1 ker‐
nels. Surely these layers cannot capture any features since they look at only one pixel
at a time? In fact, these layers serve two purposes:
• First, they are configured to output many fewer feature maps than their inputs, so
they serve as bottleneck layers, meaning they reduce dimensionality. This is par‐


they serve as bottleneck layers, meaning they reduce dimensionality. This is par‐
CNN Architectures
|
369
Download from finelybook www.finelybook.com


ticularly useful before the 3 × 3 and 5 × 5 convolutions, since these are very com‐
putationally expensive layers.
• Second, each pair of convolutional layers ([1 × 1, 3 × 3] and [1 × 1, 5 × 5]) acts
like a single, powerful convolutional layer, capable of capturing more complex
patterns. Indeed, instead of sweeping a simple linear classifier across the image
(as a single convolutional layer does), this pair of convolutional layers sweeps a
two-layer neural network across the image.
In short, you can think of the whole inception module as a convolutional layer on
steroids, able to output feature maps that capture complex patterns at various scales.
The number of convolutional kernels for each convolutional layer
is a hyperparameter. Unfortunately, this means that you have six
more hyperparameters to tweak for every inception layer you add.
Now let’s look at the architecture of the GoogLeNet CNN (see Figure 13-11). It is so


Now let’s look at the architecture of the GoogLeNet CNN (see Figure 13-11). It is so
deep that we had to represent it in three columns, but GoogLeNet is actually one tall
stack, including nine inception modules (the boxes with the spinning tops) that
actually contain three layers each. The number of feature maps output by each convo‐
lutional layer and each pooling layer is shown before the kernel size. The six numbers
in the inception modules represent the number of feature maps output by each con‐
volutional layer in the module (in the same order as in Figure 13-10). Note that all the
convolutional layers use the ReLU activation function.
370
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


Figure 13-11. GoogLeNet architecture
Let’s go through this network:
• The first two layers divide the image’s height and width by 4 (so its area is divided
by 16), to reduce the computational load.
• Then the local response normalization layer ensures that the previous layers learn
a wide variety of features (as discussed earlier).
• Two convolutional layers follow, where the first acts like a bottleneck layer. As
explained earlier, you can think of this pair as a single smarter convolutional
layer.
• Again, a local response normalization layer ensures that the previous layers cap‐
ture a wide variety of patterns.
• Next a max pooling layer reduces the image height and width by 2, again to speed
up computations.
CNN Architectures
|
371
Download from finelybook www.finelybook.com


12 “Deep Residual Learning for Image Recognition,” K. He (2015).
• Then comes the tall stack of nine inception modules, interleaved with a couple
max pooling layers to reduce dimensionality and speed up the net.
• Next, the average pooling layer uses a kernel the size of the feature maps with
VALID padding, outputting 1 × 1 feature maps: this surprising strategy is called
global average pooling. It effectively forces the previous layers to produce feature
maps that are actually confidence maps for each target class (since other kinds of
features would be destroyed by the averaging step). This makes it unnecessary to
have several fully connected layers at the top of the CNN (like in AlexNet), con‐
siderably reducing the number of parameters in the network and limiting the risk
of overfitting.
• The last layers are self-explanatory: dropout for regularization, then a fully con‐
nected layer with a softmax activation function to output estimated class proba‐
bilities.


nected layer with a softmax activation function to output estimated class proba‐
bilities.
This diagram is slightly simplified: the original GoogLeNet architecture also included
two auxiliary classifiers plugged on top of the third and sixth inception modules.
They were both composed of one average pooling layer, one convolutional layer, two
fully connected layers, and a softmax activation layer. During training, their loss
(scaled down by 70%) was added to the overall loss. The goal was to fight the vanish‐
ing gradients problem and regularize the network. However, it was shown that their
effect was relatively minor.
ResNet
Last but not least, the winner of the ILSVRC 2015 challenge was the Residual Network
(or ResNet), developed by Kaiming He et al.,12 which delivered an astounding top-5
error rate under 3.6%, using an extremely deep CNN composed of 152 layers. The
key to being able to train such a deep network is to use skip connections (also called


key to being able to train such a deep network is to use skip connections (also called
shortcut connections): the signal feeding into a layer is also added to the output of a
layer located a bit higher up the stack. Let’s see why this is useful.
When training a neural network, the goal is to make it model a target function h(x).
If you add the input x to the output of the network (i.e., you add a skip connection),
then the network will be forced to model f(x) = h(x) – x rather than h(x). This is
called residual learning (see Figure 13-12).
372
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


Figure 13-12. Residual learning
When you initialize a regular neural network, its weights are close to zero, so the net‐
work just outputs values close to zero. If you add a skip connection, the resulting net‐
work just outputs a copy of its inputs; in other words, it initially models the identity
function. If the target function is fairly close to the identity function (which is often
the case), this will speed up training considerably.
Moreover, if you add many skip connections, the network can start making progress
even if several layers have not started learning yet (see Figure 13-13). Thanks to skip
connections, the signal can easily make its way across the whole network. The deep
residual network can be seen as a stack of residual units, where each residual unit is a
small neural network with a skip connection.
Figure 13-13. Regular deep neural network (left) and deep residual network (right)
CNN Architectures
|
373
Download from finelybook www.finelybook.com


Now let’s look at ResNet’s architecture (see Figure 13-14). It is actually surprisingly
simple. It starts and ends exactly like GoogLeNet (except without a dropout layer),
and in between is just a very deep stack of simple residual units. Each residual unit is
composed of two convolutional layers, with Batch Normalization (BN) and ReLU
activation, using 3 × 3 kernels and preserving spatial dimensions (stride 1, SAME
padding).
Figure 13-14. ResNet architecture
Note that the number of feature maps is doubled every few residual units, at the same
time as their height and width are halved (using a convolutional layer with stride 2).
When this happens the inputs cannot be added directly to the outputs of the residual
unit since they don’t have the same shape (for example, this problem affects the skip
connection represented by the dashed arrow in Figure 13-14). To solve this problem,
the inputs are passed through a 1 × 1 convolutional layer with stride 2 and the right


the inputs are passed through a 1 × 1 convolutional layer with stride 2 and the right
number of output feature maps (see Figure 13-15).
Figure 13-15. Skip connection when changing feature map size and depth
374
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


13 “Very Deep Convolutional Networks for Large-Scale Image Recognition,” K. Simonyan and A. Zisserman
(2015).
14 “Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning,” C. Szegedy et al.
(2016).
ResNet-34 is the ResNet with 34 layers (only counting the convolutional layers and
the fully connected layer) containing three residual units that output 64 feature maps,
4 RUs with 128 maps, 6 RUs with 256 maps, and 3 RUs with 512 maps.
ResNets deeper than that, such as ResNet-152, use slightly different residual units.
Instead of two 3 × 3 convolutional layers with (say) 256 feature maps, they use three
convolutional layers: first a 1 × 1 convolutional layer with just 64 feature maps (4
times less), which acts a a bottleneck layer (as discussed already), then a 3 × 3 layer
with 64 feature maps, and finally another 1 × 1 convolutional layer with 256 feature
maps (4 times 64) that restores the original depth. ResNet-152 contains three such


maps (4 times 64) that restores the original depth. ResNet-152 contains three such
RUs that output 256 maps, then 8 RUs with 512 maps, a whopping 36 RUs with 1,024
maps, and finally 3 RUs with 2,048 maps.
As you can see, the field is moving rapidly, with all sorts of architectures popping out
every year. One clear trend is that CNNs keep getting deeper and deeper. They are
also getting lighter, requiring fewer and fewer parameters. At present, the ResNet
architecture is both the most powerful and arguably the simplest, so it is really the
one you should probably use for now, but keep looking at the ILSVRC challenge
every year. The 2016 winners were the Trimps-Soushen team from China with an
astounding 2.99% error rate. To achieve this they trained combinations of the previ‐
ous models and joined them into an ensemble. Depending on the task, the reduced
error rate may or may not be worth the extra complexity.


error rate may or may not be worth the extra complexity.
There are a few other architectures that you may want to look at, in particular
VGGNet13 (runner-up of the ILSVRC 2014 challenge) and Inception-v414 (which
merges the ideas of GoogLeNet and ResNet and achieves close to 3% top-5 error rate
on ImageNet classification).
There is really nothing special about implementing the various
CNN architectures we just discussed. We saw earlier how to build
all the individual building blocks, so now all you need is to assem‐
ble them to create the desired architecture. We will build ResNet-34
in the upcoming exercises and you will find full working code in
the Jupyter notebooks.
CNN Architectures
|
375
Download from finelybook www.finelybook.com


15 This name is quite misleading since this layer does not perform a deconvolution, which is a well-defined
mathematical operation (the inverse of a convolution).
TensorFlow Convolution Operations
TensorFlow also offers a few other kinds of convolutional layers:
• conv1d() creates a convolutional layer for 1D inputs. This is useful, for example,
in natural language processing, where a sentence may be represented as a 1D
array of words, and the receptive field covers a few neighboring words.
• conv3d() creates a convolutional layer for 3D inputs, such as 3D PET scan.
• atrous_conv2d() creates an atrous convolutional layer (“à trous” is French for
“with holes”). This is equivalent to using a regular convolutional layer with a fil‐
ter dilated by inserting rows and columns of zeros (i.e., holes). For example, a 1 ×
3 filter equal to [[1,2,3]] may be dilated with a dilation rate of 4, resulting in a
dilated filter [[1, 0, 0, 0, 2, 0, 0, 0, 3]]. This allows the convolutional


dilated filter [[1, 0, 0, 0, 2, 0, 0, 0, 3]]. This allows the convolutional
layer to have a larger receptive field at no computational price and using no extra
parameters.
• conv2d_transpose() creates a transpose convolutional layer, sometimes called a
deconvolutional layer,15 which upsamples an image. It does so by inserting zeros
between the inputs, so you can think of this as a regular convolutional layer using
a fractional stride. Upsampling is useful, for example, in image segmentation: in a
typical CNN, feature maps get smaller and smaller as you progress through the
network, so if you want to output an image of the same size as the input, you
need an upsampling layer.
• depthwise_conv2d() creates a depthwise convolutional layer that applies every fil‐
ter to every individual input channel independently. Thus, if there are fn filters
and fn′ input channels, then this will output fn × fn′ feature maps.


and fn′ input channels, then this will output fn × fn′ feature maps.
• separable_conv2d() creates a separable convolutional layer that first acts like a
depthwise convolutional layer, then applies a 1 × 1 convolutional layer to the
resulting feature maps. This makes it possible to apply filters to arbitrary sets of
inputs channels.
Exercises
1. What are the advantages of a CNN over a fully connected DNN for image classi‐
fication?
2. Consider a CNN composed of three convolutional layers, each with 3 × 3 kernels,
a stride of 2, and SAME padding. The lowest layer outputs 100 feature maps, the
376
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


middle one outputs 200, and the top one outputs 400. The input images are RGB
images of 200 × 300 pixels. What is the total number of parameters in the CNN?
If we are using 32-bit floats, at least how much RAM will this network require
when making a prediction for a single instance? What about when training on a
mini-batch of 50 images?
3. If your GPU runs out of memory while training a CNN, what are five things you
could try to solve the problem?
4. Why would you want to add a max pooling layer rather than a convolutional
layer with the same stride?
5. When would you want to add a local response normalization layer?
6. Can you name the main innovations in AlexNet, compared to LeNet-5? What
about the main innovations in GoogLeNet and ResNet?
7. Build your own CNN and try to achieve the highest possible accuracy on MNIST.
8. Classifying large images using Inception v3.
a. Download some images of various animals. Load them in Python, for example


a. Download some images of various animals. Load them in Python, for example
using the matplotlib.image.mpimg.imread() function. Resize and/or crop
them to 299 × 299 pixels, and ensure that they have just three channels (RGB),
with no transparency channel.
b. Download the latest pretrained Inception v3 model: the checkpoint is avail‐
able at https://goo.gl/nxSQvl.
c. Create the Inception v3 model by calling the inception_v3() function, as
shown below. This must be done within an argument scope created by the
inception_v3_arg_scope() function. Also, you must set is_training=False
and num_classes=1001 like so:
from tensorflow.contrib.slim.nets import inception
import tensorflow.contrib.slim as slim
X = tf.placeholder(tf.float32, shape=[None, 299, 299, 3])
with slim.arg_scope(inception.inception_v3_arg_scope()):
    logits, end_points = inception.inception_v3(
                             X, num_classes=1001, is_training=False)
predictions = end_points["Predictions"]


predictions = end_points["Predictions"]
saver = tf.train.Saver()
d. Open a session and use the Saver to restore the pretrained model checkpoint
you downloaded earlier.
e. Run the model to classify the images you prepared. Display the top five pre‐
dictions for each image, along with the estimated probability (the list of class
names is available at https://goo.gl/brXRtZ). How accurate is the model?
9. Transfer learning for large image classification.
Exercises
|
377
Download from finelybook www.finelybook.com


a. Create a training set containing at least 100 images per class. For example, you
could classify your own pictures based on the location (beach, mountain, city,
etc.), or alternatively you can just use an existing dataset, such as the flowers
dataset or MIT’s places dataset (requires registration, and it is huge).
b. Write a preprocessing step that will resize and crop the image to 299 × 299,
with some randomness for data augmentation.
c. Using the pretrained Inception v3 model from the previous exercise, freeze all
layers up to the bottleneck layer (i.e., the last layer before the output layer),
and replace the output layer with the appropriate number of outputs for your
new classification task (e.g., the flowers dataset has five mutually exclusive
classes so the output layer must have five neurons and use the softmax activa‐
tion function).
d. Split your dataset into a training set and a test set. Train the model on the
training set and evaluate it on the test set.


training set and evaluate it on the test set.
10. Go through TensorFlow’s DeepDream tutorial. It is a fun way to familiarize your‐
self with various ways of visualizing the patterns learned by a CNN, and to gener‐
ate art using Deep Learning.
Solutions to these exercises are available in Appendix A.
378
|
Chapter 13: Convolutional Neural Networks
Download from finelybook www.finelybook.com


CHAPTER 14
Recurrent Neural Networks
The batter hits the ball. You immediately start running, anticipating the ball’s trajec‐
tory. You track it and adapt your movements, and finally catch it (under a thunder of
applause). Predicting the future is what you do all the time, whether you are finishing
a friend’s sentence or anticipating the smell of coffee at breakfast. In this chapter, we
are going to discuss recurrent neural networks (RNN), a class of nets that can predict
the future (well, up to a point, of course). They can analyze time series data such as
stock prices, and tell you when to buy or sell. In autonomous driving systems, they
can anticipate car trajectories and help avoid accidents. More generally, they can work
on sequences of arbitrary lengths, rather than on fixed-sized inputs like all the nets we
have discussed so far. For example, they can take sentences, documents, or audio
samples as input, making them extremely useful for natural language processing


samples as input, making them extremely useful for natural language processing
(NLP) systems such as automatic translation, speech-to-text, or sentiment analysis
(e.g., reading movie reviews and extracting the rater’s feeling about the movie).
Moreover, RNNs’ ability to anticipate also makes them capable of surprising creativ‐
ity. You can ask them to predict which are the most likely next notes in a melody, then
randomly pick one of these notes and play it. Then ask the net for the next most likely
notes, play it, and repeat the process again and again. Before you know it, your net
will compose a melody such as the one produced by Google’s Magenta project. Simi‐
larly, RNNs can generate sentences, image captions, and much more. The result is not
exactly Shakespeare or Mozart yet, but who knows what they will produce a few years
from now?
In this chapter, we will look at the fundamental concepts underlying RNNs, the main


from now?
In this chapter, we will look at the fundamental concepts underlying RNNs, the main
problem they face (namely, vanishing/exploding gradients, discussed in Chapter 11),
and the solutions widely used to fight it: LSTM and GRU cells. Along the way, as
always, we will show how to implement RNNs using TensorFlow. Finally, we will take
a look at the architecture of a machine translation system.
379
Download from finelybook www.finelybook.com


Recurrent Neurons
Up to now we have mostly looked at feedforward neural networks, where the activa‐
tions flow only in one direction, from the input layer to the output layer (except for a
few networks in Appendix E). A recurrent neural network looks very much like a
feedforward neural network, except it also has connections pointing backward. Let’s
look at the simplest possible RNN, composed of just one neuron receiving inputs,
producing an output, and sending that output back to itself, as shown in Figure 14-1
(left). At each time step t (also called a frame), this recurrent neuron receives the inputs
x(t) as well as its own output from the previous time step, y(t–1). We can represent this
tiny network against the time axis, as shown in Figure 14-1 (right). This is called
unrolling the network through time.
Figure 14-1. A recurrent neuron (left), unrolled through time (right)
You can easily create a layer of recurrent neurons. At each time step t, every neuron


You can easily create a layer of recurrent neurons. At each time step t, every neuron
receives both the input vector x(t) and the output vector from the previous time step
y(t–1), as shown in Figure 14-2. Note that both the inputs and outputs are vectors now
(when there was just a single neuron, the output was a scalar).
Figure 14-2. A layer of recurrent neurons (left), unrolled through time (right)
Each recurrent neuron has two sets of weights: one for the inputs x(t) and the other for
the outputs of the previous time step, y(t–1). Let’s call these weight vectors wx and wy.
380
|
Chapter 14: Recurrent Neural Networks
Download from finelybook www.finelybook.com


1 Note that many researchers prefer to use the hyperbolic tangent (tanh) activation function in RNNs rather
than the ReLU activation function. For example, take a look at by Vu Pham et al.’s paper “Dropout Improves
Recurrent Neural Networks for Handwriting Recognition”. However, ReLU-based RNNs are also possible, as
shown in Quoc V. Le et al.’s paper “A Simple Way to Initialize Recurrent Networks of Rectified Linear Units”.
The output of a single recurrent neuron can be computed pretty much as you might
expect, as shown in Equation 14-1 (b is the bias term and ϕ(·) is the activation func‐
tion, e.g., ReLU1).
Equation 14-1. Output of a single recurrent neuron for a single instance
�t = ϕ �t
T · �x + �t −1
T · �y + b
Just like for feedforward neural networks, we can compute a whole layer’s output in
one shot for a whole mini-batch using a vectorized form of the previous equation (see
Equation 14-2).
Equation 14-2. Outputs of a layer of recurrent neurons for all instances in a mini-


Equation 14-2).
Equation 14-2. Outputs of a layer of recurrent neurons for all instances in a mini-
batch
�t = ϕ �t · �x + �t −1 · �y + �
= ϕ �t
�t −1
· �+ �with �=
�x
�y
• Y(t) is an m × nneurons matrix containing the layer’s outputs at time step t for each
instance in the mini-batch (m is the number of instances in the mini-batch and
nneurons is the number of neurons).
• X(t) is an m × ninputs matrix containing the inputs for all instances (ninputs is the
number of input features).
• Wx is an ninputs × nneurons matrix containing the connection weights for the inputs
of the current time step.
• Wy is an nneurons × nneurons matrix containing the connection weights for the out‐
puts of the previous time step.
• The weight matrices Wx and Wy are often concatenated into a single weight
matrix W of shape (ninputs + nneurons) × nneurons (see the second line of Equation
14-2).
• b is a vector of size nneurons containing each neuron’s bias term.
Recurrent Neurons
|
381


• b is a vector of size nneurons containing each neuron’s bias term.
Recurrent Neurons
|
381
Download from finelybook www.finelybook.com


Notice that Y(t) is a function of X(t) and Y(t–1), which is a function of X(t–1) and Y(t–2),
which is a function of X(t–2) and Y(t–3), and so on. This makes Y(t) a function of all the
inputs since time t = 0 (that is, X(0), X(1), …, X(t)). At the first time step, t = 0, there are
no previous outputs, so they are typically assumed to be all zeros.
Memory Cells
Since the output of a recurrent neuron at time step t is a function of all the inputs
from previous time steps, you could say it has a form of memory. A part of a neural
network that preserves some state across time steps is called a memory cell (or simply
a cell). A single recurrent neuron, or a layer of recurrent neurons, is a very basic cell,
but later in this chapter we will look at some more complex and powerful types of
cells.
In general a cell’s state at time step t, denoted h(t) (the “h” stands for “hidden”), is a
function of some inputs at that time step and its state at the previous time step: h(t) =