Figure 9-4. Visualizing the graph using TensorBoard
If you want to take a peek at the graph directly within Jupyter, you
can use the show_graph() function available in the notebook for
this chapter. It was originally written by A. Mordvintsev in his great
deepdream tutorial notebook. Another option is to install E. Jang’s
TensorFlow debugger tool which includes a Jupyter extension for
graph visualization (and more).
Name Scopes
When dealing with more complex models such as neural networks, the graph can
easily become cluttered with thousands of nodes. To avoid this, you can create name
scopes to group related nodes. For example, let’s modify the previous code to define
the error and mse ops within a name scope called "loss":
with tf.name_scope("loss") as scope:
    error = y_pred - y
    mse = tf.reduce_mean(tf.square(error), name="mse")
The name of each op defined within the scope is now prefixed with "loss/":
>>> print(error.op.name)
loss/sub
>>> print(mse.op.name)
loss/mse


>>> print(error.op.name)
loss/sub
>>> print(mse.op.name)
loss/mse
In TensorBoard, the mse and error nodes now appear inside the loss namespace,
which appears collapsed by default (Figure 9-5).
Name Scopes
|
245
Download from finelybook www.finelybook.com

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


Figure 9-5. A collapsed namescope in TensorBoard
Modularity
Suppose you want to create a graph that adds the output of two rectified linear units
(ReLU). A ReLU computes a linear function of the inputs, and outputs the result if it
is positive, and 0 otherwise, as shown in Equation 9-1.
Equation 9-1. Rectified linear unit
h�, b �= max �· �+ b, 0
The following code does the job, but it’s quite repetitive:
n_features = 3
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
w1 = tf.Variable(tf.random_normal((n_features, 1)), name="weights1")
w2 = tf.Variable(tf.random_normal((n_features, 1)), name="weights2")
b1 = tf.Variable(0.0, name="bias1")
b2 = tf.Variable(0.0, name="bias2")
z1 = tf.add(tf.matmul(X, w1), b1, name="z1")
z2 = tf.add(tf.matmul(X, w2), b2, name="z2")
relu1 = tf.maximum(z1, 0., name="relu1")
relu2 = tf.maximum(z1, 0., name="relu2")
output = tf.add(relu1, relu2, name="output")


relu2 = tf.maximum(z1, 0., name="relu2")
output = tf.add(relu1, relu2, name="output")
Such repetitive code is hard to maintain and error-prone (in fact, this code contains a
cut-and-paste error; did you spot it?). It would become even worse if you wanted to
246
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


add a few more ReLUs. Fortunately, TensorFlow lets you stay DRY (Don’t Repeat
Yourself): simply create a function to build a ReLU. The following code creates five
ReLUs and outputs their sum (note that add_n() creates an operation that will com‐
pute the sum of a list of tensors):
def relu(X):
    w_shape = (int(X.get_shape()[1]), 1)
    w = tf.Variable(tf.random_normal(w_shape), name="weights")
    b = tf.Variable(0.0, name="bias")
    z = tf.add(tf.matmul(X, w), b, name="z")
    return tf.maximum(z, 0., name="relu")
n_features = 3
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = [relu(X) for i in range(5)]
output = tf.add_n(relus, name="output")
Note that when you create a node, TensorFlow checks whether its name already
exists, and if it does it appends an underscore followed by an index to make the name
unique. So the first ReLU contains nodes named "weights", "bias", "z", and "relu"


unique. So the first ReLU contains nodes named "weights", "bias", "z", and "relu"
(plus many more nodes with their default name, such as "MatMul"); the second ReLU
contains nodes named "weights_1", "bias_1", and so on; the third ReLU contains
nodes named "weights_2", "bias_2", and so on. TensorBoard identifies such series
and collapses them together to reduce clutter (as you can see in Figure 9-6).
Figure 9-6. Collapsed node series
Modularity
|
247
Download from finelybook www.finelybook.com


Using name scopes, you can make the graph much clearer. Simply move all the con‐
tent of the relu() function inside a name scope. Figure 9-7 shows the resulting
graph. Notice that TensorFlow also gives the name scopes unique names by append‐
ing _1, _2, and so on.
def relu(X):
    with tf.name_scope("relu"):
        [...]
Figure 9-7. A clearer graph using name-scoped units
Sharing Variables
If you want to share a variable between various components of your graph, one sim‐
ple option is to create it first, then pass it as a parameter to the functions that need it.
For example, suppose you want to control the ReLU threshold (currently hardcoded
to 0) using a shared threshold variable for all ReLUs. You could just create that vari‐
able first, and then pass it to the relu() function:
def relu(X, threshold):
    with tf.name_scope("relu"):
        [...]
        return tf.maximum(z, threshold, name="max")
threshold = tf.Variable(0.0, name="threshold")


return tf.maximum(z, threshold, name="max")
threshold = tf.Variable(0.0, name="threshold")
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = [relu(X, threshold) for i in range(5)]
output = tf.add_n(relus, name="output")
This works fine: now you can control the threshold for all ReLUs using the threshold
variable. However, if there are many shared parameters such as this one, it will be
painful to have to pass them around as parameters all the time. Many people create a
Python dictionary containing all the variables in their model, and pass it around to
every function. Others create a class for each module (e.g., a ReLU class using class
variables to handle the shared parameter). Yet another option is to set the shared vari‐
able as an attribute of the relu() function upon the first call, like so:
248
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


5 Creating a ReLU class is arguably the cleanest option, but it is rather heavyweight.
def relu(X):
    with tf.name_scope("relu"):
        if not hasattr(relu, "threshold"):
            relu.threshold = tf.Variable(0.0, name="threshold")
        [...]
        return tf.maximum(z, relu.threshold, name="max")
TensorFlow offers another option, which may lead to slightly cleaner and more mod‐
ular code than the previous solutions.5 This solution is a bit tricky to understand at
first, but since it is used a lot in TensorFlow it is worth going into a bit of detail. The
idea is to use the get_variable() function to create the shared variable if it does not
exist yet, or reuse it if it already exists. The desired behavior (creating or reusing) is
controlled by an attribute of the current variable_scope(). For example, the follow‐
ing code will create a variable named "relu/threshold" (as a scalar, since shape=(),
and using 0.0 as the initial value):
with tf.variable_scope("relu"):


and using 0.0 as the initial value):
with tf.variable_scope("relu"):
    threshold = tf.get_variable("threshold", shape=(),
                                initializer=tf.constant_initializer(0.0))
Note that if the variable has already been created by an earlier call to get_vari
able(), this code will raise an exception. This behavior prevents reusing variables by
mistake. If you want to reuse a variable, you need to explicitly say so by setting the
variable scope’s reuse attribute to True (in which case you don’t have to specify the
shape or the initializer):
with tf.variable_scope("relu", reuse=True):
    threshold = tf.get_variable("threshold")
This code will fetch the existing "relu/threshold" variable, or raise an exception if it
does not exist or if it was not created using get_variable(). Alternatively, you can
set the reuse attribute to True inside the block by calling the scope’s reuse_vari
ables() method:
with tf.variable_scope("relu") as scope:
    scope.reuse_variables()


ables() method:
with tf.variable_scope("relu") as scope:
    scope.reuse_variables()
    threshold = tf.get_variable("threshold")
Once reuse is set to True, it cannot be set back to False within the
block. Moreover, if you define other variable scopes inside this one,
they will automatically inherit reuse=True. Lastly, only variables
created by get_variable() can be reused this way.
Sharing Variables
|
249
Download from finelybook www.finelybook.com


Now you have all the pieces you need to make the relu() function access the thres
hold variable without having to pass it as a parameter:
def relu(X):
    with tf.variable_scope("relu", reuse=True):
        threshold = tf.get_variable("threshold")  # reuse existing variable
        [...]
        return tf.maximum(z, threshold, name="max")
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
with tf.variable_scope("relu"):  # create the variable
    threshold = tf.get_variable("threshold", shape=(),
                                initializer=tf.constant_initializer(0.0))
relus = [relu(X) for relu_index in range(5)]
output = tf.add_n(relus, name="output")
This code first defines the relu() function, then creates the relu/threshold variable
(as a scalar that will later be initialized to 0.0) and builds five ReLUs by calling the
relu() function. The relu() function reuses the relu/threshold variable, and cre‐
ates the other ReLU nodes.


ates the other ReLU nodes.
Variables created using get_variable() are always named using
the name of their variable_scope as a prefix (e.g., "relu/thres
hold"), but for all other nodes (including variables created with
tf.Variable()) the variable scope acts like a new name scope. In
particular, if a name scope with an identical name was already cre‐
ated, then a suffix is added to make the name unique. For example,
all nodes created in the preceding code (except the threshold vari‐
able) have a name prefixed with "relu_1/" to "relu_5/", as shown
in Figure 9-8.
Figure 9-8. Five ReLUs sharing the threshold variable
It is somewhat unfortunate that the threshold variable must be defined outside the
relu() function, where all the rest of the ReLU code resides. To fix this, the following
code creates the threshold variable within the relu() function upon the first call,
then reuses it in subsequent calls. Now the relu() function does not have to worry


then reuses it in subsequent calls. Now the relu() function does not have to worry
about name scopes or variable sharing: it just calls get_variable(), which will create
250
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


or reuse the threshold variable (it does not need to know which is the case). The rest
of the code calls relu() five times, making sure to set reuse=False on the first call,
and reuse=True for the other calls.
def relu(X):
    threshold = tf.get_variable("threshold", shape=(),
                                initializer=tf.constant_initializer(0.0))
    [...]
    return tf.maximum(z, threshold, name="max")
X = tf.placeholder(tf.float32, shape=(None, n_features), name="X")
relus = []
for relu_index in range(5):
    with tf.variable_scope("relu", reuse=(relu_index >= 1)) as scope:
        relus.append(relu(X))
output = tf.add_n(relus, name="output")
The resulting graph is slightly different than before, since the shared variable lives
within the first ReLU (see Figure 9-9).
Figure 9-9. Five ReLUs sharing the threshold variable
This concludes this introduction to TensorFlow. We will discuss more advanced top‐


This concludes this introduction to TensorFlow. We will discuss more advanced top‐
ics as we go through the following chapters, in particular many operations related to
deep neural networks, convolutional neural networks, and recurrent neural networks
as well as how to scale up with TensorFlow using multithreading, queues, multiple
GPUs, and multiple servers.
Exercises
1. What are the main benefits of creating a computation graph rather than directly
executing the computations? What are the main drawbacks?
2. Is the statement a_val = a.eval(session=sess) equivalent to a_val =
sess.run(a)?
3. Is the statement a_val, b_val = a.eval(session=sess), b.eval(ses
sion=sess) equivalent to a_val, b_val = sess.run([a, b])?
4. Can you run two graphs in the same session?
Exercises
|
251
Download from finelybook www.finelybook.com


5. If you create a graph g containing a variable w, then start two threads and open a
session in each thread, both using the same graph g, will each session have its
own copy of the variable w or will it be shared?
6. When is a variable initialized? When is it destroyed?
7. What is the difference between a placeholder and a variable?
8. What happens when you run the graph to evaluate an operation that depends on
a placeholder but you don’t feed its value? What happens if the operation does
not depend on the placeholder?
9. When you run a graph, can you feed the output value of any operation, or just
the value of placeholders?
10. How can you set a variable to any value you want (during the execution phase)?
11. How many times does reverse-mode autodiff need to traverse the graph in order
to compute the gradients of the cost function with regards to 10 variables? What
about forward-mode autodiff? And symbolic differentiation?


about forward-mode autodiff? And symbolic differentiation?
12. Implement Logistic Regression with Mini-batch Gradient Descent using Tensor‐
Flow. Train it and evaluate it on the moons dataset (introduced in Chapter 5). Try
adding all the bells and whistles:
• Define the graph within a logistic_regression() function that can be reused
easily.
• Save checkpoints using a Saver at regular intervals during training, and save
the final model at the end of training.
• Restore the last checkpoint upon startup if training was interrupted.
• Define the graph using nice scopes so the graph looks good in TensorBoard.
• Add summaries to visualize the learning curves in TensorBoard.
• Try tweaking some hyperparameters such as the learning rate or the mini-
batch size and look at the shape of the learning curve.
Solutions to these exercises are available in Appendix A.
252
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


1 You can get the best of both worlds by being open to biological inspirations without being afraid to create
biologically unrealistic models, as long as they work well.
CHAPTER 10
Introduction to Artificial Neural Networks
Birds inspired us to fly, burdock plants inspired velcro, and nature has inspired many
other inventions. It seems only logical, then, to look at the brain’s architecture for
inspiration on how to build an intelligent machine. This is the key idea that inspired
artificial neural networks (ANNs). However, although planes were inspired by birds,
they don’t have to flap their wings. Similarly, ANNs have gradually become quite dif‐
ferent from their biological cousins. Some researchers even argue that we should drop
the biological analogy altogether (e.g., by saying “units” rather than “neurons”), lest
we restrict our creativity to biologically plausible systems.1
ANNs are at the very core of Deep Learning. They are versatile, powerful, and scala‐


ANNs are at the very core of Deep Learning. They are versatile, powerful, and scala‐
ble, making them ideal to tackle large and highly complex Machine Learning tasks,
such as classifying billions of images (e.g., Google Images), powering speech recogni‐
tion services (e.g., Apple’s Siri), recommending the best videos to watch to hundreds
of millions of users every day (e.g., YouTube), or learning to beat the world champion
at the game of Go by examining millions of past games and then playing against itself
(DeepMind’s AlphaGo).
In this chapter, we will introduce artificial neural networks, starting with a quick tour
of the very first ANN architectures. Then we will present Multi-Layer Perceptrons
(MLPs) and implement one using TensorFlow to tackle the MNIST digit classification
problem (introduced in Chapter 3).
253
Download from finelybook www.finelybook.com


2 “A Logical Calculus of Ideas Immanent in Nervous Activity,” W. McCulloch and W. Pitts (1943).
From Biological to Artificial Neurons
Surprisingly, ANNs have been around for quite a while: they were first introduced
back in 1943 by the neurophysiologist Warren McCulloch and the mathematician
Walter Pitts. In their landmark paper,2 “A Logical Calculus of Ideas Immanent in
Nervous Activity,” McCulloch and Pitts presented a simplified computational model
of how biological neurons might work together in animal brains to perform complex
computations using propositional logic. This was the first artificial neural network
architecture. Since then many other architectures have been invented, as we will see.
The early successes of ANNs until the 1960s led to the widespread belief that we
would soon be conversing with truly intelligent machines. When it became clear that
this promise would go unfulfilled (at least for quite a while), funding flew elsewhere


this promise would go unfulfilled (at least for quite a while), funding flew elsewhere
and ANNs entered a long dark era. In the early 1980s there was a revival of interest in
ANNs as new network architectures were invented and better training techniques
were developed. But by the 1990s, powerful alternative Machine Learning techniques
such as Support Vector Machines (see Chapter 5) were favored by most researchers,
as they seemed to offer better results and stronger theoretical foundations. Finally, we
are now witnessing yet another wave of interest in ANNs. Will this wave die out like
the previous ones did? There are a few good reasons to believe that this one is differ‐
ent and will have a much more profound impact on our lives:
• There is now a huge quantity of data available to train neural networks, and
ANNs frequently outperform other ML techniques on very large and complex
problems.
• The tremendous increase in computing power since the 1990s now makes it pos‐


problems.
• The tremendous increase in computing power since the 1990s now makes it pos‐
sible to train large neural networks in a reasonable amount of time. This is in
part due to Moore’s Law, but also thanks to the gaming industry, which has pro‐
duced powerful GPU cards by the millions.
• The training algorithms have been improved. To be fair they are only slightly dif‐
ferent from the ones used in the 1990s, but these relatively small tweaks have a
huge positive impact.
• Some theoretical limitations of ANNs have turned out to be benign in practice.
For example, many people thought that ANN training algorithms were doomed
because they were likely to get stuck in local optima, but it turns out that this is
rather rare in practice (or when it is the case, they are usually fairly close to the
global optimum).
• ANNs seem to have entered a virtuous circle of funding and progress. Amazing
products based on ANNs regularly make the headline news, which pulls more
254
|


products based on ANNs regularly make the headline news, which pulls more
254
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


3 Image by Bruce Blaus (Creative Commons 3.0). Reproduced from https://en.wikipedia.org/wiki/Neuron.
and more attention and funding toward them, resulting in more and more pro‐
gress, and even more amazing products.
Biological Neurons
Before we discuss artificial neurons, let’s take a quick look at a biological neuron (rep‐
resented in Figure 10-1). It is an unusual-looking cell mostly found in animal cerebral
cortexes (e.g., your brain), composed of a cell body containing the nucleus and most
of the cell’s complex components, and many branching extensions called dendrites,
plus one very long extension called the axon. The axon’s length may be just a few
times longer than the cell body, or up to tens of thousands of times longer. Near its
extremity the axon splits off into many branches called telodendria, and at the tip of
these branches are minuscule structures called synaptic terminals (or simply synap‐


these branches are minuscule structures called synaptic terminals (or simply synap‐
ses), which are connected to the dendrites (or directly to the cell body) of other neu‐
rons. Biological neurons receive short electrical impulses called signals from other
neurons via these synapses. When a neuron receives a sufficient number of signals
from other neurons within a few milliseconds, it fires its own signals.
Figure 10-1. Biological neuron3
Thus, individual biological neurons seem to behave in a rather simple way, but they
are organized in a vast network of billions of neurons, each neuron typically connec‐
ted to thousands of other neurons. Highly complex computations can be performed
by a vast network of fairly simple neurons, much like a complex anthill can emerge
from the combined efforts of simple ants. The architecture of biological neural net‐
From Biological to Artificial Neurons
|
255
Download from finelybook www.finelybook.com


4 In the context of Machine Learning, the phrase “neural networks” generally refers to ANNs, not BNNs.
5 Drawing of a cortical lamination by S. Ramon y Cajal (public domain). Reproduced from https://en.wikipe
dia.org/wiki/Cerebral_cortex.
works (BNN)4 is still the subject of active research, but some parts of the brain have
been mapped, and it seems that neurons are often organized in consecutive layers, as
shown in Figure 10-2.
Figure 10-2. Multiple layers in a biological neural network (human cortex)5
Logical Computations with Neurons
Warren McCulloch and Walter Pitts proposed a very simple model of the biological
neuron, which later became known as an artificial neuron: it has one or more binary
(on/off) inputs and one binary output. The artificial neuron simply activates its out‐
put when more than a certain number of its inputs are active. McCulloch and Pitts
showed that even with such a simplified model it is possible to build a network of


showed that even with such a simplified model it is possible to build a network of
artificial neurons that computes any logical proposition you want. For example, let’s
build a few ANNs that perform various logical computations (see Figure 10-3),
assuming that a neuron is activated when at least two of its inputs are active.
Figure 10-3. ANNs performing simple logical computations
256
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


• The first network on the left is simply the identity function: if neuron A is activa‐
ted, then neuron C gets activated as well (since it receives two input signals from
neuron A), but if neuron A is off, then neuron C is off as well.
• The second network performs a logical AND: neuron C is activated only when
both neurons A and B are activated (a single input signal is not enough to acti‐
vate neuron C).
• The third network performs a logical OR: neuron C gets activated if either neu‐
ron A or neuron B is activated (or both).
• Finally, if we suppose that an input connection can inhibit the neuron’s activity
(which is the case with biological neurons), then the fourth network computes a
slightly more complex logical proposition: neuron C is activated only if neuron A
is active and if neuron B is off. If neuron A is active all the time, then you get a
logical NOT: neuron C is active when neuron B is off, and vice versa.


logical NOT: neuron C is active when neuron B is off, and vice versa.
You can easily imagine how these networks can be combined to compute complex
logical expressions (see the exercises at the end of the chapter).
The Perceptron
The Perceptron is one of the simplest ANN architectures, invented in 1957 by Frank
Rosenblatt. It is based on a slightly different artificial neuron (see Figure 10-4) called
a linear threshold unit (LTU): the inputs and output are now numbers (instead of
binary on/off values) and each input connection is associated with a weight. The LTU
computes a weighted sum of its inputs (z = w1 x1 + w2 x2 + ⋯ + wn xn = wT · x), then
applies a step function to that sum and outputs the result: hw(x) = step (z) = step (wT ·
x).
Figure 10-4. Linear threshold unit
The most common step function used in Perceptrons is the Heaviside step function
(see Equation 10-1). Sometimes the sign function is used instead.
From Biological to Artificial Neurons
|
257


From Biological to Artificial Neurons
|
257
Download from finelybook www.finelybook.com


6 The name Perceptron is sometimes used to mean a tiny network with a single LTU.
Equation 10-1. Common step functions used in Perceptrons
heaviside z = 0 if z < 0
1 if z ≥0
sgn z =
−1 if z < 0
0
if z = 0
+1 if z > 0
A single LTU can be used for simple linear binary classification. It computes a linear
combination of the inputs and if the result exceeds a threshold, it outputs the positive
class or else outputs the negative class (just like a Logistic Regression classifier or a
linear SVM). For example, you could use a single LTU to classify iris flowers based on
the petal length and width (also adding an extra bias feature x0 = 1, just like we did in
previous chapters). Training an LTU means finding the right values for w0, w1, and w2
(the training algorithm is discussed shortly).
A Perceptron is simply composed of a single layer of LTUs,6 with each neuron con‐
nected to all the inputs. These connections are often represented using special pass‐


nected to all the inputs. These connections are often represented using special pass‐
through neurons called input neurons: they just output whatever input they are fed.
Moreover, an extra bias feature is generally added (x0 = 1). This bias feature is typi‐
cally represented using a special type of neuron called a bias neuron, which just out‐
puts 1 all the time.
A Perceptron with two inputs and three outputs is represented in Figure 10-5. This
Perceptron can classify instances simultaneously into three different binary classes,
which makes it a multioutput classifier.
Figure 10-5. Perceptron diagram
So how is a Perceptron trained? The Perceptron training algorithm proposed by
Frank Rosenblatt was largely inspired by Hebb’s rule. In his book The Organization of
Behavior, published in 1949, Donald Hebb suggested that when a biological neuron
258
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


7 Note that this solution is generally not unique: in general when the data are linearly separable, there is an
infinity of hyperplanes that can separate them.
often triggers another neuron, the connection between these two neurons grows
stronger. This idea was later summarized by Siegrid Löwel in this catchy phrase:
“Cells that fire together, wire together.” This rule later became known as Hebb’s rule
(or Hebbian learning); that is, the connection weight between two neurons is
increased whenever they have the same output. Perceptrons are trained using a var‐
iant of this rule that takes into account the error made by the network; it does not
reinforce connections that lead to the wrong output. More specifically, the Perceptron
is fed one training instance at a time, and for each instance it makes its predictions.
For every output neuron that produced a wrong prediction, it reinforces the connec‐
tion weights from the inputs that would have contributed to the correct prediction.


tion weights from the inputs that would have contributed to the correct prediction.
The rule is shown in Equation 10-2.
Equation 10-2. Perceptron learning rule (weight update)
wi, j
next step = wi, j + η y j −yj xi
• wi, j is the connection weight between the ith input neuron and the jth output neu‐
ron.
• xi is the ith input value of the current training instance.
• yj is the output of the jth output neuron for the current training instance.
• yj is the target output of the jth output neuron for the current training instance.
• η is the learning rate.
The decision boundary of each output neuron is linear, so Perceptrons are incapable
of learning complex patterns (just like Logistic Regression classifiers). However, if the
training instances are linearly separable, Rosenblatt demonstrated that this algorithm
would converge to a solution.7 This is called the Perceptron convergence theorem.
Scikit-Learn provides a Perceptron class that implements a single LTU network. It


Scikit-Learn provides a Perceptron class that implements a single LTU network. It
can be used pretty much as you would expect—for example, on the iris dataset (intro‐
duced in Chapter 4):
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import Perceptron
iris = load_iris()
X = iris.data[:, (2, 3)]  # petal length, petal width
y = (iris.target == 0).astype(np.int)  # Iris Setosa?
From Biological to Artificial Neurons
|
259
Download from finelybook www.finelybook.com


per_clf = Perceptron(random_state=42)
per_clf.fit(X, y)
y_pred = per_clf.predict([[2, 0.5]])
You may have recognized that the Perceptron learning algorithm strongly resembles
Stochastic Gradient Descent. In fact, Scikit-Learn’s Perceptron class is equivalent to
using an SGDClassifier with the following hyperparameters: loss="perceptron",
learning_rate="constant", eta0=1 (the learning rate), and penalty=None (no regu‐
larization).
Note that contrary to Logistic Regression classifiers, Perceptrons do not output a class
probability; rather, they just make predictions based on a hard threshold. This is one
of the good reasons to prefer Logistic Regression over Perceptrons.
In their 1969 monograph titled Perceptrons, Marvin Minsky and Seymour Papert
highlighted a number of serious weaknesses of Perceptrons, in particular the fact that
they are incapable of solving some trivial problems (e.g., the Exclusive OR (XOR)


they are incapable of solving some trivial problems (e.g., the Exclusive OR (XOR)
classification problem; see the left side of Figure 10-6). Of course this is true of any
other linear classification model as well (such as Logistic Regression classifiers), but
researchers had expected much more from Perceptrons, and their disappointment
was great: as a result, many researchers dropped connectionism altogether (i.e., the
study of neural networks) in favor of higher-level problems such as logic, problem
solving, and search.
However, it turns out that some of the limitations of Perceptrons can be eliminated by
stacking multiple Perceptrons. The resulting ANN is called a Multi-Layer Perceptron
(MLP). In particular, an MLP can solve the XOR problem, as you can verify by com‐
puting the output of the MLP represented on the right of Figure 10-6, for each com‐
bination of inputs: with inputs (0, 0) or (1, 1) the network outputs 0, and with inputs
(0, 1) or (1, 0) it outputs 1.


(0, 1) or (1, 0) it outputs 1.
Figure 10-6. XOR classification problem and an MLP that solves it
260
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


8 “Learning Internal Representations by Error Propagation,” D. Rumelhart, G. Hinton, R. Williams (1986).
9 This algorithm was actually invented several times by various researchers in different fields, starting with
P. Werbos in 1974.
Multi-Layer Perceptron and Backpropagation
An MLP is composed of one (passthrough) input layer, one or more layers of LTUs,
called hidden layers, and one final layer of LTUs called the output layer (see
Figure 10-7). Every layer except the output layer includes a bias neuron and is fully
connected to the next layer. When an ANN has two or more hidden layers, it is called
a deep neural network (DNN).
Figure 10-7. Multi-Layer Perceptron
For many years researchers struggled to find a way to train MLPs, without success.
But in 1986, D. E. Rumelhart et al. published a groundbreaking article8 introducing
the backpropagation training algorithm.9 Today we would describe it as Gradient


the backpropagation training algorithm.9 Today we would describe it as Gradient
Descent using reverse-mode autodiff (Gradient Descent was introduced in Chapter 4,
and autodiff was discussed in Chapter 9).
For each training instance, the algorithm feeds it to the network and computes the
output of every neuron in each consecutive layer (this is the forward pass, just like
when making predictions). Then it measures the network’s output error (i.e., the dif‐
ference between the desired output and the actual output of the network), and it
computes how much each neuron in the last hidden layer contributed to each output
neuron’s error. It then proceeds to measure how much of these error contributions
came from each neuron in the previous hidden layer—and so on until the algorithm
reaches the input layer. This reverse pass efficiently measures the error gradient
across all the connection weights in the network by propagating the error gradient


across all the connection weights in the network by propagating the error gradient
backward in the network (hence the name of the algorithm). If you check out the
From Biological to Artificial Neurons
|
261
Download from finelybook www.finelybook.com


reverse-mode autodiff algorithm in Appendix D, you will find that the forward and
reverse passes of backpropagation simply perform reverse-mode autodiff. The last
step of the backpropagation algorithm is a Gradient Descent step on all the connec‐
tion weights in the network, using the error gradients measured earlier.
Let’s make this even shorter: for each training instance the backpropagation algo‐
rithm first makes a prediction (forward pass), measures the error, then goes through
each layer in reverse to measure the error contribution from each connection (reverse
pass), and finally slightly tweaks the connection weights to reduce the error (Gradient
Descent step).
In order for this algorithm to work properly, the authors made a key change to the
MLP’s architecture: they replaced the step function with the logistic function, σ(z) =
1 / (1 + exp(–z)). This was essential because the step function contains only flat seg‐


1 / (1 + exp(–z)). This was essential because the step function contains only flat seg‐
ments, so there is no gradient to work with (Gradient Descent cannot move on a flat
surface), while the logistic function has a well-defined nonzero derivative every‐
where, allowing Gradient Descent to make some progress at every step. The backpro‐
pagation algorithm may be used with other activation functions, instead of the logistic
function. Two other popular activation functions are:
The hyperbolic tangent function tanh (z) = 2σ(2z) – 1
Just like the logistic function it is S-shaped, continuous, and differentiable, but its
output value ranges from –1 to 1 (instead of 0 to 1 in the case of the logistic func‐
tion), which tends to make each layer’s output more or less normalized (i.e., cen‐
tered around 0) at the beginning of training. This often helps speed up
convergence.
The ReLU function (introduced in Chapter 9)


convergence.
The ReLU function (introduced in Chapter 9)
ReLU (z) = max (0, z). It is continuous but unfortunately not differentiable at z =
0 (the slope changes abruptly, which can make Gradient Descent bounce
around). However, in practice it works very well and has the advantage of being
fast to compute. Most importantly, the fact that it does not have a maximum out‐
put value also helps reduce some issues during Gradient Descent (we will come
back to this in Chapter 11).
These popular activation functions and their derivatives are represented in
Figure 10-8.
262
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


Figure 10-8. Activation functions and their derivatives
An MLP is often used for classification, with each output corresponding to a different
binary class (e.g., spam/ham, urgent/not-urgent, and so on). When the classes are
exclusive (e.g., classes 0 through 9 for digit image classification), the output layer is
typically modified by replacing the individual activation functions by a shared soft‐
max function (see Figure 10-9). The softmax function was introduced in Chapter 3.
The output of each neuron corresponds to the estimated probability of the corre‐
sponding class. Note that the signal flows only in one direction (from the inputs to
the outputs), so this architecture is an example of a feedforward neural network
(FNN).
Figure 10-9. A modern MLP (including ReLU and softmax) for classification
From Biological to Artificial Neurons
|
263
Download from finelybook www.finelybook.com


Biological neurons seem to implement a roughly sigmoid (S-
shaped) activation function, so researchers stuck to sigmoid func‐
tions for a very long time. But it turns out that the ReLU activation
function generally works better in ANNs. This is one of the cases
where the biological analogy was misleading.
Training an MLP with TensorFlow’s High-Level API
The simplest way to train an MLP with TensorFlow is to use the high-level API
TF.Learn, which is quite similar to Scikit-Learn’s API. The DNNClassifier class
makes it trivial to train a deep neural network with any number of hidden layers, and
a softmax output layer to output estimated class probabilities. For example, the fol‐
lowing code trains a DNN for classification with two hidden layers (one with 300
neurons, and the other with 100 neurons) and a softmax output layer with 10
neurons:
import tensorflow as tf
feature_columns = tf.contrib.learn.infer_real_valued_columns_from_input(X_train)


feature_columns = tf.contrib.learn.infer_real_valued_columns_from_input(X_train)
dnn_clf = tf.contrib.learn.DNNClassifier(hidden_units=[300, 100], n_classes=10,
                                         feature_columns=feature_columns)
dnn_clf.fit(x=X_train, y=y_train, batch_size=50, steps=40000)
If you run this code on the MNIST dataset (after scaling it, e.g., by using Scikit-
Learn’s StandardScaler), you may actually get a model that achieves over 98.1%
accuracy on the test set! That’s better than the best model we trained in Chapter 3:
>>> from sklearn.metrics import accuracy_score
>>> y_pred = list(dnn_clf.predict(X_test))
>>> accuracy_score(y_test, y_pred)
0.98180000000000001
The TF.Learn library also provides some convenience functions to evaluate models:
>>> dnn_clf.evaluate(X_test, y_test)
{'accuracy': 0.98180002, 'global_step': 40000, 'loss': 0.073678359}
Under the hood, the DNNClassifier class creates all the neuron layers, based on the


Under the hood, the DNNClassifier class creates all the neuron layers, based on the
ReLU activation function (we can change this by setting the activation_fn hyper‐
parameter). The output layer relies on the softmax function, and the cost function is
cross entropy (introduced in Chapter 4).
The TF.Learn API is still quite new, so some of the names and func‐
tions used in these examples may evolve a bit by the time you read
this book. However, the general ideas should not change.
264
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


Training a DNN Using Plain TensorFlow
If you want more control over the architecture of the network, you may prefer to use
TensorFlow’s lower-level Python API (introduced in Chapter 9). In this section we
will build the same model as before using this API, and we will implement Mini-
batch Gradient Descent to train it on the MNIST dataset. The first step is the con‐
struction phase, building the TensorFlow graph. The second step is the execution
phase, where you actually run the graph to train the model.
Construction Phase
Let’s start. First we need to import the tensorflow library. Then we must specify the
number of inputs and outputs, and set the number of hidden neurons in each layer:
import tensorflow as tf
n_inputs = 28*28  # MNIST
n_hidden1 = 300
n_hidden2 = 100
n_outputs = 10
Next, just like you did in Chapter 9, you can use placeholder nodes to represent the
training data and targets. The shape of X is only partially defined. We know that it will


training data and targets. The shape of X is only partially defined. We know that it will
be a 2D tensor (i.e., a matrix), with instances along the first dimension and features
along the second dimension, and we know that the number of features is going to be
28 x 28 (one feature per pixel), but we don’t know yet how many instances each train‐
ing batch will contain. So the shape of X is (None, n_inputs). Similarly, we know
that y will be a 1D tensor with one entry per instance, but again we don’t know the
size of the training batch at this point, so the shape is (None).
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int64, shape=(None), name="y")
Now let’s create the actual neural network. The placeholder X will act as the input
layer; during the execution phase, it will be replaced with one training batch at a time
(note that all the instances in a training batch will be processed simultaneously by the


(note that all the instances in a training batch will be processed simultaneously by the
neural network). Now you need to create the two hidden layers and the output layer.
The two hidden layers are almost identical: they differ only by the inputs they are
connected to and by the number of neurons they contain. The output layer is also
very similar, but it uses a softmax activation function instead of a ReLU activation
function. So let’s create a neuron_layer() function that we will use to create one layer
at a time. It will need parameters to specify the inputs, the number of neurons, the
activation function, and the name of the layer:
def neuron_layer(X, n_neurons, name, activation=None):
    with tf.name_scope(name):
        n_inputs = int(X.get_shape()[1])
Training a DNN Using Plain TensorFlow
|
265
Download from finelybook www.finelybook.com


10 Using a truncated normal distribution rather than a regular normal distribution ensures that there won’t be
any large weights, which could slow down training.
11 For example, if you set all the weights to 0, then all neurons will output 0, and the error gradient will be the
same for all neurons in a given hidden layer. The Gradient Descent step will then update all the weights in
exactly the same way in each layer, so they will all remain equal. In other words, despite having hundreds of
neurons per layer, your model will act as if there were only one neuron per layer. It is not going to fly.
        stddev = 2 / np.sqrt(n_inputs)
        init = tf.truncated_normal((n_inputs, n_neurons), stddev=stddev)
        W = tf.Variable(init, name="weights")
        b = tf.Variable(tf.zeros([n_neurons]), name="biases")
        z = tf.matmul(X, W) + b
        if activation=="relu":
            return tf.nn.relu(z)
        else:
            return z
Let’s go through this code line by line:


else:
            return z
Let’s go through this code line by line:
1. First we create a name scope using the name of the layer: it will contain all the
computation nodes for this neuron layer. This is optional, but the graph will look
much nicer in TensorBoard if its nodes are well organized.
2. Next, we get the number of inputs by looking up the input matrix’s shape and
getting the size of the second dimension (the first dimension is for instances).
3. The next three lines create a W variable that will hold the weights matrix. It will be
a 2D tensor containing all the connection weights between each input and each
neuron; hence, its shape will be (n_inputs, n_neurons). It will be initialized
randomly, using a truncated10 normal (Gaussian) distribution with a standard
deviation of 2/ ninputs. Using this specific standard deviation helps the algorithm
converge much faster (we will discuss this further in Chapter 11; it is one of those


converge much faster (we will discuss this further in Chapter 11; it is one of those
small tweaks to neural networks that have had a tremendous impact on their effi‐
ciency). It is important to initialize connection weights randomly for all hidden
layers to avoid any symmetries that the Gradient Descent algorithm would be
unable to break.11
4. The next line creates a b variable for biases, initialized to 0 (no symmetry issue in
this case), with one bias parameter per neuron.
5. Then we create a subgraph to compute z = X · W + b. This vectorized implemen‐
tation will efficiently compute the weighted sums of the inputs plus the bias term
for each and every neuron in the layer, for all the instances in the batch in just
one shot.
6. Finally, if the activation parameter is set to "relu", the code returns relu(z)
(i.e., max (0, z)), or else it just returns z.
266
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


Okay, so now you have a nice function to create a neuron layer. Let’s use it to create
the deep neural network! The first hidden layer takes X as its input. The second takes
the output of the first hidden layer as its input. And finally, the output layer takes the
output of the second hidden layer as its input.
with tf.name_scope("dnn"):
    hidden1 = neuron_layer(X, n_hidden1, "hidden1", activation="relu")
    hidden2 = neuron_layer(hidden1, n_hidden2, "hidden2", activation="relu")
    logits = neuron_layer(hidden2, n_outputs, "outputs")
Notice that once again we used a name scope for clarity. Also note that logits is the
output of the neural network before going through the softmax activation function:
for optimization reasons, we will handle the softmax computation later.
As you might expect, TensorFlow comes with many handy functions to create
standard neural network layers, so there’s often no need to define your own


standard neural network layers, so there’s often no need to define your own
neuron_layer() function like we just did. For example, TensorFlow’s fully_connec
ted() function creates a fully connected layer, where all the inputs are connected to
all the neurons in the layer. It takes care of creating the weights and biases variables,
with the proper initialization strategy, and it uses the ReLU activation function by
default (we can change this using the activation_fn argument). As we will see in
Chapter 11, it also supports regularization and normalization parameters. Let’s tweak
the preceding code to use the fully_connected() function instead of our neu
ron_layer() function. Simply import the function and replace the dnn construction
section with the following code:
from tensorflow.contrib.layers import fully_connected
with tf.name_scope("dnn"):
    hidden1 = fully_connected(X, n_hidden1, scope="hidden1")
    hidden2 = fully_connected(hidden1, n_hidden2, scope="hidden2")


hidden2 = fully_connected(hidden1, n_hidden2, scope="hidden2")
    logits = fully_connected(hidden2, n_outputs, scope="outputs",
                             activation_fn=None)
The tensorflow.contrib package contains many useful functions,
but it is a place for experimental code that has not yet graduated to
be part of the main TensorFlow API. So the fully_connected()
function (and any other contrib code) may change or move in the
future.
Now that we have the neural network model ready to go, we need to define the cost
function that we will use to train it. Just as we did for Softmax Regression in Chap‐
ter 4, we will use cross entropy. As we discussed earlier, cross entropy will penalize
models that estimate a low probability for the target class. TensorFlow provides
several functions to compute cross entropy. We will use sparse_soft
max_cross_entropy_with_logits(): it computes the cross entropy based on the
Training a DNN Using Plain TensorFlow
|
267


Training a DNN Using Plain TensorFlow
|
267
Download from finelybook www.finelybook.com


“logits” (i.e., the output of the network before going through the softmax activation
function), and it expects labels in the form of integers ranging from 0 to the number
of classes minus 1 (in our case, from 0 to 9). This will give us a 1D tensor containing
the cross entropy for each instance. We can then use TensorFlow’s reduce_mean()
function to compute the mean cross entropy over all instances.
with tf.name_scope("loss"):
    xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(
                   labels=y, logits=logits)
    loss = tf.reduce_mean(xentropy, name="loss")
The sparse_softmax_cross_entropy_with_logits() function is
equivalent to applying the softmax activation function and then
computing the cross entropy, but it is more efficient, and it prop‐
erly takes care of corner cases like logits equal to 0. This is why we
did not apply the softmax activation function earlier. There is also
another function called softmax_cross_entropy_with_logits(),


another function called softmax_cross_entropy_with_logits(),
which takes labels in the form of one-hot vectors (instead of ints
from 0 to the number of classes minus 1).
We have the neural network model, we have the cost function, and now we need to
define a GradientDescentOptimizer that will tweak the model parameters to mini‐
mize the cost function. Nothing new; it’s just like we did in Chapter 9:
learning_rate = 0.01
with tf.name_scope("train"):
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    training_op = optimizer.minimize(loss)
The last important step in the construction phase is to specify how to evaluate the
model. We will simply use accuracy as our performance measure. First, for each
instance, determine if the neural network’s prediction is correct by checking whether
or not the highest logit corresponds to the target class. For this you can use the
in_top_k() function. This returns a 1D tensor full of boolean values, so we need to


in_top_k() function. This returns a 1D tensor full of boolean values, so we need to
cast these booleans to floats and then compute the average. This will give us the net‐
work’s overall accuracy.
with tf.name_scope("eval"):
    correct = tf.nn.in_top_k(logits, y, 1)
    accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
And, as usual, we need to create a node to initialize all variables, and we will also cre‐
ate a Saver to save our trained model parameters to disk:
init = tf.global_variables_initializer()
saver = tf.train.Saver()
268
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


Phew! This concludes the construction phase. This was fewer than 40 lines of code,
but it was pretty intense: we created placeholders for the inputs and the targets, we
created a function to build a neuron layer, we used it to create the DNN, we defined
the cost function, we created an optimizer, and finally we defined the performance
measure. Now on to the execution phase.
Execution Phase
This part is much shorter and simpler. First, let’s load MNIST. We could use Scikit-
Learn for that as we did in previous chapters, but TensorFlow offers its own helper
that fetches the data, scales it (between 0 and 1), shuffles it, and provides a simple
function to load one mini-batches a time. So let’s use it instead:
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/")
Now we define the number of epochs that we want to run, as well as the size of the
mini-batches:
n_epochs = 400
batch_size = 50
And now we can train the model:


mini-batches:
n_epochs = 400
batch_size = 50
And now we can train the model:
with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_test = accuracy.eval(feed_dict={X: mnist.test.images,
                                            y: mnist.test.labels})
        print(epoch, "Train accuracy:", acc_train, "Test accuracy:", acc_test)
    save_path = saver.save(sess, "./my_model_final.ckpt")
This code opens a TensorFlow session, and it runs the init node that initializes all
the variables. Then it runs the main training loop: at each epoch, the code iterates
through a number of mini-batches that corresponds to the training set size. Each


through a number of mini-batches that corresponds to the training set size. Each
mini-batch is fetched via the next_batch() method, and then the code simply runs
the training operation, feeding it the current mini-batch input data and targets. Next,
at the end of each epoch, the code evaluates the model on the last mini-batch and on
the full training set, and it prints out the result. Finally, the model parameters are
saved to disk.
Training a DNN Using Plain TensorFlow
|
269
Download from finelybook www.finelybook.com


Using the Neural Network
Now that the neural network is trained, you can use it to make predictions. To do
that, you can reuse the same construction phase, but change the execution phase like
this:
with tf.Session() as sess:
    saver.restore(sess, "./my_model_final.ckpt")
    X_new_scaled = [...]  # some new images (scaled from 0 to 1)
    Z = logits.eval(feed_dict={X: X_new_scaled})
    y_pred = np.argmax(Z, axis=1)
First the code loads the model parameters from disk. Then it loads some new images
that you want to classify. Remember to apply the same feature scaling as for the train‐
ing data (in this case, scale it from 0 to 1). Then the code evaluates the logits node.
If you wanted to know all the estimated class probabilities, you would need to apply
the softmax() function to the logits, but if you just want to predict a class, you can
simply pick the class that has the highest logit value (using the argmax() function
does the trick).
Fine-Tuning Neural Network Hyperparameters


does the trick).
Fine-Tuning Neural Network Hyperparameters
The flexibility of neural networks is also one of their main drawbacks: there are many
hyperparameters to tweak. Not only can you use any imaginable network topology
(how neurons are interconnected), but even in a simple MLP you can change the
number of layers, the number of neurons per layer, the type of activation function to
use in each layer, the weight initialization logic, and much more. How do you know
what combination of hyperparameters is the best for your task?
Of course, you can use grid search with cross-validation to find the right hyperpara‐
meters, like you did in previous chapters, but since there are many hyperparameters
to tune, and since training a neural network on a large dataset takes a lot of time, you
will only be able to explore a tiny part of the hyperparameter space in a reasonable
amount of time. It is much better to use randomized search, as we discussed in Chap‐


amount of time. It is much better to use randomized search, as we discussed in Chap‐
ter 2. Another option is to use a tool such as Oscar, which implements more complex
algorithms to help you find a good set of hyperparameters quickly.
It helps to have an idea of what values are reasonable for each hyperparameter, so you
can restrict the search space. Let’s start with the number of hidden layers.
Number of Hidden Layers
For many problems, you can just begin with a single hidden layer and you will get
reasonable results. It has actually been shown that an MLP with just one hidden layer
can model even the most complex functions provided it has enough neurons. For a
long time, these facts convinced researchers that there was no need to investigate any
270
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


deeper neural networks. But they overlooked the fact that deep networks have a much
higher parameter efficiency than shallow ones: they can model complex functions
using exponentially fewer neurons than shallow nets, making them much faster to
train.
To understand why, suppose you are asked to draw a forest using some drawing soft‐
ware, but you are forbidden to use copy/paste. You would have to draw each tree
individually, branch per branch, leaf per leaf. If you could instead draw one leaf,
copy/paste it to draw a branch, then copy/paste that branch to create a tree, and
finally copy/paste this tree to make a forest, you would be finished in no time. Real-
world data is often structured in such a hierarchical way and DNNs automatically
take advantage of this fact: lower hidden layers model low-level structures (e.g., line
segments of various shapes and orientations), intermediate hidden layers combine


segments of various shapes and orientations), intermediate hidden layers combine
these low-level structures to model intermediate-level structures (e.g., squares, cir‐
cles), and the highest hidden layers and the output layer combine these intermediate
structures to model high-level structures (e.g., faces).
Not only does this hierarchical architecture help DNNs converge faster to a good sol‐
ution, it also improves their ability to generalize to new datasets. For example, if you
have already trained a model to recognize faces in pictures, and you now want to
train a new neural network to recognize hairstyles, then you can kickstart training by
reusing the lower layers of the first network. Instead of randomly initializing the
weights and biases of the first few layers of the new neural network, you can initialize
them to the value of the weights and biases of the lower layers of the first network.
This way the network will not have to learn from scratch all the low-level structures


This way the network will not have to learn from scratch all the low-level structures
that occur in most pictures; it will only have to learn the higher-level structures (e.g.,
hairstyles).
In summary, for many problems you can start with just one or two hidden layers and
it will work just fine (e.g., you can easily reach above 97% accuracy on the MNIST
dataset using just one hidden layer with a few hundred neurons, and above 98% accu‐
racy using two hidden layers with the same total amount of neurons, in roughly the
same amount of training time). For more complex problems, you can gradually ramp
up the number of hidden layers, until you start overfitting the training set. Very com‐
plex tasks, such as large image classification or speech recognition, typically require
networks with dozens of layers (or even hundreds, but not fully connected ones, as
we will see in Chapter 13), and they need a huge amount of training data. However,


we will see in Chapter 13), and they need a huge amount of training data. However,
you will rarely have to train such networks from scratch: it is much more common to
reuse parts of a pretrained state-of-the-art network that performs a similar task.
Training will be a lot faster and require much less data (we will discuss this in Chap‐
ter 11).
Fine-Tuning Neural Network Hyperparameters
|
271
Download from finelybook www.finelybook.com


12 By Vincent Vanhoucke in his Deep Learning class on Udacity.com.
13 A few extra ANN architectures are presented in Appendix E.
Number of Neurons per Hidden Layer
Obviously the number of neurons in the input and output layers is determined by the
type of input and output your task requires. For example, the MNIST task requires 28
x 28 = 784 input neurons and 10 output neurons. As for the hidden layers, a common
practice is to size them to form a funnel, with fewer and fewer neurons at each layer—
the rationale being that many low-level features can coalesce into far fewer high-level
features. For example, a typical neural network for MNIST may have two hidden lay‐
ers, the first with 300 neurons and the second with 100. However, this practice is not
as common now, and you may simply use the same size for all hidden layers—for
example, all hidden layers with 150 neurons: that’s just one hyperparameter to tune


example, all hidden layers with 150 neurons: that’s just one hyperparameter to tune
instead of one per layer. Just like for the number of layers, you can try increasing the
number of neurons gradually until the network starts overfitting. In general you will
get more bang for the buck by increasing the number of layers than the number of
neurons per layer. Unfortunately, as you can see, finding the perfect amount of neu‐
rons is still somewhat of a black art.
A simpler approach is to pick a model with more layers and neurons than you
actually need, then use early stopping to prevent it from overfitting (and other regu‐
larization techniques, especially dropout, as we will see in Chapter 11). This has been
dubbed the “stretch pants” approach:12 instead of wasting time looking for pants that
perfectly match your size, just use large stretch pants that will shrink down to the
right size.
Activation Functions


right size.
Activation Functions
In most cases you can use the ReLU activation function in the hidden layers (or one
of its variants, as we will see in Chapter 11). It is a bit faster to compute than other
activation functions, and Gradient Descent does not get stuck as much on plateaus,
thanks to the fact that it does not saturate for large input values (as opposed to the
logistic function or the hyperbolic tangent function, which saturate at 1).
For the output layer, the softmax activation function is generally a good choice for
classification tasks (when the classes are mutually exclusive). For regression tasks,
you can simply use no activation function at all.
This concludes this introduction to artificial neural networks. In the following chap‐
ters, we will discuss techniques to train very deep nets, and distribute training across
multiple servers and GPUs. Then we will explore a few other popular neural network


multiple servers and GPUs. Then we will explore a few other popular neural network
architectures: convolutional neural networks, recurrent neural networks, and autoen‐
coders.13
272
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


Exercises
1. Draw an ANN using the original artificial neurons (like the ones in Figure 10-3)
that computes A ⊕ B (where ⊕ represents the XOR operation). Hint: A ⊕ B = (A
∧ ¬ B) ∨ (¬ A ∧ B).
2. Why is it generally preferable to use a Logistic Regression classifier rather than a
classical Perceptron (i.e., a single layer of linear threshold units trained using the
Perceptron training algorithm)? How can you tweak a Perceptron to make it
equivalent to a Logistic Regression classifier?
3. Why was the logistic activation function a key ingredient in training the first
MLPs?
4. Name three popular activation functions. Can you draw them?
5. Suppose you have an MLP composed of one input layer with 10 passthrough
neurons, followed by one hidden layer with 50 artificial neurons, and finally one
output layer with 3 artificial neurons. All artificial neurons use the ReLU activa‐
tion function.
• What is the shape of the input matrix X?


tion function.
• What is the shape of the input matrix X?
• What about the shape of the hidden layer’s weight vector Wh, and the shape of
its bias vector bh?
• What is the shape of the output layer’s weight vector Wo, and its bias vector bo?
• What is the shape of the network’s output matrix Y?
• Write the equation that computes the network’s output matrix Y as a function
of X, Wh, bh, Wo and bo.
6. How many neurons do you need in the output layer if you want to classify email
into spam or ham? What activation function should you use in the output layer?
If instead you want to tackle MNIST, how many neurons do you need in the out‐
put layer, using what activation function? Answer the same questions for getting
your network to predict housing prices as in Chapter 2.
7. What is backpropagation and how does it work? What is the difference between
backpropagation and reverse-mode autodiff?
8. Can you list all the hyperparameters you can tweak in an MLP? If the MLP over‐


8. Can you list all the hyperparameters you can tweak in an MLP? If the MLP over‐
fits the training data, how could you tweak these hyperparameters to try to solve
the problem?
9. Train a deep MLP on the MNIST dataset and see if you can get over 98% preci‐
sion. Just like in the last exercise of Chapter 9, try adding all the bells and whistles
Exercises
|
273
Download from finelybook www.finelybook.com


(i.e., save checkpoints, restore the last checkpoint in case of an interruption, add
summaries, plot learning curves using TensorBoard, and so on).
Solutions to these exercises are available in Appendix A.
274
|
Chapter 10: Introduction to Artificial Neural Networks
Download from finelybook www.finelybook.com


CHAPTER 11
Training Deep Neural Nets
In Chapter 10 we introduced artificial neural networks and trained our first deep
neural network. But it was a very shallow DNN, with only two hidden layers. What if
you need to tackle a very complex problem, such as detecting hundreds of types of
objects in high-resolution images? You may need to train a much deeper DNN, per‐
haps with (say) 10 layers, each containing hundreds of neurons, connected by hun‐
dreds of thousands of connections. This would not be a walk in the park:
• First, you would be faced with the tricky vanishing gradients problem (or the
related exploding gradients problem) that affects deep neural networks and makes
lower layers very hard to train.
• Second, with such a large network, training would be extremely slow.
• Third, a model with millions of parameters would severely risk overfitting the
training set.
In this chapter, we will go through each of these problems in turn and present techni‐


In this chapter, we will go through each of these problems in turn and present techni‐
ques to solve them. We will start by explaining the vanishing gradients problem and
exploring some of the most popular solutions to this problem. Next we will look at
various optimizers that can speed up training large models tremendously compared
to plain Gradient Descent. Finally, we will go through a few popular regularization
techniques for large neural networks.
With these tools, you will be able to train very deep nets: welcome to Deep Learning!
Vanishing/Exploding Gradients Problems
As we discussed in Chapter 10, the backpropagation algorithm works by going from
the output layer to the input layer, propagating the error gradient on the way. Once
the algorithm has computed the gradient of the cost function with regards to each
275
Download from finelybook www.finelybook.com


1 “Understanding the Difficulty of Training Deep Feedforward Neural Networks,” X. Glorot, Y Bengio (2010).
parameter in the network, it uses these gradients to update each parameter with a
Gradient Descent step.
Unfortunately, gradients often get smaller and smaller as the algorithm progresses
down to the lower layers. As a result, the Gradient Descent update leaves the lower
layer connection weights virtually unchanged, and training never converges to a good
solution. This is called the vanishing gradients problem. In some cases, the opposite
can happen: the gradients can grow bigger and bigger, so many layers get insanely
large weight updates and the algorithm diverges. This is the exploding gradients prob‐
lem, which is mostly encountered in recurrent neural networks (see Chapter 14).
More generally, deep neural networks suffer from unstable gradients; different layers
may learn at widely different speeds.


may learn at widely different speeds.
Although this unfortunate behavior has been empirically observed for quite a while
(it was one of the reasons why deep neural networks were mostly abandoned for a
long time), it is only around 2010 that significant progress was made in understand‐
ing it. A paper titled “Understanding the Difficulty of Training Deep Feedforward
Neural Networks” by Xavier Glorot and Yoshua Bengio1 found a few suspects, includ‐
ing the combination of the popular logistic sigmoid activation function and the
weight initialization technique that was most popular at the time, namely random ini‐
tialization using a normal distribution with a mean of 0 and a standard deviation of 1.
In short, they showed that with this activation function and this initialization scheme,
the variance of the outputs of each layer is much greater than the variance of its
inputs. Going forward in the network, the variance keeps increasing after each layer


inputs. Going forward in the network, the variance keeps increasing after each layer
until the activation function saturates at the top layers. This is actually made worse by
the fact that the logistic function has a mean of 0.5, not 0 (the hyperbolic tangent
function has a mean of 0 and behaves slightly better than the logistic function in deep
networks).
Looking at the logistic activation function (see Figure 11-1), you can see that when
inputs become large (negative or positive), the function saturates at 0 or 1, with a
derivative extremely close to 0. Thus when backpropagation kicks in, it has virtually
no gradient to propagate back through the network, and what little gradient exists
keeps getting diluted as backpropagation progresses down through the top layers, so
there is really nothing left for the lower layers.
276
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


2 Here’s an analogy: if you set a microphone amplifier’s knob too close to zero, people won’t hear your voice, but
if you set it too close to the max, your voice will be saturated and people won’t understand what you are say‐
ing. Now imagine a chain of such amplifiers: they all need to be set properly in order for your voice to come
out loud and clear at the end of the chain. Your voice has to come out of each amplifier at the same amplitude
as it came in.
Figure 11-1. Logistic activation function saturation
Xavier and He Initialization
In their paper, Glorot and Bengio propose a way to significantly alleviate this prob‐
lem. We need the signal to flow properly in both directions: in the forward direction
when making predictions, and in the reverse direction when backpropagating gradi‐
ents. We don’t want the signal to die out, nor do we want it to explode and saturate.
For the signal to flow properly, the authors argue that we need the variance of the


For the signal to flow properly, the authors argue that we need the variance of the
outputs of each layer to be equal to the variance of its inputs,2 and we also need the
gradients to have equal variance before and after flowing through a layer in the
reverse direction (please check out the paper if you are interested in the mathematical
details). It is actually not possible to guarantee both unless the layer has an equal
number of input and output connections, but they proposed a good compromise that
has proven to work very well in practice: the connection weights must be initialized
randomly as described in Equation 11-1, where ninputs and noutputs are the number of
input and output connections for the layer whose weights are being initialized (also
called fan-in and fan-out). This initialization strategy is often called Xavier initializa‐
tion (after the author’s first name), or sometimes Glorot initialization.
Vanishing/Exploding Gradients Problems
|
277


Vanishing/Exploding Gradients Problems
|
277
Download from finelybook www.finelybook.com


3 This simplified strategy was actually already proposed much earlier—for example, in the 1998 book Neural
Networks: Tricks of the Trade by Genevieve Orr and Klaus-Robert Müller (Springer).
4 Such as “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” K.
He et al. (2015).
Equation 11-1. Xavier initialization (when using the logistic activation function)
Normal distribution with mean 0 and standard deviation σ =
2
ninputs + noutputs
Or a uniform distribution between ‐r and +r, with r =
6
ninputs + noutputs
When the number of input connections is roughly equal to the number of output
connections, you get simpler equations (e.g., σ = 1/ ninputs or r =
3/ ninputs). We
used this simplified strategy in Chapter 10.3
Using the Xavier initialization strategy can speed up training considerably, and it is
one of the tricks that led to the current success of Deep Learning. Some recent papers4


one of the tricks that led to the current success of Deep Learning. Some recent papers4
have provided similar strategies for different activation functions, as shown in
Table 11-1. The initialization strategy for the ReLU activation function (and its var‐
iants, including the ELU activation described shortly) is sometimes called He initiali‐
zation (after the last name of its author).
Table 11-1. Initialization parameters for each type of activation function
Activation function
Uniform distribution [–r, r] Normal distribution
Logistic
r =
6
ninputs + noutputs
σ =
2
ninputs + noutputs
Hyperbolic tangent
r = 4
6
ninputs + noutputs
σ = 4
2
ninputs + noutputs
ReLU (and its variants)
r =
2
6
ninputs + noutputs
σ =
2
2
ninputs + noutputs
By default, the fully_connected() function (introduced in Chapter 10) uses Xavier
initialization (with a uniform distribution). You can change this to He initialization
by using the variance_scaling_initializer() function like this:


by using the variance_scaling_initializer() function like this:
he_init = tf.contrib.layers.variance_scaling_initializer()
hidden1 = fully_connected(X, n_hidden1, weights_initializer=he_init, scope="h1")
278
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


5 “Empirical Evaluation of Rectified Activations in Convolution Network,” B. Xu et al. (2015).
He initialization considers only the fan-in, not the average between
fan-in and fan-out like in Xavier initialization. This is also the
default for the variance_scaling_initializer() function, but
you can change this by setting the argument mode="FAN_AVG".
Nonsaturating Activation Functions
One of the insights in the 2010 paper by Glorot and Bengio was that the vanishing/
exploding gradients problems were in part due to a poor choice of activation func‐
tion. Until then most people had assumed that if Mother Nature had chosen to use
roughly sigmoid activation functions in biological neurons, they must be an excellent
choice. But it turns out that other activation functions behave much better in deep
neural networks, in particular the ReLU activation function, mostly because it does
not saturate for positive values (and also because it is quite fast to compute).


not saturate for positive values (and also because it is quite fast to compute).
Unfortunately, the ReLU activation function is not perfect. It suffers from a problem
known as the dying ReLUs: during training, some neurons effectively die, meaning
they stop outputting anything other than 0. In some cases, you may find that half of
your network’s neurons are dead, especially if you used a large learning rate. During
training, if a neuron’s weights get updated such that the weighted sum of the neuron’s
inputs is negative, it will start outputting 0. When this happen, the neuron is unlikely
to come back to life since the gradient of the ReLU function is 0 when its input is
negative.
To solve this problem, you may want to use a variant of the ReLU function, such as
the leaky ReLU. This function is defined as LeakyReLUα(z) = max(αz, z) (see
Figure 11-2). The hyperparameter α defines how much the function “leaks”: it is the


Figure 11-2). The hyperparameter α defines how much the function “leaks”: it is the
slope of the function for z < 0, and is typically set to 0.01. This small slope ensures
that leaky ReLUs never die; they can go into a long coma, but they have a chance to
eventually wake up. A recent paper5 compared several variants of the ReLU activation
function and one of its conclusions was that the leaky variants always outperformed
the strict ReLU activation function. In fact, setting α = 0.2 (huge leak) seemed to
result in better performance than α = 0.01 (small leak). They also evaluated the
randomized leaky ReLU (RReLU), where α is picked randomly in a given range during
training, and it is fixed to an average value during testing. It also performed fairly well
and seemed to act as a regularizer (reducing the risk of overfitting the training set).
Finally, they also evaluated the parametric leaky ReLU (PReLU), where α is authorized


Finally, they also evaluated the parametric leaky ReLU (PReLU), where α is authorized
to be learned during training (instead of being a hyperparameter, it becomes a
parameter that can be modified by backpropagation like any other parameter). This
Vanishing/Exploding Gradients Problems
|
279
Download from finelybook www.finelybook.com


6 “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs),” D. Clevert, T. Unterthiner,
S. Hochreiter (2015).
was reported to strongly outperform ReLU on large image datasets, but on smaller
datasets it runs the risk of overfitting the training set.
Figure 11-2. Leaky ReLU
Last but not least, a 2015 paper by Djork-Arné Clevert et al.6 proposed a new activa‐
tion function called the exponential linear unit (ELU) that outperformed all the ReLU
variants in their experiments: training time was reduced and the neural network per‐
formed better on the test set. It is represented in Figure 11-3, and Equation 11-2
shows its definition.
Equation 11-2. ELU activation function
ELUα z = α exp z −1 if z < 0
z
if z ≥0
Figure 11-3. ELU activation function
280
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


It looks a lot like the ReLU function, with a few major differences:
• First it takes on negative values when z < 0, which allows the unit to have an
average output closer to 0. This helps alleviate the vanishing gradients problem,
as discussed earlier. The hyperparameter α defines the value that the ELU func‐
tion approaches when z is a large negative number. It is usually set to 1, but you
can tweak it like any other hyperparameter if you want.
• Second, it has a nonzero gradient for z < 0, which avoids the dying units issue.
• Third, the function is smooth everywhere, including around z = 0, which helps
speed up Gradient Descent, since it does not bounce as much left and right of z =
0.
The main drawback of the ELU activation function is that it is slower to compute
than the ReLU and its variants (due to the use of the exponential function), but dur‐
ing training this is compensated by the faster convergence rate. However, at test time


ing training this is compensated by the faster convergence rate. However, at test time
an ELU network will be slower than a ReLU network.
So which activation function should you use for the hidden layers
of your deep neural networks? Although your mileage will vary, in
general ELU > leaky ReLU (and its variants) > ReLU > tanh > logis‐
tic. If you care a lot about runtime performance, then you may pre‐
fer leaky ReLUs over ELUs. If you don’t want to tweak yet another
hyperparameter, you may just use the default α values suggested
earlier (0.01 for the leaky ReLU, and 1 for ELU). If you have spare
time and computing power, you can use cross-validation to evalu‐
ate other activation functions, in particular RReLU if your network
is overfitting, or PReLU if you have a huge training set.
TensorFlow offers an elu() function that you can use to build your neural network.
Simply set the activation_fn argument when calling the fully_connected() func‐
tion, like this:


Simply set the activation_fn argument when calling the fully_connected() func‐
tion, like this:
hidden1 = fully_connected(X, n_hidden1, activation_fn=tf.nn.elu)
TensorFlow does not have a predefined function for leaky ReLUs, but it is easy
enough to define:
def leaky_relu(z, name=None):
    return tf.maximum(0.01 * z, z, name=name)
hidden1 = fully_connected(X, n_hidden1, activation_fn=leaky_relu)
Vanishing/Exploding Gradients Problems
|
281
Download from finelybook www.finelybook.com


7 “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” S. Ioffe
and C. Szegedy (2015).
Batch Normalization
Although using He initialization along with ELU (or any variant of ReLU) can signifi‐
cantly reduce the vanishing/exploding gradients problems at the beginning of train‐
ing, it doesn’t guarantee that they won’t come back during training.
In a 2015 paper,7 Sergey Ioffe and Christian Szegedy proposed a technique called
Batch Normalization (BN) to address the vanishing/exploding gradients problems,
and more generally the problem that the distribution of each layer’s inputs changes
during training, as the parameters of the previous layers change (which they call the
Internal Covariate Shift problem).
The technique consists of adding an operation in the model just before the activation
function of each layer, simply zero-centering and normalizing the inputs, then scaling


function of each layer, simply zero-centering and normalizing the inputs, then scaling
and shifting the result using two new parameters per layer (one for scaling, the other
for shifting). In other words, this operation lets the model learn the optimal scale and
mean of the inputs for each layer.
In order to zero-center and normalize the inputs, the algorithm needs to estimate the
inputs’ mean and standard deviation. It does so by evaluating the mean and standard
deviation of the inputs over the current mini-batch (hence the name “Batch Normal‐
ization”). The whole operation is summarized in Equation 11-3.
Equation 11-3. Batch Normalization algorithm
1 .
μB = 1
mB ∑
i = 1
mB
�i
2 .
σB
2 = 1
mB ∑
i = 1
mB
�i −μB
2
3 .
�i =
�i −μB
σB
2 + �
4 .
�i = γ�i + β
• μB is the empirical mean, evaluated over the whole mini-batch B.
• σB is the empirical standard deviation, also evaluated over the whole mini-batch.
• mB is the number of instances in the mini-batch.
282
|


• mB is the number of instances in the mini-batch.
282
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


• �(i) is the zero-centered and normalized input.
• γ is the scaling parameter for the layer.
• β is the shifting parameter (offset) for the layer.
• ϵ is a tiny number to avoid division by zero (typically 10–3). This is called a
smoothing term.
• z(i) is the output of the BN operation: it is a scaled and shifted version of the
inputs.
At test time, there is no mini-batch to compute the empirical mean and standard
deviation, so instead you simply use the whole training set’s mean and standard devi‐
ation. These are typically efficiently computed during training using a moving aver‐
age. So, in total, four parameters are learned for each batch-normalized layer: γ
(scale), β (offset), μ (mean), and σ (standard deviation).
The authors demonstrated that this technique considerably improved all the deep
neural networks they experimented with. The vanishing gradients problem was
strongly reduced, to the point that they could use saturating activation functions such


strongly reduced, to the point that they could use saturating activation functions such
as the tanh and even the logistic activation function. The networks were also much
less sensitive to the weight initialization. They were able to use much larger learning
rates, significantly speeding up the learning process. Specifically, they note that
“Applied to a state-of-the-art image classification model, Batch Normalization ach‐
ieves the same accuracy with 14 times fewer training steps, and beats the original
model by a significant margin. […] Using an ensemble of batch-normalized net‐
works, we improve upon the best published result on ImageNet classification: reach‐
ing 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of
human raters.” Finally, like a gift that keeps on giving, Batch Normalization also acts
like a regularizer, reducing the need for other regularization techniques (such as
dropout, described later in the chapter).


dropout, described later in the chapter).
Batch Normalization does, however, add some complexity to the model (although it
removes the need for normalizing the input data since the first hidden layer will take
care of that, provided it is batch-normalized). Moreover, there is a runtime penalty:
the neural network makes slower predictions due to the extra computations required
at each layer. So if you need predictions to be lightning-fast, you may want to check
how well plain ELU + He initialization perform before playing with Batch Normaliza‐
tion.
You may find that training is rather slow at first while Gradient
Descent is searching for the optimal scales and offsets for each
layer, but it accelerates once it has found reasonably good values.
Vanishing/Exploding Gradients Problems
|
283
Download from finelybook www.finelybook.com


Implementing Batch Normalization with TensorFlow
TensorFlow provides a batch_normalization() function that simply centers and
normalizes the inputs, but you must compute the mean and standard deviation your‐
self (based on the mini-batch data during training or on the full dataset during test‐
ing, as just discussed) and pass them as parameters to this function, and you must
also handle the creation of the scaling and offset parameters (and pass them to this
function). It is doable, but not the most convenient approach. Instead, you should use
the batch_norm() function, which handles all this for you. You can either call it
directly or tell the fully_connected() function to use it, such as in the following
code:
import tensorflow as tf
from tensorflow.contrib.layers import batch_norm
n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 100
n_outputs = 10
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
is_training = tf.placeholder(tf.bool, shape=(), name='is_training')


is_training = tf.placeholder(tf.bool, shape=(), name='is_training')
bn_params = {
    'is_training': is_training,
    'decay': 0.99,
    'updates_collections': None
}
hidden1 = fully_connected(X, n_hidden1, scope="hidden1",
                          normalizer_fn=batch_norm, normalizer_params=bn_params)
hidden2 = fully_connected(hidden1, n_hidden2, scope="hidden2",
                          normalizer_fn=batch_norm, normalizer_params=bn_params)
logits = fully_connected(hidden2, n_outputs, activation_fn=None,scope="outputs",
                         normalizer_fn=batch_norm, normalizer_params=bn_params)
Let’s walk through this code. The first lines are fairly self-explanatory, until we define
the is_training placeholder, which will either be True or False. This will be used to
tell the batch_norm() function whether it should use the current mini-batch’s mean
and standard deviation (during training) or the running averages that it keeps track
of (during testing).


of (during testing).
Next we define bn_params, which is a dictionary that defines the parameters that will
be passed to the batch_norm() function, including is_training of course. The algo‐
rithm uses exponential decay to compute the running averages, which is why it
requires the decay parameters. Given a new value v, the running average v is updated
through the equation v
v × decay + v × 1 −decay . A good decay value is typically
close to 1—for example, 0.9, 0.99, or 0.999 (you want more 9s for larger datasets and
284
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


smaller mini-batches). Finally, updates_collections should be set to None if you
want the batch_norm() function to update the running averages right before it per‐
forms batch normalization during training (i.e., when is_training=True). If you
don’t set this parameter, by default TensorFlow will just add the operations that
update the running averages to a collection of operations that you must run yourself.
Lastly, we create the layers by calling the fully_connected() function, just like we
did in Chapter 10, but this time we tell it to use the batch_norm() function (with the
parameters nb_params) to normalize the inputs right before calling the activation
function.
Note that by default batch_norm() only centers, normalizes, and shifts the inputs; it
does not scale them (i.e., γ is fixed to 1). This makes sense for layers with no activa‐
tion function or with the ReLU activation function, since the next layer’s weights can


tion function or with the ReLU activation function, since the next layer’s weights can
take care of scaling, but for any other activation function, you should add "scale":
True to bn_params.
You may have noticed that defining the preceding three layers was fairly repetitive
since several parameters were identical. To avoid repeating the same parameters over
and over again, you can create an argument scope using the arg_scope() function:
the first parameter is a list of functions, and the other parameters will be passed to
these functions automatically. The last three lines of the preceding code can be modi‐
fied like so:
[...]
with tf.contrib.framework.arg_scope(
        [fully_connected],
        normalizer_fn=batch_norm,
        normalizer_params=bn_params):
    hidden1 = fully_connected(X, n_hidden1, scope="hidden1")
    hidden2 = fully_connected(hidden1, n_hidden2, scope="hidden2")
    logits = fully_connected(hidden2, n_outputs, scope="outputs",


logits = fully_connected(hidden2, n_outputs, scope="outputs",
                             activation_fn=None)
It may not look much better than before in this small example, but if you have 10 lay‐
ers and want to set the activation function, the initializers, the normalizers, the regu‐
larizers, and so on, it will make your code much more readable.
The rest of the construction phase is the same as in Chapter 10: define the cost func‐
tion, create an optimizer, tell it to minimize the cost function, define the evaluation
operations, create a Saver, and so on.
The execution phase is also pretty much the same, with one exception. Whenever you
run an operation that depends on the batch_norm layer, you need to set the is_train
ing placeholder to True or False:
Vanishing/Exploding Gradients Problems
|
285
Download from finelybook www.finelybook.com


8 “On the difficulty of training recurrent neural networks,” R. Pascanu et al. (2013).
with tf.Session() as sess:
    sess.run(init)
    for epoch in range(n_epochs):
        [...]
        for X_batch, y_batch in zip(X_batches, y_batches):
            sess.run(training_op,
                     feed_dict={is_training: True, X: X_batch, y: y_batch})
        accuracy_score = accuracy.eval(
            feed_dict={is_training: False, X: X_test_scaled, y: y_test}))
        print(accuracy_score)
That’s all! In this tiny example with just two layers, it’s unlikely that Batch Normaliza‐
tion will have a very positive impact, but for deeper networks it can make a tremen‐
dous difference.
Gradient Clipping
A popular technique to lessen the exploding gradients problem is to simply clip the
gradients during backpropagation so that they never exceed some threshold (this is
mostly useful for recurrent neural networks; see Chapter 14). This is called Gradient


mostly useful for recurrent neural networks; see Chapter 14). This is called Gradient
Clipping.8 In general people now prefer Batch Normalization, but it’s still useful to
know about Gradient Clipping and how to implement it.
In TensorFlow, the optimizer’s minimize() function takes care of both computing the
gradients and applying them, so you must instead call the optimizer’s compute_gradi
ents() method first, then create an operation to clip the gradients using the
clip_by_value() function, and finally create an operation to apply the clipped gradi‐
ents using the optimizer’s apply_gradients() method:
threshold = 1.0
optimizer = tf.train.GradientDescentOptimizer(learning_rate)
grads_and_vars = optimizer.compute_gradients(loss)
capped_gvs = [(tf.clip_by_value(grad, -threshold, threshold), var)
              for grad, var in grads_and_vars]
training_op = optimizer.apply_gradients(capped_gvs)
You would then run this training_op at every training step, as usual. It will compute


You would then run this training_op at every training step, as usual. It will compute
the gradients, clip them between –1.0 and 1.0, and apply them. The threshold is a
hyperparameter you can tune.
Reusing Pretrained Layers
It is generally not a good idea to train a very large DNN from scratch: instead, you
should always try to find an existing neural network that accomplishes a similar task
286
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


to the one you are trying to tackle, then just reuse the lower layers of this network:
this is called transfer learning. It will not only speed up training considerably, but will
also require much less training data.
For example, suppose that you have access to a DNN that was trained to classify pic‐
tures into 100 different categories, including animals, plants, vehicles, and everyday
objects. You now want to train a DNN to classify specific types of vehicles. These
tasks are very similar, so you should try to reuse parts of the first network (see
Figure 11-4).
Figure 11-4. Reusing pretrained layers
If the input pictures of your new task don’t have the same size as
the ones used in the original task, you will have to add a prepro‐
cessing step to resize them to the size expected by the original
model. More generally, transfer learning will work only well if the
inputs have similar low-level features.
Reusing a TensorFlow Model


inputs have similar low-level features.
Reusing a TensorFlow Model
If the original model was trained using TensorFlow, you can simply restore it and
train it on the new task:
[...] # construct the original model
with tf.Session() as sess:
    saver.restore(sess, "./my_original_model.ckpt")
    [...] # Train it on your new task
Reusing Pretrained Layers
|
287
Download from finelybook www.finelybook.com


However, in general you will want to reuse only part of the original model (as we will
discuss in a moment). A simple solution is to configure the Saver to restore only a
subset of the variables from the original model. For example, the following code
restores only hidden layers 1, 2, and 3:
[...] # build new model with the same definition as before for hidden layers 1-3
init = tf.global_variables_initializer()
reuse_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                               scope="hidden[123]")
reuse_vars_dict = dict([(var.name, var.name) for var in reuse_vars])
original_saver = tf.Saver(reuse_vars_dict) # saver to restore the original model
new_saver = tf.Saver() # saver to save the new model
with tf.Session() as sess:
    sess.run(init)
    original_saver.restore("./my_original_model.ckpt") # restore layers 1 to 3
    [...] # train the new model
    new_saver.save("./my_new_model.ckpt") # save the whole model


[...] # train the new model
    new_saver.save("./my_new_model.ckpt") # save the whole model
First we build the new model, making sure to copy the original model’s hidden layers
1 to 3. We also create a node to initialize all variables. Then we get the list of all vari‐
ables that were just created with "trainable=True" (which is the default), and we
keep only the ones whose scope matches the regular expression "hidden[123]" (i.e.,
we get all trainable variables in hidden layers 1 to 3). Next we create a dictionary
mapping the name of each variable in the original model to its name in the new
model (generally you want to keep the exact same names). Then we create a Saver
that will restore only these variables, and we create another Saver to save the entire
new model, not just layers 1 to 3. We then start a session and initialize all variables in
the model, then restore the variable values from the original model’s layers 1 to 3.


the model, then restore the variable values from the original model’s layers 1 to 3.
Finally, we train the model on the new task and save it.
The more similar the tasks are, the more layers you want to reuse
(starting with the lower layers). For very similar tasks, you can try
keeping all the hidden layers and just replace the output layer.
Reusing Models from Other Frameworks
If the model was trained using another framework, you will need to load the weights
manually (e.g., using Theano code if it was trained with Theano), then assign them to
the appropriate variables. This can be quite tedious. For example, the following code
shows how you would copy the weight and biases from the first hidden layer of a
model trained using another framework:
288
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


original_w = [...] # Load the weights from the other framework
original_b = [...] # Load the biases from the other framework
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
hidden1 = fully_connected(X, n_hidden1, scope="hidden1")
[...] # # Build the rest of the model
# Get a handle on the variables created by fully_connected()
with tf.variable_scope("", default_name="", reuse=True):  # root scope
    hidden1_weights = tf.get_variable("hidden1/weights")
    hidden1_biases = tf.get_variable("hidden1/biases")
# Create nodes to assign arbitrary values to the weights and biases
original_weights = tf.placeholder(tf.float32, shape=(n_inputs, n_hidden1))
original_biases = tf.placeholder(tf.float32, shape=(n_hidden1))
assign_hidden1_weights = tf.assign(hidden1_weights, original_weights)
assign_hidden1_biases = tf.assign(hidden1_biases, original_biases)
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)


init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    sess.run(assign_hidden1_weights, feed_dict={original_weights: original_w})
    sess.run(assign_hidden1_biases, feed_dict={original_biases: original_b})
    [...] # Train the model on your new task
Freezing the Lower Layers
It is likely that the lower layers of the first DNN have learned to detect low-level fea‐
tures in pictures that will be useful across both image classification tasks, so you can
just reuse these layers as they are. It is generally a good idea to “freeze” their weights
when training the new DNN: if the lower-layer weights are fixed, then the higher-
layer weights will be easier to train (because they won’t have to learn a moving target).
To freeze the lower layers during training, the simplest solution is to give the opti‐
mizer the list of variables to train, excluding the variables from the lower layers:
train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,


train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                               scope="hidden[34]|outputs")
training_op = optimizer.minimize(loss, var_list=train_vars)
The first line gets the list of all trainable variables in hidden layers 3 and 4 and in the
output layer. This leaves out the variables in the hidden layers 1 and 2. Next we pro‐
vide this restricted list of trainable variables to the optimizer’s minimize() function.
Ta-da! Layers 1 and 2 are now frozen: they will not budge during training (these are
often called frozen layers).
Reusing Pretrained Layers
|
289
Download from finelybook www.finelybook.com