3 log2 is the binary log, log2(m) = log(m) / log(2).
5. This question applies only to linear SVMs since kernelized can only use the dual
form. The computational complexity of the primal form of the SVM problem is
proportional to the number of training instances m, while the computational
complexity of the dual form is proportional to a number between m2 and m3. So
if there are millions of instances, you should definitely use the primal form,
because the dual form will be much too slow.
6. If an SVM classifier trained with an RBF kernel underfits the training set, there
might be too much regularization. To decrease it, you need to increase gamma or C
(or both).
7. Let’s call the QP parameters for the hard-margin problem H′, f′, A′ and b′ (see
“Quadratic Programming” on page 159). The QP parameters for the soft-margin
problem have m additional parameters (np = n + 1 + m) and m additional con‐
straints (nc = 2m). They can be defined like so:


straints (nc = 2m). They can be defined like so:
• H is equal to H′, plus m columns of 0s on the right and m rows of 0s at the
bottom: �=
�′ 0 ⋯
0 0
⋮
⋱
• f is equal to f′ with m additional elements, all equal to the value of the hyper‐
parameter C.
• b is equal to b′ with m additional elements, all equal to 0.
• A is equal to A′, with an extra m × m identity matrix Im appended to the right,
– Im just below it, and the rest filled with zeros: �=
�′ �m
0 −�m
For the solutions to exercises 8, 9, and 10, please see the Jupyter notebooks available
at https://github.com/ageron/handson-ml.
Chapter 6: Decision Trees
1. The depth of a well-balanced binary tree containing m leaves is equal to log2(m)3,
rounded up. A binary Decision Tree (one that makes only binary decisions, as is
the case of all trees in Scikit-Learn) will end up more or less well balanced at the
end of training, with one leaf per training instance if it is trained without restric‐

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


end of training, with one leaf per training instance if it is trained without restric‐
tions. Thus, if the training set contains one million instances, the Decision Tree
will have a depth of log2(106) ≈ 20 (actually a bit more since the tree will generally
not be perfectly well balanced).
476
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


2. A node’s Gini impurity is generally lower than its parent’s. This is ensured by the
CART training algorithm’s cost function, which splits each node in a way that
minimizes the weighted sum of its children’s Gini impurities. However, if one
child is smaller than the other, it is possible for it to have a higher Gini impurity
than its parent, as long as this increase is more than compensated for by a
decrease of the other child’s impurity. For example, consider a node containing
four instances of class A and 1 of class B. Its Gini impurity is 1 −1
5
2 −4
5
2 = 0.32.
Now suppose the dataset is one-dimensional and the instances are lined up in the
following order: A, B, A, A, A. You can verify that the algorithm will split this
node after the second instance, producing one child node with instances A, B,
and the other child node with instances A, A, A. The first child node’s Gini
impurity is 1 −1
2
2 −1
2
2 = 0.5, which is higher than its parent. This is compensated


impurity is 1 −1
2
2 −1
2
2 = 0.5, which is higher than its parent. This is compensated
for by the fact that the other node is pure, so the overall weighted Gini impurity
is 2
5 × 0.5 + 3
5 × 0 = 0.2 , which is lower than the parent’s Gini impurity.
3. If a Decision Tree is overfitting the training set, it may be a good idea to decrease
max_depth, since this will constrain the model, regularizing it.
4. Decision Trees don’t care whether or not the training data is scaled or centered;
that’s one of the nice things about them. So if a Decision Tree underfits the train‐
ing set, scaling the input features will just be a waste of time.
5. The computational complexity of training a Decision Tree is O(n × m log(m)). So
if you multiply the training set size by 10, the training time will be multiplied by
K = (n × 10m × log(10m)) / (n × m × log(m)) = 10 × log(10m) / log(m). If m =
106, then K ≈ 11.7, so you can expect the training time to be roughly 11.7 hours.


106, then K ≈ 11.7, so you can expect the training time to be roughly 11.7 hours.
6. Presorting the training set speeds up training only if the dataset is smaller than a
few thousand instances. If it contains 100,000 instances, setting presort=True
will considerably slow down training.
For the solutions to exercises 7 and 8, please see the Jupyter notebooks available at
https://github.com/ageron/handson-ml.
Chapter 7: Ensemble Learning and Random Forests
1. If you have trained five different models and they all achieve 95% precision, you
can try combining them into a voting ensemble, which will often give you even
better results. It works better if the models are very different (e.g., an SVM classi‐
fier, a Decision Tree classifier, a Logistic Regression classifier, and so on). It is
even better if they are trained on different training instances (that’s the whole
point of bagging and pasting ensembles), but if not it will still work as long as the
models are very different.


models are very different.
Exercise Solutions
|
477
Download from finelybook www.finelybook.com


2. A hard voting classifier just counts the votes of each classifier in the ensemble
and picks the class that gets the most votes. A soft voting classifier computes the
average estimated class probability for each class and picks the class with the
highest probability. This gives high-confidence votes more weight and often per‐
forms better, but it works only if every classifier is able to estimate class probabil‐
ities
(e.g.,
for
the
SVM
classifiers
in
Scikit-Learn
you
must
set
probability=True).
3. It is quite possible to speed up training of a bagging ensemble by distributing it
across multiple servers, since each predictor in the ensemble is independent of
the others. The same goes for pasting ensembles and Random Forests, for the
same reason. However, each predictor in a boosting ensemble is built based on
the previous predictor, so training is necessarily sequential, and you will not gain
anything by distributing training across multiple servers. Regarding stacking


anything by distributing training across multiple servers. Regarding stacking
ensembles, all the predictors in a given layer are independent of each other, so
they can be trained in parallel on multiple servers. However, the predictors in one
layer can only be trained after the predictors in the previous layer have all been
trained.
4. With out-of-bag evaluation, each predictor in a bagging ensemble is evaluated
using instances that it was not trained on (they were held out). This makes it pos‐
sible to have a fairly unbiased evaluation of the ensemble without the need for an
additional validation set. Thus, you have more instances available for training,
and your ensemble can perform slightly better.
5. When you are growing a tree in a Random Forest, only a random subset of the
features is considered for splitting at each node. This is true as well for Extra-
Trees, but they go one step further: rather than searching for the best possible


Trees, but they go one step further: rather than searching for the best possible
thresholds, like regular Decision Trees do, they use random thresholds for each
feature. This extra randomness acts like a form of regularization: if a Random
Forest overfits the training data, Extra-Trees might perform better. Moreover,
since Extra-Trees don’t search for the best possible thresholds, they are much
faster to train than Random Forests. However, they are neither faster nor slower
than Random Forests when making predictions.
6. If your AdaBoost ensemble underfits the training data, you can try increasing the
number of estimators or reducing the regularization hyperparameters of the base
estimator. You may also try slightly increasing the learning rate.
7. If your Gradient Boosting ensemble overfits the training set, you should try
decreasing the learning rate. You could also use early stopping to find the right
number of predictors (you probably have too many).


number of predictors (you probably have too many).
For the solutions to exercises 8 and 9, please see the Jupyter notebooks available at
https://github.com/ageron/handson-ml.
478
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


Chapter 8: Dimensionality Reduction
1. Motivations and drawbacks:
• The main motivations for dimensionality reduction are:
— To speed up a subsequent training algorithm (in some cases it may even
remove noise and redundant features, making the training algorithm per‐
form better).
— To visualize the data and gain insights on the most important features.
— Simply to save space (compression).
• The main drawbacks are:
— Some information is lost, possibly degrading the performance of subse‐
quent training algorithms.
— It can be computationally intensive.
— It adds some complexity to your Machine Learning pipelines.
— Transformed features are often hard to interpret.
2. The curse of dimensionality refers to the fact that many problems that do not
exist in low-dimensional space arise in high-dimensional space. In Machine
Learning, one common manifestation is the fact that randomly sampled high-
dimensional vectors are generally very sparse, increasing the risk of overfitting


dimensional vectors are generally very sparse, increasing the risk of overfitting
and making it very difficult to identify patterns in the data without having plenty
of training data.
3. Once a dataset’s dimensionality has been reduced using one of the algorithms we
discussed, it is almost always impossible to perfectly reverse the operation,
because some information gets lost during dimensionality reduction. Moreover,
while some algorithms (such as PCA) have a simple reverse transformation pro‐
cedure that can reconstruct a dataset relatively similar to the original, other algo‐
rithms (such as T-SNE) do not.
4. PCA can be used to significantly reduce the dimensionality of most datasets, even
if they are highly nonlinear, because it can at least get rid of useless dimensions.
However, if there are no useless dimensions—for example, the Swiss roll—then
reducing dimensionality with PCA will lose too much information. You want to
unroll the Swiss roll, not squash it.


unroll the Swiss roll, not squash it.
5. That’s a trick question: it depends on the dataset. Let’s look at two extreme exam‐
ples. First, suppose the dataset is composed of points that are almost perfectly
aligned. In this case, PCA can reduce the dataset down to just one dimension
while still preserving 95% of the variance. Now imagine that the dataset is com‐
posed of perfectly random points, scattered all around the 1,000 dimensions. In
Exercise Solutions
|
479
Download from finelybook www.finelybook.com


this case all 1,000 dimensions are required to preserve 95% of the variance. So the
answer is, it depends on the dataset, and it could be any number between 1 and
1,000. Plotting the explained variance as a function of the number of dimensions
is one way to get a rough idea of the dataset’s intrinsic dimensionality.
6. Regular PCA is the default, but it works only if the dataset fits in memory. Incre‐
mental PCA is useful for large datasets that don’t fit in memory, but it is slower
than regular PCA, so if the dataset fits in memory you should prefer regular
PCA. Incremental PCA is also useful for online tasks, when you need to apply
PCA on the fly, every time a new instance arrives. Randomized PCA is useful
when you want to considerably reduce dimensionality and the dataset fits in
memory; in this case, it is much faster than regular PCA. Finally, Kernel PCA is
useful for nonlinear datasets.
7. Intuitively, a dimensionality reduction algorithm performs well if it eliminates a


7. Intuitively, a dimensionality reduction algorithm performs well if it eliminates a
lot of dimensions from the dataset without losing too much information. One
way to measure this is to apply the reverse transformation and measure the
reconstruction error. However, not all dimensionality reduction algorithms pro‐
vide a reverse transformation. Alternatively, if you are using dimensionality
reduction as a preprocessing step before another Machine Learning algorithm
(e.g., a Random Forest classifier), then you can simply measure the performance
of that second algorithm; if dimensionality reduction did not lose too much
information, then the algorithm should perform just as well as when using the
original dataset.
8. It can absolutely make sense to chain two different dimensionality reduction
algorithms. A common example is using PCA to quickly get rid of a large num‐
ber of useless dimensions, then applying another much slower dimensionality


ber of useless dimensions, then applying another much slower dimensionality
reduction algorithm, such as LLE. This two-step approach will likely yield the
same performance as using LLE only, but in a fraction of the time.
For the solutions to exercises 9 and 10, please see the Jupyter notebooks available at
https://github.com/ageron/handson-ml.
Chapter 9: Up and Running with TensorFlow
1. Main benefits and drawbacks of creating a computation graph rather than
directly executing the computations:
• Main benefits:
— TensorFlow can automatically compute the gradients for you (using
reverse-mode autodiff).
— TensorFlow can take care of running the operations in parallel in different
threads.
480
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


— It makes it easier to run the same model across different devices.
— It simplifies introspection—for example, to view the model in TensorBoard.
• Main drawbacks:
— It makes the learning curve steeper.
— It makes step-by-step debugging harder.
2. Yes, the statement a_val = a.eval(session=sess) is indeed equivalent to a_val
= sess.run(a).
3. No, the statement a_val, b_val = a.eval(session=sess), b.eval(ses
sion=sess) is not equivalent to a_val, b_val = sess.run([a, b]). Indeed, the
first statement runs the graph twice (once to compute a, once to compute b),
while the second statement runs the graph only once. If any of these operations
(or the ops they depend on) have side effects (e.g., a variable is modified, an item
is inserted in a queue, or a reader reads a file), then the effects will be different. If
they don’t have side effects, both statements will return the same result, but the
second statement will be faster than the first.


second statement will be faster than the first.
4. No, you cannot run two graphs in the same session. You would have to merge the
graphs into a single graph first.
5. In local TensorFlow, sessions manage variable values, so if you create a graph g
containing a variable w, then start two threads and open a local session in each
thread, both using the same graph g, then each session will have its own copy of
the variable w. However, in distributed TensorFlow, variable values are stored in
containers managed by the cluster, so if both sessions connect to the same cluster
and use the same container, then they will share the same variable value for w.
6. A variable is initialized when you call its initializer, and it is destroyed when the
session ends. In distributed TensorFlow, variables live in containers on the clus‐
ter, so closing a session will not destroy the variable. To destroy a variable, you
need to clear its container.


need to clear its container.
7. Variables and placeholders are extremely different, but beginners often confuse
them:
• A variable is an operation that holds a value. If you run the variable, it returns
that value. Before you can run it, you need to initialize it. You can change the
variable’s value (for example, by using an assignment operation). It is stateful:
the variable keeps the same value upon successive runs of the graph. It is typi‐
cally used to hold model parameters but also for other purposes (e.g., to count
the global training step).
• Placeholders technically don’t do much: they just hold information about the
type and shape of the tensor they represent, but they have no value. In fact, if
Exercise Solutions
|
481
Download from finelybook www.finelybook.com


you try to evaluate an operation that depends on a placeholder, you must feed
TensorFlow the value of the placeholder (using the feed_dict argument) or
else you will get an exception. Placeholders are typically used to feed training
or test data to TensorFlow during the execution phase. They are also useful to
pass a value to an assignment node, to change the value of a variable (e.g.,
model weights).
8. If you run the graph to evaluate an operation that depends on a placeholder but
you don’t feed its value, you get an exception. If the operation does not depend
on the placeholder, then no exception is raised.
9. When you run a graph, you can feed the output value of any operation, not just
the value of placeholders. In practice, however, this is rather rare (it can be useful,
for example, when you are caching the output of frozen layers; see Chapter 11).
10. You can specify a variable’s initial value when constructing the graph, and it will


10. You can specify a variable’s initial value when constructing the graph, and it will
be initialized later when you run the variable’s initializer during the execution
phase. If you want to change that variable’s value to anything you want during the
execution phase, then the simplest option is to create an assignment node (dur‐
ing the graph construction phase) using the tf.assign() function, passing the
variable and a placeholder as parameters. During the execution phase, you can
run the assignment operation and feed the variable’s new value using the place‐
holder.
import tensorflow as tf
x = tf.Variable(tf.random_uniform(shape=(), minval=0.0, maxval=1.0))
x_new_val = tf.placeholder(shape=(), dtype=tf.float32)
x_assign = tf.assign(x, x_new_val)
with tf.Session():
    x.initializer.run() # random number is sampled *now*
    print(x.eval()) # 0.646157 (some random number)
    x_assign.eval(feed_dict={x_new_val: 5.0})
    print(x.eval()) # 5.0


x_assign.eval(feed_dict={x_new_val: 5.0})
    print(x.eval()) # 5.0
11. Reverse-mode autodiff (implemented by TensorFlow) needs to traverse the graph
only twice in order to compute the gradients of the cost function with regards to
any number of variables. On the other hand, forward-mode autodiff would need
to run once for each variable (so 10 times if we want the gradients with regards to
10 different variables). As for symbolic differentiation, it would build a different
graph to compute the gradients, so it would not traverse the original graph at all
(except when building the new gradients graph). A highly optimized symbolic
differentiation system could potentially run the new gradients graph only once to
compute the gradients with regards to all variables, but that new graph may be
horribly complex and inefficient compared to the original graph.
482
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


12. See the Jupyter notebooks available at https://github.com/ageron/handson-ml.
Chapter 10: Introduction to Artificial Neural Networks
1. Here is a neural network based on the original artificial neurons that computes A
⊕ B (where ⊕ represents the exclusive OR), using the fact that A ⊕ B = (A ∧ ¬ B)
∨ (¬ A ∧ B). There are other solutions—for example, using the fact that A ⊕ B =
(A ∨ B) ∧ ¬(A ∧ B), or the fact that A ⊕ B = (A ∨ B) ∧ (¬ A ∨ ∧ B), and so on.
2. A classical Perceptron will converge only if the dataset is linearly separable, and it
won’t be able to estimate class probabilities. In contrast, a Logistic Regression
classifier will converge to a good solution even if the dataset is not linearly sepa‐
rable, and it will output class probabilities. If you change the Perceptron’s activa‐
tion function to the logistic activation function (or the softmax activation
function if there are multiple neurons), and if you train it using Gradient Descent


function if there are multiple neurons), and if you train it using Gradient Descent
(or some other optimization algorithm minimizing the cost function, typically
cross entropy), then it becomes equivalent to a Logistic Regression classifier.
3. The logistic activation function was a key ingredient in training the first MLPs
because its derivative is always nonzero, so Gradient Descent can always roll
down the slope. When the activation function is a step function, Gradient
Descent cannot move, as there is no slope at all.
4. The step function, the logistic function, the hyperbolic tangent, the rectified lin‐
ear unit (see Figure 10-8). See Chapter 11 for other examples, such as ELU and
variants of the ReLU.
5. Considering the MLP described in the question: suppose you have an MLP com‐
posed of one input layer with 10 passthrough neurons, followed by one hidden
layer with 50 artificial neurons, and finally one output layer with 3 artificial neu‐


layer with 50 artificial neurons, and finally one output layer with 3 artificial neu‐
rons. All artificial neurons use the ReLU activation function.
Exercise Solutions
|
483
Download from finelybook www.finelybook.com


4 When the values to predict can vary by many orders of magnitude, then you may want to predict the loga‐
rithm of the target value rather than the target value directly. Simply computing the exponential of the neural
network’s output will give you the estimated value (since exp(log v) = v).
• The shape of the input matrix X is m × 10, where m represents the training
batch size.
• The shape of the hidden layer’s weight vector Wh is 10 × 50 and the length of
its bias vector bh is 50.
• The shape of the output layer’s weight vector Wo is 50 × 3, and the length of its
bias vector bo is 3.
• The shape of the network’s output matrix Y is m × 3.
• Y = (X · Wh + bh) · Wo + bo. Note that when you are adding a bias vector to a
matrix, it is added to every single row in the matrix, which is called broadcast‐
ing.
6. To classify email into spam or ham, you just need one neuron in the output layer
of a neural network—for example, indicating the probability that the email is


of a neural network—for example, indicating the probability that the email is
spam. You would typically use the logistic activation function in the output layer
when estimating a probability. If instead you want to tackle MNIST, you need 10
neurons in the output layer, and you must replace the logistic function with the
softmax activation function, which can handle multiple classes, outputting one
probability per class. Now, if you want your neural network to predict housing
prices like in Chapter 2, then you need one output neuron, using no activation
function at all in the output layer.4
7. Backpropagation is a technique used to train artificial neural networks. It first
computes the gradients of the cost function with regards to every model parame‐
ter (all the weights and biases), and then it performs a Gradient Descent step
using these gradients. This backpropagation step is typically performed thou‐
sands or millions of times, using many training batches, until the model parame‐


sands or millions of times, using many training batches, until the model parame‐
ters converge to values that (hopefully) minimize the cost function. To compute
the gradients, backpropagation uses reverse-mode autodiff (although it wasn’t
called that when backpropagation was invented, and it has been reinvented sev‐
eral times). Reverse-mode autodiff performs a forward pass through a computa‐
tion graph, computing every node’s value for the current training batch, and then
it performs a reverse pass, computing all the gradients at once (see Appendix D
for more details). So what’s the difference? Well, backpropagation refers to the
whole process of training an artificial neural network using multiple backpropa‐
gation steps, each of which computes gradients and uses them to perform a Gra‐
dient Descent step. In contrast, reverse-mode autodiff is a simply a technique to
compute gradients efficiently, and it happens to be used by backpropagation.
484
|
Appendix A: Exercise Solutions


484
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


5 In Chapter 11 we discuss many techniques that introduce additional hyperparameters: type of weight initiali‐
zation, activation function hyperparameters (e.g., amount of leak in leaky ReLU), Gradient Clipping thres‐
hold, type of optimizer and its hyperparameters (e.g., the momentum hyperparameter when using a
MomentumOptimizer), type of regularization for each layer, and the regularization hyperparameters (e.g., drop‐
out rate when using dropout) and so on.
8. Here is a list of all the hyperparameters you can tweak in a basic MLP: the num‐
ber of hidden layers, the number of neurons in each hidden layer, and the activa‐
tion function used in each hidden layer and in the output layer.5 In general, the
ReLU activation function (or one of its variants; see Chapter 11) is a good default
for the hidden layers. For the output layer, in general you will want the logistic
activation function for binary classification, the softmax activation function for


activation function for binary classification, the softmax activation function for
multiclass classification, or no activation function for regression.
If the MLP overfits the training data, you can try reducing the number of hidden
layers and reducing the number of neurons per hidden layer.
9. See the Jupyter notebooks available at https://github.com/ageron/handson-ml.
Chapter 11: Training Deep Neural Nets
1. No, all weights should be sampled independently; they should not all have the
same initial value. One important goal of sampling weights randomly is to break
symmetries: if all the weights have the same initial value, even if that value is not
zero, then symmetry is not broken (i.e., all neurons in a given layer are equiva‐
lent), and backpropagation will be unable to break it. Concretely, this means that
all the neurons in any given layer will always have the same weights. It’s like hav‐
ing just one neuron per layer, and much slower. It is virtually impossible for such


ing just one neuron per layer, and much slower. It is virtually impossible for such
a configuration to converge to a good solution.
2. It is perfectly fine to initialize the bias terms to zero. Some people like to initialize
them just like weights, and that’s okay too; it does not make much difference.
3. A few advantages of the ELU function over the ReLU function are:
• It can take on negative values, so the average output of the neurons in any
given layer is typically closer to 0 than when using the ReLU activation func‐
tion (which never outputs negative values). This helps alleviate the vanishing
gradients problem.
• It always has a nonzero derivative, which avoids the dying units issue that can
affect ReLU units.
Exercise Solutions
|
485
Download from finelybook www.finelybook.com


• It is smooth everywhere, whereas the ReLU’s slope abruptly jumps from 0 to 1
at z = 0. Such an abrupt change can slow down Gradient Descent because it
will bounce around z = 0.
4. The ELU activation function is a good default. If you need the neural network to
be as fast as possible, you can use one of the leaky ReLU variants instead (e.g., a
simple leaky ReLU using the default hyperparameter value). The simplicity of the
ReLU activation function makes it many people’s preferred option, despite the
fact that they are generally outperformed by the ELU and leaky ReLU. However,
the ReLU activation function’s capability of outputting precisely zero can be use‐
ful in some cases (e.g., see Chapter 15). The hyperbolic tangent (tanh) can be use‐
ful in the output layer if you need to output a number between –1 and 1, but
nowadays it is not used much in hidden layers. The logistic activation function is
also useful in the output layer when you need to estimate a probability (e.g., for


also useful in the output layer when you need to estimate a probability (e.g., for
binary classification), but it is also rarely used in hidden layers (there are excep‐
tions—for example, for the coding layer of variational autoencoders; see Chap‐
ter 15). Finally, the softmax activation function is useful in the output layer to
output probabilities for mutually exclusive classes, but other than that it is rarely
(if ever) used in hidden layers.
5. If you set the momentum hyperparameter too close to 1 (e.g., 0.99999) when using
a MomentumOptimizer, then the algorithm will likely pick up a lot of speed, hope‐
fully roughly toward the global minimum, but then it will shoot right past the
minimum, due to its momentum. Then it will slow down and come back, accel‐
erate again, overshoot again, and so on. It may oscillate this way many times
before converging, so overall it will take much longer to converge than with a
smaller momentum value.


smaller momentum value.
6. One way to produce a sparse model (i.e., with most weights equal to zero) is to
train the model normally, then zero out tiny weights. For more sparsity, you can
apply ℓ1 regularization during training, which pushes the optimizer toward spar‐
sity. A third option is to combine ℓ1 regularization with dual averaging, using
TensorFlow’s FTRLOptimizer class.
7. Yes, dropout does slow down training, in general roughly by a factor of two.
However, it has no impact on inference since it is only turned on during training.
For the solutions to exercises 8, 9, and 10, please see the Jupyter notebooks available
at https://github.com/ageron/handson-ml.
486
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


Chapter 12: Distributing TensorFlow Across Devices and
Servers
1. When a TensorFlow process starts, it grabs all the available memory on all GPU
devices that are visible to it, so if you get a CUDA_ERROR_OUT_OF_MEMORY when
starting your TensorFlow program, it probably means that other processes are
running that have already grabbed all the memory on at least one visible GPU
device (most likely it is another TensorFlow process). To fix this problem, a triv‐
ial solution is to stop the other processes and try again. However, if you need all
processes to run simultaneously, a simple option is to dedicate different devices
to each process, by setting the CUDA_VISIBLE_DEVICES environment variable
appropriately for each device. Another option is to configure TensorFlow to grab
only part of the GPU memory, instead of all of it, by creating a ConfigProto, set‐
ting its gpu_options.per_process_gpu_memory_fraction to the proportion of


ting its gpu_options.per_process_gpu_memory_fraction to the proportion of
the total memory that it should grab (e.g., 0.4), and using this ConfigProto when
opening a session. The last option is to tell TensorFlow to grab memory only
when it needs it by setting the gpu_options.allow_growth to True. However,
this last option is usually not recommended because any memory that Tensor‐
Flow grabs is never released, and it is harder to guarantee a repeatable behavior
(there may be race conditions depending on which processes start first, how
much memory they need during training, and so on).
2. By pinning an operation on a device, you are telling TensorFlow that this is
where you would like this operation to be placed. However, some constraints may
prevent TensorFlow from honoring your request. For example, the operation
may have no implementation (called a kernel) for that particular type of device.
In this case, TensorFlow will raise an exception by default, but you can configure


In this case, TensorFlow will raise an exception by default, but you can configure
it to fall back to the CPU instead (this is called soft placement). Another example
is an operation that can modify a variable; this operation and the variable need to
be collocated. So the difference between pinning an operation and placing an
operation is that pinning is what you ask TensorFlow (“Please place this opera‐
tion on GPU #1”) while placement is what TensorFlow actually ends up doing
(“Sorry, falling back to the CPU”).
3. If you are running on a GPU-enabled TensorFlow installation, and you just use
the default placement, then if all operations have a GPU kernel (i.e., a GPU
implementation), yes, they will all be placed on the first GPU. However, if one or
more operations do not have a GPU kernel, then by default TensorFlow will raise
an exception. If you configure TensorFlow to fall back to the CPU instead (soft
placement), then all operations will be placed on the first GPU except the ones


placement), then all operations will be placed on the first GPU except the ones
without a GPU kernel and all the operations that must be collocated with them
(see the answer to the previous exercise).
Exercise Solutions
|
487
Download from finelybook www.finelybook.com


4. Yes, if you pin a variable to "/gpu:0", it can be used by operations placed
on /gpu:1. TensorFlow will automatically take care of adding the appropriate
operations to transfer the variable’s value across devices. The same goes for devi‐
ces located on different servers (as long as they are part of the same cluster).
5. Yes, two operations placed on the same device can run in parallel: TensorFlow
automatically takes care of running operations in parallel (on different CPU
cores or different GPU threads), as long as no operation depends on another
operation’s output. Moreover, you can start multiple sessions in parallel threads
(or processes), and evaluate operations in each thread. Since sessions are inde‐
pendent, TensorFlow will be able to evaluate any operation from one session in
parallel with any operation from another session.
6. Control dependencies are used when you want to postpone the evaluation of an


6. Control dependencies are used when you want to postpone the evaluation of an
operation X until after some other operations are run, even though these opera‐
tions are not required to compute X. This is useful in particular when X would
occupy a lot of memory and you only need it later in the computation graph, or if
X uses up a lot of I/O (for example, it requires a large variable value located on a
different device or server) and you don’t want it to run at the same time as other
I/O-hungry operations, to avoid saturating the bandwidth.
7. You’re in luck! In distributed TensorFlow, the variable values live in containers
managed by the cluster, so even if you close the session and exit the client pro‐
gram, the model parameters are still alive and well on the cluster. You simply
need to open a new session to the cluster and save the model (make sure you
don’t call the variable initializers or restore a previous model, as this would
destroy your precious new model!).


destroy your precious new model!).
For the solutions to exercises 8, 9, and 10, please see the Jupyter notebooks available
at https://github.com/ageron/handson-ml.
Chapter 13: Convolutional Neural Networks
1. These are the main advantages of a CNN over a fully connected DNN for image
classification:
• Because consecutive layers are only partially connected and because it heavily
reuses its weights, a CNN has many fewer parameters than a fully connected
DNN, which makes it much faster to train, reduces the risk of overfitting, and
requires much less training data.
• When a CNN has learned a kernel that can detect a particular feature, it can
detect that feature anywhere on the image. In contrast, when a DNN learns a
feature in one location, it can detect it only in that particular location. Since
images typically have very repetitive features, CNNs are able to generalize
488
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


much better than DNNs for image processing tasks such as classification, using
fewer training examples.
• Finally, a DNN has no prior knowledge of how pixels are organized; it does not
know that nearby pixels are close. A CNN’s architecture embeds this prior
knowledge. Lower layers typically identify features in small areas of the images,
while higher layers combine the lower-level features into larger features. This
works well with most natural images, giving CNNs a decisive head start com‐
pared to DNNs.
2. Let’s compute how many parameters the CNN has. Since its first convolutional
layer has 3 × 3 kernels, and the input has three channels (red, green, and blue),
then each feature map has 3 × 3 × 3 weights, plus a bias term. That’s 28 parame‐
ters per feature map. Since this first convolutional layer has 100 feature maps, it
has a total of 2,800 parameters. The second convolutional layer has 3 × 3 kernels,


has a total of 2,800 parameters. The second convolutional layer has 3 × 3 kernels,
and its input is the set of 100 feature maps of the previous layer, so each feature
map has 3 × 3 × 100 = 900 weights, plus a bias term. Since it has 200 feature
maps, this layer has 901 × 200 = 180,200 parameters. Finally, the third and last
convolutional layer also has 3 × 3 kernels, and its input is the set of 200 feature
maps of the previous layers, so each feature map has 3 × 3 × 200 = 1,800 weights,
plus a bias term. Since it has 400 feature maps, this layer has a total of 1,801 × 400
= 720,400 parameters. All in all, the CNN has 2,800 + 180,200 + 720,400 =
903,400 parameters.
Now let’s compute how much RAM this neural network will require (at least)
when making a prediction for a single instance. First let’s compute the feature
map size for each layer. Since we are using a stride of 2 and SAME padding, the
horizontal and vertical size of the feature maps are divided by 2 at each layer


horizontal and vertical size of the feature maps are divided by 2 at each layer
(rounding up if necessary), so as the input channels are 200 × 300 pixels, the first
layer’s feature maps are 100 × 150, the second layer’s feature maps are 50 × 75,
and the third layer’s feature maps are 25 × 38. Since 32 bits is 4 bytes and the first
convolutional layer has 100 feature maps, this first layer takes up 4 x 100 × 150 ×
100 = 6 million bytes (about 5.7 MB, considering that 1 MB = 1,024 KB and 1 KB
= 1,024 bytes). The second layer takes up 4 × 50 × 75 × 200 = 3 million bytes
(about 2.9 MB). Finally, the third layer takes up 4 × 25 × 38 × 400 = 1,520,000
bytes (about 1.4 MB). However, once a layer has been computed, the memory
occupied by the previous layer can be released, so if everything is well optimized,
only 6 + 9 = 15 million bytes (about 14.3 MB) of RAM will be required (when the
second layer has just been computed, but the memory occupied by the first layer


second layer has just been computed, but the memory occupied by the first layer
is not released yet). But wait, you also need to add the memory occupied by the
CNN’s parameters. We computed earlier that it has 903,400 parameters, each
using up 4 bytes, so this adds 3,613,600 bytes (about 3.4 MB). The total RAM
required is (at least) 18,613,600 bytes (about 17.8 MB).
Exercise Solutions
|
489
Download from finelybook www.finelybook.com


Lastly, let’s compute the minimum amount of RAM required when training the
CNN on a mini-batch of 50 images. During training TensorFlow uses backpropa‐
gation, which requires keeping all values computed during the forward pass until
the reverse pass begins. So we must compute the total RAM required by all layers
for a single instance and multiply that by 50! At that point let’s start counting in
megabytes rather than bytes. We computed before that the three layers require
respectively 5.7, 2.9, and 1.4 MB for each instance. That’s a total of 10.0 MB per
instance. So for 50 instances the total RAM is 500 MB. Add to that the RAM
required by the input images, which is 50 × 4 × 200 × 300 × 3 = 36 million bytes
(about 34.3 MB), plus the RAM required for the model parameters, which is
about 3.4 MB (computed earlier), plus some RAM for the gradients (we will
neglect them since they can be released gradually as backpropagation goes down


neglect them since they can be released gradually as backpropagation goes down
the layers during the reverse pass). We are up to a total of roughly 500.0 + 34.3 +
3.4 = 537.7 MB. And that’s really an optimistic bare minimum.
3. If your GPU runs out of memory while training a CNN, here are five things you
could try to solve the problem (other than purchasing a GPU with more RAM):
• Reduce the mini-batch size.
• Reduce dimensionality using a larger stride in one or more layers.
• Remove one or more layers.
• Use 16-bit floats instead of 32-bit floats.
• Distribute the CNN across multiple devices.
4. A max pooling layer has no parameters at all, whereas a convolutional layer has
quite a few (see the previous questions).
5. A local response normalization layer makes the neurons that most strongly acti‐
vate inhibit neurons at the same location but in neighboring feature maps, which
encourages different feature maps to specialize and pushes them apart, forcing


encourages different feature maps to specialize and pushes them apart, forcing
them to explore a wider range of features. It is typically used in the lower layers to
have a larger pool of low-level features that the upper layers can build upon.
6. The main innovations in AlexNet compared to LeNet-5 are (1) it is much larger
and deeper, and (2) it stacks convolutional layers directly on top of each other,
instead of stacking a pooling layer on top of each convolutional layer. The main
innovation in GoogLeNet is the introduction of inception modules, which make it
possible to have a much deeper net than previous CNN architectures, with fewer
parameters. Finally, ResNet’s main innovation is the introduction of skip connec‐
tions, which make it possible to go well beyond 100 layers. Arguably, its simplic‐
ity and consistency are also rather innovative.
For the solutions to exercises 7, 8, 9, and 10, please see the Jupyter notebooks avail‐
able at https://github.com/ageron/handson-ml.
490


able at https://github.com/ageron/handson-ml.
490
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


Chapter 14: Recurrent Neural Networks
1. Here are a few RNN applications:
• For a sequence-to-sequence RNN: predicting the weather (or any other time
series), machine translation (using an encoder–decoder architecture), video
captioning, speech to text, music generation (or other sequence generation),
identifying the chords of a song.
• For a sequence-to-vector RNN: classifying music samples by music genre, ana‐
lyzing the sentiment of a book review, predicting what word an aphasic patient
is thinking of based on readings from brain implants, predicting the probabil‐
ity that a user will want to watch a movie based on her watch history (this is
one of many possible implementations of collaborative filtering).
• For a vector-to-sequence RNN: image captioning, creating a music playlist
based on an embedding of the current artist, generating a melody based on a
set of parameters, locating pedestrians in a picture (e.g., a video frame from a
self-driving car’s camera).


self-driving car’s camera).
2. In general, if you translate a sentence one word at a time, the result will be terri‐
ble. For example, the French sentence “Je vous en prie” means “You are welcome,”
but if you translate it one word at a time, you get “I you in pray.” Huh? It is much
better to read the whole sentence first and then translate it. A plain sequence-to-
sequence RNN would start translating a sentence immediately after reading the
first word, while an encoder–decoder RNN will first read the whole sentence and
then translate it. That said, one could imagine a plain sequence-to-sequence
RNN that would output silence whenever it is unsure about what to say next (just
like human translators do when they must translate a live broadcast).
3. To classify videos based on the visual content, one possible architecture could be
to take (say) one frame per second, then run each frame through a convolutional
neural network, feed the output of the CNN to a sequence-to-vector RNN, and


neural network, feed the output of the CNN to a sequence-to-vector RNN, and
finally run its output through a softmax layer, giving you all the class probabili‐
ties. For training you would just use cross entropy as the cost function. If you
wanted to use the audio for classification as well, you could convert every second
of audio to a spectrograph, feed this spectrograph to a CNN, and feed the output
of this CNN to the RNN (along with the corresponding output of the other
CNN).
4. Building an RNN using dynamic_rnn() rather than static_rnn() offers several
advantages:
• It is based on a while_loop() operation that is able to swap the GPU’s memory
to the CPU’s memory during backpropagation, avoiding out-of-memory
errors.
Exercise Solutions
|
491
Download from finelybook www.finelybook.com


• It is arguably easier to use, as it can directly take a single tensor as input and
output (covering all time steps), rather than a list of tensors (one per time
step). No need to stack, unstack, or transpose.
• It generates a smaller graph, easier to visualize in TensorBoard.
5. To handle variable length input sequences, the simplest option is to set the
sequence_length parameter when calling the static_rnn() or dynamic_rnn()
functions. Another option is to pad the smaller inputs (e.g., with zeros) to make
them the same size as the largest input (this may be faster than the first option if
the input sequences all have very similar lengths). To handle variable-length out‐
put sequences, if you know in advance the length of each output sequence, you
can use the sequence_length parameter (for example, consider a sequence-to-
sequence RNN that labels every frame in a video with a violence score: the output
sequence will be exactly the same length as the input sequence). If you don’t


sequence will be exactly the same length as the input sequence). If you don’t
know in advance the length of the output sequence, you can use the padding
trick: always output the same size sequence, but ignore any outputs that come
after the end-of-sequence token (by ignoring them when computing the cost
function).
6. To distribute training and execution of a deep RNN across multiple GPUs, a
common technique is simply to place each layer on a different GPU (see Chap‐
ter 12).
For the solutions to exercises 7, 8, and 9, please see the Jupyter notebooks available at
https://github.com/ageron/handson-ml.
Chapter 15: Autoencoders
1. Here are some of the main tasks that autoencoders are used for:
• Feature extraction
• Unsupervised pretraining
• Dimensionality reduction
• Generative models
• Anomaly detection (an autoencoder is generally bad at reconstructing outliers)
2. If you want to train a classifier and you have plenty of unlabeled training data,


2. If you want to train a classifier and you have plenty of unlabeled training data,
but only a few thousand labeled instances, then you could first train a deep
autoencoder on the full dataset (labeled + unlabeled), then reuse its lower half for
the classifier (i.e., reuse the layers up to the codings layer, included) and train the
classifier using the labeled data. If you have little labeled data, you probably want
to freeze the reused layers when training the classifier.
492
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


3. The fact that an autoencoder perfectly reconstructs its inputs does not necessarily
mean that it is a good autoencoder; perhaps it is simply an overcomplete autoen‐
coder that learned to copy its inputs to the codings layer and then to the outputs.
In fact, even if the codings layer contained a single neuron, it would be possible
for a very deep autoencoder to learn to map each training instance to a different
coding (e.g., the first instance could be mapped to 0.001, the second to 0.002, the
third to 0.003, and so on), and it could learn “by heart” to reconstruct the right
training instance for each coding. It would perfectly reconstruct its inputs
without really learning any useful pattern in the data. In practice such a mapping
is unlikely to happen, but it illustrates the fact that perfect reconstructions are not
a guarantee that the autoencoder learned anything useful. However, if it produces
very bad reconstructions, then it is almost guaranteed to be a bad autoencoder.


very bad reconstructions, then it is almost guaranteed to be a bad autoencoder.
To evaluate the performance of an autoencoder, one option is to measure the
reconstruction loss (e.g., compute the MSE, the mean square of the outputs
minus the inputs). Again, a high reconstruction loss is a good sign that the
autoencoder is bad, but a low reconstruction loss is not a guarantee that it is
good. You should also evaluate the autoencoder according to what it will be used
for. For example, if you are using it for unsupervised pretraining of a classifier,
then you should also evaluate the classifier’s performance.
4. An undercomplete autoencoder is one whose codings layer is smaller than the
input and output layers. If it is larger, then it is an overcomplete autoencoder.
The main risk of an excessively undercomplete autoencoder is that it may fail to
reconstruct the inputs. The main risk of an overcomplete autoencoder is that it


reconstruct the inputs. The main risk of an overcomplete autoencoder is that it
may just copy the inputs to the outputs, without learning any useful feature.
5. To tie the weights of an encoder layer and its corresponding decoder layer, you
simply make the decoder weights equal to the transpose of the encoder weights.
This reduces the number of parameters in the model by half, often making train‐
ing converge faster with less training data, and reducing the risk of overfitting the
training set.
6. To visualize the features learned by the lower layer of a stacked autoencoder, a
common technique is simply to plot the weights of each neuron, by reshaping
each weight vector to the size of an input image (e.g., for MNIST, reshaping a
weight vector of shape [784] to [28, 28]). To visualize the features learned by
higher layers, one technique is to display the training instances that most activate
each neuron.
7. A generative model is a model capable of randomly generating outputs that


each neuron.
7. A generative model is a model capable of randomly generating outputs that
resemble the training instances. For example, once trained successfully on the
MNIST dataset, a generative model can be used to randomly generate realistic
images of digits. The output distribution is typically similar to the training data.
For example, since MNIST contains many images of each digit, the generative
model would output roughly the same number of images of each digit. Some
Exercise Solutions
|
493
Download from finelybook www.finelybook.com


generative models can be parametrized—for example, to generate only some
kinds of outputs. An example of a generative autoencoder is the variational
autoencoder.
For the solutions to exercises 8, 9, and 10, please see the Jupyter notebooks available
at https://github.com/ageron/handson-ml.
Chapter 16: Reinforcement Learning
1. Reinforcement Learning is an area of Machine Learning aimed at creating agents
capable of taking actions in an environment in a way that maximizes rewards
over time. There are many differences between RL and regular supervised and
unsupervised learning. Here are a few:
• In supervised and unsupervised learning, the goal is generally to find patterns
in the data. In Reinforcement Learning, the goal is to find a good policy.
• Unlike in supervised learning, the agent is not explicitly given the “right”
answer. It must learn by trial and error.
• Unlike in unsupervised learning, there is a form of supervision, through


• Unlike in unsupervised learning, there is a form of supervision, through
rewards. We do not tell the agent how to perform the task, but we do tell it
when it is making progress or when it is failing.
• A Reinforcement Learning agent needs to find the right balance between
exploring the environment, looking for new ways of getting rewards, and
exploiting sources of rewards that it already knows. In contrast, supervised and
unsupervised learning systems generally don’t need to worry about explora‐
tion; they just feed on the training data they are given.
• In supervised and unsupervised learning, training instances are typically inde‐
pendent (in fact, they are generally shuffled). In Reinforcement Learning, con‐
secutive observations are generally not independent. An agent may remain in
the same region of the environment for a while before it moves on, so consecu‐
tive observations will be very correlated. In some cases a replay memory is


tive observations will be very correlated. In some cases a replay memory is
used to ensure that the training algorithm gets fairly independent observa‐
tions.
2. Here are a few possible applications of Reinforcement Learning, other than those
mentioned in Chapter 16:
Music personalization
The environment is a user’s personalized web radio. The agent is the software
deciding what song to play next for that user. Its possible actions are to play
any song in the catalog (it must try to choose a song the user will enjoy) or to
play an advertisement (it must try to choose an ad that the user will be inter‐
494
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


ested in). It gets a small reward every time the user listens to a song, a larger
reward every time the user listens to an ad, a negative reward when the user
skips a song or an ad, and a very negative reward if the user leaves.
Marketing
The environment is your company’s marketing department. The agent is the
software that defines which customers a mailing campaign should be sent to,
given their profile and purchase history (for each customer it has two possi‐
ble actions: send or don’t send). It gets a negative reward for the cost of the
mailing campaign, and a positive reward for estimated revenue generated
from this campaign.
Product delivery
Let the agent control a fleet of delivery trucks, deciding what they should
pick up at the depots, where they should go, what they should drop off, and
so on. They would get positive rewards for each product delivered on time,
and negative rewards for late deliveries.


and negative rewards for late deliveries.
3. When estimating the value of an action, Reinforcement Learning algorithms typ‐
ically sum all the rewards that this action led to, giving more weight to immediate
rewards, and less weight to later rewards (considering that an action has more
influence on the near future than on the distant future). To model this, a discount
rate is typically applied at each time step. For example, with a discount rate of 0.9,
a reward of 100 that is received two time steps later is counted as only 0.92 × 100
= 81 when you are estimating the value of the action. You can think of the dis‐
count rate as a measure of how much the future is valued relative to the present:
if it is very close to 1, then the future is valued almost as much as the present. If it
is close to 0, then only immediate rewards matter. Of course, this impacts the
optimal policy tremendously: if you value the future, you may be willing to put


optimal policy tremendously: if you value the future, you may be willing to put
up with a lot of immediate pain for the prospect of eventual rewards, while if you
don’t value the future, you will just grab any immediate reward you can find,
never investing in the future.
4. To measure the performance of a Reinforcement Learning agent, you can simply
sum up the rewards it gets. In a simulated environment, you can run many epi‐
sodes and look at the total rewards it gets on average (and possibly look at the
min, max, standard deviation, and so on).
5. The credit assignment problem is the fact that when a Reinforcement Learning
agent receives a reward, it has no direct way of knowing which of its previous
actions contributed to this reward. It typically occurs when there is a large delay
between an action and the resulting rewards (e.g., during a game of Atari’s Pong,
there may be a few dozen time steps between the moment the agent hits the ball


there may be a few dozen time steps between the moment the agent hits the ball
and the moment it wins the point). One way to alleviate it is to provide the agent
with shorter-term rewards, when possible. This usually requires prior knowledge
Exercise Solutions
|
495
Download from finelybook www.finelybook.com


about the task. For example, if we want to build an agent that will learn to play
chess, instead of giving it a reward only when it wins the game, we could give it a
reward every time it captures one of the opponent’s pieces.
6. An agent can often remain in the same region of its environment for a while, so
all of its experiences will be very similar for that period of time. This can intro‐
duce some bias in the learning algorithm. It may tune its policy for this region of
the environment, but it will not perform well as soon as it moves out of this
region. To solve this problem, you can use a replay memory; instead of using
only the most immediate experiences for learning, the agent will learn based on a
buffer of its past experiences, recent and not so recent (perhaps this is why we
dream at night: to replay our experiences of the day and better learn from them?).
7. An off-policy RL algorithm learns the value of the optimal policy (i.e., the sum of


7. An off-policy RL algorithm learns the value of the optimal policy (i.e., the sum of
discounted rewards that can be expected for each state if the agent acts opti‐
mally), independently of how the agent actually acts. Q-Learning is a good exam‐
ple of such an algorithm. In contrast, an on-policy algorithm learns the value of
the policy that the agent actually executes, including both exploration and exploi‐
tation.
For the solutions to exercises 8, 9, and 10, please see the Jupyter notebooks available
at https://github.com/ageron/handson-ml.
496
|
Appendix A: Exercise Solutions
Download from finelybook www.finelybook.com


APPENDIX B
Machine Learning Project Checklist
This checklist can guide you through your Machine Learning projects. There are
eight main steps:
1. Frame the problem and look at the big picture.
2. Get the data.
3. Explore the data to gain insights.
4. Prepare the data to better expose the underlying data patterns to Machine Learn‐
ing algorithms.
5. Explore many different models and short-list the best ones.
6. Fine-tune your models and combine them into a great solution.
7. Present your solution.
8. Launch, monitor, and maintain your system.
Obviously, you should feel free to adapt this checklist to your needs.
Frame the Problem and Look at the Big Picture
1. Define the objective in business terms.
2. How will your solution be used?
3. What are the current solutions/workarounds (if any)?
4. How should you frame this problem (supervised/unsupervised, online/offline,
etc.)?
5. How should performance be measured?
6. Is the performance measure aligned with the business objective?
497


6. Is the performance measure aligned with the business objective?
497
Download from finelybook www.finelybook.com


7. What would be the minimum performance needed to reach the business objec‐
tive?
8. What are comparable problems? Can you reuse experience or tools?
9. Is human expertise available?
10. How would you solve the problem manually?
11. List the assumptions you (or others) have made so far.
12. Verify assumptions if possible.
Get the Data
Note: automate as much as possible so you can easily get fresh data.
1. List the data you need and how much you need.
2. Find and document where you can get that data.
3. Check how much space it will take.
4. Check legal obligations, and get authorization if necessary.
5. Get access authorizations.
6. Create a workspace (with enough storage space).
7. Get the data.
8. Convert the data to a format you can easily manipulate (without changing the
data itself).
9. Ensure sensitive information is deleted or protected (e.g., anonymized).
10. Check the size and type of data (time series, sample, geographical, etc.).


10. Check the size and type of data (time series, sample, geographical, etc.).
11. Sample a test set, put it aside, and never look at it (no data snooping!).
Explore the Data
Note: try to get insights from a field expert for these steps.
1. Create a copy of the data for exploration (sampling it down to a manageable size
if necessary).
2. Create a Jupyter notebook to keep a record of your data exploration.
3. Study each attribute and its characteristics:
• Name
• Type (categorical, int/float, bounded/unbounded, text, structured, etc.)
498
|
Appendix B: Machine Learning Project Checklist
Download from finelybook www.finelybook.com


• % of missing values
• Noisiness and type of noise (stochastic, outliers, rounding errors, etc.)
• Possibly useful for the task?
• Type of distribution (Gaussian, uniform, logarithmic, etc.)
4. For supervised learning tasks, identify the target attribute(s).
5. Visualize the data.
6. Study the correlations between attributes.
7. Study how you would solve the problem manually.
8. Identify the promising transformations you may want to apply.
9. Identify extra data that would be useful (go back to “Get the Data” on page 498).
10. Document what you have learned.
Prepare the Data
Notes:
• Work on copies of the data (keep the original dataset intact).
• Write functions for all data transformations you apply, for five reasons:
— So you can easily prepare the data the next time you get a fresh dataset
— So you can apply these transformations in future projects
— To clean and prepare the test set
— To clean and prepare new data instances once your solution is live


— To clean and prepare new data instances once your solution is live
— To make it easy to treat your preparation choices as hyperparameters
1. Data cleaning:
• Fix or remove outliers (optional).
• Fill in missing values (e.g., with zero, mean, median…) or drop their rows (or
columns).
2. Feature selection (optional):
• Drop the attributes that provide no useful information for the task.
3. Feature engineering, where appropriate:
• Discretize continuous features.
Machine Learning Project Checklist
|
499
Download from finelybook www.finelybook.com


• Decompose features (e.g., categorical, date/time, etc.).
• Add promising transformations of features (e.g., log(x), sqrt(x), x^2, etc.).
• Aggregate features into promising new features.
4. Feature scaling: standardize or normalize features.
Short-List Promising Models
Notes:
• If the data is huge, you may want to sample smaller training sets so you can train
many different models in a reasonable time (be aware that this penalizes complex
models such as large neural nets or Random Forests).
• Once again, try to automate these steps as much as possible.
1. Train many quick and dirty models from different categories (e.g., linear, naive
Bayes, SVM, Random Forests, neural net, etc.) using standard parameters.
2. Measure and compare their performance.
• For each model, use N-fold cross-validation and compute the mean and stan‐
dard deviation of the performance measure on the N folds.
3. Analyze the most significant variables for each algorithm.


3. Analyze the most significant variables for each algorithm.
4. Analyze the types of errors the models make.
• What data would a human have used to avoid these errors?
5. Have a quick round of feature selection and engineering.
6. Have one or two more quick iterations of the five previous steps.
7. Short-list the top three to five most promising models, preferring models that
make different types of errors.
Fine-Tune the System
Notes:
• You will want to use as much data as possible for this step, especially as you move
toward the end of fine-tuning.
• As always automate what you can.
500
|
Appendix B: Machine Learning Project Checklist
Download from finelybook www.finelybook.com


1 “Practical Bayesian Optimization of Machine Learning Algorithms,” J. Snoek, H. Larochelle, R. Adams (2012).
1. Fine-tune the hyperparameters using cross-validation.
• Treat your data transformation choices as hyperparameters, especially when
you are not sure about them (e.g., should I replace missing values with zero or
with the median value? Or just drop the rows?).
• Unless there are very few hyperparameter values to explore, prefer random
search over grid search. If training is very long, you may prefer a Bayesian
optimization approach (e.g., using Gaussian process priors, as described by
Jasper Snoek, Hugo Larochelle, and Ryan Adams).1
2. Try Ensemble methods. Combining your best models will often perform better
than running them individually.
3. Once you are confident about your final model, measure its performance on the
test set to estimate the generalization error.
Don’t tweak your model after measuring the generalization error:


Don’t tweak your model after measuring the generalization error:
you would just start overfitting the test set.
Present Your Solution
1. Document what you have done.
2. Create a nice presentation.
• Make sure you highlight the big picture first.
3. Explain why your solution achieves the business objective.
4. Don’t forget to present interesting points you noticed along the way.
• Describe what worked and what did not.
• List your assumptions and your system’s limitations.
5. Ensure your key findings are communicated through beautiful visualizations or
easy-to-remember statements (e.g., “the median income is the number-one pre‐
dictor of housing prices”).
Machine Learning Project Checklist
|
501
Download from finelybook www.finelybook.com


Launch!
1. Get your solution ready for production (plug into production data inputs, write
unit tests, etc.).
2. Write monitoring code to check your system’s live performance at regular inter‐
vals and trigger alerts when it drops.
• Beware of slow degradation too: models tend to “rot” as data evolves.
• Measuring performance may require a human pipeline (e.g., via a crowdsourc‐
ing service).
• Also monitor your inputs’ quality (e.g., a malfunctioning sensor sending ran‐
dom values, or another team’s output becoming stale). This is particularly
important for online learning systems.
3. Retrain your models on a regular basis on fresh data (automate as much as possi‐
ble).
502
|
Appendix B: Machine Learning Project Checklist
Download from finelybook www.finelybook.com


APPENDIX C
SVM Dual Problem
To understand duality, you first need to understand the Lagrange multipliers method.
The general idea is to transform a constrained optimization objective into an uncon‐
strained one, by moving the constraints into the objective function. Let’s look at a
simple example. Suppose you want to find the values of x and y that minimize the
function f(x,y) = x2 + 2y, subject to an equality constraint: 3x + 2y + 1 = 0. Using the
Lagrange multipliers method, we start by defining a new function called the Lagran‐
gian (or Lagrange function): g(x, y, α) = f(x, y) – α(3x + 2y + 1). Each constraint (in
this case just one) is subtracted from the original objective, multiplied by a new vari‐
able called a Lagrange multiplier.
Joseph-Louis Lagrange showed that if x, y  is a solution to the constrained optimiza‐
tion problem, then there must exist an α such that x, y, α  is a stationary point of the


tion problem, then there must exist an α such that x, y, α  is a stationary point of the
Lagrangian (a stationary point is a point where all partial derivatives are equal to
zero). In other words, we can compute the partial derivatives of g(x, y, α) with regards
to x, y, and α; we can find the points where these derivatives are all equal to zero; and
the solutions to the constrained optimization problem (if they exist) must be among
these stationary points.
In this example the partial derivatives are:
∂
∂x g x, y, α = 2x −3α
∂
∂y g x, y, α = 2 −2α
∂
∂α g x, y, α = −3x −2y −1
When
all
these
partial
derivatives
are
equal
to
0,
we
find
that
2x −3α = 2 −2α = −3x −2y −1 = 0, from which we can easily find that x = 3
2,
y = −11
4 , and α = 1. This is the only stationary point, and as it respects the con‐
straint, it must be the solution to the constrained optimization problem.
503
Download from finelybook www.finelybook.com


However, this method applies only to equality constraints. Fortunately, under some
regularity conditions (which are respected by the SVM objectives), this method can
be generalized to inequality constraints as well (e.g., 3x + 2y + 1 ≥ 0). The generalized
Lagrangian for the hard margin problem is given by Equation C-1, where the α(i) vari‐
ables are called the Karush–Kuhn–Tucker (KKT) multipliers, and they must be greater
or equal to zero.
Equation C-1. Generalized Lagrangian for the hard margin problem
ℒ�, b, α = 1
2�T · �−∑
i = 1
m
α i t i �T · �i + b −1
with
α i ≥0
for i = 1, 2, ⋯, m
Just like with the Lagrange multipliers method, you can compute the partial deriva‐
tives and locate the stationary points. If there is a solution, it will necessarily be
among the stationary points �, b, α  that respect the KKT conditions:
• Respect the problem’s constraints: t i
�T · �i + b ≥1
for i = 1, 2, ⋯, m,
• Verify α i ≥0
for i = 1, 2, ⋯, m,


�T · �i + b ≥1
for i = 1, 2, ⋯, m,
• Verify α i ≥0
for i = 1, 2, ⋯, m,
• Either α i = 0 or the ith constraint must be an active constraint, meaning it must
hold by equality: t i
�T · �i + b = 1. This condition is called the complemen‐
tary slackness condition. It implies that either α i = 0 or the ith instance lies on the
boundary (it is a support vector).
Note that the KKT conditions are necessary conditions for a stationary point to be a
solution of the constrained optimization problem. Under some conditions, they are
also sufficient conditions. Luckily, the SVM optimization problem happens to meet
these conditions, so any stationary point that meets the KKT conditions is guaranteed
to be a solution to the constrained optimization problem.
We can compute the partial derivatives of the generalized Lagrangian with regards to
w and b with Equation C-2.
Equation C-2. Partial derivatives of the generalized Lagrangian
∇�ℒ�, b, α = �−∑
i = 1
m
α i t i �i
∂
∂bℒ�, b, α = −∑
i = 1
m
α i t i


∇�ℒ�, b, α = �−∑
i = 1
m
α i t i �i
∂
∂bℒ�, b, α = −∑
i = 1
m
α i t i
504
|
Appendix C: SVM Dual Problem
Download from finelybook www.finelybook.com


When these partial derivatives are equal to 0, we have Equation C-3.
Equation C-3. Properties of the stationary points
�= ∑
i = 1
m
α i t i �i
∑
i = 1
m
α i t i = 0
If we plug these results into the definition of the generalized Lagrangian, some terms
disappear and we find Equation C-4.
Equation C-4. Dual form of the SVM problem
ℒ�, b, α = 1
2 ∑
i = 1
m
∑
j = 1
m
α i α j t i t j �i T · �j
−
∑
i = 1
m
α i
with
α i ≥0
for i = 1, 2, ⋯, m
The goal is now to find the vector α that minimizes this function, with α i ≥0 for all
instances. This constrained optimization problem is the dual problem we were look‐
ing for.
Once you find the optimal α, you can compute � using the first line of Equation C-3.
To compute b, you can use the fact that a support vector verifies t(i)(wT · x(i) + b) = 1,
so if the kth instance is a support vector (i.e., αk > 0), you can use it to compute
b = 1 −t k �T · �k . However, it is often prefered to compute the average over all


b = 1 −t k �T · �k . However, it is often prefered to compute the average over all
support vectors to get a more stable and precise value, as in Equation C-5.
Equation C-5. Bias term estimation using the dual form
b = 1
ns ∑
i = 1
α i > 0
m
1 −t i �T · �i
SVM Dual Problem
|
505
Download from finelybook www.finelybook.com


Download from finelybook www.finelybook.com


APPENDIX D
Autodiff
This appendix explains how TensorFlow’s autodiff feature works, and how it com‐
pares to other solutions.
Suppose you define a function f(x,y) = x2y + y + 2, and you need its partial derivatives
∂f
∂x and ∂f
∂y, typically to perform Gradient Descent (or some other optimization algo‐
rithm). Your main options are manual differentiation, symbolic differentiation,
numerical differentiation, forward-mode autodiff, and finally reverse-mode autodiff.
TensorFlow implements this last option. Let’s go through each of these options.
Manual Differentiation
The first approach is to pick up a pencil and a piece of paper and use your calculus
knowledge to derive the partial derivatives manually. For the function f(x,y) just
defined, it is not too hard; you just need to use five rules:
• The derivative of a constant is 0.
• The derivative of λx is λ (where λ is a constant).
• The derivative of xλ is λxλ – 1, so the derivative of x2 is 2x.


• The derivative of xλ is λxλ – 1, so the derivative of x2 is 2x.
• The derivative of a sum of functions is the sum of these functions’ derivatives.
• The derivative of λ times a function is λ times its derivative.
507
Download from finelybook www.finelybook.com


From these rules, you can derive Equation D-1:
Equation D-1. Partial derivatives of f(x,y)
∂f
∂x =
∂x2y
∂x
+ ∂y
∂x + ∂2
∂x = y
∂x2
∂x
+ 0 + 0 = 2xy
∂f
∂y =
∂x2y
∂y
+ ∂y
∂y + ∂2
∂y = x2 + 1 + 0 = x2 + 1
This approach can become very tedious for more complex functions, and you run the
risk of making mistakes. The good news is that deriving the mathematical equations
for the partial derivatives like we just did can be automated, through a process called
symbolic differentiation.
Symbolic Differentiation
Figure D-1 shows how symbolic differentiation works on an even simpler function,
g(x,y) = 5 + xy. The graph for that function is represented on the left. After symbolic
differentiation, we get the graph on the right, which represents the partial derivative
∂g
∂x = 0 + 0 × x + y × 1 = y (we could similarly obtain the partial derivative with
regards to y).
Figure D-1. Symbolic differentiation
The algorithm starts by getting the partial derivative of the leaf nodes. The constant


The algorithm starts by getting the partial derivative of the leaf nodes. The constant
node (5) returns the constant 0, since the derivative of a constant is always 0. The
508
|
Appendix D: Autodiff
Download from finelybook www.finelybook.com


variable x returns the constant 1 since ∂x
∂x = 1, and the variable y returns the constant
0 since ∂y
∂x = 0 (if we were looking for the partial derivative with regards to y, it would
be the reverse).
Now we have all we need to move up the graph to the multiplication node in function
g. Calculus tells us that the derivative of the product of two functions u and v is
∂u × v
∂x
= ∂v
∂x × u + ∂u
∂x × u. We can therefore construct a large part of the graph on the
right, representing 0 × x + y × 1.
Finally, we can go up to the addition node in function g. As mentioned, the derivative
of a sum of functions is the sum of these functions’ derivatives. So we just need to
create an addition node and connect it to the parts of the graph we have already com‐
puted. We get the correct partial derivative: ∂g
∂x = 0 + 0 × x + y × 1 .
However, it can be simplified (a lot). A few trivial pruning steps can be applied to this


However, it can be simplified (a lot). A few trivial pruning steps can be applied to this
graph to get rid of all unnecessary operations, and we get a much smaller graph with
just one node: ∂g
∂x = y.
In this case, simplification is fairly easy, but for a more complex function, symbolic
differentiation can produce a huge graph that may be tough to simplify and lead to
suboptimal performance. Most importantly, symbolic differentiation cannot deal with
functions defined with arbitrary code—for example, the following function discussed
in Chapter 9:
def my_func(a, b):
    z = 0
    for i in range(100):
        z = a * np.cos(z + i) + z * np.sin(b - i)
    return z
Numerical Differentiation
The simplest solution is to compute an approximation of the derivatives, numerically.
Recall that the derivative h′(x0) of a function h(x) at a point x0 is the slope of the func‐
tion at that point, or more precisely Equation D-2.
Equation D-2. Derivative of a function h(x) at point x0
h′ x =
lim
x
x0


Equation D-2. Derivative of a function h(x) at point x0
h′ x =
lim
x
x0
h x −h x0
x −x0
=
lim
�
0
h x0 + �−h x0
�
Autodiff
|
509
Download from finelybook www.finelybook.com


So if we want to calculate the partial derivative of f(x,y) with regards to x, at x = 3 and
y = 4, we can simply compute f(3 + ϵ, 4) – f(3, 4) and divide the result by ϵ, using a
very small value for ϵ. That’s exactly what the following code does:
def f(x, y):
    return x**2*y + y + 2
def derivative(f, x, y, x_eps, y_eps):
    return (f(x + x_eps, y + y_eps) - f(x, y)) / (x_eps + y_eps)
df_dx = derivative(f, 3, 4, 0.00001, 0)
df_dy = derivative(f, 3, 4, 0, 0.00001)
Unfortunately, the result is imprecise (and it gets worse for more complex functions).
The correct results are respectively 24 and 10, but instead we get:
>>> print(df_dx)
24.000039999805264
>>> print(df_dy)
10.000000000331966
Notice that to compute both partial derivatives, we have to call f() at least three times
(we called it four times in the preceding code, but it could be optimized). If there
were 1,000 parameters, we would need to call f() at least 1,001 times. When you are


were 1,000 parameters, we would need to call f() at least 1,001 times. When you are
dealing with large neural networks, this makes numerical differentiation way too
inefficient.
However, numerical differentiation is so simple to implement that it is a great tool to
check that the other methods are implemented correctly. For example, if it disagrees
with your manually derived function, then your function probably contains a mis‐
take.
Forward-Mode Autodiff
Forward-mode autodiff is neither numerical differentiation nor symbolic differentia‐
tion, but in some ways it is their love child. It relies on dual numbers, which are
(weird but fascinating) numbers of the form a + bϵ where a and b are real numbers
and ϵ is an infinitesimal number such that ϵ2 = 0 (but ϵ ≠ 0). You can think of the
dual number 42 + 24ϵ as something akin to 42.0000⋯000024 with an infinite num‐
ber of 0s (but of course this is simplified just to give you some idea of what dual num‐


ber of 0s (but of course this is simplified just to give you some idea of what dual num‐
bers are). A dual number is represented in memory as a pair of floats. For example, 42
+ 24ϵ is represented by the pair (42.0, 24.0).
510
|
Appendix D: Autodiff
Download from finelybook www.finelybook.com


Dual numbers can be added, multiplied, and so on, as shown in Equation D-3.
Equation D-3. A few operations with dual numbers
λ a + b�= λa + λb�
a + b�+ c + d�= a + c + b + d �
a + b�× c + d�= ac + ad + bc �+ bd �2 = ac + ad + bc �
Most importantly, it can be shown that h(a + bϵ) = h(a) + b × h′(a)ϵ, so computing
h(a + ϵ) gives you both h(a) and the derivative h′(a) in just one shot. Figure D-2
shows how forward-mode autodiff computes the partial derivative of f(x,y) with
regards to x at x = 3 and y = 4. All we need to do is compute f(3 + ϵ, 4); this will
output a dual number whose first component is equal to f(3, 4) and whose second
component is equal to ∂f
∂x 3, 4 .
Figure D-2. Forward-mode autodiff
Autodiff
|
511
Download from finelybook www.finelybook.com


To compute ∂f
∂y 3, 4  we would have to go through the graph again, but this time with
x = 3 and y = 4 + ϵ.
So forward-mode autodiff is much more accurate than numerical differentiation, but
it suffers from the same major flaw: if there were 1,000 parameters, it would require
1,000 passes through the graph to compute all the partial derivatives. This is where
reverse-mode autodiff shines: it can compute all of them in just two passes through
the graph.
Reverse-Mode Autodiff
Reverse-mode autodiff is the solution implemented by TensorFlow. It first goes
through the graph in the forward direction (i.e., from the inputs to the output) to
compute the value of each node. Then it does a second pass, this time in the reverse
direction (i.e., from the output to the inputs) to compute all the partial derivatives.
Figure D-3 represents the second pass. During the first pass, all the node values were
computed, starting from x = 3 and y = 4. You can see those values at the bottom right


computed, starting from x = 3 and y = 4. You can see those values at the bottom right
of each node (e.g., x × x = 9). The nodes are labeled n1 to n7 for clarity. The output
node is n7: f(3,4) = n7 = 42.
Figure D-3. Reverse-mode autodiff
512
|
Appendix D: Autodiff
Download from finelybook www.finelybook.com


The idea is to gradually go down the graph, computing the partial derivative of f(x,y)
with regards to each consecutive node, until we reach the variable nodes. For this,
reverse-mode autodiff relies heavily on the chain rule, shown in Equation D-4.
Equation D-4. Chain rule
∂f
∂x = ∂f
∂ni
×
∂ni
∂x
Since n7 is the output node, f = n7 so trivially ∂f
∂n7 = 1.
Let’s continue down the graph to n5: how much does f vary when n5 varies? The
answer is ∂f
∂n5 = ∂f
∂n7 ×
∂n7
∂n5. We already know that ∂f
∂n7 = 1, so all we need is
∂n7
∂n5. Since
n7 simply performs the sum n5 + n6, we find that
∂n7
∂n5 = 1, so ∂f
∂n5 = 1 × 1 = 1.
Now we can proceed to node n4: how much does f vary when n4 varies? The answer is
∂f
∂n4 = ∂f
∂n5 ×
∂n5
∂n4. Since n5 = n4 × n2, we find that
∂n5
∂n4 = n2, so ∂f
∂n4 = 1 × n2 = 4.
The process continues until we reach the bottom of the graph. At that point we will
have calculated all the partial derivatives of f(x,y) at the point x = 3 and y = 4. In this


have calculated all the partial derivatives of f(x,y) at the point x = 3 and y = 4. In this
example, we find ∂f
∂x = 24 and ∂f
∂y = 10. Sounds about right!
Reverse-mode autodiff is a very powerful and accurate technique, especially when
there are many inputs and few outputs, since it requires only one forward pass plus
one reverse pass per output to compute all the partial derivatives for all outputs with
regards to all the inputs. Most importantly, it can deal with functions defined by arbi‐
trary code. It can also handle functions that are not entirely differentiable, as long as
you ask it to compute the partial derivatives at points that are differentiable.
If you implement a new type of operation in TensorFlow and you
want to make it compatible with autodiff, then you need to provide
a function that builds a subgraph to compute its partial derivatives
with regards to its inputs. For example, suppose you implement a


with regards to its inputs. For example, suppose you implement a
function that computes the square of its input f(x) = x2. In that case
you would need to provide the corresponding derivative function f′
(x) = 2x. Note that this function does not compute a numerical
result, but instead builds a subgraph that will (later) compute the
result. This is very useful because it means that you can compute
gradients of gradients (to compute second-order derivatives, or
even higher-order derivatives).
Autodiff
|
513
Download from finelybook www.finelybook.com


Download from finelybook www.finelybook.com


APPENDIX E
Other Popular ANN Architectures
In this appendix we will give a quick overview of a few historically important neural
network architectures that are much less used today than deep Multi-Layer Percep‐
trons (Chapter 10), convolutional neural networks (Chapter 13), recurrent neural
networks (Chapter 14), or autoencoders (Chapter 15). They are often mentioned in
the literature, and some are still used in many applications, so it is worth knowing
about them. Moreover, we will discuss deep belief nets (DBNs), which were the state of
the art in Deep Learning until the early 2010s. They are still the subject of very active
research, so they may well come back with a vengeance in the near future.
Hopfield Networks
Hopfield networks were first introduced by W. A. Little in 1974, then popularized by J.
Hopfield in 1982. They are associative memory networks: you first teach them some
patterns, and then when they see a new pattern they (hopefully) output the closest


patterns, and then when they see a new pattern they (hopefully) output the closest
learned pattern. This has made them useful in particular for character recognition
before they were outperformed by other approaches. You first train the network by
showing it examples of character images (each binary pixel maps to one neuron), and
then when you show it a new character image, after a few iterations it outputs the
closest learned character.
They are fully connected graphs (see Figure E-1); that is, every neuron is connected
to every other neuron. Note that on the diagram the images are 6 × 6 pixels, so the
neural network on the left should contain 36 neurons (and 648 connections), but for
visual clarity a much smaller network is represented.
515
Download from finelybook www.finelybook.com


Figure E-1. Hopfield network
The training algorithm works by using Hebb’s rule: for each training image, the
weight between two neurons is increased if the corresponding pixels are both on or
both off, but decreased if one pixel is on and the other is off.
To show a new image to the network, you just activate the neurons that correspond to
active pixels. The network then computes the output of every neuron, and this gives
you a new image. You can then take this new image and repeat the whole process.
After a while, the network reaches a stable state. Generally, this corresponds to the
training image that most resembles the input image.
A so-called energy function is associated with Hopfield nets. At each iteration, the
energy decreases, so the network is guaranteed to eventually stabilize to a low-energy
state. The training algorithm tweaks the weights in a way that decreases the energy
level of the training patterns, so the network is likely to stabilize in one of these low-


level of the training patterns, so the network is likely to stabilize in one of these low-
energy configurations. Unfortunately, some patterns that were not in the training set
also end up with low energy, so the network sometimes stabilizes in a configuration
that was not learned. These are called spurious patterns.
Another major flaw with Hopfield nets is that they don’t scale very well—their mem‐
ory capacity is roughly equal to 14% of the number of neurons. For example, to clas‐
sify 28 × 28 images, you would need a Hopfield net with 784 fully connected neurons
and 306,936 weights. Such a network would only be able to learn about 110 different
characters (14% of 784). That’s a lot of parameters for such a small memory.
Boltzmann Machines
Boltzmann machines were invented in 1985 by Geoffrey Hinton and Terrence Sejnow‐
ski. Just like Hopfield nets, they are fully connected ANNs, but they are based on sto‐
516
|
Appendix E: Other Popular ANN Architectures


516
|
Appendix E: Other Popular ANN Architectures
Download from finelybook www.finelybook.com


chastic neurons: instead of using a deterministic step function to decide what value to
output, these neurons output 1 with some probability, and 0 otherwise. The probabil‐
ity function that these ANNs use is based on the Boltzmann distribution (used in
statistical mechanics) hence their name. Equation E-1 gives the probability that a par‐
ticular neuron will output a 1.
Equation E-1. Probability that the ith neuron will output 1
p si
next step = 1 = σ
∑j = 1
N
wi, jsj + bi
T
• sj is the jth neuron’s state (0 or 1).
• wi,j is the connection weight between the ith and jth neurons. Note that wi,i = 0.
• bi is the ith neuron’s bias term. We can implement this term by adding a bias neu‐
ron to the network.
• N is the number of neurons in the network.
• T is a number called the network’s temperature; the higher the temperature, the
more random the output is (i.e., the more the probability approaches 50%).
• σ is the logistic function.


• σ is the logistic function.
Neurons in Boltzmann machines are separated into two groups: visible units and hid‐
den units (see Figure E-2). All neurons work in the same stochastic way, but the visi‐
ble units are the ones that receive the inputs and from which outputs are read.
Figure E-2. Boltzmann machine
Other Popular ANN Architectures
|
517
Download from finelybook www.finelybook.com


Because of its stochastic nature, a Boltzmann machine will never stabilize into a fixed
configuration, but instead it will keep switching between many configurations. If it is
left running for a sufficiently long time, the probability of observing a particular con‐
figuration will only be a function of the connection weights and bias terms, not of the
original configuration (similarly, after you shuffle a deck of cards for long enough, the
configuration of the deck does not depend on the initial state). When the network
reaches this state where the original configuration is “forgotten,” it is said to be in
thermal equilibrium (although its configuration keeps changing all the time). By set‐
ting the network parameters appropriately, letting the network reach thermal equili‐
brium, and then observing its state, we can simulate a wide range of probability
distributions. This is called a generative model.
Training a Boltzmann machine means finding the parameters that will make the net‐


Training a Boltzmann machine means finding the parameters that will make the net‐
work approximate the training set’s probability distribution. For example, if there are
three visible neurons and the training set contains 75% (0, 1, 1) triplets, 10% (0, 0, 1)
triplets, and 15% (1, 1, 1) triplets, then after training a Boltzmann machine, you could
use it to generate random binary triplets with about the same probability distribu‐
tion. For example, about 75% of the time it would output the (0, 1, 1) triplet.
Such a generative model can be used in a variety of ways. For example, if it is trained
on images, and you provide an incomplete or noisy image to the network, it will
automatically “repair” the image in a reasonable way. You can also use a generative
model for classification. Just add a few visible neurons to encode the training image’s
class (e.g., add 10 visible neurons and turn on only the fifth neuron when the training


class (e.g., add 10 visible neurons and turn on only the fifth neuron when the training
image represents a 5). Then, when given a new image, the network will automatically
turn on the appropriate visible neurons, indicating the image’s class (e.g., it will turn
on the fifth visible neuron if the image represents a 5).
Unfortunately, there is no efficient technique to train Boltzmann machines. However,
fairly efficient algorithms have been developed to train restricted Boltzmann machines
(RBM).
Restricted Boltzmann Machines
An RBM is simply a Boltzmann machine in which there are no connections between
visible units or between hidden units, only between visible and hidden units. For
example, Figure E-3 represents an RBM with three visible units and four hidden
units.
518
|
Appendix E: Other Popular ANN Architectures
Download from finelybook www.finelybook.com


1 “On Contrastive Divergence Learning,” M. Á. Carreira-Perpiñán and G. Hinton (2005).
Figure E-3. Restricted Boltzmann machine
A very efficient training algorithm, called Contrastive Divergence, was introduced in
2005 by Miguel Á. Carreira-Perpiñán and Geoffrey Hinton.1 Here is how it works: for
each training instance x, the algorithm starts by feeding it to the network by setting
the state of the visible units to x1, x2, ⋯, xn. Then you compute the state of the hidden
units by applying the stochastic equation described before (Equation E-1). This gives
you a hidden vector h (where hi is equal to the state of the ith unit). Next you compute
the state of the visible units, by applying the same stochastic equation. This gives you
a vector �˙. Then once again you compute the state of the hidden units, which gives
you a vector �˙. Now you can update each connection weight by applying the rule in
Equation E-2.
Equation E-2. Contrastive divergence weight update
wi, j


Equation E-2.
Equation E-2. Contrastive divergence weight update
wi, j
next step = wi, j + η ��T −�˙�˙T
The great benefit of this algorithm it that it does not require waiting for the network
to reach thermal equilibrium: it just goes forward, backward, and forward again, and
that’s it. This makes it incomparably more efficient than previous algorithms, and it
was a key ingredient to the first success of Deep Learning based on multiple stacked
RBMs.
Deep Belief Nets
Several layers of RBMs can be stacked; the hidden units of the first-level RBM serves
as the visible units for the second-layer RBM, and so on. Such an RBM stack is called
a deep belief net (DBN).
Yee-Whye Teh, one of Geoffrey Hinton’s students, observed that it was possible to
train DBNs one layer at a time using Contrastive Divergence, starting with the lower
Other Popular ANN Architectures
|
519
Download from finelybook www.finelybook.com


2 “A Fast Learning Algorithm for Deep Belief Nets,” G. Hinton, S. Osindero, Y. Teh (2006).
layers and then gradually moving up to the top layers. This led to the groundbreaking
article that kickstarted the Deep Learning tsunami in 2006.2
Just like RBMs, DBNs learn to reproduce the probability distribution of their inputs,
without any supervision. However, they are much better at it, for the same reason that
deep neural networks are more powerful than shallow ones: real-world data is often
organized in hierarchical patterns, and DBNs take advantage of that. Their lower lay‐
ers learn low-level features in the input data, while higher layers learn high-level fea‐
tures.
Just like RBMs, DBNs are fundamentally unsupervised, but you can also train them
in a supervised manner by adding some visible units to represent the labels. More‐
over, one great feature of DBNs is that they can be trained in a semisupervised fash‐


over, one great feature of DBNs is that they can be trained in a semisupervised fash‐
ion. Figure E-4 represents such a DBN configured for semisupervised learning.
Figure E-4. A deep belief network configured for semisupervised learning
First, the RBM 1 is trained without supervision. It learns low-level features in the
training data. Then RBM 2 is trained with RBM 1’s hidden units as inputs, again
without supervision: it learns higher-level features (note that RBM 2’s hidden units
include only the three rightmost units, not the label units). Several more RBMs could
be stacked this way, but you get the idea. So far, training was 100% unsupervised.
520
|
Appendix E: Other Popular ANN Architectures
Download from finelybook www.finelybook.com


3 See this video by Geoffrey Hinton for more details and a demo: http://goo.gl/7Z5QiS.
Lastly, RBM 3 is trained using both RBM 2’s hidden units as inputs, as well as extra
visible units used to represent the target labels (e.g., a one-hot vector representing the
instance class). It learns to associate high-level features with training labels. This is
the supervised step.
At the end of training, if you feed RBM 1 a new instance, the signal will propagate up
to RBM 2, then up to the top of RBM 3, and then back down to the label units; hope‐
fully, the appropriate label will light up. This is how a DBN can be used for classifica‐
tion.
One great benefit of this semisupervised approach is that you don’t need much
labeled training data. If the unsupervised RBMs do a good enough job, then only a
small amount of labeled training instances per class will be necessary. Similarly, a
baby learns to recognize objects without supervision, so when you point to a chair


baby learns to recognize objects without supervision, so when you point to a chair
and say “chair,” the baby can associate the word “chair” with the class of objects it has
already learned to recognize on its own. You don’t need to point to every single chair
and say “chair”; only a few examples will suffice (just enough so the baby can be sure
that you are indeed referring to the chair, not to its color or one of the chair’s parts).
Quite amazingly, DBNs can also work in reverse. If you activate one of the label units,
the signal will propagate up to the hidden units of RBM 3, then down to RBM 2, and
then RBM 1, and a new instance will be output by the visible units of RBM 1. This
new instance will usually look like a regular instance of the class whose label unit you
activated. This generative capability of DBNs is quite powerful. For example, it has
been used to automatically generate captions for images, and vice versa: first a DBN is


been used to automatically generate captions for images, and vice versa: first a DBN is
trained (without supervision) to learn features in images, and another DBN is trained
(again without supervision) to learn features in sets of captions (e.g., “car” often
comes with “automobile”). Then an RBM is stacked on top of both DBNs and trained
with a set of images along with their captions; it learns to associate high-level features
in images with high-level features in captions. Next, if you feed the image DBN an
image of a car, the signal will propagate through the network, up to the top-level
RBM, and back down to the bottom of the caption DBN, producing a caption. Due to
the stochastic nature of RBMs and DBNs, the caption will keep changing randomly,
but it will generally be appropriate for the image. If you generate a few hundred cap‐
tions, the most frequently generated ones will likely be a good description of the
image.3
Self-Organizing Maps


image.3
Self-Organizing Maps
Self-organizing maps (SOM) are quite different from all the other types of neural net‐
works we have discussed so far. They are used to produce a low-dimensional repre‐
Other Popular ANN Architectures
|
521
Download from finelybook www.finelybook.com


sentation of a high-dimensional dataset, generally for visualization, clustering, or
classification. The neurons are spread across a map (typically 2D for visualization,
but it can be any number of dimensions you want), as shown in Figure E-5, and each
neuron has a weighted connection to every input (note that the diagram shows just
two inputs, but there are typically a very large number, since the whole point of
SOMs is to reduce dimensionality).
Figure E-5. Self-organizing maps
Once the network is trained, you can feed it a new instance and this will activate only
one neuron (i.e., hence one point on the map): the neuron whose weight vector is
closest to the input vector. In general, instances that are nearby in the original input
space will activate neurons that are nearby on the map. This makes SOMs useful for
visualization (in particular, you can easily identify clusters on the map), but also for
applications like speech recognition. For example, if each instance represents the