7 “Ensembles on Random Patches,” G. Louppe and P. Geurts (2012).
8 “The random subspace method for constructing decision forests,” Tin Kam Ho (1998).
>>> bag_clf.fit(X_train, y_train)
>>> bag_clf.oob_score_
0.93066666666666664
According to this oob evaluation, this BaggingClassifier is likely to achieve about
93.1% accuracy on the test set. Let’s verify this:
>>> from sklearn.metrics import accuracy_score
>>> y_pred = bag_clf.predict(X_test)
>>> accuracy_score(y_test, y_pred)
0.93600000000000005
We get 93.6% accuracy on the test set—close enough!
The oob decision function for each training instance is also available through the
oob_decision_function_ variable. In this case (since the base estimator has a pre
dict_proba() method) the decision function returns the class probabilities for each
training instance. For example, the oob evaluation estimates that the second training
instance has a 60.6% probability of belonging to the positive class (and 39.4% of


instance has a 60.6% probability of belonging to the positive class (and 39.4% of
belonging to the positive class):
>>> bag_clf.oob_decision_function_
array([[ 0.        ,  1.        ],
       [ 0.60588235,  0.39411765],
       [ 1.        ,  0.        ],
       ...
       [ 1.        ,  0.        ],
       [ 0.        ,  1.        ],
       [ 0.48958333,  0.51041667]])
Random Patches and Random Subspaces
The BaggingClassifier class supports sampling the features as well. This is con‐
trolled by two hyperparameters: max_features and bootstrap_features. They work
the same way as max_samples and bootstrap, but for feature sampling instead of
instance sampling. Thus, each predictor will be trained on a random subset of the
input features.
This is particularly useful when you are dealing with high-dimensional inputs (such
as images). Sampling both training instances and features is called the Random
Patches method.7 Keeping all training instances (i.e., bootstrap=False and max_sam

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


Patches method.7 Keeping all training instances (i.e., bootstrap=False and max_sam
ples=1.0) but sampling features (i.e., bootstrap_features=True and/or max_fea
tures smaller than 1.0) is called the Random Subspaces method.8
188
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


9 “Random Decision Forests,” T. Ho (1995).
10 The BaggingClassifier class remains useful if you want a bag of something other than Decision Trees.
11 There are a few notable exceptions: splitter is absent (forced to "random"), presort is absent (forced to
False), max_samples is absent (forced to 1.0), and base_estimator is absent (forced to DecisionTreeClassi
fier with the provided hyperparameters).
Sampling features results in even more predictor diversity, trading a bit more bias for
a lower variance.
Random Forests
As we have discussed, a Random Forest9 is an ensemble of Decision Trees, generally
trained via the bagging method (or sometimes pasting), typically with max_samples
set to the size of the training set. Instead of building a BaggingClassifier and pass‐
ing it a DecisionTreeClassifier, you can instead use the RandomForestClassifier
class, which is more convenient and optimized for Decision Trees10 (similarly, there is


class, which is more convenient and optimized for Decision Trees10 (similarly, there is
a RandomForestRegressor class for regression tasks). The following code trains a
Random Forest classifier with 500 trees (each limited to maximum 16 nodes), using
all available CPU cores:
from sklearn.ensemble import RandomForestClassifier
rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16, n_jobs=-1)
rnd_clf.fit(X_train, y_train)
y_pred_rf = rnd_clf.predict(X_test)
With a few exceptions, a RandomForestClassifier has all the hyperparameters of a
DecisionTreeClassifier (to control how trees are grown), plus all the hyperpara‐
meters of a BaggingClassifier to control the ensemble itself.11
The Random Forest algorithm introduces extra randomness when growing trees;
instead of searching for the very best feature when splitting a node (see Chapter 6), it
searches for the best feature among a random subset of features. This results in a


searches for the best feature among a random subset of features. This results in a
greater tree diversity, which (once again) trades a higher bias for a lower variance,
generally yielding an overall better model. The following BaggingClassifier is
roughly equivalent to the previous RandomForestClassifier:
bag_clf = BaggingClassifier(
        DecisionTreeClassifier(splitter="random", max_leaf_nodes=16),
        n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1
    )
Random Forests
|
189
Download from finelybook www.finelybook.com


12 “Extremely randomized trees,” P. Geurts, D. Ernst, L. Wehenkel (2005).
Extra-Trees
When you are growing a tree in a Random Forest, at each node only a random subset
of the features is considered for splitting (as discussed earlier). It is possible to make
trees even more random by also using random thresholds for each feature rather than
searching for the best possible thresholds (like regular Decision Trees do).
A forest of such extremely random trees is simply called an Extremely Randomized
Trees ensemble12 (or Extra-Trees for short). Once again, this trades more bias for a
lower variance. It also makes Extra-Trees much faster to train than regular Random
Forests since finding the best possible threshold for each feature at every node is one
of the most time-consuming tasks of growing a tree.
You can create an Extra-Trees classifier using Scikit-Learn’s ExtraTreesClassifier
class. Its API is identical to the RandomForestClassifier class. Similarly, the Extra


class. Its API is identical to the RandomForestClassifier class. Similarly, the Extra
TreesRegressor class has the same API as the RandomForestRegressor class.
It is hard to tell in advance whether a RandomForestClassifier
will perform better or worse than an ExtraTreesClassifier. Gen‐
erally, the only way to know is to try both and compare them using
cross-validation (and tuning the hyperparameters using grid
search).
Feature Importance
Lastly, if you look at a single Decision Tree, important features are likely to appear
closer to the root of the tree, while unimportant features will often appear closer to
the leaves (or not at all). It is therefore possible to get an estimate of a feature’s impor‐
tance by computing the average depth at which it appears across all trees in the forest.
Scikit-Learn computes this automatically for every feature after training. You can
access the result using the feature_importances_ variable. For example, the follow‐


access the result using the feature_importances_ variable. For example, the follow‐
ing code trains a RandomForestClassifier on the iris dataset (introduced in Chap‐
ter 4) and outputs each feature’s importance. It seems that the most important
features are the petal length (44%) and width (42%), while sepal length and width are
rather unimportant in comparison (11% and 2%, respectively):
>>> from sklearn.datasets import load_iris
>>> iris = load_iris()
>>> rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1)
>>> rnd_clf.fit(iris["data"], iris["target"])
>>> for name, score in zip(iris["feature_names"], rnd_clf.feature_importances_):
>>>     print(name, score)
sepal length (cm) 0.112492250999
190
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


13 “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting,” Yoav Freund,
Robert E. Schapire (1997).
sepal width (cm) 0.0231192882825
petal length (cm) 0.441030464364
petal width (cm) 0.423357996355
Similarly, if you train a Random Forest classifier on the MNIST dataset (introduced
in Chapter 3) and plot each pixel’s importance, you get the image represented in
Figure 7-6.
Figure 7-6. MNIST pixel importance (according to a Random Forest classifier)
Random Forests are very handy to get a quick understanding of what features
actually matter, in particular if you need to perform feature selection.
Boosting
Boosting (originally called hypothesis boosting) refers to any Ensemble method that
can combine several weak learners into a strong learner. The general idea of most
boosting methods is to train predictors sequentially, each trying to correct its prede‐
cessor. There are many boosting methods available, but by far the most popular are


cessor. There are many boosting methods available, but by far the most popular are
AdaBoost13 (short for Adaptive Boosting) and Gradient Boosting. Let’s start with Ada‐
Boost.
Boosting
|
191
Download from finelybook www.finelybook.com


14 This is just for illustrative purposes. SVMs are generally not good base predictors for AdaBoost, because they
are slow and tend to be unstable with AdaBoost.
AdaBoost
One way for a new predictor to correct its predecessor is to pay a bit more attention
to the training instances that the predecessor underfitted. This results in new predic‐
tors focusing more and more on the hard cases. This is the technique used by Ada‐
Boost.
For example, to build an AdaBoost classifier, a first base classifier (such as a Decision
Tree) is trained and used to make predictions on the training set. The relative weight
of misclassified training instances is then increased. A second classifier is trained
using the updated weights and again it makes predictions on the training set, weights
are updated, and so on (see Figure 7-7).
Figure 7-7. AdaBoost sequential training with instance weight updates
Figure 7-8 shows the decision boundaries of five consecutive predictors on the


Figure 7-8 shows the decision boundaries of five consecutive predictors on the
moons dataset (in this example, each predictor is a highly regularized SVM classifier
with an RBF kernel14). The first classifier gets many instances wrong, so their weights
get boosted. The second classifier therefore does a better job on these instances, and
so on. The plot on the right represents the same sequence of predictors except that
the learning rate is halved (i.e., the misclassified instance weights are boosted half as
much at every iteration). As you can see, this sequential learning technique has some
similarities with Gradient Descent, except that instead of tweaking a single predictor’s
192
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


parameters to minimize a cost function, AdaBoost adds predictors to the ensemble,
gradually making it better.
Figure 7-8. Decision boundaries of consecutive predictors
Once all predictors are trained, the ensemble makes predictions very much like bag‐
ging or pasting, except that predictors have different weights depending on their
overall accuracy on the weighted training set.
There is one important drawback to this sequential learning techni‐
que: it cannot be parallelized (or only partially), since each predic‐
tor can only be trained after the previous predictor has been
trained and evaluated. As a result, it does not scale as well as bag‐
ging or pasting.
Let’s take a closer look at the AdaBoost algorithm. Each instance weight w(i) is initially
set to 1
m. A first predictor is trained and its weighted error rate r1 is computed on the
training set; see Equation 7-1.
Equation 7-1. Weighted error rate of the jth predictor
rj =
∑
i = 1
y j
i ≠y i
m
w i
∑
i = 1
m
w i
where y j


rj =
∑
i = 1
y j
i ≠y i
m
w i
∑
i = 1
m
w i
where y j
i is the jth predictor’s prediction for the ith instance.
Boosting
|
193
Download from finelybook www.finelybook.com


15 The original AdaBoost algorithm does not use a learning rate hyperparameter.
The predictor’s weight αj is then computed using Equation 7-2, where η is the learn‐
ing rate hyperparameter (defaults to 1).15 The more accurate the predictor is, the
higher its weight will be. If it is just guessing randomly, then its weight will be close to
zero. However, if it is most often wrong (i.e., less accurate than random guessing),
then its weight will be negative.
Equation 7-2. Predictor weight
αj = η log
1 −rj
rj
Next the instance weights are updated using Equation 7-3: the misclassified instances
are boosted.
Equation 7-3. Weight update rule
for i = 1, 2, ⋯, m
w i
w i
if yj
i = y i
w i exp αj if yj
i ≠y i
Then all the instance weights are normalized (i.e., divided by ∑i = 1
m
w i ).
Finally, a new predictor is trained using the updated weights, and the whole process is
repeated (the new predictor’s weight is computed, the instance weights are updated,


repeated (the new predictor’s weight is computed, the instance weights are updated,
then another predictor is trained, and so on). The algorithm stops when the desired
number of predictors is reached, or when a perfect predictor is found.
To make predictions, AdaBoost simply computes the predictions of all the predictors
and weighs them using the predictor weights αj. The predicted class is the one that
receives the majority of weighted votes (see Equation 7-4).
Equation 7-4. AdaBoost predictions
y �= argmax
k
∑
j = 1
y j �= k
N
αj
where N is the number of predictors.
194
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


16 For more details, see “Multi-Class AdaBoost,” J. Zhu et al. (2006).
17 First introduced in “Arcing the Edge,” L. Breiman (1997).
Scikit-Learn actually uses a multiclass version of AdaBoost called SAMME16 (which
stands for Stagewise Additive Modeling using a Multiclass Exponential loss function).
When there are just two classes, SAMME is equivalent to AdaBoost. Moreover, if the
predictors can estimate class probabilities (i.e., if they have a predict_proba()
method), Scikit-Learn can use a variant of SAMME called SAMME.R (the R stands
for “Real”), which relies on class probabilities rather than predictions and generally
performs better.
The following code trains an AdaBoost classifier based on 200 Decision Stumps using
Scikit-Learn’s AdaBoostClassifier class (as you might expect, there is also an Ada
BoostRegressor class). A Decision Stump is a Decision Tree with max_depth=1—in
other words, a tree composed of a single decision node plus two leaf nodes. This is


other words, a tree composed of a single decision node plus two leaf nodes. This is
the default base estimator for the AdaBoostClassifier class:
from sklearn.ensemble import AdaBoostClassifier
ada_clf = AdaBoostClassifier(
        DecisionTreeClassifier(max_depth=1), n_estimators=200,
        algorithm="SAMME.R", learning_rate=0.5
    )
ada_clf.fit(X_train, y_train)
If your AdaBoost ensemble is overfitting the training set, you can
try reducing the number of estimators or more strongly regulariz‐
ing the base estimator.
Gradient Boosting
Another very popular Boosting algorithm is Gradient Boosting.17 Just like AdaBoost,
Gradient Boosting works by sequentially adding predictors to an ensemble, each one
correcting its predecessor. However, instead of tweaking the instance weights at every
iteration like AdaBoost does, this method tries to fit the new predictor to the residual
errors made by the previous predictor.


errors made by the previous predictor.
Let’s go through a simple regression example using Decision Trees as the base predic‐
tors (of course Gradient Boosting also works great with regression tasks). This is
called Gradient Tree Boosting, or Gradient Boosted Regression Trees (GBRT). First, let’s
fit a DecisionTreeRegressor to the training set (for example, a noisy quadratic train‐
ing set):
Boosting
|
195
Download from finelybook www.finelybook.com


from sklearn.tree import DecisionTreeRegressor
tree_reg1 = DecisionTreeRegressor(max_depth=2)
tree_reg1.fit(X, y)
Now train a second DecisionTreeRegressor on the residual errors made by the first
predictor:
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2)
tree_reg2.fit(X, y2)
Then we train a third regressor on the residual errors made by the second predictor:
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2)
tree_reg3.fit(X, y3)
Now we have an ensemble containing three trees. It can make predictions on a new
instance simply by adding up the predictions of all the trees:
y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))
Figure 7-9 represents the predictions of these three trees in the left column, and the
ensemble’s predictions in the right column. In the first row, the ensemble has just one
tree, so its predictions are exactly the same as the first tree’s predictions. In the second


tree, so its predictions are exactly the same as the first tree’s predictions. In the second
row, a new tree is trained on the residual errors of the first tree. On the right you can
see that the ensemble’s predictions are equal to the sum of the predictions of the first
two trees. Similarly, in the third row another tree is trained on the residual errors of
the second tree. You can see that the ensemble’s predictions gradually get better as
trees are added to the ensemble.
A simpler way to train GBRT ensembles is to use Scikit-Learn’s GradientBoostingRe
gressor class. Much like the RandomForestRegressor class, it has hyperparameters to
control the growth of Decision Trees (e.g., max_depth, min_samples_leaf, and so on),
as well as hyperparameters to control the ensemble training, such as the number of
trees (n_estimators). The following code creates the same ensemble as the previous
one:
from sklearn.ensemble import GradientBoostingRegressor


one:
from sklearn.ensemble import GradientBoostingRegressor
gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0)
gbrt.fit(X, y)
196
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


Figure 7-9. Gradient Boosting
The learning_rate hyperparameter scales the contribution of each tree. If you set it
to a low value, such as 0.1, you will need more trees in the ensemble to fit the train‐
ing set, but the predictions will usually generalize better. This is a regularization tech‐
nique called shrinkage. Figure 7-10 shows two GBRT ensembles trained with a low
learning rate: the one on the left does not have enough trees to fit the training set,
while the one on the right has too many trees and overfits the training set.
Boosting
|
197
Download from finelybook www.finelybook.com


Figure 7-10. GBRT ensembles with not enough predictors (left) and too many (right)
In order to find the optimal number of trees, you can use early stopping (see Chap‐
ter 4). A simple way to implement this is to use the staged_predict() method: it
returns an iterator over the predictions made by the ensemble at each stage of train‐
ing (with one tree, two trees, etc.). The following code trains a GBRT ensemble with
120 trees, then measures the validation error at each stage of training to find the opti‐
mal number of trees, and finally trains another GBRT ensemble using the optimal
number of trees:
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
X_train, X_val, y_train, y_val = train_test_split(X, y)
gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120)
gbrt.fit(X_train, y_train)
errors = [mean_squared_error(y_val, y_pred)
          for y_pred in gbrt.staged_predict(X_val)]


errors = [mean_squared_error(y_val, y_pred)
          for y_pred in gbrt.staged_predict(X_val)]
bst_n_estimators = np.argmin(errors)
gbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators)
gbrt_best.fit(X_train, y_train)
The validation errors are represented on the left of Figure 7-11, and the best model’s
predictions are represented on the right.
198
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


Figure 7-11. Tuning the number of trees using early stopping
It is also possible to implement early stopping by actually stopping training early
(instead of training a large number of trees first and then looking back to find the
optimal number). You can do so by setting warm_start=True, which makes Scikit-
Learn keep existing trees when the fit() method is called, allowing incremental
training. The following code stops training when the validation error does not
improve for five iterations in a row:
gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True)
min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:


error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping
The GradientBoostingRegressor class also supports a subsample hyperparameter,
which specifies the fraction of training instances to be used for training each tree. For
example, if subsample=0.25, then each tree is trained on 25% of the training instan‐
ces, selected randomly. As you can probably guess by now, this trades a higher bias
for a lower variance. It also speeds up training considerably. This technique is called
Stochastic Gradient Boosting.
Boosting
|
199
Download from finelybook www.finelybook.com


18 “Stacked Generalization,” D. Wolpert (1992).
19 Alternatively, it is possible to use out-of-fold predictions. In some contexts this is called stacking, while using a
hold-out set is called blending. However, for many people these terms are synonymous.
It is possible to use Gradient Boosting with other cost functions.
This is controlled by the loss hyperparameter (see Scikit-Learn’s
documentation for more details).
Stacking
The last Ensemble method we will discuss in this chapter is called stacking (short for
stacked generalization).18 It is based on a simple idea: instead of using trivial functions
(such as hard voting) to aggregate the predictions of all predictors in an ensemble,
why don’t we train a model to perform this aggregation? Figure 7-12 shows such an
ensemble performing a regression task on a new instance. Each of the bottom three
predictors predicts a different value (3.1, 2.7, and 2.9), and then the final predictor


predictors predicts a different value (3.1, 2.7, and 2.9), and then the final predictor
(called a blender, or a meta learner) takes these predictions as inputs and makes the
final prediction (3.0).
Figure 7-12. Aggregating predictions using a blending predictor
To train the blender, a common approach is to use a hold-out set.19 Let’s see how it
works. First, the training set is split in two subsets. The first subset is used to train the
predictors in the first layer (see Figure 7-13).
200
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


Figure 7-13. Training the first layer
Next, the first layer predictors are used to make predictions on the second (held-out)
set (see Figure 7-14). This ensures that the predictions are “clean,” since the predictors
never saw these instances during training. Now for each instance in the hold-out set
there are three predicted values. We can create a new training set using these predic‐
ted values as input features (which makes this new training set three-dimensional),
and keeping the target values. The blender is trained on this new training set, so it
learns to predict the target value given the first layer’s predictions.
Figure 7-14. Training the blender
Stacking
|
201
Download from finelybook www.finelybook.com


It is actually possible to train several different blenders this way (e.g., one using Lin‐
ear Regression, another using Random Forest Regression, and so on): we get a whole
layer of blenders. The trick is to split the training set into three subsets: the first one is
used to train the first layer, the second one is used to create the training set used to
train the second layer (using predictions made by the predictors of the first layer),
and the third one is used to create the training set to train the third layer (using pre‐
dictions made by the predictors of the second layer). Once this is done, we can make
a prediction for a new instance by going through each layer sequentially, as shown in
Figure 7-15.
Figure 7-15. Predictions in a multilayer stacking ensemble
Unfortunately, Scikit-Learn does not support stacking directly, but it is not too hard
to roll out your own implementation (see the following exercises). Alternatively, you


to roll out your own implementation (see the following exercises). Alternatively, you
can use an open source implementation such as brew (available at https://github.com/
viisar/brew).
Exercises
1. If you have trained five different models on the exact same training data, and
they all achieve 95% precision, is there any chance that you can combine these
models to get better results? If so, how? If not, why?
2. What is the difference between hard and soft voting classifiers?
202
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


3. Is it possible to speed up training of a bagging ensemble by distributing it across
multiple servers? What about pasting ensembles, boosting ensembles, random
forests, or stacking ensembles?
4. What is the benefit of out-of-bag evaluation?
5. What makes Extra-Trees more random than regular Random Forests? How can
this extra randomness help? Are Extra-Trees slower or faster than regular Ran‐
dom Forests?
6. If your AdaBoost ensemble underfits the training data, what hyperparameters
should you tweak and how?
7. If your Gradient Boosting ensemble overfits the training set, should you increase
or decrease the learning rate?
8. Load the MNIST data (introduced in Chapter 3), and split it into a training set, a
validation set, and a test set (e.g., use the first 40,000 instances for training, the
next 10,000 for validation, and the last 10,000 for testing). Then train various
classifiers, such as a Random Forest classifier, an Extra-Trees classifier, and an


classifiers, such as a Random Forest classifier, an Extra-Trees classifier, and an
SVM. Next, try to combine them into an ensemble that outperforms them all on
the validation set, using a soft or hard voting classifier. Once you have found one,
try it on the test set. How much better does it perform compared to the individ‐
ual classifiers?
9. Run the individual classifiers from the previous exercise to make predictions on
the validation set, and create a new training set with the resulting predictions:
each training instance is a vector containing the set of predictions from all your
classifiers for an image, and the target is the image’s class. Congratulations, you
have just trained a blender, and together with the classifiers they form a stacking
ensemble! Now let’s evaluate the ensemble on the test set. For each image in the
test set, make predictions with all your classifiers, then feed the predictions to the


test set, make predictions with all your classifiers, then feed the predictions to the
blender to get the ensemble’s predictions. How does it compare to the voting clas‐
sifier you trained earlier?
Solutions to these exercises are available in Appendix A.
Exercises
|
203
Download from finelybook www.finelybook.com


Download from finelybook www.finelybook.com


CHAPTER 8
Dimensionality Reduction
Many Machine Learning problems involve thousands or even millions of features for
each training instance. Not only does this make training extremely slow, it can also
make it much harder to find a good solution, as we will see. This problem is often
referred to as the curse of dimensionality.
Fortunately, in real-world problems, it is often possible to reduce the number of fea‐
tures considerably, turning an intractable problem into a tractable one. For example,
consider the MNIST images (introduced in Chapter 3): the pixels on the image bor‐
ders are almost always white, so you could completely drop these pixels from the
training set without losing much information. Figure 7-6 confirms that these pixels
are utterly unimportant for the classification task. Moreover, two neighboring pixels
are often highly correlated: if you merge them into a single pixel (e.g., by taking the
mean of the two pixel intensities), you will not lose much information.


mean of the two pixel intensities), you will not lose much information.
Reducing dimensionality does lose some information (just like
compressing an image to JPEG can degrade its quality), so even
though it will speed up training, it may also make your system per‐
form slightly worse. It also makes your pipelines a bit more com‐
plex and thus harder to maintain. So you should first try to train
your system with the original data before considering using dimen‐
sionality reduction if training is too slow. In some cases, however,
reducing the dimensionality of the training data may filter out
some noise and unnecessary details and thus result in higher per‐
formance (but in general it won’t; it will just speed up training).
Apart from speeding up training, dimensionality reduction is also extremely useful
for data visualization (or DataViz). Reducing the number of dimensions down to two
205
Download from finelybook www.finelybook.com


1 Well, four dimensions if you count time, and a few more if you are a string theorist.
2 Watch a rotating tesseract projected into 3D space at http://goo.gl/OM7ktJ. Image by Wikipedia user Nerd‐
Boy1392 (Creative Commons BY-SA 3.0). Reproduced from https://en.wikipedia.org/wiki/Tesseract.
3 Fun fact: anyone you know is probably an extremist in at least one dimension (e.g., how much sugar they put
in their coffee), if you consider enough dimensions.
(or three) makes it possible to plot a high-dimensional training set on a graph and
often gain some important insights by visually detecting patterns, such as clusters.
In this chapter we will discuss the curse of dimensionality and get a sense of what
goes on in high-dimensional space. Then, we will present the two main approaches to
dimensionality reduction (projection and Manifold Learning), and we will go
through three of the most popular dimensionality reduction techniques: PCA, Kernel
PCA, and LLE.
The Curse of Dimensionality


PCA, and LLE.
The Curse of Dimensionality
We are so used to living in three dimensions1 that our intuition fails us when we try
to imagine a high-dimensional space. Even a basic 4D hypercube is incredibly hard to
picture in our mind (see Figure 8-1), let alone a 200-dimensional ellipsoid bent in a
1,000-dimensional space.
Figure 8-1. Point, segment, square, cube, and tesseract (0D to 4D hypercubes)2
It turns out that many things behave very differently in high-dimensional space. For
example, if you pick a random point in a unit square (a 1 × 1 square), it will have only
about a 0.4% chance of being located less than 0.001 from a border (in other words, it
is very unlikely that a random point will be “extreme” along any dimension). But in a
10,000-dimensional unit hypercube (a 1 × 1 × ⋯ × 1 cube, with ten thousand 1s), this
probability is greater than 99.999999%. Most points in a high-dimensional hypercube
are very close to the border.3
206
|
Chapter 8: Dimensionality Reduction


are very close to the border.3
206
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


Here is a more troublesome difference: if you pick two points randomly in a unit
square, the distance between these two points will be, on average, roughly 0.52. If you
pick two random points in a unit 3D cube, the average distance will be roughly 0.66.
But what about two points picked randomly in a 1,000,000-dimensional hypercube?
Well, the average distance, believe it or not, will be about 408.25 (roughly
1, 000, 000/6)! This is quite counterintuitive: how can two points be so far apart
when they both lie within the same unit hypercube? This fact implies that high-
dimensional datasets are at risk of being very sparse: most training instances are
likely to be far away from each other. Of course, this also means that a new instance
will likely be far away from any training instance, making predictions much less relia‐
ble than in lower dimensions, since they will be based on much larger extrapolations.


ble than in lower dimensions, since they will be based on much larger extrapolations.
In short, the more dimensions the training set has, the greater the risk of overfitting
it.
In theory, one solution to the curse of dimensionality could be to increase the size of
the training set to reach a sufficient density of training instances. Unfortunately, in
practice, the number of training instances required to reach a given density grows
exponentially with the number of dimensions. With just 100 features (much less than
in the MNIST problem), you would need more training instances than atoms in the
observable universe in order for training instances to be within 0.1 of each other on
average, assuming they were spread out uniformly across all dimensions.
Main Approaches for Dimensionality Reduction
Before we dive into specific dimensionality reduction algorithms, let’s take a look at
the two main approaches to reducing dimensionality: projection and Manifold
Learning.
Projection


the two main approaches to reducing dimensionality: projection and Manifold
Learning.
Projection
In most real-world problems, training instances are not spread out uniformly across
all dimensions. Many features are almost constant, while others are highly correlated
(as discussed earlier for MNIST). As a result, all training instances actually lie within
(or close to) a much lower-dimensional subspace of the high-dimensional space. This
sounds very abstract, so let’s look at an example. In Figure 8-2 you can see a 3D data‐
set represented by the circles.
Main Approaches for Dimensionality Reduction
|
207
Download from finelybook www.finelybook.com


Figure 8-2. A 3D dataset lying close to a 2D subspace
Notice that all training instances lie close to a plane: this is a lower-dimensional (2D)
subspace of the high-dimensional (3D) space. Now if we project every training
instance perpendicularly onto this subspace (as represented by the short lines con‐
necting the instances to the plane), we get the new 2D dataset shown in Figure 8-3.
Ta-da! We have just reduced the dataset’s dimensionality from 3D to 2D. Note that
the axes correspond to new features z1 and z2 (the coordinates of the projections on
the plane).
Figure 8-3. The new 2D dataset after projection
208
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


However, projection is not always the best approach to dimensionality reduction. In
many cases the subspace may twist and turn, such as in the famous Swiss roll toy data‐
set represented in Figure 8-4.
Figure 8-4. Swiss roll dataset
Simply projecting onto a plane (e.g., by dropping x3) would squash different layers of
the Swiss roll together, as shown on the left of Figure 8-5. However, what you really
want is to unroll the Swiss roll to obtain the 2D dataset on the right of Figure 8-5.
Figure 8-5. Squashing by projecting onto a plane (left) versus unrolling the Swiss roll
(right)
Main Approaches for Dimensionality Reduction
|
209
Download from finelybook www.finelybook.com


Manifold Learning
The Swiss roll is an example of a 2D manifold. Put simply, a 2D manifold is a 2D
shape that can be bent and twisted in a higher-dimensional space. More generally, a
d-dimensional manifold is a part of an n-dimensional space (where d < n) that locally
resembles a d-dimensional hyperplane. In the case of the Swiss roll, d = 2 and n = 3: it
locally resembles a 2D plane, but it is rolled in the third dimension.
Many dimensionality reduction algorithms work by modeling the manifold on which
the training instances lie; this is called Manifold Learning. It relies on the manifold
assumption, also called the manifold hypothesis, which holds that most real-world
high-dimensional datasets lie close to a much lower-dimensional manifold. This
assumption is very often empirically observed.
Once again, think about the MNIST dataset: all handwritten digit images have some
similarities. They are made of connected lines, the borders are white, they are more


similarities. They are made of connected lines, the borders are white, they are more
or less centered, and so on. If you randomly generated images, only a ridiculously
tiny fraction of them would look like handwritten digits. In other words, the degrees
of freedom available to you if you try to create a digit image are dramatically lower
than the degrees of freedom you would have if you were allowed to generate any
image you wanted. These constraints tend to squeeze the dataset into a lower-
dimensional manifold.
The manifold assumption is often accompanied by another implicit assumption: that
the task at hand (e.g., classification or regression) will be simpler if expressed in the
lower-dimensional space of the manifold. For example, in the top row of Figure 8-6
the Swiss roll is split into two classes: in the 3D space (on the left), the decision
boundary would be fairly complex, but in the 2D unrolled manifold space (on the
right), the decision boundary is a simple straight line.


right), the decision boundary is a simple straight line.
However, this assumption does not always hold. For example, in the bottom row of
Figure 8-6, the decision boundary is located at x1 = 5. This decision boundary looks
very simple in the original 3D space (a vertical plane), but it looks more complex in
the unrolled manifold (a collection of four independent line segments).
In short, if you reduce the dimensionality of your training set before training a
model, it will definitely speed up training, but it may not always lead to a better or
simpler solution; it all depends on the dataset.
Hopefully you now have a good sense of what the curse of dimensionality is and how
dimensionality reduction algorithms can fight it, especially when the manifold
assumption holds. The rest of this chapter will go through some of the most popular
algorithms.
210
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


Figure 8-6. The decision boundary may not always be simpler with lower dimensions
PCA
Principal Component Analysis (PCA) is by far the most popular dimensionality reduc‐
tion algorithm. First it identifies the hyperplane that lies closest to the data, and then
it projects the data onto it.
Preserving the Variance
Before you can project the training set onto a lower-dimensional hyperplane, you
first need to choose the right hyperplane. For example, a simple 2D dataset is repre‐
sented on the left of Figure 8-7, along with three different axes (i.e., one-dimensional
hyperplanes). On the right is the result of the projection of the dataset onto each of
these axes. As you can see, the projection onto the solid line preserves the maximum
variance, while the projection onto the dotted line preserves very little variance, and
the projection onto the dashed line preserves an intermediate amount of variance.
PCA
|
211
Download from finelybook www.finelybook.com


4 “On Lines and Planes of Closest Fit to Systems of Points in Space,” K. Pearson (1901).
Figure 8-7. Selecting the subspace onto which to project
It seems reasonable to select the axis that preserves the maximum amount of var‐
iance, as it will most likely lose less information than the other projections. Another
way to justify this choice is that it is the axis that minimizes the mean squared dis‐
tance between the original dataset and its projection onto that axis. This is the rather
simple idea behind PCA.4
Principal Components
PCA identifies the axis that accounts for the largest amount of variance in the train‐
ing set. In Figure 8-7, it is the solid line. It also finds a second axis, orthogonal to the
first one, that accounts for the largest amount of remaining variance. In this 2D
example there is no choice: it is the dotted line. If it were a higher-dimensional data‐
set, PCA would also find a third axis, orthogonal to both previous axes, and a fourth,


set, PCA would also find a third axis, orthogonal to both previous axes, and a fourth,
a fifth, and so on—as many axes as the number of dimensions in the dataset.
The unit vector that defines the ith axis is called the ith principal component (PC). In
Figure 8-7, the 1st PC is c1 and the 2nd PC is c2. In Figure 8-2 the first two PCs are
represented by the orthogonal arrows in the plane, and the third PC would be
orthogonal to the plane (pointing up or down).
212
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


The direction of the principal components is not stable: if you per‐
turb the training set slightly and run PCA again, some of the new
PCs may point in the opposite direction of the original PCs. How‐
ever, they will generally still lie on the same axes. In some cases, a
pair of PCs may even rotate or swap, but the plane they define will
generally remain the same.
So how can you find the principal components of a training set? Luckily, there is a
standard matrix factorization technique called Singular Value Decomposition (SVD)
that can decompose the training set matrix X into the dot product of three matrices U
· Σ · VT, where VT contains all the principal components that we are looking for, as
shown in Equation 8-1.
Equation 8-1. Principal components matrix
�T =
∣
∣
∣
�1 �2 ⋯��
∣
∣
∣
The following Python code uses NumPy’s svd() function to obtain all the principal
components of the training set, then extracts the first two PCs:
X_centered = X - X.mean(axis=0)


components of the training set, then extracts the first two PCs:
X_centered = X - X.mean(axis=0)
U, s, V = np.linalg.svd(X_centered)
c1 = V.T[:, 0]
c2 = V.T[:, 1]
PCA assumes that the dataset is centered around the origin. As we
will see, Scikit-Learn’s PCA classes take care of centering the data
for you. However, if you implement PCA yourself (as in the pre‐
ceding example), or if you use other libraries, don’t forget to center
the data first.
Projecting Down to d Dimensions
Once you have identified all the principal components, you can reduce the dimen‐
sionality of the dataset down to d dimensions by projecting it onto the hyperplane
defined by the first d principal components. Selecting this hyperplane ensures that the
projection will preserve as much variance as possible. For example, in Figure 8-2 the
3D dataset is projected down to the 2D plane defined by the first two principal com‐
ponents, preserving a large part of the dataset’s variance. As a result, the 2D projec‐


ponents, preserving a large part of the dataset’s variance. As a result, the 2D projec‐
tion looks very much like the original 3D dataset.
To project the training set onto the hyperplane, you can simply compute the dot
product of the training set matrix X by the matrix Wd, defined as the matrix contain‐
PCA
|
213
Download from finelybook www.finelybook.com


ing the first d principal components (i.e., the matrix composed of the first d columns
of VT), as shown in Equation 8-2.
Equation 8-2. Projecting the training set down to d dimensions
�d‐proj = �· �d
The following Python code projects the training set onto the plane defined by the first
two principal components:
W2 = V.T[:, :2]
X2D = X_centered.dot(W2)
There you have it! You now know how to reduce the dimensionality of any dataset
down to any number of dimensions, while preserving as much variance as possible.
Using Scikit-Learn
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did
before. The following code applies PCA to reduce the dimensionality of the dataset
down to two dimensions (note that it automatically takes care of centering the data):
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D = pca.fit_transform(X)
After fitting the PCA transformer to the dataset, you can access the principal compo‐


After fitting the PCA transformer to the dataset, you can access the principal compo‐
nents using the components_ variable (note that it contains the PCs as horizontal vec‐
tors, so, for example, the first principal component is equal to pca.components_.T[:,
0]).
Explained Variance Ratio
Another very useful piece of information is the explained variance ratio of each prin‐
cipal component, available via the explained_variance_ratio_ variable. It indicates
the proportion of the dataset’s variance that lies along the axis of each principal com‐
ponent. For example, let’s look at the explained variance ratios of the first two compo‐
nents of the 3D dataset represented in Figure 8-2:
>>> print(pca.explained_variance_ratio_)
array([ 0.84248607,  0.14631839])
This tells you that 84.2% of the dataset’s variance lies along the first axis, and 14.6%
lies along the second axis. This leaves less than 1.2% for the third axis, so it is reason‐


lies along the second axis. This leaves less than 1.2% for the third axis, so it is reason‐
able to assume that it probably carries little information.
214
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


Choosing the Right Number of Dimensions
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is
generally preferable to choose the number of dimensions that add up to a sufficiently
large portion of the variance (e.g., 95%). Unless, of course, you are reducing dimen‐
sionality for data visualization—in that case you will generally want to reduce the
dimensionality down to 2 or 3.
The following code computes PCA without reducing dimensionality, then computes
the minimum number of dimensions required to preserve 95% of the training set’s
variance:
pca = PCA()
pca.fit(X)
cumsum = np.cumsum(pca.explained_variance_ratio_)
d = np.argmax(cumsum >= 0.95) + 1
You could then set n_components=d and run PCA again. However, there is a much
better option: instead of specifying the number of principal components you want to
preserve, you can set n_components to be a float between 0.0 and 1.0, indicating the
ratio of variance you wish to preserve:


ratio of variance you wish to preserve:
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)
Yet another option is to plot the explained variance as a function of the number of
dimensions (simply plot cumsum; see Figure 8-8). There will usually be an elbow in the
curve, where the explained variance stops growing fast. You can think of this as the
intrinsic dimensionality of the dataset. In this case, you can see that reducing the
dimensionality down to about 100 dimensions wouldn’t lose too much explained var‐
iance.
Figure 8-8. Explained variance as a function of the number of dimensions
PCA
|
215
Download from finelybook www.finelybook.com


PCA for Compression
Obviously after dimensionality reduction, the training set takes up much less space.
For example, try applying PCA to the MNIST dataset while preserving 95% of its var‐
iance. You should find that each instance will have just over 150 features, instead of
the original 784 features. So while most of the variance is preserved, the dataset is
now less than 20% of its original size! This is a reasonable compression ratio, and you
can see how this can speed up a classification algorithm (such as an SVM classifier)
tremendously.
It is also possible to decompress the reduced dataset back to 784 dimensions by
applying the inverse transformation of the PCA projection. Of course this won’t give
you back the original data, since the projection lost a bit of information (within the
5% variance that was dropped), but it will likely be quite close to the original data.
The mean squared distance between the original data and the reconstructed data


The mean squared distance between the original data and the reconstructed data
(compressed and then decompressed) is called the reconstruction error. For example,
the following code compresses the MNIST dataset down to 154 dimensions, then uses
the inverse_transform() method to decompress it back to 784 dimensions.
Figure 8-9 shows a few digits from the original training set (on the left), and the cor‐
responding digits after compression and decompression. You can see that there is a
slight image quality loss, but the digits are still mostly intact.
pca = PCA(n_components = 154)
X_mnist_reduced = pca.fit_transform(X_mnist)
X_mnist_recovered = pca.inverse_transform(X_mnist_reduced)
Figure 8-9. MNIST compression preserving 95% of the variance
216
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


5 Scikit-Learn uses the algorithm described in “Incremental Learning for Robust Visual Tracking,” D. Ross et al.
(2007).
The equation of the inverse transformation is shown in Equation 8-3.
Equation 8-3. PCA inverse transformation, back to the original number of
dimensions
�recovered = �d‐proj · �d
T
Incremental PCA
One problem with the preceding implementation of PCA is that it requires the whole
training set to fit in memory in order for the SVD algorithm to run. Fortunately,
Incremental PCA (IPCA) algorithms have been developed: you can split the training
set into mini-batches and feed an IPCA algorithm one mini-batch at a time. This is
useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
instances arrive).
The following code splits the MNIST dataset into 100 mini-batches (using NumPy’s
array_split() function) and feeds them to Scikit-Learn’s IncrementalPCA class5 to
reduce the dimensionality of the MNIST dataset down to 154 dimensions (just like


reduce the dimensionality of the MNIST dataset down to 154 dimensions (just like
before). Note that you must call the partial_fit() method with each mini-batch
rather than the fit() method with the whole training set:
from sklearn.decomposition import IncrementalPCA
n_batches = 100
inc_pca = IncrementalPCA(n_components=154)
for X_batch in np.array_split(X_mnist, n_batches):
    inc_pca.partial_fit(X_batch)
X_mnist_reduced = inc_pca.transform(X_mnist)
Alternatively, you can use NumPy’s memmap class, which allows you to manipulate a
large array stored in a binary file on disk as if it were entirely in memory; the class
loads only the data it needs in memory, when it needs it. Since the IncrementalPCA
class uses only a small part of the array at any given time, the memory usage remains
under control. This makes it possible to call the usual fit() method, as you can see
in the following code:
X_mm = np.memmap(filename, dtype="float32", mode="readonly", shape=(m, n))


in the following code:
X_mm = np.memmap(filename, dtype="float32", mode="readonly", shape=(m, n))
batch_size = m // n_batches
inc_pca = IncrementalPCA(n_components=154, batch_size=batch_size)
inc_pca.fit(X_mm)
PCA
|
217
Download from finelybook www.finelybook.com


6 “Kernel Principal Component Analysis,” B. Schölkopf, A. Smola, K. Müller (1999).
Randomized PCA
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This
is a stochastic algorithm that quickly finds an approximation of the first d principal
components. Its computational complexity is O(m × d2) + O(d3), instead of O(m × n2)
+ O(n3), so it is dramatically faster than the previous algorithms when d is much
smaller than n.
rnd_pca = PCA(n_components=154, svd_solver="randomized")
X_reduced = rnd_pca.fit_transform(X_mnist)
Kernel PCA
In Chapter 5 we discussed the kernel trick, a mathematical technique that implicitly
maps instances into a very high-dimensional space (called the feature space), enabling
nonlinear classification and regression with Support Vector Machines. Recall that a
linear decision boundary in the high-dimensional feature space corresponds to a
complex nonlinear decision boundary in the original space.


complex nonlinear decision boundary in the original space.
It turns out that the same trick can be applied to PCA, making it possible to perform
complex nonlinear projections for dimensionality reduction. This is called Kernel
PCA (kPCA).6 It is often good at preserving clusters of instances after projection, or
sometimes even unrolling datasets that lie close to a twisted manifold.
For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA
with an RBF kernel (see Chapter 5 for more details about the RBF kernel and the
other kernels):
from sklearn.decomposition import KernelPCA
rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
X_reduced = rbf_pca.fit_transform(X)
Figure 8-10 shows the Swiss roll, reduced to two dimensions using a linear kernel
(equivalent to simply using the PCA class), an RBF kernel, and a sigmoid kernel
(Logistic).
218
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


Figure 8-10. Swiss roll reduced to 2D using kPCA with various kernels
Selecting a Kernel and Tuning Hyperparameters
As kPCA is an unsupervised learning algorithm, there is no obvious performance
measure to help you select the best kernel and hyperparameter values. However,
dimensionality reduction is often a preparation step for a supervised learning task
(e.g., classification), so you can simply use grid search to select the kernel and hyper‐
parameters that lead to the best performance on that task. For example, the following
code creates a two-step pipeline, first reducing dimensionality to two dimensions
using kPCA, then applying Logistic Regression for classification. Then it uses Grid
SearchCV to find the best kernel and gamma value for kPCA in order to get the best
classification accuracy at the end of the pipeline:
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
clf = Pipeline([


from sklearn.pipeline import Pipeline
clf = Pipeline([
        ("kpca", KernelPCA(n_components=2)),
        ("log_reg", LogisticRegression())
    ])
param_grid = [{
        "kpca__gamma": np.linspace(0.03, 0.05, 10),
        "kpca__kernel": ["rbf", "sigmoid"]
    }]
grid_search = GridSearchCV(clf, param_grid, cv=3)
grid_search.fit(X, y)
The best kernel and hyperparameters are then available through the best_params_
variable:
>>> print(grid_search.best_params_)
{'kpca__gamma': 0.043333333333333335, 'kpca__kernel': 'rbf'}
Kernel PCA
|
219
Download from finelybook www.finelybook.com


Another approach, this time entirely unsupervised, is to select the kernel and hyper‐
parameters that yield the lowest reconstruction error. However, reconstruction is not
as easy as with linear PCA. Here’s why. Figure 8-11 shows the original Swiss roll 3D
dataset (top left), and the resulting 2D dataset after kPCA is applied using an RBF
kernel (top right). Thanks to the kernel trick, this is mathematically equivalent to
mapping the training set to an infinite-dimensional feature space (bottom right)
using the feature map φ, then projecting the transformed training set down to 2D
using linear PCA. Notice that if we could invert the linear PCA step for a given
instance in the reduced space, the reconstructed point would lie in feature space, not
in the original space (e.g., like the one represented by an x in the diagram). Since the
feature space is infinite-dimensional, we cannot compute the reconstructed point,


feature space is infinite-dimensional, we cannot compute the reconstructed point,
and therefore we cannot compute the true reconstruction error. Fortunately, it is pos‐
sible to find a point in the original space that would map close to the reconstructed
point. This is called the reconstruction pre-image. Once you have this pre-image, you
can measure its squared distance to the original instance. You can then select the ker‐
nel and hyperparameters that minimize this reconstruction pre-image error.
Figure 8-11. Kernel PCA and the reconstruction pre-image error
220
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


7 Scikit-Learn uses the algorithm based on Kernel Ridge Regression described in Gokhan H. Bakır, Jason
Weston, and Bernhard Scholkopf, “Learning to Find Pre-images” (Tubingen, Germany: Max Planck Institute
for Biological Cybernetics, 2004).
8 “Nonlinear Dimensionality Reduction by Locally Linear Embedding,” S. Roweis, L. Saul (2000).
You may be wondering how to perform this reconstruction. One solution is to train a
supervised regression model, with the projected instances as the training set and the
original instances as the targets. Scikit-Learn will do this automatically if you set
fit_inverse_transform=True, as shown in the following code:7
rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.0433,
                    fit_inverse_transform=True)
X_reduced = rbf_pca.fit_transform(X)
X_preimage = rbf_pca.inverse_transform(X_reduced)
By default, fit_inverse_transform=False and KernelPCA has no
inverse_transform() method. This method only gets created


inverse_transform() method. This method only gets created
when you set fit_inverse_transform=True.
You can then compute the reconstruction pre-image error:
>>> from sklearn.metrics import mean_squared_error
>>> mean_squared_error(X, X_preimage)
32.786308795766132
Now you can use grid search with cross-validation to find the kernel and hyperpara‐
meters that minimize this pre-image reconstruction error.
LLE
Locally Linear Embedding (LLE)8 is another very powerful nonlinear dimensionality
reduction (NLDR) technique. It is a Manifold Learning technique that does not rely
on projections like the previous algorithms. In a nutshell, LLE works by first measur‐
ing how each training instance linearly relates to its closest neighbors (c.n.), and then
looking for a low-dimensional representation of the training set where these local
relationships are best preserved (more details shortly). This makes it particularly


relationships are best preserved (more details shortly). This makes it particularly
good at unrolling twisted manifolds, especially when there is not too much noise.
For example, the following code uses Scikit-Learn’s LocallyLinearEmbedding class to
unroll the Swiss roll. The resulting 2D dataset is shown in Figure 8-12. As you can
see, the Swiss roll is completely unrolled and the distances between instances are
locally well preserved. However, distances are not preserved on a larger scale: the left
part of the unrolled Swiss roll is squeezed, while the right part is stretched. Neverthe‐
less, LLE did a pretty good job at modeling the manifold.
LLE
|
221
Download from finelybook www.finelybook.com


from sklearn.manifold import LocallyLinearEmbedding
lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
X_reduced = lle.fit_transform(X)
Figure 8-12. Unrolled Swiss roll using LLE
Here’s how LLE works: first, for each training instance x(i), the algorithm identifies its
k closest neighbors (in the preceding code k = 10), then tries to reconstruct x(i) as a
linear function of these neighbors. More specifically, it finds the weights wi,j such that
the squared distance between x(i) and ∑j = 1
m
wi, j�j  is as small as possible, assuming
wi,j = 0 if x(j) is not one of the k closest neighbors of x(i). Thus the first step of LLE is
the constrained optimization problem described in Equation 8-4, where W is the
weight matrix containing all the weights wi,j. The second constraint simply normalizes
the weights for each training instance x(i).
222
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


Equation 8-4. LLE step 1: linearly modeling local relationships
�= argmin
�
∑
i = 1
m
∥�i −∑
j = 1
m
wi, j�j ∥
2
subject to
wi, j = 0
if �j is not one of the k c.n. of �i
∑
j = 1
m
wi, j = 1 for i = 1, 2, ⋯, m
After this step, the weight matrix � (containing the weights wi, j) encodes the local
linear relationships between the training instances. Now the second step is to map the
training instances into a d-dimensional space (where d < n) while preserving these
local relationships as much as possible. If z(i) is the image of x(i) in this d-dimensional
space, then we want the squared distance between z(i) and ∑j = 1
m
wi, j�j  to be as small
as possible. This idea leads to the unconstrained optimization problem described in
Equation 8-5. It looks very similar to the first step, but instead of keeping the instan‐
ces fixed and finding the optimal weights, we are doing the reverse: keeping the
weights fixed and finding the optimal position of the instances’ images in the low-


weights fixed and finding the optimal position of the instances’ images in the low-
dimensional space. Note that Z is the matrix containing all z(i).
Equation 8-5. LLE step 2: reducing dimensionality while preserving relationships
�= argmin
�
∑
i = 1
m
∥�i −∑
j = 1
m
wi, j�j ∥
2
Scikit-Learn’s LLE implementation has the following computational complexity:
O(m log(m)n log(k)) for finding the k nearest neighbors, O(mnk3) for optimizing the
weights, and O(dm2) for constructing the low-dimensional representations. Unfortu‐
nately, the m2 in the last term makes this algorithm scale poorly to very large datasets.
Other Dimensionality Reduction Techniques
There are many other dimensionality reduction techniques, several of which are
available in Scikit-Learn. Here are some of the most popular:
• Multidimensional Scaling (MDS) reduces dimensionality while trying to preserve
the distances between the instances (see Figure 8-13).
Other Dimensionality Reduction Techniques
|
223


Other Dimensionality Reduction Techniques
|
223
Download from finelybook www.finelybook.com


9 The geodesic distance between two nodes in a graph is the number of nodes on the shortest path between
these nodes.
• Isomap creates a graph by connecting each instance to its nearest neighbors, then
reduces dimensionality while trying to preserve the geodesic distances9 between
the instances.
• t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality
while trying to keep similar instances close and dissimilar instances apart. It is
mostly used for visualization, in particular to visualize clusters of instances in
high-dimensional space (e.g., to visualize the MNIST images in 2D).
• Linear Discriminant Analysis (LDA) is actually a classification algorithm, but dur‐
ing training it learns the most discriminative axes between the classes, and these
axes can then be used to define a hyperplane onto which to project the data. The
benefit is that the projection will keep classes as far apart as possible, so LDA is a


benefit is that the projection will keep classes as far apart as possible, so LDA is a
good technique to reduce dimensionality before running another classification
algorithm such as an SVM classifier.
Figure 8-13. Reducing the Swiss roll to 2D using various techniques
Exercises
1. What are the main motivations for reducing a dataset’s dimensionality? What are
the main drawbacks?
2. What is the curse of dimensionality?
3. Once a dataset’s dimensionality has been reduced, is it possible to reverse the
operation? If so, how? If not, why?
4. Can PCA be used to reduce the dimensionality of a highly nonlinear dataset?
5. Suppose you perform PCA on a 1,000-dimensional dataset, setting the explained
variance ratio to 95%. How many dimensions will the resulting dataset have?
224
|
Chapter 8: Dimensionality Reduction
Download from finelybook www.finelybook.com


6. In what cases would you use vanilla PCA, Incremental PCA, Randomized PCA,
or Kernel PCA?
7. How can you evaluate the performance of a dimensionality reduction algorithm
on your dataset?
8. Does it make any sense to chain two different dimensionality reduction algo‐
rithms?
9. Load the MNIST dataset (introduced in Chapter 3) and split it into a training set
and a test set (take the first 60,000 instances for training, and the remaining
10,000 for testing). Train a Random Forest classifier on the dataset and time how
long it takes, then evaluate the resulting model on the test set. Next, use PCA to
reduce the dataset’s dimensionality, with an explained variance ratio of 95%.
Train a new Random Forest classifier on the reduced dataset and see how long it
takes. Was training much faster? Next evaluate the classifier on the test set: how
does it compare to the previous classifier?
10. Use t-SNE to reduce the MNIST dataset down to two dimensions and plot the


10. Use t-SNE to reduce the MNIST dataset down to two dimensions and plot the
result using Matplotlib. You can use a scatterplot using 10 different colors to rep‐
resent each image’s target class. Alternatively, you can write colored digits at the
location of each instance, or even plot scaled-down versions of the digit images
themselves (if you plot all digits, the visualization will be too cluttered, so you
should either draw a random sample or plot an instance only if no other instance
has already been plotted at a close distance). You should get a nice visualization
with well-separated clusters of digits. Try using other dimensionality reduction
algorithms such as PCA, LLE, or MDS and compare the resulting visualizations.
Solutions to these exercises are available in Appendix A.
Exercises
|
225
Download from finelybook www.finelybook.com


Download from finelybook www.finelybook.com

PART II
Neural Networks and Deep Learning
Download from finelybook www.finelybook.com

Download from finelybook www.finelybook.com


CHAPTER 9
Up and Running with TensorFlow
TensorFlow is a powerful open source software library for numerical computation,
particularly well suited and fine-tuned for large-scale Machine Learning. Its basic
principle is simple: you first define in Python a graph of computations to perform
(for example, the one in Figure 9-1), and then TensorFlow takes that graph and runs
it efficiently using optimized C++ code.
Figure 9-1. A simple computation graph
Most importantly, it is possible to break up the graph into several chunks and run
them in parallel across multiple CPUs or GPUs (as shown in Figure 9-2). TensorFlow
also supports distributed computing, so you can train colossal neural networks on
humongous training sets in a reasonable amount of time by splitting the computa‐
tions across hundreds of servers (see Chapter 12). TensorFlow can train a network
with millions of parameters on a training set composed of billions of instances with


with millions of parameters on a training set composed of billions of instances with
millions of features each. This should come as no surprise, since TensorFlow was
229
Download from finelybook www.finelybook.com


1 TensorFlow is not limited to neural networks or even Machine Learning; you could run quantum physics sim‐
ulations if you wanted.
developed by the Google Brain team and it powers many of Google’s large-scale serv‐
ices, such as Google Cloud Speech, Google Photos, and Google Search.
Figure 9-2. Parallel computation on multiple CPUs/GPUs/servers
When TensorFlow was open-sourced in November 2015, there were already many
popular open source libraries for Deep Learning (Table 9-1 lists a few), and to be fair
most of TensorFlow’s features already existed in one library or another. Nevertheless,
TensorFlow’s clean design, scalability, flexibility,1 and great documentation (not to
mention Google’s name) quickly boosted it to the top of the list. In short, TensorFlow
was designed to be flexible, scalable, and production-ready, and existing frameworks
arguably hit only two out of the three of these. Here are some of TensorFlow’s high‐
lights:


arguably hit only two out of the three of these. Here are some of TensorFlow’s high‐
lights:
• It runs not only on Windows, Linux, and macOS, but also on mobile devices,
including both iOS and Android.
230
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


2 Not to be confused with the TFLearn library, which is an independent project.
• It provides a very simple Python API called TF.Learn2 (tensorflow.con
trib.learn), compatible with Scikit-Learn. As you will see, you can use it to
train various types of neural networks in just a few lines of code. It was previ‐
ously an independent project called Scikit Flow (or skflow).
• It also provides another simple API called TF-slim (tensorflow.contrib.slim)
to simplify building, training, and evaluating neural networks.
• Several other high-level APIs have been built independently on top of Tensor‐
Flow, such as Keras or Pretty Tensor.
• Its main Python API offers much more flexibility (at the cost of higher complex‐
ity) to create all sorts of computations, including any neural network architecture
you can think of.
• It includes highly efficient C++ implementations of many ML operations, partic‐
ularly those needed to build neural networks. There is also a C++ API to define


ularly those needed to build neural networks. There is also a C++ API to define
your own high-performance operations.
• It provides several advanced optimization nodes to search for the parameters that
minimize a cost function. These are very easy to use since TensorFlow automati‐
cally takes care of computing the gradients of the functions you define. This is
called automatic differentiating (or autodiff).
• It also comes with a great visualization tool called TensorBoard that allows you to
browse through the computation graph, view learning curves, and more.
• Google also launched a cloud service to run TensorFlow graphs.
• Last but not least, it has a dedicated team of passionate and helpful developers,
and a growing community contributing to improving it. It is one of the most
popular open source projects on GitHub, and more and more great projects are
being built on top of it (for examples, check out the resources page on https://


being built on top of it (for examples, check out the resources page on https://
www.tensorflow.org/, or https://github.com/jtoy/awesome-tensorflow). To ask
technical questions, you should use http://stackoverflow.com/ and tag your ques‐
tion with "tensorflow". You can file bugs and feature requests through GitHub.
For general discussions, join the Google group.
In this chapter, we will go through the basics of TensorFlow, from installation to cre‐
ating, running, saving, and visualizing simple computational graphs. Mastering these
basics is important before you build your first neural network (which we will do in
the next chapter).
Up and Running with TensorFlow
|
231
Download from finelybook www.finelybook.com


Table 9-1. Open source Deep Learning libraries (not an exhaustive list)
Library
API
Platforms
Started by
Year
Caffe
Python, C++, Matlab Linux, macOS, Windows
Y. Jia, UC Berkeley (BVLC)
2013
Deeplearning4j Java, Scala, Clojure
Linux, macOS, Windows, Android
A. Gibson, J.Patterson
2014
H2O
Python, R
Linux, macOS, Windows
H2O.ai
2014
MXNet
Python, C++, others
Linux, macOS, Windows, iOS, Android
DMLC
2015
TensorFlow
Python, C++
Linux, macOS, Windows, iOS, Android
Google
2015
Theano
Python
Linux, macOS, iOS
University of Montreal
2010
Torch
C++, Lua
Linux, macOS, iOS, Android
R. Collobert, K. Kavukcuoglu, C.
Farabet
2002
Installation
Let’s get started! Assuming you installed Jupyter and Scikit-Learn by following the
installation instructions in Chapter 2, you can simply use pip to install TensorFlow. If
you created an isolated environment using virtualenv, you first need to activate it:
$ cd $ML_PATH               # Your ML working directory (e.g., $HOME/ml)
$ source env/bin/activate


$ cd $ML_PATH               # Your ML working directory (e.g., $HOME/ml)
$ source env/bin/activate
Next, install TensorFlow:
$ pip3 install --upgrade tensorflow
For GPU support, you need to install tensorflow-gpu instead of
tensorflow. See Chapter 12 for more details.
To test your installation, type the following command. It should output the version of
TensorFlow you installed.
$ python3 -c 'import tensorflow; print(tensorflow.__version__)'
1.0.0
Creating Your First Graph and Running It in a Session
The following code creates the graph represented in Figure 9-1:
import tensorflow as tf
x = tf.Variable(3, name="x")
y = tf.Variable(4, name="y")
f = x*x*y + y + 2
232
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


3 In distributed TensorFlow, variable values are stored on the servers instead of the session, as we will see in
Chapter 12.
That’s all there is to it! The most important thing to understand is that this code does
not actually perform any computation, even though it looks like it does (especially the
last line). It just creates a computation graph. In fact, even the variables are not ini‐
tialized yet. To evaluate this graph, you need to open a TensorFlow session and use it
to initialize the variables and evaluate f. A TensorFlow session takes care of placing
the operations onto devices such as CPUs and GPUs and running them, and it holds
all the variable values.3 The following code creates a session, initializes the variables,
and evaluates, and f then closes the session (which frees up resources):
>>> sess = tf.Session()
>>> sess.run(x.initializer)
>>> sess.run(y.initializer)
>>> result = sess.run(f)
>>> print(result)
42
>>> sess.close()


>>> sess.run(y.initializer)
>>> result = sess.run(f)
>>> print(result)
42
>>> sess.close()
Having to repeat sess.run() all the time is a bit cumbersome, but fortunately there is
a better way:
with tf.Session() as sess:
    x.initializer.run()
    y.initializer.run()
    result = f.eval()
Inside the with block, the session is set as the default session. Calling x.initial
izer.run() is equivalent to calling tf.get_default_session().run(x.initial
izer),
and
similarly
f.eval()
is
equivalent
to
calling
tf.get_default_session().run(f). This makes the code easier to read. Moreover,
the session is automatically closed at the end of the block.
Instead of manually running the initializer for every single variable, you can use the
global_variables_initializer() function. Note that it does not actually perform
the initialization immediately, but rather creates a node in the graph that will initialize
all variables when it is run:


all variables when it is run:
init = tf.global_variables_initializer()  # prepare an init node
with tf.Session() as sess:
    init.run()  # actually initialize all the variables
    result = f.eval()
Inside Jupyter or within a Python shell you may prefer to create an InteractiveSes
sion. The only difference from a regular Session is that when an InteractiveSes
sion is created it automatically sets itself as the default session, so you don’t need a
Creating Your First Graph and Running It in a Session
|
233
Download from finelybook www.finelybook.com


with block (but you do need to close the session manually when you are done with
it):
>>> sess = tf.InteractiveSession()
>>> init.run()
>>> result = f.eval()
>>> print(result)
42
>>> sess.close()
A TensorFlow program is typically split into two parts: the first part builds a compu‐
tation graph (this is called the construction phase), and the second part runs it (this is
the execution phase). The construction phase typically builds a computation graph
representing the ML model and the computations required to train it. The execution
phase generally runs a loop that evaluates a training step repeatedly (for example, one
step per mini-batch), gradually improving the model parameters. We will go through
an example shortly.
Managing Graphs
Any node you create is automatically added to the default graph:
>>> x1 = tf.Variable(1)
>>> x1.graph is tf.get_default_graph()
True
In most cases this is fine, but sometimes you may want to manage multiple independ‐


True
In most cases this is fine, but sometimes you may want to manage multiple independ‐
ent graphs. You can do this by creating a new Graph and temporarily making it the
default graph inside a with block, like so:
>>> graph = tf.Graph()
>>> with graph.as_default():
...     x2 = tf.Variable(2)
...
>>> x2.graph is graph
True
>>> x2.graph is tf.get_default_graph()
False
In Jupyter (or in a Python shell), it is common to run the same
commands more than once while you are experimenting. As a
result, you may end up with a default graph containing many
duplicate nodes. One solution is to restart the Jupyter kernel (or
the Python shell), but a more convenient solution is to just reset the
default graph by running tf.reset_default_graph().
234
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


Lifecycle of a Node Value
When you evaluate a node, TensorFlow automatically determines the set of nodes
that it depends on and it evaluates these nodes first. For example, consider the follow‐
ing code:
w = tf.constant(3)
x = w + 2
y = x + 5
z = x * 3
with tf.Session() as sess:
    print(y.eval())  # 10
    print(z.eval())  # 15
First, this code defines a very simple graph. Then it starts a session and runs the
graph to evaluate y: TensorFlow automatically detects that y depends on w, which
depends on x, so it first evaluates w, then x, then y, and returns the value of y. Finally,
the code runs the graph to evaluate z. Once again, TensorFlow detects that it must
first evaluate w and x. It is important to note that it will not reuse the result of the
previous evaluation of w and x. In short, the preceding code evaluates w and x twice.
All node values are dropped between graph runs, except variable values, which are


All node values are dropped between graph runs, except variable values, which are
maintained by the session across graph runs (queues and readers also maintain some
state, as we will see in Chapter 12). A variable starts its life when its initializer is run,
and it ends when the session is closed.
If you want to evaluate y and z efficiently, without evaluating w and x twice as in the
previous code, you must ask TensorFlow to evaluate both y and z in just one graph
run, as shown in the following code:
with tf.Session() as sess:
    y_val, z_val = sess.run([y, z])
    print(y_val)  # 10
    print(z_val)  # 15
In single-process TensorFlow, multiple sessions do not share any
state, even if they reuse the same graph (each session would have its
own copy of every variable). In distributed TensorFlow (see Chap‐
ter 12), variable state is stored on the servers, not in the sessions, so
multiple sessions can share the same variables.
Linear Regression with TensorFlow


multiple sessions can share the same variables.
Linear Regression with TensorFlow
TensorFlow operations (also called ops for short) can take any number of inputs and
produce any number of outputs. For example, the addition and multiplication ops
each take two inputs and produce one output. Constants and variables take no input
Lifecycle of a Node Value
|
235
Download from finelybook www.finelybook.com


4 Note that housing.target is a 1D array, but we need to reshape it to a column vector to compute theta.
Recall that NumPy’s reshape() function accepts –1 (meaning “unspecified”) for one of the dimensions: that
dimension will be computed based on the array’s length and the remaining dimensions.
(they are called source ops). The inputs and outputs are multidimensional arrays,
called tensors (hence the name “tensor flow”). Just like NumPy arrays, tensors have a
type and a shape. In fact, in the Python API tensors are simply represented by NumPy
ndarrays. They typically contain floats, but you can also use them to carry strings
(arbitrary byte arrays).
In the examples so far, the tensors just contained a single scalar value, but you can of
course perform computations on arrays of any shape. For example, the following code
manipulates 2D arrays to perform Linear Regression on the California housing data‐


manipulates 2D arrays to perform Linear Regression on the California housing data‐
set (introduced in Chapter 2). It starts by fetching the dataset; then it adds an extra
bias input feature (x0 = 1) to all training instances (it does so using NumPy so it runs
immediately); then it creates two TensorFlow constant nodes, X and y, to hold this
data and the targets,4 and it uses some of the matrix operations provided by Tensor‐
Flow to define theta. These matrix functions—transpose(), matmul(), and
matrix_inverse()—are self-explanatory, but as usual they do not perform any com‐
putations immediately; instead, they create nodes in the graph that will perform them
when the graph is run. You may recognize that the definition of theta corresponds to
the Normal Equation (θ = XT · X)–1 · XT · y; see Chapter 4). Finally, the code creates a
session and uses it to evaluate theta.
import numpy as np
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()


from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()
m, n = housing.data.shape
housing_data_plus_bias = np.c_[np.ones((m, 1)), housing.data]
X = tf.constant(housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
XT = tf.transpose(X)
theta = tf.matmul(tf.matmul(tf.matrix_inverse(tf.matmul(XT, X)), XT), y)
with tf.Session() as sess:
    theta_value = theta.eval()
The main benefit of this code versus computing the Normal Equation directly using
NumPy is that TensorFlow will automatically run this on your GPU card if you have
one (provided you installed TensorFlow with GPU support, of course; see Chapter 12
for more details).
236
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


Implementing Gradient Descent
Let’s try using Batch Gradient Descent (introduced in Chapter 4) instead of the Nor‐
mal Equation. First we will do this by manually computing the gradients, then we will
use TensorFlow’s autodiff feature to let TensorFlow compute the gradients automati‐
cally, and finally we will use a couple of TensorFlow’s out-of-the-box optimizers.
When using Gradient Descent, remember that it is important to
first normalize the input feature vectors, or else training may be
much slower. You can do this using TensorFlow, NumPy, Scikit-
Learn’s StandardScaler, or any other solution you prefer. The fol‐
lowing code assumes that this normalization has already been
done.
Manually Computing the Gradients
The following code should be fairly self-explanatory, except for a few new elements:
• The random_uniform() function creates a node in the graph that will generate a
tensor containing random values, given its shape and value range, much like
NumPy’s rand() function.


NumPy’s rand() function.
• The assign() function creates a node that will assign a new value to a variable.
In this case, it implements the Batch Gradient Descent step θ(next step) = θ –
η∇θMSE(θ).
• The main loop executes the training step over and over again (n_epochs times),
and every 100 iterations it prints out the current Mean Squared Error (mse). You
should see the MSE go down at every iteration.
n_epochs = 1000
learning_rate = 0.01
X = tf.constant(scaled_housing_data_plus_bias, dtype=tf.float32, name="X")
y = tf.constant(housing.target.reshape(-1, 1), dtype=tf.float32, name="y")
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0), name="theta")
y_pred = tf.matmul(X, theta, name="predictions")
error = y_pred - y
mse = tf.reduce_mean(tf.square(error), name="mse")
gradients = 2/m * tf.matmul(tf.transpose(X), error)
training_op = tf.assign(theta, theta - learning_rate * gradients)
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)


init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    for epoch in range(n_epochs):
Implementing Gradient Descent
|
237
Download from finelybook www.finelybook.com


if epoch % 100 == 0:
            print("Epoch", epoch, "MSE =", mse.eval())
        sess.run(training_op)
    best_theta = theta.eval()
Using autodiff
The preceding code works fine, but it requires mathematically deriving the gradients
from the cost function (MSE). In the case of Linear Regression, it is reasonably easy,
but if you had to do this with deep neural networks you would get quite a headache:
it would be tedious and error-prone. You could use symbolic differentiation to auto‐
matically find the equations for the partial derivatives for you, but the resulting code
would not necessarily be very efficient.
To understand why, consider the function f(x)= exp(exp(exp(x))). If you know calcu‐
lus, you can figure out its derivative f′(x) = exp(x) × exp(exp(x)) × exp(exp(exp(x))).
If you code f(x) and f′(x) separately and exactly as they appear, your code will not be
as efficient as it could be. A more efficient solution would be to write a function that


as efficient as it could be. A more efficient solution would be to write a function that
first computes exp(x), then exp(exp(x)), then exp(exp(exp(x))), and returns all three.
This gives you f(x) directly (the third term), and if you need the derivative you can
just multiply all three terms and you are done. With the naïve approach you would
have had to call the exp function nine times to compute both f(x) and f′(x). With this
approach you just need to call it three times.
It gets worse when your function is defined by some arbitrary code. Can you find the
equation (or the code) to compute the partial derivatives of the following function?
Hint: don’t even try.
def my_func(a, b):
    z = 0
    for i in range(100):
        z = a * np.cos(z + i) + z * np.sin(b - i)
    return z
Fortunately, TensorFlow’s autodiff feature comes to the rescue: it can automatically
and efficiently compute the gradients for you. Simply replace the gradients = ...


and efficiently compute the gradients for you. Simply replace the gradients = ...
line in the Gradient Descent code in the previous section with the following line, and
the code will continue to work just fine:
gradients = tf.gradients(mse, [theta])[0]
The gradients() function takes an op (in this case mse) and a list of variables (in this
case just theta), and it creates a list of ops (one per variable) to compute the gradi‐
ents of the op with regards to each variable. So the gradients node will compute the
gradient vector of the MSE with regards to theta.
238
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


There are four main approaches to computing gradients automatically. They are sum‐
marized in Table 9-2. TensorFlow uses reverse-mode autodiff, which is perfect (effi‐
cient and accurate) when there are many inputs and few outputs, as is often the case
in neural networks. It computes all the partial derivatives of the outputs with regards
to all the inputs in just noutputs + 1 graph traversals.
Table 9-2. Main solutions to compute gradients automatically
Technique
Nb of graph traversals to
compute all gradients
Accuracy
Supports
arbitrary code
Comment
Numerical differentiation ninputs + 1
Low
Yes
Trivial to implement
Symbolic differentiation
N/A
High
No
Builds a very different graph
Forward-mode autodiff
ninputs
High
Yes
Uses dual numbers
Reverse-mode autodiff
noutputs + 1
High
Yes
Implemented by TensorFlow
If you are interested in how this magic works, check out Appendix D.
Using an Optimizer
So TensorFlow computes the gradients for you. But it gets even easier: it also provides


So TensorFlow computes the gradients for you. But it gets even easier: it also provides
a number of optimizers out of the box, including a Gradient Descent optimizer. You
can simply replace the preceding gradients = ... and training_op = ... lines
with the following code, and once again everything will just work fine:
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(mse)
If you want to use a different type of optimizer, you just need to change one line. For
example, you can use a momentum optimizer (which often converges much faster
than Gradient Descent; see Chapter 11) by defining the optimizer like this:
optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate,
                                       momentum=0.9)
Feeding Data to the Training Algorithm
Let’s try to modify the previous code to implement Mini-batch Gradient Descent. For


Let’s try to modify the previous code to implement Mini-batch Gradient Descent. For
this, we need a way to replace X and y at every iteration with the next mini-batch. The
simplest way to do this is to use placeholder nodes. These nodes are special because
they don’t actually perform any computation, they just output the data you tell them
to output at runtime. They are typically used to pass the training data to TensorFlow
during training. If you don’t specify a value at runtime for a placeholder, you get an
exception.
To create a placeholder node, you must call the placeholder() function and specify
the output tensor’s data type. Optionally, you can also specify its shape, if you want to
Feeding Data to the Training Algorithm
|
239
Download from finelybook www.finelybook.com


enforce it. If you specify None for a dimension, it means “any size.” For example, the
following code creates a placeholder node A, and also a node B = A + 5. When we
evaluate B, we pass a feed_dict to the eval() method that specifies the value of A.
Note that A must have rank 2 (i.e., it must be two-dimensional) and there must be
three columns (or else an exception is raised), but it can have any number of rows.
>>> A = tf.placeholder(tf.float32, shape=(None, 3))
>>> B = A + 5
>>> with tf.Session() as sess:
...     B_val_1 = B.eval(feed_dict={A: [[1, 2, 3]]})
...     B_val_2 = B.eval(feed_dict={A: [[4, 5, 6], [7, 8, 9]]})
...
>>> print(B_val_1)
[[ 6.  7.  8.]]
>>> print(B_val_2)
[[  9.  10.  11.]
 [ 12.  13.  14.]]
You can actually feed the output of any operations, not just place‐
holders. In this case TensorFlow does not try to evaluate these
operations; it uses the values you feed it.
To implement Mini-batch Gradient Descent, we only need to tweak the existing code


To implement Mini-batch Gradient Descent, we only need to tweak the existing code
slightly. First change the definition of X and y in the construction phase to make them
placeholder nodes:
X = tf.placeholder(tf.float32, shape=(None, n + 1), name="X")
y = tf.placeholder(tf.float32, shape=(None, 1), name="y")
Then define the batch size and compute the total number of batches:
batch_size = 100
n_batches = int(np.ceil(m / batch_size))
Finally, in the execution phase, fetch the mini-batches one by one, then provide the
value of X and y via the feed_dict parameter when evaluating a node that depends
on either of them.
def fetch_batch(epoch, batch_index, batch_size):
    [...] # load the data from disk
    return X_batch, y_batch
with tf.Session() as sess:
    sess.run(init)
    for epoch in range(n_epochs):
        for batch_index in range(n_batches):
            X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)


X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
240
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


best_theta = theta.eval()
We don’t need to pass the value of X and y when evaluating theta
since it does not depend on either of them.
Saving and Restoring Models
Once you have trained your model, you should save its parameters to disk so you can
come back to it whenever you want, use it in another program, compare it to other
models, and so on. Moreover, you probably want to save checkpoints at regular inter‐
vals during training so that if your computer crashes during training you can con‐
tinue from the last checkpoint rather than start over from scratch.
TensorFlow makes saving and restoring a model very easy. Just create a Saver node at
the end of the construction phase (after all variable nodes are created); then, in the
execution phase, just call its save() method whenever you want to save the model,
passing it the session and path of the checkpoint file:
[...]
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0), name="theta")
[...]


[...]
theta = tf.Variable(tf.random_uniform([n + 1, 1], -1.0, 1.0), name="theta")
[...]
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session() as sess:
    sess.run(init)
    for epoch in range(n_epochs):
        if epoch % 100 == 0:  # checkpoint every 100 epochs
            save_path = saver.save(sess, "/tmp/my_model.ckpt")
        sess.run(training_op)
    best_theta = theta.eval()
    save_path = saver.save(sess, "/tmp/my_model_final.ckpt")
Restoring a model is just as easy: you create a Saver at the end of the construction
phase just like before, but then at the beginning of the execution phase, instead of ini‐
tializing the variables using the init node, you call the restore() method of the
Saver object:
with tf.Session() as sess:
    saver.restore(sess, "/tmp/my_model_final.ckpt")
    [...]
Saving and Restoring Models
|
241
Download from finelybook www.finelybook.com


By default a Saver saves and restores all variables under their own name, but if you
need more control, you can specify which variables to save or restore, and what
names to use. For example, the following Saver will save or restore only the theta
variable under the name weights:
saver = tf.train.Saver({"weights": theta})
Visualizing the Graph and Training Curves Using
TensorBoard
So now we have a computation graph that trains a Linear Regression model using
Mini-batch Gradient Descent, and we are saving checkpoints at regular intervals.
Sounds sophisticated, doesn’t it? However, we are still relying on the print() func‐
tion to visualize progress during training. There is a better way: enter TensorBoard. If
you feed it some training stats, it will display nice interactive visualizations of these
stats in your web browser (e.g., learning curves). You can also provide it the graph’s
definition and it will give you a great interface to browse through it. This is very use‐


definition and it will give you a great interface to browse through it. This is very use‐
ful to identify errors in the graph, to find bottlenecks, and so on.
The first step is to tweak your program a bit so it writes the graph definition and
some training stats—for example, the training error (MSE)—to a log directory that
TensorBoard will read from. You need to use a different log directory every time you
run your program, or else TensorBoard will merge stats from different runs, which
will mess up the visualizations. The simplest solution for this is to include a time‐
stamp in the log directory name. Add the following code at the beginning of the pro‐
gram:
from datetime import datetime
now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
root_logdir = "tf_logs"
logdir = "{}/run-{}/".format(root_logdir, now)
Next, add the following code at the very end of the construction phase:
mse_summary = tf.summary.scalar('MSE', mse)


mse_summary = tf.summary.scalar('MSE', mse)
file_writer = tf.summary.FileWriter(logdir, tf.get_default_graph())
The first line creates a node in the graph that will evaluate the MSE value and write it
to a TensorBoard-compatible binary log string called a summary. The second line cre‐
ates a FileWriter that you will use to write summaries to logfiles in the log directory.
The first parameter indicates the path of the log directory (in this case something like
tf_logs/run-20160906091959/, relative to the current directory). The second
(optional) parameter is the graph you want to visualize. Upon creation, the File
Writer creates the log directory if it does not already exist (and its parent directories
if needed), and writes the graph definition in a binary logfile called an events file.
242
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com


Next you need to update the execution phase to evaluate the mse_summary node regu‐
larly during training (e.g., every 10 mini-batches). This will output a summary that
you can then write to the events file using the file_writer. Here is the updated code:
    [...]
    for batch_index in range(n_batches):
        X_batch, y_batch = fetch_batch(epoch, batch_index, batch_size)
        if batch_index % 10 == 0:
            summary_str = mse_summary.eval(feed_dict={X: X_batch, y: y_batch})
            step = epoch * n_batches + batch_index
            file_writer.add_summary(summary_str, step)
        sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
    [...]
Avoid logging training stats at every single training step, as this
would significantly slow down training.
Finally, you want to close the FileWriter at the end of the program:
file_writer.close()
Now run this program: it will create the log directory and write an events file in this


Now run this program: it will create the log directory and write an events file in this
directory, containing both the graph definition and the MSE values. Open up a shell
and go to your working directory, then type ls -l tf_logs/run* to list the contents
of the log directory:
$ cd $ML_PATH               # Your ML working directory (e.g., $HOME/ml)
$ ls -l tf_logs/run*
total 40
-rw-r--r-- 1 ageron staff 18620 Sep 6 11:10 events.out.tfevents.1472553182.mymac
If you run the program a second time, you should see a second directory in the
tf_logs/ directory:
$ ls -l tf_logs/
total 0
drwxr-xr-x  3 ageron  staff  102 Sep  6 10:07 run-20160906091959
drwxr-xr-x  3 ageron  staff  102 Sep  6 10:22 run-20160906092202
Great! Now it’s time to fire up the TensorBoard server. You need to activate your vir‐
tualenv environment if you created one, then start the server by running the tensor
board command, pointing it to the root log directory. This starts the TensorBoard


board command, pointing it to the root log directory. This starts the TensorBoard
web server, listening on port 6006 (which is “goog” written upside down):
$ source env/bin/activate
$ tensorboard --logdir tf_logs/
Starting TensorBoard  on port 6006
(You can navigate to http://0.0.0.0:6006)
Visualizing the Graph and Training Curves Using TensorBoard
|
243
Download from finelybook www.finelybook.com


Next open a browser and go to http://0.0.0.0:6006/ (or http://localhost:6006/). Wel‐
come to TensorBoard! In the Events tab you should see MSE on the right. If you click
on it, you will see a plot of the MSE during training, for both runs (Figure 9-3). You
can check or uncheck the runs you want to see, zoom in or out, hover over the curve
to get details, and so on.
Figure 9-3. Visualizing training stats using TensorBoard
Now click on the Graphs tab. You should see the graph shown in Figure 9-4.
To reduce clutter, the nodes that have many edges (i.e., connections to other nodes)
are separated out to an auxiliary area on the right (you can move a node back and
forth between the main graph and the auxiliary area by right-clicking on it). Some
parts of the graph are also collapsed by default. For example, try hovering over the
gradients node, then click on the ⊕ icon to expand this subgraph. Next, in this sub‐
graph, try expanding the mse_grad subgraph.
244
|


graph, try expanding the mse_grad subgraph.
244
|
Chapter 9: Up and Running with TensorFlow
Download from finelybook www.finelybook.com