linear: all the weights for the high-degree polynomial features are equal to zero. In
other words, Lasso Regression automatically performs feature selection and outputs a
sparse model (i.e., with few nonzero feature weights).
You can get a sense of why this is the case by looking at Figure 4-19: on the top-left
plot, the background contours (ellipses) represent an unregularized MSE cost func‐
tion (α = 0), and the white circles show the Batch Gradient Descent path with that
cost function. The foreground contours (diamonds) represent the ℓ1 penalty, and the
triangles show the BGD path for this penalty only (α → ∞). Notice how the path first
130
|
Chapter 4: Training Models
Download from finelybook www.finelybook.com


15 You can think of a subgradient vector at a nondifferentiable point as an intermediate vector between the gra‐
dient vectors around that point.
reaches θ1 = 0, then rolls down a gutter until it reaches θ2 = 0. On the top-right plot,
the contours represent the same cost function plus an ℓ1 penalty with α = 0.5. The
global minimum is on the θ2 = 0 axis. BGD first reaches θ2 = 0, then rolls down the
gutter until it reaches the global minimum. The two bottom plots show the same
thing but uses an ℓ2 penalty instead. The regularized minimum is closer to θ = 0 than
the unregularized minimum, but the weights do not get fully eliminated.
Figure 4-19. Lasso versus Ridge regularization
On the Lasso cost function, the BGD path tends to bounce across
the gutter toward the end. This is because the slope changes
abruptly at θ2 = 0. You need to gradually reduce the learning rate in
order to actually converge to the global minimum.

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


order to actually converge to the global minimum.
The Lasso cost function is not differentiable at θi = 0 (for i = 1, 2, ⋯, n), but Gradient
Descent still works fine if you use a subgradient vector g15 instead when any θi = 0.
Equation 4-11 shows a subgradient vector equation you can use for Gradient Descent
with the Lasso cost function.
Regularized Linear Models
|
131
Download from finelybook www.finelybook.com


Equation 4-11. Lasso Regression subgradient vector
g θ, J = ∇θ MSE θ + α
sign θ1
sign θ2
⋮
sign θn
where sign θi =
−1 if θi < 0
0
if θi = 0
+1 if θi > 0
Here is a small Scikit-Learn example using the Lasso class. Note that you could
instead use an SGDRegressor(penalty="l1").
>>> from sklearn.linear_model import Lasso
>>> lasso_reg = Lasso(alpha=0.1)
>>> lasso_reg.fit(X, y)
>>> lasso_reg.predict([[1.5]])
array([ 1.53788174])
Elastic Net
Elastic Net is a middle ground between Ridge Regression and Lasso Regression. The
regularization term is a simple mix of both Ridge and Lasso’s regularization terms,
and you can control the mix ratio r. When r = 0, Elastic Net is equivalent to Ridge
Regression, and when r = 1, it is equivalent to Lasso Regression (see Equation 4-12).
Equation 4-12. Elastic Net cost function
J θ = MSE θ + rα ∑
i = 1
n
θi + 1 −r
2
α ∑
i = 1
n
θi
2
So when should you use Linear Regression, Ridge, Lasso, or Elastic Net? It is almost


i = 1
n
θi
2
So when should you use Linear Regression, Ridge, Lasso, or Elastic Net? It is almost
always preferable to have at least a little bit of regularization, so generally you should
avoid plain Linear Regression. Ridge is a good default, but if you suspect that only a
few features are actually useful, you should prefer Lasso or Elastic Net since they tend
to reduce the useless features’ weights down to zero as we have discussed. In general,
Elastic Net is preferred over Lasso since Lasso may behave erratically when the num‐
ber of features is greater than the number of training instances or when several fea‐
tures are strongly correlated.
Here is a short example using Scikit-Learn’s ElasticNet (l1_ratio corresponds to
the mix ratio r):
>>> from sklearn.linear_model import ElasticNet
>>> elastic_net = ElasticNet(alpha=0.1, l1_ratio=0.5)
>>> elastic_net.fit(X, y)
>>> elastic_net.predict([[1.5]])
array([ 1.54333232])
132
|
Chapter 4: Training Models


>>> elastic_net.predict([[1.5]])
array([ 1.54333232])
132
|
Chapter 4: Training Models
Download from finelybook www.finelybook.com


Early Stopping
A very different way to regularize iterative learning algorithms such as Gradient
Descent is to stop training as soon as the validation error reaches a minimum. This is
called early stopping. Figure 4-20 shows a complex model (in this case a high-degree
Polynomial Regression model) being trained using Batch Gradient Descent. As the
epochs go by, the algorithm learns and its prediction error (RMSE) on the training set
naturally goes down, and so does its prediction error on the validation set. However,
after a while the validation error stops decreasing and actually starts to go back up.
This indicates that the model has started to overfit the training data. With early stop‐
ping you just stop training as soon as the validation error reaches the minimum. It is
such a simple and efficient regularization technique that Geoffrey Hinton called it a
“beautiful free lunch.”
Figure 4-20. Early stopping regularization


“beautiful free lunch.”
Figure 4-20. Early stopping regularization
With Stochastic and Mini-batch Gradient Descent, the curves are
not so smooth, and it may be hard to know whether you have
reached the minimum or not. One solution is to stop only after the
validation error has been above the minimum for some time (when
you are confident that the model will not do any better), then roll
back the model parameters to the point where the validation error
was at a minimum.
Here is a basic implementation of early stopping:
from sklearn.base import clone
Regularized Linear Models
|
133
Download from finelybook www.finelybook.com


sgd_reg = SGDRegressor(n_iter=1, warm_start=True, penalty=None,
                       learning_rate="constant", eta0=0.0005)
minimum_val_error = float("inf")
best_epoch = None
best_model = None
for epoch in range(1000):
    sgd_reg.fit(X_train_poly_scaled, y_train)  # continues where it left off
    y_val_predict = sgd_reg.predict(X_val_poly_scaled)
    val_error = mean_squared_error(y_val_predict, y_val)
    if val_error < minimum_val_error:
        minimum_val_error = val_error
        best_epoch = epoch
        best_model = clone(sgd_reg)
Note that with warm_start=True, when the fit() method is called, it just continues
training where it left off instead of restarting from scratch.
Logistic Regression
As we discussed in Chapter 1, some regression algorithms can be used for classifica‐
tion as well (and vice versa). Logistic Regression (also called Logit Regression) is com‐
monly used to estimate the probability that an instance belongs to a particular class


monly used to estimate the probability that an instance belongs to a particular class
(e.g., what is the probability that this email is spam?). If the estimated probability is
greater than 50%, then the model predicts that the instance belongs to that class
(called the positive class, labeled “1”), or else it predicts that it does not (i.e., it
belongs to the negative class, labeled “0”). This makes it a binary classifier.
Estimating Probabilities
So how does it work? Just like a Linear Regression model, a Logistic Regression
model computes a weighted sum of the input features (plus a bias term), but instead
of outputting the result directly like the Linear Regression model does, it outputs the
logistic of this result (see Equation 4-13).
Equation 4-13. Logistic Regression model estimated probability (vectorized form)
p = hθ �= σ θT · �
The logistic—also called the logit, noted σ(·)—is a sigmoid function (i.e., S-shaped)


The logistic—also called the logit, noted σ(·)—is a sigmoid function (i.e., S-shaped)
that outputs a number between 0 and 1. It is defined as shown in Equation 4-14 and
Figure 4-21.
134
|
Chapter 4: Training Models
Download from finelybook www.finelybook.com


Equation 4-14. Logistic function
σ t =
1
1 + exp
−t
Figure 4-21. Logistic function
Once the Logistic Regression model has estimated the probability p = hθ(x) that an
instance x belongs to the positive class, it can make its prediction ŷ easily (see Equa‐
tion 4-15).
Equation 4-15. Logistic Regression model prediction
y = 0 if p < 0 . 5,
1 if p ≥0 . 5 .
Notice that σ(t) < 0.5 when t < 0, and σ(t) ≥ 0.5 when t ≥ 0, so a Logistic Regression
model predicts 1 if θT · x is positive, and 0 if it is negative.
Training and Cost Function
Good, now you know how a Logistic Regression model estimates probabilities and
makes predictions. But how is it trained? The objective of training is to set the param‐
eter vector θ so that the model estimates high probabilities for positive instances (y =
1) and low probabilities for negative instances (y = 0). This idea is captured by the
cost function shown in Equation 4-16 for a single training instance x.


cost function shown in Equation 4-16 for a single training instance x.
Equation 4-16. Cost function of a single training instance
c θ =
−log p
if y = 1,
−log 1 −p if y = 0 .
This cost function makes sense because – log(t) grows very large when t approaches
0, so the cost will be large if the model estimates a probability close to 0 for a positive
Logistic Regression
|
135
Download from finelybook www.finelybook.com


instance, and it will also be very large if the model estimates a probability close to 1
for a negative instance. On the other hand, – log(t) is close to 0 when t is close to 1, so
the cost will be close to 0 if the estimated probability is close to 0 for a negative
instance or close to 1 for a positive instance, which is precisely what we want.
The cost function over the whole training set is simply the average cost over all train‐
ing instances. It can be written in a single expression (as you can verify easily), called
the log loss, shown in Equation 4-17.
Equation 4-17. Logistic Regression cost function (log loss)
J θ = −1
m ∑
i = 1
m
y i log p i
+ 1 −y i log 1 −p i
The bad news is that there is no known closed-form equation to compute the value of
θ that minimizes this cost function (there is no equivalent of the Normal Equation).
But the good news is that this cost function is convex, so Gradient Descent (or any


But the good news is that this cost function is convex, so Gradient Descent (or any
other optimization algorithm) is guaranteed to find the global minimum (if the learn‐
ing rate is not too large and you wait long enough). The partial derivatives of the cost
function with regards to the jth model parameter θj is given by Equation 4-18.
Equation 4-18. Logistic cost function partial derivatives
∂
∂θj
J θ = 1
m ∑
i = 1
m
σ θT · �i
−y i xj
i
This equation looks very much like Equation 4-5: for each instance it computes the
prediction error and multiplies it by the jth feature value, and then it computes the
average over all training instances. Once you have the gradient vector containing all
the partial derivatives you can use it in the Batch Gradient Descent algorithm. That’s
it: you now know how to train a Logistic Regression model. For Stochastic GD you
would of course just take one instance at a time, and for Mini-batch GD you would
use a mini-batch at a time.
Decision Boundaries


use a mini-batch at a time.
Decision Boundaries
Let’s use the iris dataset to illustrate Logistic Regression. This is a famous dataset that
contains the sepal and petal length and width of 150 iris flowers of three different
species: Iris-Setosa, Iris-Versicolor, and Iris-Virginica (see Figure 4-22).
136
|
Chapter 4: Training Models
Download from finelybook www.finelybook.com


16 Photos reproduced from the corresponding Wikipedia pages. Iris-Virginica photo by Frank Mayfield (Crea‐
tive Commons BY-SA 2.0), Iris-Versicolor photo by D. Gordon E. Robertson (Creative Commons BY-SA 3.0),
and Iris-Setosa photo is public domain.
Figure 4-22. Flowers of three iris plant species16
Let’s try to build a classifier to detect the Iris-Virginica type based only on the petal
width feature. First let’s load the data:
>>> from sklearn import datasets
>>> iris = datasets.load_iris()
>>> list(iris.keys())
['data', 'target_names', 'feature_names', 'target', 'DESCR']
>>> X = iris["data"][:, 3:]  # petal width
>>> y = (iris["target"] == 2).astype(np.int)  # 1 if Iris-Virginica, else 0
Now let’s train a Logistic Regression model:
from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression()
log_reg.fit(X, y)
Let’s look at the model’s estimated probabilities for flowers with petal widths varying
from 0 to 3 cm (Figure 4-23):


from 0 to 3 cm (Figure 4-23):
X_new = np.linspace(0, 3, 1000).reshape(-1, 1)
y_proba = log_reg.predict_proba(X_new)
plt.plot(X_new, y_proba[:, 1], "g-", label="Iris-Virginica")
plt.plot(X_new, y_proba[:, 0], "b--", label="Not Iris-Virginica")
# + more Matplotlib code to make the image look pretty
Logistic Regression
|
137
Download from finelybook www.finelybook.com


17 It is the the set of points x such that θ0 + θ1x1 + θ2x2 = 0, which defines a straight line.
Figure 4-23. Estimated probabilities and decision boundary
The petal width of Iris-Virginica flowers (represented by triangles) ranges from 1.4
cm to 2.5 cm, while the other iris flowers (represented by squares) generally have a
smaller petal width, ranging from 0.1 cm to 1.8 cm. Notice that there is a bit of over‐
lap. Above about 2 cm the classifier is highly confident that the flower is an Iris-
Virginica (it outputs a high probability to that class), while below 1 cm it is highly
confident that it is not an Iris-Virginica (high probability for the “Not Iris-Virginica”
class). In between these extremes, the classifier is unsure. However, if you ask it to
predict the class (using the predict() method rather than the predict_proba()
method), it will return whichever class is the most likely. Therefore, there is a decision


method), it will return whichever class is the most likely. Therefore, there is a decision
boundary at around 1.6 cm where both probabilities are equal to 50%: if the petal
width is higher than 1.6 cm, the classifier will predict that the flower is an Iris-
Virginica, or else it will predict that it is not (even if it is not very confident):
>>> log_reg.predict([[1.7], [1.5]])
array([1, 0])
Figure 4-24 shows the same dataset but this time displaying two features: petal width
and length. Once trained, the Logistic Regression classifier can estimate the probabil‐
ity that a new flower is an Iris-Virginica based on these two features. The dashed line
represents the points where the model estimates a 50% probability: this is the model’s
decision boundary. Note that it is a linear boundary.17 Each parallel line represents the
points where the model outputs a specific probability, from 15% (bottom left) to 90%
(top right). All the flowers beyond the top-right line have an over 90% chance of


(top right). All the flowers beyond the top-right line have an over 90% chance of
being Iris-Virginica according to the model.
138
|
Chapter 4: Training Models
Download from finelybook www.finelybook.com


Figure 4-24. Linear decision boundary
Just like the other linear models, Logistic Regression models can be regularized using
ℓ1 or ℓ2 penalties. Scitkit-Learn actually adds an ℓ2 penalty by default.
The hyperparameter controlling the regularization strength of a
Scikit-Learn LogisticRegression model is not alpha (as in other
linear models), but its inverse: C. The higher the value of C, the less
the model is regularized.
Softmax Regression
The Logistic Regression model can be generalized to support multiple classes directly,
without having to train and combine multiple binary classifiers (as discussed in
Chapter 3). This is called Softmax Regression, or Multinomial Logistic Regression.
The idea is quite simple: when given an instance x, the Softmax Regression model
first computes a score sk(x) for each class k, then estimates the probability of each
class by applying the softmax function (also called the normalized exponential) to the


class by applying the softmax function (also called the normalized exponential) to the
scores. The equation to compute sk(x) should look familiar, as it is just like the equa‐
tion for Linear Regression prediction (see Equation 4-19).
Equation 4-19. Softmax score for class k
sk �= θk
T · �
Note that each class has its own dedicated parameter vector θk. All these vectors are
typically stored as rows in a parameter matrix Θ.
Once you have computed the score of every class for the instance x, you can estimate
the probability pk that the instance belongs to class k by running the scores through
Logistic Regression
|
139
Download from finelybook www.finelybook.com


the softmax function (Equation 4-20): it computes the exponential of every score,
then normalizes them (dividing by the sum of all the exponentials).
Equation 4-20. Softmax function
pk = σ ��
k =
exp sk �
∑j = 1
K
exp sj �
• K is the number of classes.
• s(x) is a vector containing the scores of each class for the instance x.
• σ(s(x))k is the estimated probability that the instance x belongs to class k given
the scores of each class for that instance.
Just like the Logistic Regression classifier, the Softmax Regression classifier predicts
the class with the highest estimated probability (which is simply the class with the
highest score), as shown in Equation 4-21.
Equation 4-21. Softmax Regression classifier prediction
y = argmax
k
σ ��
k = argmax
k
sk �= argmax
k
θk
T · �
• The argmax operator returns the value of a variable that maximizes a function. In
this equation, it returns the value of k that maximizes the estimated probability
σ(s(x))k.


this equation, it returns the value of k that maximizes the estimated probability
σ(s(x))k.
The Softmax Regression classifier predicts only one class at a time
(i.e., it is multiclass, not multioutput) so it should be used only with
mutually exclusive classes such as different types of plants. You
cannot use it to recognize multiple people in one picture.
Now that you know how the model estimates probabilities and makes predictions,
let’s take a look at training. The objective is to have a model that estimates a high
probability for the target class (and consequently a low probability for the other
classes). Minimizing the cost function shown in Equation 4-22, called the cross
entropy, should lead to this objective because it penalizes the model when it estimates
a low probability for a target class. Cross entropy is frequently used to measure how
well a set of estimated class probabilities match the target classes (we will use it again
several times in the following chapters).
140


several times in the following chapters).
140
|
Chapter 4: Training Models
Download from finelybook www.finelybook.com


Equation 4-22. Cross entropy cost function
J Θ = −1
m ∑
i = 1
m
∑
k = 1
K
yk
i log pk
i
• yk
i  is equal to 1 if the target class for the ith instance is k; otherwise, it is equal to
0.
Notice that when there are just two classes (K = 2), this cost function is equivalent to
the Logistic Regression’s cost function (log loss; see Equation 4-17).
Cross Entropy
Cross entropy originated from information theory. Suppose you want to efficiently
transmit information about the weather every day. If there are eight options (sunny,
rainy, etc.), you could encode each option using 3 bits since 23 = 8. However, if you
think it will be sunny almost every day, it would be much more efficient to code
“sunny” on just one bit (0) and the other seven options on 4 bits (starting with a 1).
Cross entropy measures the average number of bits you actually send per option. If
your assumption about the weather is perfect, cross entropy will just be equal to the


your assumption about the weather is perfect, cross entropy will just be equal to the
entropy of the weather itself (i.e., its intrinsic unpredictability). But if your assump‐
tions are wrong (e.g., if it rains often), cross entropy will be greater by an amount
called the Kullback–Leibler divergence.
The cross entropy between two probability distributions p and q is defined as
H p, q = −∑x p x log q x  (at least when the distributions are discrete).
The gradient vector of this cost function with regards to θk is given by Equation 4-23:
Equation 4-23. Cross entropy gradient vector for class k
∇θk J Θ = 1
m ∑
i = 1
m
pk
i −yk
i �i
Now you can compute the gradient vector for every class, then use Gradient Descent
(or any other optimization algorithm) to find the parameter matrix Θ that minimizes
the cost function.
Let’s use Softmax Regression to classify the iris flowers into all three classes. Scikit-
Learn’s LogisticRegression uses one-versus-all by default when you train it on more


Learn’s LogisticRegression uses one-versus-all by default when you train it on more
than two classes, but you can set the multi_class hyperparameter to "multinomial"
to switch it to Softmax Regression instead. You must also specify a solver that sup‐
ports Softmax Regression, such as the "lbfgs" solver (see Scikit-Learn’s documenta‐
Logistic Regression
|
141
Download from finelybook www.finelybook.com


tion for more details). It also applies ℓ2 regularization by default, which you can
control using the hyperparameter C.
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]
softmax_reg = LogisticRegression(multi_class="multinomial",solver="lbfgs", C=10)
softmax_reg.fit(X, y)
So the next time you find an iris with 5 cm long and 2 cm wide petals, you can ask
your model to tell you what type of iris it is, and it will answer Iris-Virginica (class 2)
with 94.2% probability (or Iris-Versicolor with 5.8% probability):
>>> softmax_reg.predict([[5, 2]])
array([2])
>>> softmax_reg.predict_proba([[5, 2]])
array([[  6.33134078e-07,   5.75276067e-02,   9.42471760e-01]])
Figure 4-25 shows the resulting decision boundaries, represented by the background
colors. Notice that the decision boundaries between any two classes are linear. The
figure also shows the probabilities for the Iris-Versicolor class, represented by the


figure also shows the probabilities for the Iris-Versicolor class, represented by the
curved lines (e.g., the line labeled with 0.450 represents the 45% probability bound‐
ary). Notice that the model can predict a class that has an estimated probability below
50%. For example, at the point where all decision boundaries meet, all classes have an
equal estimated probability of 33%.
Figure 4-25. Softmax Regression decision boundaries
Exercises
1. What Linear Regression training algorithm can you use if you have a training set
with millions of features?
2. Suppose the features in your training set have very different scales. What algo‐
rithms might suffer from this, and how? What can you do about it?
142
|
Chapter 4: Training Models
Download from finelybook www.finelybook.com


3. Can Gradient Descent get stuck in a local minimum when training a Logistic
Regression model?
4. Do all Gradient Descent algorithms lead to the same model provided you let
them run long enough?
5. Suppose you use Batch Gradient Descent and you plot the validation error at
every epoch. If you notice that the validation error consistently goes up, what is
likely going on? How can you fix this?
6. Is it a good idea to stop Mini-batch Gradient Descent immediately when the vali‐
dation error goes up?
7. Which Gradient Descent algorithm (among those we discussed) will reach the
vicinity of the optimal solution the fastest? Which will actually converge? How
can you make the others converge as well?
8. Suppose you are using Polynomial Regression. You plot the learning curves and
you notice that there is a large gap between the training error and the validation
error. What is happening? What are three ways to solve this?


error. What is happening? What are three ways to solve this?
9. Suppose you are using Ridge Regression and you notice that the training error
and the validation error are almost equal and fairly high. Would you say that the
model suffers from high bias or high variance? Should you increase the regulari‐
zation hyperparameter α or reduce it?
10. Why would you want to use:
• Ridge Regression instead of Linear Regression?
• Lasso instead of Ridge Regression?
• Elastic Net instead of Lasso?
11. Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime.
Should you implement two Logistic Regression classifiers or one Softmax Regres‐
sion classifier?
12. Implement Batch Gradient Descent with early stopping for Softmax Regression
(without using Scikit-Learn).
Solutions to these exercises are available in Appendix A.
Exercises
|
143
Download from finelybook www.finelybook.com


Download from finelybook www.finelybook.com


CHAPTER 5
Support Vector Machines
A Support Vector Machine (SVM) is a very powerful and versatile Machine Learning
model, capable of performing linear or nonlinear classification, regression, and even
outlier detection. It is one of the most popular models in Machine Learning, and any‐
one interested in Machine Learning should have it in their toolbox. SVMs are partic‐
ularly well suited for classification of complex but small- or medium-sized datasets.
This chapter will explain the core concepts of SVMs, how to use them, and how they
work.
Linear SVM Classification
The fundamental idea behind SVMs is best explained with some pictures. Figure 5-1
shows part of the iris dataset that was introduced at the end of Chapter 4. The two
classes can clearly be separated easily with a straight line (they are linearly separable).
The left plot shows the decision boundaries of three possible linear classifiers. The
model whose decision boundary is represented by the dashed line is so bad that it


model whose decision boundary is represented by the dashed line is so bad that it
does not even separate the classes properly. The other two models work perfectly on
this training set, but their decision boundaries come so close to the instances that
these models will probably not perform as well on new instances. In contrast, the
solid line in the plot on the right represents the decision boundary of an SVM classi‐
fier; this line not only separates the two classes but also stays as far away from the
closest training instances as possible. You can think of an SVM classifier as fitting the
widest possible street (represented by the parallel dashed lines) between the classes.
This is called large margin classification.
145
Download from finelybook www.finelybook.com


Figure 5-1. Large margin classification
Notice that adding more training instances “off the street” will not affect the decision
boundary at all: it is fully determined (or “supported”) by the instances located on the
edge of the street. These instances are called the support vectors (they are circled in
Figure 5-1).
SVMs are sensitive to the feature scales, as you can see in
Figure 5-2: on the left plot, the vertical scale is much larger than the
horizontal scale, so the widest possible street is close to horizontal.
After feature scaling (e.g., using Scikit-Learn’s StandardScaler),
the decision boundary looks much better (on the right plot).
Figure 5-2. Sensitivity to feature scales
Soft Margin Classification
If we strictly impose that all instances be off the street and on the right side, this is
called hard margin classification. There are two main issues with hard margin classifi‐
cation. First, it only works if the data is linearly separable, and second it is quite sensi‐


cation. First, it only works if the data is linearly separable, and second it is quite sensi‐
tive to outliers. Figure 5-3 shows the iris dataset with just one additional outlier: on
the left, it is impossible to find a hard margin, and on the right the decision boundary
ends up very different from the one we saw in Figure 5-1 without the outlier, and it
will probably not generalize as well.
146
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


Figure 5-3. Hard margin sensitivity to outliers
To avoid these issues it is preferable to use a more flexible model. The objective is to
find a good balance between keeping the street as large as possible and limiting the
margin violations (i.e., instances that end up in the middle of the street or even on the
wrong side). This is called soft margin classification.
In Scikit-Learn’s SVM classes, you can control this balance using the C hyperparame‐
ter: a smaller C value leads to a wider street but more margin violations. Figure 5-4
shows the decision boundaries and margins of two soft margin SVM classifiers on a
nonlinearly separable dataset. On the left, using a high C value the classifier makes
fewer margin violations but ends up with a smaller margin. On the right, using a low
C value the margin is much larger, but many instances end up on the street. However,
it seems likely that the second classifier will generalize better: in fact even on this


it seems likely that the second classifier will generalize better: in fact even on this
training set it makes fewer prediction errors, since most of the margin violations are
actually on the correct side of the decision boundary.
Figure 5-4. Fewer margin violations versus large margin
If your SVM model is overfitting, you can try regularizing it by
reducing C.
The following Scikit-Learn code loads the iris dataset, scales the features, and then
trains a linear SVM model (using the LinearSVC class with C = 0.1 and the hinge loss
Linear SVM Classification
|
147
Download from finelybook www.finelybook.com


function, described shortly) to detect Iris-Virginica flowers. The resulting model is
represented on the right of Figure 5-4.
import numpy as np
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
iris = datasets.load_iris()
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = (iris["target"] == 2).astype(np.float64)  # Iris-Virginica
svm_clf = Pipeline((
        ("scaler", StandardScaler()),
        ("linear_svc", LinearSVC(C=1, loss="hinge")),
    ))
svm_clf.fit(X_scaled, y)
Then, as usual, you can use the model to make predictions:
>>> svm_clf.predict([[5.5, 1.7]])
array([ 1.])
Unlike Logistic Regression classifiers, SVM classifiers do not out‐
put probabilities for each class.
Alternatively, you could use the SVC class, using SVC(kernel="linear", C=1), but it
is much slower, especially with large training sets, so it is not recommended. Another


is much slower, especially with large training sets, so it is not recommended. Another
option is to use the SGDClassifier class, with SGDClassifier(loss="hinge",
alpha=1/(m*C)). This applies regular Stochastic Gradient Descent (see Chapter 4) to
train a linear SVM classifier. It does not converge as fast as the LinearSVC class, but it
can be useful to handle huge datasets that do not fit in memory (out-of-core train‐
ing), or to handle online classification tasks.
The LinearSVC class regularizes the bias term, so you should center
the training set first by subtracting its mean. This is automatic if
you scale the data using the StandardScaler. Moreover, make sure
you set the loss hyperparameter to "hinge", as it is not the default
value. Finally, for better performance you should set the dual
hyperparameter to False, unless there are more features than
training instances (we will discuss duality later in the chapter).
148
|
Chapter 5: Support Vector Machines


148
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


Nonlinear SVM Classification
Although linear SVM classifiers are efficient and work surprisingly well in many
cases, many datasets are not even close to being linearly separable. One approach to
handling nonlinear datasets is to add more features, such as polynomial features (as
you did in Chapter 4); in some cases this can result in a linearly separable dataset.
Consider the left plot in Figure 5-5: it represents a simple dataset with just one feature
x1. This dataset is not linearly separable, as you can see. But if you add a second fea‐
ture x2 = (x1)2, the resulting 2D dataset is perfectly linearly separable.
Figure 5-5. Adding features to make a dataset linearly separable
To implement this idea using Scikit-Learn, you can create a Pipeline containing a
PolynomialFeatures transformer (discussed in “Polynomial Regression” on page
121), followed by a StandardScaler and a LinearSVC. Let’s test this on the moons
dataset (see Figure 5-6):
from sklearn.datasets import make_moons


dataset (see Figure 5-6):
from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
polynomial_svm_clf = Pipeline((
        ("poly_features", PolynomialFeatures(degree=3)),
        ("scaler", StandardScaler()),
        ("svm_clf", LinearSVC(C=10, loss="hinge"))
    ))
polynomial_svm_clf.fit(X, y)
Nonlinear SVM Classification
|
149
Download from finelybook www.finelybook.com


Figure 5-6. Linear SVM classifier using polynomial features
Polynomial Kernel
Adding polynomial features is simple to implement and can work great with all sorts
of Machine Learning algorithms (not just SVMs), but at a low polynomial degree it
cannot deal with very complex datasets, and with a high polynomial degree it creates
a huge number of features, making the model too slow.
Fortunately, when using SVMs you can apply an almost miraculous mathematical
technique called the kernel trick (it is explained in a moment). It makes it possible to
get the same result as if you added many polynomial features, even with very high-
degree polynomials, without actually having to add them. So there is no combinato‐
rial explosion of the number of features since you don’t actually add any features. This
trick is implemented by the SVC class. Let’s test it on the moons dataset:
from sklearn.svm import SVC
poly_kernel_svm_clf = Pipeline((
        ("scaler", StandardScaler()),


from sklearn.svm import SVC
poly_kernel_svm_clf = Pipeline((
        ("scaler", StandardScaler()),
        ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
    ))
poly_kernel_svm_clf.fit(X, y)
This code trains an SVM classifier using a 3rd-degree polynomial kernel. It is repre‐
sented on the left of Figure 5-7. On the right is another SVM classifier using a 10th-
degree polynomial kernel. Obviously, if your model is overfitting, you might want to
150
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


reduce the polynomial degree. Conversely, if it is underfitting, you can try increasing
it. The hyperparameter coef0 controls how much the model is influenced by high-
degree polynomials versus low-degree polynomials.
Figure 5-7. SVM classifiers with a polynomial kernel
A common approach to find the right hyperparameter values is to
use grid search (see Chapter 2). It is often faster to first do a very
coarse grid search, then a finer grid search around the best values
found. Having a good sense of what each hyperparameter actually
does can also help you search in the right part of the hyperparame‐
ter space.
Adding Similarity Features
Another technique to tackle nonlinear problems is to add features computed using a
similarity function that measures how much each instance resembles a particular
landmark. For example, let’s take the one-dimensional dataset discussed earlier and
add two landmarks to it at x1 = –2 and x1 = 1 (see the left plot in Figure 5-8). Next,


add two landmarks to it at x1 = –2 and x1 = 1 (see the left plot in Figure 5-8). Next,
let’s define the similarity function to be the Gaussian Radial Basis Function (RBF)
with γ = 0.3 (see Equation 5-1).
Equation 5-1. Gaussian RBF
ϕγ �, ℓ= exp −γ∥�−ℓ∥2
It is a bell-shaped function varying from 0 (very far away from the landmark) to 1 (at
the landmark). Now we are ready to compute the new features. For example, let’s look
at the instance x1 = –1: it is located at a distance of 1 from the first landmark, and 2
from the second landmark. Therefore its new features are x2 = exp (–0.3 × 12) ≈ 0.74
and x3 = exp (–0.3 × 22) ≈ 0.30. The plot on the right of Figure 5-8 shows the trans‐
formed dataset (dropping the original features). As you can see, it is now linearly
separable.
Nonlinear SVM Classification
|
151
Download from finelybook www.finelybook.com


Figure 5-8. Similarity features using the Gaussian RBF
You may wonder how to select the landmarks. The simplest approach is to create a
landmark at the location of each and every instance in the dataset. This creates many
dimensions and thus increases the chances that the transformed training set will be
linearly separable. The downside is that a training set with m instances and n features
gets transformed into a training set with m instances and m features (assuming you
drop the original features). If your training set is very large, you end up with an
equally large number of features.
Gaussian RBF Kernel
Just like the polynomial features method, the similarity features method can be useful
with any Machine Learning algorithm, but it may be computationally expensive to
compute all the additional features, especially on large training sets. However, once
again the kernel trick does its SVM magic: it makes it possible to obtain a similar


again the kernel trick does its SVM magic: it makes it possible to obtain a similar
result as if you had added many similarity features, without actually having to add
them. Let’s try the Gaussian RBF kernel using the SVC class:
rbf_kernel_svm_clf = Pipeline((
        ("scaler", StandardScaler()),
        ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001))
    ))
rbf_kernel_svm_clf.fit(X, y)
This model is represented on the bottom left of Figure 5-9. The other plots show
models trained with different values of hyperparameters gamma (γ) and C. Increasing
gamma makes the bell-shape curve narrower (see the left plot of Figure 5-8), and as a
result each instance’s range of influence is smaller: the decision boundary ends up
being more irregular, wiggling around individual instances. Conversely, a small gamma
value makes the bell-shaped curve wider, so instances have a larger range of influ‐
ence, and the decision boundary ends up smoother. So γ acts like a regularization


ence, and the decision boundary ends up smoother. So γ acts like a regularization
hyperparameter: if your model is overfitting, you should reduce it, and if it is under‐
fitting, you should increase it (similar to the C hyperparameter).
152
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


1 “A Dual Coordinate Descent Method for Large-scale Linear SVM,” Lin et al. (2008).
Figure 5-9. SVM classifiers using an RBF kernel
Other kernels exist but are used much more rarely. For example, some kernels are
specialized for specific data structures. String kernels are sometimes used when classi‐
fying text documents or DNA sequences (e.g., using the string subsequence kernel or
kernels based on the Levenshtein distance).
With so many kernels to choose from, how can you decide which
one to use? As a rule of thumb, you should always try the linear
kernel first (remember that LinearSVC is much faster than SVC(ker
nel="linear")), especially if the training set is very large or if it
has plenty of features. If the training set is not too large, you should
try the Gaussian RBF kernel as well; it works well in most cases.
Then if you have spare time and computing power, you can also
experiment with a few other kernels using cross-validation and grid


experiment with a few other kernels using cross-validation and grid
search, especially if there are kernels specialized for your training
set’s data structure.
Computational Complexity
The LinearSVC class is based on the liblinear library, which implements an optimized
algorithm for linear SVMs.1 It does not support the kernel trick, but it scales almost
Nonlinear SVM Classification
|
153
Download from finelybook www.finelybook.com


2 “Sequential Minimal Optimization (SMO),” J. Platt (1998).
linearly with the number of training instances and the number of features: its training
time complexity is roughly O(m × n).
The algorithm takes longer if you require a very high precision. This is controlled by
the tolerance hyperparameter ϵ (called tol in Scikit-Learn). In most classification
tasks, the default tolerance is fine.
The SVC class is based on the libsvm library, which implements an algorithm that sup‐
ports the kernel trick.2 The training time complexity is usually between O(m2 × n)
and O(m3 × n). Unfortunately, this means that it gets dreadfully slow when the num‐
ber of training instances gets large (e.g., hundreds of thousands of instances). This
algorithm is perfect for complex but small or medium training sets. However, it scales
well with the number of features, especially with sparse features (i.e., when each
instance has few nonzero features). In this case, the algorithm scales roughly with the


instance has few nonzero features). In this case, the algorithm scales roughly with the
average number of nonzero features per instance. Table 5-1 compares Scikit-Learn’s
SVM classification classes.
Table 5-1. Comparison of Scikit-Learn classes for SVM classification
Class
Time complexity
Out-of-core support
Scaling required
Kernel trick
LinearSVC
O(m × n)
No
Yes
No
SGDClassifier O(m × n)
Yes
Yes
No
SVC
O(m² × n) to O(m³ × n) No
Yes
Yes
SVM Regression
As we mentioned earlier, the SVM algorithm is quite versatile: not only does it sup‐
port linear and nonlinear classification, but it also supports linear and nonlinear
regression. The trick is to reverse the objective: instead of trying to fit the largest pos‐
sible street between two classes while limiting margin violations, SVM Regression
tries to fit as many instances as possible on the street while limiting margin violations
(i.e., instances off the street). The width of the street is controlled by a hyperparame‐


(i.e., instances off the street). The width of the street is controlled by a hyperparame‐
ter ϵ. Figure 5-10 shows two linear SVM Regression models trained on some random
linear data, one with a large margin (ϵ = 1.5) and the other with a small margin (ϵ =
0.5).
154
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


Figure 5-10. SVM Regression
Adding more training instances within the margin does not affect the model’s predic‐
tions; thus, the model is said to be ϵ-insensitive.
You can use Scikit-Learn’s LinearSVR class to perform linear SVM Regression. The
following code produces the model represented on the left of Figure 5-10 (the train‐
ing data should be scaled and centered first):
from sklearn.svm import LinearSVR
svm_reg = LinearSVR(epsilon=1.5)
svm_reg.fit(X, y)
To tackle nonlinear regression tasks, you can use a kernelized SVM model. For exam‐
ple, Figure 5-11 shows SVM Regression on a random quadratic training set, using a
2nd-degree polynomial kernel. There is little regularization on the left plot (i.e., a large
C value), and much more regularization on the right plot (i.e., a small C value).
Figure 5-11. SVM regression using a 2nd-degree polynomial kernel
SVM Regression
|
155
Download from finelybook www.finelybook.com


The following code produces the model represented on the left of Figure 5-11 using
Scikit-Learn’s SVR class (which supports the kernel trick). The SVR class is the regres‐
sion equivalent of the SVC class, and the LinearSVR class is the regression equivalent
of the LinearSVC class. The LinearSVR class scales linearly with the size of the train‐
ing set (just like the LinearSVC class), while the SVR class gets much too slow when
the training set grows large (just like the SVC class).
from sklearn.svm import SVR
svm_poly_reg = SVR(kernel="poly", degree=2, C=100, epsilon=0.1)
svm_poly_reg.fit(X, y)
SVMs can also be used for outlier detection; see Scikit-Learn’s doc‐
umentation for more details.
Under the Hood
This section explains how SVMs make predictions and how their training algorithms
work, starting with linear SVM classifiers. You can safely skip it and go straight to the
exercises at the end of this chapter if you are just getting started with Machine Learn‐


exercises at the end of this chapter if you are just getting started with Machine Learn‐
ing, and come back later when you want to get a deeper understanding of SVMs.
First, a word about notations: in Chapter 4 we used the convention of putting all the
model parameters in one vector θ, including the bias term θ0 and the input feature
weights θ1 to θn, and adding a bias input x0 = 1 to all instances. In this chapter, we will
use a different convention, which is more convenient (and more common) when you
are dealing with SVMs: the bias term will be called b and the feature weights vector
will be called w. No bias feature will be added to the input feature vectors.
Decision Function and Predictions
The linear SVM classifier model predicts the class of a new instance x by simply com‐
puting the decision function wT · x + b = w1 x1 + ⋯ + wn xn + b: if the result is posi‐
tive, the predicted class ŷ is the positive class (1), or else it is the negative class (0); see
Equation 5-2.


Equation 5-2.
Equation 5-2. Linear SVM classifier prediction
y = 0 if �T · �+ b < 0,
1 if �T · �+ b ≥0
156
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


3 More generally, when there are n features, the decision function is an n-dimensional hyperplane, and the deci‐
sion boundary is an (n – 1)-dimensional hyperplane.
Figure 5-12 shows the decision function that corresponds to the model on the right of
Figure 5-4: it is a two-dimensional plane since this dataset has two features (petal
width and petal length). The decision boundary is the set of points where the decision
function is equal to 0: it is the intersection of two planes, which is a straight line (rep‐
resented by the thick solid line).3
Figure 5-12. Decision function for the iris dataset
The dashed lines represent the points where the decision function is equal to 1 or –1:
they are parallel and at equal distance to the decision boundary, forming a margin
around it. Training a linear SVM classifier means finding the value of w and b that
make this margin as wide as possible while avoiding margin violations (hard margin)
or limiting them (soft margin).
Training Objective


or limiting them (soft margin).
Training Objective
Consider the slope of the decision function: it is equal to the norm of the weight vec‐
tor, ∥ w ∥. If we divide this slope by 2, the points where the decision function is equal
to ±1 are going to be twice as far away from the decision boundary. In other words,
dividing the slope by 2 will multiply the margin by 2. Perhaps this is easier to visual‐
ize in 2D in Figure 5-13. The smaller the weight vector w, the larger the margin.
Under the Hood
|
157
Download from finelybook www.finelybook.com


4 Zeta (ζ) is the 8th letter of the Greek alphabet.
Figure 5-13. A smaller weight vector results in a larger margin
So we want to minimize ∥ w ∥ to get a large margin. However, if we also want to avoid
any margin violation (hard margin), then we need the decision function to be greater
than 1 for all positive training instances, and lower than –1 for negative training
instances. If we define t(i) = –1 for negative instances (if y(i) = 0) and t(i) = 1 for positive
instances (if y(i) = 1), then we can express this constraint as t(i)(wT · x(i) + b) ≥ 1 for all
instances.
We can therefore express the hard margin linear SVM classifier objective as the con‐
strained optimization problem in Equation 5-3.
Equation 5-3. Hard margin linear SVM classifier objective
minimize
�, b
1
2�T · �
subject to
t i �T · �i + b ≥1
for i = 1, 2, ⋯, m
We are minimizing 1
2wT · w, which is equal to 1
2∥ w ∥2, rather than
minimizing ∥ w ∥. This is because it will give the same result (since


2∥ w ∥2, rather than
minimizing ∥ w ∥. This is because it will give the same result (since
the values of w and b that minimize a value also minimize half of
its square), but 1
2∥ w ∥2 has a nice and simple derivative (it is just
w) while ∥ w ∥ is not differentiable at w = 0. Optimization algo‐
rithms work much better on differentiable functions.
To get the soft margin objective, we need to introduce a slack variable ζ(i) ≥ 0 for each
instance:4 ζ(i) measures how much the ith instance is allowed to violate the margin. We
now have two conflicting objectives: making the slack variables as small as possible to
reduce the margin violations, and making 1
2wT · w as small as possible to increase the
margin. This is where the C hyperparameter comes in: it allows us to define the trade‐
158
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


5 To learn more about Quadratic Programming, you can start by reading Stephen Boyd and Lieven Vanden‐
berghe, Convex Optimization (Cambridge, UK: Cambridge University Press, 2004) or watch Richard Brown’s
series of video lectures.
off between these two objectives. This gives us the constrained optimization problem
in Equation 5-4.
Equation 5-4. Soft margin linear SVM classifier objective
minimize
�, b, ζ
1
2�T · �+ C ∑
i = 1
m
ζ i
subject to
t i �T · �i + b ≥1 −ζ i
and
ζ i ≥0
for i = 1, 2, ⋯, m
Quadratic Programming
The hard margin and soft margin problems are both convex quadratic optimization
problems with linear constraints. Such problems are known as Quadratic Program‐
ming (QP) problems. Many off-the-shelf solvers are available to solve QP problems
using a variety of techniques that are outside the scope of this book.5 The general
problem formulation is given by Equation 5-5.
Equation 5-5. Quadratic Programming problem
Minimize
�
1
2�T · �· �
+
�T · �
subject to
�· �≤�
where
�


Minimize
�
1
2�T · �· �
+
�T · �
subject to
�· �≤�
where
�
is an np‐dimensional vector (np = number of parameters),
�
is an np × np matrix,
�
is an np‐dimensional vector,
�
is an nc × np matrix (nc = number of constraints),
�
is an nc‐dimensional vector.
Note that the expression A · p ≤ b actually defines nc constraints: pT · a(i) ≤ b(i) for i =
1, 2, ⋯, nc, where a(i) is the vector containing the elements of the ith row of A and b(i) is
the ith element of b.
You can easily verify that if you set the QP parameters in the following way, you get
the hard margin linear SVM classifier objective:
• np = n + 1, where n is the number of features (the +1 is for the bias term).
Under the Hood
|
159
Download from finelybook www.finelybook.com


6 The objective function is convex, and the inequality constraints are continuously differentiable and convex
functions.
• nc = m, where m is the number of training instances.
• H is the np × np identity matrix, except with a zero in the top-left cell (to ignore
the bias term).
• f = 0, an np-dimensional vector full of 0s.
• b = 1, an nc-dimensional vector full of 1s.
• a(i) = –t(i) �˙(i), where �˙(i) is equal to x(i) with an extra bias feature �˙0 = 1.
So one way to train a hard margin linear SVM classifier is just to use an off-the-shelf
QP solver by passing it the preceding parameters. The resulting vector p will contain
the bias term b = p0 and the feature weights wi = pi for i = 1, 2, ⋯, m. Similarly, you
can use a QP solver to solve the soft margin problem (see the exercises at the end of
the chapter).
However, to use the kernel trick we are going to look at a different constrained opti‐
mization problem.
The Dual Problem


mization problem.
The Dual Problem
Given a constrained optimization problem, known as the primal problem, it is possi‐
ble to express a different but closely related problem, called its dual problem. The sol‐
ution to the dual problem typically gives a lower bound to the solution of the primal
problem, but under some conditions it can even have the same solutions as the pri‐
mal problem. Luckily, the SVM problem happens to meet these conditions,6 so you
can choose to solve the primal problem or the dual problem; both will have the same
solution. Equation 5-6 shows the dual form of the linear SVM objective (if you are
interested in knowing how to derive the dual problem from the primal problem, see
Appendix C).
Equation 5-6. Dual form of the linear SVM objective
minimize
α
1
2 ∑
i = 1
m
∑
j = 1
m
α i α j t i t j �i T · �j
−
∑
i = 1
m
α i
subject to
α i ≥0
for i = 1, 2, ⋯, m
160
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


Once you find the vector α that minimizes this equation (using a QP solver), you can
compute � and b that minimize the primal problem by using Equation 5-7.
Equation 5-7. From the dual solution to the primal solution
�= ∑
i = 1
m
α i t i �i
b = 1
ns ∑
i = 1
α i > 0
m
1 −t i �T · �i
The dual problem is faster to solve than the primal when the number of training
instances is smaller than the number of features. More importantly, it makes the ker‐
nel trick possible, while the primal does not. So what is this kernel trick anyway?
Kernelized SVM
Suppose you want to apply a 2nd-degree polynomial transformation to a two-
dimensional training set (such as the moons training set), then train a linear SVM
classifier on the transformed training set. Equation 5-8 shows the 2nd-degree polyno‐
mial mapping function ϕ that you want to apply.
Equation 5-8. Second-degree polynomial mapping
ϕ �= ϕ
x1
x2
=
x1
2
2 x1x2
x2
2


Equation 5-8. Second-degree polynomial mapping
ϕ �= ϕ
x1
x2
=
x1
2
2 x1x2
x2
2
Notice that the transformed vector is three-dimensional instead of two-dimensional.
Now let’s look at what happens to a couple of two-dimensional vectors, a and b, if we
apply this 2nd-degree polynomial mapping and then compute the dot product of the
transformed vectors (See Equation 5-9).
Under the Hood
|
161
Download from finelybook www.finelybook.com


Equation 5-9. Kernel trick for a 2nd-degree polynomial mapping
ϕ �T · ϕ �
=
a1
2
2 a1a2
a2
2
T
·
b1
2
2 b1b2
b2
2
= a1
2b1
2 + 2a1b1a2b2 + a2
2b2
2
= a1b1 + a2b2
2 =
a1
a2
T
·
b1
b2
2
= �T · �
2
How about that? The dot product of the transformed vectors is equal to the square of
the dot product of the original vectors: ϕ(a)T · ϕ(b) = (aT · b)2.
Now here is the key insight: if you apply the transformation ϕ to all training instan‐
ces, then the dual problem (see Equation 5-6) will contain the dot product ϕ(x(i))T ·
ϕ(x(j)). But if ϕ is the 2nd-degree polynomial transformation defined in Equation 5-8,
then you can replace this dot product of transformed vectors simply by �i T · �j
2
.
So you don’t actually need to transform the training instances at all: just replace the
dot product by its square in Equation 5-6. The result will be strictly the same as if you
went through the trouble of actually transforming the training set then fitting a linear


went through the trouble of actually transforming the training set then fitting a linear
SVM algorithm, but this trick makes the whole process much more computationally
efficient. This is the essence of the kernel trick.
The function K(a, b) = (aT · b)2 is called a 2nd-degree polynomial kernel. In Machine
Learning, a kernel is a function capable of computing the dot product ϕ(a)T · ϕ(b)
based only on the original vectors a and b, without having to compute (or even to
know about) the transformation ϕ. Equation 5-10 lists some of the most commonly
used kernels.
Equation 5-10. Common kernels
Linear:
K �, �= �T · �
Polynomial:
K �, �= γ�T · �+ r
d
Gaussian RBF:
K �, �= exp −γ∥�−�∥2
Sigmoid:
K �, �= tanh γ�T · �+ r
162
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


Mercer’s Theorem
According to Mercer’s theorem, if a function K(a, b) respects a few mathematical con‐
ditions called Mercer’s conditions (K must be continuous, symmetric in its arguments
so K(a, b) = K(b, a), etc.), then there exists a function ϕ that maps a and b into
another space (possibly with much higher dimensions) such that K(a, b) = ϕ(a)T ·
ϕ(b). So you can use K as a kernel since you know ϕ exists, even if you don’t know
what ϕ is. In the case of the Gaussian RBF kernel, it can be shown that ϕ actually
maps each training instance to an infinite-dimensional space, so it’s a good thing you
don’t need to actually perform the mapping!
Note that some frequently used kernels (such as the Sigmoid kernel) don’t respect all
of Mercer’s conditions, yet they generally work well in practice.
There is still one loose end we must tie. Equation 5-7 shows how to go from the dual
solution to the primal solution in the case of a linear SVM classifier, but if you apply


solution to the primal solution in the case of a linear SVM classifier, but if you apply
the kernel trick you end up with equations that include ϕ(x(i)). In fact, � must have
the same number of dimensions as ϕ(x(i)), which may be huge or even infinite, so you
can’t compute it. But how can you make predictions without knowing �? Well, the
good news is that you can plug in the formula for � from Equation 5-7 into the deci‐
sion function for a new instance x(n), and you get an equation with only dot products
between input vectors. This makes it possible to use the kernel trick, once again
(Equation 5-11).
Equation 5-11. Making predictions with a kernelized SVM
h�, b ϕ �n
= �T · ϕ �n
+ b = ∑
i = 1
m
α i t i ϕ �i
T
· ϕ �n
+ b
= ∑
i = 1
m
α i t i ϕ �i T · ϕ �n
+ b
=
∑
i = 1
α i > 0
m
α i t i K �i , �n
+ b
Note that since α(i) ≠ 0 only for support vectors, making predictions involves comput‐
ing the dot product of the new input vector x(n) with only the support vectors, not all


ing the dot product of the new input vector x(n) with only the support vectors, not all
the training instances. Of course, you also need to compute the bias term b, using the
same trick (Equation 5-12).
Under the Hood
|
163
Download from finelybook www.finelybook.com


Equation 5-12. Computing the bias term using the kernel trick
b = 1
ns ∑
i = 1
α i > 0
m
1 −t i �T · ϕ �i
= 1
ns ∑
i = 1
α i > 0
m
1 −t i ∑
j = 1
m
α j t j ϕ �j
T
· ϕ �i
= 1
ns ∑
i = 1
α i > 0
m
1 −t i
∑
j = 1
α j > 0
m
α j t j K �i , �j
If you are starting to get a headache, it’s perfectly normal: it’s an unfortunate side
effects of the kernel trick.
Online SVMs
Before concluding this chapter, let’s take a quick look at online SVM classifiers (recall
that online learning means learning incrementally, typically as new instances arrive).
For linear SVM classifiers, one method is to use Gradient Descent (e.g., using
SGDClassifier) to minimize the cost function in Equation 5-13, which is derived
from the primal problem. Unfortunately it converges much more slowly than the
methods based on QP.
Equation 5-13. Linear SVM classifier cost function
J �, b = 1
2�T · �
+
C ∑
i = 1
m
max 0, 1 −t i �T · �i + b
The first sum in the cost function will push the model to have a small weight vector


The first sum in the cost function will push the model to have a small weight vector
w, leading to a larger margin. The second sum computes the total of all margin viola‐
tions. An instance’s margin violation is equal to 0 if it is located off the street and on
the correct side, or else it is proportional to the distance to the correct side of the
street. Minimizing this term ensures that the model makes the margin violations as
small and as few as possible
Hinge Loss
The function max(0, 1 – t) is called the hinge loss function (represented below). It is
equal to 0 when t ≥ 1. Its derivative (slope) is equal to –1 if t < 1 and 0 if t > 1. It is not
differentiable at t = 1, but just like for Lasso Regression (see “Lasso Regression” on
page 130) you can still use Gradient Descent using any subderivative at t = 0 (i.e., any
value between –1 and 0).
164
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


7 “Incremental and Decremental Support Vector Machine Learning,” G. Cauwenberghs, T. Poggio (2001).
8 “Fast Kernel Classifiers with Online and Active Learning,“ A. Bordes, S. Ertekin, J. Weston, L. Bottou (2005).
It is also possible to implement online kernelized SVMs—for example, using “Incre‐
mental and Decremental SVM Learning”7 or “Fast Kernel Classifiers with Online and
Active Learning.”8 However, these are implemented in Matlab and C++. For large-
scale nonlinear problems, you may want to consider using neural networks instead
(see Part II).
Exercises
1. What is the fundamental idea behind Support Vector Machines?
2. What is a support vector?
3. Why is it important to scale the inputs when using SVMs?
4. Can an SVM classifier output a confidence score when it classifies an instance?
What about a probability?
5. Should you use the primal or the dual form of the SVM problem to train a model
on a training set with millions of instances and hundreds of features?


on a training set with millions of instances and hundreds of features?
6. Say you trained an SVM classifier with an RBF kernel. It seems to underfit the
training set: should you increase or decrease γ (gamma)? What about C?
7. How should you set the QP parameters (H, f, A, and b) to solve the soft margin
linear SVM classifier problem using an off-the-shelf QP solver?
8. Train a LinearSVC on a linearly separable dataset. Then train an SVC and a
SGDClassifier on the same dataset. See if you can get them to produce roughly
the same model.
9. Train an SVM classifier on the MNIST dataset. Since SVM classifiers are binary
classifiers, you will need to use one-versus-all to classify all 10 digits. You may
Exercises
|
165
Download from finelybook www.finelybook.com


want to tune the hyperparameters using small validation sets to speed up the pro‐
cess. What accuracy can you reach?
10. Train an SVM regressor on the California housing dataset.
Solutions to these exercises are available in Appendix A.
166
|
Chapter 5: Support Vector Machines
Download from finelybook www.finelybook.com


CHAPTER 6
Decision Trees
Like SVMs, Decision Trees are versatile Machine Learning algorithms that can per‐
form both classification and regression tasks, and even multioutput tasks. They are
very powerful algorithms, capable of fitting complex datasets. For example, in Chap‐
ter 2 you trained a DecisionTreeRegressor model on the California housing dataset,
fitting it perfectly (actually overfitting it).
Decision Trees are also the fundamental components of Random Forests (see Chap‐
ter 7), which are among the most powerful Machine Learning algorithms available
today.
In this chapter we will start by discussing how to train, visualize, and make predic‐
tions with Decision Trees. Then we will go through the CART training algorithm
used by Scikit-Learn, and we will discuss how to regularize trees and use them for
regression tasks. Finally, we will discuss some of the limitations of Decision Trees.
Training and Visualizing a Decision Tree


Training and Visualizing a Decision Tree
To understand Decision Trees, let’s just build one and take a look at how it makes pre‐
dictions. The following code trains a DecisionTreeClassifier on the iris dataset
(see Chapter 4):
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
iris = load_iris()
X = iris.data[:, 2:] # petal length and width
y = iris.target
tree_clf = DecisionTreeClassifier(max_depth=2)
tree_clf.fit(X, y)
167
Download from finelybook www.finelybook.com


1 Graphviz is an open source graph visualization software package, available at http://www.graphviz.org/.
You can visualize the trained Decision Tree by first using the export_graphviz()
method to output a graph definition file called iris_tree.dot:
from sklearn.tree import export_graphviz
export_graphviz(
        tree_clf,
        out_file=image_path("iris_tree.dot"),
        feature_names=iris.feature_names[2:],
        class_names=iris.target_names,
        rounded=True,
        filled=True
    )
Then you can convert this .dot file to a variety of formats such as PDF or PNG using
the dot command-line tool from the graphviz package.1 This command line converts
the .dot file to a .png image file:
$ dot -Tpng iris_tree.dot -o iris_tree.png
Your first decision tree looks like Figure 6-1.
Figure 6-1. Iris Decision Tree
168
|
Chapter 6: Decision Trees
Download from finelybook www.finelybook.com


Making Predictions
Let’s see how the tree represented in Figure 6-1 makes predictions. Suppose you find
an iris flower and you want to classify it. You start at the root node (depth 0, at the
top): this node asks whether the flower’s petal length is smaller than 2.45 cm. If it is,
then you move down to the root’s left child node (depth 1, left). In this case, it is a leaf
node (i.e., it does not have any children nodes), so it does not ask any questions: you
can simply look at the predicted class for that node and the Decision Tree predicts
that your flower is an Iris-Setosa (class=setosa).
Now suppose you find another flower, but this time the petal length is greater than
2.45 cm. You must move down to the root’s right child node (depth 1, right), which is
not a leaf node, so it asks another question: is the petal width smaller than 1.75 cm? If
it is, then your flower is most likely an Iris-Versicolor (depth 2, left). If not, it is likely


it is, then your flower is most likely an Iris-Versicolor (depth 2, left). If not, it is likely
an Iris-Virginica (depth 2, right). It’s really that simple.
One of the many qualities of Decision Trees is that they require
very little data preparation. In particular, they don’t require feature
scaling or centering at all.
A node’s samples attribute counts how many training instances it applies to. For
example, 100 training instances have a petal length greater than 2.45 cm (depth 1,
right), among which 54 have a petal width smaller than 1.75 cm (depth 2, left). A
node’s value attribute tells you how many training instances of each class this node
applies to: for example, the bottom-right node applies to 0 Iris-Setosa, 1 Iris-
Versicolor, and 45 Iris-Virginica. Finally, a node’s gini attribute measures its impur‐
ity: a node is “pure” (gini=0) if all training instances it applies to belong to the same
class. For example, since the depth-1 left node applies only to Iris-Setosa training


class. For example, since the depth-1 left node applies only to Iris-Setosa training
instances, it is pure and its gini score is 0. Equation 6-1 shows how the training algo‐
rithm computes the gini score Gi of the ith node. For example, the depth-2 left node
has a gini score equal to 1 – (0/54)2 – (49/54)2 – (5/54)2 ≈ 0.168. Another impurity
measure is discussed shortly.
Equation 6-1. Gini impurity
Gi = 1 −∑
k = 1
n
pi, k
2
• pi,k is the ratio of class k instances among the training instances in the ith node.
Making Predictions
|
169
Download from finelybook www.finelybook.com


Scikit-Learn uses the CART algorithm, which produces only binary
trees: nonleaf nodes always have two children (i.e., questions only
have yes/no answers). However, other algorithms such as ID3 can
produce Decision Trees with nodes that have more than two chil‐
dren.
Figure 6-2 shows this Decision Tree’s decision boundaries. The thick vertical line rep‐
resents the decision boundary of the root node (depth 0): petal length = 2.45 cm.
Since the left area is pure (only Iris-Setosa), it cannot be split any further. However,
the right area is impure, so the depth-1 right node splits it at petal width = 1.75 cm
(represented by the dashed line). Since max_depth was set to 2, the Decision Tree
stops right there. However, if you set max_depth to 3, then the two depth-2 nodes
would each add another decision boundary (represented by the dotted lines).
Figure 6-2. Decision Tree decision boundaries
Model Interpretation: White Box Versus Black Box


Figure 6-2. Decision Tree decision boundaries
Model Interpretation: White Box Versus Black Box
As you can see Decision Trees are fairly intuitive and their decisions are easy to inter‐
pret. Such models are often called white box models. In contrast, as we will see, Ran‐
dom Forests or neural networks are generally considered black box models. They
make great predictions, and you can easily check the calculations that they performed
to make these predictions; nevertheless, it is usually hard to explain in simple terms
why the predictions were made. For example, if a neural network says that a particu‐
lar person appears on a picture, it is hard to know what actually contributed to this
prediction: did the model recognize that person’s eyes? Her mouth? Her nose? Her
shoes? Or even the couch that she was sitting on? Conversely, Decision Trees provide
nice and simple classification rules that can even be applied manually if need be (e.g.,
for flower classification).
170
|


for flower classification).
170
|
Chapter 6: Decision Trees
Download from finelybook www.finelybook.com


Estimating Class Probabilities
A Decision Tree can also estimate the probability that an instance belongs to a partic‐
ular class k: first it traverses the tree to find the leaf node for this instance, and then it
returns the ratio of training instances of class k in this node. For example, suppose
you have found a flower whose petals are 5 cm long and 1.5 cm wide. The corre‐
sponding leaf node is the depth-2 left node, so the Decision Tree should output the
following probabilities: 0% for Iris-Setosa (0/54), 90.7% for Iris-Versicolor (49/54),
and 9.3% for Iris-Virginica (5/54). And of course if you ask it to predict the class, it
should output Iris-Versicolor (class 1) since it has the highest probability. Let’s check
this:
>>> tree_clf.predict_proba([[5, 1.5]])
array([[ 0. ,  0.90740741,  0.09259259]])
>>> tree_clf.predict([[5, 1.5]])
array([1])
Perfect! Notice that the estimated probabilities would be identical anywhere else in


array([1])
Perfect! Notice that the estimated probabilities would be identical anywhere else in
the bottom-right rectangle of Figure 6-2—for example, if the petals were 6 cm long
and 1.5 cm wide (even though it seems obvious that it would most likely be an Iris-
Virginica in this case).
The CART Training Algorithm
Scikit-Learn uses the Classification And Regression Tree (CART) algorithm to train
Decision Trees (also called “growing” trees). The idea is really quite simple: the algo‐
rithm first splits the training set in two subsets using a single feature k and a thres‐
hold tk (e.g., “petal length ≤ 2.45 cm”). How does it choose k and tk? It searches for the
pair (k, tk) that produces the purest subsets (weighted by their size). The cost function
that the algorithm tries to minimize is given by Equation 6-2.
Equation 6-2. CART cost function for classification
J k, tk =
mleft
m Gleft +
mright
m
Gright
where
Gleft/right measures the impurity of the left/right subset,


mleft
m Gleft +
mright
m
Gright
where
Gleft/right measures the impurity of the left/right subset,
mleft/right is the number of instances in the left/right subset.
Once it has successfully split the training set in two, it splits the subsets using the
same logic, then the sub-subsets and so on, recursively. It stops recursing once it rea‐
ches the maximum depth (defined by the max_depth hyperparameter), or if it cannot
find a split that will reduce impurity. A few other hyperparameters (described in a
Estimating Class Probabilities
|
171
Download from finelybook www.finelybook.com


2 P is the set of problems that can be solved in polynomial time. NP is the set of problems whose solutions can
be verified in polynomial time. An NP-Hard problem is a problem to which any NP problem can be reduced
in polynomial time. An NP-Complete problem is both NP and NP-Hard. A major open mathematical ques‐
tion is whether or not P = NP. If P ≠ NP (which seems likely), then no polynomial algorithm will ever be
found for any NP-Complete problem (except perhaps on a quantum computer).
3 log2 is the binary logarithm. It is equal to log2(m) = log(m) / log(2).
moment) control additional stopping conditions (min_samples_split, min_sam
ples_leaf, min_weight_fraction_leaf, and max_leaf_nodes).
As you can see, the CART algorithm is a greedy algorithm: it greed‐
ily searches for an optimum split at the top level, then repeats the
process at each level. It does not check whether or not the split will
lead to the lowest possible impurity several levels down. A greedy


lead to the lowest possible impurity several levels down. A greedy
algorithm often produces a reasonably good solution, but it is not
guaranteed to be the optimal solution.
Unfortunately, finding the optimal tree is known to be an NP-Complete problem:2 it
requires O(exp(m)) time, making the problem intractable even for fairly small train‐
ing sets. This is why we must settle for a “reasonably good” solution.
Computational Complexity
Making predictions requires traversing the Decision Tree from the root to a leaf.
Decision Trees are generally approximately balanced, so traversing the Decision Tree
requires going through roughly O(log2(m)) nodes.3 Since each node only requires
checking the value of one feature, the overall prediction complexity is just O(log2(m)),
independent of the number of features. So predictions are very fast, even when deal‐
ing with large training sets.
However, the training algorithm compares all features (or less if max_features is set)


However, the training algorithm compares all features (or less if max_features is set)
on all samples at each node. This results in a training complexity of O(n × m log(m)).
For small training sets (less than a few thousand instances), Scikit-Learn can speed up
training by presorting the data (set presort=True), but this slows down training con‐
siderably for larger training sets.
Gini Impurity or Entropy?
By default, the Gini impurity measure is used, but you can select the entropy impurity
measure instead by setting the criterion hyperparameter to "entropy". The concept
of entropy originated in thermodynamics as a measure of molecular disorder:
entropy approaches zero when molecules are still and well ordered. It later spread to a
wide variety of domains, including Shannon’s information theory, where it measures
172
|
Chapter 6: Decision Trees
Download from finelybook www.finelybook.com


4 A reduction of entropy is often called an information gain.
5 See Sebastian Raschka’s interesting analysis for more details.
the average information content of a message:4 entropy is zero when all messages are
identical. In Machine Learning, it is frequently used as an impurity measure: a set’s
entropy is zero when it contains instances of only one class. Equation 6-3 shows the
definition of the entropy of the ith node. For example, the depth-2 left node in
Figure 6-1 has an entropy equal to −49
54 log
49
54 −5
54 log
5
54  ≈ 0.31.
Equation 6-3. Entropy
Hi = −
∑
k = 1
pi, k ≠0
n
pi, k log pi, k
So should you use Gini impurity or entropy? The truth is, most of the time it does not
make a big difference: they lead to similar trees. Gini impurity is slightly faster to
compute, so it is a good default. However, when they differ, Gini impurity tends to
isolate the most frequent class in its own branch of the tree, while entropy tends to
produce slightly more balanced trees.5


produce slightly more balanced trees.5
Regularization Hyperparameters
Decision Trees make very few assumptions about the training data (as opposed to lin‐
ear models, which obviously assume that the data is linear, for example). If left
unconstrained, the tree structure will adapt itself to the training data, fitting it very
closely, and most likely overfitting it. Such a model is often called a nonparametric
model, not because it does not have any parameters (it often has a lot) but because the
number of parameters is not determined prior to training, so the model structure is
free to stick closely to the data. In contrast, a parametric model such as a linear model
has a predetermined number of parameters, so its degree of freedom is limited,
reducing the risk of overfitting (but increasing the risk of underfitting).
To avoid overfitting the training data, you need to restrict the Decision Tree’s freedom


To avoid overfitting the training data, you need to restrict the Decision Tree’s freedom
during training. As you know by now, this is called regularization. The regularization
hyperparameters depend on the algorithm used, but generally you can at least restrict
the maximum depth of the Decision Tree. In Scikit-Learn, this is controlled by the
max_depth hyperparameter (the default value is None, which means unlimited).
Reducing max_depth will regularize the model and thus reduce the risk of overfitting.
The DecisionTreeClassifier class has a few other parameters that similarly restrict
the shape of the Decision Tree: min_samples_split (the minimum number of sam‐
Regularization Hyperparameters
|
173
Download from finelybook www.finelybook.com


ples a node must have before it can be split), min_samples_leaf (the minimum num‐
ber of samples a leaf node must have), min_weight_fraction_leaf (same as
min_samples_leaf but expressed as a fraction of the total number of weighted
instances), max_leaf_nodes (maximum number of leaf nodes), and max_features
(maximum number of features that are evaluated for splitting at each node). Increas‐
ing min_* hyperparameters or reducing max_* hyperparameters will regularize the
model.
Other algorithms work by first training the Decision Tree without
restrictions, then pruning (deleting) unnecessary nodes. A node
whose children are all leaf nodes is considered unnecessary if the
purity improvement it provides is not statistically significant. Stan‐
dard statistical tests, such as the χ2 test, are used to estimate the
probability that the improvement is purely the result of chance
(which is called the null hypothesis). If this probability, called the p-


(which is called the null hypothesis). If this probability, called the p-
value, is higher than a given threshold (typically 5%, controlled by
a hyperparameter), then the node is considered unnecessary and its
children are deleted. The pruning continues until all unnecessary
nodes have been pruned.
Figure 6-3 shows two Decision Trees trained on the moons dataset (introduced in
Chapter 5). On the left, the Decision Tree is trained with the default hyperparameters
(i.e., no restrictions), and on the right the Decision Tree is trained with min_sam
ples_leaf=4. It is quite obvious that the model on the left is overfitting, and the
model on the right will probably generalize better.
Figure 6-3. Regularization using min_samples_leaf
174
|
Chapter 6: Decision Trees
Download from finelybook www.finelybook.com


Regression
Decision Trees are also capable of performing regression tasks. Let’s build a regres‐
sion tree using Scikit-Learn’s DecisionTreeRegressor class, training it on a noisy
quadratic dataset with max_depth=2:
from sklearn.tree import DecisionTreeRegressor
tree_reg = DecisionTreeRegressor(max_depth=2)
tree_reg.fit(X, y)
The resulting tree is represented on Figure 6-4.
Figure 6-4. A Decision Tree for regression
This tree looks very similar to the classification tree you built earlier. The main differ‐
ence is that instead of predicting a class in each node, it predicts a value. For example,
suppose you want to make a prediction for a new instance with x1 = 0.6. You traverse
the tree starting at the root, and you eventually reach the leaf node that predicts
value=0.1106. This prediction is simply the average target value of the 110 training
instances associated to this leaf node. This prediction results in a Mean Squared Error
(MSE) equal to 0.0151 over these 110 instances.


(MSE) equal to 0.0151 over these 110 instances.
This model’s predictions are represented on the left of Figure 6-5. If you set
max_depth=3, you get the predictions represented on the right. Notice how the pre‐
dicted value for each region is always the average target value of the instances in that
region. The algorithm splits each region in a way that makes most training instances
as close as possible to that predicted value.
Regression
|
175
Download from finelybook www.finelybook.com


Figure 6-5. Predictions of two Decision Tree regression models
The CART algorithm works mostly the same way as earlier, except that instead of try‐
ing to split the training set in a way that minimizes impurity, it now tries to split the
training set in a way that minimizes the MSE. Equation 6-4 shows the cost function
that the algorithm tries to minimize.
Equation 6-4. CART cost function for regression
J k, tk =
mleft
m MSEleft +
mright
m
MSEright
where
MSEnode =
∑
i ∈node
ynode −y i 2
ynode =
1
mnode
∑
i ∈node
y i
Just like for classification tasks, Decision Trees are prone to overfitting when dealing
with regression tasks. Without any regularization (i.e., using the default hyperpara‐
meters), you get the predictions on the left of Figure 6-6. It is obviously overfitting
the training set very badly. Just setting min_samples_leaf=10 results in a much more
reasonable model, represented on the right of Figure 6-6.
Figure 6-6. Regularizing a Decision Tree regressor
176
|


Figure 6-6. Regularizing a Decision Tree regressor
176
|
Chapter 6: Decision Trees
Download from finelybook www.finelybook.com


6 It randomly selects the set of features to evaluate at each node.
Instability
Hopefully by now you are convinced that Decision Trees have a lot going for them:
they are simple to understand and interpret, easy to use, versatile, and powerful.
However they do have a few limitations. First, as you may have noticed, Decision
Trees love orthogonal decision boundaries (all splits are perpendicular to an axis),
which makes them sensitive to training set rotation. For example, Figure 6-7 shows a
simple linearly separable dataset: on the left, a Decision Tree can split it easily, while
on the right, after the dataset is rotated by 45°, the decision boundary looks unneces‐
sarily convoluted. Although both Decision Trees fit the training set perfectly, it is very
likely that the model on the right will not generalize well. One way to limit this prob‐
lem is to use PCA (see Chapter 8), which often results in a better orientation of the
training data.


training data.
Figure 6-7. Sensitivity to training set rotation
More generally, the main issue with Decision Trees is that they are very sensitive to
small variations in the training data. For example, if you just remove the widest Iris-
Versicolor from the iris training set (the one with petals 4.8 cm long and 1.8 cm wide)
and train a new Decision Tree, you may get the model represented in Figure 6-8. As
you can see, it looks very different from the previous Decision Tree (Figure 6-2).
Actually, since the training algorithm used by Scikit-Learn is stochastic6 you may
get very different models even on the same training data (unless you set the
random_state hyperparameter).
Instability
|
177
Download from finelybook www.finelybook.com


Figure 6-8. Sensitivity to training set details
Random Forests can limit this instability by averaging predictions over many trees, as
we will see in the next chapter.
Exercises
1. What is the approximate depth of a Decision Tree trained (without restrictions)
on a training set with 1 million instances?
2. Is a node’s Gini impurity generally lower or greater than its parent’s? Is it gener‐
ally lower/greater, or always lower/greater?
3. If a Decision Tree is overfitting the training set, is it a good idea to try decreasing
max_depth?
4. If a Decision Tree is underfitting the training set, is it a good idea to try scaling
the input features?
5. If it takes one hour to train a Decision Tree on a training set containing 1 million
instances, roughly how much time will it take to train another Decision Tree on a
training set containing 10 million instances?
6. If your training set contains 100,000 instances, will setting presort=True speed
up training?


6. If your training set contains 100,000 instances, will setting presort=True speed
up training?
7. Train and fine-tune a Decision Tree for the moons dataset.
a. Generate a moons dataset using make_moons(n_samples=10000, noise=0.4).
b. Split it into a training set and a test set using train_test_split().
178
|
Chapter 6: Decision Trees
Download from finelybook www.finelybook.com


c. Use grid search with cross-validation (with the help of the GridSearchCV
class) to find good hyperparameter values for a DecisionTreeClassifier.
Hint: try various values for max_leaf_nodes.
d. Train it on the full training set using these hyperparameters, and measure
your model’s performance on the test set. You should get roughly 85% to 87%
accuracy.
8. Grow a forest.
a. Continuing the previous exercise, generate 1,000 subsets of the training set,
each containing 100 instances selected randomly. Hint: you can use Scikit-
Learn’s ShuffleSplit class for this.
b. Train one Decision Tree on each subset, using the best hyperparameter values
found above. Evaluate these 1,000 Decision Trees on the test set. Since they
were trained on smaller sets, these Decision Trees will likely perform worse
than the first Decision Tree, achieving only about 80% accuracy.
c. Now comes the magic. For each test set instance, generate the predictions of


c. Now comes the magic. For each test set instance, generate the predictions of
the 1,000 Decision Trees, and keep only the most frequent prediction (you can
use SciPy’s mode() function for this). This gives you majority-vote predictions
over the test set.
d. Evaluate these predictions on the test set: you should obtain a slightly higher
accuracy than your first model (about 0.5 to 1.5% higher). Congratulations,
you have trained a Random Forest classifier!
Solutions to these exercises are available in Appendix A.
Exercises
|
179
Download from finelybook www.finelybook.com


Download from finelybook www.finelybook.com


CHAPTER 7
Ensemble Learning and Random Forests
Suppose you ask a complex question to thousands of random people, then aggregate
their answers. In many cases you will find that this aggregated answer is better than
an expert’s answer. This is called the wisdom of the crowd. Similarly, if you aggregate
the predictions of a group of predictors (such as classifiers or regressors), you will
often get better predictions than with the best individual predictor. A group of pre‐
dictors is called an ensemble; thus, this technique is called Ensemble Learning, and an
Ensemble Learning algorithm is called an Ensemble method.
For example, you can train a group of Decision Tree classifiers, each on a different
random subset of the training set. To make predictions, you just obtain the predic‐
tions of all individual trees, then predict the class that gets the most votes (see the last
exercise in Chapter 6). Such an ensemble of Decision Trees is called a Random Forest,


exercise in Chapter 6). Such an ensemble of Decision Trees is called a Random Forest,
and despite its simplicity, this is one of the most powerful Machine Learning algo‐
rithms available today.
Moreover, as we discussed in Chapter 2, you will often use Ensemble methods near
the end of a project, once you have already built a few good predictors, to combine
them into an even better predictor. In fact, the winning solutions in Machine Learn‐
ing competitions often involve several Ensemble methods (most famously in the Net‐
flix Prize competition).
In this chapter we will discuss the most popular Ensemble methods, including bag‐
ging, boosting, stacking, and a few others. We will also explore Random Forests.
Voting Classifiers
Suppose you have trained a few classifiers, each one achieving about 80% accuracy.
You may have a Logistic Regression classifier, an SVM classifier, a Random Forest
classifier, a K-Nearest Neighbors classifier, and perhaps a few more (see Figure 7-1).
181


classifier, a K-Nearest Neighbors classifier, and perhaps a few more (see Figure 7-1).
181
Download from finelybook www.finelybook.com


Figure 7-1. Training diverse classifiers
A very simple way to create an even better classifier is to aggregate the predictions of
each classifier and predict the class that gets the most votes. This majority-vote classi‐
fier is called a hard voting classifier (see Figure 7-2).
Figure 7-2. Hard voting classifier predictions
Somewhat surprisingly, this voting classifier often achieves a higher accuracy than the
best classifier in the ensemble. In fact, even if each classifier is a weak learner (mean‐
ing it does only slightly better than random guessing), the ensemble can still be a
strong learner (achieving high accuracy), provided there are a sufficient number of
weak learners and they are sufficiently diverse.
182
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


How is this possible? The following analogy can help shed some light on this mystery.
Suppose you have a slightly biased coin that has a 51% chance of coming up heads,
and 49% chance of coming up tails. If you toss it 1,000 times, you will generally get
more or less 510 heads and 490 tails, and hence a majority of heads. If you do the
math, you will find that the probability of obtaining a majority of heads after 1,000
tosses is close to 75%. The more you toss the coin, the higher the probability (e.g.,
with 10,000 tosses, the probability climbs over 97%). This is due to the law of large
numbers: as you keep tossing the coin, the ratio of heads gets closer and closer to the
probability of heads (51%). Figure 7-3 shows 10 series of biased coin tosses. You can
see that as the number of tosses increases, the ratio of heads approaches 51%. Eventu‐
ally all 10 series end up so close to 51% that they are consistently above 50%.
Figure 7-3. The law of large numbers


Figure 7-3. The law of large numbers
Similarly, suppose you build an ensemble containing 1,000 classifiers that are individ‐
ually correct only 51% of the time (barely better than random guessing). If you pre‐
dict the majority voted class, you can hope for up to 75% accuracy! However, this is
only true if all classifiers are perfectly independent, making uncorrelated errors,
which is clearly not the case since they are trained on the same data. They are likely to
make the same types of errors, so there will be many majority votes for the wrong
class, reducing the ensemble’s accuracy.
Ensemble methods work best when the predictors are as independ‐
ent from one another as possible. One way to get diverse classifiers
is to train them using very different algorithms. This increases the
chance that they will make very different types of errors, improving
the ensemble’s accuracy.
Voting Classifiers
|
183
Download from finelybook www.finelybook.com


The following code creates and trains a voting classifier in Scikit-Learn, composed of
three diverse classifiers (the training set is the moons dataset, introduced in Chap‐
ter 5):
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
log_clf = LogisticRegression()
rnd_clf = RandomForestClassifier()
svm_clf = SVC()
voting_clf = VotingClassifier(
        estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
        voting='hard'
    )
voting_clf.fit(X_train, y_train)
Let’s look at each classifier’s accuracy on the test set:
>>> from sklearn.metrics import accuracy_score
>>> for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
>>>     clf.fit(X_train, y_train)
>>>     y_pred = clf.predict(X_test)
>>>     print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
LogisticRegression 0.864
RandomForestClassifier 0.872
SVC 0.888
VotingClassifier 0.896


LogisticRegression 0.864
RandomForestClassifier 0.872
SVC 0.888
VotingClassifier 0.896
There you have it! The voting classifier slightly outperforms all the individual classifi‐
ers.
If all classifiers are able to estimate class probabilities (i.e., they have a pre
dict_proba() method), then you can tell Scikit-Learn to predict the class with the
highest class probability, averaged over all the individual classifiers. This is called soft
voting. It often achieves higher performance than hard voting because it gives more
weight to highly confident votes. All you need to do is replace voting="hard" with
voting="soft" and ensure that all classifiers can estimate class probabilities. This is
not the case of the SVC class by default, so you need to set its probability hyperpara‐
meter to True (this will make the SVC class use cross-validation to estimate class prob‐
abilities, slowing down training, and it will add a predict_proba() method). If you


abilities, slowing down training, and it will add a predict_proba() method). If you
modify the preceding code to use soft voting, you will find that the voting classifier
achieves over 91% accuracy!
184
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


1 “Bagging Predictors,” L. Breiman (1996).
2 In statistics, resampling with replacement is called bootstrapping.
3 “Pasting small votes for classification in large databases and on-line,” L. Breiman (1999).
4 Bias and variance were introduced in Chapter 4.
Bagging and Pasting
One way to get a diverse set of classifiers is to use very different training algorithms,
as just discussed. Another approach is to use the same training algorithm for every
predictor, but to train them on different random subsets of the training set. When
sampling is performed with replacement, this method is called bagging1 (short for
bootstrap aggregating2). When sampling is performed without replacement, it is called
pasting.3
In other words, both bagging and pasting allow training instances to be sampled sev‐
eral times across multiple predictors, but only bagging allows training instances to be
sampled several times for the same predictor. This sampling and training process is
represented in Figure 7-4.


represented in Figure 7-4.
Figure 7-4. Pasting/bagging training set sampling and training
Once all predictors are trained, the ensemble can make a prediction for a new
instance by simply aggregating the predictions of all predictors. The aggregation
function is typically the statistical mode (i.e., the most frequent prediction, just like a
hard voting classifier) for classification, or the average for regression. Each individual
predictor has a higher bias than if it were trained on the original training set, but
aggregation reduces both bias and variance.4 Generally, the net result is that the
Bagging and Pasting
|
185
Download from finelybook www.finelybook.com


5 max_samples can alternatively be set to a float between 0.0 and 1.0, in which case the max number of instances
to sample is equal to the size of the training set times max_samples.
ensemble has a similar bias but a lower variance than a single predictor trained on the
original training set.
As you can see in Figure 7-4, predictors can all be trained in parallel, via different
CPU cores or even different servers. Similarly, predictions can be made in parallel.
This is one of the reasons why bagging and pasting are such popular methods: they
scale very well.
Bagging and Pasting in Scikit-Learn
Scikit-Learn offers a simple API for both bagging and pasting with the BaggingClas
sifier class (or BaggingRegressor for regression). The following code trains an
ensemble of 500 Decision Tree classifiers,5 each trained on 100 training instances ran‐
domly sampled from the training set with replacement (this is an example of bagging,


domly sampled from the training set with replacement (this is an example of bagging,
but if you want to use pasting instead, just set bootstrap=False). The n_jobs param‐
eter tells Scikit-Learn the number of CPU cores to use for training and predictions
(–1 tells Scikit-Learn to use all available cores):
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
bag_clf = BaggingClassifier(
        DecisionTreeClassifier(), n_estimators=500,
        max_samples=100, bootstrap=True, n_jobs=-1
    )
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
The BaggingClassifier automatically performs soft voting
instead of hard voting if the base classifier can estimate class proba‐
bilities (i.e., if it has a predict_proba() method), which is the case
with Decision Trees classifiers.
Figure 7-5 compares the decision boundary of a single Decision Tree with the deci‐
sion boundary of a bagging ensemble of 500 trees (from the preceding code), both


sion boundary of a bagging ensemble of 500 trees (from the preceding code), both
trained on the moons dataset. As you can see, the ensemble’s predictions will likely
generalize much better than the single Decision Tree’s predictions: the ensemble has a
comparable bias but a smaller variance (it makes roughly the same number of errors
on the training set, but the decision boundary is less irregular).
186
|
Chapter 7: Ensemble Learning and Random Forests
Download from finelybook www.finelybook.com


6 As m grows, this ratio approaches 1 – exp(–1) ≈ 63.212%.
Figure 7-5. A single Decision Tree versus a bagging ensemble of 500 trees
Bootstrapping introduces a bit more diversity in the subsets that each predictor is
trained on, so bagging ends up with a slightly higher bias than pasting, but this also
means that predictors end up being less correlated so the ensemble’s variance is
reduced. Overall, bagging often results in better models, which explains why it is gen‐
erally preferred. However, if you have spare time and CPU power you can use cross-
validation to evaluate both bagging and pasting and select the one that works best.
Out-of-Bag Evaluation
With bagging, some instances may be sampled several times for any given predictor,
while others may not be sampled at all. By default a BaggingClassifier samples m
training instances with replacement (bootstrap=True), where m is the size of the
training set. This means that only about 63% of the training instances are sampled on


training set. This means that only about 63% of the training instances are sampled on
average for each predictor.6 The remaining 37% of the training instances that are not
sampled are called out-of-bag (oob) instances. Note that they are not the same 37%
for all predictors.
Since a predictor never sees the oob instances during training, it can be evaluated on
these instances, without the need for a separate validation set or cross-validation. You
can evaluate the ensemble itself by averaging out the oob evaluations of each predic‐
tor.
In Scikit-Learn, you can set oob_score=True when creating a BaggingClassifier to
request an automatic oob evaluation after training. The following code demonstrates
this. The resulting evaluation score is available through the oob_score_ variable:
>>> bag_clf = BaggingClassifier(
>>>         DecisionTreeClassifier(), n_estimators=500,
>>>         bootstrap=True, n_jobs=-1, oob_score=True)
Bagging and Pasting
|
187


>>>         bootstrap=True, n_jobs=-1, oob_score=True)
Bagging and Pasting
|
187
Download from finelybook www.finelybook.com