• Machine Learning is about making machines get better at some task by learning
from data, instead of having to explicitly code rules.
• There are many different types of ML systems: supervised or not, batch or online,
instance-based or model-based, and so on.
• In a ML project you gather data in a training set, and you feed the training set to
a learning algorithm. If the algorithm is model-based it tunes some parameters to
fit the model to the training set (i.e., to make good predictions on the training set
itself), and then hopefully it will be able to make good predictions on new cases
as well. If the algorithm is instance-based, it just learns the examples by heart and
uses a similarity measure to generalize to new instances.
• The system will not perform well if your training set is too small, or if the data is
not representative, noisy, or polluted with irrelevant features (garbage in, garbage
out). Lastly, your model needs to be neither too simple (in which case it will


out). Lastly, your model needs to be neither too simple (in which case it will
underfit) nor too complex (in which case it will overfit).
There’s just one last important topic to cover: once you have trained a model, you
don’t want to just “hope” it generalizes to new cases. You want to evaluate it, and fine-
tune it if necessary. Let’s see how.
Testing and Validating
The only way to know how well a model will generalize to new cases is to actually try
it out on new cases. One way to do that is to put your model in production and moni‐
tor how well it performs. This works well, but if your model is horribly bad, your
users will complain—not the best idea.
A better option is to split your data into two sets: the training set and the test set. As
these names imply, you train your model using the training set, and you test it using
the test set. The error rate on new cases is called the generalization error (or out-of-

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


the test set. The error rate on new cases is called the generalization error (or out-of-
sample error), and by evaluating your model on the test set, you get an estimation of
this error. This value tells you how well your model will perform on instances it has
never seen before.
If the training error is low (i.e., your model makes few mistakes on the training set)
but the generalization error is high, it means that your model is overfitting the train‐
ing data.
It is common to use 80% of the data for training and hold out 20%
for testing.
Testing and Validating
|
29
Download from finelybook www.finelybook.com


11 “The Lack of A Priori Distinctions Between Learning Algorithms,” D. Wolperts (1996).
So evaluating a model is simple enough: just use a test set. Now suppose you are hesi‐
tating between two models (say a linear model and a polynomial model): how can
you decide? One option is to train both and compare how well they generalize using
the test set.
Now suppose that the linear model generalizes better, but you want to apply some
regularization to avoid overfitting. The question is: how do you choose the value of
the regularization hyperparameter? One option is to train 100 different models using
100 different values for this hyperparameter. Suppose you find the best hyperparame‐
ter value that produces a model with the lowest generalization error, say just 5% error.
So you launch this model into production, but unfortunately it does not perform as
well as expected and produces 15% errors. What just happened?


well as expected and produces 15% errors. What just happened?
The problem is that you measured the generalization error multiple times on the test
set, and you adapted the model and hyperparameters to produce the best model for
that set. This means that the model is unlikely to perform as well on new data.
A common solution to this problem is to have a second holdout set called the valida‐
tion set. You train multiple models with various hyperparameters using the training
set, you select the model and hyperparameters that perform best on the validation set,
and when you’re happy with your model you run a single final test against the test set
to get an estimate of the generalization error.
To avoid “wasting” too much training data in validation sets, a common technique is
to use cross-validation: the training set is split into complementary subsets, and each
model is trained against a different combination of these subsets and validated


model is trained against a different combination of these subsets and validated
against the remaining parts. Once the model type and hyperparameters have been
selected, a final model is trained using these hyperparameters on the full training set,
and the generalized error is measured on the test set.
No Free Lunch Theorem
A model is a simplified version of the observations. The simplifications are meant to
discard the superfluous details that are unlikely to generalize to new instances. How‐
ever, to decide what data to discard and what data to keep, you must make assump‐
tions. For example, a linear model makes the assumption that the data is
fundamentally linear and that the distance between the instances and the straight line
is just noise, which can safely be ignored.
In a famous 1996 paper,11 David Wolpert demonstrated that if you make absolutely
no assumption about the data, then there is no reason to prefer one model over any


no assumption about the data, then there is no reason to prefer one model over any
other. This is called the No Free Lunch (NFL) theorem. For some datasets the best
30
|
Chapter 1: The Machine Learning Landscape
Download from finelybook www.finelybook.com


model is a linear model, while for other datasets it is a neural network. There is no
model that is a priori guaranteed to work better (hence the name of the theorem). The
only way to know for sure which model is best is to evaluate them all. Since this is not
possible, in practice you make some reasonable assumptions about the data and you
evaluate only a few reasonable models. For example, for simple tasks you may evalu‐
ate linear models with various levels of regularization, and for a complex problem you
may evaluate various neural networks.
Exercises
In this chapter we have covered some of the most important concepts in Machine
Learning. In the next chapters we will dive deeper and write more code, but before we
do, make sure you know how to answer the following questions:
1. How would you define Machine Learning?
2. Can you name four types of problems where it shines?
3. What is a labeled training set?
4. What are the two most common supervised tasks?


3. What is a labeled training set?
4. What are the two most common supervised tasks?
5. Can you name four common unsupervised tasks?
6. What type of Machine Learning algorithm would you use to allow a robot to
walk in various unknown terrains?
7. What type of algorithm would you use to segment your customers into multiple
groups?
8. Would you frame the problem of spam detection as a supervised learning prob‐
lem or an unsupervised learning problem?
9. What is an online learning system?
10. What is out-of-core learning?
11. What type of learning algorithm relies on a similarity measure to make predic‐
tions?
12. What is the difference between a model parameter and a learning algorithm’s
hyperparameter?
13. What do model-based learning algorithms search for? What is the most common
strategy they use to succeed? How do they make predictions?
14. Can you name four of the main challenges in Machine Learning?


14. Can you name four of the main challenges in Machine Learning?
15. If your model performs great on the training data but generalizes poorly to new
instances, what is happening? Can you name three possible solutions?
16. What is a test set and why would you want to use it?
Exercises
|
31
Download from finelybook www.finelybook.com


17. What is the purpose of a validation set?
18. What can go wrong if you tune hyperparameters using the test set?
19. What is cross-validation and why would you prefer it to a validation set?
Solutions to these exercises are available in Appendix A.
32
|
Chapter 1: The Machine Learning Landscape
Download from finelybook www.finelybook.com


1 The example project is completely fictitious; the goal is just to illustrate the main steps of a Machine Learning
project, not to learn anything about the real estate business.
CHAPTER 2
End-to-End Machine Learning Project
In this chapter, you will go through an example project end to end, pretending to be a
recently hired data scientist in a real estate company.1 Here are the main steps you will
go through:
1. Look at the big picture.
2. Get the data.
3. Discover and visualize the data to gain insights.
4. Prepare the data for Machine Learning algorithms.
5. Select a model and train it.
6. Fine-tune your model.
7. Present your solution.
8. Launch, monitor, and maintain your system.
Working with Real Data
When you are learning about Machine Learning it is best to actually experiment with
real-world data, not just artificial datasets. Fortunately, there are thousands of open
datasets to choose from, ranging across all sorts of domains. Here are a few places
you can look to get data:


you can look to get data:
• Popular open data repositories:
33
Download from finelybook www.finelybook.com


2 The original dataset appeared in R. Kelley Pace and Ronald Barry, “Sparse Spatial Autoregressions,” Statistics
& Probability Letters 33, no. 3 (1997): 291–297.
— UC Irvine Machine Learning Repository
— Kaggle datasets
— Amazon’s AWS datasets
• Meta portals (they list open data repositories):
— http://dataportals.org/
— http://opendatamonitor.eu/
— http://quandl.com/
• Other pages listing many popular open data repositories:
— Wikipedia’s list of Machine Learning datasets
— Quora.com question
— Datasets subreddit
In this chapter we chose the California Housing Prices dataset from the StatLib repos‐
itory2 (see Figure 2-1). This dataset was based on data from the 1990 California cen‐
sus. It is not exactly recent (you could still afford a nice house in the Bay Area at the
time), but it has many qualities for learning, so we will pretend it is recent data. We
also added a categorical attribute and removed a few features for teaching purposes.
Figure 2-1. California housing prices
34


Figure 2-1. California housing prices
34
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


3 A piece of information fed to a Machine Learning system is often called a signal in reference to Shannon’s
information theory: you want a high signal/noise ratio.
Look at the Big Picture
Welcome to Machine Learning Housing Corporation! The first task you are asked to
perform is to build a model of housing prices in California using the California cen‐
sus data. This data has metrics such as the population, median income, median hous‐
ing price, and so on for each block group in California. Block groups are the smallest
geographical unit for which the US Census Bureau publishes sample data (a block
group typically has a population of 600 to 3,000 people). We will just call them “dis‐
tricts” for short.
Your model should learn from this data and be able to predict the median housing
price in any district, given all the other metrics.
Since you are a well-organized data scientist, the first thing you do
is to pull out your Machine Learning project checklist. You can


is to pull out your Machine Learning project checklist. You can
start with the one in Appendix B; it should work reasonably well
for most Machine Learning projects but make sure to adapt it to
your needs. In this chapter we will go through many checklist
items, but we will also skip a few, either because they are self-
explanatory or because they will be discussed in later chapters.
Frame the Problem
The first question to ask your boss is what exactly is the business objective; building a
model is probably not the end goal. How does the company expect to use and benefit
from this model? This is important because it will determine how you frame the
problem, what algorithms you will select, what performance measure you will use to
evaluate your model, and how much effort you should spend tweaking it.
Your boss answers that your model’s output (a prediction of a district’s median hous‐
ing price) will be fed to another Machine Learning system (see Figure 2-2), along


ing price) will be fed to another Machine Learning system (see Figure 2-2), along
with many other signals.3 This downstream system will determine whether it is worth
investing in a given area or not. Getting this right is critical, as it directly affects reve‐
nue.
Look at the Big Picture
|
35
Download from finelybook www.finelybook.com


Figure 2-2. A Machine Learning pipeline for real estate investments
Pipelines
A sequence of data processing components is called a data pipeline. Pipelines are very
common in Machine Learning systems, since there is a lot of data to manipulate and
many data transformations to apply.
Components typically run asynchronously. Each component pulls in a large amount
of data, processes it, and spits out the result in another data store, and then some time
later the next component in the pipeline pulls this data and spits out its own output,
and so on. Each component is fairly self-contained: the interface between components
is simply the data store. This makes the system quite simple to grasp (with the help of
a data flow graph), and different teams can focus on different components. Moreover,
if a component breaks down, the downstream components can often continue to run
normally (at least for a while) by just using the last output from the broken compo‐


normally (at least for a while) by just using the last output from the broken compo‐
nent. This makes the architecture quite robust.
On the other hand, a broken component can go unnoticed for some time if proper
monitoring is not implemented. The data gets stale and the overall system’s perfor‐
mance drops.
The next question to ask is what the current solution looks like (if any). It will often
give you a reference performance, as well as insights on how to solve the problem.
Your boss answers that the district housing prices are currently estimated manually
by experts: a team gathers up-to-date information about a district (excluding median
housing prices), and they use complex rules to come up with an estimate. This is
costly and time-consuming, and their estimates are not great; their typical error rate
is about 15%.
Okay, with all this information you are now ready to start designing your system.
First, you need to frame the problem: is it supervised, unsupervised, or Reinforce‐


First, you need to frame the problem: is it supervised, unsupervised, or Reinforce‐
ment Learning? Is it a classification task, a regression task, or something else? Should
36
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


4 The standard deviation, generally denoted σ (the Greek letter sigma), is the square root of the variance, which
is the average of the squared deviation from the mean.
5 When a feature has a bell-shaped normal distribution (also called a Gaussian distribution), which is very com‐
mon, the “68-95-99.7” rule applies: about 68% of the values fall within 1σ of the mean, 95% within 2σ, and
99.7% within 3σ.
you use batch learning or online learning techniques? Before you read on, pause and
try to answer these questions for yourself.
Have you found the answers? Let’s see: it is clearly a typical supervised learning task
since you are given labeled training examples (each instance comes with the expected
output, i.e., the district’s median housing price). Moreover, it is also a typical regres‐
sion task, since you are asked to predict a value. More specifically, this is a multivari‐
ate regression problem since the system will use multiple features to make a prediction


ate regression problem since the system will use multiple features to make a prediction
(it will use the district’s population, the median income, etc.). In the first chapter, you
predicted life satisfaction based on just one feature, the GDP per capita, so it was a
univariate regression problem. Finally, there is no continuous flow of data coming in
the system, there is no particular need to adjust to changing data rapidly, and the data
is small enough to fit in memory, so plain batch learning should do just fine.
If the data was huge, you could either split your batch learning
work across multiple servers (using the MapReduce technique, as
we will see later), or you could use an online learning technique
instead.
Select a Performance Measure
Your next step is to select a performance measure. A typical performance measure for
regression problems is the Root Mean Square Error (RMSE). It measures the standard


regression problems is the Root Mean Square Error (RMSE). It measures the standard
deviation4 of the errors the system makes in its predictions. For example, an RMSE
equal to 50,000 means that about 68% of the system’s predictions fall within $50,000
of the actual value, and about 95% of the predictions fall within $100,000 of the actual
value.5 Equation 2-1 shows the mathematical formula to compute the RMSE.
Equation 2-1. Root Mean Square Error (RMSE)
RMSE �, h =
1
m ∑
i = 1
m
h �i
−y i 2
Look at the Big Picture
|
37
Download from finelybook www.finelybook.com


6 Recall that the transpose operator flips a column vector into a row vector (and vice versa).
Notations
This equation introduces several very common Machine Learning notations that we
will use throughout this book:
• m is the number of instances in the dataset you are measuring the RMSE on.
— For example, if you are evaluating the RMSE on a validation set of 2,000 dis‐
tricts, then m = 2,000.
• x(i) is a vector of all the feature values (excluding the label) of the ith instance in
the dataset, and y(i) is its label (the desired output value for that instance).
— For example, if the first district in the dataset is located at longitude –118.29°,
latitude 33.91°, and it has 1,416 inhabitants with a median income of $38,372,
and the median house value is $156,400 (ignoring the other features for now),
then:
�1 =
−118 . 29
33 . 91
1, 416
38, 372
and:
y 1 = 156, 400
• X is a matrix containing all the feature values (excluding labels) of all instances in


• X is a matrix containing all the feature values (excluding labels) of all instances in
the dataset. There is one row per instance and the ith row is equal to the transpose
of x(i), noted (x(i))T.6
— For example, if the first district is as just described, then the matrix X looks
like this:
�=
�1 T
�2 T
⋮
�1999 T
�2000 T
= −118 . 29 33 . 91 1, 416 38, 372
⋮
⋮
⋮
⋮
38
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


• h is your system’s prediction function, also called a hypothesis. When your system
is given an instance’s feature vector x(i), it outputs a predicted value ŷ(i) = h(x(i))
for that instance (ŷ is pronounced “y-hat”).
— For example, if your system predicts that the median housing price in the first
district is $158,400, then ŷ(1) = h(x(1)) = 158,400. The prediction error for this
district is ŷ(1) – y(1) = 2,000.
• RMSE(X,h) is the cost function measured on the set of examples using your
hypothesis h.
We use lowercase italic font for scalar values (such as m or y(i)) and function names
(such as h), lowercase bold font for vectors (such as x(i)), and uppercase bold font for
matrices (such as X).
Even though the RMSE is generally the preferred performance measure for regression
tasks, in some contexts you may prefer to use another function. For example, suppose
that there are many outlier districts. In that case, you may consider using the Mean


that there are many outlier districts. In that case, you may consider using the Mean
Absolute Error (also called the Average Absolute Deviation; see Equation 2-2):
Equation 2-2. Mean Absolute Error
MAE �, h = 1
m ∑
i = 1
m
h �i
−y i
Both the RMSE and the MAE are ways to measure the distance between two vectors:
the vector of predictions and the vector of target values. Various distance measures,
or norms, are possible:
• Computing the root of a sum of squares (RMSE) corresponds to the Euclidian
norm: it is the notion of distance you are familiar with. It is also called the ℓ2
norm, noted ∥ · ∥2 (or just ∥ · ∥).
• Computing the sum of absolutes (MAE) corresponds to the ℓ1 norm, noted ∥ · ∥1.
It is sometimes called the Manhattan norm because it measures the distance
between two points in a city if you can only travel along orthogonal city blocks.
• More generally, the ℓk norm of a vector v containing n elements is defined as
∥�∥k =
v0
k + v1
k + ⋯+ vn
k
1


∥�∥k =
v0
k + v1
k + ⋯+ vn
k
1
k. ℓ0 just gives the cardinality of the vector (i.e.,
the number of elements), and ℓ∞ gives the maximum absolute value in the vector.
• The higher the norm index, the more it focuses on large values and neglects small
ones. This is why the RMSE is more sensitive to outliers than the MAE. But when
Look at the Big Picture
|
39
Download from finelybook www.finelybook.com


7 The latest version of Python 3 is recommended. Python 2.7+ should work fine too, but it is deprecated.
outliers are exponentially rare (like in a bell-shaped curve), the RMSE performs
very well and is generally preferred.
Check the Assumptions
Lastly, it is good practice to list and verify the assumptions that were made so far (by
you or others); this can catch serious issues early on. For example, the district prices
that your system outputs are going to be fed into a downstream Machine Learning
system, and we assume that these prices are going to be used as such. But what if the
downstream system actually converts the prices into categories (e.g., “cheap,”
“medium,” or “expensive”) and then uses those categories instead of the prices them‐
selves? In this case, getting the price perfectly right is not important at all; your sys‐
tem just needs to get the category right. If that’s so, then the problem should have


tem just needs to get the category right. If that’s so, then the problem should have
been framed as a classification task, not a regression task. You don’t want to find this
out after working on a regression system for months.
Fortunately, after talking with the team in charge of the downstream system, you are
confident that they do indeed need the actual prices, not just categories. Great! You’re
all set, the lights are green, and you can start coding now!
Get the Data
It’s time to get your hands dirty. Don’t hesitate to pick up your laptop and walk
through the following code examples in a Jupyter notebook. The full Jupyter note‐
book is available at https://github.com/ageron/handson-ml.
Create the Workspace
First you will need to have Python installed. It is probably already installed on your
system. If not, you can get it at https://www.python.org/.7
Next you need to create a workspace directory for your Machine Learning code and


Next you need to create a workspace directory for your Machine Learning code and
datasets. Open a terminal and type the following commands (after the $ prompts):
$ export ML_PATH="$HOME/ml"      # You can change the path if you prefer
$ mkdir -p $ML_PATH
You will need a number of Python modules: Jupyter, NumPy, Pandas, Matplotlib, and
Scikit-Learn. If you already have Jupyter running with all these modules installed,
you can safely skip to “Download the Data” on page 43. If you don’t have them yet,
there are many ways to install them (and their dependencies). You can use your sys‐
tem’s packaging system (e.g., apt-get on Ubuntu, or MacPorts or HomeBrew on
40
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


8 We will show the installation steps using pip in a bash shell on a Linux or macOS system. You may need to
adapt these commands to your own system. On Windows, we recommend installing Anaconda instead.
9 You may need to have administrator rights to run this command; if so, try prefixing it with sudo.
macOS), install a Scientific Python distribution such as Anaconda and use its packag‐
ing system, or just use Python’s own packaging system, pip, which is included by
default with the Python binary installers (since Python 2.7.9).8 You can check to see if
pip is installed by typing the following command:
$ pip3 --version
pip 9.0.1 from [...]/lib/python3.5/site-packages (python 3.5)
You should make sure you have a recent version of pip installed, at the very least >1.4
to support binary module installation (a.k.a. wheels). To upgrade the pip module,
type:9
$ pip3 install --upgrade pip
Collecting pip
[...]
Successfully installed pip-9.0.1
Creating an Isolated Environment


Collecting pip
[...]
Successfully installed pip-9.0.1
Creating an Isolated Environment
If you would like to work in an isolated environment (which is strongly recom‐
mended so you can work on different projects without having conflicting library ver‐
sions), install virtualenv by running the following pip command:
$ pip3 install --user --upgrade virtualenv
Collecting virtualenv
[...]
Successfully installed virtualenv
Now you can create an isolated Python environment by typing:
$ cd $ML_PATH
$ virtualenv env
Using base prefix '[...]'
New python executable in [...]/ml/env/bin/python3.5
Also creating executable in [...]/ml/env/bin/python
Installing setuptools, pip, wheel...done.
Now every time you want to activate this environment, just open a terminal and type:
$ cd $ML_PATH
$ source env/bin/activate
While the environment is active, any package you install using pip will be installed in
this isolated environment, and Python will only have access to these packages (if you


this isolated environment, and Python will only have access to these packages (if you
also want access to the system’s site packages, you should create the environment
Get the Data
|
41
Download from finelybook www.finelybook.com


10 Note that Jupyter can handle multiple versions of Python, and even many other languages such as R or
Octave.
using virtualenv’s --system-site-packages option). Check out virtualenv’s docu‐
mentation for more information.
Now you can install all the required modules and their dependencies using this sim‐
ple pip command:
$ pip3 install --upgrade jupyter matplotlib numpy pandas scipy scikit-learn
Collecting jupyter
  Downloading jupyter-1.0.0-py2.py3-none-any.whl
Collecting matplotlib
  [...]
To check your installation, try to import every module like this:
$ python3 -c "import jupyter, matplotlib, numpy, pandas, scipy, sklearn"
There should be no output and no error. Now you can fire up Jupyter by typing:
$ jupyter notebook
[I 15:24 NotebookApp] Serving notebooks from local directory: [...]/ml
[I 15:24 NotebookApp] 0 active kernels
[I 15:24 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/


[I 15:24 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/
[I 15:24 NotebookApp] Use Control-C to stop this server and shut down all
kernels (twice to skip confirmation).
A Jupyter server is now running in your terminal, listening to port 8888. You can visit
this server by opening your web browser to http://localhost:8888/ (this usually hap‐
pens automatically when the server starts). You should see your empty workspace
directory (containing only the env directory if you followed the preceding virtualenv
instructions).
Now create a new Python notebook by clicking on the New button and selecting the
appropriate Python version10 (see Figure 2-3).
This does three things: first, it creates a new notebook file called Untitled.ipynb in
your workspace; second, it starts a Jupyter Python kernel to run this notebook; and
third, it opens this notebook in a new tab. You should start by renaming this note‐


third, it opens this notebook in a new tab. You should start by renaming this note‐
book to “Housing” (this will automatically rename the file to Housing.ipynb) by click‐
ing Untitled and typing the new name.
42
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


Figure 2-3. Your workspace in Jupyter
A notebook contains a list of cells. Each cell can contain executable code or formatted
text. Right now the notebook contains only one empty code cell, labeled “In [1]:”. Try
typing print("Hello world!") in the cell, and click on the play button (see
Figure 2-4) or press Shift-Enter. This sends the current cell to this notebook’s Python
kernel, which runs it and returns the output. The result is displayed below the cell,
and since we reached the end of the notebook, a new cell is automatically created. Go
through the User Interface Tour from Jupyter’s Help menu to learn the basics.
Figure 2-4. Hello world Python notebook
Download the Data
In typical environments your data would be available in a relational database (or
some other common datastore) and spread across multiple tables/documents/files. To
Get the Data
|
43
Download from finelybook www.finelybook.com


11 You might also need to check legal constraints, such as private fields that should never be copied to unsafe
datastores.
12 In a real project you would save this code in a Python file, but for now you can just write it in your Jupyter
notebook.
access it, you would first need to get your credentials and access authorizations,11 and
familiarize yourself with the data schema. In this project, however, things are much
simpler: you will just download a single compressed file, housing.tgz, which contains a
comma-separated value (CSV) file called housing.csv with all the data.
You could use your web browser to download it, and run tar xzf housing.tgz to
decompress the file and extract the CSV file, but it is preferable to create a small func‐
tion to do that. It is useful in particular if data changes regularly, as it allows you to
write a small script that you can run whenever you need to fetch the latest data (or


write a small script that you can run whenever you need to fetch the latest data (or
you can set up a scheduled job to do that automatically at regular intervals). Auto‐
mating the process of fetching the data is also useful if you need to install the dataset
on multiple machines.
Here is the function to fetch the data:12
import os
import tarfile
from six.moves import urllib
DOWNLOAD_ROOT = "https://raw.githubusercontent.com/ageron/handson-ml/master/"
HOUSING_PATH = "datasets/housing"
HOUSING_URL = DOWNLOAD_ROOT + HOUSING_PATH + "/housing.tgz"
def fetch_housing_data(housing_url=HOUSING_URL, housing_path=HOUSING_PATH):
    if not os.path.isdir(housing_path):
        os.makedirs(housing_path)
    tgz_path = os.path.join(housing_path, "housing.tgz")
    urllib.request.urlretrieve(housing_url, tgz_path)
    housing_tgz = tarfile.open(tgz_path)
    housing_tgz.extractall(path=housing_path)
    housing_tgz.close()


housing_tgz.extractall(path=housing_path)
    housing_tgz.close()
Now when you call fetch_housing_data(), it creates a datasets/housing directory in
your workspace, downloads the housing.tgz file, and extracts the housing.csv from it in
this directory.
Now let’s load the data using Pandas. Once again you should write a small function to
load the data:
import pandas as pd
def load_housing_data(housing_path=HOUSING_PATH):
    csv_path = os.path.join(housing_path, "housing.csv")
    return pd.read_csv(csv_path)
44
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


This function returns a Pandas DataFrame object containing all the data.
Take a Quick Look at the Data Structure
Let’s take a look at the top five rows using the DataFrame’s head() method (see
Figure 2-5).
Figure 2-5. Top five rows in the dataset
Each row represents one district. There are 10 attributes (you can see the first 6 in the
screenshot): longitude, latitude, housing_median_age, total_rooms, total_bed
rooms,
population,
households,
median_income,
median_house_value, and
ocean_proximity.
The info() method is useful to get a quick description of the data, in particular the
total number of rows, and each attribute’s type and number of non-null values (see
Figure 2-6).
Figure 2-6. Housing info
Get the Data
|
45
Download from finelybook www.finelybook.com


There are 20,640 instances in the dataset, which means that it is fairly small by
Machine Learning standards, but it’s perfect to get started. Notice that the total_bed
rooms attribute has only 20,433 non-null values, meaning that 207 districts are miss‐
ing this feature. We will need to take care of this later.
All attributes are numerical, except the ocean_proximity field. Its type is object, so it
could hold any kind of Python object, but since you loaded this data from a CSV file
you know that it must be a text attribute. When you looked at the top five rows, you
probably noticed that the values in that column were repetitive, which means that it is
probably a categorical attribute. You can find out what categories exist and how many
districts belong to each category by using the value_counts() method:
>>> housing["ocean_proximity"].value_counts()
<1H OCEAN     9136
INLAND        6551
NEAR OCEAN    2658
NEAR BAY      2290
ISLAND           5
Name: ocean_proximity, dtype: int64


NEAR OCEAN    2658
NEAR BAY      2290
ISLAND           5
Name: ocean_proximity, dtype: int64
Let’s look at the other fields. The describe() method shows a summary of the
numerical attributes (Figure 2-7).
Figure 2-7. Summary of each numerical attribute
The count, mean, min, and max rows are self-explanatory. Note that the null values are
ignored (so, for example, count of total_bedrooms is 20,433, not 20,640). The std
row shows the standard deviation (which measures how dispersed the values are).
The 25%, 50%, and 75% rows show the corresponding percentiles: a percentile indi‐
cates the value below which a given percentage of observations in a group of observa‐
tions falls. For example, 25% of the districts have a housing_median_age lower than
46
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


18, while 50% are lower than 29 and 75% are lower than 37. These are often called the
25th percentile (or 1st quartile), the median, and the 75th percentile (or 3rd quartile).
Another quick way to get a feel of the type of data you are dealing with is to plot a
histogram for each numerical attribute. A histogram shows the number of instances
(on the vertical axis) that have a given value range (on the horizontal axis). You can
either plot this one attribute at a time, or you can call the hist() method on the
whole dataset, and it will plot a histogram for each numerical attribute (see
Figure 2-8). For example, you can see that slightly over 800 districts have a
median_house_value equal to about $500,000.
%matplotlib inline   # only in a Jupyter notebook
import matplotlib.pyplot as plt
housing.hist(bins=50, figsize=(20,15))
plt.show()
Figure 2-8. A histogram for each numerical attribute
Get the Data
|
47
Download from finelybook www.finelybook.com


The hist() method relies on Matplotlib, which in turn relies on a
user-specified graphical backend to draw on your screen. So before
you can plot anything, you need to specify which backend Matplot‐
lib should use. The simplest option is to use Jupyter’s magic com‐
mand %matplotlib inline. This tells Jupyter to set up Matplotlib
so it uses Jupyter’s own backend. Plots are then rendered within the
notebook itself. Note that calling show() is optional in a Jupyter
notebook, as Jupyter will automatically display plots when a cell is
executed.
Notice a few things in these histograms:
1. First, the median income attribute does not look like it is expressed in US dollars
(USD). After checking with the team that collected the data, you are told that the
data has been scaled and capped at 15 (actually 15.0001) for higher median
incomes, and at 0.5 (actually 0.4999) for lower median incomes. Working with
preprocessed attributes is common in Machine Learning, and it is not necessarily


preprocessed attributes is common in Machine Learning, and it is not necessarily
a problem, but you should try to understand how the data was computed.
2. The housing median age and the median house value were also capped. The lat‐
ter may be a serious problem since it is your target attribute (your labels). Your
Machine Learning algorithms may learn that prices never go beyond that limit.
You need to check with your client team (the team that will use your system’s out‐
put) to see if this is a problem or not. If they tell you that they need precise pre‐
dictions even beyond $500,000, then you have mainly two options:
a. Collect proper labels for the districts whose labels were capped.
b. Remove those districts from the training set (and also from the test set, since
your system should not be evaluated poorly if it predicts values beyond
$500,000).
3. These attributes have very different scales. We will discuss this later in this chap‐
ter when we explore feature scaling.


ter when we explore feature scaling.
4. Finally, many histograms are tail heavy: they extend much farther to the right of
the median than to the left. This may make it a bit harder for some Machine
Learning algorithms to detect patterns. We will try transforming these attributes
later on to have more bell-shaped distributions.
Hopefully you now have a better understanding of the kind of data you are dealing
with.
48
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


13 You will often see people set the random seed to 42. This number has no special property, other than to be
The Answer to the Ultimate Question of Life, the Universe, and Everything.
Wait! Before you look at the data any further, you need to create a
test set, put it aside, and never look at it.
Create a Test Set
It may sound strange to voluntarily set aside part of the data at this stage. After all,
you have only taken a quick glance at the data, and surely you should learn a whole
lot more about it before you decide what algorithms to use, right? This is true, but
your brain is an amazing pattern detection system, which means that it is highly
prone to overfitting: if you look at the test set, you may stumble upon some seemingly
interesting pattern in the test data that leads you to select a particular kind of
Machine Learning model. When you estimate the generalization error using the test
set, your estimate will be too optimistic and you will launch a system that will not


set, your estimate will be too optimistic and you will launch a system that will not
perform as well as expected. This is called data snooping bias.
Creating a test set is theoretically quite simple: just pick some instances randomly,
typically 20% of the dataset, and set them aside:
import numpy as np
def split_train_test(data, test_ratio):
    shuffled_indices = np.random.permutation(len(data))
    test_set_size = int(len(data) * test_ratio)
    test_indices = shuffled_indices[:test_set_size]
    train_indices = shuffled_indices[test_set_size:]
    return data.iloc[train_indices], data.iloc[test_indices]
You can then use this function like this:
>>> train_set, test_set = split_train_test(housing, 0.2)
>>> print(len(train_set), "train +", len(test_set), "test")
16512 train + 4128 test
Well, this works, but it is not perfect: if you run the program again, it will generate a
different test set! Over time, you (or your Machine Learning algorithms) will get to


different test set! Over time, you (or your Machine Learning algorithms) will get to
see the whole dataset, which is what you want to avoid.
One solution is to save the test set on the first run and then load it in subsequent
runs. Another option is to set the random number generator’s seed (e.g., np.ran
dom.seed(42))13 before calling np.random.permutation(), so that it always generates
the same shuffled indices.
Get the Data
|
49
Download from finelybook www.finelybook.com


14 The location information is actually quite coarse, and as a result many districts will have the exact same ID, so
they will end up in the same set (test or train). This introduces some unfortunate sampling bias.
But both these solutions will break next time you fetch an updated dataset. A com‐
mon solution is to use each instance’s identifier to decide whether or not it should go
in the test set (assuming instances have a unique and immutable identifier). For
example, you could compute a hash of each instance’s identifier, keep only the last
byte of the hash, and put the instance in the test set if this value is lower or equal to
51 (~20% of 256). This ensures that the test set will remain consistent across multiple
runs, even if you refresh the dataset. The new test set will contain 20% of the new
instances, but it will not contain any instance that was previously in the training set.
Here is a possible implementation:
import hashlib


Here is a possible implementation:
import hashlib
def test_set_check(identifier, test_ratio, hash):
    return hash(np.int64(identifier)).digest()[-1] < 256 * test_ratio
def split_train_test_by_id(data, test_ratio, id_column, hash=hashlib.md5):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio, hash))
    return data.loc[~in_test_set], data.loc[in_test_set]
Unfortunately, the housing dataset does not have an identifier column. The simplest
solution is to use the row index as the ID:
housing_with_id = housing.reset_index()   # adds an `index` column
train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, "index")
If you use the row index as a unique identifier, you need to make sure that new data
gets appended to the end of the dataset, and no row ever gets deleted. If this is not
possible, then you can try to use the most stable features to build a unique identifier.


possible, then you can try to use the most stable features to build a unique identifier.
For example, a district’s latitude and longitude are guaranteed to be stable for a few
million years, so you could combine them into an ID like so:14
housing_with_id["id"] = housing["longitude"] * 1000 + housing["latitude"]
train_set, test_set = split_train_test_by_id(housing_with_id, 0.2, "id")
Scikit-Learn provides a few functions to split datasets into multiple subsets in various
ways. The simplest function is train_test_split, which does pretty much the same
thing as the function split_train_test defined earlier, with a couple of additional
features. First there is a random_state parameter that allows you to set the random
generator seed as explained previously, and second you can pass it multiple datasets
with an identical number of rows, and it will split them on the same indices (this is
very useful, for example, if you have a separate DataFrame for labels):
50
|


very useful, for example, if you have a separate DataFrame for labels):
50
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


from sklearn.model_selection import train_test_split
train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)
So far we have considered purely random sampling methods. This is generally fine if
your dataset is large enough (especially relative to the number of attributes), but if it
is not, you run the risk of introducing a significant sampling bias. When a survey
company decides to call 1,000 people to ask them a few questions, they don’t just pick
1,000 people randomly in a phone booth. They try to ensure that these 1,000 people
are representative of the whole population. For example, the US population is com‐
posed of 51.3% female and 48.7% male, so a well-conducted survey in the US would
try to maintain this ratio in the sample: 513 female and 487 male. This is called strati‐
fied sampling: the population is divided into homogeneous subgroups called strata,
and the right number of instances is sampled from each stratum to guarantee that the


and the right number of instances is sampled from each stratum to guarantee that the
test set is representative of the overall population. If they used purely random sam‐
pling, there would be about 12% chance of sampling a skewed test set with either less
than 49% female or more than 54% female. Either way, the survey results would be
significantly biased.
Suppose you chatted with experts who told you that the median income is a very
important attribute to predict median housing prices. You may want to ensure that
the test set is representative of the various categories of incomes in the whole dataset.
Since the median income is a continuous numerical attribute, you first need to create
an income category attribute. Let’s look at the median income histogram more closely
(see Figure 2-9):
Figure 2-9. Histogram of income categories
Most median income values are clustered around 2–5 (tens of thousands of dollars),


Most median income values are clustered around 2–5 (tens of thousands of dollars),
but some median incomes go far beyond 6. It is important to have a sufficient num‐
Get the Data
|
51
Download from finelybook www.finelybook.com


ber of instances in your dataset for each stratum, or else the estimate of the stratum’s
importance may be biased. This means that you should not have too many strata, and
each stratum should be large enough. The following code creates an income category
attribute by dividing the median income by 1.5 (to limit the number of income cate‐
gories), and rounding up using ceil (to have discrete categories), and then merging
all the categories greater than 5 into category 5:
housing["income_cat"] = np.ceil(housing["median_income"] / 1.5)
housing["income_cat"].where(housing["income_cat"] < 5, 5.0, inplace=True)
Now you are ready to do stratified sampling based on the income category. For this
you can use Scikit-Learn’s StratifiedShuffleSplit class:
from sklearn.model_selection import StratifiedShuffleSplit
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing["income_cat"]):


for train_index, test_index in split.split(housing, housing["income_cat"]):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]
Let’s see if this worked as expected. You can start by looking at the income category
proportions in the full housing dataset:
>>> housing["income_cat"].value_counts() / len(housing)
3.0    0.350581
2.0    0.318847
4.0    0.176308
5.0    0.114438
1.0    0.039826
Name: income_cat, dtype: float64
With similar code you can measure the income category proportions in the test set.
Figure 2-10 compares the income category proportions in the overall dataset, in the
test set generated with stratified sampling, and in a test set generated using purely
random sampling. As you can see, the test set generated using stratified sampling has
income category proportions almost identical to those in the full dataset, whereas the
test set generated using purely random sampling is quite skewed.


test set generated using purely random sampling is quite skewed.
Figure 2-10. Sampling bias comparison of stratified versus purely random sampling
52
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


Now you should remove the income_cat attribute so the data is back to its original
state:
for set in (strat_train_set, strat_test_set):
    set.drop(["income_cat"], axis=1, inplace=True)
We spent quite a bit of time on test set generation for a good reason: this is an often
neglected but critical part of a Machine Learning project. Moreover, many of these
ideas will be useful later when we discuss cross-validation. Now it’s time to move on
to the next stage: exploring the data.
Discover and Visualize the Data to Gain Insights
So far you have only taken a quick glance at the data to get a general understanding of
the kind of data you are manipulating. Now the goal is to go a little bit more in depth.
First, make sure you have put the test set aside and you are only exploring the train‐
ing set. Also, if the training set is very large, you may want to sample an exploration
set, to make manipulations easy and fast. In our case, the set is quite small so you can


set, to make manipulations easy and fast. In our case, the set is quite small so you can
just work directly on the full set. Let’s create a copy so you can play with it without
harming the training set:
housing = strat_train_set.copy()
Visualizing Geographical Data
Since there is geographical information (latitude and longitude), it is a good idea to
create a scatterplot of all districts to visualize the data (Figure 2-11):
housing.plot(kind="scatter", x="longitude", y="latitude")
Figure 2-11. A geographical scatterplot of the data
Discover and Visualize the Data to Gain Insights
|
53
Download from finelybook www.finelybook.com


15 If you are reading this in grayscale, grab a red pen and scribble over most of the coastline from the Bay Area
down to San Diego (as you might expect). You can add a patch of yellow around Sacramento as well.
This looks like California all right, but other than that it is hard to see any particular
pattern. Setting the alpha option to 0.1 makes it much easier to visualize the places
where there is a high density of data points (Figure 2-12):
housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.1)
Figure 2-12. A better visualization highlighting high-density areas
Now that’s much better: you can clearly see the high-density areas, namely the Bay
Area and around Los Angeles and San Diego, plus a long line of fairly high density in
the Central Valley, in particular around Sacramento and Fresno.
More generally, our brains are very good at spotting patterns on pictures, but you
may need to play around with visualization parameters to make the patterns stand
out.


may need to play around with visualization parameters to make the patterns stand
out.
Now let’s look at the housing prices (Figure 2-13). The radius of each circle represents
the district’s population (option s), and the color represents the price (option c). We
will use a predefined color map (option cmap) called jet, which ranges from blue
(low values) to red (high prices):15
housing.plot(kind="scatter", x="longitude", y="latitude", alpha=0.4,
    s=housing["population"]/100, label="population",
    c="median_house_value", cmap=plt.get_cmap("jet"), colorbar=True,
)
plt.legend()
54
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


Figure 2-13. California housing prices
This image tells you that the housing prices are very much related to the location
(e.g., close to the ocean) and to the population density, as you probably knew already.
It will probably be useful to use a clustering algorithm to detect the main clusters, and
add new features that measure the proximity to the cluster centers. The ocean prox‐
imity attribute may be useful as well, although in Northern California the housing
prices in coastal districts are not too high, so it is not a simple rule.
Looking for Correlations
Since the dataset is not too large, you can easily compute the standard correlation
coefficient (also called Pearson’s r) between every pair of attributes using the corr()
method:
corr_matrix = housing.corr()
Now let’s look at how much each attribute correlates with the median house value:
>>> corr_matrix["median_house_value"].sort_values(ascending=False)
median_house_value    1.000000
median_income         0.687170


median_house_value    1.000000
median_income         0.687170
total_rooms           0.135231
housing_median_age    0.114220
households            0.064702
Discover and Visualize the Data to Gain Insights
|
55
Download from finelybook www.finelybook.com


total_bedrooms        0.047865
population           -0.026699
longitude            -0.047279
latitude             -0.142826
Name: median_house_value, dtype: float64
The correlation coefficient ranges from –1 to 1. When it is close to 1, it means that
there is a strong positive correlation; for example, the median house value tends to go
up when the median income goes up. When the coefficient is close to –1, it means
that there is a strong negative correlation; you can see a small negative correlation
between the latitude and the median house value (i.e., prices have a slight tendency to
go down when you go north). Finally, coefficients close to zero mean that there is no
linear correlation. Figure 2-14 shows various plots along with the correlation coeffi‐
cient between their horizontal and vertical axes.
Figure 2-14. Standard correlation coefficient of various datasets (source: Wikipedia;
public domain image)
The correlation coefficient only measures linear correlations (“if x


public domain image)
The correlation coefficient only measures linear correlations (“if x
goes up, then y generally goes up/down”). It may completely miss
out on nonlinear relationships (e.g., “if x is close to zero then y gen‐
erally goes up”). Note how all the plots of the bottom row have a
correlation coefficient equal to zero despite the fact that their axes
are clearly not independent: these are examples of nonlinear rela‐
tionships. Also, the second row shows examples where the correla‐
tion coefficient is equal to 1 or –1; notice that this has nothing to
do with the slope. For example, your height in inches has a correla‐
tion coefficient of 1 with your height in feet or in nanometers.
Another way to check for correlation between attributes is to use Pandas’
scatter_matrix function, which plots every numerical attribute against every other
numerical attribute. Since there are now 11 numerical attributes, you would get 112 =
56
|
Chapter 2: End-to-End Machine Learning Project


56
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


121 plots, which would not fit on a page, so let’s just focus on a few promising
attributes that seem most correlated with the median housing value (Figure 2-15):
from pandas.tools.plotting import scatter_matrix
attributes = ["median_house_value", "median_income", "total_rooms",
              "housing_median_age"]
scatter_matrix(housing[attributes], figsize=(12, 8))
Figure 2-15. Scatter matrix
The main diagonal (top left to bottom right) would be full of straight lines if Pandas
plotted each variable against itself, which would not be very useful. So instead Pandas
displays a histogram of each attribute (other options are available; see Pandas’ docu‐
mentation for more details).
The most promising attribute to predict the median house value is the median
income, so let’s zoom in on their correlation scatterplot (Figure 2-16):
housing.plot(kind="scatter", x="median_income", y="median_house_value",
             alpha=0.1)
Discover and Visualize the Data to Gain Insights
|
57


alpha=0.1)
Discover and Visualize the Data to Gain Insights
|
57
Download from finelybook www.finelybook.com


Figure 2-16. Median income versus median house value
This plot reveals a few things. First, the correlation is indeed very strong; you can
clearly see the upward trend and the points are not too dispersed. Second, the price
cap that we noticed earlier is clearly visible as a horizontal line at $500,000. But this
plot reveals other less obvious straight lines: a horizontal line around $450,000,
another around $350,000, perhaps one around $280,000, and a few more below that.
You may want to try removing the corresponding districts to prevent your algorithms
from learning to reproduce these data quirks.
Experimenting with Attribute Combinations
Hopefully the previous sections gave you an idea of a few ways you can explore the
data and gain insights. You identified a few data quirks that you may want to clean up
before feeding the data to a Machine Learning algorithm, and you found interesting
correlations between attributes, in particular with the target attribute. You also


correlations between attributes, in particular with the target attribute. You also
noticed that some attributes have a tail-heavy distribution, so you may want to trans‐
form them (e.g., by computing their logarithm). Of course, your mileage will vary
considerably with each project, but the general ideas are similar.
One last thing you may want to do before actually preparing the data for Machine
Learning algorithms is to try out various attribute combinations. For example, the
total number of rooms in a district is not very useful if you don’t know how many
households there are. What you really want is the number of rooms per household.
Similarly, the total number of bedrooms by itself is not very useful: you probably
want to compare it to the number of rooms. And the population per household also
58
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


seems like an interesting attribute combination to look at. Let’s create these new
attributes:
housing["rooms_per_household"] = housing["total_rooms"]/housing["households"]
housing["bedrooms_per_room"] = housing["total_bedrooms"]/housing["total_rooms"]
housing["population_per_household"]=housing["population"]/housing["households"]
And now let’s look at the correlation matrix again:
>>> corr_matrix = housing.corr()
>>> corr_matrix["median_house_value"].sort_values(ascending=False)
median_house_value          1.000000
median_income               0.687170
rooms_per_household         0.199343
total_rooms                 0.135231
housing_median_age          0.114220
households                  0.064702
total_bedrooms              0.047865
population_per_household   -0.021984
population                 -0.026699
longitude                  -0.047279
latitude                   -0.142826
bedrooms_per_room          -0.260070
Name: median_house_value, dtype: float64


bedrooms_per_room          -0.260070
Name: median_house_value, dtype: float64
Hey, not bad! The new bedrooms_per_room attribute is much more correlated with
the median house value than the total number of rooms or bedrooms. Apparently
houses with a lower bedroom/room ratio tend to be more expensive. The number of
rooms per household is also more informative than the total number of rooms in a
district—obviously the larger the houses, the more expensive they are.
This round of exploration does not have to be absolutely thorough; the point is to
start off on the right foot and quickly gain insights that will help you get a first rea‐
sonably good prototype. But this is an iterative process: once you get a prototype up
and running, you can analyze its output to gain more insights and come back to this
exploration step.
Prepare the Data for Machine Learning Algorithms
It’s time to prepare the data for your Machine Learning algorithms. Instead of just


It’s time to prepare the data for your Machine Learning algorithms. Instead of just
doing this manually, you should write functions to do that, for several good reasons:
• This will allow you to reproduce these transformations easily on any dataset (e.g.,
the next time you get a fresh dataset).
• You will gradually build a library of transformation functions that you can reuse
in future projects.
• You can use these functions in your live system to transform the new data before
feeding it to your algorithms.
Prepare the Data for Machine Learning Algorithms
|
59
Download from finelybook www.finelybook.com


• This will make it possible for you to easily try various transformations and see
which combination of transformations works best.
But first let’s revert to a clean training set (by copying strat_train_set once again),
and let’s separate the predictors and the labels since we don’t necessarily want to apply
the same transformations to the predictors and the target values (note that drop()
creates a copy of the data and does not affect strat_train_set):
housing = strat_train_set.drop("median_house_value", axis=1)
housing_labels = strat_train_set["median_house_value"].copy()
Data Cleaning
Most Machine Learning algorithms cannot work with missing features, so let’s create
a few functions to take care of them. You noticed earlier that the total_bedrooms
attribute has some missing values, so let’s fix this. You have three options:
• Get rid of the corresponding districts.
• Get rid of the whole attribute.
• Set the values to some value (zero, the mean, the median, etc.).


• Set the values to some value (zero, the mean, the median, etc.).
You can accomplish these easily using DataFrame’s dropna(), drop(), and fillna()
methods:
housing.dropna(subset=["total_bedrooms"])    # option 1
housing.drop("total_bedrooms", axis=1)       # option 2
median = housing["total_bedrooms"].median()
housing["total_bedrooms"].fillna(median)     # option 3
If you choose option 3, you should compute the median value on the training set, and
use it to fill the missing values in the training set, but also don’t forget to save the
median value that you have computed. You will need it later to replace missing values
in the test set when you want to evaluate your system, and also once the system goes
live to replace missing values in new data.
Scikit-Learn provides a handy class to take care of missing values: Imputer. Here is
how to use it. First, you need to create an Imputer instance, specifying that you want


how to use it. First, you need to create an Imputer instance, specifying that you want
to replace each attribute’s missing values with the median of that attribute:
from sklearn.preprocessing import Imputer
imputer = Imputer(strategy="median")
Since the median can only be computed on numerical attributes, we need to create a
copy of the data without the text attribute ocean_proximity:
housing_num = housing.drop("ocean_proximity", axis=1)
60
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


16 For more details on the design principles, see “API design for machine learning software: experiences from
the scikit-learn project,” L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Müller, et al. (2013).
Now you can fit the imputer instance to the training data using the fit() method:
imputer.fit(housing_num)
The imputer has simply computed the median of each attribute and stored the result
in its statistics_ instance variable. Only the total_bedrooms attribute had missing
values, but we cannot be sure that there won’t be any missing values in new data after
the system goes live, so it is safer to apply the imputer to all the numerical attributes:
>>> imputer.statistics_
array([ -118.51 , 34.26 , 29. , 2119. , 433. , 1164. , 408. , 3.5414])
>>> housing_num.median().values
array([ -118.51 , 34.26 , 29. , 2119. , 433. , 1164. , 408. , 3.5414])
Now you can use this “trained” imputer to transform the training set by replacing
missing values by the learned medians:


missing values by the learned medians:
X = imputer.transform(housing_num)
The result is a plain Numpy array containing the transformed features. If you want to
put it back into a Pandas DataFrame, it’s simple:
housing_tr = pd.DataFrame(X, columns=housing_num.columns)
Scikit-Learn Design
Scikit-Learn’s API is remarkably well designed. The main design principles are:16
• Consistency. All objects share a consistent and simple interface:
— Estimators. Any object that can estimate some parameters based on a dataset
is called an estimator (e.g., an imputer is an estimator). The estimation itself is
performed by the fit() method, and it takes only a dataset as a parameter (or
two for supervised learning algorithms; the second dataset contains the
labels). Any other parameter needed to guide the estimation process is con‐
sidered a hyperparameter (such as an imputer’s strategy), and it must be set
as an instance variable (generally via a constructor parameter).


as an instance variable (generally via a constructor parameter).
— Transformers. Some estimators (such as an imputer) can also transform a
dataset; these are called transformers. Once again, the API is quite simple: the
transformation is performed by the transform() method with the dataset to
transform as a parameter. It returns the transformed dataset. This transforma‐
tion generally relies on the learned parameters, as is the case for an imputer.
All transformers also have a convenience method called fit_transform()
Prepare the Data for Machine Learning Algorithms
|
61
Download from finelybook www.finelybook.com


17 Some predictors also provide methods to measure the confidence of their predictions.
that is equivalent to calling fit() and then transform() (but sometimes
fit_transform() is optimized and runs much faster).
— Predictors. Finally, some estimators are capable of making predictions given a
dataset; they are called predictors. For example, the LinearRegression model
in the previous chapter was a predictor: it predicted life satisfaction given a
country’s GDP per capita. A predictor has a predict() method that takes a
dataset of new instances and returns a dataset of corresponding predictions. It
also has a score() method that measures the quality of the predictions given
a test set (and the corresponding labels in the case of supervised learning
algorithms).17
• Inspection. All the estimator’s hyperparameters are accessible directly via public
instance variables (e.g., imputer.strategy), and all the estimator’s learned


instance variables (e.g., imputer.strategy), and all the estimator’s learned
parameters are also accessible via public instance variables with an underscore
suffix (e.g., imputer.statistics_).
• Nonproliferation of classes. Datasets are represented as NumPy arrays or SciPy
sparse matrices, instead of homemade classes. Hyperparameters are just regular
Python strings or numbers.
• Composition. Existing building blocks are reused as much as possible. For
example, it is easy to create a Pipeline estimator from an arbitrary sequence of
transformers followed by a final estimator, as we will see.
• Sensible defaults. Scikit-Learn provides reasonable default values for most
parameters, making it easy to create a baseline working system quickly.
Handling Text and Categorical Attributes
Earlier we left out the categorical attribute ocean_proximity because it is a text
attribute so we cannot compute its median. Most Machine Learning algorithms pre‐


attribute so we cannot compute its median. Most Machine Learning algorithms pre‐
fer to work with numbers anyway, so let’s convert these text labels to numbers.
Scikit-Learn provides a transformer for this task called LabelEncoder:
>>> from sklearn.preprocessing import LabelEncoder
>>> encoder = LabelEncoder()
>>> housing_cat = housing["ocean_proximity"]
>>> housing_cat_encoded = encoder.fit_transform(housing_cat)
>>> housing_cat_encoded
array([1, 1, 4, ..., 1, 0, 3])
62
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


18 NumPy’s reshape() function allows one dimension to be –1, which means “unspecified”: the value is inferred
from the length of the array and the remaining dimensions.
19 See SciPy’s documentation for more details.
This is better: now we can use this numerical data in any ML algorithm. You can look
at the mapping that this encoder has learned using the classes_ attribute (“<1H
OCEAN” is mapped to 0, “INLAND” is mapped to 1, etc.):
>>> print(encoder.classes_)
['<1H OCEAN' 'INLAND' 'ISLAND' 'NEAR BAY' 'NEAR OCEAN']
One issue with this representation is that ML algorithms will assume that two nearby
values are more similar than two distant values. Obviously this is not the case (for
example, categories 0 and 4 are more similar than categories 0 and 1). To fix this
issue, a common solution is to create one binary attribute per category: one attribute
equal to 1 when the category is “<1H OCEAN” (and 0 otherwise), another attribute


equal to 1 when the category is “<1H OCEAN” (and 0 otherwise), another attribute
equal to 1 when the category is “INLAND” (and 0 otherwise), and so on. This is
called one-hot encoding, because only one attribute will be equal to 1 (hot), while the
others will be 0 (cold).
Scikit-Learn provides a OneHotEncoder encoder to convert integer categorical values
into one-hot vectors. Let’s encode the categories as one-hot vectors. Note that
fit_transform() expects a 2D array, but housing_cat_encoded is a 1D array, so we
need to reshape it:18
>>> from sklearn.preprocessing import OneHotEncoder
>>> encoder = OneHotEncoder()
>>> housing_cat_1hot = encoder.fit_transform(housing_cat_encoded.reshape(-1,1))
>>> housing_cat_1hot
<16513x5 sparse matrix of type '<class 'numpy.float64'>'

with 16513 stored elements in Compressed Sparse Row format>
Notice that the output is a SciPy sparse matrix, instead of a NumPy array. This is very


Notice that the output is a SciPy sparse matrix, instead of a NumPy array. This is very
useful when you have categorical attributes with thousands of categories. After one-
hot encoding we get a matrix with thousands of columns, and the matrix is full of
zeros except for one 1 per row. Using up tons of memory mostly to store zeros would
be very wasteful, so instead a sparse matrix only stores the location of the nonzero
elements. You can use it mostly like a normal 2D array,19 but if you really want to con‐
vert it to a (dense) NumPy array, just call the toarray() method:
>>> housing_cat_1hot.toarray()
array([[ 0.,  1.,  0.,  0.,  0.],
       [ 0.,  1.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  1.],
       ...,
       [ 0.,  1.,  0.,  0.,  0.],
Prepare the Data for Machine Learning Algorithms
|
63
Download from finelybook www.finelybook.com


[ 1.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  1.,  0.]])
We can apply both transformations (from text categories to integer categories, then
from integer categories to one-hot vectors) in one shot using the LabelBinarizer
class:
>>> from sklearn.preprocessing import LabelBinarizer
>>> encoder = LabelBinarizer()
>>> housing_cat_1hot = encoder.fit_transform(housing_cat)
>>> housing_cat_1hot
array([[0, 1, 0, 0, 0],
       [0, 1, 0, 0, 0],
       [0, 0, 0, 0, 1],
       ...,
       [0, 1, 0, 0, 0],
       [1, 0, 0, 0, 0],
       [0, 0, 0, 1, 0]])
Note that this returns a dense NumPy array by default. You can get a sparse matrix
instead by passing sparse_output=True to the LabelBinarizer constructor.
Custom Transformers
Although Scikit-Learn provides many useful transformers, you will need to write
your own for tasks such as custom cleanup operations or combining specific
attributes. You will want your transformer to work seamlessly with Scikit-Learn func‐


attributes. You will want your transformer to work seamlessly with Scikit-Learn func‐
tionalities (such as pipelines), and since Scikit-Learn relies on duck typing (not inher‐
itance), all you need is to create a class and implement three methods: fit()
(returning self), transform(), and fit_transform(). You can get the last one for
free by simply adding TransformerMixin as a base class. Also, if you add BaseEstima
tor as a base class (and avoid *args and **kargs in your constructor) you will get
two extra methods (get_params() and set_params()) that will be useful for auto‐
matic hyperparameter tuning. For example, here is a small transformer class that adds
the combined attributes we discussed earlier:
from sklearn.base import BaseEstimator, TransformerMixin
rooms_ix, bedrooms_ix, population_ix, household_ix = 3, 4, 5, 6
class CombinedAttributesAdder(BaseEstimator, TransformerMixin):
    def __init__(self, add_bedrooms_per_room = True): # no *args or **kargs


def __init__(self, add_bedrooms_per_room = True): # no *args or **kargs
        self.add_bedrooms_per_room = add_bedrooms_per_room
    def fit(self, X, y=None):
        return self  # nothing else to do
    def transform(self, X, y=None):
        rooms_per_household = X[:, rooms_ix] / X[:, household_ix]
        population_per_household = X[:, population_ix] / X[:, household_ix]
        if self.add_bedrooms_per_room:
            bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]
64
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


return np.c_[X, rooms_per_household, population_per_household,
                         bedrooms_per_room]
        else:
            return np.c_[X, rooms_per_household, population_per_household]
attr_adder = CombinedAttributesAdder(add_bedrooms_per_room=False)
housing_extra_attribs = attr_adder.transform(housing.values)
In this example the transformer has one hyperparameter, add_bedrooms_per_room,
set to True by default (it is often helpful to provide sensible defaults). This hyperpara‐
meter will allow you to easily find out whether adding this attribute helps the
Machine Learning algorithms or not. More generally, you can add a hyperparameter
to gate any data preparation step that you are not 100% sure about. The more you
automate these data preparation steps, the more combinations you can automatically
try out, making it much more likely that you will find a great combination (and sav‐
ing you a lot of time).
Feature Scaling


ing you a lot of time).
Feature Scaling
One of the most important transformations you need to apply to your data is feature
scaling. With few exceptions, Machine Learning algorithms don’t perform well when
the input numerical attributes have very different scales. This is the case for the hous‐
ing data: the total number of rooms ranges from about 6 to 39,320, while the median
incomes only range from 0 to 15. Note that scaling the target values is generally not
required.
There are two common ways to get all attributes to have the same scale: min-max
scaling and standardization.
Min-max scaling (many people call this normalization) is quite simple: values are
shifted and rescaled so that they end up ranging from 0 to 1. We do this by subtract‐
ing the min value and dividing by the max minus the min. Scikit-Learn provides a
transformer called MinMaxScaler for this. It has a feature_range hyperparameter
that lets you change the range if you don’t want 0–1 for some reason.


that lets you change the range if you don’t want 0–1 for some reason.
Standardization is quite different: first it subtracts the mean value (so standardized
values always have a zero mean), and then it divides by the variance so that the result‐
ing distribution has unit variance. Unlike min-max scaling, standardization does not
bound values to a specific range, which may be a problem for some algorithms (e.g.,
neural networks often expect an input value ranging from 0 to 1). However, standard‐
ization is much less affected by outliers. For example, suppose a district had a median
income equal to 100 (by mistake). Min-max scaling would then crush all the other
values from 0–15 down to 0–0.15, whereas standardization would not be much affec‐
ted. Scikit-Learn provides a transformer called StandardScaler for standardization.
Prepare the Data for Machine Learning Algorithms
|
65
Download from finelybook www.finelybook.com


As with all the transformations, it is important to fit the scalers to
the training data only, not to the full dataset (including the test set).
Only then can you use them to transform the training set and the
test set (and new data).
Transformation Pipelines
As you can see, there are many data transformation steps that need to be executed in
the right order. Fortunately, Scikit-Learn provides the Pipeline class to help with
such sequences of transformations. Here is a small pipeline for the numerical
attributes:
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
num_pipeline = Pipeline([
        ('imputer', Imputer(strategy="median")),
        ('attribs_adder', CombinedAttributesAdder()),
        ('std_scaler', StandardScaler()),
    ])
housing_num_tr = num_pipeline.fit_transform(housing_num)
The Pipeline constructor takes a list of name/estimator pairs defining a sequence of


The Pipeline constructor takes a list of name/estimator pairs defining a sequence of
steps. All but the last estimator must be transformers (i.e., they must have a
fit_transform() method). The names can be anything you like.
When you call the pipeline’s fit() method, it calls fit_transform() sequentially on
all transformers, passing the output of each call as the parameter to the next call, until
it reaches the final estimator, for which it just calls the fit() method.
The pipeline exposes the same methods as the final estimator. In this example, the last
estimator is a StandardScaler, which is a transformer, so the pipeline has a trans
form() method that applies all the transforms to the data in sequence (it also has a
fit_transform method that we could have used instead of calling fit() and then
transform()).
You now have a pipeline for numerical values, and you also need to apply the LabelBi
narizer on the categorical values: how can you join these transformations into a sin‐


narizer on the categorical values: how can you join these transformations into a sin‐
gle pipeline? Scikit-Learn provides a FeatureUnion class for this. You give it a list of
transformers (which can be entire transformer pipelines), and when its transform()
method is called it runs each transformer’s transform() method in parallel, waits for
their output, and then concatenates them and returns the result (and of course calling
its fit() method calls all each transformer’s fit() method). A full pipeline handling
both numerical and categorical attributes may look like this:
66
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


20 But check out Pull Request #3886, which may introduce a ColumnTransformer class making attribute-specific
transformations easy. You could also run pip3 install sklearn-pandas to get a DataFrameMapper class with
a similar objective.
from sklearn.pipeline import FeatureUnion
num_attribs = list(housing_num)
cat_attribs = ["ocean_proximity"]
num_pipeline = Pipeline([
        ('selector', DataFrameSelector(num_attribs)),
        ('imputer', Imputer(strategy="median")),
        ('attribs_adder', CombinedAttributesAdder()),
        ('std_scaler', StandardScaler()),
    ])
cat_pipeline = Pipeline([
        ('selector', DataFrameSelector(cat_attribs)),
        ('label_binarizer', LabelBinarizer()),
    ])
full_pipeline = FeatureUnion(transformer_list=[
        ("num_pipeline", num_pipeline),
        ("cat_pipeline", cat_pipeline),
    ])
And you can run the whole pipeline simply:
>>> housing_prepared = full_pipeline.fit_transform(housing)
>>> housing_prepared


>>> housing_prepared = full_pipeline.fit_transform(housing)
>>> housing_prepared
array([[ 0.73225807, -0.67331551,  0.58426443, ...,  0.        ,
         0.        ,  0.        ],
       [-0.99102923,  1.63234656, -0.92655887, ...,  0.        ,
         0.        ,  0.        ],
       [...]
>>> housing_prepared.shape
(16513, 17)
Each subpipeline starts with a selector transformer: it simply transforms the data by
selecting the desired attributes (numerical or categorical), dropping the rest, and con‐
verting the resulting DataFrame to a NumPy array. There is nothing in Scikit-Learn
to handle Pandas DataFrames,20 so we need to write a simple custom transformer for
this task:
from sklearn.base import BaseEstimator, TransformerMixin
class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
Prepare the Data for Machine Learning Algorithms
|
67


return self
Prepare the Data for Machine Learning Algorithms
|
67
Download from finelybook www.finelybook.com


def transform(self, X):
        return X[self.attribute_names].values
Select and Train a Model
At last! You framed the problem, you got the data and explored it, you sampled a
training set and a test set, and you wrote transformation pipelines to clean up and
prepare your data for Machine Learning algorithms automatically. You are now ready
to select and train a Machine Learning model.
Training and Evaluating on the Training Set
The good news is that thanks to all these previous steps, things are now going to be
much simpler than you might think. Let’s first train a Linear Regression model, like
we did in the previous chapter:
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(housing_prepared, housing_labels)
Done! You now have a working Linear Regression model. Let’s try it out on a few
instances from the training set:
>>> some_data = housing.iloc[:5]
>>> some_labels = housing_labels.iloc[:5]


>>> some_data = housing.iloc[:5]
>>> some_labels = housing_labels.iloc[:5]
>>> some_data_prepared = full_pipeline.transform(some_data)
>>> print("Predictions:\t", lin_reg.predict(some_data_prepared))
Predictions:
 [ 303104.   44800.  308928.  294208.  368704.]
>>> print("Labels:\t\t", list(some_labels))
Labels:
 [359400.0, 69700.0, 302100.0, 301300.0, 351900.0]
It works, although the predictions are not exactly accurate (e.g., the second prediction
is off by more than 50%!). Let’s measure this regression model’s RMSE on the whole
training set using Scikit-Learn’s mean_squared_error function:
>>> from sklearn.metrics import mean_squared_error
>>> housing_predictions = lin_reg.predict(housing_prepared)
>>> lin_mse = mean_squared_error(housing_labels, housing_predictions)
>>> lin_rmse = np.sqrt(lin_mse)
>>> lin_rmse
68628.413493824875
Okay, this is better than nothing but clearly not a great score: most districts’


68628.413493824875
Okay, this is better than nothing but clearly not a great score: most districts’
median_housing_values range between $120,000 and $265,000, so a typical predic‐
tion error of $68,628 is not very satisfying. This is an example of a model underfitting
the training data. When this happens it can mean that the features do not provide
enough information to make good predictions, or that the model is not powerful
enough. As we saw in the previous chapter, the main ways to fix underfitting are to
68
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


select a more powerful model, to feed the training algorithm with better features, or
to reduce the constraints on the model. This model is not regularized, so this rules
out the last option. You could try to add more features (e.g., the log of the popula‐
tion), but first let’s try a more complex model to see how it does.
Let’s train a DecisionTreeRegressor. This is a powerful model, capable of finding
complex nonlinear relationships in the data (Decision Trees are presented in more
detail in Chapter 6). The code should look familiar by now:
from sklearn.tree import DecisionTreeRegressor
tree_reg = DecisionTreeRegressor()
tree_reg.fit(housing_prepared, housing_labels)
Now that the model is trained, let’s evaluate it on the training set:
>>> housing_predictions = tree_reg.predict(housing_prepared)
>>> tree_mse = mean_squared_error(housing_labels, housing_predictions)
>>> tree_rmse = np.sqrt(tree_mse)
>>> tree_rmse
0.0


>>> tree_rmse = np.sqrt(tree_mse)
>>> tree_rmse
0.0
Wait, what!? No error at all? Could this model really be absolutely perfect? Of course,
it is much more likely that the model has badly overfit the data. How can you be sure?
As we saw earlier, you don’t want to touch the test set until you are ready to launch a
model you are confident about, so you need to use part of the training set for train‐
ing, and part for model validation.
Better Evaluation Using Cross-Validation
One way to evaluate the Decision Tree model would be to use the train_test_split
function to split the training set into a smaller training set and a validation set, then
train your models against the smaller training set and evaluate them against the vali‐
dation set. It’s a bit of work, but nothing too difficult and it would work fairly well.
A great alternative is to use Scikit-Learn’s cross-validation feature. The following code


A great alternative is to use Scikit-Learn’s cross-validation feature. The following code
performs K-fold cross-validation: it randomly splits the training set into 10 distinct
subsets called folds, then it trains and evaluates the Decision Tree model 10 times,
picking a different fold for evaluation every time and training on the other 9 folds.
The result is an array containing the 10 evaluation scores:
from sklearn.model_selection import cross_val_score
scores = cross_val_score(tree_reg, housing_prepared, housing_labels,
                         scoring="neg_mean_squared_error", cv=10)
rmse_scores = np.sqrt(-scores)
Select and Train a Model
|
69
Download from finelybook www.finelybook.com


Scikit-Learn cross-validation features expect a utility function
(greater is better) rather than a cost function (lower is better), so
the scoring function is actually the opposite of the MSE (i.e., a neg‐
ative value), which is why the preceding code computes -scores
before calculating the square root.
Let’s look at the results:
>>> def display_scores(scores):
...     print("Scores:", scores)
...     print("Mean:", scores.mean())
...     print("Standard deviation:", scores.std())
...
>>> display_scores(tree_rmse_scores)
Scores: [ 74678.4916885   64766.2398337   69632.86942005  69166.67693232
          71486.76507766  73321.65695983  71860.04741226  71086.32691692
          76934.2726093   69060.93319262]
Mean: 71199.4280043
Standard deviation: 3202.70522793
Now the Decision Tree doesn’t look as good as it did earlier. In fact, it seems to per‐
form worse than the Linear Regression model! Notice that cross-validation allows


form worse than the Linear Regression model! Notice that cross-validation allows
you to get not only an estimate of the performance of your model, but also a measure
of how precise this estimate is (i.e., its standard deviation). The Decision Tree has a
score of approximately 71,200, generally ±3,200. You would not have this information
if you just used one validation set. But cross-validation comes at the cost of training
the model several times, so it is not always possible.
Let’s compute the same scores for the Linear Regression model just to be sure:
>>> lin_scores = cross_val_score(lin_reg, housing_prepared, housing_labels,
...                              scoring="neg_mean_squared_error", cv=10)
...
>>> lin_rmse_scores = np.sqrt(-lin_scores)
>>> display_scores(lin_rmse_scores)
Scores: [ 70423.5893262   65804.84913139  66620.84314068  72510.11362141
          66414.74423281  71958.89083606  67624.90198297  67825.36117664
          72512.36533141  68028.11688067]


72512.36533141  68028.11688067]
Mean: 68972.377566
Standard deviation: 2493.98819069
That’s right: the Decision Tree model is overfitting so badly that it performs worse
than the Linear Regression model.
Let’s try one last model now: the RandomForestRegressor. As we will see in Chap‐
ter 7, Random Forests work by training many Decision Trees on random subsets of
the features, then averaging out their predictions. Building a model on top of many
other models is called Ensemble Learning, and it is often a great way to push ML algo‐
rithms even further. We will skip most of the code since it is essentially the same as
for the other models:
70
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


>>> from sklearn.ensemble import RandomForestRegressor
>>> forest_reg = RandomForestRegressor()
>>> forest_reg.fit(housing_prepared, housing_labels)
>>> [...]
>>> forest_rmse
22542.396440343684
>>> display_scores(forest_rmse_scores)
Scores: [ 53789.2879722   50256.19806622  52521.55342602  53237.44937943
          52428.82176158  55854.61222549  52158.02291609  50093.66125649
          53240.80406125  52761.50852822]
Mean: 52634.1919593
Standard deviation: 1576.20472269
Wow, this is much better: Random Forests look very promising. However, note that
the score on the training set is still much lower than on the validation sets, meaning
that the model is still overfitting the training set. Possible solutions for overfitting are
to simplify the model, constrain it (i.e., regularize it), or get a lot more training data.
However, before you dive much deeper in Random Forests, you should try out many
other models from various categories of Machine Learning algorithms (several Sup‐


other models from various categories of Machine Learning algorithms (several Sup‐
port Vector Machines with different kernels, possibly a neural network, etc.), without
spending too much time tweaking the hyperparameters. The goal is to shortlist a few
(two to five) promising models.
You should save every model you experiment with, so you can
come back easily to any model you want. Make sure you save both
the hyperparameters and the trained parameters, as well as the
cross-validation scores and perhaps the actual predictions as well.
This will allow you to easily compare scores across model types,
and compare the types of errors they make. You can easily save
Scikit-Learn models by using Python’s pickle module, or using
sklearn.externals.joblib, which is more efficient at serializing
large NumPy arrays:
from sklearn.externals import joblib
joblib.dump(my_model, "my_model.pkl")
# and later...
my_model_loaded = joblib.load("my_model.pkl")
Fine-Tune Your Model


# and later...
my_model_loaded = joblib.load("my_model.pkl")
Fine-Tune Your Model
Let’s assume that you now have a shortlist of promising models. You now need to
fine-tune them. Let’s look at a few ways you can do that.
Fine-Tune Your Model
|
71
Download from finelybook www.finelybook.com


Grid Search
One way to do that would be to fiddle with the hyperparameters manually, until you
find a great combination of hyperparameter values. This would be very tedious work,
and you may not have time to explore many combinations.
Instead you should get Scikit-Learn’s GridSearchCV to search for you. All you need to
do is tell it which hyperparameters you want it to experiment with, and what values to
try out, and it will evaluate all the possible combinations of hyperparameter values,
using cross-validation. For example, the following code searches for the best combi‐
nation of hyperparameter values for the RandomForestRegressor:
from sklearn.model_selection import GridSearchCV
param_grid = [
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
  ]
forest_reg = RandomForestRegressor()
grid_search = GridSearchCV(forest_reg, param_grid, cv=5,


]
forest_reg = RandomForestRegressor()
grid_search = GridSearchCV(forest_reg, param_grid, cv=5,
                           scoring='neg_mean_squared_error')
grid_search.fit(housing_prepared, housing_labels)
When you have no idea what value a hyperparameter should have,
a simple approach is to try out consecutive powers of 10 (or a
smaller number if you want a more fine-grained search, as shown
in this example with the n_estimators hyperparameter).
This param_grid tells Scikit-Learn to first evaluate all 3 × 4 = 12 combinations of
n_estimators and max_features hyperparameter values specified in the first dict
(don’t worry about what these hyperparameters mean for now; they will be explained
in Chapter 7), then try all 2 × 3 = 6 combinations of hyperparameter values in the
second dict, but this time with the bootstrap hyperparameter set to False instead of
True (which is the default value for this hyperparameter).


True (which is the default value for this hyperparameter).
All in all, the grid search will explore 12 + 6 = 18 combinations of RandomForestRe
gressor hyperparameter values, and it will train each model five times (since we are
using five-fold cross validation). In other words, all in all, there will be 18 × 5 = 90
rounds of training! It may take quite a long time, but when it is done you can get the
best combination of parameters like this:
>>> grid_search.best_params_
{'max_features': 6, 'n_estimators': 30}
72
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


Since 30 is the maximum value of n_estimators that was evalu‐
ated, you should probably evaluate higher values as well, since the
score may continue to improve.
You can also get the best estimator directly:
>>> grid_search.best_estimator_
RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features=6, max_leaf_nodes=None, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=30, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)
If GridSearchCV is initialized with refit=True (which is the
default), then once it finds the best estimator using cross-
validation, it retrains it on the whole training set. This is usually a
good idea since feeding it more data will likely improve its perfor‐
mance.
And of course the evaluation scores are also available:
>>> cvres = grid_search.cv_results_
... for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):


... for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
...     print(np.sqrt(-mean_score), params)
...
64912.0351358 {'max_features': 2, 'n_estimators': 3}
55535.2786524 {'max_features': 2, 'n_estimators': 10}
52940.2696165 {'max_features': 2, 'n_estimators': 30}
60384.0908354 {'max_features': 4, 'n_estimators': 3}
52709.9199934 {'max_features': 4, 'n_estimators': 10}
50503.5985321 {'max_features': 4, 'n_estimators': 30}
59058.1153485 {'max_features': 6, 'n_estimators': 3}
52172.0292957 {'max_features': 6, 'n_estimators': 10}
49958.9555932 {'max_features': 6, 'n_estimators': 30}
59122.260006 {'max_features': 8, 'n_estimators': 3}
52441.5896087 {'max_features': 8, 'n_estimators': 10}
50041.4899416 {'max_features': 8, 'n_estimators': 30}
62371.1221202 {'bootstrap': False, 'max_features': 2, 'n_estimators': 3}
54572.2557534 {'bootstrap': False, 'max_features': 2, 'n_estimators': 10}
59634.0533132 {'bootstrap': False, 'max_features': 3, 'n_estimators': 3}


59634.0533132 {'bootstrap': False, 'max_features': 3, 'n_estimators': 3}
52456.0883904 {'bootstrap': False, 'max_features': 3, 'n_estimators': 10}
58825.665239 {'bootstrap': False, 'max_features': 4, 'n_estimators': 3}
52012.9945396 {'bootstrap': False, 'max_features': 4, 'n_estimators': 10}
In this example, we obtain the best solution by setting the max_features hyperpara‐
meter to 6, and the n_estimators hyperparameter to 30. The RMSE score for this
combination is 49,959, which is slightly better than the score you got earlier using the
Fine-Tune Your Model
|
73
Download from finelybook www.finelybook.com


default hyperparameter values (which was 52,634). Congratulations, you have suc‐
cessfully fine-tuned your best model!
Don’t forget that you can treat some of the data preparation steps as
hyperparameters. For example, the grid search will automatically
find out whether or not to add a feature you were not sure about
(e.g., using the add_bedrooms_per_room hyperparameter of your
CombinedAttributesAdder transformer). It may similarly be used
to automatically find the best way to handle outliers, missing fea‐
tures, feature selection, and more.
Randomized Search
The grid search approach is fine when you are exploring relatively few combinations,
like in the previous example, but when the hyperparameter search space is large, it is
often preferable to use RandomizedSearchCV instead. This class can be used in much
the same way as the GridSearchCV class, but instead of trying out all possible combi‐
nations, it evaluates a given number of random combinations by selecting a random


nations, it evaluates a given number of random combinations by selecting a random
value for each hyperparameter at every iteration. This approach has two main bene‐
fits:
• If you let the randomized search run for, say, 1,000 iterations, this approach will
explore 1,000 different values for each hyperparameter (instead of just a few val‐
ues per hyperparameter with the grid search approach).
• You have more control over the computing budget you want to allocate to hyper‐
parameter search, simply by setting the number of iterations.
Ensemble Methods
Another way to fine-tune your system is to try to combine the models that perform
best. The group (or “ensemble”) will often perform better than the best individual
model (just like Random Forests perform better than the individual Decision Trees
they rely on), especially if the individual models make very different types of errors.
We will cover this topic in more detail in Chapter 7.
Analyze the Best Models and Their Errors


We will cover this topic in more detail in Chapter 7.
Analyze the Best Models and Their Errors
You will often gain good insights on the problem by inspecting the best models. For
example, the RandomForestRegressor can indicate the relative importance of each
attribute for making accurate predictions:
>>> feature_importances = grid_search.best_estimator_.feature_importances_
>>> feature_importances
array([  7.14156423e-02,   6.76139189e-02,   4.44260894e-02,
74
|
Chapter 2: End-to-End Machine Learning Project
Download from finelybook www.finelybook.com


1.66308583e-02,   1.66076861e-02,   1.82402545e-02,
         1.63458761e-02,   3.26497987e-01,   6.04365775e-02,
         1.13055290e-01,   7.79324766e-02,   1.12166442e-02,
         1.53344918e-01,   8.41308969e-05,   2.68483884e-03,
         3.46681181e-03])
Let’s display these importance scores next to their corresponding attribute names:
>>> extra_attribs = ["rooms_per_hhold", "pop_per_hhold", "bedrooms_per_room"]
>>> cat_one_hot_attribs = list(encoder.classes_)
>>> attributes = num_attribs + extra_attribs + cat_one_hot_attribs
>>> sorted(zip(feature_importances, attributes), reverse=True)
[(0.32649798665134971, 'median_income'),
 (0.15334491760305854, 'INLAND'),
 (0.11305529021187399, 'pop_per_hhold'),
 (0.07793247662544775, 'bedrooms_per_room'),
 (0.071415642259275158, 'longitude'),
 (0.067613918945568688, 'latitude'),
 (0.060436577499703222, 'rooms_per_hhold'),
 (0.04442608939578685, 'housing_median_age'),
 (0.018240254462909437, 'population'),


(0.04442608939578685, 'housing_median_age'),
 (0.018240254462909437, 'population'),
 (0.01663085833886218, 'total_rooms'),
 (0.016607686091288865, 'total_bedrooms'),
 (0.016345876147580776, 'households'),
 (0.011216644219017424, '<1H OCEAN'),
 (0.0034668118081117387, 'NEAR OCEAN'),
 (0.0026848388432755429, 'NEAR BAY'),
 (8.4130896890070617e-05, 'ISLAND')]
With this information, you may want to try dropping some of the less useful features
(e.g., apparently only one ocean_proximity category is really useful, so you could try
dropping the others).
You should also look at the specific errors that your system makes, then try to under‐
stand why it makes them and what could fix the problem (adding extra features or, on
the contrary, getting rid of uninformative ones, cleaning up outliers, etc.).
Evaluate Your System on the Test Set
After tweaking your models for a while, you eventually have a system that performs