applications like speech recognition. For example, if each instance represents the
audio recording of a person pronouncing a vowel, then different pronunciations of
the vowel “a” will activate neurons in the same area of the map, while instances of the
vowel “e” will activate neurons in another area, and intermediate sounds will gener‐
ally activate intermediate neurons on the map.
One important difference with the other dimensionality reduction
techniques discussed in Chapter 8 is that all instances get mapped
to a discrete number of points in the low-dimensional space (one
point per neuron). When there are very few neurons, this techni‐
que is better described as clustering rather than dimensionality
reduction.
522
|
Appendix E: Other Popular ANN Architectures
Download from finelybook www.finelybook.com


4 You can imagine a class of young children with roughly similar skills. One child happens to be slightly better
at basketball. This motivates her to practice more, especially with her friends. After a while, this group of
friends gets so good at basketball that other kids cannot compete. But that’s okay, because the other kids spe‐
cialize in other topics. After a while, the class is full of little specialized groups.
The training algorithm is unsupervised. It works by having all the neurons compete
against each other. First, all the weights are initialized randomly. Then a training
instance is picked randomly and fed to the network. All neurons compute the dis‐
tance between their weight vector and the input vector (this is very different from the
artificial neurons we have seen so far). The neuron that measures the smallest dis‐
tance wins and tweaks its weight vector to be even slightly closer to the input vector,

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


tance wins and tweaks its weight vector to be even slightly closer to the input vector,
making it more likely to win future competitions for other inputs similar to this one.
It also recruits its neighboring neurons, and they too update their weight vector to be
slightly closer to the input vector (but they don’t update their weights as much as the
winner neuron). Then the algorithm picks another training instance and repeats the
process, again and again. This algorithm tends to make nearby neurons gradually
specialize in similar inputs.4
Other Popular ANN Architectures
|
523
Download from finelybook www.finelybook.com


Download from finelybook www.finelybook.com


Index
Symbols
__call__(), 385
ε-greedy policy, 459, 464
ε-insensitive, 155
χ 2 test (see chi square test)
ℓ 0 norm, 39
ℓ 1 and ℓ 2 regularization, 303-304
ℓ 1 norm, 39, 130, 139, 300, 303
ℓ 2 norm, 39, 128-130, 139, 142, 303, 307
ℓ k norm, 39
ℓ ∞ norm, 39
A
accuracy, 4, 83-84
actions, evaluating, 447-448
activation functions, 262-264
active constraints, 504
actors, 463
actual class, 85
AdaBoost, 192-195
Adagrad, 296-298
Adam optimization, 293, 298-300
adaptive learning rate, 297
adaptive moment optimization, 298
agents, 438
AlexNet architecture, 367-368
algorithms
preparing data for, 59-68
AlphaGo, 14, 253, 437, 453
Anaconda, 41
anomaly detection, 12
Apple’s Siri, 253
apply_gradients(), 286, 450
area under the curve (AUC), 92
arg_scope(), 285
array_split(), 217
artificial neural networks (ANNs), 253-274
Boltzmann Machines, 516-518
deep belief networks (DBNs), 519-521
evolution of, 254
Hopfield Networks, 515-516
hyperparameter fine-tuning, 270-272
overview, 253-255


evolution of, 254
Hopfield Networks, 515-516
hyperparameter fine-tuning, 270-272
overview, 253-255
Perceptrons, 257-264
self-organizing maps, 521-523
training a DNN with TensorFlow, 265-270
artificial neuron, 256
(see also artificial neural network (ANN))
assign(), 237
association rule learning, 12
associative memory networks, 515
assumptions, checking, 40
asynchronous updates, 348-349
asynchrous communication, 329-334
atrous_conv2d(), 376
attention mechanism, 409
attributes, 9, 45-48
(see also data structure)
combinations of, 58-59
preprocessed, 48
target, 48
autodiff, 238-239, 507-513
forward-mode, 510-512
manual differentiation, 507
numerical differentiation, 509
reverse-mode, 512-513
symbolic differentiation, 508-509
autoencoders, 411-435
525
Download from finelybook www.finelybook.com


adversarial, 433
contractive, 432
denoising, 424-425
efficient data representations, 412
generative stochastic network (GSN), 433
overcomplete, 424
PCA with undercomplete linear autoen‐
coder, 413
reconstructions, 413
sparse, 426-428
stacked, 415-424
stacked convolutional, 433
undercomplete, 413
variational, 428-432
visualizing features, 421-422
winner-take-all (WTA), 433
automatic differentiating, 231
autonomous driving systems, 379
Average Absolute Deviation, 39
average pooling layer, 364
avg_pool(), 364
B
backpropagation, 261-262, 275, 291, 422
backpropagation through time (BPTT), 389
bagging and pasting, 185-188
out-of-bag evaluation, 187-188
in Scikit-Learn, 186-187
bandwidth saturation, 349-351
BasicLSTMCell, 401
BasicRNNCell, 397-398
Batch Gradient Descent, 114-117, 130
batch learning, 14-15
Batch Normalization, 282-286, 374
operation summary, 282
with TensorFlow, 284-286
batch(), 341
batch_join(), 341
batch_norm(), 284-285
Bellman Optimality Equation, 455


batch(), 341
batch_join(), 341
batch_norm(), 284-285
Bellman Optimality Equation, 455
between-graph replication, 344
bias neurons, 258
bias term, 106
bias/variance tradeoff, 126
biases, 267
binary classifiers, 82, 134
biological neurons, 254-256
black box models, 170
blending, 200-203
Boltzmann Machines, 516-518
(see also restricted Boltzman machines
(RBMs))
boosting, 191-200
AdaBoost, 192-195
Gradient Boosting, 195-200
bootstrap aggregation (see bagging)
bootstrapping, 72, 185, 442, 469
bottleneck layers, 369
brew, 202
C
Caffe model zoo, 291
call__(), 398
CART (Classification and Regression Tree)
algorithm, 170-171, 176
categorical attributes, 62-64
cell wrapper, 392
chi square test, 174
classification versus regression, 8, 101
classifiers
binary, 82
error analysis, 96-99
evaluating, 96
MNIST dataset, 79-81
multiclass, 93-96
multilabel, 100-101
multioutput, 101-102
performance measures, 82-93
precision of, 85
voting, 181-184
clip_by_value(), 286
closed-form equation, 105, 128, 136


precision of, 85
voting, 181-184
clip_by_value(), 286
closed-form equation, 105, 128, 136
cluster specification, 324
clustering algorithms, 10
clusters, 323
coding space, 429
codings, 411
complementary slackness condition, 504
components_, 214
computational complexity, 110, 153, 172
compute_gradients(), 286, 449
concat(), 369
config.gpu_options, 318
ConfigProto, 317
confusion matrix, 84-86, 96-99
connectionism, 260
constrained optimization, 158, 503
Contrastive Divergence, 519
526
|
Index
Download from finelybook www.finelybook.com


control dependencies, 323
conv1d(), 376
conv2d_transpose(), 376
conv3d(), 376
convergence rate, 117
convex function, 113
convolution kernels, 357, 365, 370
convolutional neural networks (CNNs),
353-378
architectures, 365-376
AlexNet, 367-368
GoogleNet, 368-372
LeNet5, 366-367
ResNet, 372-375
convolutional layer, 355-363, 370, 376
feature maps, 358-360
filters, 357
memory requirement, 362-363
evolution of, 354
pooling layer, 363-365
TensorFlow implementation, 360-362
Coordinator class, 338-340
correlation coefficient, 55-58
correlations, finding, 55-58
cost function, 20, 39
in AdaBoost, 193
in adagrad, 297
in artificial neural networks, 264, 267-268
in autodiff, 238
in batch normalization, 285
cross entropy, 367
deep Q-Learning, 465
in Elastic Net, 132
in Gradient Descent, 105, 111-112, 114,
117-119, 200, 275
in Logistic Regression, 135-136
in PG algorithms, 449
in variational autoencoders, 430
in Lasso Regression, 130-131
in Linear Regression, 108, 113


in variational autoencoders, 430
in Lasso Regression, 130-131
in Linear Regression, 108, 113
in Momentum optimization, 294-295
in pretrained layers reuse, 293
in ridge regression, 127-129
in RNNs, 389, 393
stale gradients and, 349
creative sequences, 396
credit assignment problem, 447-448
critics, 463
cross entropy, 140-141, 264, 428, 449
cross-validation, 30, 69-71, 83-84
CUDA library, 315
cuDNN library, 315
curse of dimensionality, 205-207
(see also dimensionality reduction)
custom transformers, 64-65
D
data, 30
(see also test data; training data)
creating workspace for, 40-43
downloading, 43-45
finding correlations in, 55-58
making assumptions about, 30
preparing for Machine Learning algorithms,
59-68
test-set creation, 49-53
working with real data, 33
data augmentation, 309-310
data cleaning, 60-62
data mining, 6
data parallelism, 347-351
asynchronous updates, 348-349
bandwidth saturation, 349-351
synchronous updates, 348
TensorFlow implementation, 351
data pipeline, 36


synchronous updates, 348
TensorFlow implementation, 351
data pipeline, 36
data snooping bias, 49
data structure, 45-48
data visualization, 53-55
DataFrame, 60
dataquest, xvi
decay, 284
decision boundaries, 136-139, 142, 170
decision function, 87, 156-157
Decision Stumps, 195
decision threshold, 87
Decision Trees, 69-70, 167-179, 181
binary trees, 170
class probability estimates, 171
computational complexity, 172
decision boundaries, 170
GINI impurity, 172
instability with, 177-178
numbers of children, 170
predictions, 169-171
Random Forests (see Random Forests)
regression tasks, 175-176
regularization hyperparameters, 173-174
Index
|
527
Download from finelybook www.finelybook.com


training and visualizing, 167-169
decoder, 412
deconvolutional layer, 376
deep autoencoders (see stacked autoencoders)
deep belief networks (DBNs), 13, 519-521
Deep Learning, 437
(see also Reinforcement Learning; Tensor‐
Flow)
about, xiii, xvi
libraries, 230-231
deep neural networks (DNNs), 261, 275-312
(see also Multi-Layer Perceptrons (MLP))
faster optimizers for, 293-302
regularization, 302-310
reusing pretrained layers, 286-293
training guidelines overview, 310
training with TensorFlow, 265-270
training with TF.Learn, 264
unstable gradients, 276
vanishing and exploding gradients, 275-286
Deep Q-Learning, 460-469
Ms. Pac Man example, 460-469
deep Q-network, 460
deep RNNs, 396-400
applying dropout, 399
distributing across multiple GPUs, 397
long sequence difficulties, 400
truncated backpropagation through time,
400
DeepMind, 14, 253, 437, 460
degrees of freedom, 27, 126
denoising autoencoders, 424-425
depth concat layer, 369
depth radius, 368
depthwise_conv2d(), 376
dequeue(), 332


depth concat layer, 369
depth radius, 368
depthwise_conv2d(), 376
dequeue(), 332
dequeue_many(), 332, 334
dequeue_up_to(), 333-334
dequeuing data, 331
describe(), 46
device blocks, 327
device(), 319
dimensionality reduction, 12, 205-225, 411
approaches to
Manifold Learning, 210
projection, 207-209
choosing the right number of dimensions,
215
curse of dimensionality, 205-207
and data visualization, 205
Isomap, 224
LLE (Locally Linear Embedding), 221-223
Multidimensional Scaling, 223-224
PCA (Principal Component Analysis),
211-218
t-Distributed Stochastic Neighbor Embed‐
ding (t-SNE), 224
discount rate, 447
distributed computing, 229
distributed sessions, 328-329
DNNClassifier, 264
drop(), 60
dropconnect, 307
dropna(), 60
dropout, 272, 399
dropout rate, 304
dropout(), 306
DropoutWrapper, 399
DRY (Don’t Repeat Yourself), 247
Dual Averaging, 300
dual numbers, 510
dual problem, 160
duality, 503
dying ReLUs, 279
dynamic placements, 320
dynamic placer, 318
Dynamic Programming, 456


duality, 503
dying ReLUs, 279
dynamic placements, 320
dynamic placer, 318
Dynamic Programming, 456
dynamic unrolling through time, 387
dynamic_rnn(), 387, 398, 409
E
early stopping, 133-134, 198, 272, 303
Elastic Net, 132
embedded device blocks, 327
Embedded Reber grammars, 410
embeddings, 405-407
embedding_lookup(), 406
encoder, 412
Encoder–Decoder, 383
end-of-sequence (EOS) token, 388
energy functions, 516
enqueuing data, 330
Ensemble Learning, 70, 74, 181-203
bagging and pasting, 185-188
boosting, 191-200
in-graph versus between-graph replication,
343-345
Random Forests, 189-191
528
|
Index
Download from finelybook www.finelybook.com


(see also Random Forests)
random patches and random subspaces, 188
stacking, 200-202
entropy impurity measure, 172
environments, in reinforcement learning,
438-447, 459, 464
episodes (in RL), 444, 448-449, 451-452, 469
epochs, 118
ε-insensitive, 155
equality contraints, 504
error analysis, 96-99
estimators, 61
Euclidian norm, 39
eval(), 240
evaluating models, 29-31
explained variance, 215
explained variance ratio, 214
exploding gradients, 276
(see also gradients, vanishing and explod‐
ing)
exploration policies, 459
exponential decay, 284
exponential linear unit (ELU), 280-281
exponential scheduling, 301
Extra-Trees, 190
F
F-1 score, 86-87
face-recognition, 100
fake X server, 443
false positive rate (FPR), 91-93
fan-in, 277, 279
fan-out, 277, 279
feature detection, 411
feature engineering, 25
feature extraction, 12
feature importance, 190-191
feature maps, 220, 357-360, 374
feature scaling, 65
feature selection, 26, 74, 130, 191, 499
feature space, 218, 220


feature scaling, 65
feature selection, 26, 74, 130, 191, 499
feature space, 218, 220
feature vector, 39, 107, 156, 237
features, 9
FeatureUnion, 66
feedforward neural network (FNN), 263
feed_dict, 240
FIFOQueue, 330, 333
fillna(), 60
first-in first-out (FIFO) queues, 330
first-order partial derivatives (Jacobians), 300
fit(), 61, 66, 217
fitness function, 20
fit_inverse_transform=, 221
fit_transform(), 61, 66
folds, 69, 81, 83-84
Follow The Regularized Leader (FTRL), 300
forget gate, 402
forward-mode autodiff, 510-512
framing a problem, 35-37
frozen layers, 289-290
fully_connected(), 267, 278, 284-285, 417
G
game play (see reinforcement learning)
gamma value, 152
gate controllers, 402
Gaussian distribution, 37, 429, 431
Gaussian RBF, 151
Gaussian RBF kernel, 152-153, 163
generalization error, 29
generalized Lagrangian, 504-505
generative autoencoders, 428
generative models, 411, 518
genetic algorithms, 440
geodesic distance, 224
get_variable(), 249-250
GINI impurity, 169, 172


genetic algorithms, 440
geodesic distance, 224
get_variable(), 249-250
GINI impurity, 169, 172
global average pooling, 372
global_step, 466
global_variables(), 308
global_variables_initializer(), 233
Glorot initialization, 276-279
Google, 230
Google Images, 253
Google Photos, 13
GoogleNet architecture, 368-372
gpu_options.per_process_gpu_memory_frac‐
tion, 317
gradient ascent, 441
Gradient Boosted Regression Trees (GBRT),
195
Gradient Boosting, 195-200
Gradient Descent (GD), 105, 111-121, 164, 275,
294, 296
algorithm comparisons, 119-121
automatically computing gradients, 238-239
Batch GD, 114-117, 130
defining, 111
Index
|
529
Download from finelybook www.finelybook.com


local minimum versus global minimum, 112
manually computing gradients, 237
Mini-batch GD, 119-121, 239-241
optimizer, 239
Stochastic GD, 117-119, 148
with TensorFlow, 237-239
Gradient Tree Boosting, 195
GradientDescentOptimizer, 268
gradients(), 238
gradients, vanishing and exploding, 275-286,
400
Batch Normalization, 282-286
Glorot and He initialization, 276-279
gradient clipping, 286
nonsaturating activation functions, 279-281
graphviz, 168
greedy algorithm, 172
grid search, 71-74, 151
group(), 464
GRU (Gated Recurrent Unit) cell, 404-405
H
hailstone sequence, 412
hard margin classification, 146-147
hard voting classifiers, 181-184
harmonic mean, 86
He initialization, 276-279
Heaviside step function, 257
Hebb's rule, 258, 516
Hebbian learning, 259
hidden layers, 261
hierarchical clustering, 10
hinge loss function, 164
histograms, 47-48
hold-out sets, 200
(see also blenders)
Hopfield Networks, 515-516
hyperbolic tangent (htan activation function),
262, 272, 276, 278, 381


Hopfield Networks, 515-516
hyperbolic tangent (htan activation function),
262, 272, 276, 278, 381
hyperparameters, 28, 65, 72-74, 76, 111, 151,
154, 270
(see also neural network hyperparameters)
hyperplane, 157, 210-211, 213, 224
hypothesis, 39
manifold, 210
hypothesis boosting (see boosting)
hypothesis function, 107
hypothesis, null, 174
I
identity matrix, 128, 160
ILSVRC ImageNet challenge, 365
image classification, 365
impurity measures, 169, 172
in-graph replication, 343
inception modules, 369
Inception-v4, 375
incremental learning, 16, 217
inequality constraints, 504
inference, 22, 311, 363, 408
info(), 45
information gain, 173
information theory, 172
init node, 241
input gate, 402
input neurons, 258
input_put_keep_prob, 399
instance-based learning, 17, 21
InteractiveSession, 233
intercept term, 106
Internal Covariate Shift problem, 282
inter_op_parallelism_threads, 322
intra_op_parallelism_threads, 322
inverse_transform(), 221
in_top_k(), 268
irreducible error, 127


intra_op_parallelism_threads, 322
inverse_transform(), 221
in_top_k(), 268
irreducible error, 127
isolated environment, 41-42
Isomap, 224
is_training, 284-285, 399
J
jobs, 323
join(), 325, 339
Jupyter, 40, 42, 48
K
K-fold cross-validation, 69-71, 83
k-Nearest Neighbors, 21, 100
Karush–Kuhn–Tucker (KKT) conditions, 504
keep probability, 306
Keras, 231
Kernel PCA (kPCA), 218-221
kernel trick, 150, 152, 161-164, 218
kernelized SVM, 161-164
kernels, 150-153, 321
Kullback–Leibler divergence, 141, 426
530
|
Index
Download from finelybook www.finelybook.com


L
l1_l2_regularizer(), 303
LabelBinarizer, 66
labels, 8, 37
Lagrange function, 504-505
Lagrange multiplier, 503
landmarks, 151-152
large margin classification, 145-146
Lasso Regression, 130-132
latent loss, 430
latent space, 429
law of large numbers, 183
leaky ReLU, 279
learning rate, 16, 111, 115-118
learning rate scheduling, 118, 300-302
LeNet-5 architecture, 355, 366-367
Levenshtein distance, 153
liblinear library, 153
libsvm library, 154
Linear Discriminant Analysis (LDA), 224
linear models
early stopping, 133-134
Elastic Net, 132
Lasso Regression, 130-132
Linear Regression (see Linear Regression)
regression (see Linear Regression)
Ridge Regression, 127-129, 132
SVM, 145-148
Linear Regression, 20, 68, 105-121, 132
computational complexity, 110
Gradient Descent in, 111-121
learning curves in, 123-127
Normal Equation, 108-110
regularizing models (see regularization)
using Stochastic Gradient Descent (SGD),
119
with TensorFlow, 235-236
linear SVM classification, 145-148


119
with TensorFlow, 235-236
linear SVM classification, 145-148
linear threshold units (LTUs), 257
Lipschitz continuous, 113
LLE (Locally Linear Embedding), 221-223
load_sample_images(), 360
local receptive field, 354
local response normalization, 368
local sessions, 328
location invariance, 363
log loss, 136
logging placements, 320-320
logistic function, 134
Logistic Regression, 9, 134-142
decision boundaries, 136-139
estimating probablities, 134-135
Softmax Regression model, 139-142
training and cost function, 135-136
log_device_placement, 320
LSTM (Long Short-Term Memory) cell,
401-405
M
machine control (see reinforcement learning)
Machine Learning
large-scale projects (see TensorFlow)
notations, 38-39
process example, 33-77
project checklist, 35, 497-502
resources on, xvi-xvii
uses for, xiii-xiv
Machine Learning basics
attributes, 9
challenges, 22-29
algorithm problems, 26-28
training data problems, 25
definition, 4
features, 9
overview, 3
reasons for using, 4-7


training data problems, 25
definition, 4
features, 9
overview, 3
reasons for using, 4-7
spam filter example, 4-6
summary, 28
testing and validating, 29-31
types of systems, 7-22
batch and online learning, 14-17
instance-based versus model-based
learning, 17-22
supervised/unsupervised learning, 8-14
workflow example, 18-22
machine translation (see natural language pro‐
cessing (NLP))
make(), 442
Manhattan norm, 39
manifold assumption/hypothesis, 210
Manifold Learning, 210, 221
(see also LLE (Locally Linear Embedding)
MapReduce, 37
margin violations, 147
Markov chains, 453
Markov decision processes, 453-457
master service, 325
Matplotlib, 40, 48, 91, 97
Index
|
531
Download from finelybook www.finelybook.com


max margin learning, 293
max pooling layer, 363
max-norm regularization, 307-308
max_norm(), 308
max_norm_regularizer(), 308
max_pool(), 364
Mean Absolute Error (MAE), 39-40
mean coding, 429
Mean Square Error (MSE), 107, 237, 426
measure of similarity, 17
memmap, 217
memory cells, 346, 382
Mercer's theorem, 163
meta learner (see blending)
min-max scaling, 65
Mini-batch Gradient Descent, 119-121, 136,
239-241
mini-batches, 15
minimize(), 286, 289, 449, 466
min_after_dequeue, 333
MNIST dataset, 79-81
model parallelism, 345-347
model parameters, 114, 116, 133, 156, 159, 234,
268, 389
defining, 19
model selection, 19
model zoos, 291
model-based learning, 18-22
models
analyzing, 74-75
evaluating on test set, 75-76
moments, 298
Momentum optimization, 294-295
Monte Carlo tree search, 453
Multi-Layer Perceptrons (MLP), 253, 260-263,
446
training with TF.Learn, 264
multiclass classifiers, 93-96
Multidimensional Scaling (MDS), 223
multilabel classifiers, 100-101


multiclass classifiers, 93-96
Multidimensional Scaling (MDS), 223
multilabel classifiers, 100-101
Multinomial Logistic Regression (see Softmax
Regression)
multinomial(), 446
multioutput classifiers, 101-102
MultiRNNCell, 398
multithreaded readers, 338-340
multivariate regression, 37
N
naive Bayes classifiers, 94
name scopes, 245
natural language processing (NLP), 379,
405-410
encoder-decoder network for machine
translation, 407-410
TensorFlow tutorials, 405, 408
word embeddings, 405-407
Nesterov Accelerated Gradient (NAG), 295-296
Nesterov momentum optimization, 295-296
network topology, 270
neural network hyperparameters, 270-272
activation functions, 272
neurons per hidden layer, 272
number of hidden layers, 270-271
neural network policies, 444-447
neurons
biological, 254-256
logical computations with, 256
neuron_layer(), 267
next_batch(), 269
No Free Lunch theorem, 30
node edges, 244
nonlinear dimensionality reduction (NLDR),
221
(see also Kernel PCA; LLE (Locally Linear


nonlinear dimensionality reduction (NLDR),
221
(see also Kernel PCA; LLE (Locally Linear
Embedding))
nonlinear SVM classification, 149-154
computational complexity, 153
Gaussian RBF kernel, 152-153
with polynomial features, 149-150
polynomial kernel, 150-151
similarity features, adding, 151-152
nonparametric models, 173
nonresponse bias, 25
nonsaturating activation functions, 279-281
normal distribution (see Gaussian distribution)
Normal Equation, 108-110
normalization, 65
normalized exponential, 139
norms, 39
notations, 38-39
NP-Complete problems, 172
null hypothesis, 174
numerical differentiation, 509
NumPy, 40
NumPy arrays, 63
NVidia Compute Capability, 314
532
|
Index
Download from finelybook www.finelybook.com


nvidia-smi, 318
n_components, 215
O
observation space, 446
off-policy algorithm, 459
offline learning, 14
one-hot encoding, 63
one-versus-all (OvA) strategy, 94, 141, 165
one-versus-one (OvO) strategy, 94
online learning, 15-17
online SVMs, 164-165
OpenAI Gym, 441-444
operation_timeout_in_ms, 345
Optical Character Recognition (OCR), 3
optimal state value, 455
optimizers, 293-302
AdaGrad, 296-298
Adam optimization, 293, 298-300
Gradient Descent (see Gradient Descent
optimizer)
learning rate scheduling, 300-302
Momentum optimization, 294-295
Nesterov Accelerated Gradient (NAG),
295-296
RMSProp, 298
out-of-bag evaluation, 187-188
out-of-core learning, 16
out-of-memory (OOM) errors, 386
out-of-sample error, 29
OutOfRangeError, 337, 339
output gate, 402
output layer, 261
OutputProjectionWrapper, 392-395
output_put_keep_prob, 399
overcomplete autoencoder, 424
overfitting, 26-28, 49, 147, 152, 173, 176, 272
avoiding through regularization, 302-310
P
p-value, 174
PaddingFIFOQueue, 334


avoiding through regularization, 302-310
P
p-value, 174
PaddingFIFOQueue, 334
Pandas, 40, 44
scatter_matrix, 56-57
parallel distributed computing, 313-352
data parallelism, 347-351
in-graph versus between-graph replication,
343-345
model parallelism, 345-347
multiple devices across multiple servers,
323-342
asynchronous communication using
queues, 329-334
loading training data, 335-342
master and worker services, 325
opening a session, 325
pinning operations across tasks, 326
sharding variables, 327
sharing state across sessions, 328-329
multiple devices on a single machine,
314-323
control dependencies, 323
installation, 314-316
managing the GPU RAM, 317-318
parallel execution, 321-322
placing operations on devices, 318-321
one neural network per device, 342-343
parameter efficiency, 271
parameter matrix, 139
parameter server (ps), 324
parameter space, 114
parameter vector, 107, 111, 135, 139
parametric models, 173
partial derivative, 114
partial_fit(), 217
Pearson's r, 55


parametric models, 173
partial derivative, 114
partial_fit(), 217
Pearson's r, 55
peephole connections, 403
penalties (see rewards, in RL)
percentiles, 46
Perceptron convergence theorem, 259
Perceptrons, 257-264
versus Logistic Regression, 260
training, 258-259
performance measures, 37-40
confusion matrix, 84-86
cross-validation, 83-84
precision and recall, 86-90
ROC (receiver operating characteristic)
curve, 91-93
performance scheduling, 301
permutation(), 49
PG algorithms, 448
photo-hosting services, 13
pinning operations, 326
pip, 41
Pipeline constructor, 66-68
pipelines, 36
placeholder nodes, 239
Index
|
533
Download from finelybook www.finelybook.com


placers (see simple placer; dynamic placer)
policy, 440
policy gradients, 441 (see PG algorithms)
policy space, 440
polynomial features, adding, 149-150
polynomial kernel, 150-151, 162
Polynomial Regression, 106, 121-123
learning curves in, 123-127
pooling kernel, 363
pooling layer, 363-365
power scheduling, 301
precision, 85
precision and recall, 86-90
F-1 score, 86-87
precision/recall (PR) curve, 92
precision/recall tradeoff, 87-90
predetermined piecewise constant learning
rate, 301
predict(), 62
predicted class, 85
predictions, 84-86, 156-157, 169-171
predictors, 8, 62
preloading training data, 335
PReLU (parametric leaky ReLU), 279
preprocessed attributes, 48
pretrained layers reuse, 286-293
auxiliary task, 292-293
caching frozen layers, 290
freezing lower layers, 289
model zoos, 291
other frameworks, 288
TensorFlow model, 287-288
unsupervised pretraining, 291-292
upper layers, 290
Pretty Tensor, 231
primal problem, 160
principal component, 212


upper layers, 290
Pretty Tensor, 231
primal problem, 160
principal component, 212
Principal Component Analysis (PCA), 211-218
explained variance ratios, 214
finding principal components, 212-213
for compression, 216-217
Incremental PCA, 217-218
Kernel PCA (kPCA), 218-221
projecting down to d dimensions, 213
Randomized PCA, 218
Scikit Learn for, 214
variance, preserving, 211-212
probabilistic autoencoders, 428
probabilities, estimating, 134-135, 171
producer functions, 341
projection, 207-209
propositional logic, 254
pruning, 174, 509
Python
isolated environment in, 41-42
notebooks in, 42-43
pickle, 71
pip, 41
Q
Q-Learning algorithm, 458-469
approximate Q-Learning, 460
deep Q-Learning, 460-469
Q-Value Iteration Algorithm, 456
Q-Values, 456
Quadratic Programming (QP) Problems,
159-160
quantizing, 351
queries per second (QPS), 343
QueueRunner, 338-340
queues, 329-334
closing, 333
dequeuing data, 331
enqueuing data, 330
first-in first-out (FIFO), 330
of tuples, 332
PaddingFIFOQueue, 334


enqueuing data, 330
first-in first-out (FIFO), 330
of tuples, 332
PaddingFIFOQueue, 334
RandomShuffleQueue, 333
q_network(), 463
R
Radial Basis Function (RBF), 151
Random Forests, 70-72, 94, 167, 178, 181,
189-191
Extra-Trees, 190
feature importance, 190-191
random initialization, 111, 116, 118, 276
Random Patches and Random Subspaces, 188
randomized leaky ReLU (RReLU), 279
Randomized PCA, 218
randomized search, 74, 270
RandomShuffleQueue, 333, 337
random_uniform(), 237
reader operations, 335
recall, 85
recognition network, 412
reconstruction error, 216
reconstruction loss, 413, 428, 430
534
|
Index
Download from finelybook www.finelybook.com


reconstruction pre-image, 220
reconstructions, 413
recurrent neural networks (RNNs), 379-410
deep RNNs, 396-400
exploration policies, 459
GRU cell, 404-405
input and output sequences, 382-383
LSTM cell, 401-405
natural language processing (NLP), 405-410
in TensorFlow, 384-388
dynamic unrolling through time, 387
static unrolling through time, 385-386
variable length input sequences, 387
variable length output sequences, 388
training, 389-396
backpropagation through time (BPTT),
389
creative sequences, 396
sequence classifiers, 389-391
time series predictions, 392-396
recurrent neurons, 380-383
memory cells, 382
reduce_mean(), 268
reduce_sum(), 427-428, 430, 466
regression, 8
Decision Trees, 175-176
regression models
linear, 68
regression versus classification, 101
regularization, 27-28, 30, 127-134
data augmentation, 309-310
Decision Trees, 173-174
dropout, 304-307
early stopping, 133-134, 303
Elastic Net, 132
Lasso Regression, 130-132
max-norm, 307-308
Ridge Regression, 127-129


Elastic Net, 132
Lasso Regression, 130-132
max-norm, 307-308
Ridge Regression, 127-129
shrinkage, 197
ℓ 1 and ℓ 2 regularization, 303-304
REINFORCE algorithms, 448
Reinforcement Learning (RL), 13-14, 437-470
actions, 447-448
credit assignment problem, 447-448
discount rate, 447
examples of, 438
Markov decision processes, 453-457
neural network policies, 444-447
OpenAI gym, 441-444
PG algorithms, 448-453
policy search, 440-441
Q-Learning algorithm, 458-469
rewards, learning to optimize, 438-439
Temporal Difference (TD) Learning,
457-458
ReLU (rectified linear units), 246-248
ReLU activation, 374
ReLU function, 262, 272, 278-281
relu(z), 266
render(), 442
replay memory, 464
replica_device_setter(), 327
request_stop(), 339
reset(), 442
reset_default_graph(), 234
reshape(), 395
residual errors, 195-196
residual learning, 372
residual network (ResNet), 291, 372-375
residual units, 373
ResNet, 372-375
resource containers, 328-329
restore(), 241
restricted Boltzmann machines (RBMs), 13,


resource containers, 328-329
restore(), 241
restricted Boltzmann machines (RBMs), 13,
291, 518
reuse_variables(), 249
reverse-mode autodiff, 512-513
rewards, in RL, 438-439
rgb_array, 443
Ridge Regression, 127-129, 132
RMSProp, 298
ROC (receiver operating characteristic) curve,
91-93
Root Mean Square Error (RMSE), 37-40, 107
RReLU (randomized leaky ReLU), 279
run(), 233, 345
S
Sampled Softmax, 409
sampling bias, 24-25, 51
sampling noise, 24
save(), 241
Saver node, 241
Scikit Flow, 231
Scikit-Learn, 40
about, xiv
bagging and pasting in, 186-187
CART algorithm, 170-171, 176
Index
|
535
Download from finelybook www.finelybook.com


cross-validation, 69-71
design principles, 61-62
imputer, 60-62
LinearSVR class, 156
MinMaxScaler, 65
min_ and max_ hyperparameters, 173
PCA implementation, 214
Perceptron class, 259
Pipeline constructor, 66-68, 149
Randomized PCA, 218
Ridge Regression with, 129
SAMME, 195
SGDClassifier, 82, 87-88, 94
SGDRegressor, 119
sklearn.base.BaseEstimator, 64, 67, 84
sklearn.base.clone(), 83, 133
sklearn.base.TransformerMixin, 64, 67
sklearn.datasets.fetch_california_housing(),
236
sklearn.datasets.fetch_mldata(), 79
sklearn.datasets.load_iris(), 137, 148, 167,
190, 259
sklearn.datasets.load_sample_images(),
360-361
sklearn.datasets.make_moons(), 149, 178
sklearn.decomposition.IncrementalPCA,
217
sklearn.decomposition.KernelPCA,
218-219, 221
sklearn.decomposition.PCA, 214
sklearn.ensemble.AdaBoostClassifier, 195
sklearn.ensemble.BaggingClassifier, 186-189
sklearn.ensemble.GradientBoostingRegres‐
sor, 196, 198-199
sklearn.ensemble.RandomForestClassifier,
92, 95, 184


sor, 196, 198-199
sklearn.ensemble.RandomForestClassifier,
92, 95, 184
sklearn.ensemble.RandomForestRegressor,
70, 72-74, 189-190, 196
sklearn.ensemble.VotingClassifier, 184
sklearn.externals.joblib, 71
sklearn.linear_model.ElasticNet, 132
sklearn.linear_model.Lasso, 132
sklearn.linear_model.LinearRegression,
20-21, 62, 68, 110, 120, 122, 124-125
sklearn.linear_model.LogisticRegression,
137, 139, 141, 184, 219
sklearn.linear_model.Perceptron, 259
sklearn.linear_model.Ridge, 129
sklearn.linear_model.SGDClassifier, 82
sklearn.linear_model.SGDRegressor,
119-120, 129, 132-133
sklearn.manifold.LocallyLinearEmbedding,
221-222
sklearn.metrics.accuracy_score(), 184, 188,
264
sklearn.metrics.confusion_matrix(), 85, 96
sklearn.metrics.f1_score(), 87, 100
sklearn.metrics.mean_squared_error(),
68-69, 76, 124, 133, 198-199, 221
sklearn.metrics.precision_recall_curve(), 88
sklearn.metrics.precision_score(), 86, 90
sklearn.metrics.recall_score(), 86, 90
sklearn.metrics.roc_auc_score(), 92-93


sklearn.metrics.recall_score(), 86, 90
sklearn.metrics.roc_auc_score(), 92-93
sklearn.metrics.roc_curve(), 91-92
sklearn.model_selection.cross_val_pre‐
dict(), 84, 88, 92, 96, 100
sklearn.model_selection.cross_val_score(),
69-70, 83-84
sklearn.model_selection.GridSearchCV,
72-74, 77, 96, 179, 219
sklearn.model_selection.StratifiedKFold, 83
sklearn.model_selection.StratifiedShuffleS‐
plit, 52
sklearn.model_selection.train_test_split(),
50, 69, 124, 178, 198
sklearn.multiclass.OneVsOneClassifier, 95
sklearn.neighbors.KNeighborsClassifier,
100, 102
sklearn.neighbors.KNeighborsRegressor, 22
sklearn.pipeline.FeatureUnion, 66
sklearn.pipeline.Pipeline, 66, 125, 148-149,
219
sklearn.preprocessing.Imputer, 60, 66
sklearn.preprocessing.LabelBinarizer, 64, 66
sklearn.preprocessing.LabelEncoder, 62
sklearn.preprocessing.OneHotEncoder, 63
sklearn.preprocessing.PolynomialFeatures,
122-123, 125, 128, 149
sklearn.preprocessing.StandardScaler,
65-66, 96, 114, 128, 146, 148-150, 152,
237, 264


sklearn.preprocessing.StandardScaler,
65-66, 96, 114, 128, 146, 148-150, 152,
237, 264
sklearn.svm.LinearSVC, 147-149, 153-154,
156, 165
sklearn.svm.LinearSVR, 155-156
sklearn.svm.SVC, 148, 150, 152-154, 156,
165, 184
sklearn.svm.SVR, 77, 156
536
|
Index
Download from finelybook www.finelybook.com


sklearn.tree.DecisionTreeClassifier, 173,
179, 186-187, 189, 195
sklearn.tree.DecisionTreeRegressor, 69, 167,
175, 195-196
sklearn.tree.export_graphviz(), 168
StandardScaler, 114, 237, 264
SVM classification classes, 154
TF.Learn, 231
user guide, xvi
score(), 62
search space, 74, 270
second-order partial derivatives (Hessians), 300
self-organizing maps (SOMs), 521-523
semantic hashing, 434
semisupervised learning, 13
sensitivity, 85, 91
sentiment analysis, 379
separable_conv2d(), 376
sequences, 379
sequence_length, 387-388, 409
Shannon's information theory, 172
shortcut connections, 372
show(), 48
show_graph(), 245
shrinkage, 197
shuffle_batch(), 341
shuffle_batch_join(), 341
sigmoid function, 134
sigmoid_cross_entropy_with_logits(), 428
similarity function, 151-152
simulated annealing, 118
simulated environments, 442
(see also OpenAI Gym)
Singular Value Decomposition (SVD), 213
skewed datasets, 84
skip connections, 310, 372
slack variable, 158
smoothing terms, 283, 297, 299, 430


skip connections, 310, 372
slack variable, 158
smoothing terms, 283, 297, 299, 430
soft margin classification, 146-148
soft placements, 321
soft voting, 184
softmax function, 139, 263, 264
Softmax Regression, 139-142
source ops, 236, 322
spam filters, 3-6, 8
sparse autoencoders, 426-428
sparse matrix, 63
sparse models, 130, 300
sparse_softmax_cross_entropy_with_logits(),
268
sparsity loss, 426
specificity, 91
speech recognition, 6
spurious patterns, 516
stack(), 385
stacked autoencoders, 415-424
TensorFlow implementation, 416
training one-at-a-time, 418-420
tying weights, 417-418
unsupervised pretraining with, 422-424
visualizing the reconstructions, 420-421
stacked denoising autoencoders, 422, 424
stacked denoising encoders, 424
stacked generalization (see stacking)
stacking, 200-202
stale gradients, 348
standard correlation coefficient, 55
standard deviation, 37
standardization, 65
StandardScaler, 66, 237, 264
state-action values, 456
states tensor, 388
state_is_tuple, 398, 401


StandardScaler, 66, 237, 264
state-action values, 456
states tensor, 388
state_is_tuple, 398, 401
static unrolling through time, 385-386
static_rnn(), 385-386, 409
stationary point, 503-505
statistical mode, 185
statistical significance, 174
stemming, 103
step functions, 257
step(), 443
Stochastic Gradient Boosting, 199
Stochastic Gradient Descent (SGD), 117-119,
148, 260
training, 136
Stochastic Gradient Descent (SGD) classifier,
82, 129
stochastic neurons, 516
stochastic policy, 440
stratified sampling, 51-53, 83
stride, 357
string kernels, 153
string_input_producer(), 341
strong learners, 182
subderivatives, 164
subgradient vector, 131
subsample, 199, 363
Index
|
537
Download from finelybook www.finelybook.com


supervised learning, 8-9
Support Vector Machines (SVMs), 94, 145-166
decision function and predictions, 156-157
dual problem, 503-505
kernelized SVM, 161-164
linear classification, 145-148
mechanics of, 156-165
nonlinear classification, 149-154
online SVMs, 164-165
Quadratic Programming (QP) problems,
159-160
SVM regression, 154-165
the dual problem, 160
training objective, 157-159
support vectors, 146
svd(), 213
symbolic differentiation, 238, 508-509
synchronous updates, 348
T
t-Distributed Stochastic Neighbor Embedding
(t-SNE), 224
tail heavy, 48
target attributes, 48
target_weights, 409
tasks, 323
Temporal Difference (TD) Learning, 457-458
tensor processing units (TPUs), 315
TensorBoard, 231
TensorFlow, 229-252
about, xiv
autodiff, 238-239, 507-513
Batch Normalization with, 284-286
construction phase, 234
control dependencies, 323
convenience functions, 341
convolutional layers, 376
convolutional neural networks and, 360-362
data parallelism and, 351


convolutional layers, 376
convolutional neural networks and, 360-362
data parallelism and, 351
denoising autoencoders, 425-425
dropout with, 306
dynamic placer, 318
execution phase, 234
feeding data to the training algorithm,
239-241
Gradient Descent with, 237-239
graphs, managing, 234
initial graph creation and session run,
232-234
installation, 232
l1 and l2 regularization with, 303
learning schedules in, 302
Linear Regression with, 235-236
max pooling layer in, 364
max-norm regularization with, 307
model zoo, 291
modularity, 246-248
Momentum optimization in, 295
name scopes, 245
neural network policies, 446
NLP tutorials, 405, 408
node value lifecycle, 235
operations (ops), 235
optimizer, 239
overview, 229-231
parallel distributed computing (see parallel
distributed computing with TensorFlow)
Python API
construction, 265-269
execution, 269
using the neural network, 270
queues (see queues)
reusing pretrained layers, 287-288
RNNs in, 384-388
(see also recurrent neural networks


reusing pretrained layers, 287-288
RNNs in, 384-388
(see also recurrent neural networks
(RNNs))
saving and restoring models, 241-242
sharing variables, 248-251
simple placer, 318
sklearn.metrics.accuracy_score(), 286
sparse autoencoders with, 427
and stacked autoencoders, 416
TensorBoard, 242-245
tf.abs(), 303
tf.add(), 246, 303-304
tf.add_n(), 247-248, 250-251
tf.add_to_collection(), 308
tf.assign(), 237, 288, 307-308, 482
tf.bfloat16, 350
tf.bool, 284, 306
tf.cast(), 268, 391
tf.clip_by_norm(), 307-308
tf.clip_by_value(), 286
tf.concat(), 312, 369, 446, 450
tf.ConfigProto, 317, 320-321, 345, 487
tf.constant(), 235-237, 319-320, 323,
325-326
tf.constant_initializer(), 249-251
538
|
Index
Download from finelybook www.finelybook.com


tf.container(), 328-330, 351-352, 481
tf.contrib.framework.arg_scope(), 285, 416,
430
tf.contrib.layers.batch_norm(), 284-285
tf.contrib.layers.convolution2d(), 463
tf.contrib.layers.fully_connected(), 267
tf.contrib.layers.l1_regularizer(), 303, 308
tf.contrib.layers.l2_regularizer(), 303,
416-417
tf.contrib.layers.variance_scaling_initial‐
izer(), 278-279, 391, 416-417, 430, 446,
450, 463
tf.contrib.learn.DNNClassifier, 264
tf.contrib.learn.infer_real_valued_col‐
umns_from_input(), 264
tf.contrib.rnn.BasicLSTMCell, 401, 403
tf.contrib.rnn.BasicRNNCell, 385-387, 390,
392-393, 395, 397-399, 401
tf.contrib.rnn.DropoutWrapper, 399
tf.contrib.rnn.GRUCell, 405
tf.contrib.rnn.LSTMCell, 403
tf.contrib.rnn.MultiRNNCell, 397-399
tf.contrib.rnn.OutputProjectionWrapper,
392-394
tf.contrib.rnn.RNNCell, 398
tf.contrib.rnn.static_rnn(), 385-387,
409-410, 491-492
tf.contrib.slim module, 231, 377
tf.contrib.slim.nets module (nets), 377
tf.control_dependencies(), 323
tf.decode_csv(), 336, 340


tf.contrib.slim.nets module (nets), 377
tf.control_dependencies(), 323
tf.decode_csv(), 336, 340
tf.device(), 319-321, 326-327, 397-398
tf.exp(), 430-431
tf.FIFOQueue, 330, 332-333, 336, 340
tf.float32, 236, 482
tf.get_collection(), 288-289, 304, 308, 416,
463
tf.get_default_graph(), 234, 242
tf.get_default_session(), 233
tf.get_variable(), 249-251, 288, 303-308
tf.global_variables(), 308
tf.global_variables_initializer(), 233, 237
tf.gradients(), 238
tf.Graph, 232, 234, 242, 335, 343
tf.GraphKeys.REGULARIZATION_LOS‐
SES, 304, 416
tf.GraphKeys.TRAINABLE_VARIABLES,
288-289, 463
tf.group(), 464
tf.int32, 321-332, 337, 387, 390, 406, 466
tf.int64, 265
tf.InteractiveSession, 233
TF.Learn, 264
tf.log(), 427, 430, 446, 450
tf.matmul(), 236-237, 246, 265, 384, 417,
420, 425, 427-428
tf.matrix_inverse(), 236
tf.maximum(), 246, 248-251, 281
tf.multinomial(), 446, 450
tf.name_scope(), 245, 248-249, 265,
267-268, 419-420
tf.nn.conv2d(), 360-361
tf.nn.dynamic_rnn(), 386-387, 390, 392,


267-268, 419-420
tf.nn.conv2d(), 360-361
tf.nn.dynamic_rnn(), 386-387, 390, 392,
395, 397-399, 409-410, 491-492
tf.nn.elu(), 281, 416-417, 430, 446, 450
tf.nn.embedding_lookup(), 406
tf.nn.in_top_k(), 268, 391
tf.nn.max_pool(), 364-365
tf.nn.relu(), 265, 392-393, 395, 463
tf.nn.sigmoid_cross_entropy_with_logits(),
428, 431, 449-450
tf.nn.sparse_soft‐
max_cross_entropy_with_logits(),
267-268, 390
tf.one_hot(), 466
tf.PaddingFIFOQueue, 334
tf.placeholder(), 239-240, 482
tf.placeholder_with_default(), 425
tf.RandomShuffleQueue, 333, 337-338,
340-341
tf.random_normal(), 246, 384, 425, 430
tf.random_uniform(), 237, 241, 406, 482
tf.reduce_mean(), 237, 245, 267-268, 303,
390-391, 414, 416, 418, 420, 425, 427,
466
tf.reduce_sum(), 303, 427-428, 430-431,
465-466
tf.reset_default_graph(), 234
tf.reshape(), 395, 463
tf.RunOptions, 345
tf.Session, 233, 482
tf.shape(), 425, 430
tf.square(), 237, 245, 393, 414, 416, 418, 420,
425, 427, 430-431, 466
tf.stack(), 336, 340, 386
tf.string, 336, 340


425, 427, 430-431, 466
tf.stack(), 336, 340, 386
tf.string, 336, 340
tf.summary.FileWriter, 242-243
tf.summary.scalar(), 242
Index
|
539
Download from finelybook www.finelybook.com


tf.tanh(), 384
tf.TextLineReader, 336, 340
tf.to_float(), 449-450
tf.train.AdamOptimizer, 293, 299, 390, 393,
414, 416-417, 419, 427, 431, 449-450, 466
tf.train.ClusterSpec, 324
tf.train.Coordinator, 338-340
tf.train.exponential_decay(), 302
tf.train.GradientDescentOptimizer, 239,
268, 286, 293, 295
tf.train.MomentumOptimizer, 239, 295-296,
302, 311, 351, 485-486
tf.train.QueueRunner, 338-341
tf.train.replica_device_setter(), 327-328
tf.train.RMSPropOptimizer, 298
tf.train.Saver, 241-242, 268, 377, 399, 450,
466
tf.train.Server, 324
tf.train.start_queue_runners(), 341
tf.transpose(), 236-237, 386, 417
tf.truncated_normal(), 265
tf.unstack(), 385-387, 395, 492
tf.Variable, 232, 482
tf.variable_scope(), 249-251, 288, 307-308,
328, 391, 463
tf.zeros(), 265, 384, 417
truncated backpropagation through time,
400
visualizing graph and training curves,
242-245
TensorFlow Serving, 343
tensorflow.contrib, 267
test set, 29, 49-53, 81
testing and validating, 29-31
text attributes, 62-64


test set, 29, 49-53, 81
testing and validating, 29-31
text attributes, 62-64
TextLineReader, 336
TF-slim, 231
TF.Learn, 231, 264
thermal equilibrium, 518
thread pools (inter-op/intra-op, in TensorFlow,
322
threshold variable, 248-251
Tikhonov regularization, 127
time series data, 379
toarray(), 63
tolerance hyperparameter, 154
trainable, 288
training data, 4
insufficient quantities, 22
irrelevant features, 25
loading, 335-342
nonrepresentative, 24
overfitting, 26-28
poor quality, 25
underfitting, 28
training instance, 4
training models, 20, 105-143
learning curves in, 123-127
Linear Regression, 105, 106-121
Logistic Regression, 134-142
overview, 105-106
Polynomial Regression, 106, 121-123
training objectives, 157-159
training set, 4, 29, 53, 60, 68-69
cost function of, 135-136
shuffling, 81
transfer learning, 286-293
(see also pretrained layers reuse)
transform(), 61, 66
transformation pipelines, 66-68
transformers, 61
transformers, custom, 64-65
transpose(), 385


transformation pipelines, 66-68
transformers, 61
transformers, custom, 64-65
transpose(), 385
true negative rate (TNR), 91
true positive rate (TPR), 85, 91
truncated backpropagation through time, 400
tuples, 332
tying weights, 417
U
underfitting, 28, 68, 152
univariate regression, 37
unstack(), 385
unsupervised learning, 10-12
anomaly detection, 12
association rule learning, 10, 12
clustering, 10
dimensionality reduction algorithm, 12
visualization algorithms, 11
unsupervised pretraining, 291-292, 422-424
upsampling, 376
utility function, 20
V
validation set, 30
Value Iteration, 455
value_counts(), 46
vanishing gradients, 276
540
|
Index
Download from finelybook www.finelybook.com


(see also gradients, vanishing and explod‐
ing)
variables, sharing, 248-251
variable_scope(), 249-250
variance
bias/variance tradeoff, 126
variance preservation, 211-212
variance_scaling_initializer(), 278
variational autoencoders, 428-432
VGGNet, 375
visual cortex, 354
visualization, 242-245
visualization algorithms, 11-12
voice recognition, 353
voting classifiers, 181-184
W
warmup phase, 349
weak learners, 182
weight-tying, 417
weights, 267, 288
freezing, 289
while_loop(), 387
white box models, 170
worker, 324
worker service, 325
worker_device, 327
workspace directory, 40-43
X
Xavier initialization, 276-279
Y
YouTube, 253
Z
zero padding, 356, 361
Index
|
541
Download from finelybook www.finelybook.com


About the Author
Aurélien Géron is a Machine Learning consultant. A former Googler, he led the You‐
Tube video classification team from 2013 to 2016. He was also a founder and CTO of
Wifirst from 2002 to 2012, a leading Wireless ISP in France; and a founder and CTO
of Polyconseil in 2001, the firm that now manages the electric car sharing service
Autolib’.
Before this he worked as an engineer in a variety of domains: finance (JP Morgan and
Société Générale), defense (Canada’s DOD), and healthcare (blood transfusion). He
published a few technical books (on C++, WiFi, and internet architectures), and was
a Computer Science lecturer in a French engineering school.
A few fun facts: he taught his three children to count in binary with their fingers (up
to 1023), he studied microbiology and evolutionary genetics before going into soft‐
ware engineering, and his parachute didn’t open on the second jump.
Colophon
The animal on the cover of Hands-On Machine Learning with Scikit-Learn and Ten‐


Colophon
The animal on the cover of Hands-On Machine Learning with Scikit-Learn and Ten‐
sorFlow is the far eastern fire salamander (Salamandra infraimmaculata), an amphib‐
ian found in the Middle East. They have black skin featuring large yellow spots on
their back and head. These spots are a warning coloration meant to keep predators at
bay. Full-grown salamanders can be over a foot in length.
Far eastern fire salamanders live in subtropical shrubland and forests near rivers or
other freshwater bodies. They spend most of their life on land, but lay their eggs in
the water. They subsist mostly on a diet of insects, worms, and small crustaceans, but
occasionally eat other salamanders. Males of the species have been known to live up
to 23 years, while females can live up to 21 years.
Although not yet endangered, the far eastern fire salamander population is in decline.
Primary threats include damming of rivers (which disrupts the salamander’s breed‐


Primary threats include damming of rivers (which disrupts the salamander’s breed‐
ing) and pollution. They are also threatened by the recent introduction of predatory
fish, such as the mosquitofish. These fish were intended to control the mosquito pop‐
ulation, but they also feed on young salamanders.
Many of the animals on O’Reilly covers are endangered; all of them are important to
the world. To learn more about how you can help, go to animals.oreilly.com.
The cover image is from Wood’s Illustrated Natural History. The cover fonts are URW
Typewriter and Guardian Sans. The text font is Adobe Minion Pro; the heading font
is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono.
Download from finelybook www.finelybook.com