Caching the Frozen Layers
Since the frozen layers won’t change, it is possible to cache the output of the topmost
frozen layer for each training instance. Since training goes through the whole dataset
many times, this will give you a huge speed boost as you will only need to go through
the frozen layers once per training instance (instead of once per epoch). For example,
you could first run the whole training set through the lower layers (assuming you
have enough RAM):
hidden2_outputs = sess.run(hidden2, feed_dict={X: X_train})
Then during training, instead of building batches of training instances, you would
build batches of outputs from hidden layer 2 and feed them to the training operation:
import numpy as np
n_epochs = 100
n_batches = 500
for epoch in range(n_epochs):
    shuffled_idx = rnd.permutation(len(hidden2_outputs))
    hidden2_batches = np.array_split(hidden2_outputs[shuffled_idx], n_batches)
    y_batches = np.array_split(y_train[shuffled_idx], n_batches)


y_batches = np.array_split(y_train[shuffled_idx], n_batches)
    for hidden2_batch, y_batch in zip(hidden2_batches, y_batches):
        sess.run(training_op, feed_dict={hidden2: hidden2_batch, y: y_batch})
The last line runs the training operation defined earlier (which freezes layers 1 and 2),
and feeds it a batch of outputs from the second hidden layer (as well as the targets for
that batch). Since we give TensorFlow the output of hidden layer 2, it does not try to
evaluate it (or any node it depends on).
Tweaking, Dropping, or Replacing the Upper Layers
The output layer of the original model should usually be replaced since it is most
likely not useful at all for the new task, and it may not even have the right number of
outputs for the new task.
Similarly, the upper hidden layers of the original model are less likely to be as useful
as the lower layers, since the high-level features that are most useful for the new task

Hands on Machine Learning with Scikit Learn and TensorFlow.pdf


as the lower layers, since the high-level features that are most useful for the new task
may differ significantly from the ones that were most useful for the original task. You
want to find the right number of layers to reuse.
Try freezing all the copied layers first, then train your model and see how it performs.
Then try unfreezing one or two of the top hidden layers to let backpropagation tweak
them and see if performance improves. The more training data you have, the more
layers you can unfreeze.
If you still cannot get good performance, and you have little training data, try drop‐
ping the top hidden layer(s) and freeze all remaining hidden layers again. You can
290
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


iterate until you find the right number of layers to reuse. If you have plenty of train‐
ing data, you may try replacing the top hidden layers instead of dropping them, and
even add more hidden layers.
Model Zoos
Where can you find a neural network trained for a task similar to the one you want to
tackle? The first place to look is obviously in your own catalog of models. This is one
good reason to save all your models and organize them so you can retrieve them later
easily. Another option is to search in a model zoo. Many people train Machine Learn‐
ing models for various tasks and kindly release their pretrained models to the public.
TensorFlow has its own model zoo available at https://github.com/tensorflow/models.
In particular, it contains most of the state-of-the-art image classification nets such as
VGG, Inception, and ResNet (see Chapter 13, and check out the models/slim direc‐
tory), including the code, the pretrained models, and tools to download popular
image datasets.


tory), including the code, the pretrained models, and tools to download popular
image datasets.
Another popular model zoo is Caffe’s Model Zoo. It also contains many computer
vision models (e.g., LeNet, AlexNet, ZFNet, GoogLeNet, VGGNet, inception) trained
on various datasets (e.g., ImageNet, Places Database, CIFAR10, etc.). Saumitro Das‐
gupta wrote a converter, which is available at https://github.com/ethereon/caffe-
tensorflow.
Unsupervised Pretraining
Suppose you want to tackle a complex task for which you don’t have much labeled
training data, but unfortunately you cannot find a model trained on a similar task.
Don’t lose all hope! First, you should of course try to gather more labeled training
data, but if this is too hard or too expensive, you may still be able to perform unsuper‐
vised pretraining (see Figure 11-5). That is, if you have plenty of unlabeled training
data, you can try to train the layers one by one, starting with the lowest layer and then


data, you can try to train the layers one by one, starting with the lowest layer and then
going up, using an unsupervised feature detector algorithm such as Restricted Boltz‐
mann Machines (RBMs; see Appendix E) or autoencoders (see Chapter 15). Each
layer is trained on the output of the previously trained layers (all layers except the one
being trained are frozen). Once all layers have been trained this way, you can fine-
tune the network using supervised learning (i.e., with backpropagation).
This is a rather long and tedious process, but it often works well; in fact, it is this
technique that Geoffrey Hinton and his team used in 2006 and which led to the
revival of neural networks and the success of Deep Learning. Until 2010, unsuper‐
vised pretraining (typically using RBMs) was the norm for deep nets, and it was only
after the vanishing gradients problem was alleviated that it became much more com‐
mon to train DNNs purely using backpropagation. However, unsupervised pretrain‐


mon to train DNNs purely using backpropagation. However, unsupervised pretrain‐
ing (today typically using autoencoders rather than RBMs) is still a good option when
Reusing Pretrained Layers
|
291
Download from finelybook www.finelybook.com


9 Another option is to come up with a supervised task for which you can easily gather a lot of labeled training
data, then use transfer learning, as explained earlier. For example, if you want to train a model to identify your
friends in pictures, you could download millions of faces on the internet and train a classifier to detect
whether two faces are identical or not, then use this classifier to compare a new picture with each picture of
your friends.
you have a complex task to solve, no similar model you can reuse, and little labeled
training data but plenty of unlabeled training data.9
Figure 11-5. Unsupervised pretraining
Pretraining on an Auxiliary Task
One last option is to train a first neural network on an auxiliary task for which you
can easily obtain or generate labeled training data, then reuse the lower layers of that
network for your actual task. The first neural network’s lower layers will learn feature


network for your actual task. The first neural network’s lower layers will learn feature
detectors that will likely be reusable by the second neural network.
For example, if you want to build a system to recognize faces, you may only have a
few pictures of each individual—clearly not enough to train a good classifier. Gather‐
ing hundreds of pictures of each person would not be practical. However, you could
gather a lot of pictures of random people on the internet and train a first neural net‐
work to detect whether or not two different pictures feature the same person. Such a
292
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


10 At least for now: research is moving fast, especially in the field of optimization. Be sure to take a look at the
latest and greatest optimizers every time a new version of TensorFlow is released.
network would learn good feature detectors for faces, so reusing its lower layers
would allow you to train a good face classifier using little training data.
It is often rather cheap to gather unlabeled training examples, but quite expensive to
label them. In this situation, a common technique is to label all your training exam‐
ples as “good,” then generate many new training instances by corrupting the good
ones, and label these corrupted instances as “bad.” Then you can train a first neural
network to classify instances as good or bad. For example, you could download mil‐
lions of sentences, label them as “good,” then randomly change a word in each sen‐
tence and label the resulting sentences as “bad.” If a neural network can tell that “The


tence and label the resulting sentences as “bad.” If a neural network can tell that “The
dog sleeps” is a good sentence but “The dog they” is bad, it probably knows quite a lot
about language. Reusing its lower layers will likely help in many language processing
tasks.
Another approach is to train a first network to output a score for each training
instance, and use a cost function that ensures that a good instance’s score is greater
than a bad instance’s score by at least some margin. This is called max margin learn‐
ing.
Faster Optimizers
Training a very large deep neural network can be painfully slow. So far we have seen
four ways to speed up training (and reach a better solution): applying a good initiali‐
zation strategy for the connection weights, using a good activation function, using
Batch Normalization, and reusing parts of a pretrained network. Another huge speed
boost comes from using a faster optimizer than the regular Gradient Descent opti‐


boost comes from using a faster optimizer than the regular Gradient Descent opti‐
mizer. In this section we will present the most popular ones: Momentum optimiza‐
tion, Nesterov Accelerated Gradient, AdaGrad, RMSProp, and finally Adam
optimization.
Spoiler alert: the conclusion of this section is that you should almost always use
Adam optimization,10 so if you don’t care about how it works, simply replace your
GradientDescentOptimizer with an AdamOptimizer and skip to the next section!
With just this small change, training will typically be several times faster. However,
Adam optimization does have three hyperparameters that you can tune (plus the
learning rate); the default values usually work fine, but if you ever need to tweak them
it may be helpful to know what they do. Adam optimization combines several ideas
from other optimization algorithms, so it is useful to look at these algorithms first.
Faster Optimizers
|
293
Download from finelybook www.finelybook.com


11 “Some methods of speeding up the convergence of iteration methods,” B. Polyak (1964).
Momentum optimization
Imagine a bowling ball rolling down a gentle slope on a smooth surface: it will start
out slowly, but it will quickly pick up momentum until it eventually reaches terminal
velocity (if there is some friction or air resistance). This is the very simple idea behind
Momentum optimization, proposed by Boris Polyak in 1964.11 In contrast, regular
Gradient Descent will simply take small regular steps down the slope, so it will take
much more time to reach the bottom.
Recall that Gradient Descent simply updates the weights θ by directly subtracting the
gradient of the cost function J(θ) with regards to the weights (∇θJ(θ)) multiplied by
the learning rate η. The equation is: θ ← θ – η∇θJ(θ). It does not care about what the
earlier gradients were. If the local gradient is tiny, it goes very slowly.
Momentum optimization cares a great deal about what previous gradients were: at


Momentum optimization cares a great deal about what previous gradients were: at
each iteration, it adds the local gradient to the momentum vector m (multiplied by the
learning rate η), and it updates the weights by simply subtracting this momentum
vector (see Equation 11-4). In other words, the gradient is used as an acceleration, not
as a speed. To simulate some sort of friction mechanism and prevent the momentum
from growing too large, the algorithm introduces a new hyperparameter β, simply
called the momentum, which must be set between 0 (high friction) and 1 (no friction).
A typical momentum value is 0.9.
Equation 11-4. Momentum algorithm
1 .
�
β�+ η∇θJ θ
2 .
θ
θ −�
You can easily verify that if the gradient remains constant, the terminal velocity (i.e.,
the maximum size of the weight updates) is equal to that gradient multiplied by the
learning rate η multiplied by
1
1 −β. For example, if β = 0.9, then the terminal velocity


learning rate η multiplied by
1
1 −β. For example, if β = 0.9, then the terminal velocity
is equal to 10 times the gradient times the learning rate, so Momentum optimization
ends up going 10 times faster than Gradient Descent! This allows Momentum opti‐
mization to escape from plateaus much faster than Gradient Descent. In particular,
we saw in Chapter 4 that when the inputs have very different scales the cost function
will look like an elongated bowl (see Figure 4-7). Gradient Descent goes down the
steep slope quite fast, but then it takes a very long time to go down the valley. In con‐
trast, Momentum optimization will roll down the bottom of the valley faster and
faster until it reaches the bottom (the optimum). In deep neural networks that don’t
use Batch Normalization, the upper layers will often end up having inputs with very
294
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


12 “A Method for Unconstrained Convex Minimization Problem with the Rate of Convergence O(1/k2),” Yurii
Nesterov (1983).
different scales, so using Momentum optimization helps a lot. It can also help roll
past local optima.
Due to the momentum, the optimizer may overshoot a bit, then
come back, overshoot again, and oscillate like this many times
before stabilizing at the minimum. This is one of the reasons why it
is good to have a bit of friction in the system: it gets rid of these
oscillations and thus speeds up convergence.
Implementing Momentum optimization in TensorFlow is a no-brainer: just replace
the GradientDescentOptimizer with the MomentumOptimizer, then lie back and
profit!
optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate,
                                       momentum=0.9)
The one drawback of Momentum optimization is that it adds yet another hyperpara‐
meter to tune. However, the momentum value of 0.9 usually works well in practice


meter to tune. However, the momentum value of 0.9 usually works well in practice
and almost always goes faster than Gradient Descent.
Nesterov Accelerated Gradient
One small variant to Momentum optimization, proposed by Yurii Nesterov in 1983,12
is almost always faster than vanilla Momentum optimization. The idea of Nesterov
Momentum optimization, or Nesterov Accelerated Gradient (NAG), is to measure the
gradient of the cost function not at the local position but slightly ahead in the direc‐
tion of the momentum (see Equation 11-5). The only difference from vanilla
Momentum optimization is that the gradient is measured at θ + βm rather than at θ.
Equation 11-5. Nesterov Accelerated Gradient algorithm
1 .
�
β�+ η∇θJ θ + β�
2 .
θ
θ −�
This small tweak works because in general the momentum vector will be pointing in
the right direction (i.e., toward the optimum), so it will be slightly more accurate to
use the gradient measured a bit farther in that direction rather than using the gradi‐


use the gradient measured a bit farther in that direction rather than using the gradi‐
ent at the original position, as you can see in Figure 11-6 (where ∇1 represents the
gradient of the cost function measured at the starting point θ, and ∇2 represents the
gradient at the point located at θ + βm). As you can see, the Nesterov update ends up
Faster Optimizers
|
295
Download from finelybook www.finelybook.com


slightly closer to the optimum. After a while, these small improvements add up and
NAG ends up being significantly faster than regular Momentum optimization. More‐
over, note that when the momentum pushes the weights across a valley, ∇1 continues
to push further across the valley, while ∇2 pushes back toward the bottom of the val‐
ley. This helps reduce oscillations and thus converges faster.
Figure 11-6. Regular versus Nesterov Momentum optimization
NAG will almost always speed up training compared to regular Momentum optimi‐
zation. To use it, simply set use_nesterov=True when creating the MomentumOptim
izer:
optimizer = tf.train.MomentumOptimizer(learning_rate=learning_rate,
                                       momentum=0.9, use_nesterov=True)
AdaGrad
Consider the elongated bowl problem again: Gradient Descent starts by quickly going
down the steepest slope, then slowly goes down the bottom of the valley. It would be


down the steepest slope, then slowly goes down the bottom of the valley. It would be
nice if the algorithm could detect this early on and correct its direction to point a bit
more toward the global optimum.
296
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


13 “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization,” J. Duchi et al. (2011).
The AdaGrad algorithm13 achieves this by scaling down the gradient vector along the
steepest dimensions (see Equation 11-6):
Equation 11-6. AdaGrad algorithm
1 .
�
�+ ∇θJ θ ⊗∇θJ θ
2 .
θ
θ −η ∇θJ θ ⊘
�+ �
The first step accumulates the square of the gradients into the vector s (the ⊗ symbol
represents the element-wise multiplication). This vectorized form is equivalent to
computing si ← si + (∂ / ∂ θi J(θ))2 for each element si of the vector s; in other words,
each si accumulates the squares of the partial derivative of the cost function with
regards to parameter θi. If the cost function is steep along the ith dimension, then si
will get larger and larger at each iteration.
The second step is almost identical to Gradient Descent, but with one big difference:
the gradient vector is scaled down by a factor of �+ � (the ⊘ symbol represents the


the gradient vector is scaled down by a factor of �+ � (the ⊘ symbol represents the
element-wise division, and ϵ is a smoothing term to avoid division by zero, typically
set
to
10–10).
This
vectorized
form
is
equivalent
to
computing
θi
θi −η ∂/∂θi J θ / si + � for all parameters θi (simultaneously).
In short, this algorithm decays the learning rate, but it does so faster for steep dimen‐
sions than for dimensions with gentler slopes. This is called an adaptive learning rate.
It helps point the resulting updates more directly toward the global optimum (see
Figure 11-7). One additional benefit is that it requires much less tuning of the learn‐
ing rate hyperparameter η.
Figure 11-7. AdaGrad versus Gradient Descent
Faster Optimizers
|
297
Download from finelybook www.finelybook.com


14 This algorithm was created by Tijmen Tieleman and Geoffrey Hinton in 2012, and presented by Geoffrey
Hinton in his Coursera class on neural networks (slides: http://goo.gl/RsQeis; video: https://goo.gl/XUbIyJ).
Amusingly, since the authors have not written a paper to describe it, researchers often cite “slide 29 in lecture
6” in their papers.
15 “Adam: A Method for Stochastic Optimization,” D. Kingma, J. Ba (2015).
AdaGrad often performs well for simple quadratic problems, but unfortunately it
often stops too early when training neural networks. The learning rate gets scaled
down so much that the algorithm ends up stopping entirely before reaching the
global optimum. So even though TensorFlow has an AdagradOptimizer, you should
not use it to train deep neural networks (it may be efficient for simpler tasks such as
Linear Regression, though).
RMSProp
Although AdaGrad slows down a bit too fast and ends up never converging to the


RMSProp
Although AdaGrad slows down a bit too fast and ends up never converging to the
global optimum, the RMSProp algorithm14 fixes this by accumulating only the gradi‐
ents from the most recent iterations (as opposed to all the gradients since the begin‐
ning of training). It does so by using exponential decay in the first step (see Equation
11-7).
Equation 11-7. RMSProp algorithm
1 .
�
β�+ 1 −β ∇θJ θ ⊗∇θJ θ
2 .
θ
θ −η ∇θJ θ ⊘
�+ �
The decay rate β is typically set to 0.9. Yes, it is once again a new hyperparameter, but
this default value often works well, so you may not need to tune it at all.
As you might expect, TensorFlow has an RMSPropOptimizer class:
optimizer = tf.train.RMSPropOptimizer(learning_rate=learning_rate,
                                      momentum=0.9, decay=0.9, epsilon=1e-10)
Except on very simple problems, this optimizer almost always performs much better
than AdaGrad. It also generally performs better than Momentum optimization and


than AdaGrad. It also generally performs better than Momentum optimization and
Nesterov Accelerated Gradients. In fact, it was the preferred optimization algorithm
of many researchers until Adam optimization came around.
Adam Optimization
Adam,15 which stands for adaptive moment estimation, combines the ideas of Momen‐
tum optimization and RMSProp: just like Momentum optimization it keeps track of
an exponentially decaying average of past gradients, and just like RMSProp it keeps
298
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


16 These are estimations of the mean and (uncentered) variance of the gradients. The mean is often called the
first moment, while the variance is often called the second moment, hence the name of the algorithm.
track of an exponentially decaying average of past squared gradients (see Equation
11-8).16
Equation 11-8. Adam algorithm
1 .
�
β1�+ 1 −β1 ∇θJ θ
2 .
�
β2�+ 1 −β2 ∇θJ θ ⊗∇θJ θ
3 .
�
�
1 −β1
T
4 .
�
�
1 −β2
T
5 .
θ
θ −η �⊘
�+ �
• T represents the iteration number (starting at 1).
If you just look at steps 1, 2, and 5, you will notice Adam’s close similarity to both
Momentum optimization and RMSProp. The only difference is that step 1 computes
an exponentially decaying average rather than an exponentially decaying sum, but
these are actually equivalent except for a constant factor (the decaying average is just
1 – β1 times the decaying sum). Steps 3 and 4 are somewhat of a technical detail: since


1 – β1 times the decaying sum). Steps 3 and 4 are somewhat of a technical detail: since
m and s are initialized at 0, they will be biased toward 0 at the beginning of training,
so these two steps will help boost m and s at the beginning of training.
The momentum decay hyperparameter β1 is typically initialized to 0.9, while the scal‐
ing decay hyperparameter β2 is often initialized to 0.999. As earlier, the smoothing
term ϵ is usually initialized to a tiny number such as 10–8. These are the default values
for TensorFlow’s AdamOptimizer class, so you can simply use:
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
In fact, since Adam is an adaptive learning rate algorithm (like AdaGrad and
RMSProp), it requires less tuning of the learning rate hyperparameter η. You can
often use the default value η = 0.001, making Adam even easier to use than Gradient
Descent.
Faster Optimizers
|
299
Download from finelybook www.finelybook.com


17 “Primal-Dual Subgradient Methods for Convex Problems,” Yurii Nesterov (2005).
18 “Ad Click Prediction: a View from the Trenches,” H. McMahan et al. (2013).
All the optimization techniques discussed so far only rely on the
first-order partial derivatives (Jacobians). The optimization litera‐
ture contains amazing algorithms based on the second-order partial
derivatives (the Hessians). Unfortunately, these algorithms are very
hard to apply to deep neural networks because there are n2 Hessi‐
ans per output (where n is the number of parameters), as opposed
to just n Jacobians per output. Since DNNs typically have tens of
thousands of parameters, the second-order optimization algo‐
rithms often don’t even fit in memory, and even when they do,
computing the Hessians is just too slow.
Training Sparse Models
All the optimization algorithms just presented produce dense models, meaning that
most parameters will be nonzero. If you need a blazingly fast model at runtime, or if


most parameters will be nonzero. If you need a blazingly fast model at runtime, or if
you need it to take up less memory, you may prefer to end up with a sparse model
instead.
One trivial way to achieve this is to train the model as usual, then get rid of the tiny
weights (set them to 0).
Another option is to apply strong ℓ1 regularization during training, as it pushes the
optimizer to zero out as many weights as it can (as discussed in Chapter 4 about Lasso
Regression).
However, in some cases these techniques may remain insufficient. One last option is
to apply Dual Averaging, often called Follow The Regularized Leader (FTRL), a techni‐
que proposed by Yurii Nesterov.17 When used with ℓ1 regularization, this technique
often leads to very sparse models. TensorFlow implements a variant of FTRL called
FTRL-Proximal18 in the FTRLOptimizer class.
Learning Rate Scheduling
Finding a good learning rate can be tricky. If you set it way too high, training may


Finding a good learning rate can be tricky. If you set it way too high, training may
actually diverge (as we discussed in Chapter 4). If you set it too low, training will
eventually converge to the optimum, but it will take a very long time. If you set it
slightly too high, it will make progress very quickly at first, but it will end up dancing
around the optimum, never settling down (unless you use an adaptive learning rate
optimization algorithm such as AdaGrad, RMSProp, or Adam, but even then it may
take time to settle). If you have a limited computing budget, you may have to inter‐
300
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


rupt training before it has converged properly, yielding a suboptimal solution (see
Figure 11-8).
Figure 11-8. Learning curves for various learning rates η
You may be able to find a fairly good learning rate by training your network several
times during just a few epochs using various learning rates and comparing the learn‐
ing curves. The ideal learning rate will learn quickly and converge to good solution.
However, you can do better than a constant learning rate: if you start with a high
learning rate and then reduce it once it stops making fast progress, you can reach a
good solution faster than with the optimal constant learning rate. There are many dif‐
ferent strategies to reduce the learning rate during training. These strategies are called
learning schedules (we briefly introduced this concept in Chapter 4), the most com‐
mon of which are:
Predetermined piecewise constant learning rate
For example, set the learning rate to η0 = 0.1 at first, then to η1 = 0.001 after 50


For example, set the learning rate to η0 = 0.1 at first, then to η1 = 0.001 after 50
epochs. Although this solution can work very well, it often requires fiddling
around to figure out the right learning rates and when to use them.
Performance scheduling
Measure the validation error every N steps (just like for early stopping) and
reduce the learning rate by a factor of λ when the error stops dropping.
Exponential scheduling
Set the learning rate to a function of the iteration number t: η(t) = η0 10–t/r. This
works great, but it requires tuning η0 and r. The learning rate will drop by a fac‐
tor of 10 every r steps.
Power scheduling
Set the learning rate to η(t) = η0 (1 + t/r)–c. The hyperparameter c is typically set
to 1. This is similar to exponential scheduling, but the learning rate drops much
more slowly.
Faster Optimizers
|
301
Download from finelybook www.finelybook.com


19 “An Empirical Study of Learning Rates in Deep Neural Networks for Speech Recognition,” A. Senior et al.
(2013).
A 2013 paper19 by Andrew Senior et al. compared the performance of some of the
most popular learning schedules when training deep neural networks for speech rec‐
ognition using Momentum optimization. The authors concluded that, in this setting,
both performance scheduling and exponential scheduling performed well, but they
favored exponential scheduling because it is simpler to implement, is easy to tune,
and converged slightly faster to the optimal solution.
Implementing a learning schedule with TensorFlow is fairly straightforward:
initial_learning_rate = 0.1
decay_steps = 10000
decay_rate = 1/10
global_step = tf.Variable(0, trainable=False)
learning_rate = tf.train.exponential_decay(initial_learning_rate, global_step,
                                           decay_steps, decay_rate)
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum=0.9)


optimizer = tf.train.MomentumOptimizer(learning_rate, momentum=0.9)
training_op = optimizer.minimize(loss, global_step=global_step)
After setting the hyperparameter values, we create a nontrainable variable
global_step (initialized to 0) to keep track of the current training iteration number.
Then we define an exponentially decaying learning rate (with η0 = 0.1 and r = 10,000)
using TensorFlow’s exponential_decay() function. Next, we create an optimizer (in
this example, a MomentumOptimizer) using this decaying learning rate. Finally, we cre‐
ate the training operation by calling the optimizer’s minimize() method; since we
pass it the global_step variable, it will kindly take care of incrementing it. That’s it!
Since AdaGrad, RMSProp, and Adam optimization automatically reduce the learning
rate during training, it is not necessary to add an extra learning schedule. For other
optimization algorithms, using exponential decay or performance scheduling can


optimization algorithms, using exponential decay or performance scheduling can
considerably speed up convergence.
Avoiding Overfitting Through Regularization
With four parameters I can fit an elephant and with five I can make him wiggle his
trunk.
—John von Neumann, cited by Enrico Fermi in Nature 427
Deep neural networks typically have tens of thousands of parameters, sometimes
even millions. With so many parameters, the network has an incredible amount of
freedom and can fit a huge variety of complex datasets. But this great flexibility also
means that it is prone to overfitting the training set.
302
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


With millions of parameters you can fit the whole zoo. In this section we will present
some of the most popular regularization techniques for neural networks, and how to
implement them with TensorFlow: early stopping, ℓ1 and ℓ2 regularization, dropout,
max-norm regularization, and data augmentation.
Early Stopping
To avoid overfitting the training set, a great solution is early stopping (introduced in
Chapter 4): just interrupt training when its performance on the validation set starts
dropping.
One way to implement this with TensorFlow is to evaluate the model on a validation
set at regular intervals (e.g., every 50 steps), and save a “winner” snapshot if it outper‐
forms previous “winner” snapshots. Count the number of steps since the last “win‐
ner” snapshot was saved, and interrupt training when this number reaches some limit
(e.g., 2,000 steps). Then restore the last “winner” snapshot.
Although early stopping works very well in practice, you can usually get much higher


Although early stopping works very well in practice, you can usually get much higher
performance out of your network by combining it with other regularization techni‐
ques.
ℓ1 and ℓ2 Regularization
Just like you did in Chapter 4 for simple linear models, you can use ℓ1 and ℓ2 regulari‐
zation to constrain a neural network’s connection weights (but typically not its bia‐
ses).
One way to do this using TensorFlow is to simply add the appropriate regularization
terms to your cost function. For example, assuming you have just one hidden layer
with weights weights1 and one output layer with weights weights2, then you can
apply ℓ1 regularization like this:
[...] # construct the neural network
base_loss = tf.reduce_mean(xentropy, name="avg_xentropy")
reg_losses = tf.reduce_sum(tf.abs(weights1)) + tf.reduce_sum(tf.abs(weights2))
loss = tf.add(base_loss, scale * reg_losses, name="loss")
However, if there are many layers, this approach is not very convenient. Fortunately,


However, if there are many layers, this approach is not very convenient. Fortunately,
TensorFlow provides a better option. Many functions that create variables (such as
get_variable() or fully_connected()) accept a *_regularizer argument for each
created variable (e.g., weights_regularizer). You can pass any function that takes
weights as an argument and returns the corresponding regularization loss. The
l1_regularizer(), l2_regularizer(), and l1_l2_regularizer() functions return
such functions. The following code puts all this together:
with arg_scope(
        [fully_connected],
Avoiding Overfitting Through Regularization
|
303
Download from finelybook www.finelybook.com


20 “Improving neural networks by preventing co-adaptation of feature detectors,” G. Hinton et al. (2012).
21 “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” N. Srivastava et al. (2014).
        weights_regularizer=tf.contrib.layers.l1_regularizer(scale=0.01)):
    hidden1 = fully_connected(X, n_hidden1, scope="hidden1")
    hidden2 = fully_connected(hidden1, n_hidden2, scope="hidden2")
    logits = fully_connected(hidden2, n_outputs, activation_fn=None,scope="out")
This code creates a neural network with two hidden layers and one output layer, and
it also creates nodes in the graph to compute the ℓ1 regularization loss corresponding
to each layer’s weights. TensorFlow automatically adds these nodes to a special collec‐
tion containing all the regularization losses. You just need to add these regularization
losses to your overall loss, like this:
reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)


reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
loss = tf.add_n([base_loss] + reg_losses, name="loss")
Don’t forget to add the regularization losses to your overall loss, or
else they will simply be ignored.
Dropout
The most popular regularization technique for deep neural networks is arguably
dropout. It was proposed20 by G. E. Hinton in 2012 and further detailed in a paper21 by
Nitish Srivastava et al., and it has proven to be highly successful: even the state-of-
the-art neural networks got a 1–2% accuracy boost simply by adding dropout. This
may not sound like a lot, but when a model already has 95% accuracy, getting a 2%
accuracy boost means dropping the error rate by almost 40% (going from 5% error to
roughly 3%).
It is a fairly simple algorithm: at every training step, every neuron (including the
input neurons but excluding the output neurons) has a probability p of being tem‐


input neurons but excluding the output neurons) has a probability p of being tem‐
porarily “dropped out,” meaning it will be entirely ignored during this training step,
but it may be active during the next step (see Figure 11-9). The hyperparameter p is
called the dropout rate, and it is typically set to 50%. After training, neurons don’t get
dropped anymore. And that’s all (except for a technical detail we will discuss momen‐
tarily).
304
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


Figure 11-9. Dropout regularization
It is quite surprising at first that this rather brutal technique works at all. Would a
company perform better if its employees were told to toss a coin every morning to
decide whether or not to go to work? Well, who knows; perhaps it would! The com‐
pany would obviously be forced to adapt its organization; it could not rely on any sin‐
gle person to fill in the coffee machine or perform any other critical tasks, so this
expertise would have to be spread across several people. Employees would have to
learn to cooperate with many of their coworkers, not just a handful of them. The
company would become much more resilient. If one person quit, it wouldn’t make
much of a difference. It’s unclear whether this idea would actually work for compa‐
nies, but it certainly does for neural networks. Neurons trained with dropout cannot
co-adapt with their neighboring neurons; they have to be as useful as possible on


co-adapt with their neighboring neurons; they have to be as useful as possible on
their own. They also cannot rely excessively on just a few input neurons; they must
pay attention to each of their input neurons. They end up being less sensitive to slight
changes in the inputs. In the end you get a more robust network that generalizes bet‐
ter.
Another way to understand the power of dropout is to realize that a unique neural
network is generated at each training step. Since each neuron can be either present or
absent, there is a total of 2N possible networks (where N is the total number of drop‐
pable neurons). This is such a huge number that it is virtually impossible for the same
neural network to be sampled twice. Once you have run a 10,000 training steps, you
have essentially trained 10,000 different neural networks (each with just one training
instance). These neural networks are obviously not independent since they share


instance). These neural networks are obviously not independent since they share
many of their weights, but they are nevertheless all different. The resulting neural
network can be seen as an averaging ensemble of all these smaller neural networks.
There is one small but important technical detail. Suppose p = 50, in which case dur‐
ing testing a neuron will be connected to twice as many input neurons as it was (on
average) during training. To compensate for this fact, we need to multiply each neu‐
Avoiding Overfitting Through Regularization
|
305
Download from finelybook www.finelybook.com


ron’s input connection weights by 0.5 after training. If we don’t, each neuron will get a
total input signal roughly twice as large as what the network was trained on, and it is
unlikely to perform well. More generally, we need to multiply each input connection
weight by the keep probability (1 – p) after training. Alternatively, we can divide each
neuron’s output by the keep probability during training (these alternatives are not
perfectly equivalent, but they work equally well).
To implement dropout using TensorFlow, you can simply apply the dropout() func‐
tion to the input layer and to the output of every hidden layer. During training, this
function randomly drops some items (setting them to 0) and divides the remaining
items by the keep probability. After training, this function does nothing at all. The
following code applies dropout regularization to our three-layer neural network:
from tensorflow.contrib.layers import dropout
[...]


from tensorflow.contrib.layers import dropout
[...]
is_training = tf.placeholder(tf.bool, shape=(), name='is_training')
keep_prob = 0.5
X_drop = dropout(X, keep_prob, is_training=is_training)
hidden1 = fully_connected(X_drop, n_hidden1, scope="hidden1")
hidden1_drop = dropout(hidden1, keep_prob, is_training=is_training)
hidden2 = fully_connected(hidden1_drop, n_hidden2, scope="hidden2")
hidden2_drop = dropout(hidden2, keep_prob, is_training=is_training)
logits = fully_connected(hidden2_drop, n_outputs, activation_fn=None,
                         scope="outputs")
You want to use the dropout() function in tensorflow.con
trib.layers, not the one in tensorflow.nn. The first one turns off
(no-op) when not training, which is what you want, while the sec‐
ond one does not.
Of course, just like you did earlier for Batch Normalization, you need to set is_train
ing to True when training, and to False when testing.


ing to True when training, and to False when testing.
If you observe that the model is overfitting, you can increase the dropout rate (i.e.,
reduce the keep_prob hyperparameter). Conversely, you should try decreasing the
dropout rate (i.e., increasing keep_prob) if the model underfits the training set. It can
also help to increase the dropout rate for large layers, and reduce it for small ones.
Dropout does tend to significantly slow down convergence, but it usually results in a
much better model when tuned properly. So, it is generally well worth the extra time
and effort.
306
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


Dropconnect is a variant of dropout where individual connections
are dropped randomly rather than whole neurons. In general drop‐
out performs better.
Max-Norm Regularization
Another regularization technique that is quite popular for neural networks is called
max-norm regularization: for each neuron, it constrains the weights w of the incom‐
ing connections such that ∥ w ∥2 ≤ r, where r is the max-norm hyperparameter and
∥ · ∥2 is the ℓ2 norm.
We typically implement this constraint by computing ∥w∥2 after each training step
and clipping w if needed (�
�
r
∥�∥2).
Reducing r increases the amount of regularization and helps reduce overfitting. Max-
norm regularization can also help alleviate the vanishing/exploding gradients prob‐
lems (if you are not using Batch Normalization).
TensorFlow does not provide an off-the-shelf max-norm regularizer, but it is not too
hard to implement. The following code creates a node clip_weights that will clip the


hard to implement. The following code creates a node clip_weights that will clip the
weights variable along the second axis so that each row vector has a maximum norm
of 1.0:
threshold = 1.0
clipped_weights = tf.clip_by_norm(weights, clip_norm=threshold, axes=1)
clip_weights = tf.assign(weights, clipped_weights)
You would then apply this operation after each training step, like so:
with tf.Session() as sess:
    [...]
    for epoch in range(n_epochs):
        [...]
        for X_batch, y_batch in zip(X_batches, y_batches):
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
            clip_weights.eval()
You may wonder how to get access to the weights variable of each layer. For this you
can simply use a variable scope like this:
hidden1 = fully_connected(X, n_hidden1, scope="hidden1")
with tf.variable_scope("hidden1", reuse=True):
    weights1 = tf.get_variable("weights")
Alternatively, you can use the root variable scope:


weights1 = tf.get_variable("weights")
Alternatively, you can use the root variable scope:
hidden1 = fully_connected(X, n_hidden1, scope="hidden1")
hidden2 = fully_connected(hidden1, n_hidden2, scope="hidden2")
Avoiding Overfitting Through Regularization
|
307
Download from finelybook www.finelybook.com


[...]
with tf.variable_scope("", default_name="", reuse=True):  # root scope
    weights1 = tf.get_variable("hidden1/weights")
    weights2 = tf.get_variable("hidden2/weights")
If you don’t know what the name of a variable is, you can either use TensorBoard to
find out or simply use the global_variables() function and print out all the variable
names:
for variable in tf.global_variables():
    print(variable.name)
Although the preceding solution should work fine, it is a bit messy. A cleaner solution
is to create a max_norm_regularizer() function and use it just like the earlier l1_reg
ularizer() function:
def max_norm_regularizer(threshold, axes=1, name="max_norm",
                         collection="max_norm"):
    def max_norm(weights):
        clipped = tf.clip_by_norm(weights, clip_norm=threshold, axes=axes)
        clip_weights = tf.assign(weights, clipped, name=name)
        tf.add_to_collection(collection, clip_weights)


tf.add_to_collection(collection, clip_weights)
        return None  # there is no regularization loss term
    return max_norm
This function returns a parametrized max_norm() function that you can use like any
other regularizer:
max_norm_reg = max_norm_regularizer(threshold=1.0)
hidden1 = fully_connected(X, n_hidden1, scope="hidden1",
                          weights_regularizer=max_norm_reg)
Note that max-norm regularization does not require adding a regularization loss term
to your overall loss function, so the max_norm() function returns None. But you still
need to be able to run the clip_weights operation after each training step, so you
need to be able to get a handle on it. This is why the max_norm() function adds the
clip_weights node to a collection of max-norm clipping operations. You need to
fetch these clipping operations and run them after each training step:
clip_all_weights = tf.get_collection("max_norm")
with tf.Session() as sess:
    [...]


clip_all_weights = tf.get_collection("max_norm")
with tf.Session() as sess:
    [...]
    for epoch in range(n_epochs):
        [...]
        for X_batch, y_batch in zip(X_batches, y_batches):
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
            sess.run(clip_all_weights)
Much cleaner code, isn’t it?
308
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


Data Augmentation
One last regularization technique, data augmentation, consists of generating new
training instances from existing ones, artificially boosting the size of the training set.
This will reduce overfitting, making this a regularization technique. The trick is to
generate realistic training instances; ideally, a human should not be able to tell which
instances were generated and which ones were not. Moreover, simply adding white
noise will not help; the modifications you apply should be learnable (white noise is
not).
For example, if your model is meant to classify pictures of mushrooms, you can
slightly shift, rotate, and resize every picture in the training set by various amounts
and add the resulting pictures to the training set (see Figure 11-10). This forces the
model to be more tolerant to the position, orientation, and size of the mushrooms in
the picture. If you want the model to be more tolerant to lighting conditions, you can


the picture. If you want the model to be more tolerant to lighting conditions, you can
similarly generate many images with various contrasts. Assuming the mushrooms are
symmetrical, you can also flip the pictures horizontally. By combining these transfor‐
mations you can greatly increase the size of your training set.
Figure 11-10. Generating new training instances from existing ones
It is often preferable to generate training instances on the fly during training rather
than wasting storage space and network bandwidth. TensorFlow offers several image
manipulation operations such as transposing (shifting), rotating, resizing, flipping,
and cropping, as well as adjusting the brightness, contrast, saturation, and hue (see
Avoiding Overfitting Through Regularization
|
309
Download from finelybook www.finelybook.com


the API documentation for more details). This makes it easy to implement data aug‐
mentation for image datasets.
Another powerful technique to train very deep neural networks is
to add skip connections (a skip connection is when you add the
input of a layer to the output of a higher layer). We will explore this
idea in Chapter 13 when we talk about deep residual networks.
Practical Guidelines
In this chapter, we have covered a wide range of techniques and you may be wonder‐
ing which ones you should use. The configuration in Table 11-2 will work fine in
most cases.
Table 11-2. Default DNN configuration
Initialization
He initialization
Activation function
ELU
Normalization
Batch Normalization
Regularization
Dropout
Optimizer
Adam
Learning rate schedule
None
Of course, you should try to reuse parts of a pretrained neural network if you can
find one that solves a similar problem.
This default configuration may need to be tweaked:


find one that solves a similar problem.
This default configuration may need to be tweaked:
• If you can’t find a good learning rate (convergence was too slow, so you increased
the training rate, and now convergence is fast but the network’s accuracy is sub‐
optimal), then you can try adding a learning schedule such as exponential decay.
• If your training set is a bit too small, you can implement data augmentation.
• If you need a sparse model, you can add some ℓ1 regularization to the mix (and
optionally zero out the tiny weights after training). If you need an even sparser
model, you can try using FTRL instead of Adam optimization, along with ℓ1 reg‐
ularization.
• If you need a lightning-fast model at runtime, you may want to drop Batch Nor‐
malization, and possibly replace the ELU activation function with the leaky
ReLU. Having a sparse model will also help.
With these guidelines, you are now ready to train very deep nets—well, if you are


With these guidelines, you are now ready to train very deep nets—well, if you are
very patient, that is! If you use a single machine, you may have to wait for days or
310
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


even months for training to complete. In the next chapter we will discuss how to use
distributed TensorFlow to train and run models across many servers and GPUs.
Exercises
1. Is it okay to initialize all the weights to the same value as long as that value is
selected randomly using He initialization?
2. Is it okay to initialize the bias terms to 0?
3. Name three advantages of the ELU activation function over ReLU.
4. In which cases would you want to use each of the following activation functions:
ELU, leaky ReLU (and its variants), ReLU, tanh, logistic, and softmax?
5. What may happen if you set the momentum hyperparameter too close to 1 (e.g.,
0.99999) when using a MomentumOptimizer?
6. Name three ways you can produce a sparse model.
7. Does dropout slow down training? Does it slow down inference (i.e., making
predictions on new instances)?
8. Deep Learning.
a. Build a DNN with five hidden layers of 100 neurons each, He initialization,
and the ELU activation function.


and the ELU activation function.
b. Using Adam optimization and early stopping, try training it on MNIST but
only on digits 0 to 4, as we will use transfer learning for digits 5 to 9 in the
next exercise. You will need a softmax output layer with five neurons, and as
always make sure to save checkpoints at regular intervals and save the final
model so you can reuse it later.
c. Tune the hyperparameters using cross-validation and see what precision you
can achieve.
d. Now try adding Batch Normalization and compare the learning curves: is it
converging faster than before? Does it produce a better model?
e. Is the model overfitting the training set? Try adding dropout to every layer
and try again. Does it help?
9. Transfer learning.
a. Create a new DNN that reuses all the pretrained hidden layers of the previous
model, freezes them, and replaces the softmax output layer with a fresh new
one.
b. Train this new DNN on digits 5 to 9, using only 100 images per digit, and time


one.
b. Train this new DNN on digits 5 to 9, using only 100 images per digit, and time
how long it takes. Despite this small number of examples, can you achieve
high precision?
Exercises
|
311
Download from finelybook www.finelybook.com


c. Try caching the frozen layers, and train the model again: how much faster is it
now?
d. Try again reusing just four hidden layers instead of five. Can you achieve a
higher precision?
e. Now unfreeze the top two hidden layers and continue training: can you get
the model to perform even better?
10. Pretraining on an auxiliary task.
a. In this exercise you will build a DNN that compares two MNIST digit images
and predicts whether they represent the same digit or not. Then you will reuse
the lower layers of this network to train an MNIST classifier using very little
training data. Start by building two DNNs (let’s call them DNN A and B), both
similar to the one you built earlier but without the output layer: each DNN
should have five hidden layers of 100 neurons each, He initialization, and ELU
activation. Next, add a single output layer on top of both DNNs. You should
use TensorFlow’s concat() function with axis=1 to concatenate the outputs


use TensorFlow’s concat() function with axis=1 to concatenate the outputs
of both DNNs along the horizontal axis, then feed the result to the output
layer. This output layer should contain a single neuron using the logistic acti‐
vation function.
b. Split the MNIST training set in two sets: split #1 should containing 55,000
images, and split #2 should contain contain 5,000 images. Create a function
that generates a training batch where each instance is a pair of MNIST images
picked from split #1. Half of the training instances should be pairs of images
that belong to the same class, while the other half should be images from dif‐
ferent classes. For each pair, the training label should be 0 if the images are
from the same class, or 1 if they are from different classes.
c. Train the DNN on this training set. For each image pair, you can simultane‐
ously feed the first image to DNN A and the second image to DNN B. The


ously feed the first image to DNN A and the second image to DNN B. The
whole network will gradually learn to tell whether two images belong to the
same class or not.
d. Now create a new DNN by reusing and freezing the hidden layers of DNN A
and adding a softmax output layer on with 10 neurons. Train this network on
split #2 and see if you can achieve high performance despite having only 500
images per class.
Solutions to these exercises are available in Appendix A.
312
|
Chapter 11: Training Deep Neural Nets
Download from finelybook www.finelybook.com


CHAPTER 12
Distributing TensorFlow Across
Devices and Servers
In Chapter 11 we discussed several techniques that can considerably speed up train‐
ing: better weight initialization, Batch Normalization, sophisticated optimizers, and
so on. However, even with all of these techniques, training a large neural network on
a single machine with a single CPU can take days or even weeks.
In this chapter we will see how to use TensorFlow to distribute computations across
multiple devices (CPUs and GPUs) and run them in parallel (see Figure 12-1). First
we will distribute computations across multiple devices on just one machine, then on
multiple devices across multiple machines.
Figure 12-1. Executing a TensorFlow graph across multiple devices in parallel
313
Download from finelybook www.finelybook.com


TensorFlow’s support of distributed computing is one of its main highlights com‐
pared to other neural network frameworks. It gives you full control over how to split
(or replicate) your computation graph across devices and servers, and it lets you par‐
allelize and synchronize operations in flexible ways so you can choose between all
sorts of parallelization approaches.
We will look at some of the most popular approaches to parallelizing the execution
and training of a neural network. Instead of waiting for weeks for a training algo‐
rithm to complete, you may end up waiting for just a few hours. Not only does this
save an enormous amount of time, it also means that you can experiment with vari‐
ous models much more easily, and frequently retrain your models on fresh data.
Other great use cases of parallelization include exploring a much larger hyperparame‐
ter space when fine-tuning your model, and running large ensembles of neural net‐
works efficiently.


works efficiently.
But we must learn to walk before we can run. Let’s start by parallelizing simple graphs
across several GPUs on a single machine.
Multiple Devices on a Single Machine
You can often get a major performance boost simply by adding GPU cards to a single
machine. In fact, in many cases this will suffice; you won’t need to use multiple
machines at all. For example, you can typically train a neural network just as fast
using 8 GPUs on a single machine rather than 16 GPUs across multiple machines
(due to the extra delay imposed by network communications in a multimachine
setup).
In this section we will look at how to set up your environment so that TensorFlow can
use multiple GPU cards on one machine. Then we will look at how you can distribute
operations across available devices and execute them in parallel.
Installation
In order to run TensorFlow on multiple GPU cards, you first need to make sure your


Installation
In order to run TensorFlow on multiple GPU cards, you first need to make sure your
GPU cards have NVidia Compute Capability (greater or equal to 3.0). This includes
Nvidia’s Titan, Titan X, K20, and K40 cards (if you own another card, you can check
its compatibility at https://developer.nvidia.com/cuda-gpus).
314
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


If you don’t own any GPU cards, you can use a hosting service with
GPU capability such as Amazon AWS. Detailed instructions to set
up TensorFlow 0.9 with Python 3.5 on an Amazon AWS GPU
instance are available in Žiga Avsec’s helpful blog post. It should
not be too hard to update it to the latest version of TensorFlow.
Google also released a cloud service called Cloud Machine Learning
to run TensorFlow graphs. In May 2016, they announced that their
platform now includes servers equipped with tensor processing units
(TPUs), processors specialized for Machine Learning that are much
faster than GPUs for many ML tasks. Of course, another option is
simply to buy your own GPU card. Tim Dettmers wrote a great
blog post to help you choose, and he updates it fairly regularly.
You must then download and install the appropriate version of the CUDA and
cuDNN libraries (CUDA 8.0 and cuDNN 5.1 if you are using the binary installation


cuDNN libraries (CUDA 8.0 and cuDNN 5.1 if you are using the binary installation
of TensorFlow 1.0.0), and set a few environment variables so TensorFlow knows
where to find CUDA and cuDNN. The detailed installation instructions are likely to
change fairly quickly, so it is best that you follow the instructions on TensorFlow’s
website.
Nvidia’s Compute Unified Device Architecture library (CUDA) allows developers to
use CUDA-enabled GPUs for all sorts of computations (not just graphics accelera‐
tion). Nvidia’s CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated
library of primitives for DNNs. It provides optimized implementations of common
DNN computations such as activation layers, normalization, forward and backward
convolutions, and pooling (see Chapter 13). It is part of Nvidia’s Deep Learning SDK
(note that it requires creating an Nvidia developer account in order to download it).
TensorFlow uses CUDA and cuDNN to control the GPU cards and accelerate com‐


TensorFlow uses CUDA and cuDNN to control the GPU cards and accelerate com‐
putations (see Figure 12-2).
Figure 12-2. TensorFlow uses CUDA and cuDNN to control GPUs and boost DNNs
You can use the nvidia-smi command to check that CUDA is properly installed. It
lists the available GPU cards, as well as processes running on each card:
Multiple Devices on a Single Machine
|
315
Download from finelybook www.finelybook.com


$ nvidia-smi
Wed Sep 16 09:50:03 2016
+------------------------------------------------------+
| NVIDIA-SMI 352.63     Driver Version: 352.63         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   27C    P8    17W / 125W |     11MiB /  4095MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |


|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
Finally, you must install TensorFlow with GPU support. If you created an isolated
environment using virtualenv, you first need to activate it:
$ cd $ML_PATH               # Your ML working directory (e.g., $HOME/ml)
$ source env/bin/activate
Then install the appropriate GPU-enabled version of TensorFlow:
$ pip3 install --upgrade tensorflow-gpu
Now you can open up a Python shell and check that TensorFlow detects and uses
CUDA and cuDNN properly by importing TensorFlow and creating a session:
>>> import tensorflow as tf
I [...]/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally


I [...]/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I [...]/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I [...]/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I [...]/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I [...]/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
>>> sess = tf.Session()
[...]
I [...]/gpu_init.cc:102] Found device 0 with properties:
name: GRID K520
major: 3 minor: 0 memoryClockRate (GHz) 0.797
pciBusID 0000:00:03.0
Total memory: 4.00GiB
Free memory: 3.95GiB
I [...]/gpu_init.cc:126] DMA: 0
I [...]/gpu_init.cc:136] 0:   Y
I [...]/gpu_device.cc:839] Creating TensorFlow device
(/gpu:0) -> (device: 0, name: GRID K520, pci bus id: 0000:00:03.0)
Looks good! TensorFlow detected the CUDA and cuDNN libraries, and it used the
CUDA library to detect the GPU card (in this case an Nvidia Grid K520 card).
316
|


CUDA library to detect the GPU card (in this case an Nvidia Grid K520 card).
316
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Managing the GPU RAM
By default TensorFlow automatically grabs all the RAM in all available GPUs the first
time you run a graph, so you will not be able to start a second TensorFlow program
while the first one is still running. If you try, you will get the following error:
E [...]/cuda_driver.cc:965] failed to allocate 3.66G (3928915968 bytes) from
device: CUDA_ERROR_OUT_OF_MEMORY
One solution is to run each process on different GPU cards. To do this, the simplest
option is to set the CUDA_VISIBLE_DEVICES environment variable so that each process
only sees the appropriate GPU cards. For example, you could start two programs like
this:
$ CUDA_VISIBLE_DEVICES=0,1 python3 program_1.py
# and in another terminal:
$ CUDA_VISIBLE_DEVICES=3,2 python3 program_2.py
Program #1 will only see GPU cards 0 and 1 (numbered 0 and 1, respectively), and
program #2 will only see GPU cards 2 and 3 (numbered 1 and 0, respectively). Every‐
thing will work fine (see Figure 12-3).


thing will work fine (see Figure 12-3).
Figure 12-3. Each program gets two GPUs for itself
Another option is to tell TensorFlow to grab only a fraction of the memory. For
example, to make TensorFlow grab only 40% of each GPU’s memory, you must create
a ConfigProto object, set its gpu_options.per_process_gpu_memory_fraction
option to 0.4, and create the session using this configuration:
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config)
Now two programs like this one can run in parallel using the same GPU cards (but
not three, since 3 × 0.4 > 1). See Figure 12-4.
Multiple Devices on a Single Machine
|
317
Download from finelybook www.finelybook.com


1 “TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems,” Google Research
(2015).
Figure 12-4. Each program gets all four GPUs, but with only 40% of the RAM each
If you run the nvidia-smi command while both programs are running, you should
see that each process holds roughly 40% of the total RAM of each card:
$ nvidia-smi
[...]
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      5231    C   python                                        1677MiB |
|    0      5262    C   python                                        1677MiB |
|    1      5231    C   python                                        1677MiB |
|    1      5262    C   python                                        1677MiB |


|    1      5262    C   python                                        1677MiB |
[...]
Yet another option is to tell TensorFlow to grab memory only when it needs it. To do
this you must set config.gpu_options.allow_growth to True. However, TensorFlow
never releases memory once it has grabbed it (to avoid memory fragmentation) so
you may still run out of memory after a while. It may be harder to guarantee a deter‐
ministic behavior using this option, so in general you probably want to stick with one
of the previous options.
Okay, now you have a working GPU-enabled TensorFlow installation. Let’s see how
to use it!
Placing Operations on Devices
The TensorFlow whitepaper1 presents a friendly dynamic placer algorithm that auto‐
magically distributes operations across all available devices, taking into account
things like the measured computation time in previous runs of the graph, estimations
of the size of the input and output tensors to each operation, the amount of RAM


of the size of the input and output tensors to each operation, the amount of RAM
available in each device, communication delay when transferring data in and out of
318
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


devices, hints and constraints from the user, and more. Unfortunately, this sophistica‐
ted algorithm is internal to Google; it was not released in the open source version of
TensorFlow. The reason it was left out seems to be that in practice a small set of place‐
ment rules specified by the user actually results in more efficient placement than what
the dynamic placer is capable of. However, the TensorFlow team is working on
improving the dynamic placer, and perhaps it will eventually be good enough to be
released.
Until then TensorFlow relies on the simple placer, which (as its name suggests) is very
basic.
Simple placement
Whenever you run a graph, if TensorFlow needs to evaluate a node that is not placed
on a device yet, it uses the simple placer to place it, along with all other nodes that are
not placed yet. The simple placer respects the following rules:
• If a node was already placed on a device in a previous run of the graph, it is left
on that device.


on that device.
• Else, if the user pinned a node to a device (described next), the placer places it on
that device.
• Else, it defaults to GPU #0, or the CPU if there is no GPU.
As you can see, placing operations on the appropriate device is mostly up to you. If
you don’t do anything, the whole graph will be placed on the default device. To pin
nodes onto a device, you must create a device block using the device() function. For
example, the following code pins the variable a and the constant b on the CPU, but
the multiplication node c is not pinned on any device, so it will be placed on the
default device:
with tf.device("/cpu:0"):
    a = tf.Variable(3.0)
    b = tf.constant(4.0)
c = a * b
The "/cpu:0" device aggregates all CPUs on a multi-CPU system.
There is currently no way to pin nodes on specific CPUs or to use
just a subset of all CPUs.
Multiple Devices on a Single Machine
|
319
Download from finelybook www.finelybook.com


Logging placements
Let’s check that the simple placer respects the placement constraints we have just
defined. For this you can set the log_device_placement option to True; this tells the
placer to log a message whenever it places a node. For example:
>>> config = tf.ConfigProto()
>>> config.log_device_placement = True
>>> sess = tf.Session(config=config)
I [...] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GRID K520,
pci bus id: 0000:00:03.0)
[...]
>>> x.initializer.run(session=sess)
I [...] a: /job:localhost/replica:0/task:0/cpu:0
I [...] a/read: /job:localhost/replica:0/task:0/cpu:0
I [...] mul: /job:localhost/replica:0/task:0/gpu:0
I [...] a/Assign: /job:localhost/replica:0/task:0/cpu:0
I [...] b: /job:localhost/replica:0/task:0/cpu:0
I [...] a/initial_value: /job:localhost/replica:0/task:0/cpu:0
>>> sess.run(c)
12
The lines starting with "I" for Info are the log messages. When we create a session,


12
The lines starting with "I" for Info are the log messages. When we create a session,
TensorFlow logs a message to tell us that it has found a GPU card (in this case the
Grid K520 card). Then the first time we run the graph (in this case when initializing
the variable a), the simple placer is run and places each node on the device it was
assigned to. As expected, the log messages show that all nodes are placed on "/cpu:0"
except the multiplication node, which ends up on the default device "/gpu:0" (you
can safely ignore the prefix /job:localhost/replica:0/task:0 for now; we will talk
about it in a moment). Notice that the second time we run the graph (to compute c),
the placer is not used since all the nodes TensorFlow needs to compute c are already
placed.
Dynamic placement function
When you create a device block, you can specify a function instead of a device name.
TensorFlow will call this function for each operation it needs to place in the device


TensorFlow will call this function for each operation it needs to place in the device
block, and the function must return the name of the device to pin the operation on.
For example, the following code pins all the variable nodes to "/cpu:0" (in this case
just the variable a) and all other nodes to "/gpu:0":
def variables_on_cpu(op):
    if op.type == "Variable":
        return "/cpu:0"
    else:
        return "/gpu:0"
with tf.device(variables_on_cpu):
    a = tf.Variable(3.0)
320
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


b = tf.constant(4.0)
    c = a * b
You can easily implement more complex algorithms, such as pinning variables across
GPUs in a round-robin fashion.
Operations and kernels
For a TensorFlow operation to run on a device, it needs to have an implementation
for that device; this is called a kernel. Many operations have kernels for both CPUs
and GPUs, but not all of them. For example, TensorFlow does not have a GPU kernel
for integer variables, so the following code will fail when TensorFlow tries to place the
variable i on GPU #0:
>>> with tf.device("/gpu:0"):
...     i = tf.Variable(3)
[...]
>>> sess.run(i.initializer)
Traceback (most recent call last):
[...]
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device
to node 'Variable': Could not satisfy explicit device specification
Note that TensorFlow infers that the variable must be of type int32 since the initiali‐
zation value is an integer. If you change the initialization value to 3.0 instead of 3, or


zation value is an integer. If you change the initialization value to 3.0 instead of 3, or
if you explicitly set dtype=tf.float32 when creating the variable, everything will
work fine.
Soft placement
By default, if you try to pin an operation on a device for which the operation has no
kernel, you get the exception shown earlier when TensorFlow tries to place the opera‐
tion on the device. If you prefer TensorFlow to fall back to the CPU instead, you can
set the allow_soft_placement configuration option to True:
with tf.device("/gpu:0"):
    i = tf.Variable(3)
config = tf.ConfigProto()
config.allow_soft_placement = True
sess = tf.Session(config=config)
sess.run(i.initializer)  # the placer runs and falls back to /cpu:0
So far we have discussed how to place nodes on different devices. Now let’s see how
TensorFlow will run these nodes in parallel.
Parallel Execution
When TensorFlow runs a graph, it starts by finding out the list of nodes that need to


When TensorFlow runs a graph, it starts by finding out the list of nodes that need to
be evaluated, and it counts how many dependencies each of them has. TensorFlow
Multiple Devices on a Single Machine
|
321
Download from finelybook www.finelybook.com


then starts evaluating the nodes with zero dependencies (i.e., source nodes). If these
nodes are placed on separate devices, they obviously get evaluated in parallel. If they
are placed on the same device, they get evaluated in different threads, so they may run
in parallel too (in separate GPU threads or CPU cores).
TensorFlow manages a thread pool on each device to parallelize operations (see
Figure 12-5). These are called the inter-op thread pools. Some operations have multi‐
threaded kernels: they can use other thread pools (one per device) called the intra-op
thread pools.
Figure 12-5. Parallelized execution of a TensorFlow graph
For example, in Figure 12-5, operations A, B, and C are source ops, so they can
immediately be evaluated. Operations A and B are placed on GPU #0, so they are sent
to this device’s inter-op thread pool, and immediately evaluated in parallel. Operation
A happens to have a multithreaded kernel; its computations are split in three parts,


A happens to have a multithreaded kernel; its computations are split in three parts,
which are executed in parallel by the intra-op thread pool. Operation C goes to GPU
#1’s inter-op thread pool.
As soon as operation C finishes, the dependency counters of operations D and E will
be decremented and will both reach 0, so both operations will be sent to the inter-op
thread pool to be executed.
You can control the number of threads per inter-op pool by setting
the inter_op_parallelism_threads option. Note that the first ses‐
sion you start creates the inter-op thread pools. All other sessions
will just reuse them unless you set the use_per_session_threads
option to True. You can control the number of threads per intra-op
pool by setting the intra_op_parallelism_threads option.
322
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Control Dependencies
In some cases, it may be wise to postpone the evaluation of an operation even though
all the operations it depends on have been executed. For example, if it uses a lot of
memory but its value is needed only much further in the graph, it would be best to
evaluate it at the last moment to avoid needlessly occupying RAM that other opera‐
tions may need. Another example is a set of operations that depend on data located
outside of the device. If they all run at the same time, they may saturate the device’s
communication bandwidth, and they will end up all waiting on I/O. Other operations
that need to communicate data will also be blocked. It would be preferable to execute
these communication-heavy operations sequentially, allowing the device to perform
other operations in parallel.
To postpone evaluation of some nodes, a simple solution is to add control dependen‐
cies. For example, the following code tells TensorFlow to evaluate x and y only after a


cies. For example, the following code tells TensorFlow to evaluate x and y only after a
and b have been evaluated:
a = tf.constant(1.0)
b = a + 2.0
with tf.control_dependencies([a, b]):
    x = tf.constant(3.0)
    y = tf.constant(4.0)
z = x + y
Obviously, since z depends on x and y, evaluating z also implies waiting for a and b to
be evaluated, even though it is not explicitly in the control_dependencies() block.
Also, since b depends on a, we could simplify the preceding code by just creating a
control dependency on [b] instead of [a, b], but in some cases “explicit is better
than implicit.”
Great! Now you know:
• How to place operations on multiple devices in any way you please
• How these operations get executed in parallel
• How to create control dependencies to optimize parallel execution
It’s time to distribute computations across multiple servers!
Multiple Devices Across Multiple Servers
To run a graph across multiple servers, you first need to define a cluster. A cluster is


To run a graph across multiple servers, you first need to define a cluster. A cluster is
composed of one or more TensorFlow servers, called tasks, typically spread across
several machines (see Figure 12-6). Each task belongs to a job. A job is just a named
group of tasks that typically have a common role, such as keeping track of the model
Multiple Devices Across Multiple Servers
|
323
Download from finelybook www.finelybook.com


parameters (such a job is usually named "ps" for parameter server), or performing
computations (such a job is usually named "worker").
Figure 12-6. TensorFlow cluster
The following cluster specification defines two jobs, "ps" and "worker", containing
one task and two tasks, respectively. In this example, machine A hosts two Tensor‐
Flow servers (i.e., tasks), listening on different ports: one is part of the "ps" job, and
the other is part of the "worker" job. Machine B just hosts one TensorFlow server,
part of the "worker" job.
cluster_spec = tf.train.ClusterSpec({
    "ps": [
        "machine-a.example.com:2221",  # /job:ps/task:0
    ],
    "worker": [
        "machine-a.example.com:2222",  # /job:worker/task:0
        "machine-b.example.com:2222",  # /job:worker/task:1
    ]})
To start a TensorFlow server, you must create a Server object, passing it the cluster
specification (so it can communicate with other servers) and its own job name and


specification (so it can communicate with other servers) and its own job name and
task number. For example, to start the first worker task, you would run the following
code on machine A:
server = tf.train.Server(cluster_spec, job_name="worker", task_index=0)
It is usually simpler to just run one task per machine, but the previous example dem‐
onstrates that TensorFlow allows you to run multiple tasks on the same machine if
324
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


2 You can even start multiple tasks in the same process. It may be useful for tests, but it is not recommended in
production.
3 It is the next version of Google’s internal Stubby service, which Google has used successfully for over a decade.
See http://grpc.io/ for more details.
you want.2 If you have several servers on one machine, you will need to ensure that
they don’t all try to grab all the RAM of every GPU, as explained earlier. For example,
in Figure 12-6 the "ps" task does not see the GPU devices, since presumably its pro‐
cess was launched with CUDA_VISIBLE_DEVICES="". Note that the CPU is shared by
all tasks located on the same machine.
If you want the process to do nothing other than run the TensorFlow server, you can
block the main thread by telling it to wait for the server to finish using the join()
method (otherwise the server will be killed as soon as your main thread exits). Since
there is currently no way to stop the server, this will actually block forever:


there is currently no way to stop the server, this will actually block forever:
server.join()  # blocks until the server stops (i.e., never)
Opening a Session
Once all the tasks are up and running (doing nothing yet), you can open a session on
any of the servers, from a client located in any process on any machine (even from a
process running one of the tasks), and use that session like a regular local session. For
example:
a = tf.constant(1.0)
b = a + 2
c = a * 3
with tf.Session("grpc://machine-b.example.com:2222") as sess:
    print(c.eval())  # 9.0
This client code first creates a simple graph, then opens a session on the TensorFlow
server located on machine B (which we will call the master), and instructs it to evalu‐
ate c. The master starts by placing the operations on the appropriate devices. In this
example, since we did not pin any operation on any device, the master simply places
them all on its own default device—in this case, machine B’s GPU device. Then it just


them all on its own default device—in this case, machine B’s GPU device. Then it just
evaluates c as instructed by the client, and it returns the result.
The Master and Worker Services
The client uses the gRPC protocol (Google Remote Procedure Call) to communicate
with the server. This is an efficient open source framework to call remote functions
and get their outputs across a variety of platforms and languages.3 It is based on
HTTP2, which opens a connection and leaves it open during the whole session,
allowing efficient bidirectional communication once the connection is established.
Multiple Devices Across Multiple Servers
|
325
Download from finelybook www.finelybook.com


Data is transmitted in the form of protocol buffers, another open source Google tech‐
nology. This is a lightweight binary data interchange format.
All servers in a TensorFlow cluster may communicate with any
other server in the cluster, so make sure to open the appropriate
ports on your firewall.
Every TensorFlow server provides two services: the master service and the worker ser‐
vice. The master service allows clients to open sessions and use them to run graphs. It
coordinates the computations across tasks, relying on the worker service to actually
execute computations on other tasks and get their results.
This architecture gives you a lot of flexibility. One client can connect to multiple
servers by opening multiple sessions in different threads. One server can handle mul‐
tiple sessions simultaneously from one or more clients. You can run one client per
task (typically within the same process), or just one client to control all tasks. All
options are open.


options are open.
Pinning Operations Across Tasks
You can use device blocks to pin operations on any device managed by any task, by
specifying the job name, task index, device type, and device index. For example, the
following code pins a to the CPU of the first task in the "ps" job (that’s the CPU on
machine A), and it pins b to the second GPU managed by the first task of the
"worker" job (that’s GPU #1 on machine A). Finally, c is not pinned to any device, so
the master places it on its own default device (machine B’s GPU #0 device).
with tf.device("/job:ps/task:0/cpu:0")
    a = tf.constant(1.0)
with tf.device("/job:worker/task:0/gpu:1")
    b = a + 2
c = a + b
As earlier, if you omit the device type and index, TensorFlow will default to the task’s
default device; for example, pinning an operation to "/job:ps/task:0" will place it
on the default device of the first task of the "ps" job (machine A’s CPU). If you also


on the default device of the first task of the "ps" job (machine A’s CPU). If you also
omit the task index (e.g., "/job:ps"), TensorFlow defaults to "/task:0". If you omit
the job name and the task index, TensorFlow defaults to the session’s master task.
326
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Sharding Variables Across Multiple Parameter Servers
As we will see shortly, a common pattern when training a neural network on a dis‐
tributed setup is to store the model parameters on a set of parameter servers (i.e., the
tasks in the "ps" job) while other tasks focus on computations (i.e., the tasks in the
"worker" job). For large models with millions of parameters, it is useful to shard
these parameters across multiple parameter servers, to reduce the risk of saturating a
single parameter server’s network card. If you were to manually pin every variable to
a different parameter server, it would be quite tedious. Fortunately, TensorFlow pro‐
vides the replica_device_setter() function, which distributes variables across all
the "ps" tasks in a round-robin fashion. For example, the following code pins five
variables to two parameter servers:
with tf.device(tf.train.replica_device_setter(ps_tasks=2):
    v1 = tf.Variable(1.0)  # pinned to /job:ps/task:0


v1 = tf.Variable(1.0)  # pinned to /job:ps/task:0
    v2 = tf.Variable(2.0)  # pinned to /job:ps/task:1
    v3 = tf.Variable(3.0)  # pinned to /job:ps/task:0
    v4 = tf.Variable(4.0)  # pinned to /job:ps/task:1
    v5 = tf.Variable(5.0)  # pinned to /job:ps/task:0
Instead of passing the number of ps_tasks, you can pass the cluster spec clus
ter=cluster_spec and TensorFlow will simply count the number of tasks in the "ps"
job.
If you create other operations in the block, beyond just variables, TensorFlow auto‐
matically pins them to "/job:worker", which will default to the first device managed
by the first task in the "worker" job. You can pin them to another device by setting
the worker_device parameter, but a better approach is to use embedded device
blocks. An inner device block can override the job, task, or device defined in an outer
block. For example:
with tf.device(tf.train.replica_device_setter(ps_tasks=2)):


block. For example:
with tf.device(tf.train.replica_device_setter(ps_tasks=2)):
    v1 = tf.Variable(1.0)  # pinned to /job:ps/task:0 (+ defaults to /cpu:0)
    v2 = tf.Variable(2.0)  # pinned to /job:ps/task:1 (+ defaults to /cpu:0)
    v3 = tf.Variable(3.0)  # pinned to /job:ps/task:0 (+ defaults to /cpu:0)
    [...]
    s = v1 + v2            # pinned to /job:worker (+ defaults to task:0/gpu:0)
    with tf.device("/gpu:1"):
        p1 = 2 * s         # pinned to /job:worker/gpu:1 (+ defaults to /task:0)
        with tf.device("/task:1"):
            p2 = 3 * s     # pinned to /job:worker/task:1/gpu:1
This example assumes that the parameter servers are CPU-only,
which is typically the case since they only need to store and com‐
municate parameters, not perform intensive computations.
Multiple Devices Across Multiple Servers
|
327
Download from finelybook www.finelybook.com


Sharing State Across Sessions Using Resource Containers
When you are using a plain local session (not the distributed kind), each variable’s
state is managed by the session itself; as soon as it ends, all variable values are lost.
Moreover, multiple local sessions cannot share any state, even if they both run the
same graph; each session has its own copy of every variable (as we discussed in Chap‐
ter 9). In contrast, when you are using distributed sessions, variable state is managed
by resource containers located on the cluster itself, not by the sessions. So if you create
a variable named x using one client session, it will automatically be available to any
other session on the same cluster (even if both sessions are connected to a different
server). For example, consider the following client code:
# simple_client.py
import tensorflow as tf
import sys
x = tf.Variable(0.0, name="x")
increment_x = tf.assign(x, x + 1)
with tf.Session(sys.argv[1]) as sess:


increment_x = tf.assign(x, x + 1)
with tf.Session(sys.argv[1]) as sess:
    if sys.argv[2:]==["init"]:
        sess.run(x.initializer)
    sess.run(increment_x)
    print(x.eval())
Let’s suppose you have a TensorFlow cluster up and running on machines A and B,
port 2222. You could launch the client, have it open a session with the server on
machine A, and tell it to initialize the variable, increment it, and print its value by
launching the following command:
$ python3 simple_client.py grpc://machine-a.example.com:2222 init
1.0
Now if you launch the client with the following command, it will connect to the
server on machine B and magically reuse the same variable x (this time we don’t ask
the server to initialize the variable):
$ python3 simple_client.py grpc://machine-b.example.com:2222
2.0
This feature cuts both ways: it’s great if you want to share variables across multiple
sessions, but if you want to run completely independent computations on the same


sessions, but if you want to run completely independent computations on the same
cluster you will have to be careful not to use the same variable names by accident.
One way to ensure that you won’t have name clashes is to wrap all of your construc‐
tion phase inside a variable scope with a unique name for each computation, for
example:
with tf.variable_scope("my_problem_1"):
    [...] # Construction phase of problem 1
328
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


A better option is to use a container block:
with tf.container("my_problem_1"):
    [...] # Construction phase of problem 1
This will use a container dedicated to problem #1, instead of the default one (whose
name is an empty string ""). One advantage is that variable names remain nice and
short. Another advantage is that you can easily reset a named container. For example,
the following command will connect to the server on machine A and ask it to reset
the container named "my_problem_1", which will free all the resources this container
used (and also close all sessions open on the server). Any variable managed by this
container must be initialized before you can use it again:
tf.Session.reset("grpc://machine-a.example.com:2222", ["my_problem_1"])
Resource containers make it easy to share variables across sessions in flexible ways.
For example, Figure 12-7 shows four clients running different graphs on the same


For example, Figure 12-7 shows four clients running different graphs on the same
cluster, but sharing some variables. Clients A and B share the same variable x man‐
aged by the default container, while clients C and D share another variable named x
managed by the container named "my_problem_1". Note that client C even uses vari‐
ables from both containers.
Figure 12-7. Resource containers
Resource containers also take care of preserving the state of other stateful operations,
namely queues and readers. Let’s take a look at queues first.
Asynchronous Communication Using TensorFlow Queues
Queues are another great way to exchange data between multiple sessions; for exam‐
ple, one common use case is to have a client create a graph that loads the training data
and pushes it into a queue, while another client creates a graph that pulls the data
from the queue and trains a model (see Figure 12-8). This can speed up training con‐
Multiple Devices Across Multiple Servers
|
329


Multiple Devices Across Multiple Servers
|
329
Download from finelybook www.finelybook.com


siderably because the training operations don’t have to wait for the next mini-batch at
every step.
Figure 12-8. Using queues to load the training data asynchronously
TensorFlow provides various kinds of queues. The simplest kind is the first-in first-
out (FIFO) queue. For example, the following code creates a FIFO queue that can
store up to 10 tensors containing two float values each:
q = tf.FIFOQueue(capacity=10, dtypes=[tf.float32], shapes=[[2]],
                 name="q", shared_name="shared_q")
To share variables across sessions, all you had to do was to specify
the same name and container on both ends. With queues Tensor‐
Flow does not use the name attribute but instead uses shared_name,
so it is important to specify it (even if it is the same as the name).
And, of course, use the same container.
Enqueuing data
To push data to a queue, you must create an enqueue operation. For example, the fol‐
lowing code pushes three training instances to the queue:
# training_data_loader.py


lowing code pushes three training instances to the queue:
# training_data_loader.py
import tensorflow as tf
q = [...]
training_instance = tf.placeholder(tf.float32, shape=(2))
enqueue = q.enqueue([training_instance])
with tf.Session("grpc://machine-a.example.com:2222") as sess:
   sess.run(enqueue, feed_dict={training_instance: [1., 2.]})
   sess.run(enqueue, feed_dict={training_instance: [3., 4.]})
   sess.run(enqueue, feed_dict={training_instance: [5., 6.]})
330
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Instead of enqueuing instances one by one, you can enqueue several at a time using
an enqueue_many operation:
[...]
training_instances = tf.placeholder(tf.float32, shape=(None, 2))
enqueue_many = q.enqueue([training_instances])
with tf.Session("grpc://machine-a.example.com:2222") as sess:
   sess.run(enqueue_many,
            feed_dict={training_instances: [[1., 2.], [3., 4.], [5., 6.]]})
Both examples enqueue the same three tensors to the queue.
Dequeuing data
To pull the instances out of the queue, on the other end, you need to use a dequeue
operation:
# trainer.py
import tensorflow as tf
q = [...]
dequeue = q.dequeue()
with tf.Session("grpc://machine-a.example.com:2222") as sess:
   print(sess.run(dequeue))  # [1., 2.]
   print(sess.run(dequeue))  # [3., 4.]
   print(sess.run(dequeue))  # [5., 6.]
In general you will want to pull a whole mini-batch at once, instead of pulling just
one instance at a time. To do so, you must use a dequeue_many operation, specifying


one instance at a time. To do so, you must use a dequeue_many operation, specifying
the mini-batch size:
[...]
batch_size = 2
dequeue_mini_batch= q.dequeue_many(batch_size)
with tf.Session("grpc://machine-a.example.com:2222") as sess:
   print(sess.run(dequeue_mini_batch))  # [[1., 2.], [4., 5.]]
   print(sess.run(dequeue_mini_batch))  # blocked waiting for another instance
When a queue is full, the enqueue operation will block until items are pulled out by a
dequeue operation. Similarly, when a queue is empty (or you are using
dequeue_many() and there are fewer items than the mini-batch size), the dequeue
operation will block until enough items are pushed into the queue using an enqueue
operation.
Multiple Devices Across Multiple Servers
|
331
Download from finelybook www.finelybook.com


Queues of tuples
Each item in a queue can be a tuple of tensors (of various types and shapes) instead of
just a single tensor. For example, the following queue stores pairs of tensors, one of
type int32 and shape (), and the other of type float32 and shape [3,2]:
q = tf.FIFOQueue(capacity=10, dtypes=[tf.int32, tf.float32], shapes=[[],[3,2]],
                 name="q", shared_name="shared_q")
The enqueue operation must be given pairs of tensors (note that each pair represents
only one item in the queue):
a = tf.placeholder(tf.int32, shape=())
b = tf.placeholder(tf.float32, shape=(3, 2))
enqueue = q.enqueue((a, b))
with tf.Session([...]) as sess:
    sess.run(enqueue, feed_dict={a: 10, b:[[1., 2.], [3., 4.], [5., 6.]]})
    sess.run(enqueue, feed_dict={a: 11, b:[[2., 4.], [6., 8.], [0., 2.]]})
    sess.run(enqueue, feed_dict={a: 12, b:[[3., 6.], [9., 2.], [5., 8.]]})
On the other end, the dequeue() function now creates a pair of dequeue operations:
dequeue_a, dequeue_b = q.dequeue()


dequeue_a, dequeue_b = q.dequeue()
In general, you should run these operations together:
with tf.Session([...]) as sess:
    a_val, b_val = sess.run([dequeue_a, dequeue_b])
    print(a_val) # 10
    print(b_val) # [[1., 2.], [3., 4.], [5., 6.]]
If you run dequeue_a on its own, it will dequeue a pair and return
only the first element; the second element will be lost (and simi‐
larly, if you run dequeue_b on its own, the first element will be
lost).
The dequeue_many() function also returns a pair of operations:
batch_size = 2
dequeue_as, dequeue_bs = q.dequeue_many(batch_size)
You can use it as you would expect:
with tf.Session([...]) as sess:
    a, b = sess.run([dequeue_a, dequeue_b])
    print(a) # [10, 11]
    print(b) # [[[1., 2.], [3., 4.], [5., 6.]], [[2., 4.], [6., 8.], [0., 2.]]]
    a, b = sess.run([dequeue_a, dequeue_b])  # blocked waiting for another pair
332
|
Chapter 12: Distributing TensorFlow Across Devices and Servers
Download from finelybook www.finelybook.com


Closing a queue
It is possible to close a queue to signal to the other sessions that no more data will be
enqueued:
close_q = q.close()
with tf.Session([...]) as sess:
    [...]
    sess.run(close_q)
Subsequent executions of enqueue or enqueue_many operations will raise an excep‐
tion. By default, any pending enqueue request will be honored, unless you call
q.close(cancel_pending_enqueues=True).
Subsequent executions of dequeue or dequeue_many operations will continue to suc‐
ceed as long as there are items in the queue, but they will fail when there are not
enough items left in the queue. If you are using a dequeue_many operation and there
are a few instances left in the queue, but fewer than the mini-batch size, they will be
lost. You may prefer to use a dequeue_up_to operation instead; it behaves exactly like
dequeue_many except when a queue is closed and there are fewer than batch_size
instances left in the queue, in which case it just returns them.
RandomShuffleQueue


instances left in the queue, in which case it just returns them.
RandomShuffleQueue
TensorFlow also supports a couple more types of queues, including RandomShuffle
Queue, which can be used just like a FIFOQueue except that items are dequeued in a
random order. This can be useful to shuffle training instances at each epoch during
training. First, let’s create the queue:
q = tf.RandomShuffleQueue(capacity=50, min_after_dequeue=10,
                          dtypes=[tf.float32], shapes=[()],
                          name="q", shared_name="shared_q")
The min_after_dequeue specifies the minimum number of items that must remain in
the queue after a dequeue operation. This ensures that there will be enough instances
in the queue to have enough randomness (once the queue is closed, the
min_after_dequeue limit is ignored). Now suppose that you enqueued 22 items in
this queue (floats 1. to 22.). Here is how you could dequeue them:
dequeue = q.dequeue_many(5)
with tf.Session([...]) as sess:


dequeue = q.dequeue_many(5)
with tf.Session([...]) as sess:
   print(sess.run(dequeue)) # [ 20.  15.  11.  12.   4.]   (17 items left)
   print(sess.run(dequeue)) # [  5.  13.   6.   0.  17.]   (12 items left)
   print(sess.run(dequeue)) # 12 - 5 < 10: blocked waiting for 3 more instances
Multiple Devices Across Multiple Servers
|
333
Download from finelybook www.finelybook.com


PaddingFifoQueue
A PaddingFIFOQueue can also be used just like a FIFOQueue except that it accepts ten‐
sors of variable sizes along any dimension (but with a fixed rank). When you are
dequeuing them with a dequeue_many or dequeue_up_to operation, each tensor is
padded with zeros along every variable dimension to make it the same size as the
largest tensor in the mini-batch. For example, you could enqueue 2D tensors (matri‐
ces) of arbitrary sizes:
q = tf.PaddingFIFOQueue(capacity=50, dtypes=[tf.float32], shapes=[(None, None)]
                        name="q", shared_name="shared_q")
v = tf.placeholder(tf.float32, shape=(None, None))
enqueue = q.enqueue([v])
with tf.Session([...]) as sess:
   sess.run(enqueue, feed_dict={v: [[1., 2.], [3., 4.], [5., 6.]]})       # 3x2
   sess.run(enqueue, feed_dict={v: [[1.]]})                               # 1x1
   sess.run(enqueue, feed_dict={v: [[7., 8., 9., 5.], [6., 7., 8., 9.]]}) # 2x4