|
|
--- |
|
|
title: Loss |
|
|
--- |
|
|
# Loss |
|
|
|
|
|
In Caffe, as in most of machine learning, learning is driven by a **loss** function (also known as an **error**, **cost**, or **objective** function). |
|
|
A loss function specifies the goal of learning by mapping parameter settings (i.e., the current network weights) to a scalar value specifying the "badness" of these parameter settings. |
|
|
Hence, the goal of learning is to find a setting of the weights that *minimizes* the loss function. |
|
|
|
|
|
The loss in Caffe is computed by the Forward pass of the network. |
|
|
Each layer takes a set of input (`bottom`) blobs and produces a set of output (`top`) blobs. |
|
|
Some of these layers' outputs may be used in the loss function. |
|
|
A typical choice of loss function for one-versus-all classification tasks is the `SoftmaxWithLoss` function, used in a network definition as follows, for example: |
|
|
|
|
|
layer { |
|
|
name: "loss" |
|
|
type: "SoftmaxWithLoss" |
|
|
bottom: "pred" |
|
|
bottom: "label" |
|
|
top: "loss" |
|
|
} |
|
|
|
|
|
In a `SoftmaxWithLoss` function, the `top` blob is a scalar (empty shape) which averages the loss (computed from predicted labels `pred` and actuals labels `label`) over the entire mini-batch. |
|
|
|
|
|
### Loss weights |
|
|
|
|
|
For nets with multiple layers producing a loss (e.g., a network that both classifies the input using a `SoftmaxWithLoss` layer and reconstructs it using a `EuclideanLoss` layer), *loss weights* can be used to specify their relative importance. |
|
|
|
|
|
By convention, Caffe layer types with the suffix `Loss` contribute to the loss function, but other layers are assumed to be purely used for intermediate computations. |
|
|
However, any layer can be used as a loss by adding a field `loss_weight: <float>` to a layer definition for each `top` blob produced by the layer. |
|
|
Layers with the suffix `Loss` have an implicit `loss_weight: 1` for the first `top` blob (and `loss_weight: 0` for any additional `top`s); other layers have an implicit `loss_weight: 0` for all `top`s. |
|
|
So, the above `SoftmaxWithLoss` layer could be equivalently written as: |
|
|
|
|
|
layer { |
|
|
name: "loss" |
|
|
type: "SoftmaxWithLoss" |
|
|
bottom: "pred" |
|
|
bottom: "label" |
|
|
top: "loss" |
|
|
loss_weight: 1 |
|
|
} |
|
|
|
|
|
However, *any* layer able to backpropagate may be given a non-zero `loss_weight`, allowing one to, for example, regularize the activations produced by some intermediate layer(s) of the network if desired. |
|
|
For non-singleton outputs with an associated non-zero loss, the loss is computed simply by summing over all entries of the blob. |
|
|
|
|
|
The final loss in Caffe, then, is computed by summing the total weighted loss over the network, as in the following pseudo-code: |
|
|
|
|
|
loss := 0 |
|
|
for layer in layers: |
|
|
for top, loss_weight in layer.tops, layer.loss_weights: |
|
|
loss += loss_weight * sum(top) |
|
|
|
