python - What is "metrics" in Keras?

Question

It is not yet clear for me what metrics are (as given in the code below). What exactly are they evaluating? Why do we need to define them in the model? Why we can have multiple metrics in one model? And more importantly what is the mechanics behind all this? Any scientific reference is also appreciated.

model.compile(loss='mean_squared_error',
              optimizer='sgd',
              metrics=['mae', 'acc'])

Answer 1

So in order to understand what metrics are, it's good to start by understanding what a loss function is. Neural networks are mostly trained using gradient methods by an iterative process of decreasing a loss function.

A loss is designed to have two crucial properties - first, the smaller its value is, the better your model fits your data, and second, it should be differentiable. So, knowing this, we could fully define what a metric is: it's a function that, given predicted values and ground truth values from examples, provides you with a scalar measure of a "fitness" of your model, to the data you have. So, as you may see, a loss function is a metric, but the opposite doesn't always hold. To understand these differences, let's look at the most common examples of metrics usage:

1. Measure a performance of your network using non-differentiable functions: e.g. accuracy is not differentiable (not even continuous) so you cannot directly optimize your network w.r.t. to it. However, you could use it in order to choose the model with the best accuracy.

2. Obtain values of different loss functions when your final loss is a combination of a few of them: Let's assume that your loss has a regularization term which measures how your weights differ from 0, and a term which measures the fitness of your model. In this case, you could use metrics in order to have a separate track of how the fitness of your model changes across epochs.

3. Track a measure with respect to which you don't want to directly optimize your model: so - let's assume that you are solving a multidimensional regression problem where you are mostly concerned about mse, but at the same time you are interested in how a cosine-distance of your solution is changing in time. Then, it's the best to use metrics.

I hope that the explanation presented above made obvious what metrics are used for, and why you could use multiple metrics in one model. So now, let's say a few words about mechanics of their usage in keras. There are two ways of computing them while training:

1. Using metrics defined while compilation: this is what you directly asked. In this case, keras is defining a separate tensor for each metric you defined, to have it computed while training. This usually makes computation faster, but this comes at a cost of additional compilations, and the fact that metrics should be defined in terms of keras.backend functions.

2. Using keras.callback: It is nice that you can use Callbacks in order to compute your metrics. As each callback has a default attribute of model, you could compute a variety of metrics using model.predict or model parameters while training. Moreover, it makes it possible to compute it, not only epoch-wise, but also batch-wise, or training-wise. This comes at a cost of slower computations, and more complicated logic - as you need to define metrics on your own.

Here you can find a list of available metrics, as well as an example on how you could define your own.