python - How to use TensorFlow metrics in Keras

Question

There seem to be several threads/issues on this already but it doesn't appear to me that this has been solved:

How can I use tensorflow metric function within keras models?

https://github.com/fchollet/keras/issues/6050

https://github.com/fchollet/keras/issues/3230

People seem to either run into problems around variable initialization or the metric being 0.

I need to calculate different segmentation metrics and would like to include tf.metric.mean_iou in my Keras model. This is the best I have been able to come up with so far:

def mean_iou(y_true, y_pred):
   score, up_opt = tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)
   K.get_session().run(tf.local_variables_initializer())
   return score

model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=[mean_iou])

This code does not throw any errors but mean_iou always returns 0. I believe this is because up_opt is not evaluated. I have seen that prior to TF 1.3 people have suggested to use something along the lines of control_flow_ops.with_dependencies([up_opt], score) to achieve this. This does not seem possible in TF 1.3 anymore.

In summary, how do I evaluate TF 1.3 metrics in Keras 2.0.6? This seems like quite an important feature.

Answer 1

you can still usecontrol_dependencies

def mean_iou(y_true, y_pred):
   score, up_opt = tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)
   K.get_session().run(tf.local_variables_initializer())
   with tf.control_dependencies([up_opt]):
       score = tf.identity(score)
   return score