monitoring - How can I 'join' two metrics in a Prometheus query?

Question

I am using the consul exporter to ingest the health and status of my services into Prometheus. I'd like to fire alerts when the status of services and nodes in Consul is critical and then use tags extracted from Consul when routing those alerts.

I understand from this discussion that service tags are likely to be exported as a separate metric, but I'm not sure how to join one series with another so I can leverage the tags with the health status.

For example, the following query:

max(consul_health_service_status{status="critical"}) by (service_name, status,node) == 1

could return:

{node="app-server-02",service_name="app-server",status="critical"} 1

but I'd also like 'env' from this series:

consul_service_tags{node="app-server-02",service_name="app-server",env="prod"} 1

to get joined along node and service_name to pass the following to the Alertmanager as a single series:

{node="app-server-02",service_name="app-server",status="critical",env="prod"} 1

I could then match 'env' in my routing.

Is there any way to do this? It doesn't look to me like any operations or functions give me the ability to group or join like this. As far as I can see, the tags would already need to be labels on the consul_health_service_status metric.

question from:https://stackoverflow.com/questions/44461801/how-can-i-join-two-metrics-in-a-prometheus-query

Answer 1

You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

The important part here is the operation + on(service_name,node) group_left(env):

• the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
• group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)