Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
3.3k views
in Technique[技术] by (71.8m points)

prometheus-rules如何监控每一台机是否存在指定的值?

使用dcgm-exporter监控,打算监控每台机的显卡是否正常。

一开始是检查UP是否打开,发现出错的显卡机器状态也是UP

groups:
- name: dcgm
  rules:
  - alert: dcgm_down
    expr: up{job="gpu_worker"} == 0
    for: 5m
    labels:
      severity: 1
      team: node_down
    annotations:
      summary: "主机显卡报错!"
      value: '告警值:{{ $value }}'
      description: "{{ $labels.instance }} 检测到主机显卡报错, 请检查!!!"

想到的办法是监控指定的值,使用count统计有多少个值,再减去实际显卡机器。这个做法不是很好,有一台机器的显卡故障后,不清楚是那台机器,要每一台都检查下

    expr: 40 - count(DCGM_FI_DEV_MEM_CLOCK)  != 0 

目前告警只是提示有多少台故障,无法查看故障机器的IP
image

有什么办法可以检查每台是否存在指定的值或查看故障的IP

显卡机器正常显示的内容

DCGM_FI_DEV_MEM_CLOCK{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 5005
DCGM_FI_DEV_GPU_TEMP{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 40
DCGM_FI_DEV_PCIE_REPLAY_COUNTER{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 0
DCGM_FI_DEV_GPU_UTIL{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 42
DCGM_FI_DEV_MEM_COPY_UTIL{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 2
DCGM_FI_DEV_ENC_UTIL{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 0
DCGM_FI_DEV_DEC_UTIL{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 0
DCGM_FI_DEV_XID_ERRORS{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 31
DCGM_FI_DEV_POWER_VIOLATION{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 0
DCGM_FI_DEV_SYNC_BOOST_VIOLATION{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 0
DCGM_FI_DEV_FB_FREE{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 11176
DCGM_FI_DEV_FB_USED{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 0
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL{gpu="0", UUID="GPU-c38c0a66-4a28-634a-efe6-3021ccdb6d21", device="nvidia0"} 0

显卡机器故障显示的内容

DCGM_FI_DEV_XID_ERRORS{gpu="0", UUID="GPU-13f86750-22fd-89d9-aa3c-749442184ce5", device="nvidia0"} 45
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL{gpu="0", UUID="GPU-13f86750-22fd-89d9-aa3c-749442184ce5", device="nvidia0"} 0

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

已找到方法解决,expr表达式为:

expr:  up{job="gpu_worker"} == 1 UNLESS ON(instance)  DCGM_FI_DEV_SM_CLOCK

参考链接:prometheus - alerting missing metric for many hosts in alertmanager - Stack Overflow


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...