K8S pod內存告警分析

發布時間：2023-07-03 01:49:58 作者：網友整理

背景：

目前prometheus 給pod的內存告警閥值設置的85%，由于JVM 設置最高申請內存為pod limit 的75%，通過arthas-boot查看到堆內存和元空間占用內存之和跟prometheus告警值不同。

一、排查步驟：

1、prometheus 告警使用參數

使用container_memory_rss值進行告警

      - alert: Pod內存使用率
        expr: |
           sum(container_memory_rss{image!=""}) by(pod, namespace) / 
           sum(container_spec_memory_limit_bytes{image!=""}) by(pod, namespace) * 100 != +inf > 85
        for: 5m
        labels:
          severity: warning 
        annotations:
          summary: "命名空間: {{ $labels.namespace }} | Pod名稱: {{ $labels.pod }} 內存使用大于85% (當前值: {{ $value }})"

2、找到pod運行node節點

[root@pro-k8s-master ~]# kubectl -n msApp get pod -o wide|grep mayi
mayi-center-9d5d588c5-d7sgd     1/1     Running   0          7d     10.100.235.43   k8s-node1   <none>  <none>
mayi-center-9d5d588c5-mzd9p     1/1     Running   0          7d     10.100.6.41     k8s-node2   <none>  <none>

3、通過Docker state 查看容器資源

[root@k8s-node1 ~]# docker ps |grep mayi-center-9d5d588c5-d7sgd 
[root@pro-node1 ~]# docker ps |grep mayi-center-9d5d588c5-d7sgd 
b6b3733024c2   192.168.0.45/middleground-business-center/pro_mayi-center            "sh -c 'JAVA $JAVA_O…"   7 days ago          Up 7 days                               k8s_mayi-center_mayi-center-9d5d588c5-d7sgd_msapp_eac4e708-bd0a-483c-b5d4-734f95c9f1c7_0
5770fe14e7aa   registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1                 "/pause"                 7 days ago          Up 7 days                               k8s_POD_mayi-center-9d5d588c5-d7sgd_msapp_eac4e708-bd0a-483c-b5d4-734f95c9f1c7_0
[root@k8s-node1 ~]#docker state b6b3733024c2
CONTAINER ID        NAME                                                                                                         CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
b6b3733024c2        k8s_mayi-center_mayi-center-9d5d588c5-d7sgd_msapp_eac4e708-bd0a-483c-b5d4-734f95c9f1c7_0   4.31%               1.788GiB / 2GiB     89.40%              0B / 0B             1.52MB / 137MB      132

4、通過top -p 查看容器內存情況

[root@pro-k8s-node1 ~]# top -p 794890
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                       
794890 root      20   0 7483084   1.8g  15448 S   6.7  2.9 392:48.42 java

5、通過ps查看

[root@pro-k8s-node1 ~]# ps -ef|grep mayi
root     277280 257311  0 10:33 pts/0    00:00:00 grep --color=auto mayi
root     794890 794871  3 Sep08 ?        06:32:39 java -XX:+UseContainerSupport .....mayi-center ..... -jar /app.jar
[root@pro-k8s-node1 ~]# ps -e -o 'pid,comm,args,pcpu,rsz,vsz,stime,user,uid' | grep 794890
279183 grep            grep --color=auto 794890     0.0   960 112712 10:35 root         0
794890 java            java -XX:+UseContainerSuppo  3.8 1883224 7483084 Sep08 root      0