1.前言:
我們的監控系統是zabbix,最近剛做完jvm還有Tomcat的監控,需要吐槽以下內容:
- zabbix官方提供的jvm和tomcat監控模版,存在item書寫格式的坑,導致很多內容無法獲取到監控。
- 網友寫的zabbix監控JAVA文章,99%就只有環境的搭建,沒有zabbix item鍵值,所以環境監控上了,item鍵值不知道怎么寫。
本文監控jvm內容如下:
內存使用狀態:堆內存(Heap memory)和非堆內存(No Heap memory),包括已用值、最大值、已提交; 堆內存內存池:新生代(eden space),survivor space,老年代(old gen)的內存使用狀態; 非堆內存內存池:代碼緩存(Code cache),元空間(meta space),壓縮類空間(compressed class space); 類加載:加載總數,已加載,已卸載。 Java線程:總開啟線程,活動線程,線程峰值。
本文監控tomcat內容如下:
Tomcat請求數:包括每秒請求數,每秒出錯數; Tomcat網絡流量統計:包括進流量統計,出流量統計; Tomcat線程:包括最大線程數,當前線程數,當前繁忙線程數。
我沒有在zabbix中添加垃圾回收(gc)的監控,而是在catalina配置里面添加gc日輸入來給開發進行分析,設置方式如下。
CATALINA_OPTS=”-XX:ParallelGCThreads=4 -XX:+PrintGCDetails -Xloggc:日志存放路徑”
監控效果如下:
2.監控環境搭建
環境介紹:
[root@tomcat-01 ~]# /usr/local/tomcat/bin/version.sh Server version: Apache Tomcat/8.0.23 Server built: May 19 2015 14:58:38 UTC Server number: 8.0.23.0 OS Name: linux OS Version: 2.6.32-573.22.1.el6.x86_64 Architecture: amd64 JVM Version: 1.8.0_65-b17 JVM Vendor: Oracle Corporation
2.1.Server端配置
1、 Zabbix server端安裝java jdk環境并開啟javaGateway支持java監控。
javaGateway 啟用方式如下:
Zabbix通過rpm包安裝:只需要安裝zabbix-java-gateway包即可。
Zabbix通過編譯安裝:在編譯時需要加上—enable-java以支持jmx監控,如果之前沒加需要重新編譯。
2、修改zabbix-java-gateway配置文件
[root@zabbix ~]# vim /etc/zabbix/zabbix_java_gateway.conf LISTEN_IP="192.168.10.3" LISTEN_PORT=10052 PID_FILE="/var/run/zabbix/zabbix_java.pid" START_POLLERS=5 修改zabbix-server配置文件 [root@zabbix ~]# vim /etc/zabbix/zabbix_server.conf JavaGateway=192.168.10.3 JavaGatewayPort=10052 StartJavaPollers=5 啟動zabbix-java-gateway服務 [root@zabbix ~]# /etc/init.d/zabbix-java-gateway start
2.2.Tomcat服務器配置
1、下載catalina-jmx-remote.jar包,到tomcat安裝目錄下的lib目錄
wget -O /usr/local/tomcat/lib/catalina-jmx-remote.jar http://archive.apache.org/dist/tomcat/tomcat-8/v8.0.23/bin/extras/catalina-jmx-remote.jar
2、修改catalina.sh添加如下內容
CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
3、重啟tomcat與監控測試
(1)下載cmdline-jmxclient-0.10.3.jar文件,下載地址http://crawler.archive.org/cmdline-jmxclient/downloads.html
(2)本地執行如下命令查看tomcat的堆內存信息
[root@tomcat-01 ~]# java -jar /root/cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=Memory HeapMemoryUsage 11/04/2016 15:23:16 +0800 org.archive.jmx.Client HeapMemoryUsage: committed: 2146959360 init: 2147483648 max: 2146959360 used: 407611808
3.監控數據采集
3.1.堆內存
Tomcat本地查看堆內存信息:
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Memory HeapMemoryUsage 11/04/2016 15:36:58 +0800 org.archive.jmx.Client HeapMemoryUsage: committed: 2145910784 init: 2147483648 max: 2145910784 used: 741540536
zabbix監控堆內存鍵值:
堆內存最大值:jmx["java.lang:type=Memory","HeapMemoryUsage.max"] 已用堆內存:jmx["java.lang:type=Memory","HeapMemoryUsage.used"] 已提交堆內存:jmx["java.lang:type=Memory","HeapMemoryUsage.committed"]
一個完整的zabbix item填寫方式如下,不同內容填寫不同的鍵值即可:
3.2.內存池eden space:
Tomcat本地查看eden space:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=PS Eden Space Usage
zabbix監控eden區域鍵值:
最大空間:jmx["java.lang:type=MemoryPool,name=PS Eden Space",Usage.max] 已用空間:jmx["java.lang:type=MemoryPool,name=PS Eden Space",Usage.used] 提交空間:jmx["java.lang:type=MemoryPool,name=PS Eden Space",Usage.committed]
3.3.內存池survivor space:
Tomcat本地查看Survivor space區域:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=PS Survivor Space Usage
zabbix監控Survivor 鍵值:
jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.committed] jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.max] jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.used]
3.4.內存池old gen:
Tomcat本地查看old gen區域使用:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS Old Gen,type=MemoryPool Usage
zabbix監控old gen鍵值:
jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.committed] jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.max] jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.used]
3.5.非堆內存:
Tomcat 本地查看非堆內存使用:
java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Memory NonHeapMemoryUsag1 zabbix監控非堆內存使用 1jmx["java.lang:type=Memory","NonHeapMemoryUsag.committed"] jmx["java.lang:type=Memory","NonHeapMemoryUsag.used"]
3.6.內存池meta space:
Tomcat 本地查看meta space區域使用:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Metaspace Usage
zabbix監控mete space區域鍵值
jmx["java.lang:type=MemoryPool,name=Metaspace",Usage.committed] jmx["java.lang:type=MemoryPool,name=Metaspace",Usage.used]
3.7.內存池code cache:
Tomcat 本地查看code cache區域使用:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Code Cache Usage
zabbix監控code cache區域使用:
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.committed] jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.max] jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.used]
3.8.內存池compressed class space:
Tomcat 本地查看compressed class space區域使用:
java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Compressed Class Space Usage
zabbix監控 compressed class space區域使用鍵值:
jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.committed] jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.max] jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.used]
3.9.類加載:
Tomcat本地查看類加載信息:
加載總數: java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=ClassLoading TotalLoadedClassCoun 已加載: java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=ClassLoading LoadedClassCount 已卸載: java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=ClassLoading UnloadedClassCount
Zabbix監控類加載鍵值:
加載總數: jmx["java.lang:type=ClassLoading","TotalLoadedClassCount"] 已加載: jmx["java.lang:type=ClassLoading","LoadedClassCount"] 已卸載: jmx["java.lang:type=ClassLoading","UnloadedClassCount"]
3.10.java線程:
tomcat本地查看java線程:
總開啟線程: java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Threading TotalStartedThreadCount 活動線程: java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Threading PeakThreadCount 線程峰值: java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Threading PeakThreadCount
Zabbix監控java線程鍵值:
總開啟線程: jmx["java.lang:type=Threading","TotalStartedThreadCount"] 活動線程: jmx["java.lang:type=Threading","ThreadCount"] 線程峰值: jmx["java.lang:type=Threading","PeakThreadCount"]
3.11.tomcat線程:
本地查看tomcat線程信息:
最大線程:java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name="http-nio-8080",type=ThreadPool maxThreads 當前線程:java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name="http-nio-8080",type=ThreadPool currentThreadCount 繁忙線程:java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name="http-nio-8080",type=ThreadPool currentThreadsBusy
Zabbix監控tomcat線程鍵值:
最大線程:jmx["Catalina:type=ThreadPool,name="http-nio-8080"",maxThreads] 當前線程:jmx["Catalina:type=ThreadPool,name="http-nio-8080"",currentThreadCount] 繁忙線程 jmx["Catalina:type=ThreadPool,name="http-nio-8080"",currentThreadsBusy]
3.12.網絡流量:
Tomcat本地查看接收的字節:
接收的字節:java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name="http-nio-8080",type=GlobalRequestProcessor bytesReceived 發送的字節:java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name="http-nio-8080",type=GlobalRequestProcessor bytesSent
Zabbix監控tomcat接受字節鍵值:
接收的字節:jmx["Catalina:type=GlobalRequestProcessor,name="http-nio-8080"",bytesReceived] 發送的字節:jmx["Catalina:type=GlobalRequestProcessor,name="http-nio-8080"",bytesSent]
3.13.tomcat請求,出錯請求:
tomcat本地查看tomcat請求數:
tomcat請求數: java -jar cmdline-jmxclient-0.10.3.jar - 192.168.10.46:8090 Catalina:name="http-nio-8080",type=GlobalRequestProcessor requestCount tomcat出錯請求: java -jar cmdline-jmxclient-0.10.3.jar - 192.168.10.46:8090 Catalina:name="http-nio-8080",type=GlobalRequestProcessor errorCount
zabbix監控tomcat請求數:
tomcat請求數: jmx["Catalina:type=GlobalRequestProcessor,name="http-nio-8080"",requestCount] tomcat出錯請求:jmx["Catalina:type=GlobalRequestProcessor,name="http-nio-8080"",errorCount]
4.問題的解答
最近有網友聯系我說,看著我的博文搭建完了環境,獲取不到數據,我幫助排查了一下,找到了問題所在,特此記錄。
網友的報錯如下:
# java -jar /root/cmdline-jmxclient-0.10.3.jar - 127.0.0.1:9080 java.lang:type=MemoryPool,name=PS Eden Space Usag 11/11/2016 10:03:37 +0800 org.archive.jmx.Client java.lang:name=PS Eden Space,type=MemoryPool is not a registered bean
4.1.解決思路
- 如果你使用命令行監控獲取不到數據,那么就先使用jconsole看看有沒有數據,
- 如果jconsole有數據,那么就往下看,反之檢查你的環境。
- 如果jconsole有數據,但是命令行沒有數據,那么就是Mbean的Object Name或者屬性有問題,網友都是復制我的,但是自己本地環境和我的環境不一樣所以導致獲取不到數據,查詢自己的本地Mbean方法有兩種,分別是圖形和命令行,網友的報錯說事自己的Eden Space空間有問題,那么我就貼出來查看本地內存池的Object Name和屬性的方法。
(1)通過jconsole查看:
(2)通過命令行查看:直接使用java -jar cmdline-jmxclient-0.10.3.jar – 127.0.0.1:8090命令可以獲取所有的Mbean信息,輸出太多這里就不貼輸出結果了。我通過grep命令獲取所有內存池的監控信息方式如下。
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 | grep MemoryPool java.lang:name=Compressed Class Space,type=MemoryPool java.lang:name=Metaspace,type=MemoryPool java.lang:name=PS Old Gen,type=MemoryPool java.lang:name=PS Eden Space,type=MemoryPool java.lang:name=PS Survivor Space,type=MemoryPool java.lang:name=Code Cache,type=MemoryPool
然后我要獲取Eden Space的所有屬性信息方式如下:(一些特殊字符需要轉義)
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS Eden Space,type=MemoryPool Attributes: Usage: Usage (type=javax.management.openmbean.CompositeData) PeakUsage: PeakUsage (type=javax.management.openmbean.CompositeData) MemoryManagerNames: MemoryManagerNames (type=[Ljava.lang.String;) UsageThreshold: UsageThreshold (type=long) UsageThresholdExceeded: UsageThresholdExceeded (type=boolean) UsageThresholdCount: UsageThresholdCount (type=long) UsageThresholdSupported: UsageThresholdSupported (type=boolean) CollectionUsageThreshold: CollectionUsageThreshold (type=long) CollectionUsageThresholdExceeded: CollectionUsageThresholdExceeded (type=boolean) CollectionUsageThresholdCount: CollectionUsageThresholdCount (type=long) CollectionUsage: CollectionUsage (type=javax.management.openmbean.CompositeData) CollectionUsageThresholdSupported: CollectionUsageThresholdSupported (type=boolean) Valid: Valid (type=boolean) Name: Name (type=java.lang.String) Type: Type (type=java.lang.String) ObjectName: ObjectName (type=javax.management.ObjectName) Operations: resetPeakUsage: resetPeakUsage Parameters 0, return type=void
那么我們在查看Eden Space空間的使用信息。就看到已經出來數據了
[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS Eden Space,type=MemoryPool Usage 12/03/2016 08:33:58 +0800 org.archive.jmx.Client Usage: committed: 712507392 init: 537395200 max: 712507392 used: 396006304