目錄
- 01 安裝 Nginx 和 ab 工具
- 1.1 安裝 nginx
- 1.2 啟動(dòng) Nginx 并測(cè)試
- 02 使用 Filebeat 采集日志并展示
- 2.1 配置 filebeat 采集 Nginx 日志
- 2.2 使用 Kibana 展示 Nginx 日志數(shù)據(jù)
- 03 采集 JSON 格式的 Nginx 日志
- 3.1 修改 Nginx 日志為 Json 格式
- 3.2 Filebeat 重新采集 Nginx 日志
- 3.3 自定義存儲(chǔ) Filebeat 采集日志的 ES 索引
- 04 使用 Filebeat 多節(jié)點(diǎn)采集 Nginx 日志數(shù)據(jù)
- 4.1 在其他節(jié)點(diǎn)上安裝 Filebeat 和 Nginx
- 4.2 在其他節(jié)點(diǎn)上配置 Filebeat 和 Nginx
- 4.3 在其他節(jié)點(diǎn)上啟動(dòng) Filebeat 和 Nginx
- 05 收集 Nginx 錯(cuò)誤日志
- 5.1 配置 Filebeat 采集 Nginx 錯(cuò)誤日志
- 5.2 配置 Filebeat 采集日志時(shí)拆分錯(cuò)誤日志和正常日志
- 06 Filebeat 采集 Nginx 日志的最終配置文件
01 安裝 Nginx 和 ab 工具
1.1 安裝 nginx
sudo apt-get install nginx -y # 安裝Nginx sudo apt-get install apache2-utils -y # Ubuntu安裝ab工具 sudo yum -y install httpd-tools 0y # CentOS安裝ab工具
在線安裝完成后,Nginx主要文件目錄構(gòu)成如下
/etc/nginx # 配置文件 /etc/nginx/sites-available # 虛擬主機(jī) /usr/sbin/nginx # 啟動(dòng)程序文件 /var/log/nginx # 日志目錄,包含access.log和error.log
1.2 啟動(dòng) Nginx 并測(cè)試
啟動(dòng)Nginx,并使用netstat
命令查看端口
systemctl start nginx # 啟動(dòng)nginx netstat -lntup|grep nginx # 查看nginx是否啟動(dòng)成功
使用壓力測(cè)試工具測(cè)試Nginx,其中-c
選項(xiàng)表示并發(fā)請(qǐng)求結(jié)果數(shù),-n
選項(xiàng)表示請(qǐng)求次數(shù)。下面命令表示進(jìn)行100次請(qǐng)求,10個(gè)并發(fā)請(qǐng)求壓力測(cè)試結(jié)果。另外,ab壓測(cè)工具的一個(gè)缺陷是需要在壓測(cè)URL后加上/
符號(hào)。
ab -c 10 -n 100 172.16.255.131/ ab -c 10 -n 100 172.16.255.131/test.html/
壓力測(cè)試完成之后,查看Nginx日志可以得到如下結(jié)果
root@master:/etc/nginx# tail -f /var/log/nginx/access.log 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" 172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3"
02 使用 Filebeat 采集日志并展示
2.1 配置 filebeat 采集 Nginx 日志
在/etc/filebeat/filebeat.yml
配置文件中對(duì)filebeat進(jìn)行配置,將nginx的日志路徑/var/log/nginx/access.log
添加到filebeat的輸入配置中
vim /etc/filebeat/filebeat.yml # 打開(kāi)filebeat配置文件 # 采集日志數(shù)據(jù)配置 filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log
2.2 使用 Kibana 展示 Nginx 日志數(shù)據(jù)
采集日志數(shù)據(jù)
配置好filebeat采集Nginx日志數(shù)據(jù)之后,在ES-head中可以看到如下日志內(nèi)容
創(chuàng)建日志數(shù)據(jù)索引
然后在服務(wù)器啟動(dòng)kibana并使用瀏覽器訪問(wèn)http://115.156.128.172:5601/
進(jìn)入kibana。在該頁(yè)面中選擇添加數(shù)據(jù)Add your data
然后,選擇創(chuàng)建索引create index pattern
在提示欄中,選擇對(duì)應(yīng)filebeat采集的日志數(shù)據(jù)創(chuàng)建索引,并選擇時(shí)間戳@timestamp
查看日志數(shù)據(jù)
完成索引創(chuàng)建之后,使用Discovery查看日志數(shù)據(jù)
在日志數(shù)據(jù)通過(guò)設(shè)置日志范圍和日志字段查看指定日志內(nèi)容,也可以通過(guò)全文搜索和添加過(guò)濾器的功能查看指定數(shù)據(jù)
03 采集 JSON 格式的 Nginx 日志
默認(rèn)情況下Nginx的日志數(shù)據(jù)是一條文本條目,日志條目中的字段無(wú)法拆分顯示。采用怎樣的方式可以將這種非結(jié)構(gòu)化的日志內(nèi)容轉(zhuǎn)化成格式化的內(nèi)容呢?
# 將如下日志條目 message:172.16.255.131 - - [28/Jul/2021:07:19:53 +0000] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" # 轉(zhuǎn)換成JSON半結(jié)構(gòu)化數(shù)據(jù)如下 { "IP Address": 172.16.255.131, "Time": [28/Jul/2021:07:19:53 +0000], "HTTP Request": GET / HTTP/1.0 200 612, "Agent": ApacheBench/2.3 }
3.1 修改 Nginx 日志為 Json 格式
一種方法是直接將Nginx產(chǎn)生的日志采用Json格式保存,編輯Nginx的配置文件/etc/nginx/nginx.conf
,添加日志保存樣式
vim /etc/nginx/nginx.conf # 找到http中的logging settings # 添加如下內(nèi)容 log_format log_json '{ "@time_local": "$time_local", ' '"remote_addr": "$remote_addr", ' '"referer": "$http_referer", ' '"request": "$request", ' '"status": $status, ' '"bytes": $body_bytes_sent, ' '"agent": "$http_user_agent", ' '"x_forwarded": "$http_x_forwarded_for", ' '"up_addr": "$upstream_addr",' '"up_host": "$upstream_http_host",' '"up_resp_time": "$upstream_response_time",' '"request_time": "$request_time"' ' }';
測(cè)試nginx配置文件修改是否有效,得到如下輸出說(shuō)明成功
root@master:/home/wang# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
清空Nginx原有的日志數(shù)據(jù),重新啟動(dòng)Nginx并使用ab工具重新對(duì)其進(jìn)行壓測(cè),產(chǎn)生新的日志數(shù)據(jù)
> /var/log/nginx/access.log # 清空Nginx原有的日志數(shù)據(jù) systemctl restart nginx # 重新啟動(dòng)Nginx ab -c 10 -n 100 172.16.255.131/ # 使用ab工具重新進(jìn)行壓測(cè)
查看Nginx日志,可以看到Nginx日志已經(jīng)被重新以Json格式存儲(chǔ)
root@master:/home/wang# tail -f /var/log/nginx/access.log { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" } { "@timestamp": "30/Jul/2021:03:17:09 +0000", "remote_addr": "172.16.255.131", "referer": "-", "request": "GET / HTTP/1.0", "status": 200, "bytes": 612, "agent": "ApacheBench/2.3", "x_forwarded": "-", "up_addr": "-","up_host": "-","up_resp_time": "-","request_time": "0.000" }
3.2 Filebeat 重新采集 Nginx 日志
配置Filebeat識(shí)別Json格式日志
修改Nginx保存日志格式為Json之后,還需要對(duì)采集日志的Filebeat進(jìn)行重新配置,如果不對(duì)其進(jìn)行配置識(shí)別Json格式日志,被采集的日志仍然會(huì)以文本條目的形式被采集。
# 打開(kāi)filebeat配置文件 vim /etc/filebeat/filebeat.yml # 添加配置內(nèi)容,配置內(nèi)容可以參考官方手冊(cè) - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/nginx/access.log # 添加如下三行配置內(nèi)容,識(shí)別Json格式日志文件,將日志條目以Json格式解析 json.keys_under_root: true json.overwrite_keys: true tags: ["access"]
清空Nginx日志,并重新啟動(dòng)filebeat;filebeat監(jiān)控Nginx日志后,采用ab壓測(cè)工具生成日志并采集。
> /var/log/nginx/access.log systemctl restart filebeat ab -c 10 -n 100 172.16.255.131/
采用ES-head查看采集的日志數(shù)據(jù)可以看到日志數(shù)據(jù)以Json格式保存在ES中
使用Kibana查看Json格式的日志條目
創(chuàng)建新的Kibana索引后,使用Discovery查看日志數(shù)據(jù),并可以通過(guò)日志條目級(jí)的字段到達(dá)更有效的日志分析目的
3.3 自定義存儲(chǔ) Filebeat 采集日志的 ES 索引
之前使用Filebeat采集Nginx日志都是采用默認(rèn)的索引創(chuàng)建方式形如filebeat-7.13.2-2021.07.30-000001
,為了更好的識(shí)別索引和擴(kuò)大日志采集的時(shí)間跨度,需要自定義存儲(chǔ)索引名稱。
自定義存儲(chǔ)索引通過(guò)配置Filebeat實(shí)現(xiàn),在Filebeat的配置文件中對(duì)輸出進(jìn)行配置如下:
# 打開(kāi)filebeat配置文件 vim /etc/filebeat/filebeat.yml # 添加配置內(nèi)容,配置內(nèi)容可以參考官方手冊(cè) # ---------------------------- Elasticsearch template setting ---------------------- setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false setup.template.name: "nginx" # 名字和index中的名字一致 setup.template.pattern: "nginx-*" setup.template.enabled: false setup.template.overwrite: true setup.ilm.enabled: false # ---------------------------- Elasticsearch Output ---------------------------- output.elasticsearch: # Array of hosts to connect to. hosts: ["172.16.255.131:9200"] # 添加如下五行內(nèi)容 # add index settings by wanghaihua at 2021.07.30 index: "nginx-%{[agent.version]}-%{+yyyy.MM}" # 限定日期形式{+yyyy.MM}
采用這種配置可以自定義索引樣式,并自定義Kibana中搜索的字段,刪除冗余的解析字段
04 使用 Filebeat 多節(jié)點(diǎn)采集 Nginx 日志數(shù)據(jù)
4.1 在其他節(jié)點(diǎn)上安裝 Filebeat 和 Nginx
從已經(jīng)安裝好的節(jié)點(diǎn)上將filebeat安裝包拷貝到其他節(jié)點(diǎn)上
# 拷貝安裝包 scp filebeat-7.13.2-amd64.deb wang@172.16.255.132:/opt/es/ scp filebeat-7.13.2-amd64.deb wang@172.16.255.139:/opt/es/ # 在其他節(jié)點(diǎn)上安裝filebeat cd /opt/es/ sudo dpkg -i filebeat-7.13.2-amd64.deb # 安裝deb包 # 安裝Nginx sudo apt-get install nginx -y
4.2 在其他節(jié)點(diǎn)上配置 Filebeat 和 Nginx
從已經(jīng)安裝好的節(jié)點(diǎn)上將filebeat的配置文件拷貝到其他節(jié)點(diǎn)上
# 拷貝filebeat配置文件到一個(gè)暫存目錄(直接拷貝到etc目錄下可能存在權(quán)限問(wèn)題) scp /etc/filebeat/filebeat.yml wang@172.16.255.132:/opt/es/ scp /etc/filebeat/filebeat.yml wang@172.16.255.139:/opt/es/ scp /etc/nginx/nginx.conf wang@172.16.255.132:/opt/es/ scp /etc/nginx/nginx.conf wang@172.16.255.139:/opt/es/ # 在對(duì)應(yīng)節(jié)點(diǎn)上將配置文件移動(dòng)到對(duì)應(yīng)目錄覆蓋原始配置文件 mv /opt/es/filebeat.yml /etc/filebeat/ mv /opt/es/nginx.conf /etc/nginx/ # 修改用戶權(quán)限 chown -R root:root /etc/nginx/nginx.conf chown -R root:root /etc/filebeat/filebeat.yml
4.3 在其他節(jié)點(diǎn)上啟動(dòng) Filebeat 和 Nginx
啟動(dòng)Filebeat和Nginx并使用master節(jié)點(diǎn)的ab工具進(jìn)行壓測(cè)產(chǎn)生日志數(shù)據(jù)
# 啟動(dòng)Filebeat和Nginx systemctl start nginx systemctl start filebeat # 使用master節(jié)點(diǎn)的ab工具進(jìn)行壓測(cè)產(chǎn)生日志數(shù)據(jù) ab -n 100 -c 20 http://172.16.255.132/node1.html ab -n 100 -c 20 http://172.16.255.139/node2.html # 查看產(chǎn)生的日志數(shù)據(jù)是否為Json格式 tail -f /var/log/nginx/access.log
Filebeat的配置文件將日志數(shù)據(jù)采集并存儲(chǔ)在ES中,多個(gè)節(jié)點(diǎn)的日志數(shù)據(jù)被聚合在一個(gè)ES索引中保存。
05 收集 Nginx 錯(cuò)誤日志
收集錯(cuò)誤日志的需求:要能夠區(qū)分錯(cuò)誤日志和正常日志,要能夠是使用單獨(dú)索引存儲(chǔ)錯(cuò)誤日志
5.1 配置 Filebeat 采集 Nginx 錯(cuò)誤日志
在filebeat配置文件etc/filebeat/filebeat.yml
的inputs
選項(xiàng)中添加如下內(nèi)容
- type: log enabled: true paths: - /var/log/nginx/error.log tags: ["error"]
5.2 配置 Filebeat 采集日志時(shí)拆分錯(cuò)誤日志和正常日志
在filebeat配置文件etc/filebeat/filebeat.yml
的輸入中加入tags
標(biāo)識(shí)采集的不同類型日志數(shù)據(jù),然后在索引設(shè)置中配置如下對(duì)日志進(jìn)行拆分
# 在`output`中配置通過(guò)tags區(qū)分日志 output.elasticsearch: hosts: ["172.16.255.131:9200"] indices: - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}" when.contains: tags: "access" - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM}" when.contains: tags: "error"
讓其他節(jié)點(diǎn)上采集Nginx日志的Filebeat的配置文件于上述配置一致,直接將該filebeat的配置文件拷貝到其他節(jié)點(diǎn)上覆蓋
# 拷貝filebeat配置文件到一個(gè)暫存目錄(直接拷貝到etc目錄下可能存在權(quán)限問(wèn)題) scp /etc/filebeat/filebeat.yml wang@172.16.255.132:/opt/es/ scp /etc/filebeat/filebeat.yml wang@172.16.255.139:/opt/es/ # 在對(duì)應(yīng)節(jié)點(diǎn)上將配置文件移動(dòng)到對(duì)應(yīng)目錄覆蓋原始配置文件 mv /opt/es/filebeat.yml /etc/filebeat/ # 修改用戶權(quán)限 chown -R root:root /etc/filebeat/filebeat.yml
06 Filebeat 采集 Nginx 日志的最終配置文件
# ============================== Filebeat inputs =============================== filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log json.keys_under_root: true json.overwrite_keys: true tags: ["access"] - type: log enabled: true paths: - /var/log/nginx/error.log tags: ["error"] # ======================= Elasticsearch template setting ======================= setup.template.settings: index.number_of_shards: 1 setup.template.name: "nginx" setup.template.pattern: "nginx-*" setup.template.enabled: false setup.template.overwrite: true setup.ilm.enabled: false # ================================== Outputs =========================== # ---------------------------- Elasticsearch Output ---------------------------- output.elasticsearch: hosts: ["172.16.255.131:9200"] indices: - index: "nginx-access-%{[agent.version]}-%{+yyyy.MM}" when.contains: tags: "access" - index: "nginx-error-%{[agent.version]}-%{+yyyy.MM}" when.contains: tags: "error" # ================================== Logging =================================== logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0644