以nginx錯(cuò)誤日志為例，演示日志處理流程

站在用戶的角度思考問題，與客戶深入溝通，找到石首網(wǎng)站設(shè)計(jì)與石首網(wǎng)站推廣的解決方案，憑借多年的經(jīng)驗(yàn)，讓設(shè)計(jì)與互聯(lián)網(wǎng)技術(shù)結(jié)合，創(chuàng)造個(gè)性化、用戶體驗(yàn)好的作品，建站類型包括：網(wǎng)站制作、成都網(wǎng)站建設(shè)、企業(yè)官網(wǎng)、英文網(wǎng)站、手機(jī)端網(wǎng)站、網(wǎng)站推廣、國際域名空間、雅安服務(wù)器托管、企業(yè)郵箱。業(yè)務(wù)覆蓋石首地區(qū)。

filebeat--logstash--es

filebeat--kafka--logstash--es

#filebeat使用systemd管理
/usr/lib/systemd/system/filebeat.service?
[Unit]
Description=Filebeat
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/usr/local/filebeat/filebeat?-c?/usr/local/filebeat/filebeat.yml
Restart=always
[Install]
WantedBy=multi-user.target

#logstash使用systemd管理
#如果有多個(gè)logstash配置文件,可以使用-f指定目錄
/usr/lib/systemd/system/logstash.service?
[Unit]
Description=logstash
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
Environment=JAVA_HOME=/usr/java/jdk1.8.0_211
ExecStart=/usr/local/logstash/bin/logstash?-f?/usr/local/logstash/config/logstash.conf?-l?/usr/local/logstash/logs
Restart=always
[Install]
WantedBy=multi-user.target

#啟動(dòng)nginx容器，映射日志目錄
docker?run?-d?--name=nginx?--net=host?-v?/tmp/nginx_log:/var/log/nginx?nginx

#nginx錯(cuò)誤日志：

2019/09/21 17:00:08 [error] 7#7: *9 open() "/usr/share/nginx/html/api" failed (2: No such file or directory), client: 192.168.3.102, server: localhost, request: "GET /api HTTP/1.1", host: "192.168.3.100"

• filebeat--logstash--es示例

#filebeat輸出logstash示例
/usr/local/filebeat/filebeat.yml?
filebeat.inputs:
-?type:?log
??paths:
????-?/tmp/nginx_log/error.log
??
??multiline.pattern:?^\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2}
??#匹配nginx日志時(shí)間格式??2019/09/21?17:00:08
??multiline.negate:?true
??multiline.match:?after
??exclude_files:?[".gz$"]
??tail_files:?true
??
??#增加輸出字段，tags為數(shù)組形式，fields.id為鍵值對(duì)形式
??tags:?["nginx-100"]
??fields:
????id:?"nginx-100"
output.logstash:
??hosts:?["192.168.3.100:5044","192.168.3.101:5044"]
??loadbalance:?true

#輸出到單個(gè)logstash
#output.logstash:
#??hosts:?["127.0.0.1:5044"]

#logstash輸出到es示例；根據(jù)fileds.id來劃分索引
/usr/local/logstash/config/logstash.conf?
input?{
??beats?{
????port?=>?5044
??}
}
output?{
??elasticsearch?{
????hosts?=>?["http://192.168.3.100:9200","http://192.168.3.101:9200","http://192.168.3.102:9200"]
????index?=>?"%{[fields][id]}-%{+YYYY.MM.dd}"
????user?=>?"elastic"
????password?=>?"HkqZIHZsuXSv6B5OwqJ7"
??}
}

• filebeat--kafka--logstash--es示例

#filebeat輸出到kafka示例
/usr/local/filebeat/filebeat.yml?
filebeat.inputs:
-?type:?log
??paths:
????-?/tmp/nginx_log/error.log
??
??multiline.pattern:?^\d{4}/\d{2}/\d{2}\s\d{2}:\d{2}:\d{2}
??#匹配nginx日志時(shí)間格式??2019/09/21?17:00:08
??multiline.negate:?true
??multiline.match:?after
??exclude_files:?[".gz$"]
??tail_files:?true
??
??#增加輸出字段，tags為數(shù)組形式，fields.id為鍵值對(duì)形式
??tags:?["nginx-kafka-100"]
??fields:
????id:?"nginx-kafka-100"
output.kafka:
??hosts:?["192.168.3.100:9092",?"192.168.3.101:9092",?"192.168.3.102:9092"]

??topic:?'%{[fields.id]}'
??partition.round_robin:
????reachable_only:?false

??required_acks:?1
??compression:?gzip
??max_message_bytes:?1000000

#kafka輸出到es示例
/usr/local/logstash/config/logstash.conf
input?{
????kafka?{
??????group_id?=>?"logstash"
??????topics?=>?["nginx-kafka-100"]
??????bootstrap_servers?=>?"192.168.3.100:9092,192.168.3.101:9092,192.168.3.102:9092"
??????consumer_threads?=>?"1"
??????fetch_max_bytes?=>?"26214400"
??????codec?=>?plain
??}
}
filter?{
??json?{
????source?=>?"message"
??}
}
output?{
??elasticsearch?{
????hosts?=>?["http://192.168.3.100:9200","http://192.168.3.101:9200","http://192.168.3.102:9200"]
????index?=>?"%{[fields][id]}-%{+YYYY.MM.dd}"
????user?=>?"elastic"
????password?=>?"HkqZIHZsuXSv6B5OwqJ7"
??}
}

參考：

https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html

https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

文章名稱：filebeat日志收集
本文URL：http://vcdvsql.cn/article36/gjccpg.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供App開發(fā)、、網(wǎng)站內(nèi)鏈、網(wǎng)站改版、虛擬主機(jī)、服務(wù)器托管

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來源： 創(chuàng)新互聯(lián)