Elasticstack官网:https://www.elastic.co本文档仅限搭建过程参考,使用相关的文档,不在本文档讨论范围之内。一切依据的核心即是Elasticstack官网。查看支持的操作系统:Elasticstack各版本软件支持的系统等:https://www.elastic.co/support/matrix#show_os必备软件:Elasticsearch:elasticsearch-5.3.0.tar.gzLogstash:logstash-5.3.0.tar.gzKibana:kibana-5.3.0-linux-x86_64.tar.gzBeats:filebeat-5.3.0-linux-x86_64.tar.gzX-pack:x-pack-5.3.0.zipjdk1.8:jdk-8u121-linux-i586.tar搭建流程:Elasticsearch:https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html修改时区(如必要):cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime安装JDK1.8,先安装yum install -y glibc.i686root权限下sysctl -w vm.max_map_count=262144/etc/security/limits.conf 增加 * soft nproc 2048 (第一列为Linux账户名)* hard nproc 4096(第一列为Linux账户)* soft nofile 65536* hard nofile 131072/etc/security/limits.d/90-nproc.conf 修改* soft nproc 2048sudo sysctl -p生产环境需要的配置Elasticsearch通过name判断所属的集群 ● path.data and path.logs ○ 不要和安装目录放一起,防止升级的时候数据丢失 ○ 可以配置多个目录 ● cluster.name ○ 各节点通过cluster.name加入集群 ● node.name ○ 保证重启机器名称不变 ○ 可以设置成机器名例如node.name:${HOSTNAME} ● bootstrap.memory_lock ○ bootstrap.memory_lock: true(遇到问题,见博客园博客) ○ /etc/sysctl.conf 增加vm.swappiness=0 ● network.host ○ 一旦配置标志进入生产环境 ● discovery.zen.ping.unicast.hosts ○ 自动入集群 ● discovery.zen.minimum_master_nodes ○ 避免脑裂生产环境需要设置bootstrap.memory_lock: true节点如下配置,承担的角色为负载均衡node.master: falsenode.data: falsenode.ingest: false配置文件elasticsearch.yml参考:cluster.name: rokid-testnode.name: node-41node.master: truenode.data: true path.data: /home/zhangzhenghai/elk/data/elasticsearch/datapath.logs: /home/zhangzhenghai/elk/data/elasticsearch/logsbootstrap.memory_lock: true bootstrap.system_call_filter: false network.host: test41http.port: 9200discovery.zen.ping.unicast.hosts: - test41 - test42 - test43discovery.zen.minimum_master_nodes: 2 action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*启动方式:bin/elasticsearch -d -p pid 停止方式:kill `cat pid`重启方式:kill `cat pid` bin/elasticsearch -d -p pid Logstash:参考https://www.elastic.co/guide/en/logstash/current/index.htmlgrok正则表达式参考:https://grokdebug.herokuapp.com/配置文件:input { beats { port => "5043" }}filter { grok { match => { "message" => "%{HOSTNAME:hostabc} %{DAY:zhouji} %{WORD:month} %{MONTHDAY:jihao} %{TIME:shijian} %{TZ:biaozhun} %{YEAR:nian} %{TIMESTAMP_ISO8601:shijianquan} \[%{WORD:zhonglei}\] %{WORD:caozuo} %{NOTSPACE:info}" } }}output { file { codec => line {format => "%{message}"} path => "/home/zhangzhenghai/elk/data/logstash/%{hostabc}.log" } elasticsearch { hosts => ["test41:9200","test42:9200","test43:9200"] index => "%{hostabc}" document_type => "%{hostabc}" flush_size => 100 idle_flush_time => 10 user => "elastic" password => "baoshan" }}上面的正则表达式对应的原始日志参考:test-41.dev.rokid-inc.com Tue Apr 25 14:54:36 CST 2017 2017-04-16 23:37:44,282 [DEBUG] add service:com.rokid.open.nlp.facade.NLPService启动方式(自动加载配置文件,修改配置文件不用重启Logstash):bin/logstash -f config/firtst.conf --config.reload.automaticKibana:配置文件参考:server.port: 5601server.host: "test43" elasticsearch.url: "http://test43:9200"elasticsearch.preserveHost: truekibana.index: ".kibana"kibana.defaultAppId: "discover"elasticsearch.username: "elastic"elasticsearch.password: "xxx"启动方式:bin/kibanaX-Packbin/elasticsearch-plugin install x-pack 或者线下安装方式bin/elasticsearch-plugin install file:///path/to/file/x-pack-5.3.1.zipbin/kibana-plugin install file:///path/to/file/x-pack-5.3.1.zip查看是否安装成功:bin/kibana-plugin listbin/elasticsearch-plugin list(如是elastic集群,需要将所有节点安装x-pack后重启,默认用户名密码elastic/changeme才生效)elasticsearch.yml配置如下信息action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*卸载x-packbin/elasticsearch-plugin remove x-packBeats(可直接写入Elasticseatch):filebeat-5.3.0-linux-x86_64.tar.gz配置文件参考:filebeat.prospectors:- input_type: log paths: - /home/zhangzhenghai/elk/data/filebeat/testdate.log document_type: typetest42output.elasticsearch: hosts: ["test41:9200","test42:9200","test43:9200"] index: "indextest42" username: "elastic" password: "xxx"output.logstash: hosts: ["test39:5043"]启动方式:./filebeat -e -c filebeat.yml -d "publish"可能短期内将会部署到线上,随着部署的进行,逐步更新完善本文档。(发现每当需要文档的时候,经常嫌弃写得少;当需要自己写文档的时候,总感觉没得写?OMG)
如更新忘记更新此文档,请参考链接:
http://note.youdao.com/noteshare?id=d9b4d5c0f5991c63c5b8ae965722f619