| 注册
请输入搜索内容

热门搜索

Java Linux MySQL PHP JavaScript Hibernate jQuery Nginx
jopen
9年前发布

使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践

原文  http://wsgzao.github.io/post/elk/

使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践

前言

Elasticsearch + Logstash + Kibana(ELK)是一套开源的日志管理方案,分析网站的访问情况时我们一般会借助Google/百度/CNZZ等方式嵌入JS做数据统计,但是当网 站访问异常或者被攻击时我们需要在后台分析如Nginx的具体日志,而Nginx日志分割/GoAccess/Awstats都是相对简单的单节点解决方 案,针对分布式集群或者数据量级较大时会显得心有余而力不足,而ELK的出现可以使我们从容面对新的挑战。

  • Logstash:负责日志的收集,处理和储存
  • Elasticsearch:负责日志检索和分析
  • Kibana:负责日志的可视化

ELK(Elasticsearch + Logstash + Kibana)

更新记录

2015年08月31日 - 初稿

阅读原文 - http://wsgzao.github.io/post/elk/

扩展阅读

CentOS 7.x安装ELK(Elasticsearch+Logstash+Kibana) - http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/

Centos 6.5 安装nginx日志分析系统 elasticsearch + logstash + redis + kibana - http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582

logstash-forwarder and grok examples - https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413/

三斗室 - http://chenlinux.com/

elastic - https://www.elastic.co/guide

LTMP索引 - http://wsgzao.github.io/index/#LTMP

组件预览

JDK - http://www.oracle.com/technetwork/java/javase/downloads/index.html

Elasticsearch - https://www.elastic.co/downloads/elasticsearch

Logstash - https://www.elastic.co/downloads/logstash

Kibana - https://www.elastic.co/downloads/kibana

redis - http://redis.io/download

设置FQDN

创建SSL证书的时候需要配置FQDN

#修改hostname  cat /etc/hostname  elk    #修改hosts  cat /etc/hosts    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4  ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6    127.0.0.1 10-10-87-19  10.10.87.19 elk.ooxx.com elk    #刷新环境  hostname -F /etc/hostname    #复查结果  hostname -f  elk.ooxx.com    hostname  elk

服务端

Java

cat /etc/redhat-release  CentOS release 6.5 (Final)    yum install java-1.7.0-openjdk  java -version    java version "1.7.0_85"  OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)  OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)    ···    ### Elasticsearch    ``` bash    #下载安装  wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm  yum localinstall elasticsearch-1.7.1.noarch.rpm     #启动相关服务  service elasticsearch start  service elasticsearch status    #查看Elasticsearch的配置文件  rpm -qc elasticsearch    /etc/elasticsearch/elasticsearch.yml  /etc/elasticsearch/logging.yml  /etc/init.d/elasticsearch  /etc/sysconfig/elasticsearch  /usr/lib/sysctl.d/elasticsearch.conf  /usr/lib/systemd/system/elasticsearch.service  /usr/lib/tmpfiles.d/elasticsearch.conf    #查看端口使用情况  netstat -nltp    Active Internet connections (only servers)  Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name     tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java             tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java             tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd             tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd      #测试访问  curl -X GET http://localhost:9200/

Kibana

#下载tar包  wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz  #解压  tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/  cd /usr/local/  mv kibana-4.1.1-linux-x64 kibana    #创建kibana服务  vi /etc/rc.d/init.d/kibana    #!/bin/bash  ### BEGIN INIT INFO  # Provides:          kibana  # Default-Start:     2 3 4 5  # Default-Stop:      0 1 6  # Short-Description: Runs kibana daemon  # Description: Runs the kibana daemon as a non-root user  ### END INIT INFO    # Process name  NAME=kibana  DESC="Kibana4"  PROG="/etc/init.d/kibana"    # Configure location of Kibana bin  KIBANA_BIN=/usr/local/kibana/bin    # PID Info  PID_FOLDER=/var/run/kibana/  PID_FILE=/var/run/kibana/$NAME.pid  LOCK_FILE=/var/lock/subsys/$NAME  PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN  DAEMON=$KIBANA_BIN/$NAME    # Configure User to run daemon process  DAEMON_USER=root  # Configure logging location  KIBANA_LOG=/var/log/kibana.log    # Begin Script  RETVAL=0    if [ `id -u` -ne 0 ]; then          echo "You need root privileges to run this script"          exit 1  fi    # Function library  . /etc/init.d/functions     start() {          echo -n "Starting $DESC : "    pid=`pidofproc -p $PID_FILE kibana`          if [ -n "$pid" ] ; then                  echo "Already running."                  exit 0          else          # Start Daemon  if [ ! -d "$PID_FOLDER" ] ; then                          mkdir $PID_FOLDER                  fi  daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &                  sleep 2                  pidofproc node > $PID_FILE                  RETVAL=$?                  [[ $? -eq 0 ]] && success || failure  echo                  [ $RETVAL = 0 ] && touch $LOCK_FILE                  return $RETVAL          fi  }    reload()  {      echo "Reload command is not implemented for this service."      return $RETVAL  }    stop() {          echo -n "Stopping $DESC : "          killproc -p $PID_FILE $DAEMON          RETVAL=$?  echo          [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE  }     case "$1" in    start)          start  ;;    stop)          stop          ;;    status)          status -p $PID_FILE $DAEMON          RETVAL=$?          ;;    restart)          stop          start          ;;    reload)  reload  ;;    *)  # Invalid Arguments, print the following message.          echo "Usage: $0 {start|stop|status|restart}" >&2  exit 2          ;;  esac    #修改启动权限  chmod +x /etc/rc.d/init.d/kibana    #启动kibana服务  service kibana start  service kibana status    #查看端口  netstat -nltp    Active Internet connections (only servers)  Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name     tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java             tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java             tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd             tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node             tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd

Logstash

#下载rpm包  wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm  #安装  yum localinstall logstash-1.5.4-1.noarch.rpm     #设置ssl,之前设置的FQDN是elk.ooxx.com  cd /etc/pki/tls  #openssl req -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=logstash.example.com  openssl req -subj '/CN=elk.ooxx.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt       #创建一个01-logstash-initial.conf文件  cat > /etc/logstash/conf.d/01-logstash-initial.conf << EOF  input {    lumberjack {      port => 5000      type => "logs"      ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"      ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"    }  }      filter {    if [type] == "syslog" {      grok {        match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }        add_field => [ "received_at", "%{@timestamp}" ]        add_field => [ "received_from", "%{host}" ]      }      syslog_pri { }      date {        match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]      }    }  }    output {    elasticsearch { host => localhost }    stdout { codec => rubydebug }  }  EOF    #启动logstash服务  service logstash start  service logstash status    #查看5000端口  netstat -nltp    Active Internet connections (only servers)  Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name     tcp        0      0 0.0.0.0:9200                0.0.0.0:*                   LISTEN      1765/java             tcp        0      0 0.0.0.0:9300                0.0.0.0:*                   LISTEN      1765/java             tcp        0      0 0.0.0.0:9301                0.0.0.0:*                   LISTEN      2309/java             tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1509/sshd             tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      1876/node             tcp        0      0 0.0.0.0:5000                0.0.0.0:*                   LISTEN      2309/java             tcp        0      0 :::22                       :::*                        LISTEN      1509/sshd        #启动服务  service logstash-forwarder start  service logstash-forwarder status    #访问Kibana,Time-field name 选择 @timestamp  http://localhost:5601/    #增加节点和客户端配置一样,注意同步证书  /etc/pki/tls/certs/logstash-forwarder.crt

客户端

Logstash Forwarder

#登陆到客户端,安装Logstash Forwarder  wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm  yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm    #查看logstash-forwarder的配置文件位置  rpm -qc logstash-forwarder  /etc/logstash-forwarder.conf    #备份配置文件  cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save    #编辑 /etc/logstash-forwarder.conf,需要根据实际情况进行修改    cat > /etc/logstash-forwarder.conf << EOF  {    "network": {      "servers": [ "elk.ooxx.com:5000" ],        "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",        "timeout": 15    },      "files": [      {        "paths": [          "/var/log/messages",          "/var/log/secure"        ],          "fields": { "type": "syslog" }      }    ]  }  EOF

配置Nginx日志策略

#修改客户端配置  vi /etc/logstash-forwarder.conf    {    "network": {      "servers": [ "elk.ooxx.com:5000" ],        "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",        "timeout": 15    },      "files": [      {        "paths": [          "/var/log/messages",          "/var/log/secure"        ],        "fields": { "type": "syslog" }      }, {        "paths": [          "/app/local/nginx/logs/access.log"        ],        "fields": { "type": "nginx" }      }    ]  }    #服务端增加patterns  mkdir /opt/logstash/patterns  vi /opt/logstash/patterns/nginx    NGUSERNAME [a-zA-Z\.\@\-\+_%]+  NGUSER %{NGUSERNAME}  NGINXACCESS %{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}    #官网pattern的debug在线工具  https://grokdebug.herokuapp.com/    #修改logstash权限  chown -R logstash:logstash /opt/logstash/patterns    #修改服务端配置  vi /etc/logstash/conf.d/01-logstash-initial.conf     input {    lumberjack {      port => 5000      type => "logs"      ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"      ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"    }  }      filter {    if [type] == "syslog" {      grok {        match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }        add_field => [ "received_at", "%{@timestamp}" ]        add_field => [ "received_from", "%{host}" ]      }      syslog_pri { }      date {        match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]      }    }    if [type] == "nginx" {      grok {         match => { "message" => "%{NGINXACCESS}" }      }    }  }    output {    elasticsearch { host => localhost }    stdout { codec => rubydebug }  }

其它注意事项

修改kibana端口

#编辑kibana.yaml  vi /usr/local/kibana/config/kibana.yml      # Kibana is served by a back end server. This controls which port to use.  #port: 5601  port: 80    # The host to bind the server to.  host: "0.0.0.0"    # The Elasticsearch instance to use for all your queries.  elasticsearch_url: "http://localhost:9200"    # preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,  # then the host you use to connect to *this* Kibana instance will be sent.  elasticsearch_preserve_host: true    # Kibana uses an index in Elasticsearch to store saved searches, visualizations  # and dashboards. It will create a new index if it doesn't already exist.  kibana_index: ".kibana"    # If your Elasticsearch is protected with basic auth, this is the user credentials  # used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana  # users will still need to authenticate with Elasticsearch (which is proxied thorugh  # the Kibana server)  # kibana_elasticsearch_username: user  # kibana_elasticsearch_password: pass    # If your Elasticsearch requires client certificate and key  # kibana_elasticsearch_client_crt: /path/to/your/client.crt  # kibana_elasticsearch_client_key: /path/to/your/client.key    # If you need to provide a CA certificate for your Elasticsarech instance, put  # the path of the pem file here.  # ca: /path/to/your/CA.pem    # The default application to load.  default_app_id: "discover"    # Time in milliseconds to wait for elasticsearch to respond to pings, defaults to  # request_timeout setting  # ping_timeout: 1500    # Time in milliseconds to wait for responses from the back end or elasticsearch.  # This must be > 0  request_timeout: 300000    # Time in milliseconds for Elasticsearch to wait for responses from shards.  # Set to 0 to disable.  shard_timeout: 0    # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying  # startup_timeout: 5000    # Set to false to have a complete disregard for the validity of the SSL  # certificate.  verify_ssl: true    # SSL for outgoing requests from the Kibana Server (PEM formatted)  # ssl_key_file: /path/to/your/server.key  # ssl_cert_file: /path/to/your/server.crt    # Set the path to where you would like the process id file to be created.  # pid_file: /var/run/kibana.pid    # If you would like to send the log output to a file you can set the path below.  # This will also turn off the STDOUT log output.  # log_file: ./kibana.log    # Plugins that are included in the build, and no longer found in the plugins/ folder  bundled_plugin_ids:   - plugins/dashboard/index   - plugins/discover/index   - plugins/doc/index   - plugins/kibana/index   - plugins/markdown_vis/index   - plugins/metric_vis/index   - plugins/settings/index   - plugins/table_vis/index   - plugins/vis_types/index   - plugins/visualize/index

JVM调优

#修改elasticsearch.in.sh  vi /usr/share/elasticsearch/bin/elasticsearch.in.sh    if [ "x$ES_MIN_MEM" = "x" ]; then      ES_MIN_MEM=1g  fi  if [ "x$ES_MAX_MEM" = "x" ]; then      ES_MAX_MEM=1g

 本文由用户 jopen 自行上传分享,仅供网友学习交流。所有权归原作者,若您的权利被侵害,请联系管理员。
 转载本站原创文章,请注明出处,并保留原始链接、图片水印。
 本站是一个以用户分享为主的开源技术平台,欢迎各类分享!
 本文地址:https://www.open-open.com/lib/view/open1441088477784.html
ElasticSearch 搜索引擎