墨 痕

fatesai#gmail.com

OPS

目录
  1. 1. 软件版本列表
  2. 2. Kafka
    1. 2.1. 安装
    2. 2.2. 配置
      1. 2.2.1. zookeeper配置
      2. 2.2.2. Kafka配置
      3. 2.2.3. kafka常用操作
      4. 2.2.4. 测试Kafka
  3. 3. Elasticsearch
    1. 3.1. 安装
    2. 3.2. 配置
      1. 3.2.1. 文件打开数配置
      2. 3.2.2. ES集群配置
      3. 3.2.3. ES内存设置
    3. 3.3. Elasticsearch插件安装
  4. 4. Search-Guard
    1. 4.1. search-guard-ssl
      1. 4.1.1. 更新openssl
      2. 4.1.2. 安装search-guard-ssl
      3. 4.1.3. 制作HTTPS证书
    2. 4.2. 安装search-guard
    3. 4.3. 配置search-guard
    4. 4.4. search-guard权限设置
      1. 4.4.1. 创建用户及设置密码——sg_internal_users.yml
      2. 4.4.2. 定义角色与用户映射关系——sg_roles_mapping.yml
      3. 4.4.3. 定义角色(用户组)权限——sg_roles.yml
      4. 4.4.4. 定制权限别名——sg_action_groups.yml
    5. 4.5. 启动Elasticsearch集群
    6. 4.6. search-guard初始化
      1. 4.6.1. 设置searchguard索引自动分片
  5. 5. Kibana
    1. 5.1. 安装
    2. 5.2. 配置
    3. 5.3. 访问
  6. 6. Logstash
    1. 6.1. 安装
    2. 6.2. 配置
      1. 6.2.1. Logstash采集Nginx
      2. 6.2.2. 启动
  7. 7. Sentinl
    1. 7.1. 安装
    2. 7.2. search-guard权限设置
    3. 7.3. kibana设置
    4. 7.4. 配置Watch监控
  8. 8. Supervisord
    1. 8.1. 安装
    2. 8.2. 配置
    3. 8.3. 启动

  ELK(Elasticsearch/Logstash/Kibana)是目前对日志采集分析比较适合的解决方案；Kafka是一个分布式高可用高吞吐的消息队列软件，在日志量大及对数据可靠性要求高的场景使用；Search-Guard是一款开源的、对Elasticsearch提供安全及权限控制功能的插件；Sentinl是一款类似Watch的Kibana插件，提供监控、报警和报告功能。
  这整个系统为日志的采集、存储、查询、图表分析和监控报警提供一套完整的解决方案。

软件版本列表

Kafka

安装

1
2
3
4
tar -zxf kafka_2.11-0.10.0.0.tgz
mv kafka_2.11-0.10.0.0 /usr/local/kafka
mkdir /data/zookeeper
chown -R osadmin:osadmin /data/zookeeper

配置

zookeeper配置

  • /usr/local/kafka/config/zookeeper.properties

    1
    2
    3
    4
    5
    6
    7
    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/data/zookeeper
    clientPort=2181
    server.0=10.201.3.33:2888:3888
    server.1=10.201.3.30:2888:3888

  • 设置集群节点myid
     根据配置文件中server.X设置集群节点的myid

    1
    echo X > /data/zookeeper/myid

  • zookeeper调整JVM内存大小
     zookeeper默认的JVM堆内存大小为512M，可视具体情况调整
     /usr/local/kafka/bin/zookeeper-server-start.sh

    1
    export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"

  • 启动zookeeper
     手动启动zookeeper(更好的方式是使用Supervisord来管理)

    1
    nohup /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &> /usr/local/kafka/logs/zookeeper.log &

Kafka配置

  • /usr/local/kafka/config/server.properties

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    #id唯一
    broker.id=0
    port=9092
    advertised.host.name=10.201.3.33
    #host.name唯一
    host.name=10.201.3.33
    num.network.threads=3
    num.io.threads=8
    socket.send.buffer.bytes=102400
    socket.receive.buffer.bytes=102400
    socket.request.max.bytes=104857600
    log.dirs=/usr/local/kafka/logs
    num.partitions=5
    num.recovery.threads.per.data.dir=1
    log.retention.hours=72
    log.segment.bytes=1073741824
    log.retention.check.interval.ms=300000
    zookeeper.connect=10.201.3.33:2181,10.201.3.30:2181
    zookeeper.connection.timeout.ms=6000
    delete.topic.enable=true

  • Kafka配置集群注意事项

    1. echo X > /data/zookeeper/myid
    2. /usr/local/kafka/config/server.properties
      • broker.id
      • host.name

  • Kafka调优项

    • 调整JVM内存大小
       Kafka默认的JVM堆内存大小为1G，如果需要承载较大日志量可视具体情况调整JVM堆内存大小，建议JVM堆的内存大小最好不要超过4G。
       /usr/local/kafka/bin/kafka-server-start.sh

      1
      export KAFKA_HEAP_OPTS="-Xmx3G -Xms3G"

    • 线程数调整
       在需要处理大量日志的场景时可调整num.network.threads和num.io.threads

      1
      2
      num.network.threads=nproc
      num.io.threads=2*nproc

  • 启动Kafka
     手动启动kafka(更好的方式是使用Supervisord来管理)

    1
    nohup /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &> /usr/local/kafka/logs/kafka.log &

kafka常用操作

  • 创建topic

    1
    /usr/local/kafka/bin/kafka-topics.sh --create --zookeeper 10.201.3.30:2181,10.201.3.33:2181 --replication-factor 2 --partitions 2 --topic test

  • 查看topic

    1
    2
    3
    4
    5
    #列出所有topic
    /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 10.201.3.30:2181,10.201.3.33:2181

    #查看具体topic
    /usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 10.201.3.30:2181,10.201.3.33:2181 --topic test

  • 删除topic

    1
    /usr/local/kafka/bin/kafka-topics.sh --delete --zookeeper 10.201.3.30:2181,10.201.3.33:2181 --topic test

  • 查看logstash消费Kafka队列情况

    1
    2
    3
    4
    #列出所有consumer group
    /usr/local/kafka/bin/kafka-consumer-groups.sh --zookeeper 10.201.3.30:2181,10.201.3.33:2181 --list
    #默认所有logstash消费者在logstash的group中
    /usr/local/kafka/bin/kafka-consumer-groups.sh --zookeeper 10.201.3.30:2181,10.201.3.33:2181 --describe --group logstash

  • 动态增加Kafka partition
     将testtopic的partition增加到12个

    1
    /usr/local/kafka/bin/kafka-topics.sh --zookeeper 10.201.3.30:2181,10.201.3.33:2181 --alter --topic test --partitions 12

测试Kafka

  • 生产消息

    1
    /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 10.201.3.33:9092,10.201.3.30:9092 --topic test

  • 消费消息

    1
    /usr/local/kafka/bin/kafka-console-consumer.sh --zookeeper  10.201.3.30:2181,10.201.3.33:2181 --topic test --from-beginning

Elasticsearch

安装

1
2
3
4
tar -zxf elasticsearch-2.4.4.tar.gz
mv elasticsearch-2.4.4 /usr/local/elasticsearch
mkdir /usr/local/elasticsearch/{logs,plugins} /data
chown -R osadmin:osadmin /usr/local/elasticsearch/

配置

文件打开数配置

1
ulimit -n 655350

 /etc/security/limits.conf

1
2
3
4
* soft nofile 655350
* hard nofile 655350
osadmin soft memlock unlimited
osadmin hard memlock unlimited

ES集群配置

 /usr/local/elasticsearch/config/elasticsearch.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
path.data: /data
path.logs: /usr/local/elasticsearch/logs
path.plugins: /usr/local/elasticsearch/plugins
network.host: 0.0.0.0
http.port: 9200
bootstrap.mlockall: true
indices.fielddata.cache.size: 75%
indices.breaker.fielddata.limit: 85%
threadpool.search.queue_size: 10000

#Cluster
cluster.name: elk-cluster
node.name: "10.201.3.49"
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: true
discovery.zen.ping.unicast.hosts: ["10.201.3.49", "10.201.3.33", "10.201.3.30"]
#cluster.routing.allocation.disk.threshold_enabled: false
#cluster.routing.allocation.disk.watermark.low: 90%
#cluster.routing.allocation.disk.watermark.high: 95%

  • path.data：索引数据的存储路径
  • path.logs：日志文件的存储路径
  • path.plugins：插件安装路径
  • network.host：监听IP
  • http.port：监听端口
  • bootstrap.mlockall：锁内存，使ES不使用swap
  • indices.fielddata.cache.size：节点用于 fielddata 的最大内存(达到阀值旧数据将被交换出内存)
  • indices.breaker.fielddata.limit：JVM 堆内存大小(确保 indices.breaker.fielddata.limit 的值大于 indices.fielddata.cache.size 的值)

  • threadpool.search.queue_size：ES搜索队列大小(kibana查询量大时需要增大此值)

  • cluster.name：集群名称(cluster.name相同的节点将自动组成一个集群)

  • node.name：集群节点名称
  • node.master：允许节点成为主节点
  • node.data：允许节点存储数据
  • discovery.zen.ping.multicast.enabled：允许组播发现节点
  • discovery.zen.ping.unicast.hosts：集群初始节点列表(加速发现节点)

ES内存设置

/usr/local/elasticsearch/bin/elasticsearch.in.sh
 内存充足情况尽量分配多内存给Elasticsearch，一般认为64bit机器最大分配内存不超过32G

1
2
ES_MIN_MEM=10g
ES_MAX_MEM=10g

Elasticsearch插件安装

  • head

    1
    2
    3
    4
    5
    cd /usr/local/elasticsearch
    /usr/local/elasticsearch/bin/plugin install mobz/elasticsearch-head

    #访问head插件
    http://10.201.3.49:9200/_plugin/head

  • kopf

    1
    2
    3
    4
    5
    cd /usr/local/elasticsearch
    /usr/local/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/2.1.1

    #访问kopf插件
    http://10.201.3.49:9200/_plugin/kopf

  • curator
      curator用于管理Elasticsearch索引

    • 安装

      1
      pip install elasticsearch-curator==3.5.1

    • 查看索引

      1
      2
      #查看前缀sd-3-centos33-nginx且旧于30天的索引
      curator --timeout 36000 --host localhost show indices --older-than 30 --time-unit days --timestring '%Y.%m.%d' --prefix sd-3-centos33-nginx

    • 关闭索引

      1
      2
      #关闭前缀sd-3-centos33-nginx且旧于30天的索引
      curator --timeout 36000 --host localhost close indices --older-than 30 --time-unit days --timestring '%Y.%m.%d' --prefix sd-3-centos33-nginx

    • 删除索引

      1
      2
      #删除前缀sd-3-centos33-nginx-的所有索引
      curator --timeout 36000 --host localhost delete indices --time-unit days --timestring %Y.%m.%d --prefix sd-3-centos33-nginx-

    • 配合search-guard执行方式
       使用search-guard后所有对Elasticsearch的连接都强制使用HTTPS方式

      1
      curator --http_auth <user>:<password> --use_ssl --timeout 36000 --ssl-no-validate --host localhost show indices --time-unit days --timestring '%Y.%m.%d' --prefix test-nginx

Search-Guard

  Elasticsearch自身并没有认证和权限控制功能，导致安全性问题十分严重。Elasticsearch2.x的shield插件、Elasticsearch5.x的x-pack插件弥补了这个安全问题，但这两个插件都是需要付费才能长期使用的，search-guard是开源免费并能提供权限控制的插件。使用search-guard最好JDK的版本在1.8以上。

search-guard-ssl

  search-guard2.x版本需要依赖search-guard-ssl，search-guard-ssl需要使用openssl 1.0.1k以上版本。

更新openssl

  只需要要在使用search-guard-ssl生成证书的机器上更新openssl即可。

1
2
3
4
5
6
7
8
9
10
11
12
wget http://www.openssl.org/source/openssl-1.0.1k.tar.gz
tar -zxf openssl-1.0.1k.tar.gz && cd openssl-1.0.1k
./config shared zlib
make && make install
mv /usr/bin/openssl /usr/bin/openssl.old && mv /usr/include/openssl /usr/include/openssl.old
ln -s /usr/local/ssl/bin/openssl /usr/bin/openssl
ln -s /usr/local/ssl/include/openssl /usr/include/openssl
echo "/usr/local/ssl/lib" >> /etc/ld.so.conf
ldconfig -v

#查看openssl版本
openssl version

安装search-guard-ssl

1
2
cd /usr/local/elasticsearch
bin/plugin install -b com.floragunn/search-guard-ssl/2.4.4.19

制作HTTPS证书

  search-guard强制只能使用HTTPS方式访问Elasticsearch，所以需要使用search-guard-ssl制作HTTPS的相关证书。

1
2
git clone https://github.com/floragunncom/search-guard-ssl.git
cd search-guard-ssl/example-pki-scripts/

  • 制作HTTPS证书脚本
      search-guard-ssl提供制作证书相关的脚本

    • gen_client_node_cert.sh
      制作客户端证书
    • gen_node_cert.sh
      创建节点证书
    • gen_root_ca.sh
      创建根证书

  • 修改证书信息
      根据自身情况修改gen_client_node_cert.sh、gen_node_cert.sh证书相关的dname信息(不修改dname相关信息也无碍)

    • CN: 公用名称
    • OU: 组织单位名称
    • O: 组织名称
    • L: 城市名称
    • S: 省份名称

    • C: 国家名称

    • gen_client_node_cert.sh

      1
      -dname "CN=$CLIENT_NAME, OU=client, O=client, L=SZ, C=CN"

    • gen_node_cert.sh

      1
      -dname "CN=$NODE_NAME, OU=test, O=test, L=SZ, C=CN"

  • 修改example.sh脚本相关密码
      example.sh是search-guard-ssl提供的样例脚本，把CA、TrustStore和KeyStore改成自己设置的密码

    1
    2
    3
    4
    5
    6
    #!/bin/bash
    set -e
    ./clean.sh
    ./gen_root_ca.sh <your_CA_password> <your_TrustStore_password>
    ./gen_node_cert.sh <your_node_name> <your_KeyStore_password> <your_CA_password>
    ./gen_client_node_cert.sh admin <your_KeyStore_password> <your_CA_password>

  • 拷贝证书
      将生成的证书拷贝到所有节点相应的目录下。search-guard官方推荐每个ES节点的节点证书不同，这里所有节点都是使用的同一个节点证书。

    1
    2
    cp truststore.jks node-<your_node_name>-keystore.jks /usr/local/elasticsearch/config/
    cp truststore.jks admin-keystore.jks /usr/local/elasticsearch/plugins/search-guard-2/sgconfig/

安装search-guard

1
2
cd /usr/local/elasticsearch
bin/plugin install -b com.floragunn/search-guard-2/2.4.4.10

配置search-guard

  search-guard需要在Elasticsearch配置文件/usr/local/elasticsearch/config/elasticsearch.yml中新增配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#search-guard
searchguard.authcz.admin_dn:
  - CN=admin,OU=client, O=client, L=SZ, C=CN

#search-guard ssl
searchguard.ssl.transport.keystore_filepath: node-<your_node_name>-keystore.jks
searchguard.ssl.transport.keystore_password: your_KeyStore_password
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: your_TrustStore_password
searchguard.ssl.transport.enforce_hostname_verification: false
#search-guard https
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node-<your_node_name>-keystore.jks
searchguard.ssl.http.keystore_password: your_KeyStore_password
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: your_TrustStore_password

search-guard权限设置

  search-guard的权限配置文件都在/usr/local/elasticsearch/plugins/search-guard-2/sgconfig目录下

  • sg_config.yml
    主配置文件，定义认证类型等。一般不需要改动
  • sg_internal_users.yml
    本地用户文件，定义用户密码以及对应的权限(密码生成脚本：plugins/search-guard-2/tools/hash.sh)
  • sg_roles_mapping.yml
    定义角色(sg_roles.yml)和用户(sg_internal_users.yml)的映射关系
  • sg_roles.yml
    角色权限配置文件
  • sg_action_groups.yml
    定义权限别名，把多个单独的权限整合并配置别名，简化配置

创建用户及设置密码——sg_internal_users.yml

  创建具体的用户和密码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#管理员权限帐号
admin:
  hash: $2a$12$rVW6Elg3PUBTJIAXRZ881.kWfG4OL/vTwX0ksJ8uUUGEC763J08SK

#kibana server帐号
#用于在kibana.yml中配置，该帐号用于创建和管理.kibana索引
kibana_server:
  hash: $2a$12$QrVeCyf7JFwq/2.2BjLP0O0g8C1hlHunoyxWFe57nlbJGHI/A/Wda

#kibana 登录帐号
kibana_admin:
  hash: $2a$12$.vAGeZm8FdFMIeDz8DlWqeWCO/Uiy12v6xJEbpGnhahSfSuPwbYOy

#logstash帐号
#用于在logstash中配置，允许logstash往Elasticsearch中写入数据
logstash:
  hash: $2a$12$zbVAUMH5thQvnCDKNfMLv.QDsdFYdoiK3V70.tkC8tMVF8EvP0nf2

定义角色与用户映射关系——sg_roles_mapping.yml

  search-guard中的角色其实相当于用户组的概念，对角色(用户组)进行权限控制，用户加入角色(用户组)中，以此来管理权限。
  用sg_admin角色(用户组)为例，sg_admin为管理员角色(用户组)，该角色(用户组)下有名为admin的用户。所以admin用户就拥有了管理员角色(组)的权限

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#管理员角色
sg_admin:
  users:
    - admin

#kibana server帐号
sg_kibana_server:
  users:
    - kibana_server

#kibana 登录帐号
sg_kibana_admin:
  users:
    - kibana_admin

#logstash帐号
sg_logstash:
  users:
    - logstash

定义角色(用户组)权限——sg_roles.yml

  定义不同角色(用户组)的角色有不同的权限

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
#管理员角色
#拥有所有权限
sg_admin:
  cluster:
    - '*'
  indices:
    '*':
      '*':
        - '*'

#kibana server角色
#对.kibana索引有所有权限，并且对集群有获取节点信息和健康状态的权限
sg_kibana_server:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '?kibana':
      '*':
        - ALL

#kibana登录角色
sg_kibana_admin:
  indices:
    '_all':
      '*':
        - indices:data/read/mget*
        - indices:data/read/get*
        - indices:data/read/search*
        - indices:data/read/msearch*
    'logstash-*':
      '*':
        - ALL
    '?kibana':
      '*':
        - ALL

#logstash角色
#对Elasticsearch有创建索引和写入的权限
sg_logstash:
  cluster:
    - indices:admin/template/get
    - indices:admin/template/put
  indices:
    '*':
      '*':
        - WRITE
        - CREATE_INDEX
        - indices:data/read/search
        - indices:data/read/scroll

定制权限别名——sg_action_groups.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
ALL:
  - "indices:*"
CREATE_INDEX:
  - "indices:admin/create"
WRITE:
  - "indices:data/write*"
READ:
  - "indices:data/read*"
DELETE:
  - "indices:data/write/delete*"
CRUD:
  - READ
  - WRITE

启动Elasticsearch集群

  将证书都拷贝到集群所有节点，ES配置文件都新增相关配置后就可启动集群

1
/usr/local/elasticsearch/bin/elasticsearch -d

search-guard初始化

  Elasticsearch所有节点启动完成后需要对search-guard进行初始化，生成searchguard索引。search-guard根据权限配置文件将数据写入到searchguard索引中，当权限设置变更时，只需要重新初始化searchguard索引即可，不需要重启集群。

1
2
cd /usr/local/elasticsearch/
plugins/search-guard-2/tools/sgadmin.sh -cd plugins/search-guard-2/sgconfig/ -ks plugins/search-guard-2/sgconfig/admin-keystore.jks -ts plugins/search-guard-2/sgconfig/truststore.jks -tspass <your_TrustStore_password> -kspass <your_KeyStore_password> -icl -nhnv

设置searchguard索引自动分片

  初始化后生成的searchguard索引是不会随着集群节点增加而自动增加分片的，需要设置searchguard索引成自动分片。

1
2
cd /usr/local/elasticsearch/
plugins/search-guard-2/tools/sgadmin.sh -cd plugins/search-guard-2/sgconfig/ -ks plugins/search-guard-2/sgconfig/admin-keystore.jks -ts plugins/search-guard-2/sgconfig/truststore.jks -tspass <your_TrustStore_password> -kspass <your_KeyStore_password> -icl -era

Kibana

安装

1
2
3
4
tar -zxf kibana-4.6.4-linux-x86_64.tar.gz
mv kibana-4.6.4-linux-x86_64 /usr/local/kibana
useradd kibana
chown -R kibana:kibana /usr/local/kibana/

配置

/usr/local/kibana/config/kibana.yml

1
2
3
4
5
6
7
8
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "https://localhost:9200"
kibana.index: ".kibana"
elasticsearch.username: "kibana_server"
elasticsearch.password: "kibana"
elasticsearch.ssl.ca: /usr/local/kibana/root-ca.pem
elasticsearch.ssl.verify: false

访问

http://kibana_server_ip:5601
输入不同用户及密码会根据用户做权限的控制

Logstash

安装

1
2
3
4
tar -zxf logstash-2.4.1.tar.gz
mv logstash-2.4.1 /usr/local/logstash
chown -R osadmin:osadmin /usr/local/logstash/
mkdir /usr/local/logstash/config

配置

Logstash使用grok正则表达式对日志进行匹配并json化后存入Elasticsearch
grok正则在线测试

Logstash采集Nginx

  • 客户端配置
     采集的客户端用的是filebeat，直接通过rpm安装。
     /etc/filebeat/filebeat.yml

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    filebeat.prospectors:
    - input_type: log
      paths:
        - /usr/local/nginx/logs/access.log
      #multiline:
      #  pattern: '^(20[0-9]{2}(-[0-9]{2}){2} [0-9]{2}(:[0-9]{2}){2})'
      #  negate: true
      #  match: after

    output.kafka:
      hosts: ["10.201.5.30:9092", "10.201.5.31:9092"]
      topic: 'nginx'
      partition.round_robin:
        reachable_only: false

      required_acks: 1
      compression: gzip
      max_message_bytes: 1000000

  • 服务端配置
    /usr/local/logstash/config/logstash.conf
    root-ca.pem是使用search-guard-ssl生成的证书原路径——search-guard-ssl/example-pki-scripts/ca/root-ca.pem

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    input {
    	kafka {
    		zk_connect => "10.201.3.33:2181,10.201.3.30:2181"
    		topic_id => "nginx"
    		codec => json
    		reset_beginning => false
    		consumer_threads => 5
    		decorate_events => true
    	}
    }

    filter {
    	grok {
    		patterns_dir => ["/usr/local/logstash/patterns"]
    		match => ["message", "%{NGINXACCESS}"]
    		overwrite => ["message"]
    	}
    	geoip {
    		source => "xforward"
    		target => "geoip"
    		database => "/usr/local/logstash/maps/GeoLiteCity.dat"
    	}
    }

       output {
          elasticsearch {
             user => logstash
             password => logstash
             ssl => true
             ssl_certificate_verification => false
             cacert => "/usr/local/logstash/config/root-ca.pem"
             hosts => ["10.201.3.49:9200","10.201.3.33:9200","10.201.3.30:9200"]
             index => "nginx-%{+YYYY.MM.dd}"
          }
       }

  • grok正则
    /usr/local/logstash/patterns/nginx

    1
    NGINXACCESS %{IP:ip} \- \- \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:uri} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} \"(?:-|%{IP:xforward}[%{IP}\, ]*)\" "%{NUMBER:request_time:float} %{NUMBER:request_length} %{NUMBER:connection_requests}" %{QS:other}

    Nginx日志格式

    1
    2
    3
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                         '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for" "$request_time $request_length $connection_requests" "$http_syncsession| $http_sessionkey| $http_cookie| $http_accept| $http_content_length| $http_x_forwarded_proto"';

启动

测试Logstash配置文件是否正确

1
/usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf --configtest --verbose

启动Logstash(尽量使用Supervisord管理)

1
/usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstash.conf

Sentinl

  Sentinl是一个开源的Kibana插件，用于监控报警和报告，可弥补ELK缺乏报警的功能。Sentinl项目地址

安装

1
2
cd /usr/local/kibana/
bin/kibana plugin --install sentinl -u https://github.com/sirensolutions/sentinl/releases/download/snapshot/sentinl-latest.tar.gz

search-guard权限设置

  Sentinl需要查询Elasticsearch的数据，所以需要给kibana.yml中的用户相应的权限
  sg_roles.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
sg_kibana_server:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '?kibana':
      '*':
        - ALL
    'watcher*':
      '*':
        - ALL
    '*':
      '*':
        - indices:data/read/search*

kibana设置

  需要在kibana.yml中设置发邮帐号和监控相对应的Elasticsearch索引。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
sentinl:
 es:
   timefield: '@timestamp'
   default_index: watcher
   type: watch
   alarm_index: watcher_alarms
 sentinl:
   history: 20
   results: 50
 settings:
   email:
     active: true
     user: test@163.com
     password: <password>
     host: smtp.163.com
     ssl: true
   report:
     active: true
     tmp_path: /tmp/

配置Watch监控

  登录Kibana后进入Sentinl点击Watch创建监控规则。规则中的email from需要和kibana.yml中邮箱的user一致。
  下面这个示例是5分钟内nginx-索引日志中statushttp状态码为502的记录出现超过3次则报警

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
{
  "_index": "watcher",
  "_type": "watch",
  "_id": "ops_1",
  "_score": 1,
  "_source": {
    "disable": false,
    "uuid": "ops_1",
    "trigger": {
      "schedule": {
        "later": "every 5 minutes"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "<nginx-{now/d}>"
          ],
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "must": {
                  "match": {
                    "status": "502"
                  }
                },
                "filter": [
                  {
                    "range": {
                      "@timestamp": {
                        "gt": "now-5m"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 3"
      }
    },
    "transform": {},
    "actions": {
      "email_admin": {
        "throttle_period": "5m",
        "email": {
          "to": "admin@gmail.com",
          "from": "test@163.com",
          "subject": "Sentinl Alarm",
          "priority": "high",
          "body": "Found {{payload.hits.total}} 502 Events"
        }
      }
    }
  }
}

Supervisord

  使用Supervisord来管理Logstash、Zookeeper和Kafka的启动/停止

安装

1
2
pip install -U setuptools
pip install supervisor

配置

生成模板配置文件

1
echo_supervisord_conf > /etc/supervisord.conf

启动

  • 启动Supervisor

    1
    supervisord

  • supervisorctl
    通过supervisorctl控制程序的启动，也可以通过Web界面管理http://supervisor_server_ip:9001

    1
    supervisorctl  [start|stop|restart|reread|update] program_name
ELK
← Prev Next →

Powered: Hexo, Theme: Nadya remastered from NadyMain