墨痕

2015-12-28 OPS

EFK日志系统——Fluentd代替Logstash & GeoIP世界地图

1. 安装
2. Fluentd配置
1. 2.1. Fluentd Aggregator
2. 2.2. Fluentd Collector
3. GeoIP
4. 展示

安装

Elasticsearch、Kibana详见Elasticsearch、Kibana安装
Fluentd详见Fluentd安装

Fluentd插件

fluent-plugin-secure_forward、fluent-plugin-elasticsearch
secure_forward插件可选，也可用内置的forward插件
1
$> td-agent-gem install fluent-plugin-secure-forward fluent-plugin-elasticsearch

fluent-plugin-geoip
此插件也可选，如果不需要统计IP位置，Kibana不需要地图可忽略。

#若没安装EPEL，则需事先安装
#$> rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
#$> rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm

#若安装EPEL后yum报错Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again。须更新ca
#yum --disablerepo="epel" -y update ca-certificates

$> yum install geoip-devel
$> td-agent-gem install fluent-plugin-geoip

Fluentd配置

如果之前对Fluentd没有了解，可先看看此篇博客文章有个大概的认识。
Fluentd Collector(A)收集日志转发到Fluentd Aggregator(B)，再通过插件fluent-plugin-elasticsearch存入Elasticsearch

1	access.log(A)——>Fluentd-tail(A)——>Fluentd-secure_forward(A)——>Fluentd-secure_forward(B)——>Fluentd-elasticsearch(B)——>Elasticsearch——>Kibana

Fluentd Aggregator

日志汇聚端配置/etc/td-agent/td-agent.conf
若传输需要加密，则需要先生成证书和key，详见此博客文章
auth验证也是可选的

<source>
  type secure_forward
  port 22222
  shared_key secret_string
  self_hostname logs_server.fluent

  secure false
  #若传输内容需要加密则用此配置
  #secure true
  #ca_cert_path        /etc/td-agent/ca/ca_cert.pem
  #ca_private_key_path /etc/td-agent/ca/ca_key.pem
  #ca_private_key_passphrase  passphrase_for_private_CA_secret_key

  authentication yes
  <user>
    username your_usrname
    password your_password
  </user>
</source>


<match efk_test.**>
  type elasticsearch
  host localhost
  port 9200
  index_name fluentd
  include_tag_key true
  tag_key @log_name
  type_name nginx
  logstash_format true
  logstash_prefix node1   	#Elasticsearch存储indices的前缀(必须小写)，默认值为：logstash
  utc_index false
  flush_interval 10s
</match>

Fluentd Collector

日志收集客户端配置/etc/td-agent/td-agent.conf
若传输内容需要加密，则需从汇聚端将证书拷贝到客户端
auth验证可选，若汇聚端没有设置则不需要

<source>
    type          tail
    tag           efk_test
    path          /var/wwwlog/efk_test/access.log
    pos_file      /tmp/efk_test.fluentd.pos
    format        /^(?<ip>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<forward>[^ ]*) (?<reqlen>[^ ]*) (?<reqtime>[^ ]*) (?<backend>[^ ]*) (?<domain>[^ ]*)/
    time_format   %d/%b/%Y:%H:%M:%S %z
</source>


<match efk_test.**>
  type secure_forward
  shared_key secret_string
  self_hostname node1.fluentd

  secure false
  #若传输内容需要加密则用此配置
  #secure true
  #ca_cert_path        /etc/td-agent/ca/ca_cert.pem

  <server>
    host logs_server.fluentd
    port 22222
    username your_usrname
    password your_password
  </server>
</match>

GeoIP

通过GeoIP库查询地理位置，并通过Kibana地图展示

1	access.log(A)——>Fluentd-tail(A)——>Fluentd-secure_forward(A)——>Fluentd-secure_forward(B)——>Fluentd-geoip(B)——>Fluentd-elasticsearch(B)——>Elasticsearch——>Kibana

Elasticsearch经纬格式配置

在配置Fluentd之前必须做此步骤!否则Kibana找不到geo_point类型字段无法在世界地图上展示。
fluent-plugin-geoip插件在存储经纬字段时使用的是string类型(Fluentd默认存入Elasticsearch的字段类型都是string)，但Kibana规定必须使用geo_point类型字段。所以在Elasticsearch存储时必须将经纬字段设置成geo_point。最简单的办法是通过Elasticsearch的模板映射来解决。

添加Elasticsearch模板

curl -X PUT http://localhost:9200/_template/geoip_type_template -d'{
  "template": "*",
  "mappings": {
    "geoip_nginx": {
      "properties": {
        "geoip_location": {
          "type": "geo_point"
        }
      }
    }
  }
}'

添加名为geoip_type_template的模板，模板的应用对象为所有已存在的模板*，对存在type_name为geoip_nginx的数据，将字段geoip_location的类型设置为geo_point
若Elasticsearch生成indices，可调用Elasticsearch API查看mapping映射类型

1	curl -XGET localhost:9200/indices_name/_mapping

Fluentd Aggregator配置

免费版GeoIP库下载地址

<source>
  type secure_forward
  port 22222
  shared_key secret_string
  self_hostname logs_server.fluent

  secure false
  #若传输内容需要加密则用此配置
  #secure true
  #ca_cert_path        /etc/td-agent/ca/ca_cert.pem
  #ca_private_key_path /etc/td-agent/ca/ca_key.pem
  #ca_private_key_passphrase  passphrase_for_private_CA_secret_key

  authentication yes
  <user>
    username your_usrname
    password your_password
  </user>
</source>


<match geoip.**>
  type geoip
  geoip_lookup_key  ip   	#设置ip地址所在的key值
  geoip_database    "/etc/td-agent/geoipdb/GeoLiteCity.dat" 	#指定geoip库

  <record>
    country  ${country_code['ip']}
    geoip_location	${latitude['ip']},${longitude['ip']}		#geo_point类型字段
  </record>

  remove_tag_prefix    geoip.
  tag   es.${tag}

  skip_adding_null_record   true	#跳过IP库查找不到的记录。false则会在查找不到的记录中添加'{country:null, geoip_location null}'
  log_level info
  flush_interval   5s
</match>


<match es.**>
  type elasticsearch
  host localhost
  port 9200
  index_name fluentd
  include_tag_key true
  tag_key @log_name
  logstash_format true
  type_name geoip_nginx		#geoip_type_template模板根据此type_name匹配
  logstash_prefix node1   	#Elasticsearch存储indices的前缀(必须小写)，默认值为：logstash
  utc_index false
  flush_interval 10s
</match>

Fluentd Collector配置

<source>
    type          tail
    tag           geoip
    path          /var/wwwlog/efk_test/access.log
    pos_file      /tmp/efk_test.fluentd.pos
    format        /^(?<ip>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<forward>[^ ]*) (?<reqlen>[^ ]*) (?<reqtime>[^ ]*) (?<backend>[^ ]*) (?<domain>[^ ]*)/
    time_format   %d/%b/%Y:%H:%M:%S %z
</source>


<match geoip.**>
  type secure_forward
  shared_key secret_string
  self_hostname node1.fluentd

  secure false
  #若传输内容需要加密则用此配置
  #secure true
  #ca_cert_path        /etc/td-agent/ca/ca_cert.pem

  <server>
    host logs_server.fluentd
    port 22222
    username your_usrname
    password your_password
  </server>
</match>

展示

图为某接口的日志统计情况，目前只有初步统计图，更详细的数据图有待添加。

EFK/ELK