elasticsearch + fluentd + kibana4(EFK)

elasticsearch 安裝

tar xfvz elasticsearch-2.3.2.tar.gz
mv elasticsearch-2.3.2 /usr/local/
elasticsearch -Des.insecure.allow.root=true
nohup /usr/local/elasticsearch-2.3.2/bin/elasticsearch -Des.insecure.allow.root=true &
curl 'localhost:9200/_cat/indices?v'

test >curl -XGET localhost:9200/

curl http://localhost:9200/_aliases?pretty=1

設定
vi /usr/local/elasticsearch-2.3.2/config/elasticsearch.yml

fluentd 安裝

 rpm -ivh td-agent-2.3.2-0.el6.x86_64.rpm

安裝 plugin
1.https://github.com/uken/fluent-plugin-elasticsearch
2.https://github.com/shivaken/fluent-plugin-better-timestamp


手動到 rubygems.org 抓取
或直接執行
$ sudo /usr/sbin/td-agent-gem install fluent-plugin-secure-forward $ sudo /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch

sudo /usr/sbin/td-agent-gem 'fluent-plugin-elasticsearch', '~> 1.4'

gem install --local your.gem
$ sudo /usr/sbin/td-agent-gem install --local /home/henry/fluent-plugin-secure-forward-0.3.2
$ sudo /usr/sbin/td-agent-gem install --local /home/henry/fluent-plugin-elasticsearch-1.4.0

把gem copy 到 plugin 資料夾
cp /home/henry/*.gem /etc/td-agent/plugin/

強制執行安裝 本機gem
sudo /usr/sbin/td-agent-gem install --force --local *.gem

設定檔位置

sudo vi /etc/td-agent/td-agent.conf

/opt/td-agent/

$ cd /opt/td-agent/embedded/lib/ruby/gems/2.1.0/bin/
$ sudo ./secure-forward-ca-generate /opt/td-agent/ passphrase_for_private_CA_secret_key

touch /var/log/bps_log/fluentd.bps.pos

啟動

 sudo service td-agent start
 sudo service td-agent stop

vim /var/log/td-agent/td-agent.log tail -f /var/log/td-agent/td-agent.log

kibana


tar xfvz kibana-4.5.3-linux-x64.tar.gz
cp kibana-4.5.3-linux-x64 /usr/local
run kibana
sh /usr/local/kibana-4.5.3-linux-x64/bin/kibana
http://yourhost.com:5601




  type tdlog
  apikey YOUR_API_KEY

  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td

 
    type file
    path /var/log/td-agent/failed_records
 



  @type elasticsearch
  host 112.121.96.159
  port 9200
  index_name bps
  type_name fluentd



  @type elasticsearch
  host 112.121.96.159
  port 9200
  index_name bps
  type_name fluentd



 type tail
 format multiline
 format_firstline /^(\[\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}.\d{3}\])/
 format1 /^(?\[\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}.\d{3}\]) \[(?.*)\] \[(?.....)\] \$\[(?.*?)\]\$/
 path /var/log/bps_log/bps_SIT.log
 pos_file /var/log/bps_log/fluentd.bps.pos
 read_from_head true
 tag bps_log
 refresh_interval 30



 type tail
 format multiline
 format_firstline /^(\[\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}.\d{3}\])/
 format1 /^(?\[\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}.\d{3}\]) \[(?.*)\] \[(?.....)\] \$\[(?.*?)\]\$/
 path /var/log/bps_log/bps_SIT_NCCC.log
 pos_file /var/log/bps_log/fluentd.bps_NCCC.pos
 read_from_head true
 tag bps_NCCC_log
 refresh_interval 30


  @type record_transformer
  enable_ruby
 
    timestamp ${ require 'time'; Time.now.utc.iso8601(3) }