elk stack logstash

info

1. logstash

2. downloads

3. documentation

   1. 2.3(current)

   2. 5.0

4. install steps

   1. downloads

       #### AAA install with apt-get
       # 1. import pgp key
       # download and install the public signing key
       $ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
      
       # 2. add the repository definition
       $ echo "deb https://packages.elastic.co/logstash/2.3/debian stable main" | \
       sudo tee -a /etc/apt/sources.list
      
       # 2.3 install
       $ sudo apt-get update && sudo apt-get install logstash
      
       #### BBB install from tar.gz
       $ curl -O https://download.elastic.co/logstash/logstash/logstash-5.0.0-alpha3.tar.gz
       $ tar zxvf logstash-5.0.0-alpha3.tar.gz
      

   2. prepare a logstash.conf config file

   3. run

       $ bin/logstash agent -f logstash.conf
      

5. config

   1. config file json format

       $ cat demo.conf
       # this is a comment
       input {
           ...
       }
      
       filter {
           ...
       }
      
       output {
           ...
       }
      

docker-logstash

1. git repo

processing pipeline

1. input plugins

   1. file

   2. syslog

   3. redis

   4. beats

2. filter plugins

   1. grok

       parse and structure arbitrary text
      

   2. mutate

       transform: rename remove replace modify
      

   3. drop

   4. clone

   5. geoip

3. output plugins

   1. elasticsearch

   2. file

   3. graphite

   4. statsd

4. codec plugins

   1. json

   2. multiline

demo

1. basic logstash pipeline

   1. test installation

       $ cd logstash-2.3.0
       $ bin/logstash -e 'input { stdin { } } output{ stdout{ } }'
      

   2. -e flag

        specify a configuration from command line
      

2. rubydebug

        $ bin/logstash -e 'input { stdin { } } output{ stdout { codec => rubydebug } }'
        hello
   

3. elasticsearch output

   1. command

       # if connect to another container
       # change `elasticsearch { host = localhost }`
       #     to `elasticsearch { hosts => ["192.168.99.100:9200"] }`
       $ bin/logstash -e 'input { stdin { } } output{ elasticsearch {  } }'
      

   2. check indexes

       $ curl http://localhost:9200/_search?pretty
       {
           "name" : "Ellie Phimster",
           "cluster_name" : "elasticsearch",
           "version" : {
               "number" : "2.3.3",
               "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
               "build_timestamp" : "2016-05-17T15:40:04Z",
               "build_snapshot" : false,
               "lucene_version" : "5.5.0"
           },
           "tagline" : "You Know, for Search"
       }