Joel Corrêa Blog

Logstash

25 Aug 2014

Version checked 1.4.2

A tool for managing events and logs.

You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.

Highlights

Logs Events BigData Filter

Trade-off

Extensible SOC ELK stack integration JRuby Poor documentation

Features

  • Transform, and process logs to and from anywhere
  • Provide search and analytics

Inputs, Filters and Outputs (Plugins)

  • Input: Stream source
  • Filter: Extracts what you are interested from the stream.
  • Output: Sends the extracted stuff somewhere.
  • Codec: Decode (via inputs) and Encode (via outputs) a message

Examples

Simplest configuration example

input { stdin {} }
output { stdout {} }

> ./logstash agent -f my.conf
> 1
2014-08-26T02:42:21.416+0000 joelcorrea.local 1

Using codecs

input { stdin {} }

output {
  stdout { codec => json }
}

> 1
{"message":"1","@version":"1","@timestamp":"2014-08-26T02:44:29.809Z","host":"joelcorrea.local"}

Grok: A collection of pre-defined patterns

Have a look on them (You can debug it)

input {
  stdin {}
}
filter {
  grok { 
  	match => [ "message", "name: %{WORD:custom_name}" ] 
  }
  mutate { lowercase => [ "custom_name" ]
}
output {
    stdout{codec => json}
}

> name: Joel
{"message":"name: Joel","@version":"1","@timestamp":"2014-08-26T02:56:57.461Z","host":"joelcorrea.local","custom_name":"joel"}

References

Related Posts