Logstash  Family  Introduc4on

Owen

What  is  a  log

•  Oxford  Dic4onary  – a  thick  piece  of  wood  that  is  cut  from  or  has  fallen  from  a  tree  

–  (also  logbook)  an  official  record  of  events  during  a  par4cular  period  of  4me,  especially  a  journey  on  a  ship    

•  4me  +  data        

In  theory,  life  cycle  of  log

Record

Transmit

Analyze Store

Delete

In  design,  life  cycle  of  log

Record

Transmit

Store

Delete

In  fact,  life  cycle  of  log

Record

Delete

Problems

•  Logging  to  a  database  or  filesysytem  •  Logging  has  placed  a  load  on  the  database  and  filesystem  

•  Mul4ple  log    formats  •  No  easy  way  to  search  logs  •  No  easy  method  to  gather  sta4s4cs  

Find  the  logs  of  16  computers  6  months  ago  ?

Why  use  Logstash?

•  A lot choices! �•  But we want a free & high-integrality &

easy to use solution •  splunk (finding your faults, just like mom) •  facebookarchive/scribe (2682 ★) •  Graylog2(Server+WUI 1683 ★) •  fluentd (2038 ★) •  logstash (2689 ★)

logstash  and  other  things

hRps://www.youtube.com/watch?v=RuUFnog29M4

Logstash

•  Open  Source,  Apache  Licence  •  WriRen  in  JRuby,  Runs  on  JVM  •  Plugins  easily  wriRen  in  Ruby  •  Process  mul4ple  format  (  input,  output  )  •  Logstash  Family!  (  Elas4cSearch  ,  Kibana)  

LogStash  Family  architecture

Elas4cSearch

•  A  response  to  the  claim  :  “Search  is  hard”  •  Powerful  indexing  &  search  tool  •  search  &  index  data  available  Rescully  as  JSON  over  HTTP  

Kibana

All-­‐in-­‐one!

How logstash works?

•  logstash process events, not (only) loglines!

•  “The logstash agent is a processing pipeline with 3 stages: –  inputs -> filters -> outputs.” – separate threads

•  “Inputs generate events, filters modify them, outputs ship them elsewhere.”

•  -- [the life of an event in logstash] �

In  my  thinking,  Event  Life  Cycle

Input

filter    output

In  fact,  Event  Life  Cycle

event  (Input  -­‐>  output)

event  -­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐  input  filter    output

Logstash  is  a  wooden  tube

Input

Input

Input

filter

output

codec

filter filter

output

output

Logstash  plugins  Workflow  

•  inputs  – How  events  get  into  LogStash.  

•  codecs    –   convert  an  incoming  format  into  an  internal    representa4on  

•  filters  –   processing  ac4ons  on  events  :  modify  events  or  drop  events  

•  outputs  – How  output  events  from  LogStash

Logstash  plugins

What  is  an  event!?

•  A  @4mestamp  (ISO  8601  4mestamp)  •  A  messsage  field  (  data  )  •  A  @version  •  host  (  the  host  of  sender)  •  type(  syslog,  irc,  etc)  

Exersice:  Hello  Word!

java  -­‐jar  logstash-­‐1.1.12-­‐flatjar.jar  agent  -­‐f  hello.conf

java  -­‐jar  logstash.jar  agent  -­‐f  hello.conf

Input  

•  tcp  •  udp  •  unix  •  file  •  syslog  •  redis  •  logstash-­‐fowarder(former  Lumberjack)

Codecs

•  plain  •  json  •  rubydebug  •  mul4line

Outputs  

•  mongodb  •  elas4cSearch  •  email  •  file  •  jira  

Exercise:  Mul4ple  input  &  output

logstash-­‐forwader

•  ♫  I'm  a  lumberjack  and  I'm  ok!  I  sleep  when  idle,  then  I  ship  logs  all  day!  I  parse  your  logs,  I  eat  the  JVM  agent  for  lunch!  ♫  

•  WriRen  in  Go  •  lumberjack  is  reserved  for  protocol  •  Resource  Usage  Concerns  •  Need  an  SSL  CA  to  verify  the  server  

lumberjack

•  Encryp4on  &  Authen4ca4on  (TLS)  •  Compression  (  reduce  bandwidth)  •  Sequence  &  ack  behavior  like  TCP  •  Low  latency  •  Reliable  Aplica4on-­‐Level  message  transport

Forwarder  Sample

Filters  

•  date  •  grok  •  drop  •  geoIP  •  mutate    •  mul4line  

Exercise:  Parse  Data

filter  config  

powerful  grok •  Parse  arbitrary  text  and  structure  it.  •  The  syntax  for  a  grok  paRern  is      – %{SYNTAX:SEMANTIC}  

•  55.3.244.1        GET          /index.html          15824  – %{IP:client}    – %{WORD:method}    – %{URIPATHPARAM:request}    – %{NUMBER:bytes}  

•  hRps://github.com/elas4csearch/logstash/blob/v1.4.2/paRerns/grok-­‐paRerns  

grok  sample  

drop

mutate  

•  Muta4ons  on  fields.    –   rename  –   remove  –  replace  –  join  – split  – upper  –  lower

mul4line

•  Codecs  &  filter

Reference

•  hRps://www.digitalocean.com/community/tutorials/how-­‐to-­‐use-­‐logstash-­‐and-­‐kibana-­‐to-­‐centralize-­‐and-­‐visualize-­‐logs-­‐on-­‐ubuntu-­‐14-­‐04  

•  hRp://www.vmdoh.com/blog/centralizing-­‐logs-­‐lumberjack-­‐logstash-­‐and-­‐elas4csearch  

•  hRp://jpmens.net/2012/08/09/i-­‐grok-­‐how-­‐to-­‐mutate-­‐a-­‐file-­‐with-­‐logstash/  

•  hRp://gleenders.blogspot.tw/2014/02/logstash-­‐glassfish.html